16-825 Assignment 5: Point Cloud Classification and Segmentation

Farnaz Alam Ahmed

Q1. Classification Model

Model. A vanilla PointNet (no T-Nets). A shared per-point MLP (Conv1d with kernel size 1: 3→64→64→64→128→1024, each followed by BatchNorm and ReLU) maps every point to a 1024-D feature. A max-pool over the points gives one order-invariant global feature per object, and an MLP head (1024→512→256→3, BatchNorm, ReLU, dropout 0.3) outputs the class scores. Trained for 50 epochs with Adam (lr 0.001, batch 32) on all 10,000 points per object.

Test accuracy: 97.17% (926 / 953 test objects correct; best checkpoint at epoch 25).

Random test samples

Five random test objects (fixed seed) with their predicted class. All five are classified correctly.

Test object 605
#605 · true: chair · pred: chair
Test object 486
#486 · true: chair · pred: chair
Test object 256
#256 · true: chair · pred: chair
Test object 293
#293 · true: chair · pred: chair
Test object 807
#807 · true: lamp · pred: lamp

Failure cases

True classMisclassifiedPredicted as
Chair0 / 617–
Vase9 / 102lamp ×7, chair ×2
Lamp18 / 234vase ×17, chair ×1
Vase 619 predicted as lamp
#619 · true: vase · pred: lamp
Lamp 726 predicted as vase
#726 · true: lamp · pred: vase
Chair 35 predicted as lamp with 100 points
#35 (100 points, from Q3) · true: chair · pred: lamp

Interpretation.

Q2. Segmentation Model

Model. PointNet segmentation network. A local MLP (3→64→64) gives a 64-D per-point feature. A second MLP (64→64→128→1024) followed by a max-pool gives a 1024-D global feature for each chair. The global feature is copied to every point and concatenated with that point's local feature (64 + 1024 = 1088), and a per-point head (1088→512→256→128→6) outputs scores for the 6 part classes. Trained for 30 epochs (Adam, lr 0.001, batch 32) on an AWS g5.xlarge GPU.

Test accuracy: 87.65% of all points labelled correctly (617 test chairs; best checkpoint at epoch 25).

Colours: back seat legs arms part 1 part 0 (white, invisible on the white background)

#471good · 99.25%
Chair 471 ground truth
Ground truth
Chair 471 prediction
Prediction
#434good · 99.20%
Chair 434 ground truth
Ground truth
Chair 434 prediction
Prediction
#0typical · 91.81%
Chair 0 ground truth
Ground truth
Chair 0 prediction
Prediction
#351bad · 42.27%
Chair 351 ground truth
Ground truth
Chair 351 prediction
Prediction
#577bad · 44.14%
Chair 577 ground truth
Ground truth
Chair 577 prediction
Prediction

Interpretation. The model segments ordinary chairs almost perfectly (>99% on #471 and #434): thin, clearly separated legs, a flat seat and a distinct back, which is the most common shape in the training data. On a typical chair (#0, 91.8%), the errors are concentrated at the boundaries between parts, e.g. where the legs meet the seat. The two worst chairs are unusual, legless designs: #351 is a closed tub/barrel chair, and #577 is an upholstered armchair with a solid base. In both, the model labels the entire lower half as legs, whereas the ground truth labels it as seat/back/arms. This shows that PointNet relies heavily on a point's height within the object. Because each point is processed independently and only combined with one global feature, the model cannot see that the bottom of these chairs is a solid surface rather than thin rods.

Q3. Robustness Analysis

Both experiments use the trained Q1 and Q2 models without retraining. Only the test inputs are changed (--num_points and a new --rot_deg option in eval_cls.py / eval_seg.py). The bold rows are the Q1/Q2 baseline.

Experiment 1: Number of points per object

Procedure. For each test object, a random subset of N ∈ {10000, 5000, 1000, 500, 100, 50, 10} points is kept (with the matching per-point labels for segmentation), and both models are evaluated on this subset.

Points per objectClassification acc.Segmentation acc.
10000 (Q1/Q2 baseline)97.17%87.65%
500097.17%87.80%
100097.27%88.05%
50096.85%87.65%
10094.33%82.17%
5087.09%77.46%
1026.65%68.57%

Classification: the same objects with 10,000 vs 100 points

Object 605 with 10000 points
#605 · 10000 pts · pred chair
Object 605 with 100 points
#605 · 100 pts · pred chair
Object 807 with 10000 points
#807 · 10000 pts · pred lamp
Object 807 with 100 points
#807 · 100 pts · pred lamp
Vase 619 with 100 points
#619 · 100 pts · true vase, pred lamp

Segmentation: chair #0 as the number of points decreases

10000 ptsbaseline
Chair 0 ground truth, 10000 points
Ground truth
Chair 0 prediction, 10000 points
Prediction
500 pts
Chair 0 ground truth, 500 points
Ground truth
Chair 0 prediction, 500 points
Prediction
100 pts
Chair 0 ground truth, 100 points
Ground truth
Chair 0 prediction, 100 points
Prediction
50 pts
Chair 0 ground truth, 50 points
Ground truth
Chair 0 prediction, 50 points
Prediction

Interpretation. Both models are very robust to sparsity: accuracy is essentially unchanged down to 500 points (5% of the training resolution) and only drops noticeably below 100 points. This is a direct consequence of the max-pool. The global feature is determined by a small set of "critical points" that give the maximum response for each feature, and a random subset of a few hundred points still contains enough of them to describe the shape. At 10 points, the shape is essentially gone: classification collapses to 26.7%, worse than always predicting "chair". Segmentation degrades more gracefully (68.6%), because a point's height alone is already a strong cue for which part it belongs to (legs low, back high).

Experiment 2: Rotating the input

Procedure. Every test point cloud is rotated about the x-axis by θ ∈ {0°, 15°, 30°, 45°, 60°, 90°, 180°}, i.e. tilted forwards/backwards (the y-axis points up in this data). Labels are unchanged. All 10,000 points are used.

Rotation about xClassification acc.Segmentation acc.
0° (Q1/Q2 baseline)97.17%87.65%
15°94.54%84.88%
30°83.42%75.24%
45°54.04%63.05%
60°35.68%44.31%
90°21.20%31.36%
180°62.43%37.39%

Classification: the same objects at 0°, 45° and 90°

#486chair
Chair 486 at 0 degrees
0° · pred chair
Chair 486 at 45 degrees
45° · pred lamp
Chair 486 at 90 degrees
90° · pred vase
#605chair
Chair 605 at 0 degrees
0° · pred chair
Chair 605 at 45 degrees
45° · pred chair
Chair 605 at 90 degrees
90° · pred vase
#807lamp
Lamp 807 at 0 degrees
0° · pred lamp
Lamp 807 at 45 degrees
45° · pred vase
Lamp 807 at 90 degrees
90° · pred vase

Segmentation: chair #0 under rotation

0°baseline
Chair 0 ground truth, 0 degrees
Ground truth
Chair 0 prediction, 0 degrees
Prediction
30°
Chair 0 ground truth, 30 degrees
Ground truth
Chair 0 prediction, 30 degrees
Prediction
90°
Chair 0 ground truth, 90 degrees
Ground truth
Chair 0 prediction, 90 degrees
Prediction
180°
Chair 0 ground truth, 180 degrees
Ground truth
Chair 0 prediction, 180 degrees
Prediction

Interpretation. Unlike sparsity, rotation hurts both models badly. Accuracy is still reasonable at 15° but falls quickly beyond 30°, and at 90° the classifier is at 21% and calls almost everything a vase (a chair lying on its back has a compact, box-like outline). Without T-Nets or rotation augmentation, PointNet has no mechanism for rotation invariance. Every training object was upright, so the learned features encode upright patterns such as "horizontal surface at mid-height" (seat) or "thin vertical rod" (lamp pole). 180° is better than 90° for classification (62%) because many vases and lamps are roughly symmetric top-to-bottom, but segmentation stays poor (37%): parts are recognised largely by height, and upside down the legs are at the top. Rotation augmentation during training, or a T-Net that predicts a canonical alignment, would be the natural fixes.

Q4. Bonus: Locality with DGCNN

Model. I implemented DGCNN (Dynamic Graph CNN, Wang et al. 2019). Its building block, EdgeConv, adds the locality that PointNet lacks:

  1. For each point xi, find its k = 20 nearest neighbours xj.
  2. Build an edge feature for each neighbour: [xi, xj − xi]. The relative offset describes the local surface (flat, curved, rod-like) around the point.
  3. Apply a shared MLP to every edge (Conv2d 1×1 + BatchNorm + LeakyReLU), then max-pool over the k neighbours.
  4. Repeat. After the first layer the neighbour graph is rebuilt in feature space (hence "dynamic"), so points that look alike, e.g. the tips of different legs, can become neighbours even when they are far apart.

Classification: 4 EdgeConv layers (64, 64, 128, 256) → concatenate (512) → 1024 → global max-pool → MLP (512, 256, dropout 0.5) → 3 classes. Segmentation: 3 EdgeConv layers (64 each) → concatenated local feature (192) + global max-pooled feature (1024) copied to every point → per-point head (256, 256, 128) → 6 parts. The kNN is implemented in plain PyTorch. Because the pairwise-distance matrix is N×N, both models are trained and tested on a random 1,024 points per object (a new random subset every batch). Trained for 40 epochs (cls) and 50 epochs (seg) with Adam, lr 0.001, batch 32.

Test accuracy compared with Q1 and Q2

ModelPointsClassification acc.Segmentation acc.
PointNet (Q1 / Q2)1000097.17%87.65%
PointNet (Q3, same input size as DGCNN)102497.27%88.05%
DGCNN (Q4)102498.32%90.17%

DGCNN makes 16 classification mistakes instead of 27 (lamp → vase errors drop from 17 to 5) and improves segmentation by about 2 points at the same input size.

Classification visualisations

The same five random objects as Q1 are all still classified correctly. The remaining failures:

Chair 543 predicted as lamp by DGCNN
#543 · true: chair · pred: lamp
Vase 619 predicted as lamp by DGCNN
#619 · true: vase · pred: lamp (same as Q1)
Lamp 750 predicted as vase by DGCNN
#750 · true: lamp · pred: vase
Lamp 726
#726 (Q1 failure) · DGCNN: lamp (fixed)

The bulbous-base lamp #726 that PointNet called a vase is now correct, likely because local features can pick up the thin connector between base and shade, which a per-point model cannot see. The flower vase #619 is still wrong, because its stem and bloom really are locally lamp-like. The single chair error (#543) is a very tall, narrow chair that looks like a pole with a small base.

Segmentation: the Q2 chairs, PointNet vs DGCNN (both at 1,024 points)

ChairPointNet @1024DGCNN @1024Change
#0 (typical)91.80%93.85%+2.1
#471 (good)98.54%99.12%+0.6
#434 (good)98.73%98.83%+0.1
#351 (tub chair)45.80%45.41%−0.4
#577 (armchair)42.97%69.04%+26.1
#577armchair
Chair 577 ground truth
Ground truth
Chair 577 PointNet prediction
PointNet · 43.0%
Chair 577 DGCNN prediction
DGCNN · 69.0%
#0typical
Chair 0 ground truth
Ground truth
Chair 0 PointNet prediction
PointNet · 91.8%
Chair 0 DGCNN prediction
DGCNN · 93.9%
#471good
Chair 471 ground truth
Ground truth
Chair 471 PointNet prediction
PointNet · 98.5%
Chair 471 DGCNN prediction
DGCNN · 99.1%
#434good
Chair 434 ground truth
Ground truth
Chair 434 PointNet prediction
PointNet · 98.7%
Chair 434 DGCNN prediction
DGCNN · 98.8%
#351tub chair
Chair 351 ground truth
Ground truth
Chair 351 PointNet prediction
PointNet · 45.8%
Chair 351 DGCNN prediction
DGCNN · 45.4%

Interpretation. The clearest benefit of locality is on armchair #577. PointNet labelled its whole solid base as legs based on height alone. DGCNN predicts almost no leg points and correctly traces the arms down the sides, because each point's neighbourhood shows a solid surface rather than a thin rod. This raises accuracy from 43% to 69%. On ordinary chairs, DGCNN mainly cleans up the boundaries between parts (+2 points on chair #0). The tub chair #351 is still a failure for both models: its body is one continuous curved wall, so local geometry alone does not reveal where the "seat" ends and the "back" begins, and the bottom half is still predicted as legs. Fixing it likely needs more such shapes in the training data rather than only a different architecture.