16-825 Assignment 5: Point Cloud Classification and Segmentation

Q1. Classification Model

Test accuracy: 97.90% (PointNet without transform blocks, all 10000 points, 953 test objects).

Random test objects


#39 pred: chair
true: chair

#256 pred: chair
true: chair

#293 pred: chair
true: chair

#485 pred: chair
true: chair

#604 pred: chair
true: chair

#806 pred: vase (wrong)
true: lamp

Failure cases (one row per class)

chair
0 of 617 chairs wrong
shown: other objects
wrongly called chair

#869 pred: chair (wrong)
true: lamp
vase
9 of 102 vases wrong

#618 pred: lamp (wrong)
true: vase

#651 pred: lamp (wrong)
true: vase
lamp
11 of 234 lamps wrong

#721 pred: vase (wrong)
true: lamp

#787 pred: vase (wrong)
true: lamp

Training: Adam, learning rate 0.001, batch size 32, 100 epochs, all 10000 points. train.py keeps the best epoch (epoch 93).

Class ids: 0 = chair, 1 = vase, 2 = lamp. Only 20 of 953 test objects are wrong. All 617 chairs are right. 9 vases are called lamps, 10 lamps are called vases, and 1 lamp is called a chair.

Why it fails: vases and lamps are both round, tall and often symmetric, so their global shapes are close. The failures are unusual shapes. Vase #618 is a closed cylinder with a lid, like a lantern. Vase #651 is a flower on a long curved stem, like a desk lamp arm with a head. Lamp #721 is a cylinder on thin rods, and lamp #787 is a round ball, so both look like vases. No chair is wrong, so the chair row shows lamp #869 instead: it has a flat rectangular frame, like a chair back. PointNet only uses one global max-pooled feature, so it does not look at small local parts that could fix these cases. There are also fewer vases (741) and lamps (1554) than chairs (4489) in training.

Q2. Segmentation Model

Test accuracy: 89.79% (fraction of correct points over all 617 test chairs, 10000 points each).

objectground truthprediction
#294
good
accuracy 94.72%

ground truth

prediction
#329
good
accuracy 98.16%

ground truth

prediction
#537
good
accuracy 95.71%

ground truth

prediction
#351
bad
accuracy 45.00%

ground truth

prediction
#235
bad
accuracy 49.56%

ground truth

prediction

Training: Adam, learning rate 0.001, batch size 32, 100 epochs, all 10000 points. train.py keeps the best epoch (epoch 97).

Colors: white = class 0 and magenta = class 1 (small, rare parts; white is hard to see), cyan = back, yellow = arms, blue = legs or base, red = seat.

Most chairs are segmented well (95% to 98% on the good examples). The errors are mostly at part borders, for example where the seat meets the back or the legs.

The two bad objects are unusual chairs. #351 is a closed barrel chair: the ground truth calls its outer wall back and arms, but the model calls the lower half legs/base, because on normal chairs the low points are legs. #235 is a sofa where the ground truth mixes arm and seat labels on the same surface, and the model gives a cleaner but different split. The model relies a lot on the height of a point, so it fails when a part is not at its usual height or when the labels are not clear.

Q3. Robustness Analysis

Experiment 1: rotate the input

Procedure: rotate every test point cloud about the z axis (this tilts the object sideways) by 15, 30, 45, 60 and 90 degrees. Use the Q1 and Q2 models with no retraining, all 10000 points (eval_cls.py / eval_seg.py with --rot_deg).

rotation (deg, about z)classification accsegmentation acc
097.90%89.79%
1592.65%82.88%
3071.88%69.98%
4532.00%60.92%
6027.70%53.95%
9025.50%43.92%

Classification (left column = Q1 result):


0 deg
#39 pred: chair
true: chair

45 deg
#39 pred: vase (wrong)
true: chair

90 deg
#39 pred: vase (wrong)
true: chair

0 deg
#256 pred: chair
true: chair

45 deg
#256 pred: vase (wrong)
true: chair

90 deg
#256 pred: vase (wrong)
true: chair

0 deg
#293 pred: chair
true: chair

45 deg
#293 pred: vase (wrong)
true: chair

90 deg
#293 pred: vase (wrong)
true: chair

Segmentation (0 deg = Q2 result):


ground truth

0 deg pred
accuracy 94.72%

45 deg pred
accuracy 58.28%

90 deg pred
accuracy 34.42%

ground truth

0 deg pred
accuracy 98.16%

45 deg pred
accuracy 64.26%

90 deg pred
accuracy 51.58%

Both models are not rotation invariant. Accuracy already drops at 15 degrees and falls fast after that. At 45 degrees, classification falls from 97.9% to 32.0%, and segmentation from 89.8% to 60.9%.

The models were trained only on upright objects, with no rotation augmentation and no transform block, so they learn features in a fixed frame. A tilted chair looks like a vase to the classifier (at 90 degrees, 724 of 953 objects are called vase). The segmentation model still labels points by their old height, so after tilting, the labels follow the wrong axis.

Experiment 2: fewer points per object

Procedure: keep only N random points per object (the same random subset for all objects, seed 0), N = 5000, 2000, 1000, 500, 100, 50, 10. Use the Q1 and Q2 models with no retraining (eval_cls.py / eval_seg.py with --num_points).

points per objectclassification accsegmentation acc
1000097.90%89.79%
500097.69%89.86%
200097.69%89.85%
100097.38%89.63%
50096.75%88.83%
10089.93%81.78%
5067.47%77.69%
1027.07%66.66%

Classification (left column = Q1 result):


10000 points
#39 pred: chair
true: chair

100 points
#39 pred: chair
true: chair

50 points
#39 pred: chair
true: chair

10000 points
#256 pred: chair
true: chair

100 points
#256 pred: chair
true: chair

50 points
#256 pred: chair
true: chair

10000 points
#293 pred: chair
true: chair

100 points
#293 pred: chair
true: chair

50 points
#293 pred: lamp (wrong)
true: chair

Segmentation (10000 points = Q2 result):


10000 points
ground truth

10000 points pred
accuracy 94.72%

100 points
ground truth

100 points pred
accuracy 90.00%

50 points
ground truth

50 points pred
accuracy 80.00%

10000 points
ground truth

10000 points pred
accuracy 98.16%

100 points
ground truth

100 points pred
accuracy 87.00%

50 points
ground truth

50 points pred
accuracy 88.00%

The models are very robust to fewer points. With 500 points (5% of the input), classification is 96.7% and segmentation is 88.8%, almost the same as with 10000 points. Accuracy drops clearly only below 100 points.

PointNet uses a max pool, so the global feature only depends on a small set of key points that cover the shape. A random subset still covers the shape well. With 50 or 10 points the shape is lost and the classifier mostly says lamp. Segmentation stays at 67% even with 10 points, because the label of a point depends a lot on its height.

Q4. Bonus: Locality (PointNet++)

Model: PointNet++ (single-scale grouping), in models.py (cls_pointnet2_model, seg_pointnet2_model). It adds locality by learning features of small neighborhoods:

Set abstraction 1: farthest point sampling picks 512 centers, a ball query (radius 0.2) takes 32 neighbors per center, and a shared MLP (64, 64, 128) with max pool encodes each neighborhood in local coordinates. Set abstraction 2: 128 centers, radius 0.4, 64 neighbors, MLP (128, 128, 256). Set abstraction 3: one global MLP (256, 512, 1024) and max pool.

Classification: the global feature goes through FC layers 512, 256, 3 with dropout. Segmentation: three feature propagation layers bring the features back to all 10000 points (inverse-distance interpolation from the 3 nearest centers, skip links, MLPs), then a per-point classifier.

Training is the same as Q1 and Q2 (Adam, 0.001, batch 32, 100 epochs, all 10000 points). Best epochs: 42 (classification) and 93 (segmentation). Sampling and grouping use pytorch3d ops.

modelclassification accsegmentation acc
PointNet (Q1, Q2)97.90%89.79%
PointNet++ (Q4)97.48%90.45%

Classification: the Q1 failure cases again.

objectPointNetPointNet++
#869
true: lamp

PointNet: chair
(wrong)
PointNet++: vase
(wrong)
#618
true: vase

PointNet: lamp
(wrong)
PointNet++: lamp
(wrong)
#651
true: vase

PointNet: lamp
(wrong)
PointNet++: vase
(right)
#721
true: lamp

PointNet: vase
(wrong)
PointNet++: lamp
(right)
#787
true: lamp

PointNet: vase
(wrong)
PointNet++: vase
(wrong)

Segmentation: the Q2 objects again.

objectground truthPointNetPointNet++
#294
ground truth

PointNet
accuracy 94.72%

PointNet++
accuracy 97.01%
#329
ground truth

PointNet
accuracy 98.16%

PointNet++
accuracy 98.04%
#537
ground truth

PointNet
accuracy 95.71%

PointNet++
accuracy 98.24%
#351
ground truth

PointNet
accuracy 45.00%

PointNet++
accuracy 47.11%
#235
ground truth

PointNet
accuracy 49.56%

PointNet++
accuracy 49.54%

Segmentation gets better: 90.45% vs 89.79%. Part borders are sharper and thin parts are cleaner (for example #537 goes from 95.7% to 98.2%), because each point now sees its local neighborhood. The two bad chairs stay bad, because their problem is the unusual shape and unclear labels, not missing detail.

Classification stays about the same: 97.48% vs 97.90% (4 more wrong objects out of 953). PointNet++ fixes 11 of the 20 PointNet failures, but makes new mistakes, mostly vases called lamps. Local features do not help much for this easy 3-class task.

Note: all numbers above use the same eval scripts, which shuffle the point order. The ball query takes the first 32 points inside the ball, so PointNet++ depends a little on the point order. With the original point order it gets 98.53% (classification) and 91.76% (segmentation). PointNet does not depend on the order.