Assignment 5: Point Cloud Classification and Segmentation

16-825 Learning for 3D Vision · Andrew ID: soumojib

1

Classification model

30 pts
Test accuracy98.43%
Test objects953

PointNet without the transformation blocks: a shared per-point MLP (3→64→64→64→128→1024), max-pooling over the points, and an MLP head (1024→512→256→3). Trained for 250 epochs on all 10,000 points per object; the reported accuracy is that of best_model.pt, which the provided train.py keeps at the epoch with the highest test accuracy. Per class: chair 99.84%, vase 94.12%, lamp 96.58%.

Random test objects

Test chair 523
#523 · true: chair · predicted: chair
Test chair 393
#393 · true: chair · predicted: chair
Test vase 648
#648 · true: vase · predicted: vase
Test vase 644
#644 · true: vase · predicted: vase
Test lamp 722
#722 · true: lamp · predicted: lamp
Test lamp 760
#760 · true: lamp · predicted: lamp

Failures (one per class)

Misclassified test chair
#406 · true: chair · predicted: lamp
Misclassified test vase
#618 · true: vase · predicted: lamp
Misclassified test lamp
#883 · true: lamp · predicted: vase
Chair predicted as lamp
From the side this chair is almost a flat panel: a tall rounded back on short legs with hardly any seat depth. Without the seat that every other chair has, its global shape is closer to a flat lamp or shade on a stand.
Vase predicted as lamp
A capped cylinder with another object visible inside it, which looks like a glass lantern. The points inside the shell (unusual for a vase, which is a hollow surface) and the lid on top pull it towards the lamp class.
Lamp predicted as vase
A squat globe on a short base, with no stem, arm or shade: exactly the silhouette of a round vase or pot. The only cue that it is a lamp is its function, not its shape.

All three failures are shapes that sit between two classes. Max-pooling keeps only the overall extent of the object in each feature direction, so a chair without a seat or a lamp without a stem has nothing that separates it from the other class. Most errors are between vases and lamps (6 vases called lamps, 8 lamps called vases); vases are also the rarest training class.

2

Segmentation model

30 pts
Test accuracy90.72%
Test chairs617

PointNet segmentation: each point's 64-d local feature (after the first two shared layers) is concatenated with the 1024-d global feature and classified per point by a shared MLP (1088→512→256→128→6). Accuracy is the fraction of correctly labelled points over all 617 test chairs. Three chairs below are random; the last two are the two with the lowest accuracy in the test set.

Chair #561: 97.54% of points correct

Ground-truth parts of chair 561
Ground truth
Predicted parts of chair 561
Prediction

Chair #310: 87.67% of points correct

Ground-truth parts of chair 310
Ground truth
Predicted parts of chair 310
Prediction

Chair #375: 95.95% of points correct

Ground-truth parts of chair 375
Ground truth
Predicted parts of chair 375
Prediction

Chair #235 (bad): 42.49% of points correct

Ground-truth parts of chair 235
Ground truth
Predicted parts of chair 235
Prediction

Chair #351 (bad): 46.08% of points correct

Ground-truth parts of chair 351
Ground truth
Predicted parts of chair 351
Prediction
ObjectAccuracy
#56197.54%
#31087.67%
#37595.95%
#235 (bad)42.49%
#351 (bad)46.08%

Interpretation. On the two ordinary chairs (#561, #375: separate legs, a flat seat and an upright back) the predictions are almost perfect, and the few errors sit on the seat/back boundary. On the armchair (#310) the errors are where the arms meet the back: nothing in the per-point feature marks an exact part boundary, so neighbouring parts bleed into each other. The two worst chairs are a boxy sofa and a closed cylindrical chair whose sides are labelled as armrests. PointNet labels them as if they followed the usual template by height: the lower half becomes legs, the middle the seat and the top the back. Its per-point decision depends on the point's own coordinates plus one global shape code, so a chair that breaks the height template is labelled from the template anyway.

3

Experiment 1: rotating the input

10 pts

Procedure. Every test object is rotated about the x axis (y is up in this data, so the object tips forward/backward) by 0–90°, and the Q1/Q2 models are evaluated unchanged on all 953 classification and 617 segmentation test objects. No rotation was used in training.

Tilt about xClassificationSegmentation
0°98.43%90.72%
15°95.70%84.75%
30°75.76%73.30%
45°47.74%60.40%
60°34.84%42.61%
90°25.08%26.84%

Classification samples

Test chair 523 at 0
0° · true: chair · predicted: chair
Test chair 523 at 45
45° · true: chair · predicted: vase
Test chair 523 at 90
90° · true: chair · predicted: vase
Test vase 648 at 0
0° · true: vase · predicted: vase
Test vase 648 at 45
45° · true: vase · predicted: lamp
Test vase 648 at 90
90° · true: vase · predicted: vase
Test lamp 722 at 0
0° · true: lamp · predicted: lamp
Test lamp 722 at 45
45° · true: lamp · predicted: lamp
Test lamp 722 at 90
90° · true: lamp · predicted: chair

Segmentation sample (chair #561; ground truth left, prediction right)

Chair 561 at 0
0° · 97.54% of points correct
Chair 561 at 45
45° · 64.09% of points correct
Chair 561 at 90
90° · 30.42% of points correct

Interpretation. PointNet has no rotation invariance and every training object is upright, so accuracy holds for small tilts (classification 95.70% at 15°) and then collapses: 47.74% at 45° and 25.08% at 90°, far below the 64.74% that always answering "chair" would score, so tipped chairs are mostly called vases or lamps (like the chair sample above). Segmentation degrades the same way (90.72% → 60.40% → 26.84%), and the 90° sample shows why: with the chair on its back, one block of legs is now the highest part, and it is labelled back and seat because that is where the back and seat sit on an upright chair.

3

Experiment 2: fewer points per object

10 pts

Procedure. The same random subset of points (fixed seed) is kept from every test object, from all 10,000 down to 50 points, and the Q1/Q2 models trained on 10,000 points are evaluated unchanged.

Points / objectClassificationSegmentation
10,000 pts98.43%90.72%
5,000 pts98.32%90.73%
1,000 pts98.11%89.94%
500 pts97.27%88.25%
100 pts93.81%80.71%
50 pts79.01%75.98%

Classification samples

Test chair 523 at 10000
10,000 pts · true: chair · predicted: chair
Test chair 523 at 1000
1,000 pts · true: chair · predicted: chair
Test chair 523 at 100
100 pts · true: chair · predicted: chair
Test vase 648 at 10000
10,000 pts · true: vase · predicted: vase
Test vase 648 at 1000
1,000 pts · true: vase · predicted: vase
Test vase 648 at 100
100 pts · true: vase · predicted: lamp
Test lamp 722 at 10000
10,000 pts · true: lamp · predicted: lamp
Test lamp 722 at 1000
1,000 pts · true: lamp · predicted: lamp
Test lamp 722 at 100
100 pts · true: lamp · predicted: lamp

Segmentation sample (chair #561; ground truth left, prediction right)

Chair 561 at 10000
10,000 pts · 97.54% of points correct
Chair 561 at 1000
1,000 pts · 96.80% of points correct
Chair 561 at 100
100 pts · 94.00% of points correct

Interpretation. PointNet is very robust to sparsity: classification stays at 98.11% with 1,000 points and 93.81% with 100, and segmentation at 89.94% and 80.71%. Each point is processed independently and max-pooling only needs the few points that are extreme in each feature direction, which a uniform subsample of the surface still contains. It breaks only at 50 points (79.01% / 75.98%), when thin parts such as legs or a lamp stem keep only one or two points.

4

Locality: PointNet++

+20 bonus

Model: PointNet++ (single-scale grouping). Locality comes from set abstraction layers: farthest-point sampling picks centroids, a ball query groups the points within a fixed radius of each, and a small shared PointNet on coordinates relative to the centroid summarises every group. Classification stacks two such layers (512 centroids, radius 0.2; 128 centroids, radius 0.4) and a global PointNet. Segmentation uses four levels (1,024 / 256 / 64 centroids, radii 0.1 / 0.2 / 0.4, then global) and propagates features back down to every input point by inverse-distance interpolation over the 3 nearest coarser points, concatenated with each level's own features (skip connections). Same training setup as Q1/Q2: 250 epochs, Adam, lr 1e-3 with cosine decay.

Task (test accuracy)PointNet (Q1 / Q2)PointNet++ (Q4)
Classification98.43%98.64%
Segmentation90.72%92.03%

The Q1 failures, classified by PointNet++

Test chair 406
#406 · true: chair · PointNet: lamp · PointNet++: lamp
Test vase 618
#618 · true: vase · PointNet: lamp · PointNet++: vase
Test lamp 883
#883 · true: lamp · PointNet: vase · PointNet++: lamp

The two bad Q2 chairs, segmented by PointNet++

Ground truth chair 235
#235 ground truth
PointNet chair 235
PointNet: 42.49%
PointNet++ chair 235
PointNet++: 48.87%
Ground truth chair 351
#351 ground truth
PointNet chair 351
PointNet: 46.08%
PointNet++ chair 351
PointNet++: 49.74%

Robustness of both models

Tilt about x0°15°30°45°60°90°
PointNet cls98.43%95.70%75.76%47.74%34.84%25.08%
PointNet++ cls98.64%97.69%79.01%48.48%42.81%39.66%
PointNet seg90.72%84.75%73.30%60.40%42.61%26.84%
PointNet++ seg92.03%90.33%80.37%66.15%42.95%27.06%
Points / object10,0005,0001,00050010050
PointNet cls98.43%98.32%98.11%97.27%93.81%79.01%
PointNet++ cls98.64%97.80%94.02%77.54%23.71%23.29%
PointNet seg90.72%90.73%89.94%88.25%80.71%75.98%
PointNet++ seg92.03%91.44%69.86%53.58%49.38%49.45%

Comparison. Locality helps most where local geometry matters: segmentation improves from 90.72% to 92.03%. On the two hard chairs it is only a few points better and makes the same mistake, labelling the closed sides by height, so local features do not fix shapes that are far from the training distribution. Classification is already near ceiling for PointNet (98.43% vs 98.64%); PointNet++ fixes two of the three Q1 failures, but not the flat chair. It degrades more slowly under small tilts (segmentation 80.37% vs 73.30% at 30°), but neither model is rotation invariant and both collapse by 60–90°. The price is robustness to density: the ball radii are fixed, so with 1,000 points the neighbourhoods hold far fewer points than in training, and accuracy falls much faster than PointNet's (classification 94.02% vs 98.11%; 23.71% at 100 points).