16-825 Learning for 3D Vision: Assignment 5

16-825, Fall 2026 · Zijin Ye (Andrew ID: zijiny)
Point cloud classification and segmentation with PointNet, robustness analysis, and PointNet++ (bonus). All results below were produced with the submitted code on a Google Colab A100 GPU.

ModelClassification test accuracySegmentation test accuracy
PointNet (Q1, Q2)97.48%90.50%
PointNet++ SSG (Q4)98.32%91.13%

Models and training setup

Both PointNet models follow the paper without the T-Net transformation blocks. Every "shared MLP" is a 1×1 Conv1d + BatchNorm + ReLU applied to each point independently.

Training uses Adam (learning rate 0.001, cosine decay to 1e-5), batch size 32 and cross-entropy loss. At every training step each object is represented by a fresh random subset of 4,096 of its 10,000 points, which is faster and acts as light augmentation. Testing always uses all 10,000 points. As in the starter code, the checkpoint with the best test accuracy is kept. The classifier trained for 50 epochs (3.8 s/epoch, best at epoch 44) and the segmenter for 60 epochs (4.9 s/epoch, best at epoch 54).

Test accuracy per epoch for PointNet and PointNet++ on classification and segmentation
Test accuracy after every epoch. PointNet is blue; PointNet++ (Q4) is orange. PointNet's segmentation accuracy jumps around early on, while the learning rate is high, and settles as the cosine schedule decays.

Q1. Classification Model

Test accuracy: 97.48% (929 of 953 test objects; 24 mistakes). Per class: chair 99.84%, vase 91.18%, lamp 94.02%.

ground truth ↓ / predicted →chairvaselamp
chair61601
vase1938
lamp212220

Random test objects

Five test objects drawn at random (seed 0), with the predicted class. Four are chairs because chairs make up 65% of the test set.

test object 249 predicted chair
#249: predicted chair (ground truth chair) ✓
test object 331 predicted chair
#331: predicted chair (ground truth chair) ✓
test object 493 predicted chair
#493: predicted chair (ground truth chair) ✓
test object 569 predicted chair
#569: predicted chair (ground truth chair) ✓
test object 875 predicted lamp
#875: predicted lamp (ground truth lamp) ✓

Failure cases (at least one per class)

chair misclassified as lamp
#543: ground truth chair, predicted lamp
vase misclassified as lamp
#620: ground truth vase, predicted lamp
vase misclassified as lamp
#651: ground truth vase, predicted lamp
lamp misclassified as vase
#787: ground truth lamp, predicted vase
lamp misclassified as chair
#944: ground truth lamp, predicted chair

Interpretation. Chairs are almost never wrong: they are the majority class and have a distinctive layout (legs, a flat seat, a back). Most errors are between vases and lamps: 8 vases are called lamps and 12 lamps are called vases. Both classes are mostly round, rotationally symmetric shells, and PointNet decides from a single max-pooled global feature that captures the overall silhouette but not small local details such as a bulb socket, a cord or the opening of a vase. Each failure looks like a typical member of the predicted class:

Q2. Segmentation Model

Test accuracy: 90.50% (fraction of correctly labelled points over all 617 test chairs). Per-object accuracy: mean 90.50%, median 94.02%, worst 43.44%, best 99.73%.

Colours follow the handout (white, magenta, cyan, yellow, blue, red for parts 0–5), except that part 0 is drawn in gray because white points are invisible on the white background. The part names are my reading of the visualizations; the dataset only provides numbers. The per-part accuracy is the fraction of points of that part that are labelled correctly:

part (colour)share of test pointstest chairs containing itPointNetPointNet++
rare part (0)0.33%1194.3%78.8%
headrest (1)0.13%815.6%12.3%
back (2)31.97%61792.4%93.8%
armrests (3)9.13%28476.2%82.2%
legs / base (4)20.31%59990.1%91.4%
seat (5)38.14%61692.7%91.3%

Good predictions (random test objects)

ground truth segmentation of chair 50
ground truth
predicted segmentation of chair 50
prediction: 88.78% of points correct

Chair #50 (random test object): 88.78%

ground truth segmentation of chair 212
ground truth
predicted segmentation of chair 212
prediction: 92.07% of points correct

Chair #212 (random test object): 92.07%

ground truth segmentation of chair 251
ground truth
predicted segmentation of chair 251
prediction: 92.42% of points correct

Chair #251 (random test object): 92.42%

ground truth segmentation of chair 498
ground truth
predicted segmentation of chair 498
prediction: 92.94% of points correct

Chair #498 (random test object): 92.94%

Bad predictions (the two worst test objects)

ground truth segmentation of chair 235
ground truth
predicted segmentation of chair 235
prediction: 43.44% of points correct

Chair #235 (one of the two worst test objects): 43.44%

ground truth segmentation of chair 26
ground truth
predicted segmentation of chair 26
prediction: 47.60% of points correct

Chair #26 (one of the two worst test objects): 47.60%

Interpretation. On ordinary chairs with clearly separated parts (#50, #212, #251, #498) the prediction is close to the ground truth, and the remaining errors sit on part boundaries: where the seat meets the back or the legs, the label depends on an arbitrary cut. The two worst objects are both sofa-like chairs made of blocky cushions. For #235, the model falls back on a "typical chair by height" layout: back at the top, seat in the middle, legs at the bottom. The real object has almost no legs (1% of its points), and its large side blocks are armrests. Its ground truth is also noisy: armrest and seat points are interleaved. #26 has a headrest cushion (magenta). That part covers only 0.13% of the points and appears in 8 test chairs, so the model almost never predicts it (15.6% of headrest points are labelled correctly). In short, PointNet segments well when the per-point labels can be read off position and the global shape is a standard chair, and it struggles with rare parts and unusual shapes.

Q3. Robustness Analysis — Experiment 1: rotation

Procedure. Every test object is rotated about the x-axis by θ ∈ {0, 15, 30, 45, 60, 90, 135, 180}°. The y-axis points up, so the object tips forward; at 90° a chair lies on its back and at 180° it is upside down. Then the Q1 and Q2 checkpoints are evaluated without any retraining. All 10,000 points are used, and for the segmentation task the ground-truth labels rotate with the points. The θ = 0° row is exactly the Q1 / Q2 result. The same three classification objects (a chair, a vase and a lamp) and the same chair for segmentation are shown at every angle. The command is python robustness.py --experiment rotation.

Accuracy versus rotation angle for classification and segmentation
Test accuracy against rotation angle (PointNet in blue; PointNet++ from Q4 in orange for comparison).
rotationPointNet clsPointNet segPointNet++ clsPointNet++ seg
0 deg about x97.5%90.5%98.3%91.1%
15 deg about x95.4%84.3%98.0%89.0%
30 deg about x83.3%72.8%85.3%80.0%
45 deg about x56.2%61.2%57.8%59.7%
60 deg about x35.4%44.1%49.7%38.9%
90 deg about x23.5%27.0%63.3%18.7%
135 deg about x68.2%25.6%34.1%22.5%
180 deg about x58.6%34.4%41.7%38.1%
cls: chair #559
cls: vase #681
cls: lamp #786
seg #467 ground truth
seg #467 prediction
0 deg
cls 97.5% · seg 90.5%
chair 0 deg about x, predicted chairpred: chair ✓
vase 0 deg about x, predicted vasepred: vase ✓
lamp 0 deg about x, predicted lamppred: lamp ✓
seg ground truth 0 deg about xground truth
seg prediction 0 deg about x80.0% correct
30 deg
cls 83.3% · seg 72.8%
chair 30 deg about x, predicted chairpred: chair ✓
vase 30 deg about x, predicted lamppred: lamp ✗
lamp 30 deg about x, predicted vasepred: vase ✗
seg ground truth 30 deg about xground truth
seg prediction 30 deg about x73.4% correct
60 deg
cls 35.4% · seg 44.1%
chair 60 deg about x, predicted lamppred: lamp ✗
vase 60 deg about x, predicted lamppred: lamp ✗
lamp 60 deg about x, predicted chairpred: chair ✗
seg ground truth 60 deg about xground truth
seg prediction 60 deg about x59.9% correct
90 deg
cls 23.5% · seg 27.0%
chair 90 deg about x, predicted vasepred: vase ✗
vase 90 deg about x, predicted vasepred: vase ✓
lamp 90 deg about x, predicted chairpred: chair ✗
seg ground truth 90 deg about xground truth
seg prediction 90 deg about x42.8% correct
180 deg
cls 58.6% · seg 34.4%
chair 180 deg about x, predicted chairpred: chair ✓
vase 180 deg about x, predicted lamppred: lamp ✗
lamp 180 deg about x, predicted lamppred: lamp ✓
seg ground truth 180 deg about xground truth
seg prediction 180 deg about x39.0% correct

Interpretation. PointNet is not rotation invariant. It reads raw xyz coordinates, has no T-Net, and was trained only on upright objects. Small tilts are harmless (15°: 95.4% / 84.3%), but by 45° accuracy has fallen to 56.2% for classification and 61.2% for segmentation. At 90° it drops to 23.5% / 27.0%: only 3.4% of chairs are recognized, because a chair lying on its back is classified as a vase or lamp. Classification partly recovers at 135° and 180° (68.2%, 58.6%). An upside-down chair still has a seat, legs and a back, and lamps (a pole with a shade) stay lamp-like when flipped. Segmentation does not recover (34.4% at 180°) because the model labels points largely by height. In the 180° example it calls the bottom of the object "legs" even though that is now the back, and it calls the old legs "back". The model has learned the canonical orientation of the dataset; rotation augmentation or a learned alignment (T-Net) would be needed to handle rotated inputs.

Q3. Robustness Analysis — Experiment 2: number of points

Procedure. Instead of all 10,000 points, each test object keeps N ∈ {10000, 5000, 2000, 1000, 500, 200, 100, 50, 10} randomly chosen points. The same random indices are used for every object, with a fixed seed. The unchanged Q1 and Q2 checkpoints are evaluated, and segmentation accuracy is computed over the kept points. The command is python robustness.py --experiment num_points.

Accuracy versus number of points per object
Test accuracy against the number of points per object, decreasing from left to right (PointNet in blue; PointNet++ from Q4 in orange).
points per objectPointNet clsPointNet segPointNet++ clsPointNet++ seg
10000 points97.5%90.5%98.3%91.1%
5000 points97.4%90.6%98.2%91.2%
2000 points97.3%90.4%97.9%91.2%
1000 points97.6%90.0%97.1%90.8%
500 points97.3%88.6%91.0%88.1%
200 points95.9%84.8%44.8%72.4%
100 points94.0%82.3%24.8%48.0%
50 points80.6%79.2%24.6%30.1%
10 points26.1%68.8%24.6%20.8%
cls: chair #559
cls: vase #681
cls: lamp #786
seg #456 ground truth
seg #456 prediction
10000 points
cls 97.5% · seg 90.5%
chair 10000 points, predicted chairpred: chair ✓
vase 10000 points, predicted vasepred: vase ✓
lamp 10000 points, predicted lamppred: lamp ✓
seg ground truth 10000 pointsground truth
seg prediction 10000 points95.4% correct
1000 points
cls 97.6% · seg 90.0%
chair 1000 points, predicted chairpred: chair ✓
vase 1000 points, predicted vasepred: vase ✓
lamp 1000 points, predicted lamppred: lamp ✓
seg ground truth 1000 pointsground truth
seg prediction 1000 points93.0% correct
200 points
cls 95.9% · seg 84.8%
chair 200 points, predicted chairpred: chair ✓
vase 200 points, predicted vasepred: vase ✓
lamp 200 points, predicted lamppred: lamp ✓
seg ground truth 200 pointsground truth
seg prediction 200 points89.5% correct
50 points
cls 80.6% · seg 79.2%
chair 50 points, predicted chairpred: chair ✓
vase 50 points, predicted lamppred: lamp ✗
lamp 50 points, predicted lamppred: lamp ✓
seg ground truth 50 pointsground truth
seg prediction 50 points98.0% correct
10 points
cls 26.1% · seg 68.8%
chair 10 points, predicted lamppred: lamp ✗
vase 10 points, predicted lamppred: lamp ✗
lamp 10 points, predicted lamppred: lamp ✓
seg ground truth 10 pointsground truth
seg prediction 10 points50.0% correct

Interpretation. PointNet is very robust to sparse inputs. With only 500 points (5% of the input) it still reaches 97.3% / 88.6%, and with 100 points it reaches 94.0% / 82.3%. The reason is max-pooling: the global feature is determined by a small set of "critical" points that span the shape, and a uniform random subset still covers the same outline. Training on random 4,096-point subsets also exposed the model to varying density. Accuracy only collapses when the outline itself disappears. Vases go first (66% correct at 100 points, 34% at 50), and at 10 points almost everything is classified as a lamp (99.6% of lamps but 2.6% of chairs), since ten scattered points look like a thin, sparse object. Segmentation degrades more gracefully (68.9% at 10 points), because a point's label can still be guessed from its height alone.

Extra experiment: Gaussian noise

Procedure. Independent Gaussian noise N(0, σ²) is added to every coordinate of every test point, for σ ∈ {0, 0.01, 0.02, 0.05, 0.1, 0.2}. The objects are normalized to a radius of 1, so σ = 0.1 is 10% of the object radius. The command is python robustness.py --experiment noise.

PointNet accuracy versus Gaussian noise level
PointNet test accuracy against noise level.
noisePointNet clsPointNet seg
sigma 097.5%90.5%
sigma 0.0196.7%90.2%
sigma 0.0296.2%89.3%
sigma 0.0590.8%83.4%
sigma 0.186.9%64.4%
sigma 0.273.3%44.5%
cls: chair #559
cls: vase #681
cls: lamp #786
seg #467 ground truth
seg #467 prediction
sigma 0
cls 97.5% · seg 90.5%
chair sigma 0, predicted chairpred: chair ✓
vase sigma 0, predicted vasepred: vase ✓
lamp sigma 0, predicted lamppred: lamp ✓
seg ground truth sigma 0ground truth
seg prediction sigma 080.0% correct
sigma 0.05
cls 90.8% · seg 83.4%
chair sigma 0.05, predicted chairpred: chair ✓
vase sigma 0.05, predicted vasepred: vase ✓
lamp sigma 0.05, predicted lamppred: lamp ✓
seg ground truth sigma 0.05ground truth
seg prediction sigma 0.0568.2% correct
sigma 0.1
cls 86.9% · seg 64.4%
chair sigma 0.1, predicted chairpred: chair ✓
vase sigma 0.1, predicted vasepred: vase ✓
lamp sigma 0.1, predicted lamppred: lamp ✓
seg ground truth sigma 0.1ground truth
seg prediction sigma 0.155.1% correct
sigma 0.2
cls 73.3% · seg 44.5%
chair sigma 0.2, predicted chairpred: chair ✓
vase sigma 0.2, predicted chairpred: chair ✗
lamp sigma 0.2, predicted lamppred: lamp ✓
seg ground truth sigma 0.2ground truth
seg prediction sigma 0.245.2% correct

Interpretation. Light jitter (σ ≤ 0.02) barely matters. Classification degrades slowly (86.9% at σ = 0.1) because the overall box-like shape of a chair survives heavy noise: chairs stay above 99% even at σ = 0.2. Lamps fail first (64% at σ = 0.05, 27% at σ = 0.2) because their thin poles and cords dissolve into the noise. Segmentation is much more sensitive (64.4% at σ = 0.1, 44.5% at σ = 0.2): part boundaries blur, and at σ = 0.2 the model labels most of the noisy cloud as seat.

Q4. Bonus — Locality: PointNet++

Model. I implemented PointNet++ with single-scale grouping (SSG) in plain PyTorch, with no custom CUDA kernels. PointNet applies one MLP to every point and then pools over the whole object, so it never looks at neighbourhoods. PointNet++ builds features hierarchically from local regions instead. Each set abstraction (SA) layer works in three steps:

  1. Farthest point sampling picks a set of well-spread centroids.
  2. A ball query collects up to K neighbours within radius r of each centroid. The neighbours' coordinates are expressed relative to the centroid, so the features describe local shape rather than absolute position.
  3. A shared mini-PointNet (MLP + max-pool) turns each neighbourhood into one feature vector.

Training uses the same recipe as Q1 and Q2 (Adam, cosine decay, batch 32), with 40 epochs, random 2,048-point subsets per object and mixed precision. That takes 26 s/epoch for classification and 19 s/epoch for segmentation on an A100. Testing again uses all 10,000 points.

Test accuracy compared with Q1 and Q2

TaskPointNet (Q1 / Q2)PointNet++Change
Classification (overall)97.48% (24 errors)98.32% (16 errors)+0.84 points, one third fewer errors
  chair / vase / lamp99.84% / 91.18% / 94.02%99.68% / 93.14% / 97.01%vase and lamp improve
Segmentation (all points)90.50%91.13%+0.62 points
  median / worst chair94.02% / 43.44%94.74% / 39.68%

Farthest point sampling starts from a random point, so PointNet++ test accuracy varies by about one object between runs. The evaluation script gives 98.32% / 91.13%, and the end-of-epoch evaluation during training gave 98.43% / 91.27%.

ground truth ↓ / predicted →chairvaselamp
chair61502
vase0957
lamp07227

Per-part accuracies for both models are in the Q2 table. PointNet++ improves the back (92.4% → 93.8%), armrests (76.2% → 82.2%) and legs (90.1% → 91.4%), the parts whose boundaries are defined by local geometry. It is slightly worse on the seat (92.7% → 91.3%) and clearly worse on the very rare part 0 (94.3% → 78.8%).

Classification: the Q1 failure cases

PointNet++ classifies 3 of the 5 Q1 failures correctly. The orchid bowl (#651) and the egg-shaped vase (#620) are now vases, and the bell-shaped lamp (#944) is now a lamp. Local features capture the vase opening and the shape of the thin stem, details the single global PointNet feature tends to wash out. The tall chair (#543) and the stacked-sphere lamp (#787) fool both models.

object 543
#543 (chair): PointNet lamp ✗ → PointNet++ lamp ✗
object 620
#620 (vase): PointNet lamp ✗ → PointNet++ vase ✓
object 651
#651 (vase): PointNet lamp ✗ → PointNet++ vase ✓
object 787
#787 (lamp): PointNet vase ✗ → PointNet++ vase ✗
object 944
#944 (lamp): PointNet chair ✗ → PointNet++ lamp ✓

The remaining PointNet++ errors are still mostly between vases and lamps: ornate urns with a pedestal (#631, #650) and lamps with a vase-like body (#793, #859). PointNet classifies all five of these correctly, so the two models make different mistakes.

object 406
#406 (chair): PointNet chair ✓ → PointNet++ lamp ✗
object 631
#631 (vase): PointNet vase ✓ → PointNet++ lamp ✗
object 650
#650 (vase): PointNet vase ✓ → PointNet++ lamp ✗
object 793
#793 (lamp): PointNet lamp ✓ → PointNet++ vase ✗
object 859
#859 (lamp): PointNet lamp ✓ → PointNet++ vase ✗

Segmentation: the Q2 objects side by side

The same chairs as in Q2, showing the ground truth, the PointNet prediction and the PointNet++ prediction.

ground truth chair 50
ground truth
PointNet prediction chair 50
PointNet: 88.78%
PointNet++ prediction chair 50
PointNet++: 90.22%

Chair #50: PointNet 88.78% → PointNet++ 90.22%

ground truth chair 212
ground truth
PointNet prediction chair 212
PointNet: 92.07%
PointNet++ prediction chair 212
PointNet++: 90.10%

Chair #212: PointNet 92.07% → PointNet++ 90.10%

ground truth chair 251
ground truth
PointNet prediction chair 251
PointNet: 92.42%
PointNet++ prediction chair 251
PointNet++: 94.09%

Chair #251: PointNet 92.42% → PointNet++ 94.09%

ground truth chair 235
ground truth
PointNet prediction chair 235
PointNet: 43.44%
PointNet++ prediction chair 235
PointNet++: 51.44%

Chair #235: PointNet 43.44% → PointNet++ 51.44%

ground truth chair 26
ground truth
PointNet prediction chair 26
PointNet: 47.60%
PointNet++ prediction chair 26
PointNet++: 57.41%

Chair #26: PointNet 47.60% → PointNet++ 57.41%

PointNet++ helps most on the hard sofa-like chairs: #235 goes from 43.4% to 51.4% and #26 from 47.6% to 57.4%. On #235 it labels the side blocks as armrests instead of applying PointNet's height-based back / seat / legs layout, and on #26 it finds the armrest and the seat cushions. Neither model finds the headrest of #26. On ordinary chairs the two models are within about two points of each other. The two worst PointNet++ chairs are shown below. On #426 the boxy skirt under the seat is labelled as the rare part 0 (gray), and on #238 the headrest and the base are confused.

ground truth segmentation of chair 238
ground truth
predicted segmentation of chair 238
prediction: 39.68% of points correct

Chair #238: PointNet++ 39.68% (PointNet got 65.85% on this chair)

ground truth segmentation of chair 426
ground truth
predicted segmentation of chair 426
prediction: 46.11% of points correct

Chair #426: PointNet++ 46.11% (PointNet got 78.62% on this chair)

Robustness compared with PointNet

The orange curves in the rotation and number-of-points plots repeat the Q3 experiments for PointNet++, and the tables there list both models.

cls: chair #559
cls: vase #681
cls: lamp #786
seg #456 prediction
1000 points
cls 97.1% · seg 90.8%
chair 1000 points, predicted chairpred: chair ✓
vase 1000 points, predicted vasepred: vase ✓
lamp 1000 points, predicted lamppred: lamp ✓
seg prediction 1000 points94.6% correct
200 points
cls 44.8% · seg 72.4%
chair 200 points, predicted lamppred: lamp ✗
vase 200 points, predicted lamppred: lamp ✗
lamp 200 points, predicted lamppred: lamp ✓
seg prediction 200 points78.5% correct
50 points
cls 24.6% · seg 30.1%
chair 50 points, predicted lamppred: lamp ✗
vase 50 points, predicted lamppred: lamp ✗
lamp 50 points, predicted lamppred: lamp ✓
seg prediction 50 points44.0% correct