Assignment 5 - Chan WangSing

Everything was run on one RTX 5090. All point clouds are rotating gifs. In the classification gifs the points are coloured by position (left-right = red, height = green, front-back = blue) so the 3D shape is easier to read.

1. Classification Model

The model is PointNet without the transform blocks. The same small MLP turns every point's (x, y, z) into 1024 features (3 → 64 → 64 → 64 → 128 → 1024, each layer with batch norm and ReLU). A max over all points gives one 1024-number description of the object, and an MLP (1024 → 512 → 256 → 3, dropout 0.3) turns it into class scores. I trained it for 250 epochs with the starter settings (Adam, learning rate 0.001, batch size 32, all 10000 points), which took about 24 minutes. The best epoch was 188.

Test accuracy: 98.01% (934 of 953 objects).

ClassTest objectsAccuracyMisclassified
Chair61799.84%1 (as lamp)
Vase10288.24%12 (all as lamp)
Lamp23497.44%6 (all as vase)

Random test objects

Two random objects per class. All six are predicted correctly.

#393: chair, predicted chair
#523: chair, predicted chair
#644: vase, predicted vase
#648: vase, predicted vase
#722: lamp, predicted lamp
#736: lamp, predicted lamp

Failures

#406: chair, predicted lamp
#670: vase, predicted lamp
#701: vase, predicted lamp
#827: lamp, predicted vase
#944: lamp, predicted vase

2. Segmentation Model

The segmentation model uses the same idea for every point. Each point first gets 64 local features (3 → 64 → 64). These go on to 1024 features (64 → 64 → 128 → 1024), and a max over the points gives a 1024-number description of the whole chair. Every point's 64 local features are joined with the 1024 global ones, and a per-point MLP (1088 → 512 → 256 → 128 → 6) gives its part scores. Same training settings as Q1: 250 epochs, about 37 minutes, best epoch 218.

Test accuracy: 90.13% of all points. Accuracy per object: mean 90.13%, median 93.18%, lowest 49.99%, highest 99.53%.

Colours: cyan = back, red = seat, blue = legs / base, yellow = armrests, magenta = headrest / pillow, grey = a rare extra part (the starter code drew this one white on the white background, so I made it grey).

#562 ground truth
#562 prediction: 99.53%
#315 ground truth
#315 prediction: 94.75%
#392 ground truth
#392 prediction: 80.54%
#523 ground truth
#523 prediction: 81.13%
#26 ground truth
#26 prediction: 49.99%
#351 ground truth
#351 prediction: 50.25%
ObjectHow it was pickedAccuracyWhat happens
#562best99.53%A plain chair with every part at its usual height.
#315random94.75%Office chair. Errors only where the armrests meet the seat.
#392random80.54%Rocking chair. The curly armrests merge into the seat.
#523random81.13%The seat / leg border is placed too high, so the front of the seat is called legs.
#26worst49.99%Boxy sofa with a pillow. The pillow is called back and the cushions are mixed up.
#351second worst50.25%Barrel chair with no legs. The bottom half is called legs anyway.

The model has mostly learned "high = back, middle = seat, low = legs". That works for ordinary chairs (half the test chairs are above 93%) and fails on unusual ones like the legless barrel chair #351. It never predicts the headrest / pillow part, which only 8 test chairs have and which is about 0.1% of the training points, so the pillow of #26 becomes back. The mistakes on the good chairs sit on the borders between parts, because every point decides from its own position and one global description, with no information about its neighbours.

3. Robustness Analysis

Both experiments use the Q1 and Q2 models as they are (no retraining) on the whole test set. The 0° and 10000-point results are the Q1 and Q2 numbers.

Experiment 1: rotated input

Procedure: rotate every test object about the z axis (the front-to-back axis), which tips it sideways, by 0° to 180°, then evaluate with all 10000 points.

Rotation0°15°30°45°60°75°90°120°150°180°
Classification98.0%95.1%75.5%40.5%29.2%25.7%24.0%23.5%26.6%60.2%
Segmentation90.1%80.9%67.5%59.7%55.3%50.2%45.8%36.1%35.8%36.0%

Classification, with the predicted class under each gif:

chair #393, 0°: chair
30°: chair
60°: chair
90°: vase
vase #644, 0°: vase
30°: vase
60°: vase
90°: vase
lamp #722, 0°: lamp
30°: lamp
60°: lamp
90°: lamp

Segmentation of chair #562 (ground truth on top, prediction below):

0°
30°
60°
90°
99.5%
66.5%
52.5%
50.0%

Experiment 2: fewer points

Procedure: keep the same random subset of N points of every test object, for N from 10000 down to 10, then evaluate. For segmentation the labels of the kept points are compared.

Points10000500020001000500200100502010
Classification98.0%98.0%97.5%97.4%97.1%94.5%79.6%41.1%24.9%24.6%
Segmentation90.1%90.1%90.0%89.4%87.7%83.7%80.3%76.6%70.9%64.8%

Classification, with the predicted class under each gif:

chair #393, 10000: chair
1000: chair
100: chair
50: lamp
vase #644, 10000: vase
1000: vase
100: vase
50: vase
lamp #722, 10000: lamp
1000: lamp
100: lamp
50: lamp

Segmentation of chair #562 (ground truth on top, prediction below):

10000 points
1000 points
100 points
50 points
99.5%
99.8%
84.0%
90.0%

4. Bonus: Locality with DGCNN

I implemented DGCNN. Its EdgeConv layer finds each point's 20 nearest neighbours, applies a shared MLP to every pair [own features, neighbour features − own features], and keeps the maximum over the 20 neighbours. That way every point learns what its neighbourhood looks like (a thin stick, a flat surface, a curved leaf). The neighbours are searched again in every layer using that layer's features (the "dynamic graph"), so points that look alike can become neighbours even when they are far apart.

The neighbour search compares every pair of points, which is too slow and too big for 10000 points. So DGCNN is trained on 2048 random points per object (a new random set every step) and tested on one fixed set of 2048 points. For a fair comparison, PointNet is also tested on exactly those 2048 points. DGCNN uses the same optimizer, learning rate and batch size as PointNet, but only 100 epochs instead of 250: classification took about 35 minutes (best epoch 57) and segmentation about 13 minutes (best epoch 81).

Test accuracyPointNet, all 10000 points (Q1, Q2)PointNet, same 2048 pointsDGCNN, 2048 points
Classification98.01%97.69%98.53%
Segmentation90.13%89.94%91.39%

Per class on the same 2048 points, PointNet → DGCNN: chair 99.84% → 100%, vase 86.27% → 93.14%, lamp 97.01% → 97.01%.

Classification

DGCNN fixes 16 of PointNet's mistakes and makes 8 new ones. These gifs show 2048 points, so they look sparser than the ones above.

chair #406
PointNet: lamp, DGCNN: chair
vase #670
PointNet: lamp, DGCNN: vase
vase #701
PointNet: lamp, DGCNN: vase
lamp #916
PointNet: vase, DGCNN: lamp
vase #620
PointNet: vase, DGCNN: lamp
lamp #750
PointNet: lamp, DGCNN: vase

The fixed objects include chair #406 and most of the plants that PointNet called lamps. Up close, leaves and bunches of stems are thin, curved blades pointing in many directions, while a lamp pole is one straight, smooth tube. PointNet's single global description blurs that difference, but a neighbourhood of 20 points picks it up. Several of the new mistakes are smooth, round shapes like #620 and #750, where there is little local detail to help.

Segmentation

DGCNN is more accurate than PointNet on 383 of the 617 test chairs.

#523 ground truth
PointNet: 81.1%
DGCNN: 89.5%
#315 ground truth
PointNet: 93.7%
DGCNN: 96.1%
#26 ground truth
PointNet: 49.0%
DGCNN: 55.6%
#351 ground truth
PointNet: 47.8%
DGCNN: 42.2%

The borders between parts get cleaner. On #523 the seat / leg border is now in the right place, since neighbours on the flat seat agree they are on a flat surface while neighbours along a leg line up vertically, and #315 goes from 93.7% to 96.1%. Locality does not fix the mistakes that come from the big picture: the barrel chair #351 still has its bottom half called legs, and the pillow of #26 is still never labelled as a pillow. Those come from the "bottom = legs" habit and from too few training examples, not from missing local detail.