16-825 Assignment 5: PointNet for Classification and Segmentation¶

Q1. Classification Model¶

PointNet without the transform blocks: a shared per-point MLP (64, 64, 64, 128, 1024), a max-pool over the points, and a classifier (512, 256, 3). Trained for 250 epochs with Adam (learning rate 0.001, batch size 32) on 10,000 points per object. Results use the best checkpoint (epoch 164).

Test accuracy: 0.9790 (933/953 right).

True class Test objects Predicted chair Predicted vase Predicted lamp Class accuracy
chair 617 616 1 0 0.998
vase 102 0 93 9 0.912
lamp 234 1 9 224 0.957

Random test point clouds¶

Two random test objects per class, with the predicted probability of each class. All six are classified correctly.

Rotating point cloud of test object 523: true class chair, predicted chairRotating point cloud of test object 393: true class chair, predicted chairRotating point cloud of test object 648: true class vase, predicted vaseRotating point cloud of test object 644: true class vase, predicted vaseRotating point cloud of test object 722: true class lamp, predicted lampRotating point cloud of test object 736: true class lamp, predicted lamp
Test index523393648644722736
True classchairchairvasevaselamplamp
Predicted classchairchairvasevaselamplamp
P(chair)1.0001.0000.0000.0000.0000.000
P(vase)0.0000.0001.0000.9990.0000.000
P(lamp)0.0000.0000.0000.0011.0001.000

Failure cases¶

The most confident mistake for each true class.

Rotating point cloud of test object 595: true class chair, predicted vaseRotating point cloud of test object 685: true class vase, predicted lampRotating point cloud of test object 827: true class lamp, predicted vase
Test index595685827
True classchairvaselamp
Predicted classvaselampvase
P(chair)0.0220.0000.000
P(vase)0.9420.0001.000
P(lamp)0.0361.0000.000

18 of the 20 errors are vases and lamps confused with each other. Chairs are almost never wrong: they are the largest training class and have a distinctive structure. PointNet classifies from one max-pooled global feature with no local information, so it mostly sees an object's overall outline, and vases and lamps often share one.

  • Chair predicted as vase (595). A low, wave-shaped lounge chair with no legs, flat seat or upright back. It is the only misclassified chair of 617. It's indeed a difficult case.
  • Vase predicted as lamp (685). A very tall, thin, tapering column, which is the outline of a floor lamp. I don't even know what this is, odd looking vase for sure.
  • Lamp predicted as vase (827). A wide, squat cylinder with no stem, arm or shade, which is the outline of a pot. I think this is quite a bad case, since lamp shouldn't have that low probability.

Q2. Segmentation Model¶

PointNet segmentation without the transform blocks. Three shared layers (64, 64, 64) give a local feature per point; two more (128, 1024) and a max-pool give a global feature. The global feature is concatenated to every point's local feature (1088 values), and a shared MLP (512, 256, 128, 128, 6) predicts the part label. Trained for 250 epochs with the same settings as Q1. Results use the best checkpoint (epoch 192).

Test accuracy: 0.9036 over all points of the 617 test chairs. Per object, the median is 0.934 and 82 objects are below 0.8.

Part label (colour in the renders) Share of test points Accuracy on points of this part
seat (red) 38.1% 0.925
back (cyan) 32.0% 0.926
legs or base (blue) 20.3% 0.890
armrests (yellow) 9.1% 0.771
rare part (white) 0.3% 0.963
headrest (magenta) 0.1% 0.170

The part names are my reading of the renders; the dataset only gives label numbers.

Segmentation results¶

Three random test objects and the two with the lowest accuracy, with the ground truth above the prediction.

Object 392Object 315Object 523Object 26Object 351
Ground truthGround truth segmentation of test object 392Ground truth segmentation of test object 315Ground truth segmentation of test object 523Ground truth segmentation of test object 26Ground truth segmentation of test object 351
PredictionPredicted segmentation of test object 392Predicted segmentation of test object 315Predicted segmentation of test object 523Predicted segmentation of test object 26Predicted segmentation of test object 351
Accuracy0.8110.9480.8680.4830.489
Selectionrandomrandomrandomlowest accuracylowest accuracy

Interpretation. Conventional chairs are segmented well (315). On 392 and 523 the parts are right and the errors are at their boundaries, for example where the legs of 523 meet the seat. The two bad predictions are unusual chairs: on the sofa-like 26 the wide armrests are labelled seat, and on the barrel-shaped 351 the whole lower half is labelled legs.

Each point is labelled from its own position plus one global feature, with no neighbourhood information. The network therefore learns where each part sits on a typical chair (low points are legs, high points are the back). That is imprecise at part boundaries and fails on unusual layouts. Armrests, which vary most between chairs, are the weakest large part (0.77).

Q3. Robustness Analysis¶

Both experiments use the Q1 and Q2 checkpoints unchanged and modify only the test data. The first column of each accuracy table is the Q1 and Q2 result.

Experiment 1: rotating the input¶

Procedure. Every test point cloud is rotated by a fixed angle about the vertical (y) axis before it is given to the model, using all 10,000 points. As an extra check, objects are also tipped 90° about a horizontal (x) axis.

Rotation about the vertical axis 0° 15° 30° 45° 60° 90° 180°
Classification accuracy 0.979 0.966 0.841 0.614 0.523 0.537 0.339
Segmentation accuracy 0.904 0.866 0.808 0.743 0.692 0.637 0.650

Tipped 90° about the x axis: classification 0.228, segmentation 0.269.

Classification samples, with the predicted class and its probability:

True class0° (Q1 result)30°90°
chair (523)Test object 523 (chair), setting n10000_rot0y
predicted chair, P = 1.00
Test object 523 (chair), setting n10000_rot30y
predicted chair, P = 1.00
Test object 523 (chair), setting n10000_rot90y
predicted chair, P = 1.00
vase (648)Test object 648 (vase), setting n10000_rot0y
predicted vase, P = 1.00
Test object 648 (vase), setting n10000_rot30y
predicted vase, P = 0.99
Test object 648 (vase), setting n10000_rot90y
predicted vase, P = 1.00
lamp (722)Test object 722 (lamp), setting n10000_rot0y
predicted lamp, P = 1.00
Test object 722 (lamp), setting n10000_rot30y
predicted lamp, P = 1.00
Test object 722 (lamp), setting n10000_rot90y
predicted chair (wrong), P = 0.84

Segmentation samples, with the accuracy on each object. Points predicted as the rare white label are invisible against the background, which is why part of the back seems to be missing at 90°.

ObjectGround truthPrediction, 0° (Q2 result)Prediction, 30°Prediction, 90°
392Ground truth segmentation of test object 392Predicted segmentation of test object 392, setting n10000_rot0y
accuracy 0.811
Predicted segmentation of test object 392, setting n10000_rot30y
accuracy 0.732
Predicted segmentation of test object 392, setting n10000_rot90y
accuracy 0.551
315Ground truth segmentation of test object 315Predicted segmentation of test object 315, setting n10000_rot0y
accuracy 0.948
Predicted segmentation of test object 315, setting n10000_rot30y
accuracy 0.849
Predicted segmentation of test object 315, setting n10000_rot90y
accuracy 0.476
523Ground truth segmentation of test object 523Predicted segmentation of test object 523, setting n10000_rot0y
accuracy 0.868
Predicted segmentation of test object 523, setting n10000_rot30y
accuracy 0.835
Predicted segmentation of test object 523, setting n10000_rot90y
accuracy 0.534

Interpretation. Neither model is robust to rotation. A 15° turn costs little, but classification loses 14 points at 30° and is at chance level (0.34) by 180°. All training objects share one upright, forward-facing pose, and the network has no transform blocks or rotation augmentation, so its features depend on absolute coordinates. Segmentation holds up better about the vertical axis (0.64 against 0.54 at 90°), because height is preserved and height alone separates legs, seat and back fairly well. Tipping the object removes that cue and both tasks collapse. In the samples, the near-symmetric vase is unaffected, while the lamp with a side arm is classified as a chair at 90°.

Experiment 2: reducing the number of points¶

Procedure. Only a random subset of each object's 10,000 points is given to the model (the same indices for every object, fixed seed), with no rotation. Segmentation accuracy is measured on the points that were kept.

Points per object 10,000 5,000 1,000 500 100 50
Classification accuracy 0.979 0.979 0.974 0.967 0.915 0.754
Segmentation accuracy 0.904 0.904 0.900 0.886 0.808 0.767

Classification samples. Sparse clouds are drawn with larger points.

True class10,000 points (Q1 result)500 points50 points
chair (523)Test object 523 (chair), setting n10000_rot0y
predicted chair, P = 1.00
Test object 523 (chair), setting n500_rot0y
predicted chair, P = 1.00
Test object 523 (chair), setting n50_rot0y
predicted chair, P = 1.00
vase (648)Test object 648 (vase), setting n10000_rot0y
predicted vase, P = 1.00
Test object 648 (vase), setting n500_rot0y
predicted lamp (wrong), P = 0.88
Test object 648 (vase), setting n50_rot0y
predicted lamp (wrong), P = 1.00
lamp (722)Test object 722 (lamp), setting n10000_rot0y
predicted lamp, P = 1.00
Test object 722 (lamp), setting n500_rot0y
predicted lamp, P = 1.00
Test object 722 (lamp), setting n50_rot0y
predicted lamp, P = 1.00

Segmentation samples.

ObjectGround truthPrediction, 10,000 points (Q2 result)Prediction, 500 pointsPrediction, 50 points
392Ground truth segmentation of test object 392Predicted segmentation of test object 392, setting n10000_rot0y
accuracy 0.811
Predicted segmentation of test object 392, setting n500_rot0y
accuracy 0.816
Predicted segmentation of test object 392, setting n50_rot0y
accuracy 0.900
315Ground truth segmentation of test object 315Predicted segmentation of test object 315, setting n10000_rot0y
accuracy 0.948
Predicted segmentation of test object 315, setting n500_rot0y
accuracy 0.958
Predicted segmentation of test object 315, setting n50_rot0y
accuracy 0.780
523Ground truth segmentation of test object 523Predicted segmentation of test object 523, setting n10000_rot0y
accuracy 0.868
Predicted segmentation of test object 523, setting n500_rot0y
accuracy 0.830
Predicted segmentation of test object 523, setting n50_rot0y
accuracy 0.740

Interpretation. Both models are very robust to losing points. With 500 points, 5% of the input, classification drops from 0.979 to 0.967 and segmentation from 0.904 to 0.886. Each point is processed independently, and the global feature is a max over the points, so it is set by a few of them: a random subset that still covers the outline gives nearly the same result. Accuracy only breaks down when the subset no longer outlines the shape (0.754 and 0.767 at 50 points). In the samples, the potted-plant vase is mistaken for a lamp once its foliage thins out to a narrow column.

Q4. Bonus Question: Locality¶

Model¶

I implemented DGCNN (Wang et al., 2019). Its EdgeConv layer finds each point's k = 20 nearest neighbours in the current feature space (recomputed at every layer), applies a shared MLP to every (point, neighbour) pair, and max-pools over the neighbours. Unlike PointNet, every layer therefore sees local shape.

  • Classification: four EdgeConv layers (64, 64, 128, 256), concatenated and mapped to 1024; global max-pool and average-pool (2048); MLP (512, 256, 3) with dropout 0.5.
  • Segmentation: three EdgeConv layers (64, 64, 64) give a 192-dimensional local feature; a shared layer to 1024 and a max-pool give the global feature, which is concatenated to each point's local feature; shared MLP (256, 256, 128, 6).

Both are simplified from the paper: no spatial transformer, and one MLP layer per EdgeConv.

Settings¶

Setting PointNet (Q1 and Q2) DGCNN classification DGCNN segmentation
Parameters 0.81 M and 0.89 M 1.80 M 0.63 M
Points per object in training 10,000 1,024 2,048
Neighbours per point (k) none 20 20
Optimizer Adam, learning rate 0.001, constant same same
Batch size 32 32 32
Epochs 250 200, in two stages 200, in three stages
Data augmentation none a new random subset of points for every batch same
Dropout none 0.5 0.5
Activation ReLU LeakyReLU (0.2) LeakyReLU (0.2)
Checkpoint reported best test accuracy (epochs 164 and 192) best test accuracy (epoch 187) best test accuracy (epoch 53)
Time per epoch, including the test pass about 6 s and 9 s about 11 s about 9 s
  • Fewer points. The cost of the neighbour search grows with the square of the number of points, so DGCNN trains on a random subset, redrawn for every batch.
  • Staged training. Classification ran for 60 epochs and was continued from its epoch-50 checkpoint to 200. Segmentation ran for 100 epochs and was continued from epoch 90 and, after an interruption, from epoch 180, to 200. The optimizer state restarted at each continuation.
  • Evaluation. DGCNN is tested on a fixed random subset of the size it was trained on. PointNet is tested on the same subset for comparison.

Test accuracy¶

Task PointNet, all 10,000 points (Q1, Q2) PointNet, same subset as DGCNN DGCNN
Classification (subset of 1,024 points) 0.9790 0.9748 0.9853
Segmentation (subset of 2,048 points) 0.9036 0.9031 0.9142

Given all 10,000 points, DGCNN falls to 0.887 for classification and 0.891 for segmentation, because its 20 neighbours then cover a much smaller region than in training. It has to be tested at its training density.

Classification compared with Q1¶

Errors on the 953 test objects, both models on the same 1,024 points:

True class Test objects PointNet errors DGCNN errors
chair 617 1 1
vase 102 12 7
lamp 234 11 6
all 953 24 14

The three Q1 failure cases, then DGCNN's most confident mistake for each class:

Object 595Object 685Object 827Object 406Object 673Object 859
Input (1,024 points)Test object 595 (chair), 1,024 pointsTest object 685 (vase), 1,024 pointsTest object 827 (lamp), 1,024 pointsTest object 406 (chair), 1,024 pointsTest object 673 (vase), 1,024 pointsTest object 859 (lamp), 1,024 points
True classchairvaselampchairvaselamp
PointNetvase (wrong)
P = 0.83
lamp (wrong)
P = 1.00
vase (wrong)
P = 1.00
chair
P = 0.88
lamp (wrong)
P = 1.00
lamp
P = 1.00
DGCNNchair
P = 1.00
lamp (wrong)
P = 1.00
vase (wrong)
P = 0.70
vase (wrong)
P = 0.63
lamp (wrong)
P = 1.00
vase (wrong)
P = 1.00

Segmentation compared with Q2¶

Accuracy by part, both models on the same 2,048 points:

Part label (colour in the renders) Share of test points PointNet accuracy DGCNN accuracy
seat (red) 38.1% 0.921 0.931
back (cyan) 32.0% 0.926 0.939
legs or base (blue) 20.3% 0.903 0.898
armrests (yellow) 9.1% 0.756 0.805
rare part (white) 0.3% 0.951 0.950
headrest (magenta) 0.1% 0.162 0.033
all points 100% 0.903 0.914

DGCNN is more accurate than PointNet on 397 of the 617 chairs and less accurate on 215. The number of chairs below 0.8 falls from 80 to 68.

The five Q2 objects, then DGCNN's second-worst object (96):

Object 392Object 315Object 523Object 26Object 351Object 96
Ground truthGround truth segmentation of test object 392, 2,048 pointsGround truth segmentation of test object 315, 2,048 pointsGround truth segmentation of test object 523, 2,048 pointsGround truth segmentation of test object 26, 2,048 pointsGround truth segmentation of test object 351, 2,048 pointsGround truth segmentation of test object 96, 2,048 points
PointNetPointNet segmentation of test object 392
accuracy 0.835
PointNet segmentation of test object 315
accuracy 0.946
PointNet segmentation of test object 523
accuracy 0.844
PointNet segmentation of test object 26
accuracy 0.505
PointNet segmentation of test object 351
accuracy 0.452
PointNet segmentation of test object 96
accuracy 0.537
DGCNNDGCNN segmentation of test object 392
accuracy 0.829
DGCNN segmentation of test object 315
accuracy 0.950
DGCNN segmentation of test object 523
accuracy 0.917
DGCNN segmentation of test object 26
accuracy 0.528
DGCNN segmentation of test object 351
accuracy 0.463
DGCNN segmentation of test object 96
accuracy 0.504

Interpretation¶

Classification. DGCNN is better: 0.985 against 0.975 on the same points, with errors falling from 24 to 14, almost all in the vase and lamp confusion from Q1. Seeing local shape helps separate classes that share an outline: the lounge chair 595 is now a chair with probability 1.00. It is not a cure, since 685 and 827 are wrong for both models and DGCNN makes new mistakes (406, 859). The margin is only 10 objects, but it is consistent: over the last 50 epochs, test accuracy averaged 0.981 for DGCNN and 0.968 for PointNet.

Segmentation. DGCNN's best checkpoint is 1.1 points better (0.914 against 0.903), mostly from the armrests (0.756 to 0.805) and from sharper part boundaries, as on chair 523 (0.844 to 0.917). Chairs with unusual layouts (26, 351, 96) stay near 0.5 for both models. The caveat is that DGCNN's training was unstable: over its last 50 epochs, test accuracy swung between 0.74 and 0.91, where PointNet stayed between 0.87 and 0.90. The reported checkpoint is from epoch 53 and later epochs never beat it, so the margin, selected on the test set, is optimistic. A decaying learning rate would likely help; I did not test it.

Cost. DGCNN is more expensive per point, has to train on a subset of the points, and only works at that density. PointNet is indifferent to the number of points. Training for longer mattered little: the best classification accuracy was 0.982 after 60 epochs and 0.985 after 200.