16-825 Assignment 5: PointNet for Classification and Segmentation¶
Q1. Classification Model¶
PointNet without the transform blocks: a shared per-point MLP (64, 64, 64, 128, 1024), a max-pool over the points, and a classifier (512, 256, 3). Trained for 250 epochs with Adam (learning rate 0.001, batch size 32) on 10,000 points per object. Results use the best checkpoint (epoch 164).
Test accuracy: 0.9790 (933/953 right).
| True class | Test objects | Predicted chair | Predicted vase | Predicted lamp | Class accuracy |
|---|---|---|---|---|---|
| chair | 617 | 616 | 1 | 0 | 0.998 |
| vase | 102 | 0 | 93 | 9 | 0.912 |
| lamp | 234 | 1 | 9 | 224 | 0.957 |
Random test point clouds¶
Two random test objects per class, with the predicted probability of each class. All six are classified correctly.
![]() | ![]() | ![]() | ![]() | ![]() | ![]() | |
| Test index | 523 | 393 | 648 | 644 | 722 | 736 |
| True class | chair | chair | vase | vase | lamp | lamp |
| Predicted class | chair | chair | vase | vase | lamp | lamp |
| P(chair) | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | 0.000 |
| P(vase) | 0.000 | 0.000 | 1.000 | 0.999 | 0.000 | 0.000 |
| P(lamp) | 0.000 | 0.000 | 0.000 | 0.001 | 1.000 | 1.000 |
Failure cases¶
The most confident mistake for each true class.
![]() | ![]() | ![]() | |
| Test index | 595 | 685 | 827 |
| True class | chair | vase | lamp |
| Predicted class | vase | lamp | vase |
| P(chair) | 0.022 | 0.000 | 0.000 |
| P(vase) | 0.942 | 0.000 | 1.000 |
| P(lamp) | 0.036 | 1.000 | 0.000 |
18 of the 20 errors are vases and lamps confused with each other. Chairs are almost never wrong: they are the largest training class and have a distinctive structure. PointNet classifies from one max-pooled global feature with no local information, so it mostly sees an object's overall outline, and vases and lamps often share one.
- Chair predicted as vase (595). A low, wave-shaped lounge chair with no legs, flat seat or upright back. It is the only misclassified chair of 617. It's indeed a difficult case.
- Vase predicted as lamp (685). A very tall, thin, tapering column, which is the outline of a floor lamp. I don't even know what this is, odd looking vase for sure.
- Lamp predicted as vase (827). A wide, squat cylinder with no stem, arm or shade, which is the outline of a pot. I think this is quite a bad case, since lamp shouldn't have that low probability.
Q2. Segmentation Model¶
PointNet segmentation without the transform blocks. Three shared layers (64, 64, 64) give a local feature per point; two more (128, 1024) and a max-pool give a global feature. The global feature is concatenated to every point's local feature (1088 values), and a shared MLP (512, 256, 128, 128, 6) predicts the part label. Trained for 250 epochs with the same settings as Q1. Results use the best checkpoint (epoch 192).
Test accuracy: 0.9036 over all points of the 617 test chairs. Per object, the median is 0.934 and 82 objects are below 0.8.
| Part label (colour in the renders) | Share of test points | Accuracy on points of this part |
|---|---|---|
| seat (red) | 38.1% | 0.925 |
| back (cyan) | 32.0% | 0.926 |
| legs or base (blue) | 20.3% | 0.890 |
| armrests (yellow) | 9.1% | 0.771 |
| rare part (white) | 0.3% | 0.963 |
| headrest (magenta) | 0.1% | 0.170 |
The part names are my reading of the renders; the dataset only gives label numbers.
Segmentation results¶
Three random test objects and the two with the lowest accuracy, with the ground truth above the prediction.
| Object 392 | Object 315 | Object 523 | Object 26 | Object 351 | |
|---|---|---|---|---|---|
| Ground truth | ![]() | ![]() | ![]() | ![]() | ![]() |
| Prediction | ![]() | ![]() | ![]() | ![]() | ![]() |
| Accuracy | 0.811 | 0.948 | 0.868 | 0.483 | 0.489 |
| Selection | random | random | random | lowest accuracy | lowest accuracy |
Interpretation. Conventional chairs are segmented well (315). On 392 and 523 the parts are right and the errors are at their boundaries, for example where the legs of 523 meet the seat. The two bad predictions are unusual chairs: on the sofa-like 26 the wide armrests are labelled seat, and on the barrel-shaped 351 the whole lower half is labelled legs.
Each point is labelled from its own position plus one global feature, with no neighbourhood information. The network therefore learns where each part sits on a typical chair (low points are legs, high points are the back). That is imprecise at part boundaries and fails on unusual layouts. Armrests, which vary most between chairs, are the weakest large part (0.77).
Q3. Robustness Analysis¶
Both experiments use the Q1 and Q2 checkpoints unchanged and modify only the test data. The first column of each accuracy table is the Q1 and Q2 result.
Experiment 1: rotating the input¶
Procedure. Every test point cloud is rotated by a fixed angle about the vertical (y) axis before it is given to the model, using all 10,000 points. As an extra check, objects are also tipped 90° about a horizontal (x) axis.
| Rotation about the vertical axis | 0° | 15° | 30° | 45° | 60° | 90° | 180° |
|---|---|---|---|---|---|---|---|
| Classification accuracy | 0.979 | 0.966 | 0.841 | 0.614 | 0.523 | 0.537 | 0.339 |
| Segmentation accuracy | 0.904 | 0.866 | 0.808 | 0.743 | 0.692 | 0.637 | 0.650 |
Tipped 90° about the x axis: classification 0.228, segmentation 0.269.
Classification samples, with the predicted class and its probability:
| True class | 0° (Q1 result) | 30° | 90° |
|---|---|---|---|
| chair (523) | ![]() predicted chair, P = 1.00 | ![]() predicted chair, P = 1.00 | ![]() predicted chair, P = 1.00 |
| vase (648) | ![]() predicted vase, P = 1.00 | ![]() predicted vase, P = 0.99 | ![]() predicted vase, P = 1.00 |
| lamp (722) | ![]() predicted lamp, P = 1.00 | ![]() predicted lamp, P = 1.00 | ![]() predicted chair (wrong), P = 0.84 |
Segmentation samples, with the accuracy on each object. Points predicted as the rare white label are invisible against the background, which is why part of the back seems to be missing at 90°.
| Object | Ground truth | Prediction, 0° (Q2 result) | Prediction, 30° | Prediction, 90° |
|---|---|---|---|---|
| 392 | ![]() | ![]() accuracy 0.811 | ![]() accuracy 0.732 | ![]() accuracy 0.551 |
| 315 | ![]() | ![]() accuracy 0.948 | ![]() accuracy 0.849 | ![]() accuracy 0.476 |
| 523 | ![]() | ![]() accuracy 0.868 | ![]() accuracy 0.835 | ![]() accuracy 0.534 |
Interpretation. Neither model is robust to rotation. A 15° turn costs little, but classification loses 14 points at 30° and is at chance level (0.34) by 180°. All training objects share one upright, forward-facing pose, and the network has no transform blocks or rotation augmentation, so its features depend on absolute coordinates. Segmentation holds up better about the vertical axis (0.64 against 0.54 at 90°), because height is preserved and height alone separates legs, seat and back fairly well. Tipping the object removes that cue and both tasks collapse. In the samples, the near-symmetric vase is unaffected, while the lamp with a side arm is classified as a chair at 90°.
Experiment 2: reducing the number of points¶
Procedure. Only a random subset of each object's 10,000 points is given to the model (the same indices for every object, fixed seed), with no rotation. Segmentation accuracy is measured on the points that were kept.
| Points per object | 10,000 | 5,000 | 1,000 | 500 | 100 | 50 |
|---|---|---|---|---|---|---|
| Classification accuracy | 0.979 | 0.979 | 0.974 | 0.967 | 0.915 | 0.754 |
| Segmentation accuracy | 0.904 | 0.904 | 0.900 | 0.886 | 0.808 | 0.767 |
Classification samples. Sparse clouds are drawn with larger points.
| True class | 10,000 points (Q1 result) | 500 points | 50 points |
|---|---|---|---|
| chair (523) | ![]() predicted chair, P = 1.00 | ![]() predicted chair, P = 1.00 | ![]() predicted chair, P = 1.00 |
| vase (648) | ![]() predicted vase, P = 1.00 | ![]() predicted lamp (wrong), P = 0.88 | ![]() predicted lamp (wrong), P = 1.00 |
| lamp (722) | ![]() predicted lamp, P = 1.00 | ![]() predicted lamp, P = 1.00 | ![]() predicted lamp, P = 1.00 |
Segmentation samples.
| Object | Ground truth | Prediction, 10,000 points (Q2 result) | Prediction, 500 points | Prediction, 50 points |
|---|---|---|---|---|
| 392 | ![]() | ![]() accuracy 0.811 | ![]() accuracy 0.816 | ![]() accuracy 0.900 |
| 315 | ![]() | ![]() accuracy 0.948 | ![]() accuracy 0.958 | ![]() accuracy 0.780 |
| 523 | ![]() | ![]() accuracy 0.868 | ![]() accuracy 0.830 | ![]() accuracy 0.740 |
Interpretation. Both models are very robust to losing points. With 500 points, 5% of the input, classification drops from 0.979 to 0.967 and segmentation from 0.904 to 0.886. Each point is processed independently, and the global feature is a max over the points, so it is set by a few of them: a random subset that still covers the outline gives nearly the same result. Accuracy only breaks down when the subset no longer outlines the shape (0.754 and 0.767 at 50 points). In the samples, the potted-plant vase is mistaken for a lamp once its foliage thins out to a narrow column.
Q4. Bonus Question: Locality¶
Model¶
I implemented DGCNN (Wang et al., 2019). Its EdgeConv layer finds each point's k = 20 nearest neighbours in the current feature space (recomputed at every layer), applies a shared MLP to every (point, neighbour) pair, and max-pools over the neighbours. Unlike PointNet, every layer therefore sees local shape.
- Classification: four EdgeConv layers (64, 64, 128, 256), concatenated and mapped to 1024; global max-pool and average-pool (2048); MLP (512, 256, 3) with dropout 0.5.
- Segmentation: three EdgeConv layers (64, 64, 64) give a 192-dimensional local feature; a shared layer to 1024 and a max-pool give the global feature, which is concatenated to each point's local feature; shared MLP (256, 256, 128, 6).
Both are simplified from the paper: no spatial transformer, and one MLP layer per EdgeConv.
Settings¶
| Setting | PointNet (Q1 and Q2) | DGCNN classification | DGCNN segmentation |
|---|---|---|---|
| Parameters | 0.81 M and 0.89 M | 1.80 M | 0.63 M |
| Points per object in training | 10,000 | 1,024 | 2,048 |
| Neighbours per point (k) | none | 20 | 20 |
| Optimizer | Adam, learning rate 0.001, constant | same | same |
| Batch size | 32 | 32 | 32 |
| Epochs | 250 | 200, in two stages | 200, in three stages |
| Data augmentation | none | a new random subset of points for every batch | same |
| Dropout | none | 0.5 | 0.5 |
| Activation | ReLU | LeakyReLU (0.2) | LeakyReLU (0.2) |
| Checkpoint reported | best test accuracy (epochs 164 and 192) | best test accuracy (epoch 187) | best test accuracy (epoch 53) |
| Time per epoch, including the test pass | about 6 s and 9 s | about 11 s | about 9 s |
- Fewer points. The cost of the neighbour search grows with the square of the number of points, so DGCNN trains on a random subset, redrawn for every batch.
- Staged training. Classification ran for 60 epochs and was continued from its epoch-50 checkpoint to 200. Segmentation ran for 100 epochs and was continued from epoch 90 and, after an interruption, from epoch 180, to 200. The optimizer state restarted at each continuation.
- Evaluation. DGCNN is tested on a fixed random subset of the size it was trained on. PointNet is tested on the same subset for comparison.
Test accuracy¶
| Task | PointNet, all 10,000 points (Q1, Q2) | PointNet, same subset as DGCNN | DGCNN |
|---|---|---|---|
| Classification (subset of 1,024 points) | 0.9790 | 0.9748 | 0.9853 |
| Segmentation (subset of 2,048 points) | 0.9036 | 0.9031 | 0.9142 |
Given all 10,000 points, DGCNN falls to 0.887 for classification and 0.891 for segmentation, because its 20 neighbours then cover a much smaller region than in training. It has to be tested at its training density.
Classification compared with Q1¶
Errors on the 953 test objects, both models on the same 1,024 points:
| True class | Test objects | PointNet errors | DGCNN errors |
|---|---|---|---|
| chair | 617 | 1 | 1 |
| vase | 102 | 12 | 7 |
| lamp | 234 | 11 | 6 |
| all | 953 | 24 | 14 |
The three Q1 failure cases, then DGCNN's most confident mistake for each class:
| Object 595 | Object 685 | Object 827 | Object 406 | Object 673 | Object 859 | |
|---|---|---|---|---|---|---|
| Input (1,024 points) | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
| True class | chair | vase | lamp | chair | vase | lamp |
| PointNet | vase (wrong) P = 0.83 | lamp (wrong) P = 1.00 | vase (wrong) P = 1.00 | chair P = 0.88 | lamp (wrong) P = 1.00 | lamp P = 1.00 |
| DGCNN | chair P = 1.00 | lamp (wrong) P = 1.00 | vase (wrong) P = 0.70 | vase (wrong) P = 0.63 | lamp (wrong) P = 1.00 | vase (wrong) P = 1.00 |
Segmentation compared with Q2¶
Accuracy by part, both models on the same 2,048 points:
| Part label (colour in the renders) | Share of test points | PointNet accuracy | DGCNN accuracy |
|---|---|---|---|
| seat (red) | 38.1% | 0.921 | 0.931 |
| back (cyan) | 32.0% | 0.926 | 0.939 |
| legs or base (blue) | 20.3% | 0.903 | 0.898 |
| armrests (yellow) | 9.1% | 0.756 | 0.805 |
| rare part (white) | 0.3% | 0.951 | 0.950 |
| headrest (magenta) | 0.1% | 0.162 | 0.033 |
| all points | 100% | 0.903 | 0.914 |
DGCNN is more accurate than PointNet on 397 of the 617 chairs and less accurate on 215. The number of chairs below 0.8 falls from 80 to 68.
The five Q2 objects, then DGCNN's second-worst object (96):
| Object 392 | Object 315 | Object 523 | Object 26 | Object 351 | Object 96 | |
|---|---|---|---|---|---|---|
| Ground truth | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
| PointNet | ![]() accuracy 0.835 | ![]() accuracy 0.946 | ![]() accuracy 0.844 | ![]() accuracy 0.505 | ![]() accuracy 0.452 | ![]() accuracy 0.537 |
| DGCNN | ![]() accuracy 0.829 | ![]() accuracy 0.950 | ![]() accuracy 0.917 | ![]() accuracy 0.528 | ![]() accuracy 0.463 | ![]() accuracy 0.504 |
Interpretation¶
Classification. DGCNN is better: 0.985 against 0.975 on the same points, with errors falling from 24 to 14, almost all in the vase and lamp confusion from Q1. Seeing local shape helps separate classes that share an outline: the lounge chair 595 is now a chair with probability 1.00. It is not a cure, since 685 and 827 are wrong for both models and DGCNN makes new mistakes (406, 859). The margin is only 10 objects, but it is consistent: over the last 50 epochs, test accuracy averaged 0.981 for DGCNN and 0.968 for PointNet.
Segmentation. DGCNN's best checkpoint is 1.1 points better (0.914 against 0.903), mostly from the armrests (0.756 to 0.805) and from sharper part boundaries, as on chair 523 (0.844 to 0.917). Chairs with unusual layouts (26, 351, 96) stay near 0.5 for both models. The caveat is that DGCNN's training was unstable: over its last 50 epochs, test accuracy swung between 0.74 and 0.91, where PointNet stayed between 0.87 and 0.90. The reported checkpoint is from epoch 53 and later epochs never beat it, so the margin, selected on the test set, is optimistic. A decaying learning rate would likely help; I did not test it.
Cost. DGCNN is more expensive per point, has to train on a subset of the points, and only works at that density. PointNet is indifferent to the number of points. Training for longer mattered little: the best classification accuracy was 0.982 after 60 epochs and 0.985 after 200.


































































