16-825 Assignment 5: Point Cloud Classification and Segmentation
Both PointNet models are vanilla PointNet without the T-Net transform blocks. They have a shared per-point MLP (3→64→64→64→128→1024), a symmetric max-pool and an MLP head. They were trained for 50 epochs with Adam (lr 1e-3, batch 32). Each training batch randomly subsamples 2048 of the 10,000 points, which acts as augmentation. All numbers below are on the full test sets with all 10,000 points per object.
1. Classification Model
Test accuracy: 97.48% (929 / 953). Best checkpoint over 50 epochs.
| GT \ Pred | chair | vase | lamp | per-class acc |
|---|---|---|---|---|
| chair (617) | 617 | 0 | 0 | 100.0% |
| vase (102) | 0 | 95 | 7 | 93.1% |
| lamp (234) | 0 | 17 | 217 | 92.7% |
Random test samples
Two random objects per class (seed 0). All six are classified correctly.






Failure cases
The model makes no chair mistakes: all 617 test chairs are correct. For the chair column I show its least confident chair instead. For vase and lamp I show the most confident mistakes.





Interpretation. All 24 errors are confusions between vase and lamp. Chairs have a very distinctive layout (seat plane, back, four thin legs), so they never get confused. Vases and lamps are both mostly rotationally symmetric objects standing on a base, and the failures are the objects that borrow the other class's shape.
- Chair #517 is a solid, boxy, house-shaped chair with no visible legs or open back. That makes it look like a closed container, which explains the 0.29 vase probability.
- Vase #651 is a shallow bowl with a plant arching over it. The plant reads as a lamp arm with a shade at the end.
- Vase #685 is a tall, thin column, which looks like a floor-lamp pole.
- Lamp #922 is a candelabra: a base with several upright stems, much like flowers in a vase.
- Lamp #827 is just a large drum shade with no visible stand. Its global shape is a wide cylinder, which is what a pot or vase looks like.
PointNet only sees a single max-pooled global feature, so it judges by overall silhouette and extent, and these objects have the other class's silhouette.
2. Segmentation Model
Test accuracy (per point, 617 chairs × 10,000 points): 88.22%. The median per-object accuracy is 92.3%. 7.5% of chairs fall below 70%, and the worst is 40.1%.
The architecture is the same encoder as Q1. Each point's 64-d local feature is concatenated with the 1024-d global feature, then passed through a per-point MLP (1088→512→256→128→6).
Part colours:
0
1
2 (back)
3 (arms)
4 (legs / base)
5 (seat).
This is the viz_seg palette, except class 0 is grey instead of white so it shows against the background.
The first three objects are typical cases, taken at the 90th, 50th and 25th percentile of per-object accuracy. The last two are the two worst objects in the test set.
| Object | Type | Ground truth | PointNet prediction | Accuracy |
|---|---|---|---|---|
| #6 | good (90th pct) | ![]() | ![]() | 97.2% |
| #69 | median | ![]() | ![]() | 92.3% |
| #504 | 25th pct | ![]() | ![]() | 84.4% |
| #163 | bad | ![]() | ![]() | 40.1% |
| #577 | bad | ![]() | ![]() | 40.5% |
Interpretation. On ordinary chairs with a clear back, seat and legs (#6, #69), the model is right almost everywhere. The remaining errors sit on the boundaries between parts, such as where the seat meets the back or the legs. PointNet labels each point from its own coordinates plus one global vector, with no notion of neighbourhood, so the cut between two touching parts comes out as a soft, noisy band.
#504 is a sofa. Its wide armrests and thick seat cushion are partly labelled as back or seat, which costs about 15%.
Both bad cases are unusual, fully upholstered, box-like chairs where the parts are not separated geometrically:
- #163 is a solid block whose top cushion is labelled class 1 (magenta). That class barely appears in other chairs, so the model paints it as back (cyan). It also places the seat/base split at a different height than the annotator.
- #577 has a skirted base that the annotation counts as seat (red). The model calls almost all of it legs/base (blue), which is the usual answer for "whatever is below the seat".
In both cases the prediction is a reasonable reading of the shape, but the annotation follows a different convention, and that is the main source of the long tail of bad objects.
3. Robustness Analysis
Both experiments use the Q1/Q2 models without any retraining, evaluated on the full test sets (python q3_robustness.py). The 0° / 10,000-point rows are the Q1/Q2 results. Each grid shows the same samples under every setting: chair #0, vase #650 and lamp #800 for classification, and chairs #0, #1 and #2 for segmentation. Captions in the classification grid are green when the prediction is correct and red when it is wrong. (These grids use the original viz_seg colours, so class 0 is white.)
3.1 Rotating the input
Procedure. Every test cloud is rotated about the x-axis (a forward tilt, since y is up in this data) by 0°, 15°, 30°, 45°, 60° and 90°. The rotation is about the origin, since the clouds are centred. Segmentation labels are unchanged.
| Rotation | cls acc | chair | vase | lamp | seg acc |
|---|---|---|---|---|---|
| 0° (Q1/Q2) | 97.48% | 1.000 | 0.931 | 0.927 | 88.22% |
| 15° | 95.07% | 0.990 | 0.843 | 0.893 | 84.14% |
| 30° | 87.30% | 0.914 | 0.618 | 0.876 | 74.11% |
| 45° | 57.92% | 0.502 | 0.451 | 0.838 | 65.28% |
| 60° | 34.10% | 0.139 | 0.471 | 0.816 | 52.38% |
| 90° | 22.98% | 0.034 | 0.765 | 0.513 | 23.54% |


Interpretation. Without a T-Net and without rotation augmentation, PointNet is not rotation invariant. It learns features from absolute coordinates in the canonical upright frame.
- Small tilts (15°) cost only a few points.
- Past 30°, both tasks collapse. At 90°, only 3% of chairs are recognised. A chair lying on its back has its "seat plane" in the wrong place, so the model votes vase or lamp.
- Segmentation degrades smoothly, because labels are tied to height: after a 90° tilt the "top" points are the back, which the model labels as legs, and vice versa. The 90° grid shows the colour bands staying horizontal while the object turns under them.
- Vase accuracy going up again at 90° is not robustness. It happens because the model starts predicting vase for most inputs.
3.2 Fewer points per object
Procedure. The same random subset of point indices (seed 0) is kept for every object: 10,000, 5,000, 1,000, 500, 100 or 50 points. Segmentation is scored on the kept points.
| # points | cls acc | chair | vase | lamp | seg acc |
|---|---|---|---|---|---|
| 10,000 (Q1/Q2) | 97.48% | 1.000 | 0.931 | 0.927 | 88.22% |
| 5,000 | 97.38% | 1.000 | 0.931 | 0.923 | 88.15% |
| 1,000 | 97.17% | 0.998 | 0.912 | 0.927 | 87.55% |
| 500 | 97.06% | 0.998 | 0.902 | 0.927 | 86.66% |
| 100 | 93.28% | 0.990 | 0.500 | 0.970 | 80.39% |
| 50 | 86.99% | 0.951 | 0.127 | 0.979 | 74.62% |


Interpretation. PointNet is very robust to sparsity. Going from 10,000 down to 500 points (5% of the input) loses only 0.4 points in classification and 1.6 in segmentation. This is the expected behaviour of a max-pooled global feature: only a small "critical set" of points determines each of the 1024 maxima, and random subsampling usually keeps a few points near each of them. Training on random 2048-point subsets helped too.
Below about 100 points, the critical points start to go missing:
- Classification. Vases are hit hardest (0.13 at 50 points). The model falls back to lamp for sparse, ambiguous shapes, which is why lamp accuracy rises.
- Segmentation. Per-point labels stay surprisingly good, because each point still knows its own position. The errors are mostly on thin parts such as arms, where only a couple of points survive.
4. Bonus: Locality with PointNet++
Model
I implemented PointNet++ (single-scale grouping) (Qi et al., 2017) in pointnet2.py. PointNet has no notion of locality: every point is processed independently until a single global max-pool. PointNet++ instead applies a small PointNet to local neighbourhoods, in a hierarchy:
- Set abstraction (SA) layers. Each layer works in three steps:
- Farthest-point sampling picks well-spread centroids.
- Ball query groups up to K neighbours within radius r of each centroid. Coordinates are made relative to the centroid.
- A shared MLP plus max-pool turns each group into one local feature.
- Classification head. An FC head (1024→512→256→3) with BatchNorm and dropout 0.5.
- Segmentation head. Three feature-propagation layers go back up the hierarchy. They interpolate features to the denser level with inverse-distance-weighted 3-NN, concatenate skip features and apply a shared MLP. A per-point classifier follows. Every point's label therefore depends on features of its local neighbourhood at several scales.
- Ball query. It picks K random points inside the ball, not the K nearest. A neighbourhood then covers the same physical area whether the cloud has 2,048 points (training) or 10,000 points (testing).
Training matched PointNet (Adam, lr 1e-3, batch 32, random 2048-point subsets), but ran for 25 epochs instead of 50, because PointNet++ is much slower on my laptop (Apple MPS).
Test accuracy
| Model | Classification | chair / vase / lamp | Segmentation | median obj | chairs < 70% |
|---|---|---|---|---|---|
| PointNet (Q1/Q2), 50 ep | 97.48% | 1.000 / 0.931 / 0.927 | 88.22% | 92.3% | 7.5% |
| PointNet++ (SSG), 25 ep | 97.80% | 0.997 / 0.931 / 0.949 | 90.41% | 94.2% | 4.4% |
Locality helps most for segmentation: +2.2 points overall, 73% of the test chairs improve, and the worst 10% of chairs gain +9.6 points on average. For classification the gain is small (+0.3 points, 21 vs 24 errors), because PointNet was already near the ceiling. PointNet++ fixes 15 of PointNet's mistakes but makes 12 new ones.
Classification: PointNet's failures, revisited






The objects PointNet++ fixes are vases holding flowers or plants (#619, #651, #663) and the candelabra (#922). Their overall silhouette is ambiguous, but the local structure is distinctive: leafy, irregular patches versus thin cylindrical candle arms. The SA layers capture exactly that kind of local structure. The two failures that remain (#685, #827) carry no local cue at all. Both are smooth, plain surfaces whose only evidence is the global shape, so locality cannot help.
Segmentation: same objects as Q2
| Object | Ground truth | PointNet (Q2) | PointNet++ |
|---|---|---|---|
| #6 | ![]() | ![]() 97.2% | ![]() 97.9% |
| #69 | ![]() | ![]() 92.3% | ![]() 93.8% |
| #504 | ![]() | ![]() 84.4% | ![]() 87.8% |
| #163 | ![]() | ![]() 40.1% | ![]() 46.6% |
| #577 | ![]() | ![]() 40.5% | ![]() 45.6% |
PointNet++ improves all five objects. The gain is most visible at part boundaries: the seat/back and seat/leg cuts are sharper and less speckled, because a point's label now comes from features of its neighbourhood rather than from its coordinates alone. The two bad chairs improve by 5–6 points but still fail. Their problem is an annotation convention (a skirted base counted as seat, a rare top-cushion class), which local geometry cannot resolve.
Robustness of PointNet++ (same Q3 experiments)
| Setting | PointNet cls | PointNet++ cls | PointNet seg | PointNet++ seg |
|---|---|---|---|---|
| rotate 30° | 87.30% | 90.98% | 74.11% | 79.31% |
| rotate 60° | 34.10% | 56.77% | 52.38% | 29.94% |
| rotate 90° | 22.98% | 49.84% | 23.54% | 19.08% |
| 1,000 points | 97.17% | 96.64% | 87.55% | 88.97% |
| 500 points | 97.06% | 87.41% | 86.66% | 84.10% |
| 100 points | 93.28% | 24.76% | 80.39% | 50.24% |
Locality is a trade-off.
- Rotation. Local patches with relative coordinates are more rotation tolerant than absolute coordinates, so PointNet++ classification degrades more slowly under rotation. Large-angle segmentation still fails, because the FP layers also use absolute positions.
- Sparsity. PointNet++ is much less robust to sparse inputs. Its fixed radii (0.2 / 0.4) and 512 FPS centroids assume a dense cloud. With 100 points, most balls contain one or two points, so the learned local features see inputs unlike anything in training, and the classifier collapses to "lamp". Random input dropout during training, or the multi-scale / multi-resolution grouping from the paper, would address this.
















