16-825 Assignment 5: Point Cloud Classification and Segmentation

Both PointNet models are vanilla PointNet without the T-Net transform blocks. They have a shared per-point MLP (3→64→64→64→128→1024), a symmetric max-pool and an MLP head. They were trained for 50 epochs with Adam (lr 1e-3, batch 32). Each training batch randomly subsamples 2048 of the 10,000 points, which acts as augmentation. All numbers below are on the full test sets with all 10,000 points per object.

1. Classification Model

Test accuracy: 97.48% (929 / 953). Best checkpoint over 50 epochs.

GT \ Predchairvaselampper-class acc
chair (617)61700100.0%
vase (102)095793.1%
lamp (234)01721792.7%

Random test samples

Two random objects per class (seed 0). All six are classified correctly.

chair 523
#523 GT chair. Pred chair (p = 1.00)
chair 393
#393 GT chair. Pred chair (p = 1.00)
vase 648
#648 GT vase. Pred vase (p = 0.90)
vase 644
#644 GT vase. Pred vase (p = 0.89)
lamp 722
#722 GT lamp. Pred lamp (p = 1.00)
lamp 736
#736 GT lamp. Pred lamp (p = 1.00)

Failure cases

The model makes no chair mistakes: all 617 test chairs are correct. For the chair column I show its least confident chair instead. For vase and lamp I show the most confident mistakes.

chair 517
#517 GT chair. Pred chair, but only p = 0.61 (vase 0.29). The hardest chair.
vase 651
#651 GT vase. Pred lamp (p = 1.00)
vase 685
#685 GT vase. Pred lamp (p = 0.99)
lamp 922
#922 GT lamp. Pred vase (p = 0.98)
lamp 827
#827 GT lamp. Pred vase (p = 0.94)

Interpretation. All 24 errors are confusions between vase and lamp. Chairs have a very distinctive layout (seat plane, back, four thin legs), so they never get confused. Vases and lamps are both mostly rotationally symmetric objects standing on a base, and the failures are the objects that borrow the other class's shape.

PointNet only sees a single max-pooled global feature, so it judges by overall silhouette and extent, and these objects have the other class's silhouette.

2. Segmentation Model

Test accuracy (per point, 617 chairs × 10,000 points): 88.22%. The median per-object accuracy is 92.3%. 7.5% of chairs fall below 70%, and the worst is 40.1%.

The architecture is the same encoder as Q1. Each point's 64-d local feature is concatenated with the 1024-d global feature, then passed through a per-point MLP (1088→512→256→128→6).

Part colours: 0 1 2 (back) 3 (arms) 4 (legs / base) 5 (seat). This is the viz_seg palette, except class 0 is grey instead of white so it shows against the background.

The first three objects are typical cases, taken at the 90th, 50th and 25th percentile of per-object accuracy. The last two are the two worst objects in the test set.

ObjectTypeGround truthPointNet predictionAccuracy
#6good (90th pct)97.2%
#69median92.3%
#50425th pct84.4%
#163bad40.1%
#577bad40.5%

Interpretation. On ordinary chairs with a clear back, seat and legs (#6, #69), the model is right almost everywhere. The remaining errors sit on the boundaries between parts, such as where the seat meets the back or the legs. PointNet labels each point from its own coordinates plus one global vector, with no notion of neighbourhood, so the cut between two touching parts comes out as a soft, noisy band.

#504 is a sofa. Its wide armrests and thick seat cushion are partly labelled as back or seat, which costs about 15%.

Both bad cases are unusual, fully upholstered, box-like chairs where the parts are not separated geometrically:

In both cases the prediction is a reasonable reading of the shape, but the annotation follows a different convention, and that is the main source of the long tail of bad objects.

3. Robustness Analysis

Both experiments use the Q1/Q2 models without any retraining, evaluated on the full test sets (python q3_robustness.py). The 0° / 10,000-point rows are the Q1/Q2 results. Each grid shows the same samples under every setting: chair #0, vase #650 and lamp #800 for classification, and chairs #0, #1 and #2 for segmentation. Captions in the classification grid are green when the prediction is correct and red when it is wrong. (These grids use the original viz_seg colours, so class 0 is white.)

3.1 Rotating the input

Procedure. Every test cloud is rotated about the x-axis (a forward tilt, since y is up in this data) by 0°, 15°, 30°, 45°, 60° and 90°. The rotation is about the origin, since the clouds are centred. Segmentation labels are unchanged.

Rotationcls accchairvaselampseg acc
0° (Q1/Q2)97.48%1.0000.9310.92788.22%
15°95.07%0.9900.8430.89384.14%
30°87.30%0.9140.6180.87674.11%
45°57.92%0.5020.4510.83865.28%
60°34.10%0.1390.4710.81652.38%
90°22.98%0.0340.7650.51323.54%
rotation cls
Classification under rotation.
rotation seg
Segmentation under rotation (GT row, then prediction row with per-object accuracy).

Interpretation. Without a T-Net and without rotation augmentation, PointNet is not rotation invariant. It learns features from absolute coordinates in the canonical upright frame.

3.2 Fewer points per object

Procedure. The same random subset of point indices (seed 0) is kept for every object: 10,000, 5,000, 1,000, 500, 100 or 50 points. Segmentation is scored on the kept points.

# pointscls accchairvaselampseg acc
10,000 (Q1/Q2)97.48%1.0000.9310.92788.22%
5,00097.38%1.0000.9310.92388.15%
1,00097.17%0.9980.9120.92787.55%
50097.06%0.9980.9020.92786.66%
10093.28%0.9900.5000.97080.39%
5086.99%0.9510.1270.97974.62%
num points cls
Classification with fewer points.
num points seg
Segmentation with fewer points.

Interpretation. PointNet is very robust to sparsity. Going from 10,000 down to 500 points (5% of the input) loses only 0.4 points in classification and 1.6 in segmentation. This is the expected behaviour of a max-pooled global feature: only a small "critical set" of points determines each of the 1024 maxima, and random subsampling usually keeps a few points near each of them. Training on random 2048-point subsets helped too.

Below about 100 points, the critical points start to go missing:

4. Bonus: Locality with PointNet++

Model

I implemented PointNet++ (single-scale grouping) (Qi et al., 2017) in pointnet2.py. PointNet has no notion of locality: every point is processed independently until a single global max-pool. PointNet++ instead applies a small PointNet to local neighbourhoods, in a hierarchy:

Training matched PointNet (Adam, lr 1e-3, batch 32, random 2048-point subsets), but ran for 25 epochs instead of 50, because PointNet++ is much slower on my laptop (Apple MPS).

Test accuracy

ModelClassificationchair / vase / lampSegmentationmedian objchairs < 70%
PointNet (Q1/Q2), 50 ep97.48%1.000 / 0.931 / 0.92788.22%92.3%7.5%
PointNet++ (SSG), 25 ep97.80%0.997 / 0.931 / 0.94990.41%94.2%4.4%

Locality helps most for segmentation: +2.2 points overall, 73% of the test chairs improve, and the worst 10% of chairs gain +9.6 points on average. For classification the gain is small (+0.3 points, 21 vs 24 errors), because PointNet was already near the ceiling. PointNet++ fixes 15 of PointNet's mistakes but makes 12 new ones.

Classification: PointNet's failures, revisited

#651 vase. PointNet lamp → PointNet++ vase (0.89)
#619 vase. PointNet lamp → PointNet++ vase
#663 vase. PointNet lamp → PointNet++ vase
#922 lamp (candelabra). PointNet vase → PointNet++ lamp (0.98)
#685 vase (column). Both predict lamp
#827 lamp (drum shade). Both predict vase

The objects PointNet++ fixes are vases holding flowers or plants (#619, #651, #663) and the candelabra (#922). Their overall silhouette is ambiguous, but the local structure is distinctive: leafy, irregular patches versus thin cylindrical candle arms. The SA layers capture exactly that kind of local structure. The two failures that remain (#685, #827) carry no local cue at all. Both are smooth, plain surfaces whose only evidence is the global shape, so locality cannot help.

Segmentation: same objects as Q2

ObjectGround truthPointNet (Q2)PointNet++
#6
97.2%

97.9%
#69
92.3%

93.8%
#504
84.4%

87.8%
#163
40.1%

46.6%
#577
40.5%

45.6%

PointNet++ improves all five objects. The gain is most visible at part boundaries: the seat/back and seat/leg cuts are sharper and less speckled, because a point's label now comes from features of its neighbourhood rather than from its coordinates alone. The two bad chairs improve by 5–6 points but still fail. Their problem is an annotation convention (a skirted base counted as seat, a rare top-cushion class), which local geometry cannot resolve.

Robustness of PointNet++ (same Q3 experiments)

SettingPointNet clsPointNet++ clsPointNet segPointNet++ seg
rotate 30°87.30%90.98%74.11%79.31%
rotate 60°34.10%56.77%52.38%29.94%
rotate 90°22.98%49.84%23.54%19.08%
1,000 points97.17%96.64%87.55%88.97%
500 points97.06%87.41%86.66%84.10%
100 points93.28%24.76%80.39%50.24%

Locality is a trade-off.

PointNet++ classification under rotation.
PointNet++ classification with fewer points.