16-825 Assignment 5: Point Cloud Classification and Segmentation

Training setup

SettingValue
OptimizerAdam, lr 1e-3
Batch size32
LossCross-entropy
PointNet cls (Q1)60 epochs, constant lr
PointNet seg (Q2)60 epochs, cosine decay to 1e-5
PointNet++ cls / seg (Q4)40 epochs, cosine decay
Train pointsRandom 2500 of 10,000 per object each batch (fits 8 GB GPU, light augmentation)
Test points10,000

Q1. Classification Model

Architecture: a shared per-point MLP (3→64→64→64→128→1024, Conv1d + BN + ReLU), then a max-pool over points to get a 1024-d global feature, then FC layers 512→256→3 with BN, ReLU and dropout 0.3.

Test accuracy: 97.17% (926 / 953), best checkpoint at epoch 58.

GT \ Predchairvaselampper-class acc
chair (617)6161099.8%
vase (102)1891287.3%
lamp (234)01322194.4%

Random test samples (all correct)

chair 524
#524: predicted chair (GT chair)
chair 392
#392: predicted chair (GT chair)
vase 645
#645: predicted vase (GT vase)
vase 641
#641: predicted vase (GT vase)
lamp 722
#722: predicted lamp (GT lamp)
lamp 736
#736: predicted lamp (GT lamp)

Failure cases (at least one per class)

chair failure
#406: GT chair, predicted vase
vase failure 1
#618: GT vase, predicted lamp
vase failure 2
#651: GT vase, predicted lamp
lamp failure 1
#726: GT lamp, predicted vase
lamp failure 2
#777: GT lamp, predicted vase

Interpretation. Nearly all errors are vase↔lamp confusions, because vases and lamps share a very similar global structure (a round body with thin parts on top). PointNet has no notion of local structure: it only max-pools independent per-point features, so it cannot pick up the fine local details that tell the two classes apart.

Q2. Segmentation Model

Architecture: per-point MLP 3→64→64 (local feature), then 64→128→1024 followed by a max-pool (global feature). The 1024-d global feature is copied to every point and concatenated with its 64-d local feature (1088-d). A per-point MLP 1088→512→256→128→6 produces the part logits.

Test accuracy: 90.17% (per-point, over 617 chairs × 10,000 points). Per-object accuracy has a median of 93.6%, a minimum of 45.3% and a maximum of 99.6%.

Left = ground truth, right = prediction. Part colors:whitemagentacyan (back)yellow (arms)blue (legs/base)red (seat)

seg 499
#499: accuracy 98.2%
seg 562
#562: accuracy 99.5%
seg 310
#310: accuracy 92.4%
seg 399
#399: accuracy 88.9%
seg 351
#351 (bad): accuracy 45.3%
seg 61
#61 (bad): accuracy 48.0%

Interpretation. Normal chairs (legs, a seat and a back) are segmented almost perfectly; the small errors sit on part boundaries. The bad cases are unusual shapes whose parts are not stacked by height. PointNet has no notion of local structure: each point is labeled from its own position plus one max-pooled global feature, so it mostly learns a height prior ("bottom = legs, middle = seat, top = back") that fails on these shapes.

Q3. Robustness Analysis

Both experiments use the Q1/Q2 checkpoints unchanged (no retraining), evaluated on the full test sets. Rows marked 0° or 10,000 are the Q1/Q2 baselines. The visualizations always show the same objects: classification #524 (chair) and segmentation #499.

Experiment 1: rotating the input

Procedure. Rotate every test point cloud about the z axis (the camera's viewing axis in the renders, so the object tilts sideways) by θ ∈ {0, 15, 30, 45, 60, 90, 135, 180}°, then evaluate. Run with python eval_cls.py --rot_deg 45 --rot_axis z (and likewise for eval_seg.py).

accuracy vs rotation
θcls accseg acc
0° (Q1/Q2)97.2%90.2%
15°89.3%84.2%
30°61.5%67.7%
45°38.7%57.2%
60°26.8%51.1%
90°25.5%40.9%
135°24.3%30.7%
180°63.7%33.2%
cls #524 at 30°: predicted chair
cls #524 at 60°: predicted vase
cls #524 at 90°: predicted vase
seg #499 at 30°: 65.6% (98.2% unrotated)
seg #499 at 60°: 54.1%
seg #499 at 90°: 34.7%

Interpretation. PointNet without the T-Net is not rotation-invariant. Max-pooling only removes the point order, not the orientation: the first layer is a linear function of the raw xyz, so a rotated chair produces different features, and every training shape is upright. Accuracy drops sharply after 15–30°; at 60° classification is 26.8%, far below always guessing "chair" (64.7%), because a tilted chair loses the "legs below, back above" layout. Segmentation still labels points by their new height, so the horizontal bands cut across the tilted chair (#499 at 90°). Rotation augmentation, the T-Net, or rotation-invariant input features would help.

Experiment 2: number of input points

Procedure. Evaluate with N ∈ {10000, 5000, 1000, 500, 100, 50, 10} points per object, using one fixed random subset of indices shared by all objects. Run with python eval_cls.py --num_points 100 (and likewise for eval_seg.py).

accuracy vs number of points
Ncls accseg acc
10000 (Q1/Q2)97.2%90.2%
500097.3%90.3%
100096.7%90.1%
50097.0%89.1%
10093.5%80.3%
5076.1%74.3%
1025.6%55.4%
cls #524, N=1000: predicted chair
cls #524, N=100: predicted chair
cls #524, N=50: predicted chair
seg #499, N=1000: 97.7%
seg #499, N=100: 92.0%
seg #499, N=50: 62.0%

Interpretation. PointNet is very robust to sparser inputs. Down to 500 points (5% of the original) both tasks lose less than 1.1 points of accuracy. This follows from the architecture: the global feature is a max over per-point features, and with a few hundred well-spread points the critical points that set each max are still nearly present. Training on random 2500-point subsets also helped. Below about 100 points the max-pooled descriptor loses the critical points for thin structures such as legs and lamp stems. Classification collapses at 10 points (25.6%): 943 of 953 objects are called lamp. Ten scattered points carry essentially no shape information, so the max-pooled feature falls into one region of feature space. Segmentation degrades more gracefully (55% at 10 points), because even with a poor global code the per-point height prior still assigns many points to the large back and seat parts.

Q4. Bonus: Locality with PointNet++

Model

I implemented PointNet++ with single-scale grouping (pn2_cls_model / pn2_seg_model in models.py), using PyTorch3D's sample_farthest_points, ball_query and knn_points. PointNet applies one global max-pool to independent points. PointNet++ instead builds features hierarchically from local neighborhoods:

Results

Modelcls test accchair / vase / lampseg test accseg per-object median
PointNet (Q1/Q2)97.17%99.8 / 87.3 / 94.490.17%93.6%
PointNet++ (SSG)97.69%99.7 / 90.2 / 95.791.42%95.0%

Farthest-point sampling uses a random start point, so PointNet++ test accuracy varies a little between runs. The best epoch scored 98.2% during training, and a fresh evaluation of the same checkpoint gives the 97.69% above.

Classification: PointNet's failure cases under PointNet++

PointNet++ makes 22 errors versus PointNet's 27, and only 9 of them are shared. It gets 4 of the 5 Q1 failure cases right:

objectGTPointNetPointNet++
#406chairvaselamp
#618vaselampvase
#651vaselampvase
#726lampvaselamp
#777lampvaselamp

The GIFs are the same as in Q1. The fixed cases are exactly the ones where the deciding evidence is a small local structure: the orchid's leaves and flower (#651), the shade-and-socket of the round-base table lamp (#726), and the chandelier's bulbs and arms (#777). PointNet's single global max-pool mixes those details away. PointNet++ encodes them first in local neighborhoods. The boxy chair #406 is still wrong, because it has no locally chair-like parts at all.

Segmentation comparison

Each GIF: ground truth (left) | PointNet++ prediction (right). Compare with the PointNet GIFs for the same objects in Q2.

#61: PointNet 48.0% → PointNet++ 77.9%. It now finds the arms (yellow) and the seat (red) inside the boxy frame, where PointNet painted the whole lower half as seat.
#399: PointNet 88.9% → PointNet++ 95.0%. The thin arm rests are cleanly separated from the back.
#351: PointNet 45.3% → PointNet++ 47.9%. The barrel chair is still mostly wrong. Its parts are fused into one smooth shell, so local geometry gives no boundary cue either.
#310: PointNet 92.4% → PointNet++ 79.0%. A regression: seat and arm labels spread into the lower frame and the side panels.
#499: 98.2% → 98.3%
#562: 99.5% → 99.5%

Interpretation. Locality helps most where the answer depends on fine local structure: thin parts (arm rests, legs, lamp stems) and part boundaries. This is where the per-object median goes up, from 93.6% to 95.0%. Overall, though, the gain is small, since both models still end with a global feature and ordinary chairs were already easy for PointNet.

Trade-off: point density. Locality needs enough points to work. With 1000 points PointNet++ still holds up (cls 96.9%, seg 88.5%, vs. PointNet's 96.7% and 90.1%). With 100 points it collapses (cls 26.4%, seg 32.4%), while PointNet keeps 93.5% and 80.3%. The ball-query radii work for dense input, but with sparse input each neighborhood has only one or two points. PointNet's per-point features don't depend on neighbors, so it handles both dense and sparse input. Training with random point dropout, or the multi-scale grouping from the PointNet++ paper, should fix this.