Assignment 5: Point Cloud Classification and Segmentation

16-825 Learning for 3D Vision · Yongyi Xiong (andrewid: yongyix)

Q1. Classification Model

PointNet classifier without the transformation blocks, trained for 250 epochs on all 10000 points per object (batch size 32). The checkpoint with the best test accuracy is used for evaluation.

ModelTest accuracy
PointNet (best epoch)97.80%
Q1 training curves
Training loss and test accuracy over 250 epochs.

Random test samples

#418
#418 · GT chair · Pred chair
#84
#84 · GT chair · Pred chair
#623
#623 · GT vase · Pred vase
#736
#736 · GT lamp · Pred lamp

Failure cases

#685
#685 · GT vase · Pred lamp
#827
#827 · GT lamp · Pred vase

Chair failure case: none exists. All 617 test chairs are classified correctly (617/617), so there is no chair failure to visualize. The failure cases above cover the other two classes.

Interpretation

Both PointNet failures are confusions between vase and lamp, the two classes whose shapes overlap the most. Vase #685 is tall, thin and slightly tapered with a narrow neck at the top. That silhouette looks much more like the pole or base of a lamp than the round body of a typical vase, so the model predicts lamp. Lamp #827 is the opposite case: it is a wide, short, hollow drum, essentially a lamp shade with no visible stand, and an open cylinder is exactly the shape of many vases.

PointNet describes the whole object with one max pooled global feature, so its decision depends mostly on the overall silhouette. When the silhouette of one class resembles the other, there is no part level cue (such as a bulb, a stand or a handle) to correct it. Chairs are never misclassified because legs, seat and back form a very distinctive structure, and chairs are also the majority class (617 of the 953 test objects), so the model has seen the widest variety of them. All errors therefore fall on the two smaller classes: per class accuracy is chair 617/617 (100%), vase 92/102 (90.2%) and lamp 223/234 (95.3%). Ten vases are predicted as lamp and eleven lamps as vase, and no object is confused with a chair.

Q2. Segmentation Model

PointNet segmentation network (per point local feature concatenated with the global max pooled feature), trained for 250 epochs on all 10000 points per chair.

ModelTest accuracy (per point)
PointNet (best epoch)90.32%
Q2 training curves
Training loss and test accuracy over 250 epochs.

Ground truth (left) vs prediction (right)

ChairAccuracyGT  |  Prediction
#505 (good)99.58%
gt 505pred 505
#397 (good)99.51%
gt 397pred 397
#297 (good)99.46%
gt 297pred 297
#093.14%
gt 0pred 0
#351 (bad)47.23%
gt 351pred 351
#235 (bad)46.66%
gt 235pred 235

#505, #397 and #297 are the three best chairs in the test set; #351 and #235 are the two worst.

Interpretation

Colors follow the label ids in viz_seg: cyan (2) is the back, yellow (3) the armrests, blue (4) the chair base (legs and other supports) and red (5) the seat. The three good chairs (#505, #397, #297) are standard chairs with a back, a thin seat and four legs at the usual heights, and the part boundaries are almost perfect. Chair #0 (93%) loses points mainly along the thin seat, where the seat meets the back and the base.

The two bad chairs have unusual shapes. #351 is a barrel chair with a closed cylindrical body and no separate base. In its ground truth the side panels are mostly yellow (armrests), the curved back is mostly cyan, and no point is labeled as base. PointNet nevertheless predicts 37.01% of the points as base, turning the lower part of the barrel into a blue band. #235 is a wide sofa-like chair. PointNet predicts horizontal bands and assigns the red seat label to 3,344 of the 6,165 points marked yellow (armrests) in the ground truth.

Both failures suggest that the network largely segments by height. The global feature tells it that the object is a chair, and the per point feature mostly encodes where the point sits vertically. This works whenever parts are at typical heights, and breaks when a chair puts different parts at the same height (armrests next to the seat) or has no separate base at all.

Q3. Robustness Analysis

Both experiments reuse the Q1 and Q2 checkpoints without retraining and only change the test inputs. All runs use seed 42 for point subsampling; the four random cls samples are the same objects as in Q1. Every setting is evaluated on the full test set (953 objects for cls, 617 chairs for seg). The sweep is scripted in run_q3_sweep.py.

3.1 Rotating the input

Procedure. Every test point cloud is rotated about the x axis (through the origin) by θ ∈ {0, 15, 30, 45, 60, 90} degrees before being fed to the model, using --rot_deg θ --rot_axis x. The labels are unchanged.

Rotation about xcls accuracyseg accuracy
0° (Q1 / Q2)97.80%90.32%
15°93.91%84.98%
30°78.17%74.06%
45°44.39%62.24%
60°34.52%41.84%
90°26.65%27.93%

Classification samples

#84 at 0 deg
0° · GT chair · Pred chair
#84 at 30 deg
30° · GT chair · Pred lamp
#84 at 60 deg
60° · GT chair · Pred lamp
#84 at 90 deg
90° · GT chair · Pred lamp
#418 at 0 deg
0° · GT chair · Pred chair
#418 at 30 deg
30° · GT chair · Pred chair
#418 at 60 deg
60° · GT chair · Pred vase
#418 at 90 deg
90° · GT chair · Pred vase

The vase #623 and the lamp #736 stay correct at every angle (see runs/q3/cls_rotx*).

Segmentation samples (chairs #0 and #397, GT left, prediction right)

gt 0 degpred 0 deg
0° · 93.14%
gt 30 degpred 30 deg
30° · 63.90%
gt 60 degpred 60 deg
60° · 36.54%
gt 90 degpred 90 deg
90° · 39.96%
gtpred
#397 · 0° · 99.51%
gtpred
#397 · 30° · 82.56%
gtpred
#397 · 60° · 41.79%
gtpred
#397 · 90° · 30.59%

Interpretation. Accuracy falls quickly with rotation: 15° already costs about 4 points (cls) and 5 points (seg), and at 45° classification is below 50%. PointNet without the T-Net has no rotation invariance. The shared MLP acts directly on absolute xyz coordinates, and every training object is upright and consistently aligned, so the learned features are tied to that canonical pose.

In classification, tilted chairs are the first to fail: #84 is predicted as lamp from 30° on, and #418 as vase from 60° on, while the vase #623 and the lamp #736 stay correct at every angle. In segmentation, the predicted labels stay arranged in roughly horizontal bands in the world frame instead of rotating with the chair. At 60° the lower part of the tilted back is labeled as base and its top as armrests, which again shows that the model assigns parts mostly by height. Training with random rotation augmentation, or adding the T-Net to learn a canonical alignment, would be the natural fix.

3.2 Number of points per object

Procedure. Each test object is subsampled to N ∈ {10000, 5000, 1000, 500, 100, 50} points with --num_points N. The same random subset of indices (seed 42) is used for every object, and for seg the labels are indexed with the same subset.

Points per objectcls accuracyseg accuracy
10000 (Q1 / Q2)97.80%90.32%
500097.59%90.35%
100097.27%90.01%
50097.27%88.85%
10094.33%82.99%
5087.20%80.75%

Classification samples (chair #418 and vase #623)

10000 points
10000 pts · Pred chair
1000 points
1000 pts · Pred chair
100 points
100 pts · Pred chair
50 points
50 pts · Pred chair
vase 10000 points
10000 pts · GT vase · Pred vase
vase 1000 points
1000 pts · GT vase · Pred vase
vase 100 points
100 pts · GT vase · Pred vase
vase 50 points
50 pts · GT vase · Pred vase

Segmentation samples (chairs #0 and #397, GT left, prediction right)

gt 10000pred 10000
10000 pts · 93.14%
gt 1000pred 1000
1000 pts · 94.00%
gt 100pred 100
100 pts · 96.00%
gt 50pred 50
50 pts · 94.00%
gtpred
#397 · 10000 pts · 99.51%
gtpred
#397 · 1000 pts · 99.30%
gtpred
#397 · 100 pts · 99.00%
gtpred
#397 · 50 pts · 100.00%

Interpretation. The model is very robust to the number of points. Down to 500 points (5% of the training density) classification loses only 0.5 points and segmentation 1.5 points. The global feature is a max pool over all points, and it is determined by a small set of critical points; a random subset still contains points close to them, so the pooled feature barely changes. Segmentation holds up for the same reason, since each label depends on the point's own feature plus that global feature.

Below 100 points, thin structures such as legs and the seat edge are covered by only a handful of points, the global feature starts to change, and accuracy drops to 87.20% (cls) and 80.75% (seg). At 50 points, chairs #0 and #397 score 94% and 100%, respectively, while overall segmentation accuracy drops to 80.75%. With so few points a single chair is a very noisy estimate, so the full test set accuracy is the reliable measure.

Q4. Bonus: Locality with DGCNN

Model

I implemented a Dynamic Graph CNN (DGCNN) in dgcnn.py. Its building block is EdgeConv: for each point xi, find its k = 20 nearest neighbors, build edge features [xi, xj − xi], apply a shared Conv2d + BatchNorm + LeakyReLU(0.2), and max pool over the k neighbors. The graph is dynamic: the second EdgeConv recomputes the kNN graph in the 64 dimensional feature space of the first layer.

StageClassificationSegmentation
EncoderEdgeConv 3→64, EdgeConv 64→64, concatenated to 128 dim per point
EmbeddingConv1d 128→1024 + BN + LeakyReLU
Global featuremax pool ⊕ mean pool → 2048max pool (1024), broadcast and concatenated with the 128 dim local features → 1152 per point
HeadFC 2048→512→256→3 (BN, LeakyReLU, dropout 0.5)Conv1d 1152→256→256→128→6 (BN, LeakyReLU)
Parameters1.32 M0.54 M
Points per object10242048
Epochs100100

This is a lighter version of the original DGCNN, whose classification model stacks four EdgeConv layers and whose part segmentation model uses three; two layers keep the training time within a Colab session. Because every EdgeConv needs an N × N distance matrix and an edge tensor of size B × 2C × N × k, cost grows quickly with N. I therefore train and evaluate on random subsets of 1024 points for classification and 2048 points for segmentation, matching the paper's main ModelNet40 experiments (1024 points) and its ShapeNet part training (2048 points).

Why locality should help. PointNet processes every point independently until the final max pool, so it never sees how a point relates to its neighbors. EdgeConv computes features from each point's neighborhood through xj − xi, which captures local geometry such as thin versus flat structures, surface direction and part boundaries. Since the second layer rebuilds the graph in feature space, a point can also aggregate from points that are far away in space but play a similar role, such as the two armrests.

Test accuracy

For a fair comparison the Q1/Q2 PointNet checkpoints are also evaluated on the same 1024 / 2048 point subsets that DGCNN sees.

TaskPointNet, 10000 pts (Q1 / Q2)PointNet, same pts as DGCNNDGCNN
cls (1024 pts)97.80%97.17%98.43%
seg (2048 pts)90.32%90.34%91.24%

Training budget differs (PointNet 250 epochs, DGCNN 100). Within its first 100 epochs, PointNet's best test accuracy is 97.80% (cls) and 89.60% (seg). Averaged over epochs 90 to 99 of each run, DGCNN reaches 97.72% (cls) and 88.74% (seg), versus 97.03% and 87.41% for PointNet. So DGCNN stays ahead under the same epoch budget, but the two runs also differ in architecture and input point count, and these checks do not isolate the cause of the gap.

DGCNN cls curves
DGCNN classification training curves.
DGCNN seg curves
DGCNN segmentation training curves.

Classification: comparison with Q1

The four random samples are the same objects as in Q1 and are all classified correctly by DGCNN. Failure cases of DGCNN:

#418
#418 · GT chair · Pred chair
#623
#623 · GT vase · Pred vase
#673
#673 · GT vase · Pred lamp
#750
#750 · GT lamp · Pred vase

All 617 test chairs are classified correctly. Fourteen errors are confusions between vase and lamp, and one vase (#663) is predicted as a chair.

Segmentation: comparison with Q2 (same chairs)

ChairPointNet @10000PointNet @2048DGCNN @2048GTPointNet pred (Q2)DGCNN pred
#093.14%93.55%94.63%gt 0pointnet 0dgcnn 0
#23546.66%47.61%51.61%gt 235pointnet 235dgcnn 235
#35147.23%44.97%47.85%gt 351pointnet 351dgcnn 351
#50599.58%99.76%98.78%gt 505pointnet 505dgcnn 505
#39799.51%99.41%99.61%gt 397pointnet 397dgcnn 397
#29799.46%99.76%99.76%gt 297pointnet 297dgcnn 297

GT and PointNet gifs show all 10000 points; the DGCNN gif shows the 2048 points it was evaluated on, so it looks sparser. DGCNN GT renders on the same 2048 points are in images/q4/gt_q4_dgcnn_*.gif.

Interpretation

DGCNN improves both tasks while seeing fewer points and training for fewer epochs: +1.26 points for classification and +0.90 points for segmentation over PointNet on the same point subsets. Per class, DGCNN gets chair 617/617, vase 95/102 (93.1%) and lamp 226/234 (96.6%), compared with 617/617, 89/102 (87.3%) and 220/234 (94.0%) for PointNet on the same 1024 points. The gain is concentrated on vase versus lamp: confusions between these two classes drop from 26 to 14.

Of the 15 DGCNN classification errors, 14 are still vase versus lamp (6 vases predicted as lamp, 8 lamps predicted as vase) and one vase is predicted as chair. The vase and lamp failures look like the PointNet ones: #673 is a tall thin vase like #685, and #750 is a round, sparse globe lamp that resembles a round vase. Locality cannot help when the whole silhouette is ambiguous.

In segmentation both models are almost perfect on the standard chairs. The gains come from the unusual ones. On the same 2048 points, PointNet scores 47.61% on #235 and 44.97% on #351, versus 51.61% and 47.85% for DGCNN. On #235 DGCNN recovers more armrests: it labels 44.9% of the ground truth armrest points correctly, compared with 31.1% for PointNet. On #351 the two models are much closer (35.5% vs 34.7% of armrest points), and DGCNN still predicts 36.9% of the points as base although the ground truth has none, so both stay below 50%. Local edge features help separate parts that sit at the same height but have different local geometry (an armrest next to the seat), which is exactly where the height based behavior of PointNet fails. The cost is speed: the kNN search adds N × N work per layer, so DGCNN is slower per step and had to use fewer input points.