Assignment 5

Name: Yuqin Jiao  |  Andrew ID: sjiao2

Q1. Classification Model (30 points)

PointNet classifier (no transformation blocks), trained with Adam for 100 epochs on 10,000 points per object.

Test accuracy: 98.01% Per class: chair 100% (617/617), vase 95.1% (97/102), lamp 94.0% (220/234).

Random test samples (all correct)

Failure cases

The model classifies all 617 test chairs correctly, so there is no chair failure on the original test set. As a substitute, this is a chair that fails after rotating it by 45° about the y axis (from Q3).

Interpretation:

Q2. Segmentation Model (30 points)

PointNet segmentation network: a per-point MLP gives local features, an MLP with max-pooling gives a global feature, and the two are concatenated for every point and passed through a per-point MLP to predict 6 part labels. Trained with Adam (lr 1e-3, batch size 16) for 100 epochs on 10,000 points per object.

Test accuracy: 89.78%

Segmentation results

ObjectGround TruthPredictionAccuracy
#397397 GT397 pred99.71%
#471471 GT471 pred99.45%
#562562 GT562 pred99.43%
#235 (bad)235 GT235 pred38.89%
#351 (bad)351 GT351 pred44.54%
Interpretation:

Q3. Robustness Analysis (20 points)

Both experiments use the models from Q1/Q2 (baselines: 98.01% cls, 89.78% seg).

Experiment 1: Rotating the input point cloud

Each test point cloud (all 10,000 points) is rotated by 0–180° about the x, y or z axis before it goes into the network (--rotation_axis, --rotation_angle). The y axis is the vertical axis of the objects.

Classification accuracy:

Axis \ Angle0°15°30°45°60°90°180°
x98.0%95.5%87.1%61.5%35.8%31.3%69.5%
y98.0%96.3%89.4%74.1%64.7%64.7%33.3%
z98.0%92.7%68.5%40.5%30.2%25.8%68.6%

Segmentation accuracy:

Axis \ Angle0°15°30°45°60°90°180°
x89.8%84.1%73.4%63.1%49.4%25.2%32.6%
y89.8%86.2%78.8%72.6%67.4%57.2%54.7%
z89.8%83.6%71.0%61.1%54.9%45.1%37.7%
Interpretation: accuracy goes down as the angle grows. Classification is robust to rotation about the vertical y axis and not as that robust about z. Under y rotation the vases and lamps stay at about 90%, while chairs prediction accuracy goes down from 100% to 2.3% at 180° (mostly predicted as lamp); a possible reason is that vases and lamps may be symmetric around the vertical axis. It shows that PointNet only works in the orientation it was trained on.

Experiment 2: Varying the number of input points

A random subset of the 10,000 points is used (same subset for all objects, as --num_points). Results are means over 5 random subsets.

Classification accuracy:

Num Points100005000100050010050
Accuracy98.0%97.8%97.3%97.0%94.2%87.3%

Segmentation accuracy:

Num Points100005000100050010050
Accuracy89.8%89.8%89.7%89.0%81.7%76.7%
Interpretation: both models are robust to fewer points: with 1000 points (10%) classification only drops to 97.3% and segmentation stays at 89.7%. Accuracy falls clearly only below about 500 points; at 50 points classification is 87.3% and segmentation 76.7%. Segmentation degrades faster, probably because every point must be labelled and small parts disappear in a sparse cloud (not verified).

Q4. Bonus: Locality (20 points)

I implemented DGCNN, each EdgeConv layer finds the k = 20 nearest neighbours of every point, applies a shared MLP to the edge features, and max-pools over the neighbours, so each point gets local context. The neighbour graph is recomputed in feature space at every layer. Both models are trained and tested with 1024 points (Adam, cosine learning-rate decay, 40 epochs for cls, 60 for seg).
Model#Points at testCls accuracySeg accuracy
PointNet (Q1/Q2)1000098.01%89.78%
PointNet (Q1/Q2)102497.25%89.68%
DGCNN102498.41%91.28%

oth models use 1024 test points.

ObjectGround TruthPointNetDGCNN
Seg #447
54.7% → 98.9%
GTPointNetDGCNN
Discussion: DGCNN is better than PointNet in both tasks at the same number of points (cls 97.25% → 98.41%, seg 89.68% → 91.28%), even compared with PointNet using all 10,000 points. In segmentation it helps most on bulky objects.