Assignment 5: Point Cloud Classification and Segmentation

ModelClassification accuracySegmentation accuracy
PointNet (Q1 / Q2)97.17%87.20%
PointNet++ (Q4)97.48%90.80%

Q1. Classification Model

Test accuracy: 97.17%

Classchairvaselamp
Accuracy99.68%91.18%93.16%
Test objects617102234

Random test point clouds

object 233object 636object 213object 222object 406object 496
GT: chairGT: vaseGT: chairGT: chairGT: chairGT: chair
Pred: chairPred: vasePred: chair Pred: chairPred: vasePred: chair

Failure cases

Ground truthchairvaselamp
Object406543651663726750
Prediction vaselamplamp lampvasevase

Interpretation.
Most of the failures are between vases and lamps, whose overall shape looks like each other. The chair class is almost never wrong (2 of 617).
Vases and Lamps have similar overall shapes and silhouettes. Since PointNet summarizes an object with a single max pooled global feature, it mostly sees the coarse outline which is similar in both classes and cannot use the other fin details that would tell the classes apart.

Q2. Segmentation Model

Test accuracy: 87.20%

Segmentation results

Colors: cyan = back, red = seat, blue = legs, yellow = arms, magenta / white = the two remaining (rare) part classes.

ObjectGround truthPredictionAccuracy
397 99.34%good
490 99.25%good
89 90.44%good
26 36.42%bad
235 42.98%bad

Interpretation.
The model works very well on ordinary chairs with a clear back, a flat seat and separate legs. The PointNet network has mostly learned where a part usually lies inside the object (for example, low points are legs), because the per-point feature is only computed from the point's own coordinates plus a global shape code. This works for the common chair shapes and fails for rare shapes where the same position belongs to a different part.

Q3. Robustness Analysis

Both experiments use the Q1 and Q2 checkpoints without any retraining.

Experiment 1: Rotation

Procedure. Every test point cloud is rotated around the z axis (the objects are upright along y, so this tilts the objects sideways) by a fixed angle before it is passed to the model

Rotation (degrees)0 (Q1 / Q2)1530456090180
Classification accuracy97.17%94.44%75.55%37.57%29.91%26.86%43.55%
Segmentation accuracy87.20%82.56%70.73%61.61%54.73%43.18%35.88%

Classification samples

Object0 degrees (Q1)30 degrees90 degrees
233
GT: chair
Pred: chair Pred: vase Pred: vase
636
GT: vase
Pred: vase Pred: vase Pred: vase
800
GT: lamp
Pred: lamp Pred: lamp Pred: vase

Segmentation samples

Object0 degrees (Q2)30 degrees90 degrees
GTPredGTPredGTPred
397 99.34% 76.95% 34.17%
89 90.44% 71.95% 62.07%

Interpretation.