16-825 Assignment 5

Q1. Classification Model

Test accuracy: 97.59%

Random test point clouds


Predicted: chair

Predicted: chair

Predicted: vase

Predicted: vase

Predicted: lamp

Predicted: lamp

Failure cases


Ground truth: chair, predicted: lamp

Ground truth: vase, predicted: lamp

Ground truth: lamp, predicted: vase

Chair predicted as lamp: This chair is almost completely flat (it looks like a folded chair), so it has no seat sticking out and no normal leg layout. Without those chair features, its thin flat shape is closer to some lamp parts.

Vase predicted as lamp: This vase is a closed cylinder with a lid, with no plant or open top. It looks like a cylindrical lamp body or lamp shade, which is also a common shape in the lamp class.

Lamp predicted as vase: This lamp is a chandelier with many thin, curved arms. That spread-out, branchy shape looks like the leaves and flowers of the plants in many vase examples, so the global feature is closer to a vase.

Only 1 of 617 chairs is misclassified, compared with 17 of 102 vases and 5 of 234 lamps. Vases and lamps are confused more often because there are fewer of them in training, and both classes contain many round, roughly symmetric shapes.

Q2. Segmentation Model

Test accuracy: 90.34%

ObjectGround truthPrediction
Object 209
Accuracy: 95.44%
Object 123
Accuracy: 91.98%
Object 387
Accuracy: 69.04%
Object 235 (bad prediction)
Accuracy: 48.35%
Object 351 (bad prediction)
Accuracy: 47.12%

Objects 209 and 123 are regular chairs with a clear back, seat and separate legs, so each part is easy to tell apart by its position and the model gets over 90% of the points right. Mistakes there happen only at boundaries between parts, for example where the seat meets the legs.

Object 387 is an armchair, and the model labels parts of the seat and armrests with the wrong part. The armrests and the back are joined together, so the boundary between them is not clear from the point positions.

The two bad predictions (235 and 351) are a sofa and a barrel-shaped chair with no separate legs. The model predicts the bottom part of these chairs as legs, because in most training chairs the bottom part is legs. Since PointNet labels each point using only its own position plus one global feature, it cannot use local shape to see that the bottom is a solid base and not legs.

Q3. Robustness Analysis

Both experiments use the trained Q1 and Q2 models (no retraining) on the full test sets. The 0° / 10000-point column is the Q1 and Q2 result.

Experiment 1: Rotating the input

Procedure: Every test point cloud is rotated about the z axis (the objects stand along y, so this tilts them sideways) by 0°, 30°, 60° and 90° before it is given to the model. All 10000 points are used.

0° (Q1/Q2)30°60°90°
Classification accuracy97.59%75.45%28.23%24.66%
Segmentation accuracy90.34%67.31%50.74%44.27%

Classification (caption: predicted class):

Object0° (Q1/Q2)30°60°90°
393 (chair)
chair

chair

vase

vase
644 (vase)
vase

vase

vase

vase
722 (lamp)
lamp

vase

lamp

lamp

Segmentation, object 209 (caption: accuracy for this object):

0° (Q1/Q2)30°60°90°
Ground truth
Prediction
95.44%

75.69%

50.11%

52.07%

Both models are very sensitive to rotation. Classification drops from 97.6% to 75.5% at 30° and to about 25-28% at 60-90°, which is worse than always predicting chair; at large angles most objects are predicted as vase (for example the tilted chair). Segmentation drops from 90.3% to 44.3% at 90°. In the segmentation example the model still splits the chair into parts, but it assigns labels partly by height, so after tilting the seat and back are labeled with the wrong parts. The models use raw xyz coordinates, all training objects are upright, and there is no rotation augmentation or input transform block, so they never learn rotation invariance.

Experiment 2: Number of points per object

Procedure: Each test object is randomly subsampled (same indices for every object, fixed seed) to 10000, 1000, 100 and 50 points before it is given to the model. No rotation is applied.

10000 (Q1/Q2)100010050
Classification accuracy97.59%96.75%93.07%80.69%
Segmentation accuracy90.34%89.72%81.74%78.86%

Classification (caption: predicted class):

Object10000 (Q1/Q2)100010050
393 (chair)
chair

chair

chair

chair
644 (vase)
vase

lamp

vase

vase
722 (lamp)
lamp

lamp

lamp

lamp

Segmentation, object 209 (caption: accuracy for this object):

10000 (Q1/Q2)100010050
Ground truth
Prediction
95.44%

94.90%

94.00%

88.00%

Both models are robust to the number of points. With 1000 points (10% of the input) accuracy drops by less than 1%, and with only 100 points classification is still 93.1% and segmentation 81.7%. This is because of the max-pool: the global feature only depends on the few points that give the largest value in each channel, so as long as the samples still cover the overall shape, the feature barely changes. At 50 points the shape is no longer covered well (thin parts such as legs and lamp poles are missed), so accuracy drops more, to 80.7% and 78.9%.