16-825 Assignment 5

Name: Yu-Rou Tuan (andrew id: yurout)

Q1. Classification Model

Test accuracy: 0.981

Random test samples

Predicted: chair

Predicted: lamp

Predicted: chair

Predicted: lamp

Failure cases

GT: vase, predicted: lamp

GT: vase, predicted: lamp

GT: lamp, predicted: vase

GT: lamp, predicted: vase

The model got every chair in the test set right, so there's no chair failure on the original data. The chair below is the one it was least sure about, and the second one fails after a 45° rotation (predicted as lamp).

GT: chair, predicted: chair (lowest confidence)

GT: chair, rotated 45°, predicted: lamp

Most mistakes are between vases and lamps. Both are usually tall and round, so the overall shape isn't always enough. The flower in a pot looks a lot like a lamp with a bent arm and a shade, and the tall vase with a lid looks like a lamp base. The other way around, the chandelier has leafy decorations that look like the plants in many vases. Chairs are easy because they have a very distinct shape and make up most of the training data.

Q2. Segmentation Model

Test accuracy: 0.905

Ground truthPredictionAccuracy
0.953
0.960
0.978
0.933
0.439 (bad)
0.480 (bad)

Normal chairs with a clear back, seat and legs are segmented well, and most errors are right at the boundary between two parts. The two bad ones are a barrel-shaped chair and a sofa. They don't have visible legs, but the model still labels the bottom half as legs, like it would for a normal chair. It seems to rely a lot on where a point is (height) rather than what the local shape looks like.

Q3. Robustness Analysis

Experiment 1: Rotation

I rotated the test point clouds around the x axis by different angles and evaluated the same models from Q1 and Q2.

Angle0° (Q1/Q2)30°60°90°
Classification0.9810.8530.3510.295
Segmentation0.9050.7190.4230.257

0°, predicted: chair

30°, predicted: chair

90°, predicted: vase

0°, acc 0.953

30°, acc 0.810

90°, acc 0.344

90°, ground truth

Both models break down quickly once the rotation is more than a small angle. All training shapes are upright, and PointNet works directly on xyz coordinates, so a rotated chair just looks like a different object. In segmentation you can see that the model still labels the bottom as legs and the top as back even when the chair is lying on its side.

Experiment 2: Number of points

I evaluated the models with fewer points per object by randomly sampling a subset of each point cloud.

Points10000 (Q1/Q2)100010010
Classification0.9810.9800.9140.269
Segmentation0.9050.8990.7930.616

10000 points, predicted: lamp

100 points, predicted: lamp

10000 points, acc 0.960

1000 points, acc 0.935

100 points, acc 0.700

This one is much more robust. Going from 10000 to 1000 points barely changes anything, and classification still works pretty well with 100 points. Since the global feature comes from a max pool, only a few key points matter, and a random subset usually still covers the overall shape. It only really fails at very few points, when whole parts like thin legs disappear.

Q4. Bonus: PointNet++

I implemented PointNet++. Instead of one global max pool, it picks center points with farthest point sampling, groups nearby points around each center, and runs a small PointNet on each group. This is repeated at larger scales, so the model learns local features. For segmentation, the features are interpolated back to all points.

PointNetPointNet++
Classification0.9810.985
Segmentation0.9050.919

GT: vase. PointNet: lamp, PointNet++: vase

GT: vase. PointNet: lamp, PointNet++: vase

Ground truthPointNetPointNet++

0.439

0.561

0.960

0.966

PointNet++ is better on both tasks. It fixes some of the vases that PointNet called lamps, and on the barrel chair it gets part of the side right instead of calling the whole bottom legs. On normal chairs the two models look about the same.