Name: Yu-Rou Tuan (andrew id: yurout)
Test accuracy: 0.981

Predicted: chair

Predicted: lamp

Predicted: chair

Predicted: lamp

GT: vase, predicted: lamp

GT: vase, predicted: lamp

GT: lamp, predicted: vase

GT: lamp, predicted: vase
The model got every chair in the test set right, so there's no chair failure on the original data. The chair below is the one it was least sure about, and the second one fails after a 45° rotation (predicted as lamp).

GT: chair, predicted: chair (lowest confidence)

GT: chair, rotated 45°, predicted: lamp
Most mistakes are between vases and lamps. Both are usually tall and round, so the overall shape isn't always enough. The flower in a pot looks a lot like a lamp with a bent arm and a shade, and the tall vase with a lid looks like a lamp base. The other way around, the chandelier has leafy decorations that look like the plants in many vases. Chairs are easy because they have a very distinct shape and make up most of the training data.
Test accuracy: 0.905
| Ground truth | Prediction | Accuracy |
|---|---|---|
![]() | ![]() | 0.953 |
![]() | ![]() | 0.960 |
![]() | ![]() | 0.978 |
![]() | ![]() | 0.933 |
![]() | ![]() | 0.439 (bad) |
![]() | ![]() | 0.480 (bad) |
Normal chairs with a clear back, seat and legs are segmented well, and most errors are right at the boundary between two parts. The two bad ones are a barrel-shaped chair and a sofa. They don't have visible legs, but the model still labels the bottom half as legs, like it would for a normal chair. It seems to rely a lot on where a point is (height) rather than what the local shape looks like.
I rotated the test point clouds around the x axis by different angles and evaluated the same models from Q1 and Q2.
| Angle | 0° (Q1/Q2) | 30° | 60° | 90° |
|---|---|---|---|---|
| Classification | 0.981 | 0.853 | 0.351 | 0.295 |
| Segmentation | 0.905 | 0.719 | 0.423 | 0.257 |

0°, predicted: chair

30°, predicted: chair

90°, predicted: vase

0°, acc 0.953

30°, acc 0.810

90°, acc 0.344

90°, ground truth
Both models break down quickly once the rotation is more than a small angle. All training shapes are upright, and PointNet works directly on xyz coordinates, so a rotated chair just looks like a different object. In segmentation you can see that the model still labels the bottom as legs and the top as back even when the chair is lying on its side.
I evaluated the models with fewer points per object by randomly sampling a subset of each point cloud.
| Points | 10000 (Q1/Q2) | 1000 | 100 | 10 |
|---|---|---|---|---|
| Classification | 0.981 | 0.980 | 0.914 | 0.269 |
| Segmentation | 0.905 | 0.899 | 0.793 | 0.616 |

10000 points, predicted: lamp

100 points, predicted: lamp

10000 points, acc 0.960

1000 points, acc 0.935

100 points, acc 0.700
This one is much more robust. Going from 10000 to 1000 points barely changes anything, and classification still works pretty well with 100 points. Since the global feature comes from a max pool, only a few key points matter, and a random subset usually still covers the overall shape. It only really fails at very few points, when whole parts like thin legs disappear.
I implemented PointNet++. Instead of one global max pool, it picks center points with farthest point sampling, groups nearby points around each center, and runs a small PointNet on each group. This is repeated at larger scales, so the model learns local features. For segmentation, the features are interpolated back to all points.
| PointNet | PointNet++ | |
|---|---|---|
| Classification | 0.981 | 0.985 |
| Segmentation | 0.905 | 0.919 |

GT: vase. PointNet: lamp, PointNet++: vase

GT: vase. PointNet: lamp, PointNet++: vase
| Ground truth | PointNet | PointNet++ |
|---|---|---|
![]() | ![]() 0.439 | ![]() 0.561 |
![]() | ![]() 0.960 | ![]() 0.966 |
PointNet++ is better on both tasks. It fixes some of the vases that PointNet called lamps, and on the barrel chair it gets part of the side right instead of calling the whole bottom legs. On normal chairs the two models look about the same.