Name: Hiroki Mii
Andrew ID: hmii
Test Accuracy: 0.980
Correct Cases
Failure Cases
The failure cases are pretty justified. Each of the failure cases involves an ambiguous looking object that even a human can mistaken. Failure case 1 and 2 are both atypical looking chairs and vases, and case 3's lamp could easily be mistaken for a vase.
Test Accuracy: 0.904
Below includes the 3 highest accuracies, followed by the 2 lowest. Prediction on left, ground truth on right.
Accuracy: 0.996
Accuracy: 0.995
Accuracy: 0.993
Accuracy: 0.497
Accuracy: 0.461
The high accuracy results clearly segment the different parts of the chair. The back, seat, and legs are distinct and separated, leading to a clean segmentation. However, some chairs have more ambiguous separation between the parts, like the couch chairs that led to poor accuracies. Unlike the first three chairs, the legs on the couch chairs are very small and hard to identify. The segmentation was off on the legs for both chair #4 and #5. Additionally, chair #4 had a low armrest that was completely missed, which the model thought was part of the seat.
| Axis | 0° (baseline) | 90° | 180° | 270° |
|---|---|---|---|---|
| x | 0.980 | 0.341 | 0.665 | 0.439 |
| y | 0.980 | 0.700 | 0.443 | 0.636 |
| z | 0.980 | 0.265 | 0.665 | 0.337 |
Point clouds were rotated 90, 180, and 270 degrees about the x-axis, y-axis, and z-axis.
PointNet is permutation invariant, but not rotation invariant. Since the model was trained only on upright objects, with no transformation blocks or rotation augmentation, it struggles when the orientation differs substantially from the training data.
For x-axis and z-axis rotations, the accuracies drop significantly for 90 and 270 degree rotations since the objects are flipped over on its sides. For 180 degree rotations, the accuracies drop, but less drastically. The model can still recognize objects that are flipped upside down to an extent.
For y-axis rotations, the accuracies also drop, but milder since the objects still remain in an upright position. The model can still classify objects turned 90 and 270 degrees, but when it's completely turned around in a 180 degree position, it struggles.
In the visualizations, vase and lamp are constantly confused between each other, while chair jumps around in predictions.
Test Accuracy: 0.341
Test Accuracy: 0.665
Test Accuracy: 0.439
Test Accuracy: 0.700
Test Accuracy: 0.443
Test Accuracy: 0.636
Test Accuracy: 0.265
Test Accuracy: 0.665
Test Accuracy: 0.337
| Axis | 0° (baseline) | 90° | 180° | 270° |
|---|---|---|---|---|
| x | 0.904 | 0.283 | 0.336 | 0.204 |
| y | 0.904 | 0.580 | 0.586 | 0.658 |
| z | 0.904 | 0.411 | 0.349 | 0.408 |
Point clouds were rotated 90, 180, and 270 degrees about the x-axis, y-axis, and z-axis.
Like classification, point cloud rotations drop segmentation accuracies because PointNet is not rotation invariant. However, the drop in accuracy is worse for segmentation than classification. For classification, when the model is not confident, it can still take a guess and have a chance of getting it right. Segmentation is harder because the model needs to know which part belongs where in a chair.
Like classification, y-axis rotations did less worse than x-axis and z-axis rotations, since the chairs are still in an upright position. The back, seat, and legs are in expected positions relative to each other that the model saw during training.
It is interesting to see that for 180 degree rotations for x-axis and z-axis, the model confuses the back and legs since the chair is flipped upside down. The seat is often detected correctly.
Test Accuracy: 0.283
Accuracy: 0.057
Accuracy: 0.128
Accuracy: 0.293
Test Accuracy: 0.336
Accuracy: 0.423
Accuracy: 0.261
Accuracy: 0.424
Test Accuracy: 0.204
Accuracy: 0.184
Accuracy: 0.164
Accuracy: 0.042
Test Accuracy: 0.580
Accuracy: 0.655
Accuracy: 0.503
Accuracy: 0.665
Test Accuracy: 0.586
Accuracy: 0.534
Accuracy: 0.242
Accuracy: 0.644
Test Accuracy: 0.658
Accuracy: 0.782
Accuracy: 0.534
Accuracy: 0.830
Test Accuracy: 0.411
Accuracy: 0.539
Accuracy: 0.476
Accuracy: 0.372
Test Accuracy: 0.349
Accuracy: 0.364
Accuracy: 0.412
Accuracy: 0.476
Test Accuracy: 0.408
Accuracy: 0.642
Accuracy: 0.441
Accuracy: 0.353
| Points | 10000 (baseline) | 5000 | 1000 | 500 | 100 | 50 |
|---|---|---|---|---|---|---|
| Test accuracy | 0.980 | 0.978 | 0.967 | 0.969 | 0.809 | 0.510 |
The number of points evaluated was decreased from its baseline of 10000 points.
As expected, as the number of points decreased, the accuracy decreased. From the baseline to 500 points, the accuracy hangs around the 96-98% range, and then dips around 100 points to 80%. Decreasing to 50 points lowers the accuracy even more to 50%.
The visualizations show that the point clouds still have clear shape around 500 points, but at 100 points it starts to become quite sparse and hard to classify.
Note: There was an unusual small accuracy increase when the point cloud went from 1000 and 500 points. This could be due to random sampling since each run keeps a different subset of points. Rerunning the experiment with different seeds may lead to more precise accuracy trends.
Test Accuracy: 0.978
Test Accuracy: 0.967
Test Accuracy: 0.969
Test Accuracy: 0.809
Test Accuracy: 0.510
| Points | 10000 (baseline) | 5000 | 1000 | 500 | 100 | 50 |
|---|---|---|---|---|---|---|
| Test accuracy | 0.904 | 0.904 | 0.900 | 0.890 | 0.818 | 0.778 |
The number of points evaluated was decreased from its baseline of 10000 points.
As expected, as the number of points decreased, the accuracy decreased. From the baseline to 500 points, the accuracy hangs around the 89-90% range, and then dips around 100 points to 82%. Decreasing to 50 points lowers the accuracy slightly more to 78%.
Compared to classification, the accuracy drop for sparser point clouds is not as severe. Despite the sparsity, the points are still spread out enough to hint at which part each point belongs to.
Test Accuracy: 0.904
Accuracy: 0.996
Accuracy: 0.994
Accuracy: 0.993
Test Accuracy: 0.900
Accuracy: 0.989
Accuracy: 0.996
Accuracy: 0.991
Test Accuracy: 0.890
Accuracy: 0.966
Accuracy: 0.996
Accuracy: 0.990
Test Accuracy: 0.818
Accuracy: 0.930
Accuracy: 0.810
Accuracy: 0.970
Test Accuracy: 0.778
Accuracy: 0.920
Accuracy: 0.720
Accuracy: 0.960
I used Claude to help generate accuracy tables for each robustness experiment.