Test accuracy: 97.59%
![]() Predicted: chair |
![]() Predicted: chair |
![]() Predicted: vase |
![]() Predicted: vase |
![]() Predicted: lamp |
![]() Predicted: lamp |
![]() Ground truth: chair, predicted: lamp |
![]() Ground truth: vase, predicted: lamp |
![]() Ground truth: lamp, predicted: vase |
Chair predicted as lamp: This chair is almost completely flat (it looks like a folded chair), so it has no seat sticking out and no normal leg layout. Without those chair features, its thin flat shape is closer to some lamp parts.
Vase predicted as lamp: This vase is a closed cylinder with a lid, with no plant or open top. It looks like a cylindrical lamp body or lamp shade, which is also a common shape in the lamp class.
Lamp predicted as vase: This lamp is a chandelier with many thin, curved arms. That spread-out, branchy shape looks like the leaves and flowers of the plants in many vase examples, so the global feature is closer to a vase.
Only 1 of 617 chairs is misclassified, compared with 17 of 102 vases and 5 of 234 lamps. Vases and lamps are confused more often because there are fewer of them in training, and both classes contain many round, roughly symmetric shapes.
Test accuracy: 90.34%
| Object | Ground truth | Prediction |
|---|---|---|
| Object 209 Accuracy: 95.44% |
![]() |
![]() |
| Object 123 Accuracy: 91.98% |
![]() |
![]() |
| Object 387 Accuracy: 69.04% |
![]() |
![]() |
| Object 235 (bad prediction) Accuracy: 48.35% |
![]() |
![]() |
| Object 351 (bad prediction) Accuracy: 47.12% |
![]() |
![]() |
Objects 209 and 123 are regular chairs with a clear back, seat and separate legs, so each part is easy to tell apart by its position and the model gets over 90% of the points right. Mistakes there happen only at boundaries between parts, for example where the seat meets the legs.
Object 387 is an armchair, and the model labels parts of the seat and armrests with the wrong part. The armrests and the back are joined together, so the boundary between them is not clear from the point positions.
The two bad predictions (235 and 351) are a sofa and a barrel-shaped chair with no separate legs. The model predicts the bottom part of these chairs as legs, because in most training chairs the bottom part is legs. Since PointNet labels each point using only its own position plus one global feature, it cannot use local shape to see that the bottom is a solid base and not legs.
Both experiments use the trained Q1 and Q2 models (no retraining) on the full test sets. The 0° / 10000-point column is the Q1 and Q2 result.
Procedure: Every test point cloud is rotated about the z axis (the objects stand along y, so this tilts them sideways) by 0°, 30°, 60° and 90° before it is given to the model. All 10000 points are used.
| 0° (Q1/Q2) | 30° | 60° | 90° | |
|---|---|---|---|---|
| Classification accuracy | 97.59% | 75.45% | 28.23% | 24.66% |
| Segmentation accuracy | 90.34% | 67.31% | 50.74% | 44.27% |
Classification (caption: predicted class):
| Object | 0° (Q1/Q2) | 30° | 60° | 90° |
|---|---|---|---|---|
| 393 (chair) | ![]() chair | ![]() chair | ![]() vase | ![]() vase |
| 644 (vase) | ![]() vase | ![]() vase | ![]() vase | ![]() vase |
| 722 (lamp) | ![]() lamp | ![]() vase | ![]() lamp | ![]() lamp |
Segmentation, object 209 (caption: accuracy for this object):
| 0° (Q1/Q2) | 30° | 60° | 90° | |
|---|---|---|---|---|
| Ground truth | ![]() | ![]() | ![]() | ![]() |
| Prediction | ![]() 95.44% | ![]() 75.69% | ![]() 50.11% | ![]() 52.07% |
Both models are very sensitive to rotation. Classification drops from 97.6% to 75.5% at 30° and to about 25-28% at 60-90°, which is worse than always predicting chair; at large angles most objects are predicted as vase (for example the tilted chair). Segmentation drops from 90.3% to 44.3% at 90°. In the segmentation example the model still splits the chair into parts, but it assigns labels partly by height, so after tilting the seat and back are labeled with the wrong parts. The models use raw xyz coordinates, all training objects are upright, and there is no rotation augmentation or input transform block, so they never learn rotation invariance.
Procedure: Each test object is randomly subsampled (same indices for every object, fixed seed) to 10000, 1000, 100 and 50 points before it is given to the model. No rotation is applied.
| 10000 (Q1/Q2) | 1000 | 100 | 50 | |
|---|---|---|---|---|
| Classification accuracy | 97.59% | 96.75% | 93.07% | 80.69% |
| Segmentation accuracy | 90.34% | 89.72% | 81.74% | 78.86% |
Classification (caption: predicted class):
| Object | 10000 (Q1/Q2) | 1000 | 100 | 50 |
|---|---|---|---|---|
| 393 (chair) | ![]() chair | ![]() chair | ![]() chair | ![]() chair |
| 644 (vase) | ![]() vase | ![]() lamp | ![]() vase | ![]() vase |
| 722 (lamp) | ![]() lamp | ![]() lamp | ![]() lamp | ![]() lamp |
Segmentation, object 209 (caption: accuracy for this object):
| 10000 (Q1/Q2) | 1000 | 100 | 50 | |
|---|---|---|---|---|
| Ground truth | ![]() | ![]() | ![]() | ![]() |
| Prediction | ![]() 95.44% | ![]() 94.90% | ![]() 94.00% | ![]() 88.00% |
Both models are robust to the number of points. With 1000 points (10% of the input) accuracy drops by less than 1%, and with only 100 points classification is still 93.1% and segmentation 81.7%. This is because of the max-pool: the global feature only depends on the few points that give the largest value in each channel, so as long as the samples still cover the overall shape, the feature barely changes. At 50 points the shape is no longer covered well (thin parts such as legs and lamp poles are missed), so accuracy drops more, to 80.7% and 78.9%.