| Model | Classification accuracy | Segmentation accuracy |
|---|---|---|
| PointNet (Q1 / Q2) | 97.17% | 87.20% |
| PointNet++ (Q4) | 97.48% | 90.80% |
Test accuracy: 97.17%
| Class | chair | vase | lamp |
|---|---|---|---|
| Accuracy | 99.68% | 91.18% | 93.16% |
| Test objects | 617 | 102 | 234 |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
| object 233 | object 636 | object 213 | object 222 | object 406 | object 496 |
| GT: chair | GT: vase | GT: chair | GT: chair | GT: chair | GT: chair |
| Pred: chair | Pred: vase | Pred: chair | Pred: chair | Pred: vase | Pred: chair |
| Ground truth | chair | vase | lamp | |||
|---|---|---|---|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
|
| Object | 406 | 543 | 651 | 663 | 726 | 750 |
| Prediction | vase | lamp | lamp | lamp | vase | vase |
Interpretation.
Most of the failures are between vases and lamps, whose overall shape looks like each other.
The chair class is almost never wrong (2 of 617).
Vases and Lamps have similar overall shapes and silhouettes. Since PointNet summarizes an object with a single max pooled global feature, it mostly sees the
coarse outline which is similar in both classes and cannot use the other fin details that would tell the classes apart.
Test accuracy: 87.20%
Colors: cyan = back, red = seat, blue = legs, yellow = arms, magenta / white = the two remaining (rare) part classes.
| Object | Ground truth | Prediction | Accuracy | |
|---|---|---|---|---|
| 397 | ![]() | ![]() |
99.34% | good |
| 490 | ![]() | ![]() |
99.25% | good |
| 89 | ![]() | ![]() |
90.44% | good |
| 26 | ![]() | ![]() |
36.42% | bad |
| 235 | ![]() | ![]() |
42.98% | bad |
Interpretation.
The model works very well on ordinary chairs with a clear back, a flat seat and separate legs.
The PointNet network has mostly learned where a part usually lies inside the object (for example, low points are
legs), because the per-point feature is only computed from the point's own coordinates plus a global shape code. This
works for the common chair shapes and fails for rare shapes where the same position belongs to a different part.
Both experiments use the Q1 and Q2 checkpoints without any retraining.
Procedure. Every test point cloud is rotated around the z axis (the objects are upright along y, so this tilts the objects sideways) by a fixed angle before it is passed to the model
| Rotation (degrees) | 0 (Q1 / Q2) | 15 | 30 | 45 | 60 | 90 | 180 |
|---|---|---|---|---|---|---|---|
| Classification accuracy | 97.17% | 94.44% | 75.55% | 37.57% | 29.91% | 26.86% | 43.55% |
| Segmentation accuracy | 87.20% | 82.56% | 70.73% | 61.61% | 54.73% | 43.18% | 35.88% |
Classification samples
| Object | 0 degrees (Q1) | 30 degrees | 90 degrees |
|---|---|---|---|
| 233 GT: chair |
Pred: chair |
Pred: vase |
Pred: vase |
| 636 GT: vase |
Pred: vase |
Pred: vase |
Pred: vase |
| 800 GT: lamp |
Pred: lamp |
Pred: lamp |
Pred: vase |
Segmentation samples
| Object | 0 degrees (Q2) | 30 degrees | 90 degrees | |||
|---|---|---|---|---|---|---|
| GT | Pred | GT | Pred | GT | Pred | |
| 397 | ![]() | 99.34% |
![]() | 76.95% |
![]() | 34.17% |
| 89 | ![]() | 90.44% |
![]() | 71.95% |
![]() | 62.07% |
Interpretation.
Procedure. Instead of all 10,000 points, a random subset of N points per object is given to the model The same random subset is used for all objects and segmentation accuracy is computed over the N sampled points.
| Number of points | 10000 (Q1 / Q2) | 5000 | 1000 | 500 | 100 | 50 |
|---|---|---|---|---|---|---|
| Classification accuracy | 97.17% | 96.96% | 96.75% | 96.43% | 91.92% | 77.33% |
| Segmentation accuracy | 87.20% | 87.21% | 87.26% | 86.80% | 81.03% | 74.32% |
Classification samples
| Object | 10000 points (Q1) | 1000 points | 100 points |
|---|---|---|---|
| 233 GT: chair |
Pred: chair |
Pred: chair |
Pred: chair |
| 636 GT: vase |
Pred: vase |
Pred: vase |
Pred: vase |
| 800 GT: lamp |
Pred: lamp |
Pred: lamp |
Pred: lamp |
Segmentation samples
| Object | 10000 points (Q2) | 1000 points | 100 points | |||
|---|---|---|---|---|---|---|
| GT | Pred | GT | Pred | GT | Pred | |
| 397 | ![]() | 99.34% |
![]() | 99.30% |
![]() | 97.00% |
| 89 | ![]() | 90.44% |
![]() | 90.50% |
![]() | 83.00% |
Interpretation.
Model. I implemented PointNet++ with single scale grouping. PointNet looks at every point on its own and only combines them in one global max pool. PointNet++ adds locality with set abstraction layers. FPS picks a set of centroids, a ball query collects the neighbours inside a radius around every centroid, and a small PointNet (shared MLP + max pool) is run on each neighbourhood, in coordinates relative to its centroid. Stacking these layers gives features of larger and larger regions.
| Task | PointNet (Q1 / Q2) | PointNet++ (Q4) |
|---|---|---|
| Classification | 97.17% | 97.48% |
| Segmentation | 87.20% | 90.80% |
| Classification, per class | chair | vase | lamp |
|---|---|---|---|
| PointNet | 99.68% | 91.18% | 93.16% |
| PointNet++ | 99.68% | 89.22% | 95.30% |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
|
| Object | 406 | 543 | 651 | 663 | 726 | 750 |
| Ground truth | chair | chair | vase | vase | lamp | lamp |
| PointNet (Q1) | vase | lamp | lamp | lamp | vase | vase |
| PointNet++ | lamp | lamp | vase | vase | lamp | lamp |
PointNet++ gets four of the six Q1 failure cases right (the vases with plants and the two unusual lamps). The two odd chairs are still wrong.
| Object | Ground truth | PointNet (Q2) | PointNet++ |
|---|---|---|---|
| 397 | ![]() |
99.34% |
96.47% |
| 490 | ![]() |
99.25% |
99.54% |
| 89 | ![]() |
90.44% |
94.56% |
| 26 | ![]() |
36.42% |
59.95% |
| 235 | ![]() |
42.98% |
46.97% |
Interpretation.