Test accuracy: 97.48% (929 of 953 test objects)






Chair. No chair in the test set is misclassified (617 of 617 correct).

Chairs only start failing once the input is rotated, so as the chair failure I show chair 393 from Q3, lying on its side.
Vase, predicted lamp (11 of 102 vases are wrong)

A heavy base with a flower stem sticking up from it, which looks like the base and pole of a table lamp.

Same as 619, with a stick instead of a stem.

A tall block on a wide base, again the shape of a table lamp.
Lamp, predicted vase (13 of 234 lamps are wrong)

A hanging chandelier with lots of curly arms and no base or pole, which looks more like a plant in a vase.

Just a drum-shaped shade with no base or pole, which looks like a wide pot.

A normal table lamp with a shade, a pole and a base, so this one is a plain model mistake. DGCNN gets it right (Q4).
Interpretation.
Test accuracy: 88.28% (fraction of correctly labeled points over all 617 test chairs)
Ground truth on the left, prediction on the right. In these chairs cyan is the back, red is the seat, blue is the legs and base and yellow is the armrests.


A normal chair. All the parts are right and the mistakes are only along the boundaries, where the seat meets the back or the legs.


Also a normal chair, with the same kind of boundary mistakes as 315.


A rocking chair. It is still mostly correct, but the curved rockers and the armrests that run into the seat make the boundaries harder.


The second worst object in the test set. The seat label spreads down into the top of the legs.


The worst object in the test set. It is a solid barrel-shaped chair, and its sides are labeled as armrests all the way down to the floor. The model labels the bottom half as legs and a band in the middle as seat, which is what a normal chair has at those heights.
Interpretation. In both bad cases the prediction follows where each part usually is in a chair more than the actual shape of this chair. I think this is because in PointNet each point only gets its own coordinates and one global feature, with nothing about its neighbors.
Procedure.
--rotation in eval_cls.py and eval_seg.py).| Rotation | 0° (Q1/Q2) | 15° | 30° | 45° | 60° | 90° | 180° |
|---|---|---|---|---|---|---|---|
| Classification | 97.48% | 95.07% | 72.30% | 43.55% | 33.68% | 28.23% | 52.05% |
| Chairs only | 100% | 99.0% | 71.5% | 28.5% | 15.1% | 6.0% | 41.3% |
| Segmentation | 88.28% | 82.32% | 69.38% | 58.70% | 53.88% | 47.78% | 34.79% |
Interpretation.
Classification, the same chair (object 393) at different angles:




A vase (object 644) and a lamp (object 722) at 90°:


Both are still correct at 90°, like 72.5% of the vases and 67.5% of the lamps, compared to only 6% of the chairs.
Segmentation, object 315 (93.85% when upright in Q2) and object 523 (89.31% when upright):






Segmentation is worst at 180°. In the upside-down chair the legs, now at the top, are labeled as back, and the back, now at the bottom, is labeled as legs. So the segmentation model has mostly learned to label parts by their height, not by their shape.


Procedure.
--num_points. The models from Q1 and Q2 were trained with all 10000 points and are not retrained.| Points per object | 10000 (Q1/Q2) | 5000 | 1000 | 500 | 100 | 50 | 10 |
|---|---|---|---|---|---|---|---|
| Classification | 97.48% | 97.48% | 97.17% | 96.96% | 92.44% | 82.27% | 31.58% |
| Segmentation | 88.28% | 88.35% | 88.41% | 87.59% | 81.72% | 78.28% | 62.40% |
Interpretation.
Classification, object 393 with fewer points:




A vase (object 644) and a lamp (object 722) with 50 points:


At 50 points only 21% of the vases are still correct while 99% of the lamps are, and at 10 points all lamps are still correct while only 11% of the chairs and none of the vases are. A few scattered points no longer show the solid body of a vase or the seat of a chair, and the thin, spread-out lamps hold up best.
Segmentation, objects 315 and 523 with fewer points:










Model. I implemented DGCNN (Dynamic Graph CNN). Its main block is EdgeConv:
[xj − xi, xi]. The difference describes the local shape around the point and xi keeps the point's own feature (its position in the first layer).| Task | PointNet, 10000 points (Q1/Q2) | PointNet, same points as DGCNN | DGCNN |
|---|---|---|---|
| Classification (1024 points) | 97.48% | 97.17% | 98.53% |
| Segmentation (2048 points) | 88.28% | 88.47% | 91.15% |
Results.
The PointNet and ground-truth GIFs below use all 10000 points, while the DGCNN GIFs use 1024 (classification) or 2048 (segmentation) points, which is why they look sparser.
Classification. The six objects from the Q1 failure cases, now predicted by DGCNN. "PointNet" in the captions is the Q1 model with 10000 points.



DGCNN fixes all three vases where PointNet was fooled by a thin part on top. With local neighborhoods it can tell a thin stem apart from a lamp pole.



Segmentation. Ground truth, PointNet from Q2 and DGCNN on the same chairs:



The biggest gain (50.13% to 78.13%). DGCNN puts the boundary between the seat and the legs much closer to the right place. I think this is because its local neighborhoods can separate the flat seat from the thin legs, while PointNet has to go mostly by height.



The rocking chair gets a little better (82.93% to 85.64%).



Stays at about 48% for both models. Other chairs label a solid base like this as legs, so I think this is an unusual labeling that neither model can learn from the rest of the training set.



A normal chair that both models already get almost right (93.85% and 94.43%).