16-825 Assignment 5: Point Cloud Classification and Segmentation

Q1. Classification Model

Test accuracy: 97.48% (929 of 953 test objects)

Random test objects

Chair, predicted chair
Chair, predicted chair
Vase, predicted vase
Vase, predicted vase
Lamp, predicted lamp
Lamp, predicted lamp

Failure cases

Chair. No chair in the test set is misclassified (617 of 617 correct).

Test object 393 rotated by 90°, predicted vase

Chairs only start failing once the input is rotated, so as the chair failure I show chair 393 from Q3, lying on its side.

Vase, predicted lamp (11 of 102 vases are wrong)

Test object 619

A heavy base with a flower stem sticking up from it, which looks like the base and pole of a table lamp.

Test object 655

Same as 619, with a stick instead of a stem.

Test object 707

A tall block on a wide base, again the shape of a table lamp.

Lamp, predicted vase (13 of 234 lamps are wrong)

Test object 777

A hanging chandelier with lots of curly arms and no base or pole, which looks more like a plant in a vase.

Test object 827

Just a drum-shaped shade with no base or pole, which looks like a wide pot.

Test object 916

A normal table lamp with a shade, a pole and a base, so this one is a plain model mistake. DGCNN gets it right (Q4).

Interpretation.

Q2. Segmentation Model

Test accuracy: 88.28% (fraction of correctly labeled points over all 617 test chairs)

Ground truth on the left, prediction on the right. In these chairs cyan is the back, red is the seat, blue is the legs and base and yellow is the armrests.

Object 315, ground truth
Object 315, prediction (93.85%)

A normal chair. All the parts are right and the mistakes are only along the boundaries, where the seat meets the back or the legs.

Object 523, ground truth
Object 523, prediction (89.31%)

Also a normal chair, with the same kind of boundary mistakes as 315.

Object 392, ground truth
Object 392, prediction (82.93%)

A rocking chair. It is still mostly correct, but the curved rockers and the armrests that run into the seat make the boundaries harder.

Object 272, ground truth
Object 272, prediction (50.13%, bad)

The second worst object in the test set. The seat label spreads down into the top of the legs.

Object 351, ground truth
Object 351, prediction (48.71%, bad)

The worst object in the test set. It is a solid barrel-shaped chair, and its sides are labeled as armrests all the way down to the floor. The model labels the bottom half as legs and a band in the middle as seat, which is what a normal chair has at those heights.

Interpretation. In both bad cases the prediction follows where each part usually is in a chair more than the actual shape of this chair. I think this is because in PointNet each point only gets its own coordinates and one global feature, with nothing about its neighbors.

Q3. Robustness Analysis

Experiment 1: Rotating the input

Procedure.

Rotation0° (Q1/Q2)15°30°45°60°90°180°
Classification97.48%95.07%72.30%43.55%33.68%28.23%52.05%
Chairs only100%99.0%71.5%28.5%15.1%6.0%41.3%
Segmentation88.28%82.32%69.38%58.70%53.88%47.78%34.79%

Interpretation.

Classification, the same chair (object 393) at different angles:

0°, predicted chair
30°, predicted chair
90°, predicted vase
180°, predicted lamp

A vase (object 644) and a lamp (object 722) at 90°:

Vase at 90°, predicted vase
Lamp at 90°, predicted lamp

Both are still correct at 90°, like 72.5% of the vases and 67.5% of the lamps, compared to only 6% of the chairs.

Segmentation, object 315 (93.85% when upright in Q2) and object 523 (89.31% when upright):

30°, ground truth
30°, prediction (73.01%)
90°, ground truth
90°, prediction (61.02%)
180°, ground truth
180°, prediction (32.37%)

Segmentation is worst at 180°. In the upside-down chair the legs, now at the top, are labeled as back, and the back, now at the bottom, is labeled as legs. So the segmentation model has mostly learned to label parts by their height, not by their shape.

Object 523 at 90°, ground truth
Object 523 at 90°, prediction (53.67%)

Experiment 2: Number of points per object

Procedure.

Points per object10000 (Q1/Q2)500010005001005010
Classification97.48%97.48%97.17%96.96%92.44%82.27%31.58%
Segmentation88.28%88.35%88.41%87.59%81.72%78.28%62.40%

Interpretation.

Classification, object 393 with fewer points:

500 points, predicted chair
100 points, predicted chair
50 points, predicted chair
10 points, predicted lamp

A vase (object 644) and a lamp (object 722) with 50 points:

Vase with 50 points, predicted lamp
Lamp with 50 points, predicted lamp

At 50 points only 21% of the vases are still correct while 99% of the lamps are, and at 10 points all lamps are still correct while only 11% of the chairs and none of the vases are. A few scattered points no longer show the solid body of a vase or the seat of a chair, and the thin, spread-out lamps hold up best.

Segmentation, objects 315 and 523 with fewer points:

500 points, ground truth
500 points, prediction (89.60%)
100 points, ground truth
100 points, prediction (82.00%)
50 points, ground truth
50 points, prediction (80.00%)
10 points, ground truth
10 points, prediction (50.00%)
Object 523, 100 points, ground truth
Object 523, 100 points, prediction (85.00%)

Q4. Bonus: Locality

Model. I implemented DGCNN (Dynamic Graph CNN). Its main block is EdgeConv:

TaskPointNet, 10000 points (Q1/Q2)PointNet, same points as DGCNNDGCNN
Classification (1024 points)97.48%97.17%98.53%
Segmentation (2048 points)88.28%88.47%91.15%

Results.

The PointNet and ground-truth GIFs below use all 10000 points, while the DGCNN GIFs use 1024 (classification) or 2048 (segmentation) points, which is why they look sparser.

Classification. The six objects from the Q1 failure cases, now predicted by DGCNN. "PointNet" in the captions is the Q1 model with 10000 points.

619, vase. PointNet: lamp, DGCNN: vase
655, vase. PointNet: lamp, DGCNN: vase
707, vase. PointNet: lamp, DGCNN: vase

DGCNN fixes all three vases where PointNet was fooled by a thin part on top. With local neighborhoods it can tell a thin stem apart from a lamp pole.

916, lamp. PointNet: vase, DGCNN: lamp
777, lamp. PointNet: vase, DGCNN: vase
827, lamp. PointNet: vase, DGCNN: vase

Segmentation. Ground truth, PointNet from Q2 and DGCNN on the same chairs:

272, ground truth
272, PointNet (50.13%)
272, DGCNN (78.13%)

The biggest gain (50.13% to 78.13%). DGCNN puts the boundary between the seat and the legs much closer to the right place. I think this is because its local neighborhoods can separate the flat seat from the thin legs, while PointNet has to go mostly by height.

392, ground truth
392, PointNet (82.93%)
392, DGCNN (85.64%)

The rocking chair gets a little better (82.93% to 85.64%).

351, ground truth
351, PointNet (48.71%)
351, DGCNN (48.14%)

Stays at about 48% for both models. Other chairs label a solid base like this as legs, so I think this is an unusual labeling that neither model can learn from the rest of the training set.

315, ground truth
315, PointNet (93.85%)
315, DGCNN (94.43%)

A normal chair that both models already get almost right (93.85% and 94.43%).