16-825 Assignment 5: Point Cloud Classification and Segmentation

Everything was trained on my RTX 5090.

1. Classification Model

Test accuracy = 97.69%

My model is PointNet without the transformation blocks. Every point goes through the same MLP (3 to 64 to 64 to 64 to 128 to 1024, written as 1x1 convolutions with batch norm and ReLU), a max pool over all the points gives one 1024-d feature for the whole object, and 3 fully connected layers (1024 to 512 to 256 to 3) give the class scores. I trained with the default settings (250 epochs, batch size 32, learning rate 0.001), and train.py kept the checkpoint from epoch 191.

True classPredicted chairPredicted vasePredicted lampAccuracy
Chair61700100%
Vase1911089.2%
Lamp11022395.3%

Random test objects

2 random test objects from each class (most of the test set is chairs, so I picked them per class).

Chairs

chair 459 chair 85

Left: predicted chair. Right: a long lounge chair, also predicted chair.

Vases

vase 637 vase 655

Left: predicted vase. Right: predicted lamp, which is wrong. It's a shallow dish with a stick coming out of it, so it looks like the base and arm of a desk lamp.

Lamps

lamp 721 lamp 871

Both predicted lamp.

Failures

Chair, predicted lamp (from the epoch 190 checkpoint)

chair 77 predicted lamp

My best model got all 617 test chairs right, so I took this failure from the checkpoint saved at epoch 190 (96.75% test accuracy), which called this chair a lamp (79% lamp). It's a tall, closed box with flat sides and no separate legs, seat or back, so it looks more like a tall column lamp than a chair. The best model got it right but was still unsure (59% chair, 41% lamp), and it's the least confident chair in the whole test set.

Vase, predicted lamp

vase 618 predicted lamp vase 651 predicted lamp

The left one is a wide short cylinder that looks just like a drum lamp shade, and the right one is a flat dish with a branch sticking up, which looks like a lamp base with an arm. The model was 100% sure both were lamps.

Lamp, predicted vase

lamp 726 predicted vase lamp 812 predicted vase

Both of these have a round, bulging base under the shade, and that base looks like the body of a vase (both were about 67% vase and 33% lamp, so the model wasn't very sure).

Almost all of the mistakes are between vases and lamps, since both are mostly round and symmetric, and the model only sees the overall shape through one max-pooled feature. Chairs never get confused because their legs and backs look very different. Vases are also the smallest class (741 training objects vs 4489 chairs and 1554 lamps), which is probably why they have the lowest accuracy.

2. Segmentation Model

Test accuracy = 90.07% (fraction of all test points labeled correctly)

The segmentation model uses the same shared MLP, but for every point it concatenates the point's own 64-d feature from the first layers with the 1024-d max-pooled feature of the whole chair, and then a per point MLP (1088 to 512 to 256 to 128 to 6) gives the part scores. I trained with the default settings (250 epochs, batch size 32, learning rate 0.001), and train.py kept the checkpoint from epoch 164. The colors are cyan for the back, red for the seat, blue for the legs and base, yellow for the arms and magenta for the small part on top of some backs.

Object 459 (random): accuracy 97.8%

459 ground truth 459 prediction

Object 85 (random): accuracy 86.4%

85 ground truth 85 prediction

Object 323 (random): accuracy 73.9%

323 ground truth 323 prediction

Object 4 (random): accuracy 69.3%

4 ground truth 4 prediction

Object 235 (bad prediction, the worst object): accuracy 44.7%

235 ground truth 235 prediction

Object 26 (bad prediction, the second worst object): accuracy 46.7%

26 ground truth 26 prediction

Ground truth is on the left and the prediction is on the right. The 4 random objects were picked randomly and the 2 bad ones are the lowest accuracy objects in the test set.

Simple chairs like 459 come out almost perfect, since the back, seat and legs are clearly separate pieces. The problems are on sofas and armchairs, where the seat, arms and back are one soft connected shape and the boundaries between them are not obvious. In 323 and 4 the model puts the boundary between the seat and the back or arms in the wrong place. The two worst objects are sofas whose labels are pretty unusual (most of 235 is labeled as arms), and the model predicts a more typical layout instead, with a seat in the middle and legs at the bottom. It also misses the small magenta part on top of 26, which is only 0.1% of the test points. The edges between parts are also noisy because PointNet labels each point from its own position and one global feature, without looking at its neighbors. Over the test set the median object accuracy is 93.5%, so most chairs are segmented well and the average is pulled down by the harder sofas.

3. Robustness Analysis

For both experiments I used the same trained models from Q1 and Q2 and only changed the test inputs (I added --rotate options to eval_cls.py and eval_seg.py, and robustness.py runs all the settings). In the data y is up and the objects are centered at the origin.

Experiment 1: Rotating the input

I rotated every test point cloud around the x axis (tilting the object forward) and around the y axis (spinning it around the up direction) by 0 to 180 degrees.

Rotation (degrees)0 (Q1/Q2)1530456090135180
Classification, tilt97.7%94.5%89.0%63.5%38.0%28.0%64.5%76.2%
Classification, spin97.7%96.9%90.6%76.6%61.9%69.9%31.3%35.6%
Segmentation, tilt90.1%83.7%73.7%63.8%42.0%25.7%24.8%34.6%
Segmentation, spin90.1%85.7%76.1%71.1%67.8%56.3%51.6%57.0%

accuracy vs rotation

Classification (chair 459, vase 637, lamp 871 from Q1, which were all predicted correctly there)

Original (Q1): chair, vase, lamp

chair original vase original lamp original

Tilted 45 degrees: predicted lamp, vase, lamp

chair tilted vase tilted lamp tilted

Spun 90 degrees: predicted vase, vase, lamp

chair spun vase spun lamp spun

Segmentation (objects 459 and 85 from Q2, ground truth then prediction)

Tilted 45 degrees: object 459 accuracy 65.1% (97.8% in Q2), object 85 accuracy 58.4% (86.4% in Q2)

459 tilted gt 459 tilted pred 85 tilted gt 85 tilted pred

Spun 90 degrees: object 459 accuracy 84.6%, object 85 accuracy 42.3%

459 spun gt 459 spun pred 85 spun gt 85 spun pred

Both models get a lot worse once the object is rotated more than about 30 degrees, since they were only trained on upright objects facing the same way and PointNet works directly on the xyz coordinates (there's no transformation block or rotation augmentation). Spinning around the up axis barely matters for vases and lamps because they're round (85 to 93% at 90 and 180 degrees), but chairs drop to 59% at 90 degrees and 6% at 180 degrees, so the model learned that chairs face one direction. Segmentation falls apart the most when tilted (26% at 90 degrees) because the parts are mostly told apart by height, legs at the bottom and back at the top, and tilting moves them to different heights. Training with random rotations would probably fix most of this.

Experiment 2: Fewer points per object

I evaluated with --num_points from 10,000 (the Q1/Q2 setting) down to 10, which randomly keeps that many of the points of every object.

Points per object10000 (Q1/Q2)500020001000500200100502010
Classification97.7%97.4%97.5%97.2%96.6%95.1%92.0%78.5%48.3%30.4%
Segmentation90.1%90.1%90.0%89.5%88.4%85.6%83.1%79.9%73.2%66.5%

accuracy vs number of points

Classification (same chair, vase and lamp as above)

100 points: predicted vase, vase, lamp

chair 100 points vase 100 points lamp 100 points

20 points: predicted lamp, lamp, lamp

chair 20 points vase 20 points lamp 20 points

Segmentation (objects 459 and 85, ground truth then prediction)

100 points: object 459 accuracy 93.0% (97.8% in Q2), object 85 accuracy 63.0% (86.4% in Q2)

459 100 points gt 459 100 points pred 85 100 points gt 85 100 points pred

20 points: object 459 accuracy 80.0%, object 85 accuracy 65.0%

459 20 points gt 459 20 points pred 85 20 points gt 85 20 points pred

Both models barely change down to about 500 points, because the max pool only needs a few points on each part of the object to get almost the same global feature. Below 100 points classification drops quickly, and with 20 points it calls almost everything a lamp (every lamp is right but none of the vases are), since a handful of scattered points looks like the thin parts of a lamp. Segmentation goes down much more slowly (73% with 20 points), probably because each point's label mostly depends on its own height in the chair, which doesn't change when the other points are removed.

4. Locality (PointNet++)

I implemented PointNet++ with single scale grouping. Instead of one max pool over the whole object, it runs a small PointNet on local neighborhoods and builds them up in layers. Each set abstraction layer picks centers with farthest point sampling, takes up to K points within a radius of each center (ball query), runs a shared MLP on their coordinates relative to the center and max pools inside each group. I used 3 layers: 512 centers with radius 0.2 and K = 32, 128 centers with radius 0.4 and K = 64, and then one group with everything, which gives a 1024-d feature for the object. For classification, fully connected layers (1024 to 512 to 256 to 3) give the class scores. For segmentation, 3 feature propagation layers bring the features back to all 10,000 points (each point takes the inverse distance weighted average of its 3 nearest centers and concatenates the features from the matching earlier layer), and a per point layer gives the 6 part scores. I used the pytorch3d ops for the sampling, ball query and nearest neighbors, and trained both with the same default settings as Q1 and Q2. train.py kept the checkpoints from epoch 170 (classification) and 180 (segmentation).

Classification test accuracy = 98.85% (PointNet in Q1: 97.69%)

Segmentation test accuracy = 91.80% (PointNet in Q2: 90.07%)

PointNet (Q1/Q2)PointNet++
Classification97.69%98.85%
Chairs100% (617/617)99.8% (616/617)
Vases89.2% (91/102)94.1% (96/102)
Lamps95.3% (223/234)98.3% (230/234)
Segmentation90.07%91.80%
Segmentation, median object93.5%95.2%

One thing I noticed is that PointNet++ depends a little on the order of the points. The ball query keeps the first K points it finds in each ball, and the points in the data files are stored nicely spread out, so in file order those points cover the ball evenly. eval_cls.py and eval_seg.py shuffle the points, and with shuffled points PointNet++ gets 98.11% and 90.64%. The numbers above use the file order (I added --keep_order to the eval scripts), which is also what train.py used to pick the best model. PointNet gets the same accuracy either way.

Classification

Q1 failures (vases 618 and 651, lamps 726 and 812)

vase 618 vase 651 lamp 726 lamp 812

PointNet predicted lamp, lamp, vase, vase. PointNet++ gets all four right (vase, vase, lamp, lamp), and also vase 655 and chair 77 from Q1. Their overall shapes look like the other class, so looking at local details probably helps.

PointNet++ failures (chair 406 predicted lamp, vase 670 predicted lamp, lamp 787 predicted vase)

chair 406 predicted lamp vase 670 predicted lamp lamp 787 predicted vase

PointNet got 406 and 787 right and 670 wrong. 406 is a chair that is almost completely flat, 670 is a thin vase full of sticks that look like the arms of a lamp, and 787 is a lamp with a big round ball for a base, which looks like the body of a vase.

Segmentation

Object45985323423526
PointNet (Q2)97.8%86.4%73.9%69.3%44.7%46.7%
PointNet++98.8%75.6%87.4%72.1%49.8%64.7%

Ground truth, then PointNet (Q2), then PointNet++.

Object 323: PointNet 73.9%, PointNet++ 87.4%

323 ground truth 323 PointNet 323 PointNet++

Object 26: PointNet 46.7%, PointNet++ 64.7%

26 ground truth 26 PointNet 26 PointNet++

Object 85: PointNet 86.4%, PointNet++ 75.6%

85 ground truth 85 PointNet 85 PointNet++

On 323 PointNet labels the lower front of the seat as legs, and on 26 it mixes up the seat, arms and legs. PointNet++ gets these mostly right (the arms of 26 go from 34% to 78% correct), probably because each point's label now also depends on the shape around it and not just its height. It's not always better though: on 85 it gets the arms and legs right but puts patches of legs and arms inside the big seat, so the accuracy goes down, and it still misses the small magenta part on top of 26. Overall it improves 5 of these 6 objects and the median object accuracy goes from 93.5% to 95.2%.