Assignment 5: Point Cloud Classification and Segmentation

Name: Sarah Pendhari  |  Andrew ID: spendhar

Q1. Classification Model

I used PointNet without the transform blocks. A shared MLP (64, 128, 1024) runs on every point. Then I max pool over the points. After that, an MLP (512, 256, 3) gives the class scores. I trained with Adam (lr 0.001) and cross entropy.

Test accuracy: 0.9801

Random test point clouds

Each caption shows the true and predicted class.

base

#249: true=chair, pred=chair
#331: true=chair, pred=chair
#493: true=chair, pred=chair
#569: true=chair, pred=chair
#693: true=vase, pred=vase
#875: true=lamp, pred=lamp

Failure cases

#406: true=chair, pred=lamp
#618: true=vase, pred=lamp
#671: true=vase, pred=chair
#767: true=lamp, pred=vase

The model got 19 of 953 test objects wrong. Sixteen of those are vase and lamp mix-ups. Only three involve a chair.

PointNet only uses one global feature. It does not look at local neighborhoods. So it sees the overall shape but misses small details. Vases and lamps have similar overall shapes. That is why most errors are between them.

Q2. Segmentation Model

I used the PointNet segmentation design. Each point has a 64-d feature. I join it with the 1024-d global feature. That gives 1088 channels per point. A per-point MLP (512, 256, 128, 6) gives the labels. Training used Adam (lr 0.001) and cross entropy.

Test accuracy: 0.8997

Ground truth is on the left. The prediction is on the right. The accuracy of each object is in the caption.

Good example

Ground truth, object 505
Prediction, object 505, accuracy 0.9970

Good example

Ground truth, object 397
Prediction, object 397, accuracy 0.9948

Good example

Ground truth, object 434
Prediction, object 434, accuracy 0.9948

Bad example

Ground truth, object 605
Prediction, object 605, accuracy 0.4628

Bad example

Ground truth, object 351
Prediction, object 351, accuracy 0.4925

The good examples have accuracy above 0.99. They are normal chairs. The seat, back, legs and arms are easy to tell apart. The model only makes small mistakes at the edges between parts.

The bad examples have accuracy below 0.50. These chairs have unusual shapes. Some parts blend together. The model mixes up the seat, back and arms. Sometimes it gives a whole part the wrong label.

This makes sense for PointNet. It labels each point using its own position and one global feature. It does not check its neighbors. So nothing forces the labels inside one part to agree.

The overall accuracy is 0.8997. But the objects range from 0.46 to 0.99. So the average hides a big gap between easy and hard chairs.

Q3. Robustness Analysis

Experiment 1: Rotating the input

Procedure: I rotated the test point clouds with a rotation matrix. I did not retrain. I tried 30 and 90 degrees around the vertical (y) axis. I also tried 90 degrees around the x axis, which tilts the object. I used the same good (object 505) and bad (object 605) chairs as in Q2.

Rotation (degrees)ClassificationSegmentation
0 (baseline)0.98010.8997
300.91080.7774
900.62220.5615
90 about x0.24240.3264

Classification (random test objects)

rot30

#249: true=chair, pred=chair
#331: true=chair, pred=chair
#493: true=chair, pred=chair
#569: true=chair, pred=chair
#693: true=vase, pred=vase
#875: true=lamp, pred=lamp

rot90

#249: true=chair, pred=chair
#331: true=chair, pred=chair
#493: true=chair, pred=vase
#569: true=chair, pred=vase
#693: true=vase, pred=vase
#875: true=lamp, pred=lamp

rotx90

#249: true=chair, pred=vase
#331: true=chair, pred=lamp
#493: true=chair, pred=vase
#569: true=chair, pred=lamp
#693: true=vase, pred=lamp
#875: true=lamp, pred=lamp

Segmentation

rot30, object 505 (good example in Q2)

Ground truth, object 505
Prediction, object 505

rot30, object 605 (bad example in Q2)

Ground truth, object 605
Prediction, object 605

rot90, object 505 (good example in Q2)

Ground truth, object 505
Prediction, object 505

rot90, object 605 (bad example in Q2)

Ground truth, object 605
Prediction, object 605

rotx90, object 505 (good example in Q2)

Ground truth, object 505
Prediction, object 505

rotx90, object 605 (bad example in Q2)

Ground truth, object 605
Prediction, object 605

Accuracy drops a lot when the objects are rotated. Classification goes from 0.9801 to 0.9108 at 30 degrees. It falls to 0.6222 at 90 degrees. Segmentation goes from 0.8997 to 0.7774, then 0.5615.

The tilt about x is the worst. Classification falls to 0.2424. That is below chance for 3 classes (0.33). So the model must be sending most tilted objects to one or two classes. Segmentation falls to 0.3264.

The good chair (505) shows the drop clearly. Its accuracy goes from 0.9970 to 0.9198 at 30 degrees and 0.7022 at 90 degrees. The bad chair (605) was already failing at 0.4628. Its numbers move around that value, so they say little. It even goes up a bit at 30 degrees (0.5781).

The reason is simple. The model only saw upright objects in training. PointNet has no rotation invariance, since I skipped the transform blocks. Its features depend on the raw coordinates. When the chair turns, the parts end up in places the model does not expect. A tilt about x moves the parts the most, so it hurts the most.

Experiment 2: Number of points

Procedure: At test time I used --num_points to sample random points from each object. I sampled without replacement. I tried 10000 (baseline), 1000 and 100 points. I did not retrain. The same good and bad chairs are shown.

Number of pointsClassificationSegmentation
10000 (baseline)0.98010.8997
10000.97800.8989
1000.94230.8334

Classification (random test objects)

n1000

#249: true=chair, pred=chair
#331: true=chair, pred=chair
#493: true=chair, pred=chair
#569: true=chair, pred=chair
#693: true=vase, pred=vase
#875: true=lamp, pred=lamp

n100

#249: true=chair, pred=chair
#331: true=chair, pred=chair
#493: true=chair, pred=chair
#569: true=chair, pred=chair
#693: true=vase, pred=vase
#875: true=lamp, pred=lamp

Segmentation

n1000, object 505 (good example in Q2)

Ground truth, object 505
Prediction, object 505

n1000, object 605 (bad example in Q2)

Ground truth, object 605
Prediction, object 605

n100, object 505 (good example in Q2)

Ground truth, object 505
Prediction, object 505

n100, object 605 (bad example in Q2)

Ground truth, object 605
Prediction, object 605

The model is quite robust to fewer points. With 1000 points, classification is 0.9780 and segmentation is 0.8989. That is almost the same as the baseline. With 100 points, classification is 0.9423 and segmentation is 0.8334. This is a clear drop, but not a collapse.

Max pooling keeps only the strongest value in each channel. A random subset of 1000 points still covers the main shape. So the global feature barely changes. With 100 points, the shape is covered thinly. The global feature changes more and some predictions flip.

The good chair (505) goes from 0.9970 to 0.9930 at 1000 points. It drops to 0.8500 at 100 points. Training used 10000 points, so very sparse inputs also look different from what the model saw.

The numbers change a little between runs. The 1000 point segmentation accuracy was 0.8989 in one run and 0.8979 in another. This is because the points are picked at random.