16-825 Assignment 5: Point Cloud Processing

Name: Hiroki Mii

Andrew ID: hmii

Q1. Classification Model

Test Accuracy: 0.980

Correct Cases

Predicted: Chair / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

Failure Cases

Predicted: Lamp / True: Chair
Predicted: Lamp / True: Vase
Predicted: Vase / True: Lamp

The failure cases are pretty justified. Each of the failure cases involves an ambiguous looking object that even a human can mistaken. Failure case 1 and 2 are both atypical looking chairs and vases, and case 3's lamp could easily be mistaken for a vase.

Q2. Segmentation Model

Test Accuracy: 0.904

Below includes the 3 highest accuracies, followed by the 2 lowest. Prediction on left, ground truth on right.

Accuracy: 0.996

Accuracy: 0.995

Accuracy: 0.993

Accuracy: 0.497

Accuracy: 0.461

The high accuracy results clearly segment the different parts of the chair. The back, seat, and legs are distinct and separated, leading to a clean segmentation. However, some chairs have more ambiguous separation between the parts, like the couch chairs that led to poor accuracies. Unlike the first three chairs, the legs on the couch chairs are very small and hard to identify. The segmentation was off on the legs for both chair #4 and #5. Additionally, chair #4 had a low armrest that was completely missed, which the model thought was part of the seat.

Q3. Robustness Analysis

Rotation - Classification

Axis0° (baseline)90°180°270°
x0.9800.3410.6650.439
y0.9800.7000.4430.636
z0.9800.2650.6650.337

Point clouds were rotated 90, 180, and 270 degrees about the x-axis, y-axis, and z-axis.

PointNet is permutation invariant, but not rotation invariant. Since the model was trained only on upright objects, with no transformation blocks or rotation augmentation, it struggles when the orientation differs substantially from the training data.

For x-axis and z-axis rotations, the accuracies drop significantly for 90 and 270 degree rotations since the objects are flipped over on its sides. For 180 degree rotations, the accuracies drop, but less drastically. The model can still recognize objects that are flipped upside down to an extent.

For y-axis rotations, the accuracies also drop, but milder since the objects still remain in an upright position. The model can still classify objects turned 90 and 270 degrees, but when it's completely turned around in a 180 degree position, it struggles.

In the visualizations, vase and lamp are constantly confused between each other, while chair jumps around in predictions.

Rotation about x-axis

90 degrees

Test Accuracy: 0.341

Predicted: Vase / True: Chair
Predicted: Lamp / True: Vase
Predicted: Vase / True: Lamp

180 degrees

Test Accuracy: 0.665

Predicted: Vase / True: Chair
Predicted: Lamp / True: Vase
Predicted: Lamp / True: Lamp

270 degrees

Test Accuracy: 0.439

Predicted: Chair / True: Chair
Predicted: Lamp / True: Vase
Predicted: Vase / True: Lamp

Rotation about y-axis

90 degrees

Test Accuracy: 0.700

Predicted: Chair / True: Chair
Predicted: Lamp / True: Vase
Predicted: Lamp / True: Lamp

180 degrees

Test Accuracy: 0.443

Predicted: Lamp / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

270 degrees

Test Accuracy: 0.636

Predicted: Chair / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

Rotation about z-axis

90 degrees

Test Accuracy: 0.265

Predicted: Vase / True: Chair
Predicted: Lamp / True: Vase
Predicted: Vase / True: Lamp

180 degrees

Test Accuracy: 0.665

Predicted: Chair / True: Chair
Predicted: Lamp / True: Vase
Predicted: Lamp / True: Lamp

270 degrees

Test Accuracy: 0.337

Predicted: Vase / True: Chair
Predicted: Lamp / True: Vase
Predicted: Vase / True: Lamp

Rotation - Segmentation

Axis0° (baseline)90°180°270°
x0.9040.2830.3360.204
y0.9040.5800.5860.658
z0.9040.4110.3490.408

Point clouds were rotated 90, 180, and 270 degrees about the x-axis, y-axis, and z-axis.

Like classification, point cloud rotations drop segmentation accuracies because PointNet is not rotation invariant. However, the drop in accuracy is worse for segmentation than classification. For classification, when the model is not confident, it can still take a guess and have a chance of getting it right. Segmentation is harder because the model needs to know which part belongs where in a chair.

Like classification, y-axis rotations did less worse than x-axis and z-axis rotations, since the chairs are still in an upright position. The back, seat, and legs are in expected positions relative to each other that the model saw during training.

It is interesting to see that for 180 degree rotations for x-axis and z-axis, the model confuses the back and legs since the chair is flipped upside down. The seat is often detected correctly.

Rotation about x-axis

90 degrees

Test Accuracy: 0.283

Accuracy: 0.057

Accuracy: 0.128

Accuracy: 0.293

180 degrees

Test Accuracy: 0.336

Accuracy: 0.423

Accuracy: 0.261

Accuracy: 0.424

270 degrees

Test Accuracy: 0.204

Accuracy: 0.184

Accuracy: 0.164

Accuracy: 0.042

Rotation about y-axis

90 degrees

Test Accuracy: 0.580

Accuracy: 0.655

Accuracy: 0.503

Accuracy: 0.665

180 degrees

Test Accuracy: 0.586

Accuracy: 0.534

Accuracy: 0.242

Accuracy: 0.644

270 degrees

Test Accuracy: 0.658

Accuracy: 0.782

Accuracy: 0.534

Accuracy: 0.830

Rotation about z-axis

90 degrees

Test Accuracy: 0.411

Accuracy: 0.539

Accuracy: 0.476

Accuracy: 0.372

180 degrees

Test Accuracy: 0.349

Accuracy: 0.364

Accuracy: 0.412

Accuracy: 0.476

270 degrees

Test Accuracy: 0.408

Accuracy: 0.642

Accuracy: 0.441

Accuracy: 0.353

Number of Points - Classification

Points10000 (baseline)5000100050010050
Test accuracy 0.9800.9780.9670.9690.8090.510

The number of points evaluated was decreased from its baseline of 10000 points.

As expected, as the number of points decreased, the accuracy decreased. From the baseline to 500 points, the accuracy hangs around the 96-98% range, and then dips around 100 points to 80%. Decreasing to 50 points lowers the accuracy even more to 50%.

The visualizations show that the point clouds still have clear shape around 500 points, but at 100 points it starts to become quite sparse and hard to classify.

Note: There was an unusual small accuracy increase when the point cloud went from 1000 and 500 points. This could be due to random sampling since each run keeps a different subset of points. Rerunning the experiment with different seeds may lead to more precise accuracy trends.

5000 Points

Test Accuracy: 0.978

Predicted: Chair / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

1000 Points

Test Accuracy: 0.967

Predicted: Chair / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

500 Points

Test Accuracy: 0.969

Predicted: Chair / True: Chair
Predicted: Vase / True: Vase
Predicted: Lamp / True: Lamp

100 Points

Test Accuracy: 0.809

Predicted: Chair / True: Chair
Predicted: Lamp / True: Vase
Predicted: Lamp / True: Lamp

50 Points

Test Accuracy: 0.510

Predicted: Chair / True: Chair
Predicted: Lamp / True: Vase
Predicted: Lamp / True: Lamp

Number of Points - Segmentation

Points10000 (baseline)5000100050010050
Test accuracy 0.9040.9040.9000.8900.8180.778

The number of points evaluated was decreased from its baseline of 10000 points.

As expected, as the number of points decreased, the accuracy decreased. From the baseline to 500 points, the accuracy hangs around the 89-90% range, and then dips around 100 points to 82%. Decreasing to 50 points lowers the accuracy slightly more to 78%.

Compared to classification, the accuracy drop for sparser point clouds is not as severe. Despite the sparsity, the points are still spread out enough to hint at which part each point belongs to.

5000 points

Test Accuracy: 0.904

Accuracy: 0.996

Accuracy: 0.994

Accuracy: 0.993

1000 points

Test Accuracy: 0.900

Accuracy: 0.989

Accuracy: 0.996

Accuracy: 0.991

500 points

Test Accuracy: 0.890

Accuracy: 0.966

Accuracy: 0.996

Accuracy: 0.990

100 points

Test Accuracy: 0.818

Accuracy: 0.930

Accuracy: 0.810

Accuracy: 0.970

50 points

Test Accuracy: 0.778

Accuracy: 0.920

Accuracy: 0.720

Accuracy: 0.960

AI Assistance

I used Claude to help generate accuracy tables for each robustness experiment.