Assignment 5 - Avi Athota

1 Classification Model

My test accuracy was 96.64%.

   

These are random test point clouds that were classified correctly. The predicted classes from left to right are: chair, vase, lamp.

   

These are failure cases. From left to right: a chair predicted as a lamp, a vase predicted as a lamp, and a lamp predicted as a vase.

Explanations of the above failure cases (from left to right):
This chair appears to be folded (only two legs visible) and very compact, which makes it not really resemble a regular chair. It was instead labelled as a lamp.
This vase has a flower sticking out of it, which looks like a lamp pole/light, which is likely why this was labeled as a lamp.
This lamp has a very large and round base which resembles a vase, which is what it was misclassified as.

2 Segmentation Model

My test accuracy was 89.81%.

 

Object 397: GT (left) vs prediction (right). Accuracy: 99.84%.

 

Object 297: GT (left) vs prediction (right). Accuracy: 99.61%.

 

Object 203: GT (left) vs prediction (right). Accuracy: 92.95%.

 

Object 142 (bad prediction): GT (left) vs prediction (right). Accuracy: 46.00%.

 

Object 577 (bad prediction): GT (left) vs prediction (right). Accuracy: 39.22%.

Here were my prediction accuracies for each object:
- Object 397: 99.84%
- Object 297: 99.61%
- Object 203: 92.95%
- Object 142: 46.00%
- Object 577: 39.22%

Interpretation: The model seems to do really well on chairs that conform to the standard geometry (back, seat, four legs), as 297 and 397 were almost perfect. Also, you can see from 203 that a common place of error is the boundaries between parts (like where the seat meets the legs in this case). 142 (a beach chair looking thing) and 577 (a sofa chair) are segmented poorly, as their shapes are pretty different from typical chairs.

3 Robustness Analysis

3.1 Rotations

I tried rotating every test point cloud around the x-axis and evaluated the same models from the above parts without retraining.

Test accuracy at each rotation:
- 0°: classification 96.64%, segmentation 89.81%
- 15°: classification 92.34%, segmentation 82.72%
- 30°: classification 73.14%, segmentation 69.93%
- 45°: classification 43.23%, segmentation 56.02%
- 60°: classification 29.91%, segmentation 34.71%
- 90°: classification 24.76%, segmentation 24.65%

   

0° (from Q1): predicted chair, vase, lamp (all correct).

   

45°: predicted chair, lamp, lamp (the vase is now wrong).

   

90°: predicted lamp, lamp, vase (all wrong).

     

0° (from Q2): GT and prediction for object 203 (92.95%) and object 397 (99.84%).

     

45°: GT and prediction for object 203 (66.72%) and object 397 (65.12%).

     

90°: GT and prediction for object 203 (30.87%) and object 397 (37.84%).

Interpretation: Both of the models are clearly very sensitive to rotation. For classification the model goes from getting all right at 0 degrees to getting 1 wrong at 45 degrees to getting all of them wrong at 90 degrees. Similarly, the segmentation accuracies start above 90%, dropping to ~60% and ~30% as the degree is changed to 45 and 90. Since all the training images are upright, the model utilizes features like height and width which break when the object is rotated.

3.2 Number of points

Instead of using all 10k points, I randomly sampled various numbers of points per object.

Test accuracy for each number of points:
- 10000 (Q1/Q2): classification 96.64%, segmentation 89.81%
- 5000: classification 96.50%, segmentation 89.73%
- 1000: classification 96.36%, segmentation 88.83%
- 500: classification 95.63%, segmentation 87.30%
- 100: classification 94.30%, segmentation 79.60%
- 50: classification 90.98%, segmentation 74.68%
(Since the points are sampled randomly, these are averaged over my eval runs, which were all within 1% of each other.)

   

10000 points (from Q1): predicted chair, vase, lamp (all correct).

   

100 points: predicted chair, vase, vase (the lamp is now wrong).

   

50 points: predicted chair, lamp, chair (only the chair is correct).

     

10000 points (from Q2): GT and prediction for object 203 (92.95%) and object 397 (99.84%).

     

100 points: GT and prediction for object 203 (86.00%) and object 397 (99.00%).

     

50 points: GT and prediction for object 203 (84.00%) and object 397 (90.00%).

Interpretation: Both models are actually surprisingly capable of handling fewer points. The classification is still at over 90% even with just 50 points. Segmentation drops a little bit more (to ~75% at 50 points). This is probably because the max-pool doesn't need so many key points to recognize overall shape for classification, while the segmentation needs a little more to be able to label every point.