Assignment 5

Q1. Classification Model

Pred: Chair, GT: Chair
Prediction 1
Pred: Vase, GT: Vase
Prediction 619
Pred: Lamp, GT: Lamp
Prediction 725
Pred: Lamp, GT: Chair
Prediction 406
Pred: Lamp, GT: Vase
Prediction 618
Pred: Vase, GT: Lamp
Prediction 726
The test accuracy is 97.48%.
In the second row, the first point cloud is misclassified as a lamp because it is an unconventional chair, which looks folded. That makes it confusing for the model. The second point cloud in the second row is just a cylinder with no features that make it distinguishable as a vase. The third point cloud in the second row has a very round body which makes it an unusual lamp and makes it look more like a vase.

Q2. Segmentation Model

Predicted
Predicted segmentation 113
Ground Truth
Ground truth segmentation 113
Predicted
Predicted segmentation 162
Ground Truth
Ground truth segmentation 162
Predicted
Predicted segmentation 275
Ground Truth
Ground truth segmentation 275
Predicted
Predicted segmentation 26
Ground Truth
Ground truth segmentation 26
Predicted
Predicted segmentation 351
Ground Truth
Ground truth segmentation 351
The test accuracy is 90.21%.
The 4th and 5th point clouds are examples of bad predictions. Clearly, the chairs are very atypical and even a human annotator would struggle to segment those chairs accurately.

Q3. Robustness Analysis

Q3.1 Rotation Analysis

A. Classification Model

Pred: lamp, GT: Chair
Prediction with rotation 1
Pred: lamp, GT: Vase
Prediction with rotation 619
Pred: chair, GT: Lamp
Prediction with rotation 725
Pred: lamp, GT: Chair
Prediction with rotation 406
Pred: vase, GT: Vase
Prediction with rotation 618
Pred: vase, GT: Lamp
Prediction with rotation 726
Test accuracy is 45.96%.
For ease of comparison, I chose to visualize the same examples as in Q1.

The approach I followed was to rotate each point cloud in the test set by a uniform random elevation between -90 and 90 degrees, and a uniform random azimuth between 0 to 360 degrees. Clearly, this model is not robust to rotations as the test accuracy has dropped precipitously. It can also be seen by the visualized examples in which the same examples without rotation are classified correctly whereas incorrectly with rotations.

B. Segmentation Model

Predicted
Predicted segmentation rotation 113
Ground Truth
Ground truth segmentation 113
Predicted
Predicted segmentation rotation 162
Ground Truth
Ground truth segmentation 162
Predicted
Predicted segmentation rotation 275
Ground Truth
Ground truth segmentation 275
Predicted
Predicted segmentation rotation 26
Ground Truth
Ground truth segmentation 26
Predicted
Predicted segmentation rotation 351
Ground Truth
Ground truth segmentation 351
The test accuracy is 48.94%.
For ease of comparison, I chose to visualize the same examples as in Q2.

Followed the same approach of random rotations as in Q3.1A. Clearly, the segmentation model is not robust to rotations either based on the drop in test accuracy. The same is clear from the visualized examples.

Q3.2 Changing number of sampled points

A. Classification Model

Pred: Chair, GT: Chair
Prediction num points 1
Pred: Vase, GT: Vase
Prediction num points 619
Pred: Vase, GT: Lamp
Prediction num points 725
Pred: Vase, GT: Chair
Prediction num points 406
Pred: Lamp, GT: Vase
Prediction num points 618
Pred: Lamp, GT: Lamp
Prediction num points 726
The test accuracy is 90.97% sampling 100 points per point cloud.
For ease of comparison, I chose to visualize the same examples as in Q1.

The model is surprisingly robust to reducing the number of sampled points. Even at 50 points per point cloud, the accuracy was around 75%. This indicates that even as few as 50-100 points provide enough structure for the model to distinguish between different classes. This is also evident from the visualized examples.

B. Segmentation Model

Predicted
Predicted segmentation num points 113
Ground Truth
Ground truth segmentation num points 113
Predicted
Predicted segmentation num points 162
Ground Truth
Ground truth segmentation num points 162
Predicted
Predicted segmentation num points 275
Ground Truth
Ground truth segmentation num points 275
Predicted
Predicted segmentation num points 26
Ground Truth
Ground truth segmentation num points 26
Predicted
Predicted segmentation num points 351
Ground Truth
Ground truth segmentation num points 351
The test accuracy is 80.97% sampling 100 points per point cloud.
For ease of comparison, I chose to visualize the same examples as in Q2.

The segmentation model is also surprisingly robust to fewers points being sampled. The test accuracy is around 75% with as few as 50 points per point cloud. This is also clear from the visualized examples.