Homework5 learning for 3D

Contact Information:

• Name: Shibo Zhao
• Email: shiboz@andrew.cmu.edu

#ipython nbconvert --to HTML your_notebook.ipynb

1. Classification Model (40 points)

In this assignment, you will implement a PointNet based architecture for classification and segmentation with point clouds (you don't need to worry about the tranformation blocks). Q1 and Q2 focus on implementing, training and testing models. Q3 asks you to quantitatively analyze model robustness. Q4 (extra point) involves locality.

Training Result

To run the code:

`python train.py --task cls`

Validation Accuracy (Accuracy/Epoch)	Training Loss (Loss/Iteration)

The accuracy of my best model was 97%. Below are a few correctly predicted examples for each class.

Good Cases

Chairs

Chairs	Chairs	Chairs

Vase

Vase	Vase	Vase

Lamp

Lamp	Lamp	Lamp

Failure Cases

Chair Accuracy=100%

Chairs: We didn't found the failure cases for the Chairs. The accuracy is the 100% for chair.

Vase Accuracy=92%

Predicted Lamp	Predicted Lamp	Predicted Chair

Lamp Accuracy=97%

Predicted Vase	Predicted Vase	Predicted Vase

Conclusion

It seems that the model relies on the structure to finish the classification. For example, the model have difficult to distinuish the circular lamps from vases since it has simialr structure and both have round cylindrical components. However, I think these cases are very diffcult to distinuguish even by human and could be solved if we use some texture information.

2. Segmentation Model (40 points)

To run the code:

`python train.py --task seg`

Validation Accuracy (Accuracy/Epoch)	Training Loss (Loss/Iteration)

On the test set, the accuracy of my best model was 88%. We show the most correctly predicted point clouds below.

Good Segmentation

GT	Predicted Chair (Accuracy=0.9122)

GT	Predicted Chair (Accuracy=0.9232)

GT	Predicted Chair (Accuracy=0.9432)

Failure Segmentation

GT	Predicted Chair (Accuracy=0.764)

GT	Predicted Chair (Accuracy=0.673)

GT	Predicted Chair (Accuracy=0.567)

Conclusion

Checking the failure cases, we can find that the model is difficult to classify the head of chair and arm of chair since it is difficult to recongnize that it is in the same component or not. The potential reason behind it is that the model is predict the 3D position of each point and not consider the connections between the 3D points.

3. Robustness Analysis

Num of Points Effect

We evaluate our model with num_points=[10000,7500,5000,2500,1000,500,100] by changing the num_points argument in the code.

Classification	Segmentation

Classification Result

We can find the classification model is very robust even if the num_points is 100 and its accracy is still high which can matain 0.95. I think the potential reason is that the chair, lamps and vase has distinct structure which is not difficult to classify.

Segmentation Result

We can find the segmentation model is also robust. The accracy is almost same when num_points decrase from 10000 to 500. We can see there is a siginificant drop when the num_point is 100.

Rotation Effect

We evaluate our model with rotation=[0,30,60,90,120,150,180] degree by rotating the point cloud along the X direction. Then we send the rotated the point cloud to our model.

Classification	Segmentation

Conclusion

I evaluate the both the robustness of segmentation model and classification model with rotation value of 0,30, 60,90,120,150,180. We can see that the both model cannot work well in such environment since model highly relies on the point position to achieve classification and segmentation. We also notice that a 180 degree rotation perform better than 90 degree since the position of points with 180 degree rotation is more close to the training set.