Name: Ruinian (Ray) Xue
AndrewID: ruinianx
Assignment 5: Point Cloud ProcessingQ1. Classification Model (30 points)Result of classificationVisualizeSuccessFailureWhy fail:Q2. Segmentation Model (30 points)Result of SegmentationVisualizeAccuracy for singular casesWhy Best:Why Worst:Q3. Robustness Analysis (20 points)Input pointsProcedureResultClassificationSegmentationExplanationRotationProcedureResultClassificationSegmentationExplanation
Implement the classification model in models.py.
Input: points clouds from across 3 classes (chairs, vases and lamps objects)
Output: propapibility distribution indicating predicted classification (Dimension: Batch * Number of Classes)
Test accuracy: 0.9790
Note: the visualization code is adapted from my own code in assignment 2. (The same applies in later visualizations)
| Chair | Vase | Lamp |
|---|---|---|
![]() | ![]() | ![]() |
| Pred: chair | Pred: vase | Pred: lamp |
| GT: chair | GT: vase | GT: lamp |
|---|---|---|
![]() | ![]() | ![]() |
| Pred: lamp | Pred: lamp | Pred: vase |
In the first case, the chair that has rec recognized as a lamp looks similar to a lamp itself. It is nearly flat and has little depth so it is difficult to recognize.
In the second case, the vase with flowers on top looks like a lamp because it has a thin middle and is wider in the top and bottom.
In the third case, the the lamp looks like the vase with some flower on it.
Based on the three failure cases, we can know that if the shape of an object is ambiguous and vague, it is easily to distinguished as another type.
Implement the segmentation model in models.py.
Input: points of chair objects (6 semantic segmentation classes)
Output: segmentation of points (Dimension: Batch * Number of Points per Object * Number of Segmentation Classes)
Test accuracy: 0.9018
| Object Index | Accuracy | Note |
|---|---|---|
| 426 | 0.3868 | worst |
| 235 | 0.5076 | 2nd worst |
| 397 | 0.9939 | 3nd best |
| 505 | 0.9951 | 2nd best |
| 471 | 0.9976 | best |
| Object | Ground Truth | Prediction |
|---|---|---|
| 426 (worst) | ![]() | ![]() |
| 235 (2nd worst) | ![]() | ![]() |
| 397 (3rd best) | ![]() | ![]() |
| 505 (2nd best) | ![]() | ![]() |
| 471 (best) | ![]() | ![]() |
For those best cases, they look really similar to a typical chair and have their own distinguish parts so the border of each part is really clear, because the shape is normal and easy to distinguish by the model. Even though the shapes of components are different, their overall structure is fixed.
The worst two cases have similarities. We can notice that the shape itself is quite different from a normal chair. We can understand that the model learns what a typical chair looks like and cannot understand a shave that is totally indifferent from a chair.
For the chair and sofa that has a thicker bottom, the model can't recognise the bottom seat clearly, and can't distinguish it from the leg or the armrest.
Conduct 2 experiments to analyze the robustness of your learned model. Each experiment is worth 10 points. A maximum of 20 points is possible for this question.
Using the fixed model or really trained in Q1 and Q2 above, I changed numbers of point in args to compare the robustness under different points.
I set number of 1000, 500, 100, and 50 as input, keeping other factors the same, and compare their result in classification and segmentation.
| Number of Points | Classification Accuracy | Segmentation Accuracy |
|---|---|---|
| 10000 (base) | 0.9790 | 0.9018 |
| 1000 | 0.9738 | 0.8978 |
| 500 | 0.9696 | 0.8818 |
| 100 | 0.8940 | 0.8113 |
| 50 | 0.6369 | 0.7745 |
| Number of Points | Success | Failure | Failure |
|---|---|---|---|
| 1000 | GT: chair, Pred: chair | GT: chair, Pred: lamp | GT: lamp, Pred: vase |
| 50 | GT: lamp, Pred: lamp | GT: chair, Pred: lamp | GT: vase, Pred: lamp |
When tring to pick up the visualization for segmentation, I noticed this chair appears multiple times. So I just put it here and compare. This result is also interesting that in segementation, despite different rotation and number of points, this chair with index 235, is almost always among the worst cases.
| Number of Points | Ground Truth | Prediction | Accuracy |
|---|---|---|---|
| 10000 (base) | ![]() | ![]() | 0.5076 |
| 1000 | ![]() | ![]() | 0.4310 |
| 50 | ![]() | ![]() | 0.2600 |
The model itself is robust as long as the number of point isn't too low. As the point number gets low, the accuracy didn't quite drop much. From 0.97 to 0.89, compared with 10000 and 100 in classification, and 0.90 to 0.81 in segementation. The result is interesting that even though in classification, the accuracy is better at the number of 10000, it drops quicker than segmentation when the number of points drops.
Why drop/fail? As the number of points drops, the shape of each component and the shape of the whole object is becoming more vague, so it is more difficult to recognize, especially in those cases that is already ambiguous. As I mentioned before, the chair with index 235 appers among the worst cases despite the setting because of its own shape.
Using the fixed model or really trained in Q1 and Q2 above, I added args named rotate_angle in evaluation part, and set angles from 15 to 180 as input, keeping other factors the same, and compare their result in classification and segmentation.
| Rotation Angle (around Z axis) | Classification Accuracy | Segmentation Accuracy |
|---|---|---|
| 0° (base) | 0.9790 | 0.9018 |
| 15° | 0.9307 | 0.8326 |
| 45° | 0.3725 | 0.6256 |
| 90° | 0.2424 | 0.4041 |
| 180° | 0.5551 | 0.3494 |
| Rotation Angle | Success | Failure | Failure |
|---|---|---|---|
| 15° | GT: chair, Pred: chair | GT: chair, Pred: lamp | GT: lamp, Pred: vase |
| 90° | GT: vase, Pred: vase | GT: chair, Pred: vase | GT: lamp, Pred: vase |
| Rotation Angle | Ground Truth | Prediction | Accuracy |
|---|---|---|---|
| 0° (base) | ![]() | ![]() | 0.5076 |
| 15° | ![]() | ![]() | 0.4185 |
| 90° | ![]() | ![]() | 0.1163 |
Different from the number of points, the model is not robust towards rotation. In most cases, the more the rotation is, the worst the result is, as sgementation keeps drop all the way to 180°. But when the rotation becomes 180°, the classification accuracy get better a little bit.
It is also easy to understand that, because the model learned how to divide a chair into different components via certain angle, and if the model turns the chair to a totally different side, the structure totally changed so the segmentation went worse. As segmentation relies heavily on position and structure of the whole object and classification just need to have a eye on the overall shape.