Assignment 5: Point Cloud Processing

Name: Ruinian (Ray) Xue

AndrewID: ruinianx

Q1. Classification Model (30 points)

Implement the classification model in models.py.

Result of classification

Test accuracy: 0.9790

Visualize

Note: the visualization code is adapted from my own code in assignment 2. (The same applies in later visualizations)

Success

ChairVaseLamp
Pred: chairPred: vasePred: lamp

Failure

GT: chairGT: vaseGT: lamp
Pred: lampPred: lampPred: vase
Why fail:

In the first case, the chair that has rec recognized as a lamp looks similar to a lamp itself. It is nearly flat and has little depth so it is difficult to recognize.

In the second case, the vase with flowers on top looks like a lamp because it has a thin middle and is wider in the top and bottom.

In the third case, the the lamp looks like the vase with some flower on it.

Based on the three failure cases, we can know that if the shape of an object is ambiguous and vague, it is easily to distinguished as another type.

Q2. Segmentation Model (30 points)

Implement the segmentation model in models.py.

Result of Segmentation

Test accuracy: 0.9018

Visualize

Accuracy for singular cases

Object IndexAccuracyNote
4260.3868worst
2350.50762nd worst
3970.99393nd best
5050.99512nd best
4710.9976best
ObjectGround TruthPrediction
426 (worst)
235 (2nd worst)
397 (3rd best)
505 (2nd best)
471 (best)
Why Best:

For those best cases, they look really similar to a typical chair and have their own distinguish parts so the border of each part is really clear, because the shape is normal and easy to distinguish by the model. Even though the shapes of components are different, their overall structure is fixed.

Why Worst:

The worst two cases have similarities. We can notice that the shape itself is quite different from a normal chair. We can understand that the model learns what a typical chair looks like and cannot understand a shave that is totally indifferent from a chair.

For the chair and sofa that has a thicker bottom, the model can't recognise the bottom seat clearly, and can't distinguish it from the leg or the armrest.

Q3. Robustness Analysis (20 points)

Conduct 2 experiments to analyze the robustness of your learned model. Each experiment is worth 10 points. A maximum of 20 points is possible for this question.

Input points

Procedure

Using the fixed model or really trained in Q1 and Q2 above, I changed numbers of point in args to compare the robustness under different points.

I set number of 1000, 500, 100, and 50 as input, keeping other factors the same, and compare their result in classification and segmentation.

Result

Number of PointsClassification AccuracySegmentation Accuracy
10000 (base)0.97900.9018
10000.97380.8978
5000.96960.8818
1000.89400.8113
500.63690.7745
Classification
Number of PointsSuccessFailureFailure
1000GT: chair, Pred: chairGT: chair, Pred: lampGT: lamp, Pred: vase
50GT: lamp, Pred: lampGT: chair, Pred: lampGT: vase, Pred: lamp
Segmentation

When tring to pick up the visualization for segmentation, I noticed this chair appears multiple times. So I just put it here and compare. This result is also interesting that in segementation, despite different rotation and number of points, this chair with index 235, is almost always among the worst cases.

Number of PointsGround TruthPredictionAccuracy
10000 (base)0.5076
10000.4310
500.2600

Explanation

The model itself is robust as long as the number of point isn't too low. As the point number gets low, the accuracy didn't quite drop much. From 0.97 to 0.89, compared with 10000 and 100 in classification, and 0.90 to 0.81 in segementation. The result is interesting that even though in classification, the accuracy is better at the number of 10000, it drops quicker than segmentation when the number of points drops.

Why drop/fail? As the number of points drops, the shape of each component and the shape of the whole object is becoming more vague, so it is more difficult to recognize, especially in those cases that is already ambiguous. As I mentioned before, the chair with index 235 appers among the worst cases despite the setting because of its own shape.

Rotation

Procedure

Using the fixed model or really trained in Q1 and Q2 above, I added args named rotate_angle in evaluation part, and set angles from 15 to 180 as input, keeping other factors the same, and compare their result in classification and segmentation.

Result

Rotation Angle (around Z axis)Classification AccuracySegmentation Accuracy
0° (base)0.97900.9018
15°0.93070.8326
45°0.37250.6256
90°0.24240.4041
180°0.55510.3494
Classification
Rotation AngleSuccessFailureFailure
15°GT: chair, Pred: chairGT: chair, Pred: lampGT: lamp, Pred: vase
90°GT: vase, Pred: vaseGT: chair, Pred: vaseGT: lamp, Pred: vase
Segmentation
Rotation AngleGround TruthPredictionAccuracy
0° (base)0.5076
15°0.4185
90°0.1163

Explanation

Different from the number of points, the model is not robust towards rotation. In most cases, the more the rotation is, the worst the result is, as sgementation keeps drop all the way to 180°. But when the rotation becomes 180°, the classification accuracy get better a little bit.

It is also easy to understand that, because the model learned how to divide a chair into different components via certain angle, and if the model turns the chair to a totally different side, the structure totally changed so the segmentation went worse. As segmentation relies heavily on position and structure of the whole object and classification just need to have a eye on the overall shape.