Assignment 5: Point Cloud Processing¶
The model struggled on the failure cases because it was trying to make predictions for more complicated structures that include elements that could easily be associated with the other classes. For instance, the model predicted a lamp for the last image above even though it was a vase. Here, we can see the structure of the vase includes elements that could be associated with a lamp, such as the stand-like base and the thin screen on top of the base. Because these structures are more complicated, the model wasn't able to properly predict on these structures.
Question 2¶
| Test Accuracy |
|---|
| 90.24% |
| Predicted | Ground Truth | Good or Bad | Prediction Accuracy |
|---|---|---|---|
 |
 |
Good | 94.6% |
 |
 |
Good | 98.4% |
 |
 |
Good | 88.5% |
 |
 |
Bad | 69% |
 |
 |
Bad | 72.8% |
For the good segmentation predictions here, we can see that each component of the chair is clearly marked out in both the predicted and ground truth, as these components are fairly distinct from each other. The seat is clearly distinct from the back of the chair, which is clearly distinct from the legs, and so on.
For the bad cases, we can see that the component demarcations get a little more blurry. We see that for the first bad case, the line between the base of the chair and the back of the chair is a little more blurry, as these components blend together. Similarly, for the second case, we see that the seat component and the x-shaped base don't become distinctly different for a little bit (almost as if they are stuck together), so the model predicts the segmented region to be further down than it really is.
Question 3¶
Experiment 1: Number of Points¶
Procedure¶
For the experiment below, I change the number of points I sample and toggle from 25 points to 100 points to 400 points and finally to 1600 points. I represent my results in the tables below, highlighting the number of points sampled, the accuracy, and visualizations of my results. I also showcase the results of sampling 1000 points (baseline).
Classification¶
| Num Points | Accuracy | Image 1 | Image 2 | Image 3 | Image 4 | Image 5 |
|---|---|---|---|---|---|---|
| 25 | 60.86% |  |
 |
 |
 |
 |
| 100 | 93.39% |  |
 |
 |
 |
 |
| 400 | 92.86% |  |
 |
 |
 |
 |
| 1000 (original) | 92.76% |  |
 |
 |
 |
 |
| 1600 | 92.44% |  |
 |
 |
 |
 |
For classification, we see a precipitous drop in performance when only sampling 25 points here. As per the visualizations above, we can see that the structure of the object we're visualizing is not at all noticeable. Therefore, our model does poorly in classifying the object. However, as the number of points increase, and our structures start becoming more and more distinct, the object is able to classify much better. However, we see a slight dip when going from 100 points to 400 points. This could be because when sampling more points, we are introducing more redundant points, which can dilute or shift which features we select when max pooling.
Segmentation¶
| Num Points | Accuracy | Image 1 | Image 2 | Image 3 | Image 4 | Image 5 |
|---|---|---|---|---|---|---|
| 25 | 64.79% |  |
 |
 |
 |
 |
| 100 | 80.49% |  |
 |
 |
 |
 |
| 400 | 88.89% |  |
 |
 |
 |
 |
| 1000 (original) | 90.24% |  |
 |
 |
 |
|
| 1600 | 90.41% |  |
 |
 |
 |
 |
For segmentation, when sampling only 25 points, our model's segmentation predictions are poor because of the same reason mentioned above: our object's structure is not noticeable at all, and therefore it's individual components would be very difficult to distinguish. However, as our structure becomes more and more clear as we sample more and more points, our model is able to better segment the object into the 6 distinct classes, especially as it has a more refined object to work with.
Experiment 2: Rotation of Input Points¶
Procedure¶
For both tasks, I rotated the input point clouds by 0 (original), 30, 45, 60, and 75 degrees along the z-axis inside their respective eval scripts, and ran the evaluation. I showcase the output of my experiments in the tables below, highlighting the rotation angle, accuracy, and some visualizations of my results.
Classification¶
| Degree | Accuracy | Image 1 | Image 2 | Image 3 | Image 4 | Image 5 |
|---|---|---|---|---|---|---|
| 0 (original) | 92.76% |  |
 |
 |
 |
 |
| 30 | 81.74% |  |
 |
 |
 |
 |
| 45 | 56.77% |  |
 |
 |
 |
 |
| 60 | 34.94% |  |
 |
 |
 |
 |
| 75 | 28.43% |  |
 |
 |
 |
 |
Here, we can clearly see how the model is not able to classify properly when our rotations of our point clouds get bigger and bigger. When we keep the rotation angle small, the point cloud representation is still relatively the same (and the geometric relationships the model has learned hasn't changed all that much). However, as our rotation angle increases, the geometric relationships and the representation that the model learned no longer apply, since our point cloud has rotated so much, which makes it difficult for the model to classify properly. We can see this in the degradataion of performance as our angle of rotation goes up.
Segmentation¶
| Degree | Accuracy | Image 1 | Image 2 | Image 3 | Image 4 | Image 5 |
|---|---|---|---|---|---|---|
| 0 (original) | 90.22% | |
|
|
|
|
| 30 | 67.66% |  |
 |
 |
 |
 |
| 45 | 55.81% |  |
 |
 |
 |
 |
| 60 | 49.47% |  |
 |
 |
 |
 |
| 75 | 42.91% |  |
 |
 |
 |
 |
Here, we see a similar issue as in the classification case. We can see that as our angle of rotation increases, our model's ability to segment decreases. This is for similar reasons as stated above - the representations the model learned, as well as the geometric relationships between parts of the chair and the points, no longer apply because our points have been rotated by so much. Therefore, our model is not able to handle large angle rotations well.