3D Vision Assignment 5: Point Cloud Processing

Q1: Classification Model

Test accuracy: 0.963

Prediction Visualizations

Example 256
Example 293
Example 486
Example 605
Example 807

Failures

Example 595
Example 916
Example 687

This model featured a simple three-layer linear architecture with a hidden dimension of 128. It ran for 250 epochs at the default learning rate. While most predictions were accurate, some of the failure cases exhibited here struggled likely because of their unconventional shapes. The chair, for example, has a curved shape with no large flat surfaces which is somewhat reminiscent of a vase. With the others, the lamps and vases have very similar structures, so these failures feature lamps that have a cylindrical vase-like structure and vases with open lamp-like bowls.

Q2: Segmentation

Test accuracy: 0.802

Results

Prediction
Ground Truth

Prediction Accuracy: 0.8920


Prediction
Ground Truth

Prediction Accuracy: 0.8622


Prediction
Ground Truth

Prediction Accuracy: 0.8027


Prediction
Ground Truth

Prediction Accuracy: 0.6172


Prediction
Ground Truth

Prediction Accuracy: 0.6941

The predictions were mostly accurate but struggled with distnguishing the boundaries of each part of the chairs. Especially in the case of the seat of the chairs, the model had trouble distinguishing where the seats ended, while other parts (back, legs) did not have as much of this issue. Notably, in the last two cases which had a much higher proportion of failure, the unconventional geometry tripped up the model; the protrusion of the seat on the sides of the second-to-last example made the model believe that they were part of the armrests, and in the last example, it had a hard time distinguishing what part of the smooth curve was armrest vs seat, which might be confusing even for a human. The lack of local feature analysis makes it harder for the model to distinguish relative features for each point to determine which class they might belong to.

Q3: Robustness Analysis

Exploration 1: Input Point Cloud Rotation

Classification

Test accuracy: 0.226

Prediction Visualizations (Original Set)

Example 256
Example 293
Example 486
Example 605
Example 807

More Failures

Example 501
Example 681
Example 927

As expected, the test accuracy dropped significantly when trained with upright models and tested on rotated ones. In this approach, I applied a 135-degree rotation around the Z axis for each input point cloud for evaluation only. The accuracy turned out to be worse than random guessing (which might have hovered more around 0.333), as the model frequently confused features across classes. Of the original randomly selected set of correct predictions, 4/5 of them failed under rotation. Lamps were a common prediction as many lamps in the training set have wildly different geometries, some of which resemble upside-down chairs.

Segmentation

Test accuracy: 0.433

Results

Prediction
Ground Truth

Prediction Accuracy: 0.2357


Prediction
Ground Truth

Prediction Accuracy: 0.3369


Prediction
Ground Truth

Prediction Accuracy: 0.3487


Prediction
Ground Truth

Prediction Accuracy: 0.2885


Prediction
Ground Truth

Prediction Accuracy: 0.3767

Similarly to classification, this model suffered greatly from the rotation. While some parts remained accurate, like the seats of each chair, the ranges for each part varied wildly and sometimes did not cover the correct parts. The model definitely seemed to learn labels based on absolute location as well, since it assigned chair backs as legs and legs as backs when point clouds were rotated. Evaluating local features might improve this somewhat, but overall the model is not very robust to rotation.

Evaluation 2: Changing Number of Points

Classification

Test accuracy: 0.929

Prediction Visualizations (Original Set)

Example 256
Example 293
Example 486
Example 605
Example 807

More Failures

Example 585
Example 685
Example 922

Test accuracy dropped slightly but not by much, which was expected. In this evaluation, I reduced the number of points from 10000 to 500 to make a sparse set of input points. The original evaluated set of randomly selected objects remained correctly predicted, but other failures were selected due to the wider pool to display the variation in failure cases. A lesser number of points reflects less input features for the model to evaluate on and apply learned features, but it was still able to identify most. Not pictured is an additional test with 5000 points, which yielded a test accuracy of 0.931; this implies that the correlation between density and accuracy has a more gradual decline after a certain threshold.

Segmentation

Test accuracy: 0.799

Results

Prediction
Ground Truth

Prediction Accuracy: 0.8880


Prediction
Ground Truth

Prediction Accuracy: 0.8500


Prediction
Ground Truth

Prediction Accuracy: 0.8040


Prediction
Ground Truth

Prediction Accuracy: 0.6460


Prediction
Ground Truth

Prediction Accuracy: 0.7260

This model did not see as much of an accuracy drop as expected. Just like with classification, I dropped the number of points from 10000 to 500 in evaluation. In most more-successful cases, the accuracy remained about the same, but in the failure cases, the accuracy actually increased slightly. This might be because the proportion of incorrect points is relatively smaller just by virtue of there being a smaller pool of points to classify, though notably in the last example the range of the armrests more closely matches the ground truth at 500 points than 10000 points (extending lower).

Q4: Locality

Classification

Test accuracy: 0.978

Original Failures

Example 595 (Original Classification)
Example 687 (Original Classification)
Example 916 (Original Classification)

Original Failures

Example 595 (New Classification)
Example 687 (New Classification)
Example 916 (New Classification)

I followed a simple PointNet++-inspired structure for injecting locality into each model. I created a KNN-based local feature block before the existing global MLP that pools features from each point's k nearby points. These features are evaluated before the global MLP and concatenated with the original coordinates. As the results show, with the same configuration of default parameters, 250 epochs, and 10000 points, the test accuracy rose slightly from 0.963 to 0.978, showing that local feature evaluation did improve the model. Above are the failure cases from the original test without locality; while example 687 still evaluated incorrectly as a lamp instead of a vase, the other two examples actually saw correct classifications with locality implemented.

Segmentation

Test accuracy: 0.864

Results

Nonlocal Prediction
Locality Prediction
Ground Truth

Prediction Accuracy: 0.942


Nonlocal Prediction
Locality Prediction
Ground Truth

Prediction Accuracy: 0.821


Nonlocal Prediction
Locality Prediction
Ground Truth

Prediction Accuracy: 0.939


Nonlocal Prediction
Locality Prediction
Ground Truth

Prediction Accuracy: 0.695


Nonlocal Prediction
Locality Prediction
Ground Truth

Prediction Accuracy: 0.737

The segmentation model locality injection followed the same structure as the classification locality method. We see a much more significant increase in test accuracy here, where locality matters even more in refining the edges of segmented parts. With the ability to see more local features, the model was able to better predict the boundaries of each region within each subset of points. Overall test accuracy jumped from 0.802 to 0.864, and each model except for one in the original set saw an increase in accuracy and better-defined boundaries. Peculiarly, the one with a drop in accuracy saw an outward creep of the chair back region, potentially due to local feature processing obscuring where the broader boundary of the chair leg might be. It also witnessed wider but more spread-out creep of the seat region which was not present in other models.