Assignment 5 - Pointcloud Classification and Segmentation

Author: Kailash Jagadeesh

Course: 16-825 Learning for 3D Vision — Carnegie Mellon University

Section 1: Classification Model

Test Accuracy of the best Model : 98.11%
Visualization Table:

Input Point-Cloud Ground Truth Predicted Class Prediction - Success/Failure
Cls_task1 Chair Chair Success
Cls_task2 Vase Vase Success
Cls_task3 Lamp Lamp Success
Cls_task4 Lamp Vase Failure
Cls_task5 Vase Lamp Failure

By checking the failure cases it is clear that the pointclouds that the model fails for has a very similar geometry for the other class. Even I was confused a little bit. Pointclouds inherently check only the structure but has no information about the semantics. So if there's an object which has a similar geometry but is a different class it is bound to fail. We should use semantic information or image information for better performance. Interestingly, there was no failure with the chairs class. I believe because of the completely different geometric figure, it would be hard to mis-classify the chair class pointclouds.

Section 2: Segmentation Model

Test Accuracy of the best Model : 90.32%
Visualization Table:

GT Point-Cloud Predicted Point-Cloud Segmentation - Success/Failure Prediction Accuracy
gt_1 segment_task1 Failure 50.82%
gt_1 segment_task1 Success 96.95%
gt_1 segment_task1 Success 97.99%
gt_1 segment_task1 Failure 54.76%
gt_1 segment_task1 Success 96.17%

By checking the failure cases I observe that the segmentation fails if the pointclouds are too big. If the object itself is very big then the segmentation gets confused and is not able to distinguish certain features. This might be worse when curved features are involved as well.

Section 3: Robustness Analysis

Section 3.1: Classification Model Performance

Section 3.1.1 : Rotation of Pointclouds

Procedure:

  1. Baseline Evaluation: First, evaluate the model on the original test set without any rotation to establish baseline accuracy (0° rotation).

  2. Rotation Transformation: For each test point cloud, apply a 3D rotation transformation around the z-axis using Euler angles. The rotation is performed by:

    • Converting the rotation angle from degrees to radians
    • Computing a 3×3 rotation matrix using scipy's Rotation.from_euler() function
    • Applying the rotation matrix to each point: rotated_points = points @ rotation_matrix.T

  3. Evaluation at Multiple Angles: Test the model at rotation angles of 15°, 30°, 45°, 60° and 90° around the z-axis. For each angle:

    • Rotate all test point clouds by that angle
    • Pass the rotated point clouds through the trained model
    • Compute predictions and compare with ground truth labels
    • Calculate accuracy and accuracy drop compared to baseline

Accuracy Performance Table:
Baseline: 0 degrees in Z axis

Angle (degrees) in Z axis Accuracy (%) Drop from Baseline (%) Visualisation
0 98.11 0.00
15 91.50 6.61
30 59.50 38.61
45 30.75 67.37
60 28.44 69.67
90 24.45 73.66

The poor performance for higher rotations is very much expected as there are no transforms encoded in the current architecture. If i visualise the rotated pointcloud i can clearly see that the current geometry does not match with the classes properly (without rotation that is). As expected, the performance worsens as rotation increases.

Section 3.1.2 : Number of Points in the Pointcloud

Procedure:

  1. Baseline Evaluation: First, evaluate the model on the full test set with all 10,000 points per object to establish baseline accuracy.

  2. Point Subsampling: For each test point cloud, randomly subsample a specified number of points (e.g., 1000, 2000, 5000, 8000) from the original 10,000 points. The subsampling is performed by:

    • Using np.random.choice(10000, num_points, replace=False) to randomly select num_points indices without replacement
    • Extracting the corresponding points: sampled_data = test_data_full[:, ind, :]
    • For segmentation, also subsample the labels: sampled_label = test_label_full[:, ind]
    • For classification, labels remain unchanged since they are per-object

  3. Evaluation at Multiple Point Counts: Test the model with different numbers of points (1000, 2000, 5000, 8000). For each point count:

    • Subsample all test point clouds to that number of points
    • Pass the subsampled point clouds through the trained model
    • Compute predictions and compare with ground truth labels
    • Calculate accuracy and accuracy drop compared to baseline (10,000 points)

Accuracy Performance Table:
Baseline: 10000 Points

Number of Points Accuracy (%) Drop from Baseline (%) Visualization
1000 97.38 0.73
2000 97.69 0.42
5000 97.80 0.31
8000 98.01 0.10
10000 98.11 0.00

The number of points in the point cloud is not affecting drastically. This makes sense considering that as long as there's enough pointclouds to capture the geometry of the object, the number of pointclouds dont affect the accuracy (drastically). We can see some improvement with increase in number of points as it gives us clearer geometry.

Section 3.2: Segmentation Model Performance

Section 3.2.1 : Rotation of Pointclouds

Procedure:
Already explained in section 3.1.1.

Accuracy/Performance Table:

Angle (degrees) Accuracy (%) Drop from Baseline (%) Ground Truth Segmentation Predicted Segmentation
0 90.32 0.00
15 82.96 7.36
30 69.88 20.44
45 59.65 30.67
60 53.23 37.10
90 45.71 44.61

Reasoning: I believe the same reasoning for the classification model can be used here as well. Due to lack of transformations the model is not able to learn to segment efficiently.

Section 3.2.2: Number of points in the PointClouds

Procedure:
Already explained in section 3.1.2.

Accuracy/Performance Table:

Number of Points Accuracy (%) Drop from Baseline (%) Ground Truth Segmentation Predicted Segmentation
1000 89.77 0.55
2000 90.23 0.09
5000 90.34 -0.02
8000 90.32 0.00
10000 90.32 0.00 gt_1 segment_task1

Reasoning: The same reasoning used in section 3.1.2 is applicable here as well.