Assignment 5 : Point Cloud Processing¶
Q1. Classification Model (40 points)¶
Model Performance Summary¶
| Metric | Value |
|---|---|
| Test Accuracy | $97.07\%$ |
1. Classification Summary¶
Per-class Accuracy¶
| Class | Correct / Total | Accuracy |
|---|---|---|
| chair | 616 / 617 | $0.9984$ |
| vase | 96 / 102 | $0.9412$ |
| lamp | 224 / 234 | $0.9573$ |
Analysis: Chair Class¶
Sample Review: Correct and Incorrect Classifications¶
Analysis: Lamp Class¶
Sample Review: Correct and Incorrect Classifications¶
Analysis: Vase Class¶
Sample Review: Correct and Incorrect Classifications¶
Improved Classification Interpretation¶
Here is the improved interpretation of the classification failure cases, focusing on clarity, grammar, and conciseness:
Analysis: Chair Failure¶
The model misclassified the chair likely because the object was folded or in a non-standard configuration. This obscured its typical structural features, preventing recognition as a normal chair shape and leading to the error.
Analysis: Lamp Failure¶
The misclassified lamp sample has a non-standard, cubical design. This obscure structure does not resemble the feature set of typical lamps in the training data, leading to the misclassification.
Analysis: Vase Failure¶
The misclassified vase exhibits a structure that closely resembles a pedestal lamp. Specifically, the model may have interpreted the slender body of the vase as the lamp's base and a small, leaf-like top feature (or rim) as the lamp's light source.
Q2. Segmentation Model (40 points)¶
Model Performance Summary¶
| Metric | Value |
|---|---|
| Overall Segmentation Accuracy | $90.29\%$ |
Objects Visualization (Good Performance)¶
Samples with High Accuracy¶
Objects Visualization (Bad Performance)¶
Samples with Low Accuracy¶
Interpretation¶
The model performs well on chairs with standard, clearly segregated components (seat, backrest, legs, etc.), similar to those shown in the top row of visuals.
However, the model fails on chairs whose structure lacks clear segregation between these different parts, as is evident in the failure examples in the second row.
Q3. Robustness Analysis (20 points)¶
1. Classification Rotation Analysis¶
1.1 X-Axis Rotation Robustness¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.9706$ |
| 30 | $0.7177$ |
| 45 | $0.3809$ |
| 60 | $0.2991$ |
| 80 | $0.2833$ |
1.2 Y-Axis Rotation Robustness¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.9748$ |
| 30 | $0.9297$ |
| 45 | $0.7838$ |
| 60 | $0.6149$ |
| 80 | $0.5719$ |
1.3 Z-Axis Rotation Robustness¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.9601$ |
| 30 | $0.6348$ |
| 45 | $0.3683$ |
| 60 | $0.3190$ |
| 80 | $0.2791$ |
2. Qualitative Visualization of Failure Modes (45° Rotation)¶
X Axis Rotation ($\theta_x = 45^{\circ}$)¶
Y Axis Rotation ($\theta_y = 45^{\circ}$)¶
Z Axis Rotation ($\theta_z = 45^{\circ}$)¶
Procedure: Generating Rotated Point Clouds¶
The process used to generate the rotated set of points involves standard 3D rotation mathematics
[Image of 3D rotation using rotation matrix] .
Given a set of 3D points $\mathbf{P}$ and a specified angle $\theta$ around a particular axis (X, Y, or Z), the Rotation Matrix ($\mathbf{R}$) is calculated. The rotated set of points $\mathbf{P}'$ is then found by multiplying the original points by the rotation matrix, which effectively applies the rotation around the origin:
$$\mathbf{P}' = \mathbf{R} \cdot \mathbf{P}$$
This exact process was used to generate rotated point sets for angles $\theta = 10^\circ, 30^\circ, 45^\circ, 60^\circ, \text{ and } 80^\circ$ across all three principal axes.
Interpretation: Rotational Robustness¶
The test results clearly demonstrate that the model's performance is significantly less robust to rotations around the X and Z axes (pitch and roll) compared to rotations around the Y-axis (yaw).
- General Trend: Across all three axes, accuracy consistently drops as the angle of rotation increases from $10^\circ$ to $80^\circ$.
- Axis Comparison: The slope of accuracy decrease is much steeper for the X and Z axes. This indicates that even slight rotations that tilt the object (X and Z axes) rapidly degrade the features the model relies on.
- Y-Axis Resilience: Rotation around the Y-axis (vertical axis), which preserves the object's general upright posture, causes a much slower and gentler drop in accuracy, confirming that the model is somewhat orientation-dependent, particularly regarding the object's vertical alignment.
2. Segmentation Rotation Robustness Test Results¶
Overall Performance Summary¶
The model's overall segmentation accuracy under rotation shows a decline, with the model being most robust to Y-axis rotation (upright spin) and least robust to X-axis rotation.
1. Rotational Accuracy Analysis¶
1.1 X-Axis Rotation Robustness (Segmentation)¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.8763$ |
| 30 | $0.7339$ |
| 45 | $0.6477$ |
| 60 | $0.5007$ |
| 80 | $0.2594$ |
1.2 Y-Axis Rotation Robustness (Segmentation)¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.8861$ |
| 30 | $0.8192$ |
| 45 | $0.7668$ |
| 60 | $0.7081$ |
| 80 | $0.6555$ |
1.3 Z-Axis Rotation Robustness (Segmentation)¶
| Rotation Angle (Degrees) | Test Accuracy |
|---|---|
| 10 | $0.8624$ |
| 30 | $0.6861$ |
| 45 | $0.5813$ |
| 60 | $0.5123$ |
| 80 | $0.4539$ |
2. Qualitative Visualization of Failure Modes ($\theta = 45^{\circ}$)¶
X Axis Rotation ($\theta_x = 45^{\circ}$)¶
Y Axis Rotation ($\theta_y = 45^{\circ}$)¶
Z Axis Rotation ($\theta_z = 45^{\circ}$)¶
Interpretation : Segmentation Rotational Robustness¶
The segmentation model's behavior under rotation mirrors the classification model's trend across all three axes, confirming a fundamental orientation dependence in the feature learning process.
- General Trend: Test accuracy consistently drops as the angle of rotation increases from $10^\circ$ to $80^\circ$.
- Axis Sensitivity: The model demonstrates significantly higher stability against Y-axis rotation (yaw or spinning upright) compared to X- and Z-axis rotations (pitch and roll).
- Conclusion: The slope of accuracy decrease for Y-axis rotation is much less steep than for the X and Z axes, indicating that the model relies heavily on the upright alignment of the input point cloud for accurate segmentation.
3. Classification Accuracy vs. Point Density¶
Quantitative Analysis¶
The results show a strong correlation between the number of input points and the final test accuracy, though the drop-off is relatively graceful until the point count is drastically reduced to 100.
| Number of Points (N) | Test Accuracy |
|---|---|
| 10,000 | $99.0\%$ |
| 5,000 | $98.3\%$ |
| 3,000 | $98.0\%$ |
| 1,000 | $97.9\%$ |
| 100 | $93.8\%$ |
Qualitative Visualization¶
The following samples demonstrate the model's ability to correctly classify shapes even when the point cloud is sparse, successfully identifying the overall geometry.
5,000 Points¶
3,000 Points¶
1,000 Points¶
100 Points¶
Improved Interpretation: Point Density vs. Classification Accuracy¶
The provided inference accurately captures the core phenomenon of the model's performance under reduced point density. I've refined the language for brevity, clarity, and stronger analytical impact.
Quantitative Insight¶
The classification accuracy exhibits a gradual decrease as the point count is reduced from 5,000 to 1,000, confirming a slight reliance on dense data. The most significant drop (approximately 5 percentage points) occurs when the density is severely reduced to 100 points.
Qualitative Insight¶
The model maintains a relatively high accuracy of $93.8\%$ even at 100 points. This suggests the network effectively learns the global geometry and topological structure of the shapes using minimal data. While the high accuracy could partially be attributed to a lower probability of local errors with fewer points to evaluate, the sustained performance indicates that the features critical for class distinction (e.g., a chair's legs vs. a vase's symmetry) are still preserved even in highly sparse point clouds.
4. Segmentation Accuracy vs. Point Density¶
Quantitative Analysis¶
The overall segmentation accuracy remains remarkably stable even when reducing the point count from 10,000 down to 1,000. A significant performance drop is only observed when the point count is severely reduced to 100.
| Number of Points (N) | Test Accuracy |
|---|---|
| 10,000 | $91.0\%$ |
| 5,000 | $90.34\%$ |
| 3,000 | $90.40\%$ |
| 1,000 | $90.03\%$ |
| 100 | $80.00\%$ |
Qualitative Visualization¶
The following samples show the model's segmentation output (Pred) compared to the ideal mask (GT) across different point densities.
5,000 Points¶
3,000 Points¶
1,000 Points¶
100 Points¶
Final Interpretation: Point Density vs. Segmentation Accuracy¶
The segmentation model exhibits a significantly sharper drop-off in performance under extreme sparsity compared to the classification model.
Key Observations:¶
- Steep Drop at 100 Points: Accuracy remains highly stable ($\approx 90-91\%$) from 10,000 down to 1,000 points. However, reducing the count further to 100 points causes a substantial drop of $\approx 11$ percentage points (from $91\%$ to $80\%$). This suggests 100 points is critically insufficient for maintaining the fine-grained local relationships needed for accurate segmentation.
- Performance Bimodality: At 100 points, the model displays bimodal behavior, achieving near $100\%$ accuracy on some test cases but failing completely ($\approx 10\%$ accuracy) on others. This indicates that while the sparse cloud sometimes captures all necessary segment boundaries, slight variations can cause catastrophic failure in feature grouping.
- Structural Requirement: Unlike classification, which only needs global features to recognize the object type, segmentation requires dense, localized points to accurately define object boundaries and distinct parts. The steep drop confirms that 100 points is structurally inadequate for the model to reliably understand and delineate the object's geometry and internal component segregation.