Assignment 5: Point Cloud Processing

Parth Burujwale Andrew ID: pburujwa
16-825: Learning for 3D Vision

1. Point Cloud Classification

Test Accuracy

The saved best baseline classifier achieved 98.1112% test accuracy: 935 correct predictions out of 953 objects. There were 18 errors: two chairs, eight vases, and eight lamps.

Random Test Examples

The following six test objects were selected with a fixed random seed. Each figure includes the ground-truth and predicted categories.

Test object 90 Test object 156 Test object 183
Classification example 90 Classification example 156 Classification example 183
Test object 359 Test object 486 Test object 900
Classification example 359 Classification example 486 Classification example 900

Failure Cases

Ground truth Prediction Test object
Chair Lamp 406
Vase Lamp 650
Lamp Vase 750
Chair misclassified as lamp Vase misclassified as lamp Lamp misclassified as vase
Chair failure Vase failure Lamp failure

Chair 406: The displayed object has a broad, low surface and thin supports, without a prominent conventional backrest in this view. Its unusual shape may make the category harder to distinguish from other supported structures.

Vase 650: Its irregular, multipart silhouette differs from a simple upright vessel. Some rounded and elongated components could resemble features found in decorative lamps.

Lamp 750: The displayed object has a container-like cylindrical body, making vase versus lamp confusion plausible when distinctive lamp components are less apparent.

These are visual interpretations of the failures, rather than verified explanations of the model’s internal decisions. The baseline compresses the object into a global maximum-pooled descriptor, which may lose some relationships between nearby parts.

2. Point Cloud Segmentation

Test Accuracy

The best baseline segmentation checkpoint achieved approximately 89.50% overall point accuracy.

Ground Truth and Predictions

Objects 426 and 10 were selected as poor baseline examples. Objects 102, 120, and 233 were selected randomly from the remaining test objects using seed 825. Ground truth and predictions use the same part-color convention within these GIFs.

Test object Ground truth Baseline prediction
426 — poor example Object 426 ground truth Object 426 baseline prediction
10 — poor example Object 10 ground truth Object 10 baseline prediction
102 — random example Object 102 ground truth Object 102 baseline prediction
120 — random example Object 120 ground truth Object 120 baseline prediction
233 — random example Object 233 ground truth Object 233 baseline prediction

The common comparison evaluator measured the following baseline accuracies on these objects:

Test object Baseline point accuracy
426 24.97%
10 45.51%
102 98.15%
120 95.59%
233 95.10%

Discussion

The three random examples have point accuracies above 95%, while the two selected poor examples expose substantial failures. This contrast shows that aggregate accuracy does not fully describe performance on individual shapes.

The baseline combines each point’s features with a global object descriptor, but it does not explicitly encode local neighborhoods. Local part boundaries and relationships between nearby surfaces are therefore learned indirectly. This motivates the neighborhood-based model in Q4. Overall point accuracy also gives more weight to parts containing more points; it is not a per-part IoU metric.

3. Robustness Analysis

Two experiments tested both trained baseline models without retraining: rotations around the Y axis and reductions in input point count. Evaluation used the saved best checkpoints in evaluation mode. Classification reports object accuracy; segmentation reports point accuracy.

Experiment 1: Rotation Around the Y Axis

Every test point cloud was rotated by 0°, 15°, 30°, 60°, or 90° around the Y axis, retaining all 10,000 points. Class labels and the segmentation labels attached to individual points were unchanged.

For angle θ, the rotation was:

\[ R_y(\theta)= \begin{bmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{bmatrix}. \]

Y-axis rotation Classification accuracy Classification drop Segmentation accuracy Segmentation drop
0° 98.1112% 0.00 pp 89.4990% 0.00 pp
15° 96.4323% 1.68 pp 85.1741% 4.32 pp
30° 83.5257% 14.59 pp 76.4612% 13.04 pp
60° 45.0157% 53.10 pp 64.3310% 25.17 pp
90° 50.4722% 47.64 pp 57.1058% 32.39 pp

Here and below, pp denotes percentage points relative to the unmodified baseline.

Classification Examples

Original orientation 90° rotation
Classification at zero rotation Classification at 90 degrees

Segmentation Examples

Original orientation 90° rotation
Segmentation at zero rotation Segmentation at 90 degrees

Interpretation

Both models are sensitive to orientation. Classification drops sharply beyond 15°, reaching 45.02% at 60°. Its recovery to 50.47% at 90° shows that degradation is not strictly monotonic with angle. Segmentation decreases across the tested angles, reaching 57.11% at 90°.

Permutation invariance does not imply rotation invariance: rotating an object changes its XYZ coordinates and the learned responses to those coordinates. The results suggest reliance on orientation patterns present in the training data. Rotation augmentation or explicit rotation-aware design would be possible follow-up experiments; they were not evaluated here.

Experiment 2: Reduced Input Point Count

Each model was evaluated using 10,000, 5,000, 1,000, 500, or 100 points per object. A fixed random permutation with seed 825 defined nested subsets for the reduced point counts within each task. The same point indices were used across objects in that task, and segmentation targets were subsampled with their corresponding points.

The 10,000-point setting used the original complete input. No retraining or additional coordinate normalization was applied.

Points per object Classification accuracy Classification drop Segmentation accuracy Segmentation drop
10,000 98.1112% 0.00 pp 89.4990% 0.00 pp
5,000 98.1112% 0.00 pp 89.5336% −0.03 pp
1,000 97.6915% 0.42 pp 89.0702% 0.43 pp
500 97.4816% 0.63 pp 87.5361% 1.96 pp
100 93.8090% 4.30 pp 70.8088% 18.69 pp

Classification Examples

10,000 input points 100 input points
Classification with full inputs Classification with 100 points

Segmentation Examples

10,000 input points 100 input points
Segmentation with full inputs Segmentation with 100 points

Interpretation

Classification is relatively robust to reduced point counts: it retains 97.48% accuracy with 500 points and 93.81% with 100 points. Coarse object shape remains informative even when many points are removed.

Segmentation is more affected at extreme sparsity. Its accuracy falls by 18.69 percentage points at 100 points, consistent with needing sufficient geometric coverage to distinguish parts. At reduced point counts, segmentation accuracy is measured on the retained points, rather than on all original points.

The slight increase at 5,000 points is only about 0.03 percentage points. Since each reduced-count setting uses one deterministic subset, this does not establish that subsampling improves segmentation generally.

4. Bonus: Learning Locality

Hierarchical Architecture

The bonus model is PointNet++ inspired, using farthest-point sampling and k-nearest-neighbor grouping rather than reproducing the original paper’s radius-based grouping architecture exactly.

Stage Operation Output feature width
Input 10,000 XYZ points 3
Local stage 1 Sample 512 centers; group 32 neighbors; shared MLP 6 → 64 → 64 → 128; max-pool neighbors 128 per center
Local stage 2 Sample 128 centers; group 32 neighbors; shared MLP 131 → 128 → 128 → 256; max-pool neighbors 256 per center
Global stage Concatenate center coordinates; shared MLP 259 → 256 → 512; max-pool centers 512 per object

Each local group includes coordinates relative to its center and the neighbors’ input features. The first stage uses absolute XYZ coordinates as its initial features, alongside the relative coordinates. The second stage groups the learned first-stage features.

Classification: The 512-dimensional global feature passes through a 512 → 256 → 128 → 3 classifier. Hidden layers use batch normalization, ReLU, and dropout with probability 0.3.

Segmentation: The global descriptor is concatenated with coarse local features. Three-neighbor interpolation propagates features from 128 centers to 512 centers, then to all input points. Interpolation weights are proportional to inverse squared Euclidean distance, with a small numerical floor. Skip connections fuse finer features at each stage. A pointwise head predicts six part logits per original point.

Training Setup

Both locality models were trained from scratch for 250 epochs with cross-entropy loss and Adam at learning rate 0.001. Classification used batch size 32; segmentation used batch size 4. All 10,000 input points were provided to the models before internal sampling.

Classification training took 49 minutes 25 seconds and segmentation training took 2 hours 3 minutes 29 seconds on an AWS g5.xlarge instance with an NVIDIA A10G. The environment used Python 3.10, PyTorch 2.7.1 with CUDA 12.8, and PyTorch3D 0.7.9.

Quantitative Comparison

Task Baseline accuracy Locality accuracy Gain
Classification 98.1112% 98.3211% +0.21 pp
Segmentation 89.4991% 91.7682% +2.27 pp

The classification model correctly labels 937 of 953 objects, two more than the baseline. The segmentation gain is larger in percentage-point terms.

Classification Comparisons

These are the same three baseline failures shown in Q1. Each comparison labels the ground truth and the prediction from each model.

Classification comparison on chair 406
Classification comparison on vase 650
Classification comparison on lamp 750

Segmentation Comparisons

The same five Q2 objects are shown with ground truth, baseline prediction, and locality prediction. Within each comparison, all panels use identical displayed points and viewpoints. Gray, magenta, cyan, yellow, blue, and red represent part IDs 0–5, respectively.

Test object Baseline accuracy Locality accuracy Change
426 24.97% 61.78% +36.81 pp
10 45.51% 88.45% +42.94 pp
102 98.15% 98.03% −0.12 pp
120 95.59% 95.67% +0.08 pp
233 95.10% 91.48% −3.62 pp
Segmentation comparison on object 426
Segmentation comparison on object 10
Segmentation comparison on object 102
Segmentation comparison on object 120
Segmentation comparison on object 233

Discussion and Limitations

The locality model improves overall performance on both tasks in these runs. Explicit neighborhood aggregation provides information about nearby surfaces and their relative arrangement before global pooling or feature propagation.

The two difficult segmentation examples improve substantially: object 426 gains 36.81 percentage points and object 10 gains 42.94 points. However, locality does not improve every example. Object 233 loses 3.62 points, and object 426 remains challenging at 61.78% accuracy.

The observed gains cannot be attributed exclusively to locality: the architecture also changes feature widths, parameterization, and the segmentation decoder. These are single-run comparisons without repeated-seed uncertainty estimates. Following the provided training loop, best checkpoints were selected using the supplied test split, so these numbers are not an independent evaluation on a separate untouched holdout set.

References

  1. Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CVPR, 2017. Paper.
  2. Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NeurIPS, 2017. Paper.
  3. PyTorch3D: GPU operations for farthest-point sampling, neighborhood queries, and point-cloud rendering. Documentation.