16-825 · Learning for 3D Vision

Point Cloud Processing

Kashish Gandhi · kgandhi2 · Assignment 5

Experiment setup

Models were trained for 250 epochs on a Tesla T4 using Adam with learning rate 0.001. Each object contains 10,000 XYZ points. Classification uses 6,784 training and 953 test objects across chair, vase, and lamp categories. Segmentation uses 4,489 training and 617 test chairs with six point labels. Batch sizes were 16 for classification and 8 for segmentation.

Following the starter training loop, the best checkpoint is selected using test accuracy each epoch. Results are from one run per model. Classification accuracy counts objects and segmentation accuracy counts points. All plots display Y vertically while inference uses the original coordinates.

Q1. Classification

A shared MLP maps each point through 3 → 64 → 128 → 256 channels with ReLU. Global max pooling yields an object feature, followed by a 256 → 128 → 3 classifier with dropout 0.3. Cross-entropy consumes raw logits.

Test accuracy 97.06% · 925 / 953 objects

Random test examples

Test #418. Ground truth and prediction both chair.
Test #418. Ground truth and prediction both chair.
Test #84. Ground truth and prediction both chair.
Test #84. Ground truth and prediction both chair.
Test #623. Ground truth and prediction both vase.
Test #623. Ground truth and prediction both vase.
Test #736. Ground truth and prediction both lamp.
Test #736. Ground truth and prediction both lamp.

One failure per class

Test #406. Chair → lamp. The tightly spaced surfaces and compact frame give this chair an unusual silhouette. The model may have relied on its broad curved surfaces more than its supporting structure.
Test #406. Chair → lamp. The tightly spaced surfaces and compact frame give this chair an unusual silhouette. The model may have relied on its broad curved surfaces more than its supporting structure.
Test #619. Vase → lamp. The vase, thin flower stem, and broad flower head resemble a base, support, and shade. This is a plausible geometric source of confusion.
Test #619. Vase → lamp. The vase, thin flower stem, and broad flower head resemble a base, support, and shade. This is a plausible geometric source of confusion.
Test #750. Lamp → vase. The broad hollow shade dominates the cloud, while its supporting frame is thin. Its container-like shape may explain the vase prediction.
Test #750. Lamp → vase. The broad hollow shade dominates the cloud, while its supporting frame is thin. Its container-like shape may explain the vase prediction.

Q2. Chair part segmentation

A shared 3 → 64 → 64 MLP produces point features. A 64 → 128 → 256 branch and global max pooling produce chair context. Concatenating this context with each 64-dimensional point feature feeds a 320 → 128 → 64 → 6 segmentation head. The model predicts six logits per point.

Test point accuracy 89.75%

The two lowest-scoring chairs and examples at the 25th, 50th, and 75th percentiles illustrate the range of predictions. Colors identify consistent label IDs across ground truth and prediction. Part descriptions below refer to visible geometry.

Chair #238 · 40.16%. The model merges the green back panel into the purple class and confuses the cyan seat and pink side region. The reclined geometry produces errors across large surfaces.
Chair #238 · 40.16%. The model merges the green back panel into the purple class and confuses the cyan seat and pink side region. The reclined geometry produces errors across large surfaces.
Chair #142 · 46.90%. The purple back and much of the cyan seat are assigned other labels, while several frame sections also change class. The thin intersecting frame is difficult for this model.
Chair #142 · 46.90%. The purple back and much of the cyan seat are assigned other labels, while several frame sections also change class. The thin intersecting frame is difficult for this model.
Chair #399 · 86.82%. The major seat, back, armrest, and leg regions are recovered. Errors are visible near connections between the thin supports and the seat.
Chair #399 · 86.82%. The major seat, back, armrest, and leg regions are recovered. Errors are visible near connections between the thin supports and the seat.
Chair #257 · 92.68%. Most large surfaces are labeled consistently. A thin yellow support next to the seat is partly assigned the cyan seat label.
Chair #257 · 92.68%. Most large surfaces are labeled consistently. A thin yellow support next to the seat is partly assigned the cyan seat label.
Chair #301 · 96.41%. The major regions agree closely with ground truth. Small errors remain around connections between the seat and frame.
Chair #301 · 96.41%. The major regions agree closely with ground truth. Small errors remain around connections between the seat and frame.

Q3. Robustness

Both experiments evaluate the original Q1 and Q2 checkpoints without retraining. The baseline uses all 10,000 points.

Experiment 1 — Rotation

Rotate every test cloud about the dataset Y axis by 30°, 60°, or 90°, keeping all points and their semantic labels. The same angle is applied to every object.

RotationClassificationSegmentation
0°97.06%89.75%
30°86.04%71.78%
60°58.55%58.13%
90°63.59%52.55%

Both models are sensitive to rotation. Classification falls by 38.51 percentage points at 60°, while segmentation falls by 37.20 points at 90°. Point-order invariance from max pooling does not imply rotation invariance, since the shared MLP consumes XYZ coordinates directly. Classification partially recovers at 90°, so its response is not monotonic across these angles.

Chair #0 changes from chair to lamp after 60° rotation.
Chair #0 changes from chair to lamp after 60° rotation.
Chair #9 changes from chair to vase after 60° rotation.
Chair #9 changes from chair to vase after 60° rotation.
Chair #35 drops from 94.87% to 10.96%. Rotation changes labels across the seat and back.
Chair #35 drops from 94.87% to 10.96%. Rotation changes labels across the seat and back.
Chair #389 drops from 91.70% to 27.25%. Large surfaces are split across incorrect classes after rotation.
Chair #389 drops from 91.70% to 27.25%. Large surfaces are split across incorrect classes after rotation.

Experiment 2 — Point subsampling

A single permutation of the 10,000 point indices is generated with NumPy default_rng(42). Its first 2,048, 1,024, and 256 indices form nested subsets shared across objects. Segmentation applies the same indices to ground-truth labels and scores only retained points. These measurements use one sampling seed.

PointsClassificationSegmentation
10,00097.06%89.75%
2,04896.85%89.48%
1,02496.64%88.75%
25695.80%84.64%

At 256 points, classification falls by 1.26 percentage points and segmentation by 5.11 points. Sparse samples often retain enough global shape to identify a category, while part labeling is more affected by the changed global feature and reduced geometric coverage. The displayed segmentation examples were selected for large drops and are more severe than the average.

Vase #618 changes from vase to lamp with 256 points.
Vase #618 changes from vase to lamp with 256 points.
Vase #697 changes from vase to lamp with 256 points.
Vase #697 changes from vase to lamp with 256 points.
Chair #426 drops from 90.33% to 44.92%. Many retained seat points receive incorrect labels.
Chair #426 drops from 90.33% to 44.92%. Many retained seat points receive incorrect labels.
Chair #354 drops from 86.34% to 48.05%. Many side and seat points are predicted yellow in the sparse cloud.
Chair #354 drops from 86.34% to 48.05%. Many side and seat points are predicted yellow in the sparse cloud.

Q4. Locality with voxel pooling

Each object is partitioned into a 12 × 12 × 12 cubic grid. The grid starts at the coordinate-wise minimum and its side length is the largest bounding-box extent. A shared 3 → 32 → 32 MLP encodes points. Cell-wise max pooling is gathered back to each point and concatenated with XYZ. The resulting 35-dimensional input feeds the original classification or segmentation network. This supplies local cell context alongside global PointNet pooling.

TaskPointNetWith voxel poolingChange
Classification97.06%97.69%+0.63 pp
Segmentation89.75%90.01%+0.26 pp

Both models use all 10,000 points and the same training schedule as their baselines. The aggregate improvements are modest in this single-run comparison. Local pooling corrects some difficult examples and introduces some new errors. Changes in parameter count and training randomness also contribute to this comparison.

Chair #406. Voxel pooling corrects the baseline lamp prediction to chair.
Chair #406. Voxel pooling corrects the baseline lamp prediction to chair.
Vase #650. The local model correctly identifies the vase despite its angular body and rounded extension.
Vase #650. The local model correctly identifies the vase despite its angular body and rounded extension.
Vase #618. The local model introduces a lamp error on an object classified correctly by the baseline.
Vase #618. The local model introduces a lamp error on an object classified correctly by the baseline.
Chair #142. Accuracy rises from 46.90% to 76.84%. Most of the purple back is recovered and frame labeling improves, with cyan errors remaining near intersections.
Chair #142. Accuracy rises from 46.90% to 76.84%. Most of the purple back is recovered and frame labeling improves, with cyan errors remaining near intersections.
Chair #238. Accuracy rises from 40.16% to 45.61%, but large errors remain. Both models miss the green back panel, while the local model changes several other seat and frame labels.
Chair #238. Accuracy rises from 40.16% to 45.61%, but large errors remain. Both models miss the green back panel, while the local model changes several other seat and frame labels.
Chair #399. Accuracy rises from 86.82% to 89.06%. The local prediction reduces some label spill near the supports and seat.
Chair #399. Accuracy rises from 86.82% to 89.06%. The local prediction reduces some label spill near the supports and seat.