Assignment 5: Point Cloud Classification and Segmentation

Contents

Training setup. All four models use AdamW, a train-only validation split, and the same seed. Checkpoints are selected by validation accuracy. Training uses 1,024 points per object. Dense evaluation uses all 10,000 original points without retraining; Q3 retains its controlled 1,024-point reference. Point counts are stated with each result.

1. Classification Model (30 points)

The PointNet classifier reaches 97.69% test accuracy over 953 objects using 10,000 points per object. The checkpoint is selected at epoch 141. A shared XYZ MLP (64–64–64–128–1024), global max pooling, and a 512–256–3 head predict chair, vase, or lamp. Transformation blocks are omitted.

True classTest objectsPredicted chairPredicted vasePredicted lampAccuracy
chair61761700100.00%
vase1020911189.22%
lamp2342922395.30%

The original 1,024-point accuracy is 97.48%; that input condition is the controlled reference for Q3.

Random test objects (2 per class, seeded)

#119#233#626
Object 119, true chairObject 233, true chairObject 626, true vase
True chair; predicted chair.
Confidence 100.00%.
True chair; predicted chair.
Confidence 100.00%.
True vase; predicted vase.
Confidence 99.99%.
#713#798#870
Object 713, true vaseObject 798, true lampObject 870, true lamp
True vase; predicted vase.
Confidence 99.97%.
True lamp; predicted lamp.
Confidence 100.00%.
True lamp; predicted lamp.
Confidence 56.40%.

Failure predictions

#650#726#758
Failure: true vase, predicted lampFailure: true lamp, predicted vaseFailure: true lamp, predicted chair
True vase; predicted lamp.
Confidence 65.97%.
True lamp; predicted vase.
Confidence 74.63%.
True lamp; predicted chair.
Confidence 61.12%.

There is an incorrect prediction of each output class: chair, vase, and lamp.

The shown chair prediction is a false positive: a lamp predicted as chair.

All 617 true chairs are classified correctly at 10,000 points, so a false-negative example on a true chair is unavailable at this input condition.

All failure GIFs above use unrotated, 10,000-point inputs.

Object 650 is a vase predicted as a lamp with 65.97% confidence. The dense turntable reveals a flared cup on a narrow neck above a tall square pedestal. That pedestal-plus-upper-volume arrangement resembles a lamp base and shade, providing a plausible geometric source of confusion. The additional points resolve its shape clearly but do not remove the class ambiguity for this checkpoint.

Object 726 is a lamp predicted as a vase with 74.63% confidence. Its rounded lower body and broad tapered shade form a compact stacked silhouette with little exposed stem. The dense points show the shade rim and the transition to the base, but the global descriptor still selects vase. This is a class error on the full original cloud, not a failure created by dropping points or rotating the input.

Object 758 is a lamp incorrectly predicted as a chair with 61.12% confidence. It has a rounded wall-mount-like disc, a curved connecting neck, and a larger framed body with several horizontal rims. Those frame and rim features may activate furniture-related cues in the pooled descriptor, despite the overall object being a lamp. This is a false-positive chair prediction; it does not mean that any ground-truth chair was misclassified.

True-chair failure under point removal

Object 543 is a genuine chair that this same PointNet checkpoint misclassifies when only 128 points are retained. Both inputs are unrotated. This additional example comes from the measured Q3 density experiment; it is not a failure at the 10,000-point baseline.

1024 points: correct chair128 points: incorrect lamp
True chair 543, baseline inputTrue chair 543, reduced-point failure
Confidence 99.43%Confidence 76.73%

At 128 points, classification accuracy over all 953 test objects is 95.91% (versus 97.48% at 1024 points). The reduced cloud is a nested subset of the reference cloud, with the true class unchanged.

Object 543 has a tall narrow back above a small seat and support frame. At 128 points, much of the geometric detail is missing and the remaining vertical silhouette is predicted as a lamp with 76.73% confidence, versus a correct chair prediction with 99.43% confidence at 1,024 points. A lamp-like stem and base is a plausible interpretation of the sparse outline, rather than a proven feature attribution. The weights, object identity, input orientation, and normalization are unchanged; only the number of retained points differs.

2. Segmentation Model (30 points)

PointNet segmentation reaches 90.17% test point accuracy over 617 chairs with 10,000 points each. Dataset part mIoU is 63.24% and instance mIoU is 81.50%. The best validation checkpoint is epoch 145.

The model joins a 64-channel per-point feature with a global 1024-channel max-pooled feature, then applies a 512–256–128–6 shared head.

Color key: Part 0 · Part 1 · Part 2 · Part 3 · Part 4 · Part 5.

#101 (95.28%)#119 (86.31%)#232 (72.04%)#426 (36.96%, bad)#605 (44.19%, bad)
Object 101: ground truthObject 119: ground truthObject 232: ground truthObject 426: ground truthObject 605: ground truth
Object 101: PointNet predictionObject 119: PointNet predictionObject 232: PointNet predictionObject 426: PointNet predictionObject 605: PointNet prediction

Top row: ground truth. Bottom row: prediction. The first three objects are seeded random examples; the last two are the lowest-accuracy full-point cases. All GIFs use 10,000 points, synchronized views, and the accuracy printed above each object. Click any cell for the full-size paired animation.

Object 101 reaches 95.28% full-point accuracy. Its large teal back, orange seat region, and purple lower frame stay separated through the orbit. Only 472 of 10,000 points are wrong, with 287 Part 4 to Part 5 and 180 Part 5 to Part 2 errors dominating. The residual disagreement is localized near adjoining structures while the main part layout remains consistent.

Object 119 reaches 86.31% accuracy. The back and seat are mostly stable, but the slender lower supports receive systematically different labels. Of 1,369 incorrect points, 768 change from orange Part 5 to purple Part 4 and 568 from Part 5 to teal Part 2. More input points make those thin structures easier to inspect; they do not by themselves correct the learned part ambiguity.

Object 232 reaches 72.04% accuracy. The spread-out curved structure contains several intersecting surfaces, and a substantial orange Part 5 region becomes teal Part 2. This accounts for 1,560 of 2,796 errors; another 549 points change from yellow Part 3 to orange Part 5. The orbit shows broad region swaps as well as boundary leakage, so the error is not merely a few isolated mislabeled samples.

Object 426 is the worst full-point prediction, at 36.96% accuracy. The teal back is mostly retained, but a broad orange Part 5 region across the lower box is assigned blue Part 0. That single swap accounts for 5,858 of 6,304 wrong points. The dense paired orbit shows a structural labeling error across a large connected surface, rather than an apparent gap caused by sparse visualization.

Object 605 is the second-worst full-point case, at 44.19% accuracy. Its broad purple Part 4 lower frame is fragmented into yellow Part 3 and teal Part 2 in the prediction. The largest confusions are 2,613 Part 4 to Part 3 points and 1,580 Part 4 to Part 2 points, out of 5,581 errors. The connected arm-and-frame arrangement makes several parts meet in a compact region, which global context and independent point features do not separate reliably here.

The original 1,024-point segmentation accuracy is 90.72%. The dense figures and scores above use the separate full-point evaluation.

Part IDDataset IoU
053.82%
16.75%
286.86%
366.32%
483.17%
582.50%

3. Robustness Analysis (20 points)

Procedure. Both experiments use the original frozen PointNet checkpoints and complete test-object splits. The controlled baseline is 1,024 points per object: 97.48% classification and 90.72% segmentation. The seed, full-cloud normalization, and weights stay fixed; there is no retraining or test-time augmentation.

Rotation

Fixed 1024 points; rotate around y; no re-normalization after rotation. Positive accuracy drops below indicate degradation relative to the 1,024-point baseline.

Rotation0°15°30°60°90°180°
Classification97.48%96.96%92.34%61.18%54.77%31.48%
Segmentation90.72%88.00%80.95%70.42%58.72%58.23%
Classification drop (pp)+0.00+0.52+5.14+36.31+42.71+66.00
Segmentation drop (pp)+0.00+2.72+9.77+20.30+32.00+32.49

Classification samples

ObjectBaseline: 1024 points, 0°Perturbed: 1024 points, 180°
#157Object 157: chair
chair
Object 157: vase
vase
#184Object 184: chair
chair
Object 184: lamp
lamp
#360Object 360: chair
chair
Object 360: lamp
lamp

Same object and camera in each pair. Cells show the predicted class or point accuracy. Segmentation GIFs show predicted parts. These animations retain the original controlled point counts and saved predictions.

Segmentation samples

ObjectBaseline: 1024 points, 0°Perturbed: 1024 points, 180°
#101Object 101: 96.48%
96.48%
Object 101: 88.48%
88.48%
#119Object 119: 85.84%
85.84%
Object 119: 79.49%
79.49%
#233Object 233: 95.80%
95.80%
Object 233: 40.53%
40.53%

Same object and camera in each pair. Cells show the predicted class or point accuracy. Segmentation GIFs show predicted parts. These animations retain the original controlled point counts and saved predictions.

The measured curves show that segmentation is already more sensitive at small angles: at 15 degrees its accuracy drops by 2.72 percentage points, versus 0.52 for classification. At 30 degrees the drops are 9.77 and 5.14 points. At 180 degrees, classification falls from 97.48% to 31.48% and segmentation from 90.72% to 58.23%. The clouds retain their point identities and labels, so this degradation exposes dependence on the training orientation rather than lost geometry. The especially large-angle cases should not be interpreted as ordinary unrotated test performance.

Number of points

Nested subsets; same checkpoint and normalization; segmentation labels sampled with points. Positive accuracy drops below indicate degradation relative to the 1,024-point baseline.

Points per object102451225612864
Classification97.48%97.59%97.59%95.91%93.28%
Segmentation90.72%90.23%87.14%82.48%76.07%
Classification drop (pp)+0.00-0.10-0.10+1.57+4.20
Segmentation drop (pp)+0.00+0.49+3.58+8.24+14.65

Classification samples

ObjectBaseline: 1024 points, 0°Perturbed: 64 points, 0°
#157Object 157: chair
chair
Object 157: chair
chair
#184Object 184: chair
chair
Object 184: chair
chair
#360Object 360: chair
chair
Object 360: chair
chair

Same object and camera in each pair. Cells show the predicted class or point accuracy. Segmentation GIFs show predicted parts. These animations retain the original controlled point counts and saved predictions.

Segmentation samples

ObjectBaseline: 1024 points, 0°Perturbed: 64 points, 0°
#101Object 101: 96.48%
96.48%
Object 101: 84.38%
84.38%
#119Object 119: 85.84%
85.84%
Object 119: 65.62%
65.62%
#233Object 233: 95.80%
95.80%
Object 233: 92.19%
92.19%

Same object and camera in each pair. Cells show the predicted class or point accuracy. Segmentation GIFs show predicted parts. These animations retain the original controlled point counts and saved predictions.

Classification remains at 97.59% for both 512 and 256 points, one additional correct object relative to the 1024-point baseline. That small increase is not evidence of a significant improvement from downsampling. At 64 points classification is still 93.28%, a 4.20-point drop, whereas segmentation falls to 76.07%, a 14.65-point drop; its dataset part mIoU also declines from 64.88% to 45.57%. The paired samples illustrate the loss of coverage of thin supports and part boundaries. Classifying a global shape tolerates missing detail better than assigning a part label to each retained point in this run.

4. Bonus Question: Locality (20 points)

Model. The compact DGCNN uses three EdgeConv blocks (64, 64, 128 channels), rebuilding an exact 20-nearest-neighbor graph in learned feature space at every block. An edge MLP processes the center feature and neighbor-minus-center difference, then max-pools over neighbors. Concatenated block features feed a 512-channel embedding. Classification combines global max and average pooling; segmentation joins per-point features with global context.

Each point therefore depends explicitly on its local neighbors. This variant omits the original transform network. Chunked neighbor search limits distance-matrix memory, while exact search remains quadratic in point count.

Full-test comparison: 10,000 points per object. Both architectures are evaluated on every classification and segmentation test object, using identical full-point inputs, normalization, orientation, and sampling seed. The PointNet scores are the main Q1 and Q2 results above. Each model uses its best validation-selected checkpoint, trained at 1,024 points; no model is retrained for dense evaluation.

TaskPointNetDGCNNGain (percentage points)Points per objectTest objects
Classification97.69%92.24%-5.4610000953
Segmentation90.17%83.73%-6.4410000617

Original 1,024-point reference results are retained below for comparison with the training input size and Q3 protocol.

Original 1,024-point referencePointNetDGCNN
Classification97.48%98.01%
Segmentation90.72%91.47%

DGCNN reduces full-test classification accuracy by 5.46 percentage points (97.69% PointNet; 92.24% DGCNN). DGCNN reduces full-test segmentation accuracy by 6.44 percentage points (90.17% PointNet; 83.73% DGCNN). Increasing the evaluation point count changes the neighborhood geometry seen by the fixed-k graph; dense inputs therefore need not improve either architecture uniformly.

Full-point classification by true classPointNet recallDGCNN recall
chair100.00%98.87%
vase89.22%34.31%
lamp95.30%100.00%

The largest class-recall decrease relative to PointNet occurs for vase: DGCNN correctly classifies 35 of 102 objects, and predicts lamp for 66. This full-test breakdown explains behavior that the selected comparison GIFs alone do not capture. A mismatch between training density and the physical scale of a fixed-k neighborhood is a possible explanation, rather than an established causal attribution.

Dense visual comparison. The following selected objects use identical 10,000-point inputs for both models. The full-test scores above use all test objects. The GIFs below show selected examples from those evaluations, with per-object scores.

Segmentation

#101#119#232#426#605
Object 101: ground truthObject 119: ground truthObject 232: ground truthObject 426: ground truthObject 605: ground truth
Object 101: PointNetObject 119: PointNetObject 232: PointNetObject 426: PointNetObject 605: PointNet
Object 101: DGCNNObject 119: DGCNNObject 232: DGCNNObject 426: DGCNNObject 605: DGCNN
PointNet: 95.28%
DGCNN: 92.55%
PointNet: 86.31%
DGCNN: 83.36%
PointNet: 72.04%
DGCNN: 48.15%
PointNet: 36.96%
DGCNN: 52.65%
PointNet: 44.19%
DGCNN: 79.32%

Rows: ground truth, PointNet, DGCNN, and per-object scores. Same full-point chairs as Q2. Camera, point count, and part colors are shared.

For the selected full-point chairs, DGCNN improves object 426 from 36.96% to 52.65% and object 605 from 44.19% to 79.32%. It performs worse on objects 101 (95.28% to 92.55%), 119 (86.31% to 83.36%), and 232 (72.04% to 48.15%). The three-way GIFs show these gains and regressions on identical input points. These are per-object scores; the aggregate table above uses all 617 test objects at 10,000 points each. Both models were trained at 1,024 points, and increasing density changes the physical extent of a fixed 20-neighbor graph.

Classification

#726: true lamp#758: true lamp#406: true chair#119: true chair#233: true chair
Object 726, shared input cloudObject 758, shared input cloudObject 406, shared input cloudObject 119, shared input cloudObject 233, shared input cloud
PointNet: vase
DGCNN: lamp
PointNet: chair
DGCNN: lamp
PointNet: chair
DGCNN: lamp
PointNet: chair
DGCNN: chair
PointNet: chair
DGCNN: chair

The point cloud is the shared input; the labels below each object compare the two models. All classification comparison GIFs use 10,000 original points.

On these full-point examples, DGCNN corrects PointNet errors on lamps 726 and 758, while it incorrectly calls chair 406 a lamp even though PointNet gets that chair right. The two remaining displayed chairs are correct for both models. These examples show both corrections and a regression. The aggregate comparison above is measured separately over the entire 953-object test set at the same 10,000-point condition.

This is a single-seed comparison of complete architectures. It includes differences in parameterization and regularization, so it does not isolate locality as the only causal factor.

Training and Evaluation Details

All models train on the course training split with a fixed 10% validation holdout; classification is stratified by class and segmentation holds out whole objects. Test results do not select checkpoints. Training uses scale, translation, and jitter augmentation, with no random rotations.

Clouds are centered and scaled to unit maximum radius before point sampling. Evaluation uses seed 825 and deterministic per-object permutations. Density subsets are nested and use aligned segmentation labels. Point accuracy counts correct retained points; dataset part mIoU averages class IoUs, omitting absent classes.

ModelEpochsTraining pointsBatch sizeLearning rateSelected epoch
pointnet_cls1501024160.001141
pointnet_seg1501024160.001145
dgcnn_cls1501024160.001109
dgcnn_seg1501024160.00195

Additional dense inference uses the existing checkpoints on cpu, without training updates. Small numerical differences can occur across CPU/GPU backends.

An independent CPU rerun reproduced all four original headline accuracies to two decimal percentage precision.

Training curves

pointnet_cls: training and validation trajectories.
pointnet_cls: training and validation trajectories.
pointnet_seg: training and validation trajectories.
pointnet_seg: training and validation trajectories.
dgcnn_cls: training and validation trajectories.
dgcnn_cls: training and validation trajectories.
dgcnn_seg: training and validation trajectories.
dgcnn_seg: training and validation trajectories.

References