16-825 · Learning for 3D Vision

Assignment 5 · Point Cloud Processing

Q1 · PointNet classification

A shared pointwise MLP encodes the XYZ coordinates. Global max pooling gives one descriptor per object, and a classification head predicts chair, vase, or lamp. The model trained for 30 epochs with 2,048 randomly sampled points per object. Validation accuracy selected the checkpoint, and the official test used all 10,000 points per object.

Test accuracy: 95.28% (908 / 953 objects)

True classCorrectTest objectsAccuracy
Chair61461799.51%
Vase8910287.25%
Lamp20523487.61%

Random test examples

One object was sampled at random from each true class.

Rotating point cloud of test chair 111
Test object #111True chair · Predicted chair
Rotating point cloud of test vase 638
Test object #638True vase · Predicted vase
Rotating point cloud of test lamp 774
Test object #774True lamp · Predicted lamp

Misclassifications

Chair 406 with a shallow side profile and rounded back
#406 · Chair predicted as lampThe chair is unusually shallow from the side. From the front, the rounded top and solid rectangular section dominate the silhouette, with little separation between the seat and back. The weak seat-depth cue may contribute to the error.
Vase 619 with a flower stem extending above its body
#619 · Vase predicted as lampThe vase includes a thin stem and broad blossoms above a tall body. This vertical arrangement resembles a lamp with a stand and head and may explain the prediction.
Lamp 726 with a rounded base and short connector below the shade
#726 · Lamp predicted as vaseThe lamp has a large rounded base and a very short connector below the shade. The base resembles the body of a vase and dominates the lower half of the cloud.

Download all classification predictions

Q2 · PointNet part segmentation

The segmentation model concatenates each point's local feature with a global max-pooled descriptor and predicts one of six part labels per point. It trained for 30 epochs with 2,048 sampled points per object. The best validation checkpoint was evaluated on all 10,000 points of each of the 617 test chairs.

Test point accuracy: 89.44% (5,518,677 / 6,170,000 points)

Ground truth and predictions

Colors indicate the same part labels throughout. Objects #142 and #351 have the two lowest test accuracies.

Chair #142 · 46.72%

Ground-truth segmentation of chair 142
Ground truth
PointNet segmentation of chair 142
PointNet prediction

The model identifies much of the back and legs. It assigns the thin armrests and several nearby rails to the seat class, accounting for much of the visible error.

Chair #351 · 49.79%

Ground-truth segmentation of chair 351
Ground truth
PointNet segmentation of chair 351
PointNet prediction

The compact, curved shell has no clearly separated legs. The model predicts a broad band of support labels around the lower shell, where the ground-truth part labels continue down from the upper surfaces.

Chair #374 · 82.55%

Ground-truth segmentation of chair 374
Ground truth
PointNet segmentation of chair 374
PointNet prediction

The model captures the wheeled base and most of the backrest. Errors cluster around the seat and the lower armrest attachments, where several parts meet.

Chair #57 · 92.93%

Ground-truth segmentation of chair 57
Ground truth
PointNet segmentation of chair 57
PointNet prediction

The four legs, crossbars, seat, and backrest are largely intact. A few support labels spill into the seat edge around the joints.

Chair #338 · 97.06%

Ground-truth segmentation of chair 338
Ground truth
PointNet segmentation of chair 338
PointNet prediction

The model separates the seat, legs, and rounded backrest cleanly. The remaining visible errors are concentrated near the seam between the backrest and seat.

Download all per-object segmentation scores

Q3 · Robustness analysis

Both experiments use the trained PointNet checkpoints from Q1 and Q2. Classification accuracy is measured over all 953 test objects. Segmentation accuracy is measured over the evaluated points from all 617 test chairs. The original 10,000-point inputs reproduce the Q1 and Q2 baselines.

Experiment 1 · Rotation about the Y axis

Rotate every test cloud by 30°, 60°, or 90° about the Y axis through the origin. Each input keeps all 10,000 points. Each segmentation label stays attached to its corresponding point during rotation.

RotationClassificationChange (pp)SegmentationChange (pp)
0° (baseline)95.28%0.0089.44%0.00
30°85.20%−10.0775.13%−14.31
60°69.05%−26.2364.14%−25.30
90°65.27%−30.0155.70%−33.74

Accuracy decreased as the tested angle increased for both tasks. The classifier changed chair #84 from class 0 to class 1 after a 90° rotation. For chair #338, segmentation dropped from 97.1% to 62.2%, with errors across the backrest and seat. The models were trained in the dataset's original orientations, and the pointwise features depend directly on the XYZ coordinates. The results show sensitivity to orientation despite max pooling across points.

Three classification examples showing predictions before and after 90-degree rotation
Classification. Original input and prediction on the left, with the 90° result on the right.
Three chair segmentation examples with ground truth and predictions before and after a 90-degree rotation
Segmentation. Each input has its ground truth and prediction side by side. Colors denote the same part labels in every panel.

Experiment 2 · Point count

Use a fixed random permutation of point indices (seed 825) to select nested subsets of 4,096, 2,048, and 512 points without replacement. Keep the original orientation and score the segmentation predictions on the retained points.

Input pointsClassificationChange (pp)SegmentationChange (pp)
10,000 (baseline)95.28%0.0089.44%0.00
4,09695.07%−0.2189.41%−0.03
2,04895.17%−0.1089.38%−0.07
51294.86%−0.4288.57%−0.87

With 512 points, classification lost 0.42 percentage points and segmentation lost 0.87 points relative to the full-cloud baselines. Scoring the full-input segmentation predictions on the same 512 retained points gave 89.40%, a 0.83-point gap from the sparse-input prediction. The retained points still capture the overall shape in the examples. Chair #338 retains its main part boundaries, and the armrest errors on chair #142 are visible at both densities. These results describe one fixed random subsample at each size.

Three classification examples showing predictions from the full cloud and from 512 retained points
Classification. The three illustrated class predictions stay the same at 512 points.
Three chair segmentation examples with ground truth and predictions for the full cloud and 512 retained points
Segmentation. The sparse predictions and ground truth are drawn at the same retained points.

Download the accuracy table with counts

Q4 Bonus · Locality

Model

I implemented a PointNet++ variant with two levels of set abstraction. The first level uses farthest-point sampling to select 128 centers and groups 24 nearest input points around each center. The second selects 32 of those centers and groups 16 neighbors. A shared MLP encodes relative XYZ coordinates and any features from the previous level, then max pooling summarizes each neighborhood. A final global max-pooling layer gives the classifier an object descriptor.

For segmentation, the decoder interpolates features from the three nearest coarse centers with inverse squared distance weights. It combines the interpolated features with first-level features and the global descriptor, interpolates again to the original points, and predicts a part label at each point. The hierarchy follows the set-abstraction and feature-propagation design of PointNet++.

Training and test accuracy

Both models trained for 30 epochs using 2,048 randomly sampled points per object. The best checkpoint from that stage then received six more epochs of fine-tuning on full 10,000-point clouds at a learning rate of 0.0003. Checkpoint selection used the same held-out splits as Q1 and Q2. All reported test scores use the full 10,000 points per object.

ModelClassification accuracySegmentation point accuracy
PointNet (Q1 and Q2)95.28% (908 / 953)89.44% (5,518,677 / 6,170,000)
PointNet++ after 2,048-point training94.54% (901 / 953)88.18% (5,440,486 / 6,170,000)
PointNet++ after full-cloud fine-tuning98.01% (934 / 953)90.01% (5,553,625 / 6,170,000)

The selected model improves classification by 2.73 percentage points and segmentation by 0.57 points. The local model also received extra full-cloud training, so the comparison measures the combined effect of the architecture and training schedule.

Classification examples

The selected model corrected 34 of the baseline's errors and introduced 8 new errors. It correctly recognized the lamp in #726, which PointNet labeled as a vase. It predicted a lamp for vase #631, whose narrow stem and wide base resemble a lamp stand.

A corrected lamp, a misclassified vase, and a chair classified correctly by both models
One seeded sample from each outcome group. The images show the same input to both models and list both predictions.

Segmentation examples

On chair #142, the local model recovers parts of the armrests, raising accuracy from 46.72% to 57.10%. It labels more of the seat correctly on #374, which improves from 82.55% to 86.19%. Both models capture the main parts of #338, with accuracy changing from 97.06% to 96.47%.

Ground truth, PointNet predictions, and PointNet++ predictions for chairs 142, 374, and 338
The three chairs are the same ones shown in Q3. All three columns use the same 10,000 points and label colors.