16-825 Assignment 5: Point Cloud Classification and Segmentation

1. Classification Model

Test accuracy: 97.38% (928 of 953 objects).

ClassCorrect / totalAccuracy
Chair616 / 61799.8%
Vase93 / 10291.2%
Lamp219 / 23493.6%

Random test point clouds

Random test samples
Ground truth and predicted class are in each title.

Failure cases

Failure cases

Interpretation.

2. Segmentation Model

Test accuracy: 90.84%

Visualizations

Each row shows ground truth (left) and prediction (right). Objects 133, 499 and 592 are random good cases; objects 351 and 26 are bad cases.

Test objectAccuracy
#13397.6%
#49998.0%
#59297.5%
#351 (bad)46.4%
#26 (bad)48.1%

#133 — 97.6%

133 ground truth
133 prediction

#499 — 98.0%

499 ground truth
499 prediction

#592 — 97.5%

592 ground truth
592 prediction

#351 — 46.4% (bad)

351 ground truth
351 prediction

#26 — 48.1% (bad)

26 ground truth
26 prediction

Interpretation.

3. Robustness Analysis

Experiment 1: Rotation

Rotate every test object around the x axis by 15° to 90° and run the models from Q1 and Q2.

Rotation0°15°30°45°60°90°
Classification accuracy97.4%95.3%85.2%53.1%35.3%29.1%
Segmentation accuracy90.8%84.5%72.3%59.4%41.0%26.6%

Classification samples

Classification with rotation

Segmentation samples

Segmentation with rotation
First row: object 133 (good case). Second row: object 351 (bad case).

Interpretation. Classification is fine up to 15° and fails quickly after 30°. Segmentation already loses at 15°. The predictions at 90° are mostly wrong for both objects. PointNet uses the raw xyz coordinates and has never seen rotated objects and rotated chair looks like a new shape.

Experiment 2: Number of points

Randomly keep 5000, 1000, 500 or 100 of the 10,000 points of each test object and run the same models. Each number is the average of 3 random draws.

Points1000050001000500100
Classification accuracy97.4%97.3%97.2%96.9%92.9%
Segmentation accuracy90.8%90.8%89.9%88.3%81.4%

Classification samples

Classification with fewer points

Segmentation samples

Segmentation with fewer points
First row: object 133 (good case). Second row: object 351 (bad case).

Interpretation. Both models are stable down to 500 points and only drop at 100 points. Max pooling over the points only needs enough points to cover the overall shape. At 100 points, thin parts such as legs and stems are often missed, and the shape is harder to recognize. Segmentation drops more than classification, probably because each point label depends on the part around that point, and with 100 points there are too few neighbors.

4. Bonus: Locality (DGCNN)

Model. DGCNN. Each EdgeConv layer finds the 20 nearest neighbors of every point and applies an MLP to the feature differences to those neighbors. The neighbors are recomputed in feature space at every layer. Classification uses 4 EdgeConv layers and pooling (max and mean) over points. Segmentation uses 3 EdgeConv layers and concatenates a global feature.

Training. Finding neighbors among 10,000 points is too expensive, so each batch uses 1024 random points per object. At test time, each object is randomly split into chunks of 1024 points. For classification, the logits of all chunks are averaged. For segmentation, each chunk is labeled separately, so every one of the 10,000 points gets a label.

ModelClassification accuracySegmentation accuracy
PointNet (Q1, Q2)97.38%90.84%
DGCNN98.64%91.20%

Classification

Classification comparison
GT = ground truth, PN = PointNet, DG = DGCNN. The last three are wrong for both.

Segmentation

Segmentation comparison
Robjects 351 and 26 (bad cases in Q2) and 133 (good case).

Interpretation.