Assignment 5: Point Cloud Classification and Segmentation

Author: Po-Ting Ko

Andrew ID: potingk

Implementation

Both models are vanilla PointNet without the transformation blocks.

Q1. Classification Model

Test accuracy: 98.01% (934 / 953)

Class Accuracy
Chair 100.00% (617 / 617)
Vase 94.12% (96 / 102)
Lamp 94.44% (221 / 234)

Random test samples

#312 #160 #629
cls 312 cls 160 cls 629
GT chair, pred chair GT chair, pred chair GT vase, pred vase
#640 #760 #864
cls 640 cls 760 cls 864
GT vase, pred vase GT lamp, pred lamp GT lamp, pred vase ✗

Failure cases

All 19 errors at 10,000 points are among vases and lamps: 12 lamps → vase, 6 vases → lamp, 1 lamp → chair. The model classifies every chair correctly in this setting. For the chair failure, I therefore use the same model with 1,000 input points (python eval_cls.py --num_points 1000 --i 595). At 1,000 points, #595 is the only misclassified chair.

Vase → lamp (#651) Vase → lamp (#673) Lamp → vase (#726)
cls 651 cls 673 cls 726
p(lamp) = 1.00 p(lamp) = 1.00 p(vase) = 0.99
Lamp → vase (#827) Lamp → chair (#944) Chair → lamp (#595, 1,000 points)
cls 827 cls 944 cls 595 at 1000 points
p(vase) = 1.00 p(chair) = 0.71 p(lamp) = 0.53, p(chair) = 0.46

Interpretation.

Since PointNet sees only one global max-pooled feature, it classifies mainly by coarse shape and proportions. Vases and lamps overlap in exactly these cues, and they are also the two smallest training classes (741 vases and 1,554 lamps vs. 4,489 chairs).

Q2. Segmentation Model

Test accuracy: 90.45% (over all 617 × 10,000 points). The median per-object accuracy is 93.9%.

Colors: cyan = back, red = seat, blue = legs / base, yellow = arms, magenta and white = rare parts (e.g. headrest). The part names are read from the renderings.

Object Ground truth Prediction Accuracy
#297 seg 297 gt seg 297 pred 99.68%
#0 seg 0 gt seg 0 pred 94.52%
#134 seg 134 gt seg 134 pred 84.98%
#500 seg 500 gt seg 500 pred 74.69%
#26 (bad) seg 26 gt seg 26 pred 48.53%
#255 (bad) seg 255 gt seg 255 pred 47.41%
#351 (bad) seg 351 gt seg 351 pred 45.39%

Interpretation.

Q3. Robustness Analysis

Both experiments use the best Q1 and Q2 checkpoints without retraining. The "0°" and "10,000 points" settings are the Q1 / Q2 results.

Experiment 1: Rotation

Procedure. Rotate every test object about the horizontal x-axis by θ ∈ {0°, 15°, 30°, 45°, 60°, 90°, 120°, 180°}. This tilts the object forward (90° lays it on its back, 180° turns it upside down). The rotation is applied to the input points only; the labels stay the same.

θ 0° 15° 30° 45° 60° 90° 120° 180°
Cls accuracy 98.01% 95.80% 86.46% 54.98% 38.41% 34.31% 67.37% 73.56%
Seg accuracy 90.45% 83.84% 72.10% 63.05% 54.39% 27.56% 26.05% 33.90%

Classification samples (rotation about x; predicted class below each).

Object 0° 30° 60° 90°
Chair #312 312 rot0 chair 312 rot30 chair 312 rot60 lamp ✗ 312 rot90 lamp ✗
Vase #629 629 rot0 vase 629 rot30 vase 629 rot60 vase 629 rot90 lamp ✗
Lamp #760 760 rot0 lamp 760 rot30 lamp 760 rot60 lamp 760 rot90 chair ✗

Segmentation samples (object #0; per-object accuracy below each).

0° 30° 60° 90°
Ground truth seg 0 rot0 gt seg 0 rot30 gt seg 0 rot60 gt seg 0 rot90 gt
Prediction seg 0 rot0 94.52% seg 0 rot30 82.30% seg 0 rot60 73.93% seg 0 rot90 31.21%

Interpretation.

Experiment 2: Number of input points

Procedure. Evaluate with N ∈ {10000, 5000, 2500, 1000, 500, 100, 50, 10} points per object. The same random subset of point indices (seed 0) is used for every object; segmentation accuracy is measured on the kept points.

N 10,000 5,000 2,500 1,000 500 100 50 10
Cls accuracy 98.01% 98.01% 97.80% 97.80% 97.38% 93.81% 83.53% 28.65%
Seg accuracy 90.45% 90.53% 90.60% 90.36% 89.37% 82.30% 78.60% 67.57%

Classification samples (predicted class below each; points are drawn larger at low N).

Object 10,000 1,000 100 50
Chair #312 312 10000 chair 312 1000 chair 312 100 chair 312 50 chair
Vase #629 629 10000 vase 629 1000 vase 629 100 lamp ✗ 629 50 lamp ✗
Lamp #760 760 10000 lamp 760 1000 lamp 760 100 lamp 760 50 lamp

Segmentation samples (prediction, per-object accuracy below each).

Object 10,000 1,000 100 50
#0 seg 0 10000 94.52% seg 0 1000 94.50% seg 0 100 86.00% seg 0 50 82.00%
#297 seg 297 10000 99.68% seg 297 1000 99.10% seg 297 100 97.00% seg 297 50 76.00%

Interpretation.

Q4. Bonus: Locality (DGCNN)

Model

I implemented DGCNN (Dynamic Graph CNN, Wang et al. 2019). PointNet processes every point independently and only combines them in the final max-pool, so it never sees local structure. DGCNN adds locality with EdgeConv:

xi′ = maxj ∈ kNN(i) hΘ( [ xj − xi, xi ] )

Training. Adam (lr 1e-3, cosine decay), 100 epochs. Classification uses random 1,024-point subsets (batch 32), segmentation random 2,048-point subsets (batch 16). Training ran on an AWS g5.xlarge (A10G 24GB): about 30 s per epoch for each task.

Testing on 10,000 points. The kNN neighbourhood depends on point density: with 10,000 points, the 20 nearest neighbours cover a much smaller region than during training. Feeding all 10,000 points directly drops classification accuracy from 98.7% (at 1,000 points) to 93.9%. So at test time I split each cloud into random disjoint subsets of the training size (1,024 or 2,048 points), predict each subset, and average the class scores (cls) or write each point's scores back (seg). Every one of the 10,000 points still gets a prediction, so the numbers below are directly comparable to Q1 and Q2.

Test accuracy

Task PointNet (Q1 / Q2) DGCNN Change
Classification 98.01% (19 errors) 98.64% (13 errors) +0.63
Segmentation 90.45% 91.89% +1.44
Class PointNet DGCNN
Chair 100.00% 100.00%
Vase 94.12% 93.14%
Lamp 94.44% 97.44%

Both computed with eval_cls.py / eval_seg.py --model dgcnn on all 10,000 points (seed 0).

Classification compared with Q1

DGCNN fixes 11 of PointNet's 19 errors and makes 5 new ones; 8 objects are wrong for both.

Object Q1 PointNet DGCNN
Vase #667 (plant) cls 667 lamp ✗ vase ✓
Lamp #726 cls 726 vase ✗ lamp ✓
Lamp #944 cls 944 chair ✗ lamp ✓
Vase #651 (orchid) cls 651 lamp ✗ lamp ✗
Lamp #827 cls 827 vase ✗ vase ✗
Vase #716 cls 716 vase ✓ lamp ✗
Vase #699 cls 699 vase ✓ lamp ✗

Segmentation compared with Q2

DGCNN is better on 428 of 617 chairs (mean per-object gain +1.4 points).

Object Ground truth Q2 PointNet DGCNN
#297 seg 297 gt seg 297 pointnet 99.68% seg 297 dgcnn 99.48%
#0 seg 0 gt seg 0 pointnet 94.52% seg 0 dgcnn 95.34%
#134 seg 134 gt seg 134 pointnet 84.98% seg 134 dgcnn 87.66%
#500 seg 500 gt seg 500 pointnet 74.69% seg 500 dgcnn 74.48%
#26 seg 26 gt seg 26 pointnet 48.53% seg 26 dgcnn 59.45%
#255 seg 255 gt seg 255 pointnet 47.41% seg 255 dgcnn 62.91%
#351 seg 351 gt seg 351 pointnet 45.39% seg 351 dgcnn 47.70%

Robustness compared with Q3

The same two experiments as in Q3, for both models. For DGCNN, inputs larger than the training size use the subset strategy above; smaller inputs are fed directly.

Accuracy vs rotation for PointNet and DGCNN

Accuracy vs number of points for PointNet and DGCNN

Setting Cls PointNet Cls DGCNN Seg PointNet Seg DGCNN
Original 98.01% 98.64% 90.45% 91.84%
Rotate 15° 95.80% 97.90% 83.84% 89.93%
Rotate 30° 86.46% 80.69% 72.10% 77.87%
Rotate 60° 38.41% 35.47% 54.39% 35.73%
Rotate 90° 34.31% 34.63% 27.56% 17.74%
500 points 97.38% 97.27% 89.37% 87.34%
100 points 93.81% 65.37% 82.30% 59.89%
50 points 83.53% 36.62% 78.60% 42.92%

The "Original" DGCNN numbers come from the sweep script and differ from the table above by ≤ 0.05 points because the test-time subsets are drawn at random.