Assignment 5: Point Cloud Classification and Segmentation

16-825 Learning for 3D Vision

Name: Alicia Yang  |  Andrew ID: aliciay

contents

  1. setup
  2. Q1. classification
  3. Q2. segmentation
  4. Q3. robustness: rotation · number of points · noise
  5. Q4. bonus: PointNet++

setup

both pointnets are the basic version without the transform blocks. cls: shared mlp 3→64→64→64→128→1024 on every point, max pool, then fc 512→256→3. seg: glue each point's 64-d feature to the 1024-d global feature and run a per-point mlp down to 6 classes. adam, batch 32, cross entropy, trained on one L40S.

PointNet clsPointNet segPointNet++ clsPointNet++ seg
params0.81M0.88M1.46M1.40M
epochs150150100100
train points / objectall 10kall 10k20484096
lr1e-31e-31e-3, cosine1e-3, cosine
train time36 min45 min28 min30 min
best test acc (epoch)97.90% (125)89.95% (132)98.32% (68)91.64% (51)
last epoch test acc96.85%87.81%98.22%91.54%

best_model.pt is the epoch with the best test acc (that's how the starter picks it), so the numbers below are a little optimistic. the last-epoch row shows by how much.

Q1. classification

test accuracy: 97.90% (933 / 953)
classacccorrect
chair100.00%617 / 617
vase88.24%90 / 102
lamp96.58%226 / 234
gt \ predchairvaselamp
chair61700
vase09012
lamp26226
train loss and test acc per epoch
train loss and test acc per epoch

random test objects

6 random ones, colored by predicted class (chair vase lamp). all correct.

#523: chair, predicted chair (1.00)
#523: chair, predicted chair (1.00)
#393: chair, predicted chair (1.00)
#393: chair, predicted chair (1.00)
#648: vase, predicted vase (0.76)
#648: vase, predicted vase (0.76)
#644: vase, predicted vase (1.00)
#644: vase, predicted vase (1.00)
#722: lamp, predicted lamp (1.00)
#722: lamp, predicted lamp (1.00)
#736: lamp, predicted lamp (1.00)
#736: lamp, predicted lamp (1.00)

failures

most confident wrong prediction for each class. no chair was wrong, so i show the closest call, plus a real chair failure from the 15° tilt test in Q3 (no new runs, just the existing results).

chair: no failures. closest call #406, predicted chair at 0.89
chair: no failures. closest call #406, predicted chair at 0.89
vase: #651 predicted lamp (1.00)
vase: #651 predicted lamp (1.00)
lamp: #922 predicted vase (1.00)
lamp: #922 predicted vase (1.00)
chair tilted 15°: #543 predicted lamp
chair tilted 15°: #543 predicted lamp
interpretation
  • all 20 mistakes are vase/lamp mixups (12 vases called lamps, 6 lamps called vases) plus 2 lamps called chairs. zero chair mistakes.
  • vase #651 is a potted orchid: tall curvy stem with flowers at the top. honestly it looks like an arc floor lamp. pointnet only gets one max-pooled global feature, so it sees "thin stem + blob on top + base" and says lamp.
  • lamp #922 is a candelabra with thin arms coming off a base, which looks like branches in a vase.
  • chairs are 2/3 of the training data and super distinctive, so none fail. the closest call (#406, p = 0.89) is a barrel chair that looks kind of like a container. tilting the test set by just 15° makes 10 chairs fail, all as lamps. #543 is a really tall ladder-back chair, and tilted it's basically a pole on a small base, so lamp.
  • vase is the smallest class (741 train vs 1554 lamps and 4489 chairs), so anything ambiguous leans lamp. that's why vase has the lowest acc (88.2%).

Q2. segmentation

test accuracy: 89.95% of all test points (617 chairs × 10k points), mIoU 0.693

per chair: mean 90.0%, median 93.2%, worst 35.4%, 29 chairs under 70%. part names are my guess from where they sit on the chair.

part% of pointsaccIoU
0: rare part 00.3%90.3%0.899
1: rare part 1 (headrest?)0.1%13.4%0.097
2: back32.0%91.5%0.860
3: arms9.1%77.7%0.655
4: legs / base20.3%89.0%0.823
5: seat38.1%92.4%0.824
train loss and test acc per epoch
train loss and test acc per epoch

5 chairs: the 2 worst, then median, 75th percentile and best

#26 (worst): 35.4% of points right. left gt, right prediction
#26 (worst): 35.4% of points right. left gt, right prediction
#351 (2nd worst): 46.4% of points right. left gt, right prediction
#351 (2nd worst): 46.4% of points right. left gt, right prediction
#36 (median): 93.2% of points right. left gt, right prediction
#36 (median): 93.2% of points right. left gt, right prediction
#561 (75th percentile): 96.5% of points right. left gt, right prediction
#561 (75th percentile): 96.5% of points right. left gt, right prediction
#471 (best): 99.6% of points right. left gt, right prediction
#471 (best): 99.6% of points right. left gt, right prediction
interpretation
  • normal chairs are fine (93.2%, 96.5%, 99.6%). the errors are mostly on part boundaries like seat/legs and seat/back. each point only sees its own xyz plus the global feature, no neighbors, so it's pretty much labeling by height and position.
  • #26 (35.4%) is a blocky sofa chair with no legs. it calls the bottom of the base legs (low points are usually legs) and merges the arms into the seat.
  • #351 (46.4%) is a barrel chair where the sides wrap around and are labeled arms. it splits that shell by height into back and legs instead. the gt is kind of ambiguous there anyway.
  • the two rare parts (0.3% and 0.1% of points) are what pull the mIoU down.

Q3. robustness

for all of these i used the trained q1/q2 models and only messed with the test inputs, no retraining. full test sets, run with python robustness.py (same as eval_cls.py / eval_seg.py with --rot, --num_points, --noise). the first column of each table is the q1/q2 result. the PointNet++ rows are for q4. red = more than 10 points below the unperturbed number. in the gifs each column is one setting, cls is colored by predicted class, seg has gt on top and predictions below.

experiment 1: rotation

rotate every test cloud about x (y is up), 0 to 180°, so things tip over until they're upside down.

test acc vs rotation
test acc vs rotation
0°15°30°45°60°90°120°180°
cls PointNet97.9%94.8%68.6%39.3%30.6%33.8%61.4%78.8%
cls PointNet++98.3%94.4%54.4%36.9%37.9%40.5%30.7%36.4%
seg PointNet90.0%85.1%74.0%60.7%42.5%23.8%23.3%35.6%
seg PointNet++91.6%89.8%79.6%63.3%42.6%23.2%21.1%37.3%
per-class cls acc vs rotation
per-class cls acc vs rotation
chair #524, PointNet/PointNet++ predicted class: chair/chair at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, vase/lamp at rot 180°
chair #524, PointNet/PointNet++ predicted class: chair/chair at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, vase/lamp at rot 180°
vase #681, PointNet/PointNet++ predicted class: vase/vase at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, lamp/lamp at rot 180°
vase #681, PointNet/PointNet++ predicted class: vase/vase at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, lamp/lamp at rot 180°
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, lamp/lamp at rot 180°
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at rot 0°, lamp/lamp at rot 45°, lamp/lamp at rot 90°, lamp/lamp at rot 180°
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at rot 0°, 53.8%/55.9% at rot 45°, 17.4%/25.8% at rot 90°, 28.4%/25.3% at rot 180°
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at rot 0°, 53.8%/55.9% at rot 45°, 17.4%/25.8% at rot 90°, 28.4%/25.3% at rot 180°
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at rot 0°, 64.4%/74.3% at rot 45°, 20.5%/12.9% at rot 90°, 43.4%/33.9% at rot 180°
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at rot 0°, 64.4%/74.3% at rot 45°, 20.5%/12.9% at rot 90°, 43.4%/33.9% at rot 180°
interpretation
  • no t-net and only upright training data, so it drops fast. cls goes 97.9% → 94.8% at 15° → 39.3% at 45°. seg goes 90.0% → 85.1% → 60.7%.
  • tilted chairs mostly turn into lamps (chair acc 24% at 45°) while lamps stay at 89%. lamp is kind of the default for weird tilted shapes.
  • cls comes back a bit at 180° (78.8%) because an upside-down chair still has a seat, back and straight legs.
  • seg never comes back (23.8% at 90°, 35.6% at 180°) since the labels are tied to height. that's worse than calling every point seat (38%).

experiment 2: number of points

keep a random N of the 10k points per object (same indices for every object), N from 10000 down to 10.

test acc vs number of points
test acc vs number of points
10,0005,0002,0001,0005001005010
cls PointNet97.9%98.0%97.5%97.2%97.5%92.5%79.0%28.8%
cls PointNet++98.3%98.3%98.2%98.1%91.5%24.7%24.6%24.6%
seg PointNet90.0%89.9%89.7%89.1%87.5%78.8%73.9%64.5%
seg PointNet++91.6%91.5%91.5%89.7%83.5%31.5%21.6%19.9%
per-class cls acc vs number of points
per-class cls acc vs number of points
chair #524, PointNet/PointNet++ predicted class: chair/chair at 10000 pts, chair/chair at 1000 pts, chair/lamp at 100 pts, lamp/lamp at 10 pts
chair #524, PointNet/PointNet++ predicted class: chair/chair at 10000 pts, chair/chair at 1000 pts, chair/lamp at 100 pts, lamp/lamp at 10 pts
vase #681, PointNet/PointNet++ predicted class: vase/vase at 10000 pts, vase/vase at 1000 pts, lamp/lamp at 100 pts, lamp/lamp at 10 pts
vase #681, PointNet/PointNet++ predicted class: vase/vase at 10000 pts, vase/vase at 1000 pts, lamp/lamp at 100 pts, lamp/lamp at 10 pts
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at 10000 pts, lamp/lamp at 1000 pts, lamp/lamp at 100 pts, lamp/lamp at 10 pts
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at 10000 pts, lamp/lamp at 1000 pts, lamp/lamp at 100 pts, lamp/lamp at 10 pts
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at 10000 pts, 97.4%/95.6% at 1000 pts, 86.0%/29.0% at 100 pts, 70.0%/40.0% at 10 pts
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at 10000 pts, 97.4%/95.6% at 1000 pts, 86.0%/29.0% at 100 pts, 70.0%/40.0% at 10 pts
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at 10000 pts, 81.1%/83.8% at 1000 pts, 72.0%/37.0% at 100 pts, 50.0%/0.0% at 10 pts
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at 10000 pts, 81.1%/83.8% at 1000 pts, 72.0%/37.0% at 100 pts, 50.0%/0.0% at 10 pts
interpretation
  • barely changes until it gets really sparse: cls 97.5% at 500 points and 92.5% at 100, seg 87.5% and 78.8%. max pooling only needs the points near the extremes of each feature (the "critical points") and a random subset still has most of them.
  • vases go first (51% at 100 points, 16% at 50) since thin rims and stems disappear. at 10 points almost everything is lamp (28.8%).
  • seg holds up better (64.5% even at 10 points) because each label mostly depends on the point's own height.

experiment 3 (extra): noise

add gaussian noise with std σ to every coordinate. objects have radius 1, so σ = 0.1 is 10% of the object.

test acc vs noise std
test acc vs noise std
00.010.020.050.1
cls PointNet97.9%97.6%97.0%92.2%90.0%
cls PointNet++98.3%98.1%97.1%66.3%10.8%
seg PointNet90.0%89.7%88.8%82.6%69.0%
seg PointNet++91.6%91.1%89.4%81.4%73.8%
chair #524, PointNet/PointNet++ predicted class: chair/chair at noise 0, chair/chair at noise 0.02, chair/chair at noise 0.05, chair/vase at noise 0.1
chair #524, PointNet/PointNet++ predicted class: chair/chair at noise 0, chair/chair at noise 0.02, chair/chair at noise 0.05, chair/vase at noise 0.1
vase #681, PointNet/PointNet++ predicted class: vase/vase at noise 0, vase/vase at noise 0.02, vase/vase at noise 0.05, lamp/vase at noise 0.1
vase #681, PointNet/PointNet++ predicted class: vase/vase at noise 0, vase/vase at noise 0.02, vase/vase at noise 0.05, lamp/vase at noise 0.1
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at noise 0, lamp/lamp at noise 0.02, lamp/lamp at noise 0.05, lamp/vase at noise 0.1
lamp #838, PointNet/PointNet++ predicted class: lamp/lamp at noise 0, lamp/lamp at noise 0.02, lamp/lamp at noise 0.05, lamp/vase at noise 0.1
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at noise 0, 95.3%/91.3% at noise 0.02, 88.6%/85.2% at noise 0.05, 61.4%/82.0% at noise 0.1
chair #393, PointNet/PointNet++ accuracy: 96.2%/95.1% at noise 0, 95.3%/91.3% at noise 0.02, 88.6%/85.2% at noise 0.05, 61.4%/82.0% at noise 0.1
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at noise 0, 91.8%/89.3% at noise 0.02, 85.4%/81.9% at noise 0.05, 75.9%/79.8% at noise 0.1
chair #523, PointNet/PointNet++ accuracy: 84.6%/87.2% at noise 0, 91.8%/89.3% at noise 0.02, 85.4%/81.9% at noise 0.05, 75.9%/79.8% at noise 0.1
interpretation

pretty robust. cls only goes 97.9% → 90.0% at σ = 0.1, lamps get hit hardest (73%) since their thin parts blur. seg drops more (69.0%) because noise pushes points across the height-based part boundaries.

Q4. bonus: PointNet++

i implemented single-scale PointNet++ in plain pytorch (pointnet2.py). each set abstraction layer picks centers with farthest point sampling, grabs up to k neighbors within radius r of each center (ball query), runs a mini pointnet on each neighborhood (coordinates relative to the center) and max pools it. so features describe local neighborhoods instead of single points, which is the locality pointnet is missing.

taskmodelparamstest accdetails
clsPointNet (q1)0.81M97.90%chair 100.0%, vase 88.2%, lamp 96.6%
clsPointNet++1.46M98.32%chair 99.8%, vase 90.2%, lamp 97.9%
segPointNet (q2)0.88M89.95%mIoU 0.693
segPointNet++1.40M91.64%mIoU 0.672
cls test acc per epoch
cls test acc per epoch
seg test acc per epoch
seg test acc per epoch

cls, same objects as q1

left PointNet, right PointNet++, colored by predicted class.

#523 (chair): PointNet says chair, PointNet++ says chair
#523 (chair): PointNet says chair, PointNet++ says chair
#393 (chair): PointNet says chair, PointNet++ says chair
#393 (chair): PointNet says chair, PointNet++ says chair
#648 (vase): PointNet says vase, PointNet++ says vase
#648 (vase): PointNet says vase, PointNet++ says vase
#644 (vase): PointNet says vase, PointNet++ says vase
#644 (vase): PointNet says vase, PointNet++ says vase
#722 (lamp): PointNet says lamp, PointNet++ says lamp
#722 (lamp): PointNet says lamp, PointNet++ says lamp
#736 (lamp): PointNet says lamp, PointNet++ says lamp
#736 (lamp): PointNet says lamp, PointNet++ says lamp
#651 (vase): PointNet says lamp, PointNet++ says vase
#651 (vase): PointNet says lamp, PointNet++ says vase
#922 (lamp): PointNet says vase, PointNet++ says lamp
#922 (lamp): PointNet says vase, PointNet++ says lamp

seg, same chairs as q2

#26: gt | PointNet 35.4% | PointNet++ 64.7%
#26: gt | PointNet 35.4% | PointNet++ 64.7%
#351: gt | PointNet 46.4% | PointNet++ 52.4%
#351: gt | PointNet 46.4% | PointNet++ 52.4%
#36: gt | PointNet 93.2% | PointNet++ 94.0%
#36: gt | PointNet 93.2% | PointNet++ 94.0%
#561: gt | PointNet 96.5% | PointNet++ 98.4%
#561: gt | PointNet 96.5% | PointNet++ 98.4%
#471: gt | PointNet 99.6% | PointNet++ 99.7%
#471: gt | PointNet 99.6% | PointNet++ 99.7%

robustness

see the PointNet++ rows in the q3 tables and plots.

interpretation
  • locality helps a little for cls (97.90% → 98.32%; 12 objects only PointNet++ gets right vs 8 only PointNet) and more for seg (89.95% → 91.64%, better on 470 of 617 chairs), mostly at part boundaries. the worst q2 chair (#26) goes 35.4% → 64.7%, since neighborhood features can tell the base is one solid block and not legs. it also fixes both q1 failures (#651 and #922).
  • mIoU actually drops a bit (0.693 → 0.672). the rare parts are smaller than a ball neighborhood so they get swallowed by the part around them.
  • it's way less robust to sparse or noisy input. at 100 points cls is 24.7% (everything becomes lamp) vs 92.5% for PointNet, and seg is 31.5% vs 78.8%. at noise 0.1 cls is 10.8% (everything becomes vase) vs 90.0%. the balls have a fixed radius of 0.2, so sparse clouds leave them almost empty, and noise of 0.1 is half the radius which wrecks the local shape. PointNet's one global max pool doesn't care as much. the paper fixes this by randomly dropping input points during training, i didn't do that here.
  • rotation: neither model is invariant. PointNet++ is a bit better at small angles for seg (79.6% vs 74.0% at 30°) but worse for cls at big angles (36.4% vs 78.8% at 180°).