Assignment 5

ModelTaskPointsEpochsBatchParamsBest test acc
PointNetcls100006032810,1150.9769
PointNetseg100006016876,0380.9059
DGCNNcls102480161,800,0670.9811
DGCNNseg10248016627,4620.9119

Q1. Classification model

Test accuracy: 0.9769 (931 / 953 objects, 10000 points per object)
ClassAccuracy#objects→ chair→ vase→ lamp
chair0.996861761502
vase0.862710208814
lamp0.974423415228

Random test predictions

chair
i249 · gt chair · pred chair
chair
i331 · gt chair · pred chair
chair
i493 · gt chair · pred chair
chair
i569 · gt chair · pred chair
vase
i693 · gt vase · pred vase
lamp
i875 · gt lamp · pred lamp

Failure cases (one per class)

chair as lamp
i406 · gt chair · pred lamp
vase as lamp
i618 · gt vase · pred lamp
lamp as vase
i777 · gt lamp · pred vase

Q2. Segmentation model

Test accuracy: 0.9059 (point-wise over 617 objects × 10000 points)
Segmentation class012345
Accuracy0.89960.24760.92800.79090.90750.9164
Share of test points0.33%0.13%31.97%9.13%20.31%38.14%

Colours: 0 1 2 back 3 armrest 4 legs 5 seat

Per-object accuracy

Objecti471i562i120i480i350i0i240i60i235i351
Accuracy0.9960.9950.9730.9680.9600.9430.9410.7830.4440.440

Good predictions

gt
i471 ground truth
pred
i471 pred · 0.996
gt
i120 ground truth
pred
i120 pred · 0.973
gt
i0 ground truth
pred
i0 pred · 0.943
gt
i60 ground truth
pred
i60 pred · 0.783

Bad predictions

gt
i235 ground truth
pred
i235 pred · 0.444
gt
i351 ground truth
pred
i351 pred · 0.440

Q3. Robustness analysis

Experiment 1 — rotating the input

Each test cloud is rotated about one axis at inference time only, weights unchanged. y is the vertical axis of the dataset.

TaskAxis0°10°20°30°45°60°90°180°
clsx0.97690.96430.86780.60860.34940.29070.26440.6422
clsy0.97690.97690.95380.90350.74500.56980.55190.3295
clsz0.97690.96430.90140.73450.32000.25810.24870.5425
segx0.90590.87890.80900.73590.62140.41730.22340.3767
segy0.90590.88400.83080.77430.73030.67160.55890.6607
segz0.90590.86980.77300.67440.57690.51680.40870.3312
0 deg
cls i0 · 0° · chair
30 deg
cls i0 · z 30° · chair
90 deg
cls i0 · z 90° · vase ✗
0 deg
cls i500 · 0° · chair
30 deg
cls i500 · z 30° · chair
90 deg
cls i500 · z 90° · vase ✗
0 deg
cls i900 · 0° · lamp
30 deg
cls i900 · z 30° · vase ✗
90 deg
cls i900 · z 90° · lamp
gt
seg i0 ground truth
0 deg
i0 · 0° · 0.943
30 deg
i0 · z 30° · 0.795
90 deg
i0 · z 90° · 0.478
gt
seg i26 ground truth
0 deg
i26 · 0° · 0.460
30 deg
i26 · z 30° · 0.315
90 deg
i26 · z 90° · 0.333

Experiment 2 — number of input points

Same weights, evaluated with --num_points from 10 to 10000 (uniform random subsample).

Points10205010025050010002500500010000
cls0.31160.52050.83000.91920.95910.97170.97590.97590.97590.9769
seg0.59160.65650.74020.79130.84470.88150.89960.90670.90670.9059
10000
cls i0 · N=10000 · chair
100
cls i0 · N=100 · chair
10
cls i0 · N=10 · chair
10000
cls i500 · N=10000 · chair
100
cls i500 · N=100 · chair
10
cls i500 · N=10 · lamp ✗
10000
cls i900 · N=10000 · lamp
100
cls i900 · N=100 · lamp
10
cls i900 · N=10 · lamp
gt
seg i0 ground truth
10000
i0 · N=10000 · 0.943
100
i0 · N=100 · 0.830
10
i0 · N=10 · 0.700
gt
seg i26 ground truth
10000
i26 · N=10000 · 0.460
100
i26 · N=100 · 0.350
10
i26 · N=10 · 0.000

Q4. Bonus — locality with DGCNN

DGCNN / EdgeConv: each layer rebuilds a k-NN graph (k = 20) in the current feature space, forms [xj − xi, xi], applies a shared MLP and max-pools over the neighbours — 4 layers for cls, 3 plus a global branch for seg. k-NN needs an O(N²) distance matrix (400 MB per sample at N = 10000, recomputed every layer), so DGCNN runs at 1024 points; PointNet is evaluated at 1024 too for a matched comparison.

ModelPointscls accuracyseg accuracy
PointNet (Q1 / Q2)100000.97690.9059
PointNet10240.97380.8979
DGCNN10240.98110.9119

Same object, both models (1024 points)

gt
i60 ground truth
pointnet
PointNet · 0.746
dgcnn
DGCNN · 0.907
gt
i0 ground truth
pointnet
PointNet · 0.938
dgcnn
DGCNN · 0.955
lamp
cls i875 · gt lamp · pred lamp
vase as lamp
cls i631 · gt vase · pred lamp ✗

Robustness, DGCNN vs PointNet

SettingModel0°/full30°60°90°
cls, rotation about yPointNet0.97690.90350.56980.5519
cls, rotation about yDGCNN0.98110.93700.60440.5257
seg, rotation about yPointNet0.90590.77430.67160.5589
seg, rotation about yDGCNN0.91190.85210.80260.7214
PointsModel10501002505001024
clsPointNet0.31160.83000.91920.95910.97170.9738
clsDGCNN0.24550.24550.58660.95380.96960.9811
segPointNet0.59160.74020.79130.84470.88150.8979
segDGCNN0.35740.53860.70070.83140.89440.9119