Test accuracy: 0.9769 (931 / 953 objects, 10000 points per object)
Class
Accuracy
#objects
→ chair
→ vase
→ lamp
chair
0.9968
617
615
0
2
vase
0.8627
102
0
88
14
lamp
0.9744
234
1
5
228
Random test predictions
i249 · gt chair · pred chairi331 · gt chair · pred chairi493 · gt chair · pred chairi569 · gt chair · pred chairi693 · gt vase · pred vasei875 · gt lamp · pred lamp
Failure cases (one per class)
i406 · gt chair · pred lampi618 · gt vase · pred lampi777 · gt lamp · pred vase
19 of the 22 errors are vase ↔ lamp; chair is near-perfect (0.997).
One max-pooled 1024-d vector keeps only extremal points, so similar silhouettes merge:
i618 is a closed cylindrical vase that reads as a drum shade; i777 a chandelier whose thin
arms vanish in the pool, leaving a vase-like stem; i406 a solid slab armchair with no thin legs
and no open space underneath.
Vase is the smallest class (102 objects), hence its lowest recall (0.863).
Q2. Segmentation model
Test accuracy: 0.9059 (point-wise over 617 objects × 10000 points)
Segmentation class
0
1
2
3
4
5
Accuracy
0.8996
0.2476
0.9280
0.7909
0.9075
0.9164
Share of test points
0.33%
0.13%
31.97%
9.13%
20.31%
38.14%
Colours:
0
1
2 back
3 armrest
4 legs
5 seat
Per-object accuracy
Object
i471
i562
i120
i480
i350
i0
i240
i60
i235
i351
Accuracy
0.996
0.995
0.973
0.968
0.960
0.943
0.941
0.783
0.444
0.440
Good predictions
i471 ground truthi471 pred · 0.996i120 ground truthi120 pred · 0.973i0 ground truthi0 pred · 0.943i60 ground truthi60 pred · 0.783
Bad predictions
i235 ground truthi235 pred · 0.444i351 ground truthi351 pred · 0.440
Bimodal: canonical chairs are nearly solved (i471 at 0.996), solid-bodied furniture collapses to ~0.44.
Both worst cases fail identically — the barrel chair i351 and the sofa i235 get their
entire lower half labelled legs.
Cause: each point is classified from its own coordinates plus one global vector. Enough to learn
"low points are legs", not enough to see there is no gap underneath.
Classes 0 and 1 are 0.46% of all points, so cross-entropy barely penalises missing them
(class 1 ends at 0.248).
Q3. Robustness analysis
Experiment 1 — rotating the input
Each test cloud is rotated about one axis at inference time only, weights unchanged.
y is the vertical axis of the dataset.
Task
Axis
0°
10°
20°
30°
45°
60°
90°
180°
cls
x
0.9769
0.9643
0.8678
0.6086
0.3494
0.2907
0.2644
0.6422
cls
y
0.9769
0.9769
0.9538
0.9035
0.7450
0.5698
0.5519
0.3295
cls
z
0.9769
0.9643
0.9014
0.7345
0.3200
0.2581
0.2487
0.5425
seg
x
0.9059
0.8789
0.8090
0.7359
0.6214
0.4173
0.2234
0.3767
seg
y
0.9059
0.8840
0.8308
0.7743
0.7303
0.6716
0.5589
0.6607
seg
z
0.9059
0.8698
0.7730
0.6744
0.5769
0.5168
0.4087
0.3312
cls i0 · 0° · chaircls i0 · z 30° · chaircls i0 · z 90° · vase ✗cls i500 · 0° · chaircls i500 · z 30° · chaircls i500 · z 90° · vase ✗cls i900 · 0° · lampcls i900 · z 30° · vase ✗cls i900 · z 90° · lamp
seg i0 ground truthi0 · 0° · 0.943i0 · z 30° · 0.795i0 · z 90° · 0.478seg i26 ground truthi26 · 0° · 0.460i26 · z 30° · 0.315i26 · z 90° · 0.333
No transformation block and no rotation augmentation → rotation is purely out of distribution.
10° is free; past 45° about x or z, accuracy drops to 0.25–0.35 — below the 0.647 of
always answering chair.
The vertical axis y is far more benign (0.552 at 90° vs 0.264 / 0.249): objects are
roughly symmetric about it and their azimuth already varies in training.
x and z tip the object over, breaking the "legs below" prior carried by the absolute coordinates.
The partial recovery at 180° comes from lamps (0.893), which still look like lamps upside down —
not from a collapse to the majority class.
Experiment 2 — number of input points
Same weights, evaluated with --num_points from 10 to 10000 (uniform random subsample).
The opposite picture: max-pooling makes classification almost density-independent — 500 points
already give 0.9717 and everything above 1000 is saturated.
The global descriptor only needs enough samples to hit the same extremal responses.
It collapses only below ~50 points, and degenerately: at N=10 every object is predicted lamp.
Segmentation degrades steadily instead (0.791 at 100 points) — it needs a label per point, and the
per-point features themselves become uninformative.
Q4. Bonus — locality with DGCNN
DGCNN / EdgeConv: each layer rebuilds a k-NN graph (k = 20) in the current feature space, forms
[xj − xi, xi], applies a shared MLP and max-pools over the
neighbours — 4 layers for cls, 3 plus a global branch for seg. k-NN needs an O(N²) distance matrix
(400 MB per sample at N = 10000, recomputed every layer), so DGCNN runs at 1024 points;
PointNet is evaluated at 1024 too for a matched comparison.
cls i875 · gt lamp · pred lampcls i631 · gt vase · pred lamp ✗
Robustness, DGCNN vs PointNet
Setting
Model
0°/full
30°
60°
90°
cls, rotation about y
PointNet
0.9769
0.9035
0.5698
0.5519
cls, rotation about y
DGCNN
0.9811
0.9370
0.6044
0.5257
seg, rotation about y
PointNet
0.9059
0.7743
0.6716
0.5589
seg, rotation about y
DGCNN
0.9119
0.8521
0.8026
0.7214
Points
Model
10
50
100
250
500
1024
cls
PointNet
0.3116
0.8300
0.9192
0.9591
0.9717
0.9738
cls
DGCNN
0.2455
0.2455
0.5866
0.9538
0.9696
0.9811
seg
PointNet
0.5916
0.7402
0.7913
0.8447
0.8815
0.8979
seg
DGCNN
0.3574
0.5386
0.7007
0.8314
0.8944
0.9119
At matched density DGCNN wins on both tasks (+0.73 cls, +1.40 seg); its 1024-point segmentation even beats
PointNet at 10000 points, with fewer parameters (627k vs 876k).
The gain sits on part boundaries — i60: 0.746 → 0.907.
More rotation-stable (seg at 60° about y: 0.803 vs 0.672), because the edge term rotates with the
object.
But more fragile when sparsified: degenerates to all-lamp at ≤50 points, where PointNet still
reaches 0.830.