16-825 Assignment 5

Q1. Classification Model

Test accuracy: 97.69% (931 / 953). Per class: chair 100%, vase 91.2%, lamp 94.4%.

true \ predictedchairvaselamp
chair61700
vase0939
lamp112221

Random test objects

test 312
#312, true chair, predicted chair
test 160
#160, true chair, predicted chair
test 629
#629, true vase, predicted vase
test 640
#640, true vase, predicted vase
test 760
#760, true lamp, predicted lamp
test 864
#864, true lamp, predicted chair (wrong)

Failure cases

No test chair was misclassified, so the chair column shows the only object predicted as chair and the least confident chair.

ChairVaseLamp
test 864
#864: true lamp, predicted chair
test 618
#618: true vase, predicted lamp
test 721
#721: true lamp, predicted vase
test 406
#406: true chair, predicted chair, but only 46% chair vs 45% lamp
test 651
#651: true vase, predicted lamp
test 726
#726: true lamp, predicted vase

Interpretation: Errors are mostly vase vs. lamp, which share shapes: cylinders (#618, #721), round bulbous bodies (#726), and a thin arching stem resembling a lamp arm (#651). Chairs, the majority class with a distinctive structure, are never missed. Wall lamp #864 has no typical lamp silhouette.

Q2. Segmentation Model

Test accuracy: 90.21% (median per-object 93.99%).

Part colors (names inferred from the labels): ■ back, ■ seat, ■ legs/base, ■ arms, ■ headrest, ■ other (e.g. footrest).

ObjectGround truthPrediction (accuracy)
#191gt 191pred 19197.13%
#389gt 389pred 38995.55%
#581gt 581pred 58187.82%
#256gt 256pred 25686.32%
#26 (bad)gt 26pred 2648.53%
#605 (bad)gt 605pred 60543.11%

Interpretation: Typical chairs are segmented almost perfectly, with errors at part boundaries. Failures are unusual shapes or ambiguous labels: the sofa with ottoman (#26), solid side panels labeled as legs but predicted as arms (#605), and a stool top labeled as back (#581).

Q3. Robustness Analysis

Q1/Q2 models, no retraining.

Experiment 1: rotation

Procedure: each test object is tilted about the x axis by a fixed angle before evaluation.

RotationClassification acc.Segmentation acc.
0° (Q1/Q2)97.69%90.21%
15°94.86%83.28%
30°86.46%69.34%
45°58.45%60.12%
60°36.94%52.26%
90°32.32%24.89%
180°75.13%35.08%
30°60°90°
Classification, chair #312 (correct at 0°) rot 30 cls
predicted chair
rot 60 cls
predicted lamp
rot 90 cls
predicted vase
Segmentation, chair #191: prediction (97.13% at 0°) rot 30 seg
69.92%
rot 60 seg
48.54%
rot 90 seg
27.89%
Ground truth rot 30 gt rot 60 gt rot 90 gt

Interpretation: Without transformation blocks or rotation augmentation, PointNet is not rotation invariant and degrades beyond ~15°. Segmentation relies heavily on height, so tilted parts are relabeled. Upside-down (180°) classification partly recovers since an inverted chair still looks chair-like.

Experiment 2: point density

Procedure: evaluate on n randomly sampled points per object (--num_points).

Points per objectClassification acc.Segmentation acc.
10000 (Q1/Q2)97.69%90.21%
500097.69%90.35%
100097.38%90.38%
50096.64%89.54%
10093.18%81.97%
5080.38%77.73%
1027.07%51.75%
1000 points100 points50 points
Classification, chair #312 1000 pts cls
predicted chair
100 pts cls
predicted chair
50 pts cls
predicted lamp
Segmentation, chair #191: prediction 1000 pts seg
97.20%
100 pts seg
85.00%
50 pts seg
84.00%
Ground truth 1000 pts gt 100 pts gt 50 pts gt

Interpretation: Max-pooling makes PointNet robust to sparsity; accuracy is nearly unchanged down to 500 points and drops only when whole parts lose their points (<100).

Q4. Locality: PointNet++

PointNet++ (single-scale grouping): set-abstraction layers sample centers by farthest point sampling, group neighbors within a radius, and apply a shared MLP + max-pool to local coordinates, building features hierarchically. For segmentation, feature propagation layers upsample to every input point (3-NN inverse-distance weights) and merge the matching encoder features.

TaskPointNet (Q1/Q2)PointNet++
Classification97.69%98.01%
Segmentation90.21%91.18%

Classification on the Q1 failure cases

test 864
#864, true lamp
PointNet: chair
PointNet++: lamp
test 618
#618, true vase
PointNet: lamp
PointNet++: vase
test 651
#651, true vase
PointNet: lamp
PointNet++: lamp
test 721
#721, true lamp
PointNet: vase
PointNet++: lamp
test 726
#726, true lamp
PointNet: vase
PointNet++: lamp
test 629
#629, true vase
PointNet: vase
PointNet++: vase

Segmentation compared with Q2

ObjectGround truthPointNet (Q2)PointNet++
#191gt 191pn 191
97.13%
pn2 191
98.43%
#581gt 581pn 581
87.82%
pn2 581
95.16%
#389gt 389pn 389
95.55%
pn2 389
88.86%
#26gt 26pn 26
48.53%
pn2 26
59.68%
#605gt 605pn 605
43.11%
pn2 605
88.58%

Robustness of PointNet++ (same experiments as Q3)

Rotation0°15°30°45°60°90°180°
Cls, PointNet97.794.986.558.436.932.375.1
Cls, PointNet++97.996.790.264.842.639.032.5
Seg, PointNet90.283.369.360.152.324.935.1
Seg, PointNet++91.289.574.960.533.620.037.2

Points10000500010005001005010
Cls, PointNet97.797.797.496.693.280.427.1
Cls, PointNet++98.198.297.891.024.624.624.6
Seg, PointNet90.290.490.489.582.077.751.8
Seg, PointNet++91.391.489.780.864.761.253.2