index

16-825 Assignment 5¶

Q1. Classification Model

Test accuracy: 0.9664

Class	Prediction	Interpretation (for failure case)
chair	chair
vase	vase
lamp	lamp
chair	vase	failure could be due to the fact that the seat part is not clearly visible in the point cloud (indicating it might be a folded chair) unlike the other chair point clouds that have a distinct 3D structure- hence this was the only misclassified chair
vase	lamp	the shape looks like a lamp due to the tall lean structure of the point cloud which is not typical for a vase
lamp	vase	the point cloud looks like a vase with each of the lights in the vase looking like flowers - not a typical shape for a lamp

Q2. Segmentation Model

Test accuracy: 0.87165

(ran with num_points=1000)

Quality	Accuracy	Interpretation (for failure case)
good	0.9220	model does a good job indentifying different classes
good	0.9440	model does a good job indentifying different classes
good	0.9810	model does a good job indentifying different classes
bad	0.6300	The model struggles to correctly segment the backrest and arms (cyan/yellow regions), confusing them with adjacent parts- the yellow and cyan colours seem to be not clearly separated in the arms of the chair, the legs and base under the seat are also confusing to the model
bad	0.3810	This severe failure case shows the model cannot handle complex chair designs with unusual components (magenta headrest and white attachment) because these are underrepresented in the training data

Q3. Robustness Analysis

3.1 Rotating input points along x-axis for classification

Procedure: I rotated the input point clouds by varying degrees (30°, 45°, 60°, 90°, 120°) around the x-axis and evaluated how classification degraded with increasing rotation angles

Accuracy at 0° rotation was 0.9664

Class	Prediction	Visualization
chair	chair
chair	chair
vase	vase
lamp	lamp
lamp	lamp

i) Angle of Rotation: 30°

Test accuracy: 0.8531

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	chair	chair
vase	chair	vase
lamp	vase	lamp
lamp	lamp	lamp

ii) Angle of Rotation: 45°

Test accuracy: 0.6558

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	chair	chair
vase	lamp	vase
lamp	vase	lamp
lamp	lamp	lamp

iii) Angle of Rotation: 60°

Test accuracy: 0.4218

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	vase	chair
vase	lamp	vase
lamp	vase	lamp
lamp	vase	lamp

iv) Angle of Rotation: 90°

Test accuracy: 0.2739

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	vase	chair
vase	vase	vase
lamp	chair	lamp
lamp	chair	lamp

v) Angle of Rotation: 120°

Test accuracy: 0.6967

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	chair	chair
vase	lamp	vase
lamp	lamp	lamp
lamp	lamp	lamp

The classification model exhibits sensitivity to rotation, with accuracy dropping from 0° to 90°, demonstrating the model lacks rotation invariance. The consistent confusion between chairs and lamps and lamps with vases across all rotation angles reveals the model relies heavily on orientation-specific features rather than learning true rotation-invariant representations. This critical weakness indicates the model was trained on canonically-oriented objects and cannot generalize to rotated instances, requiring data augmentation.

3.2 Rotating input points along x-axis for Segmentation

Procedure: I rotated the input point clouds by varying degrees (30°, 45°, 60°, 90°, 120°) around the x-axis and evaluated how segmentation degraded with increasing rotation angles

Test accuracy with 0°: 0.8716

Accuracy	Ground Truth	Pred Visualization
0.9520
0.8990
0.9230
0.9810
0.9590

i) Angle of Rotation: 30°

Test accuracy: 0.7296

Accuracy	Ground Truth	Pred Visualization	Visualization from Q2
0.7160
0.7730
0.6620
0.8740
0.8270

ii) Angle of Rotation: 45°

Test accuracy: 0.6295

Accuracy	Ground Truth	Pred Visualization	Visualization from Q2
0.6160
0.6470
0.5990
0.5710
0.7440

iii) Angle of Rotation: 60°

Test accuracy: 0.4449

Accuracy	Ground Truth	Pred Visualization	Visualization from Q2
0.4190
0.3410
0.3560
0.3170
0.6230

iv) Angle of Rotation: 90°

Test accuracy: 0.2740

Accuracy	Ground Truth	Pred Visualization	Visualization from Q2
0.2000
0.2100
0.3130
0.2630
0.2470

v) Angle of Rotation: 120°

Test accuracy: 0.2650

Accuracy	Ground Truth	Pred Visualization	Visualization from Q2
0.1620
0.2320
0.2980
0.1230
0.2470

The segmentation model shows significant degradation under rotation, with point-wise accuracy dropping from 87.16% at 0° to 72.96% at 30°, 62.95% at 45°, and collapsing to 27.40% at 90° and 26.50% at 120°, revealing severe lack of rotational invariance. Even well-segmented samples at 0° (e.g., the first chair in each table with 95.2% accuracy) experience dramatic accuracy drops demonstrating that rotation fundamentally disrupts the model's ability to recognize semantic part boundaries. This critical weakness indicates the model relies on orientation-specific geometric features rather than learning truly rotation-invariant representations.

3.3 Varying number of input points for classification

Procedure: I randomly subsampled varying numbers of points (from 100 to 750) from each test object's original 1000-point cloud and evaluated model performance for classification

Accuracy with 1000 points: 0.9664

Class	Prediction	Visualization
chair	chair
chair	chair
vase	vase
vase	vase
lamp	vase

i) num_points:100

Test accuracy: 0.9182

Class	Prediction	Prediction in Q1
chair	lamp	chair
chair	chair	chair
vase	vase	vase
vase	vase	vase
lamp	vase	vase

ii) num_points:500

Test accuracy: 0.9664

Class	Prediction	Prediction in Q1
chair	chair	chair
chair	chair	chair
vase	vase	vase
vase	vase	vase
lamp	vase	vase

iii) num_points:750

Test accuracy: 9633

Class	Prediction	Prediction in Q1
chair	chair	chair
chair	chair	chair
vase	vase	vase
vase	vase	vase
lamp	vase	vase

The classification model demonstrates strong robustness to point density variation with only a small drop in accuracy to 91.82% at 100 points. The failure case (lamp misclassified as vase at 100 points but corrected at higher densities, and chair misclassified as lamp at 100 points) reveals that sparse point clouds can cause confusion between geometrically similar objects, particularly cylindrical shapes like lamps and vases. Overall, the model requires a minimum of 100-200 points for reliable classification, with performance stabilizing at 500 points.

3.4 Varying number of input points for Segmentation

Procedure: I randomly subsampled varying numbers of points (from 100 to 750) from each test object's original 1000-point cloud and evaluated model performance for segmenation

Accuracy with 1000 points: 0.8716

Accuracy	Ground truth	Pred Visualization
0.7570
0.4600
0.5390
0.5690
0.7120

i) num_points:100

Test accuracy: 0.8344

Accuracy	Ground truth	Pred Visualization	Pred from Q2
0.6400
0.4800
0.600
0.7100
0.7550

ii) num_points:500

Test accuracy: 0.8653

Accuracy	Ground truth	Pred Visualization	Pred from Q2
0.7460
0.4600
0.5560
0.6020
0.6960

iii) num_points:750

Test accuracy: 0.8692

Accuracy	Ground truth	Pred Visualization	Pred from Q2
0.7920
0.4450
0.6030
0.5800
0.6890

The test accuracy decreases with decrease in points sampled, but segmentation model is robust to point cloud density to some extent. This is because with fewer points, we cannot reasonably capture the general shape of the object.

****16-825 Assignment 5****¶

16-825 Assignment 5¶