16-825 Assignment 5

Andrew ID: rajathc

Q1. Classification Model (40 points)

test accuracy: 0.9811122770199371

Idx	GT Class	Predicted Class	Interpretation
199	Chair	Chair	Correct Prediction
620	Vase	Vase	Correct Prediction
802	Lamp	Lamp	Correct Prediction

All point clouds in chair class were correctely classfied as chairs

	687	Vase	Lamp	The flower emerging from the vase resembles a lamp’s light source, making the entire shape look lamp-like. This visual similarity likely caused the misclassification.
	922	Lamp	Vase	The lamp has four lights arranged symmetrically, forming a bowl- or cup-like outline. This shape strongly resembles the top of a vase, leading the model to label it incorrectly.

Q2. Segmentation Model (40 points)

test accuracy: 0.9029952998379255

Result Type	Idx	Accuracy	Interpretation
Good	562	0.9971	A standard-shaped chair with clearly separated components allows the model to distinguish each part more easily, leading to higher accuracy.
Good	57	0.9940	A standard-shaped chair with clearly separated components allows the model to distinguish each part more easily, leading to higher accuracy.
Good	471	0.9928	A standard-shaped chair with clearly separated components allows the model to distinguish each part more easily, leading to higher accuracy.
Bad	235	0.4008	The seat (red) and legs (blue) are connected in the point cloud, so the boundary between these two parts is unclear. Because the red and blue regions blend together, the model cannot reliably identify where the seat ends and the legs begin, leading to poor segmentation.
Bad	351	0.4972	This chair has a non-standard design with ambiguous part boundaries. The attached pillow is also incorrectly labeled, because the model fails to recognize that the pillow should be part of the seat (red). These ambiguities cause result in low segmentation accuracy.

Q3. Robustness Analysis (20 points)

Effect of roatation (X-axis)

Procedure

We tested how sensitive the model is to rotations by applying rotations around the X-axis at inference time. Specifically, we modified eval_cls.py and eval_seg.py to rotate the input point cloud using:

if args.rot_deg != 0:
    theta = torch.tensor([np.deg2rad(args.rot_deg)], dtype=torch.float32)
    R = euler_angles_to_matrix(torch.tensor([[theta, 0.0, 0.0]], dtype=torch.float32), "XYZ")[0]
    test_data = torch.einsum("ij,bpj->bpi", R, test_data)

Results

Rotation (X Axis)	Classification Accuracy	Segmentation Accuracy
0°	0.9811	0.9030
15°	0.9549	0.8345
30°	0.9014	0.7237
45°	0.6275	0.5749
60°	0.3536	0.3667
90°	0.2487	0.2707

Interpretation

The model shows high sensitivity to rotations not seen during training. As the rotation angle increases, accuracy drops sharply—especially beyond 30°, suggesting the model lacks rotation invariance. Segmentation suffers more than classification due to finer-grained prediction requirements.

Effect of varying number of points

Procedure

We evaluated the model’s robustness to input sparsity by reducing the number of points fed into the network. This was done by modifying the --num_points flag when running eval_cls.py and eval_seg.py. The same pretrained model (trained with 10,000 points) was tested on increasingly sparser versions of the input point cloud, down to just 50 points.

Results

Number of Points	Classification Accuracy	Segmentation Accuracy
10000	0.9811	0.9030
5000	0.9811	0.9030
1000	0.9780	0.8994
500	0.9759	0.8900
100	0.9402	0.8270
50	0.8520	0.7839

Interpretation

The model is surprisingly robust to point sparsity. Performance is nearly unchanged down to 1000 points, and only degrades slightly at 500 and below. This indicates the model has learned to extract meaningful global and local features from sparse point clouds, though segmentation accuracy drops more due to finer-grain dependence on point distribution. It suggests the network generalizes well even with partial object observations.