16-825 Assignment 5

Q1 — Classification Model

Test Accuracy

default values

test accuracy: 0.8992654774396642

Visualized Test Examples

Below I show a few random test point clouds (rendered as rotating point-cloud GIFs) together with the predicted class. At least one failure example per class is included. All GIFs are rendered using the evaluation script and saved under ./data/. As a key, red is chair, green is vase, blue is lamp. 0: [1.0, 0.0, 0.0], # chair 1: [0.0, 1.0, 0.0], # vase 2: [0.0, 0.0, 1.0], # lamp

Failure Cases

Q2 — Segmentation Model

Test Accuracy

ran with default flags

test accuracy: 0.6381837925445705

Per-object Segmentation Results

I visualize segmentation for three representative test chairs (indices 0-2). For each object I show the ground-truth part labels and the predicted labels.

Most errors concentrate near part boundaries (e.g., between seat and back or between legs and base). Since the model only uses global pooling and no explicit local neighborhoods, it sometimes smooths labels over these ambiguous regions.

Additional “Bad” Segmentations

The following visualizations highlight cases where segmentation is noticeably worse, either due to aggressive subsampling or large rotations (also reused in the robustness experiments).

Q3 — Robustness Analysis

I ran two main robustness experiments: (1) varying the number of input points per object, and (2) rotating the point clouds around the z-axis. All runs use the same trained checkpoints (model_epoch_0.pt for both tasks). More detailed results are outlined in writeup.md

Experiment 1 — Varying Number of Points

For both classification and segmentation, I subsampled each test point cloud from the original 10k points down to {4096, 2048, 1024} points using random sampling without replacement (via --num_points in eval_cls.py and eval_seg.py). The model weights are kept fixed; only the input density changes.

Classification

The classification accuracy is very stable as long as a reasonable number of points are kept. This matches the intuition that PointNet aggregates features over the whole set; global max-pooling is fairly robust to moderate subsampling.

Segmentation

For segmentation, performance degrades more noticeably as the number of points decreases. Thin structures like legs become sparsely sampled, so the model has fewer points to correctly label those regions. Qualitatively, the legs and back edges become noisier and more fragmented in the low-point GIFs (see gt_points*.gif and pred_points*.gif).

Experiment 2 — Rotations Around the z-axis

In the second experiment I fixed the number of points at 1024 and rotated each test point cloud around the z-axis by {45°, 90°, 180°, 270°}, then evaluated the trained models on the rotated sets. The scripts report both the (unrotated) baseline test accuracy and the accuracy on the rotated data (rotation test accuracy).

Classification (1024 points)

The classification model is not rotation-invariant: accuracy drops a lot for large rotations, especially at 90° and 270°. Since the model was trained on a fixed canonical orientation, the global features noticeably change under rotation, causing chairs to look more like other objects (vases or lamps) from unusual viewpoints.

Segmentation (1024 points)

Segmentation accuracy is even more sensitive to rotation, and 180° rotation is particularly destructive. Many points, especially on the seat/back interface, flip to incorrect labels. In the corresponding GIFs (gt_rot*.gif vs pred_rot*.gif) you can see entire parts miscolored when the chair is upside-down or sideways.

Qualitative Rotation Visuals

Overall, both tasks are robust to moderate point subsampling but quite sensitive to large unseen rotations. This matches the design of PointNet, which is permutation-invariant but not inherently rotation-invariant.