Assignment 5 Report

Overview

In this assignment, you will implement a PointNet based architecture for classification and segmentation with point clouds (you don't need to worry about the tranformation blocks).

I implemented the tranformation blocks regardless :)

Table of Contents

Q1. Classification Model (40 points)

Correct samples

Correct chair         Correct vase         Correct lamp

Failure cases

GT: chairs         GT: vases         GT: lamps

Wrong chair         Wrong vase         Wrong lamp

Without armrest, the silhouette looks like a lamp base.         Tall neck resembled a lamp stem.         The wide base makes it look like a vase.

Q2. Segmentation Model (40 points)

Sample Accuracy GT Pred Note
Good 1 0.9962 seg gt good1 seg pred good1 Nearly perfect segmentation.
Good 2 0.9677 seg gt good2 seg pred good2 Also nearly perfect segmentation.
Good 3 0.9129 seg gt good3 seg pred good3 Looks clean to me.
Bad 1 0.6279 seg gt bad1 seg pred bad1 Wrong labels on base.
Bad 2 0.7243 seg gt bad2 seg pred bad2 Confusion between arm rest and seat region.

Q3. Robustness Analysis (20 points)

Classification:

Two stress tests on the classifier: (1) aggressive downsampling with no rotation, and (2) heavy rotation with all points kept.

Downsample sweep (rot=0)

keep fraction accuracy sample render
1.0 0.981 rot0.0_down1.0
0.5 0.981 rot0.0_down0.5
0.1 0.971 rot0.0_down0.1
0.05 0.970 rot0.0_down0.05
0.01 0.924 rot0.0_down0.01
0.005 0.769 rot0.0_down0.005
0.001 0.317 rot0.0_down0.001

Observation: accuracy stays near baseline down to 5% of points, then collapses quickly as points be come so sparse that some details are lost.

Rotation sweep (keep=1.0)

rotation (deg) accuracy sample render
0 0.981 rot0.0_down1.0
30 0.768 rot30.0_down1.0
60 0.448 rot60.0_down1.0
90 0.216 rot90.0_down1.0
120 0.193 rot120.0_down1.0
150 0.197 rot150.0_down1.0
180 0.583 rot180.0_down1.0

Observation: the model is highly orientation sensitive, recovering only near 180 degree.

Full comparison:

Downsample vs accuracy     Rotation sweep

Observation: For a 3-way classification, uniform random guess expects a 33% accuracy, but our test set is imbalanced (class counts: [617, 102, 234], majority = 64.7%), so the real lower bound can be much lower than 33%, which is shown in the experiements.

Segmentation:

Downsample sweep (rot=0)

keep fraction accuracy GT Pred
1.0 0.915 seg gt rot0 keep1 seg pred rot0 keep1
0.1 0.908 seg gt rot0 keep0.1 seg pred rot0 keep0.1
0.01 0.809 seg gt rot0 keep0.01 seg pred rot0 keep0.01
0.001 0.496 seg gt rot0 keep0.001 seg pred rot0 keep0.001

The model tolerates aggresive downsampling until key details are lost.

Rotation sweep (keep=1.0)

rotation (deg) accuracy GT Pred
0 0.915 seg gt rot0 keep1 seg pred rot0 keep1
60 0.488 seg gt rot60 keep1 seg pred rot60 keep1
120 0.270 seg gt rot120 keep1 seg pred rot120 keep1
180 0.368 seg gt rot180 keep1 seg pred rot180 keep1

The segmentations are roughly structured by vertical layers. Which makes it extremely sensitive to rotations. It also explains the slight accuracy improvement when rotation=180 degrees.

Full comparison:

Seg downsample vs accuracy     Seg rotation sweep