Assignment 2

Jianing Zhang (jianing7)

1. Exploring loss functions (10 points)

1.1 Fitting a voxel grid (4 points)

Optimized voxel grid
fit
Ground truth voxel grid
ground truth

1.2 Fitting a point cloud (3 points)

Optimized point cloud
fit
Ground truth point cloud
ground truth

1.3 Fitting a mesh (3 points)

Optimized mesh
fit
Ground truth mesh
ground truth

2. Reconstructing 3D from single view (80 points)

2.1 Image to voxel grid (20 points)

Input, test 0
input (test 0)
Prediction, test 0
prediction (F1 87.8)
Ground truth, test 0
ground truth
Input, test 150
input (test 150)
Prediction, test 150
prediction (F1 88.7)
Ground truth, test 150
ground truth
Input, test 300
input (test 300)
Prediction, test 300
prediction (F1 45.8)
Ground truth, test 300
ground truth

eval_vox

F1@0.05: 74.84

2.2 Image to point cloud (20 points)

Input, test 0
input (test 0)
Prediction, test 0
prediction (F1 84.4)
Ground truth, test 0
ground truth
Input, test 150
input (test 150)
Prediction, test 150
prediction (F1 55.6)
Ground truth, test 150
ground truth
Input, test 300
input (test 300)
Prediction, test 300
prediction (F1 71.4)
Ground truth, test 300
ground truth

eval_point

F1@0.05: 79.48

2.3 Image to mesh (20 points)

Input, test 0
input (test 0)
Prediction, test 0
prediction (F1 77.6)
Ground truth, test 0
ground truth
Input, test 150
input (test 150)
Prediction, test 150
prediction (F1 41.6)
Ground truth, test 150
ground truth
Input, test 300
input (test 300)
Prediction, test 300
prediction (F1 69.3)
Ground truth, test 300
ground truth

eval_mesh

F1@0.05: 72.89

Point cloud (79.48) > voxel (74.84) > mesh (72.89). The point decoder can place its points anywhere and F1 is measured on points, so it is the least constrained. The voxel grid is limited by its 323 resolution. The mesh is a deformed sphere, so it cannot open holes or represent thin, open parts well.

2.4 Analyse effects of hyperparams variations (10 points)

I varied w_smooth for the mesh decoder (20k steps each). Smoothness is the average Laplacian smoothness loss of the predicted meshes on test chairs (lower is smoother).

w_smooth0.1110
F1@0.0573.7172.2171.14
smoothness1.660.330.067

More smoothing gives lower F1: the Laplacian term pulls vertices toward their neighbors and rounds off thin legs and edges. The surfaces look cleaner, but they are less accurate.

Input, test 0
input (armchair)
Prediction with w_smooth 0.1, test 0
w_smooth 0.1: sharp edges, but jagged facets and folded spikes
Prediction with w_smooth 1, test 0
w_smooth 1: smoother faces, a few folds left
Prediction with w_smooth 10, test 0
w_smooth 10: round and blobby, corners and legs inflated
Ground truth, test 0
ground truth
Input, test 150
input (ladder-back)
Prediction with w_smooth 0.1, test 150
w_smooth 0.1
Prediction with w_smooth 1, test 150
w_smooth 1
Prediction with w_smooth 10, test 150
w_smooth 10
Ground truth, test 150
ground truth

Each row is the same chair predicted with w_smooth 0.1, 1 and 10. At 0.1 the boxy armchair keeps its flat sides and corners but the surface is torn into folded triangles; at 10 it turns into a rounded, inflated shape and the thin legs of the ladder-back become thick tubes.

2.5 Interpret your model (10 points)

I made two visualizations, one for what the voxel model is unsure about and one for how much the input view matters.

1. Where is the voxel model unsure?

Predicted occupancy probability of the voxel model (middle slice along each axis) next to the ground truth. The model is confident on large flat parts and unsure on thin parts and boundaries.

Voxel confidence slices

2. Does the prediction depend on the input view?

The same chair from all of its views, through the point model. The predicted shape barely changes with the viewpoint, so how hard the chair is matters more than the view.

View sensitivity, chair 0

View sensitivity, chair 150

3. Exploring other architectures / datasets (choose at least one, more than one is extra credit)

3.1 Implicit network (10 points)

The decoder takes the image feature and a 3D coordinate and predicts the occupancy there. I query it on a 32x32x32 meshgrid in (-1, 1)3: each coordinate is concatenated with the 512-d feature and passed through a small shared MLP (128 wide, three residual blocks). 22 of 678 test chairs, mostly thin ones, have no voxel above the threshold and are scored 0.

Input, test 0
input (test 0)
Prediction, test 0
prediction (F1 87.3)
Ground truth, test 0
ground truth
Input, test 150
input (test 150)
Prediction, test 150
prediction (F1 79.8)
Ground truth, test 150
ground truth
Input, test 300
input (test 300)
Prediction, test 300
prediction (F1 74.9)
Ground truth, test 300
ground truth

eval_implicit

F1@0.05: 72.62

Compared to the voxel decoder (74.84), it scores a little lower overall but handles thin shells better (test 300: 74.9 against 45.8), since one small network shared by all points learns a general rule for occupancy instead of 32,768 separate outputs. It also needed much longer training to get there.

3.2 Parametric network (10 points)

The decoder maps a random 2D point (u, v) in [0, 1]2, together with the image feature, to a 3D point, so it learns to fold a flat square onto the chair surface. One sheet has to cover legs, seat and back, so it gets stretched across the gaps between parts.

Input, test 0
input (test 0)
Prediction, test 0
prediction (F1 71.8)
Ground truth, test 0
ground truth
Input, test 150
input (test 150)
Prediction, test 150
prediction (F1 49.8)
Ground truth, test 150
ground truth
Input, test 300
input (test 300)
Prediction, test 300
prediction (F1 54.3)
Ground truth, test 300
ground truth

eval_parametric

F1@0.05: 72.48

Compared to the point decoder (79.48), it scores lower: its points form one connected surface, which looks more regular, but a single sheet cannot cover separate parts like legs without leaving points in the gaps (test 150: 49.8 against 55.6).

3.3 Extended dataset for training (10 points)

Point cloud model trained on chairs only versus on chairs, cars and planes, evaluated on each class.

test settrained on chairstrained on 3 classes
chair79.4880.01
car73.3692.81
plane66.5096.21

Training on three classes does not hurt chairs (80.01 against 79.48), even though only about half of the training data is chairs. The chair-only model cannot represent other classes: for a plane it predicts a chair-sized cloud instead of thin wings.

Input, plane
input (plane)
Trained on chairs, plane
trained on chairs (F1 30.8)
Trained on 3 classes, plane
trained on 3 classes (F1 94.0)
Ground truth, plane
ground truth
Input, car
input (car)
Trained on chairs, car
trained on chairs (F1 76.1)
Trained on 3 classes, car
trained on 3 classes (F1 98.2)
Ground truth, car
ground truth
Input, chair
input (chair)
Trained on chairs, chair
trained on chairs (F1 84.4)
Trained on 3 classes, chair
trained on 3 classes (F1 83.2)
Ground truth, chair
ground truth