16-825 Assignment 2: Single View to 3D

1. Exploring loss functions

1.1 Fitting a voxel grid

Left: optimized voxel grid. Right: ground truth.

1.2 Fitting a point cloud

Left: optimized point cloud. Right: ground truth.

1.3 Fitting a mesh

Sphere init, with smoothness loss
Sphere init, no smoothness loss
Bounding-box sphere init
Cow init, with smoothness loss

The smoothness loss gives far fewer spikes, even from a worse initialization such as a larger sphere. Even with it, the gaps between the chair legs are not modelled well. An initialization closer in scale to the target (the bounding-box sphere) gives a better result. For fun, I also tried starting from the cow mesh to see what it would give.

2. Reconstructing 3D from single view

2.1 Image to voxel grid

F1@0.05: 73.19

Test example 0
Test example 400
Test example 250
F1 vs. threshold

2.2 Image to point cloud

F1@0.05: 72.86

Test example 150
Test example 450
Test example 550
F1 vs. threshold

2.3 Image to mesh

F1@0.05: 70.02

Test example 0
Test example 500
Test example 150
F1 vs. threshold

Comparison

RepresentationF1@0.05
Voxel grid73.19
Point cloud72.86
Mesh70.02

2.4 Effect of hyperparameter variations

Early in training, the voxel model frequently predicted completely empty grids. To counter this, I added a weight to the BCE loss for ground-truth voxels that are occupied (pos_weight 2). This led to faster convergence and fewer completely empty predictions. I then added a Dice loss term (w_dice 1) on top to see whether it would further improve the model's performance.

Voxel model lossF1@0.05
BCE (pos_weight 2)71.64
BCE (pos_weight 2) + Dice73.19

Adding the Dice loss gave only a very small gain in F1, but the predictions themselves were slightly better and cleaner.

Test example 0, without Dice
Test example 0, with Dice
Test example 400, without Dice
Test example 400, with Dice
Test example 250, without Dice
Test example 250, with Dice
F1 vs. threshold, without Dice
F1 vs. threshold, with Dice

2.5 Interpreting the model

Voxel confidence map. Each cube's size and color show the predicted occupancy probability: large and green is high, small and red is low. Voxels below 0.1 are not drawn. Both models use the same scales.

Test example 0
Test example 200
Test example 400

3. Exploring other architectures / datasets

3.1 Implicit network

F1@0.05: 61.84

Test example 0
Test example 100
Test example 400
F1 vs. threshold

3.2 Parametric network

F1@0.05: 70.89

Test example 100
Test example 400
Test example 500
F1 vs. threshold

3.3 Extended dataset for training

I trained the voxel model on chairs only and on chairs, cars and planes, with the same settings, and evaluated both on the 3-class test set.

Test classTrained on chairs onlyTrained on chair + car + plane
Chair (678)73.3070.13
Car (368)53.6776.56
Plane (405)43.8887.79

The car voxels in the extended dataset are stored with the x and z axes swapped compared to chairs and planes. With the provided evaluation, even the ground-truth car voxels only score about 50 F1. I therefore swapped the x and z axes when evaluating cars. Chairs and planes use the provided evaluation unchanged.

Each example shows the input image, the predictions of both models from that same image, and the ground truth.

Chair, test example 773
Chair, test example 1111
Chair, test example 1450
Car, test example 405
Car, test example 496
Car, test example 680
Plane, test example 0
Plane, test example 101
Plane, test example 303