16-825 Assignment 2: Single View to 3D

1. Exploring Loss Functions

I fit each shape for 100,000 steps. Final loss: voxel 1.01×10−7, point 1.93×10−4, and mesh 3.04×10−4.

1.1 Voxel fitting

Fitted voxel grid
Optimized voxel grid
Voxel ground truth
Ground truth

1.2 Point-cloud fitting

Fitted point cloud
Optimized point cloud
Point-cloud ground truth
Ground truth

1.3 Mesh fitting

Fitted mesh
Deformed ico-sphere
Mesh ground truth
Ground-truth mesh

The mesh starts as a sphere. Its fixed topology makes thin legs and holes hard to fit.

2. Reconstructing 3D from a Single View

RepresentationPrecision@0.05Recall@0.05F1@0.05
Voxel67.3448.6852.85
Point cloud80.3379.7578.90
Mesh67.2774.4970.30

Point cloud works best because each point can move freely. Mesh is smooth but has fixed connectivity. Voxels lose thin parts at 32³ resolution.

2.1 Image to voxel grid

F1@0.05: 52.85
Voxel F1 curve

Three test examples

Voxel input 1
Input RGB
Voxel prediction 1
Prediction
Voxel GT 1
Ground truth
Voxel input 2
Input RGB
Voxel prediction 2
Prediction
Voxel GT 2
Ground truth
Voxel input 3
Input RGB
Voxel prediction 3
Prediction
Voxel GT 3
Ground truth

2.2 Image to point cloud

F1@0.05: 78.90
Point-cloud F1 curve

Three test examples

Point input 1
Input RGB
Point prediction 1
Prediction
Point GT 1
Ground truth
Point input 2
Input RGB
Point prediction 2
Prediction
Point GT 2
Ground truth
Point input 3
Input RGB
Point prediction 3
Prediction
Point GT 3
Ground truth

2.3 Image to mesh

F1@0.05: 70.30
Mesh F1 curve

Three test examples

Mesh input 1
Input RGB
Mesh prediction 1
Prediction
Mesh GT 1
Ground truth
Mesh input 2
Input RGB
Mesh prediction 2
Prediction
Mesh GT 2
Ground truth
Mesh input 3
Input RGB
Mesh prediction 3
Prediction
Mesh GT 3
Ground truth

2.4 Effect of Mesh Smoothness Weight

I changed only the mesh smoothness weight. Each model trained for 10,000 steps.

Smoothness weightPrecision@0.05Recall@0.05F1@0.05
0.074.0077.1475.03
0.170.8376.2873.13
1.069.0071.9870.09

A larger smoothness weight gives a lower F1 score. It makes sharp chair parts harder to fit. Weight 0.0 works best here.

2.5 Interpreting the Models

I computed the gradient of the loss with respect to the input. Each plot shows the 20 pixels with the largest gradient.

Voxel

Mean gradient 3.80×10−4; max 6.61×10−3.

Point

Mean gradient 1.23×10−5; max 3.13×10−4.

Mesh

Mean gradient 4.72×10−5; max 8.09×10−4.

The points form small groups around useful image regions. The voxel model has the largest gradients because it predicts a dense grid.

3. Additional Architectures and Data

3.1 Implicit occupancy decoder

F1@0.05: 44.41
Implicit F1 curve

The implicit model has 68.12 precision and 39.15 recall. It misses many occupied cells. Some outputs are empty, so its F1 is lower than the voxel model.

Implicit input 1
Input RGB
Implicit prediction 1
Prediction
Implicit GT 1
Ground truth
Implicit input 2
Input RGB
Implicit prediction 2
Prediction
Implicit GT 2
Ground truth
Implicit input 3
Input RGB
Implicit prediction 3
Prediction
Implicit GT 3
Ground truth

3.2 Parametric point decoder

F1@0.05: 72.52
Parametric F1 curve

This model maps 2D samples to 3D points. Its F1 is 72.52. The normal point decoder is better because its points can move more freely.

Parametric input 1
Input RGB
Parametric prediction 1
Prediction
Parametric GT 1
Ground truth
Parametric input 2
Input RGB
Parametric prediction 2
Prediction
Parametric GT 2
Ground truth
Parametric input 3
Input RGB
Parametric prediction 3
Prediction
Parametric GT 3
Ground truth

3.3 One class versus three classes

Training/evaluation settingF1@0.05
Chair-only model → chair test79.238
Three-class model → same chair test79.218
Three-class model → full test87.528
Three-class airplane / car / chair96.122 / 92.995 / 79.427
Chair-only F1 curve
Chair-only training
Three-class F1 curves
Three-class training

The chair score changes from 79.238 to 79.218. The change is very small. Training on three classes learns cars and planes without hurting the chair result.

Qualitative chair examples

Chair-only input 1
Chair-only input
Chair-only prediction 1
Chair-only prediction
Three-class input 1
Three-class input
Three-class prediction 1
Three-class prediction
Ground truth 1
Ground truth
Chair-only input 2
Chair-only input
Chair-only prediction 2
Chair-only prediction
Three-class input 2
Three-class input
Three-class prediction 2
Three-class prediction
Ground truth 2
Ground truth
Chair-only input 3
Chair-only input
Chair-only prediction 3
Chair-only prediction
Three-class input 3
Three-class input
Three-class prediction 3
Three-class prediction
Ground truth 3
Ground truth