16-825 Assignment 2: Single View to 3D

Sarah Pendhari   Andrew ID: spendhar

1. Exploring Loss Functions

Each GIF shows the ground truth (left) next to the optimized result (right), for the first training sample.

1.1 Fitting a Voxel Grid

Ground truth voxel grid next to the optimized voxel grid
Ground truth | optimized voxel grid

1.2 Fitting a Point Cloud

Ground truth point cloud next to the optimized point cloud
Ground truth | optimized point cloud

1.3 Fitting a Mesh

Ground truth mesh next to the optimized mesh
Ground truth | optimized mesh

2. Reconstructing 3D from a Single View

All models were trained for 3000 iterations using pretrained ResNet18 features (--load_feat) on --device mps. Each GIF shows input RGB, ground truth, and prediction. The same three test examples are used for all three representations.

2.1 Image to Voxel Grid

Voxel reconstruction example 1
Example 1
Voxel reconstruction example 2
Example 2
Voxel reconstruction example 3
Example 3

F1@0.05: 32.305%

F1 score versus threshold for voxels
F1 vs. threshold (voxel)

2.2 Image to Point Cloud

Point cloud reconstruction example 1
Example 1
Point cloud reconstruction example 2
Example 2
Point cloud reconstruction example 3
Example 3

F1@0.05: 75.676%

F1 score versus threshold for point clouds
F1 vs. threshold (point cloud)

2.3 Image to Mesh

Mesh reconstruction example 1
Example 1
Mesh reconstruction example 2
Example 2
Mesh reconstruction example 3
Example 3

F1@0.05: 63.156%

F1 score versus threshold for meshes
F1 vs. threshold (mesh)

Voxel vs. Point Cloud vs. Mesh

Representation F1@0.05
Point cloud 75.676%
Mesh 63.156%
Voxel grid 32.305%

2.4 Analyzing Hyperparameter Variations

I varied the marching-cubes isovalue for the voxel model. The trained model was kept fixed.

Voxel F1 score as a function of marching-cubes isovalue
Effect of marching-cubes isovalue on average F1@0.05.

The F1 score changes with the isovalue. A lower value includes more of the predicted occupancy, while a higher value produces a smaller surface.

The best value is a trade-off between keeping the predicted geometry and removing incorrect geometry.

Note: The marching-cubes isovalue is a post-processing parameter, not a training hyperparameter. I used it to measure the sensitivity of the final voxel reconstruction.

2.5 Interpreting the Model

I looked at per-example F1 scores and reconstruction errors to see how the models behave on individual examples.

Per-sample F1 distributions

Per-sample F1 distribution for voxel reconstruction
Voxel reconstruction: per-sample F1@0.05.
Per-sample F1 distribution for point-cloud reconstruction
Point-cloud reconstruction: per-sample F1@0.05.
Per-sample F1 distribution for mesh reconstruction
Mesh reconstruction: per-sample F1@0.05.

The F1 scores vary across test examples. Some objects are easier to reconstruct than others.

Spatial reconstruction error

I colored the predicted points based on their distance to the nearest ground-truth point. Larger distances indicate larger reconstruction errors.

Voxel reconstruction error heatmap
Voxel reconstruction: test example 1.
Point cloud reconstruction error heatmap
Point-cloud reconstruction: test example 1.
Mesh reconstruction error heatmap
Mesh reconstruction: test example 1.
Voxel reconstruction error heatmap example 2
Voxel reconstruction: test example 2.
Point cloud reconstruction error heatmap example 2
Point-cloud reconstruction: test example 2.
Mesh reconstruction error heatmap example 2
Mesh reconstruction: test example 2.

The errors are more noticeable around thin parts and parts that are not clearly visible in the input image.

This is especially noticeable for the voxel model because of its limited resolution. The model also has to predict the geometry that is not visible from the input view.