16-825 Assignment 2: Single View to 3D


1. Exploring loss functions

1.1 Fitting a voxel grid

Left: optimized voxel grid. Right: ground truth.

fit vox

1.2 Fitting a point cloud

Left: optimized point cloud. Right: ground truth.

fit point

1.3 Fitting a mesh

Left: prediction. Right: ground truth.

fit mesh


2. Reconstructing 3D from single view

input RGB | prediction | ground-truth mesh

2.1 Image to voxel grid

F1@0.05 = 63.40

vox 1 vox 2 vox 3

eval vox

2.2 Image to point cloud

F1@0.05 = 87.24

point 1 point 2 point 3

eval point

2.3 Image to mesh

F1@0.05 = 82.74

mesh 1 mesh 2 mesh 3

eval mesh

Quantitative comparison

Representation F1@0.05
Point cloud 87.24
Mesh 82.74
Voxel 63.40

The point cloud representation wins, as it has less constraints compared to mesh & voxels. Mesh representation is restricted by topology, whereas voxels are restricted by resolution.

2.4 Hyperparameter analysis

Mesh initialization scale

init F1@0.05
ico_sphere(4) (radius 1.0), free offsets 82.74
sphere scaled to 0.4, offsets capped at ±0.5 58.76

init scale comparison

Left-to-right: input, ico_sphere(4) (radius 1.0), scaled-init mesh, ground truth. All 3D panels rotate 360°, in a shared coordinate frame derived from the ground truth.

Baseline deforms a ico_sphere(4) with radius = 1 and free offsets, and the variant starts from the sphere scaled to 0.4 with offsets capped at ±0.5. The scaled init gives tighter, less folded shapes, but F1 score drops. This shows that initialization scale fixes the folding, but the deformation range must still cover the shape distribution.

2.5 Interpreting the model

Latent Interpolation

Interpolating the features (dimension = 512) of two images and decoding them into voxel representation gives a smooth, monotonic morph, showing that the decoder learned a continuous shape space.

latent interpolation Left to right: input image A · interpolation weights α = 0, 0.25, 0.5, 0.75, 1.0 · input image B


3. Exploring other architectures

3.1 Implicit network

F1@0.05 = 57.66

Decoder: (image feature, 3D point) → occupancy

implicit 1 implicit 2 implicit 3

eval implicit

F1@0.05
Voxel (transposed 3D conv, equal 10 000-iteration budget) 58.38
Implicit (coordinate MLP) 57.66