Assignment 2: Single View to 3D

Name: Tashvik Dhamija (tdhamija)

1.1 Fitting a Voxel Grid

Fit voxel grid
Optimized voxel grid (left) vs ground truth (right)

1.2 Fitting a Point Cloud

Fit point cloud
Optimized point cloud (left) vs ground truth (right)

1.3 Fitting a Mesh

Fit mesh
Optimized mesh (left) vs ground truth (right), w_smooth = 0.1

Effect of w_smooth

w_smooth 0.05
w_smooth = 0.05
w_smooth 0.3
w_smooth = 0.3
w_smooth 0.5
w_smooth = 0.5

2.1 Image to Voxel Grid

Trained for 6000 iterations. Architecture: ResNet18 encoder → Linear(512, 64) → reshape to (1, 4, 4, 4) → 3x ConvTranspose3d (4³ → 8³ → 16³ → 32³).

Vox input 1
Input 1
Vox prediction 1
Predicted (left) vs GT (right)
Vox input 2
Input 2
Vox prediction 2
Predicted (left) vs GT (right)
Vox input 3
Input 3
Vox prediction 3
Predicted (left) vs GT (right)

F1 Score

Vox F1 plot
F1 score vs threshold

Avg F1@0.05 = 43.7%

2.2 Image to Point Cloud

Trained for 3000 iterations, n_points = 1000. Architecture: ResNet18 encoder → Linear(512, 1024) → ReLU → Linear(1024, 1024) → ReLU → Linear(1024, n_points * 3).

Point input 1
Input 1
Point prediction 1
Predicted (left) vs GT (right)
Point input 2
Input 2
Point prediction 2
Predicted (left) vs GT (right)
Point input 3
Input 3
Point prediction 3
Predicted (left) vs GT (right)

F1 Score

Point F1 plot
F1 score vs threshold

Avg F1@0.05 = 77.7% (n_points = 1000)

2.3 Image to Mesh

Trained for 3000 iterations. Architecture: ResNet18 encoder → Linear(512, 1024) → ReLU → Linear(1024, 1024) → ReLU → Linear(1024, n_verts * 3) vertex offsets, applied to an icosphere (subdivision level 4) initial mesh.

Mesh input 1
Input 1
Mesh prediction 1
Predicted (left) vs GT (right)
Mesh input 2
Input 2
Mesh prediction 2
Predicted (left) vs GT (right)
Mesh input 3
Input 3
Mesh prediction 3
Predicted (left) vs GT (right)

F1 Score

Mesh F1 plot
F1 score vs threshold

Avg F1@0.05 = 70.5%

Comparing voxel, point cloud, and mesh

RepresentationAvg F1@0.05
Voxel grid43.7%
Point cloud77.7%
Mesh70.5%

The voxel grid performs worst because the model has to learn the spatial dependencies between neighboring voxels through the convolutional decoder, which is a harder relationship to learn. The coarse 32³ grid resolution adds to this problem, since fine details like thin chair legs are easy to lose entirely at that resolution.

The mesh performs better, but deforming a fixed topology icosphere with a fixed number of triangle primitives can force artifacts. The mesh has to bend a constrained set of triangles into shapes they are not naturally suited for, which shows up as jagged, distorted geometry.

The point cloud shows the best understanding of shape overall, but with a fixed n_points budget, the predicted points end up dense in the high confidence central body of the chair, while fewer points populate the less certain extremities, such as the backrest and the legs.

2.4 Hyperparameter Variation: n_points

Varying n_points (the number of points the point-cloud decoder predicts and that chamfer loss is computed over) for the point cloud model, keeping all other hyperparameters fixed. Each variant (500, 1000, 1500) was trained for 3000 iterations with the same architecture as 2.2.

n_points=500
n_points = 500, Avg F1@0.05 = 70.8%
n_points=1000
n_points = 1000, Avg F1@0.05 = 77.7%
n_points=1500
n_points = 1500, Avg F1@0.05 = 81.6%

F1@0.05 increases monotonically with n_points (70.8% → 77.7% → 81.6%). A denser predicted point cloud gives finer-grained coverage of the true surface, which directly improves the precision/recall computation at a fixed distance threshold. The tradeoff is compute and memory cost per sample (and diminishing returns are expected eventually), but within this range, more points reliably helps.

2.5 Model Interpretation: Latent Space Interpolation

To understand what the learned image feature represents, I encode two different test images independently with the ResNet18 encoder, then linearly interpolate between their two 512-d feature vectors (t = 0, 0.25, 0.5, 0.75, 1) and decode each interpolated feature. If the decoder learned a smooth, generalizable shape space (rather than memorizing specific outputs), the in-between predictions should look like coherent, plausible intermediate chairs rather than incoherent noise.

Voxel

Vox input A 1
Image A
Vox interpolation 1
A → B
Vox input B 1
Image B
Vox input A 2
Image A
Vox interpolation 2
A → B
Vox input B 2
Image B

Point Cloud

Point input A 1
Image A
Point interpolation 1
A → B
Point input B 1
Image B
Point input A 2
Image A
Point interpolation 2
A → B
Point input B 2
Image B

Mesh

Mesh input A 1
Image A
Mesh interpolation 1
A → B
Mesh input B 1
Image B
Mesh input A 2
Image A
Mesh interpolation 2
A → B
Mesh input B 2
Image B

The voxel model shows the clearest, smoothest morph between the two endpoint shapes across all 5 steps. Point cloud interpolation shows visible but more subtle shape change (point density/spread shifting gradually). Mesh interpolation also shows gradual change in size/shape, though the predicted mesh surfaces are consistently jagged and self-intersecting regardless of interpolation step, reflecting the same fixed topology constraint discussed in 2.3.

3.1 Implicit Network

Trained for 2000 iterations. Architecture: ResNet18 encoder → two-stage FiLM-conditioned coordinate MLP → Linear(256, 1) occupancy logit, queried at a fixed 32³ grid of points in (-1,1)³.

An implicit decoder takes a 3D coordinate and the image feature as input and outputs an occupancy value at that point, instead of predicting a whole fixed-size grid in one forward pass. The image feature conditions the coordinate pathway via two stacked FiLM (scale and shift) layers. At inference, the network is queried at the full grid to reconstruct the occupancy grid, which is then converted to a mesh via marching cubes, the same as the voxel grid pipeline in 2.1.

Implicit input 1
Input 1
Implicit prediction 1
Predicted (left) vs GT (right)
Implicit input 2
Input 2
Implicit prediction 2
Predicted (left) vs GT (right)
Implicit input 3
Input 3
Implicit prediction 3
Predicted (left) vs GT (right)

F1 Score

Implicit F1 plot
F1 score vs threshold

Avg F1@0.05 = 31.6%

3.2 Parametric Network

Trained for 2000 iterations. Architecture: ResNet18 encoder → two-stage FiLM-conditioned coordinate MLP → Linear(256, 3) output point, applied to a fixed set of n_points sampled on (-1,1)².

A parametric decoder maps sampled 2D points on a fixed (-1,1)² domain to 3D points, instead of regressing all output points directly from the image feature alone as the point cloud decoder in 2.2 does. The image feature conditions the 2D coordinate pathway via the same two-stage FiLM (scale and shift) design used in 3.1, with a final linear layer outputting 3D coordinates instead of an occupancy value.

Parametric input 1
Input 1
Parametric prediction 1
Predicted (left) vs GT (right)
Parametric input 2
Input 2
Parametric prediction 2
Predicted (left) vs GT (right)
Parametric input 3
Input 3
Parametric prediction 3
Predicted (left) vs GT (right)

F1 Score

Parametric F1 plot
F1 score vs threshold

Avg F1@0.05 = 67.6%