16-825 Assignment 2: Single View to 3D

Setup. All models were trained on a single NVIDIA V100 GPU. The image encoder is an ImageNet-pretrained ResNet18 that is fine-tuned end to end; the --load_feat option (precomputed features) was not used.

1. Exploring loss functions

1.1 Fitting a voxel grid

Optimized for 3000 iterations with Adam (lr 4e-4) from a random initialization. The fitted grid is converted to a mesh by marching cubes at probability 0.5 for rendering.

Voxel fitting: per-iteration, fitted and target voxel grids
Voxel grid, fitted grid and target grid.

1.2 Fitting a point cloud

Optimized for 3000 iterations (Adam, lr 4e-4) on 5000 points initialized from a Gaussian.

Point cloud fitting: per-iteration, fitted and target clouds
Point cloud, fitted cloud and target cloud.

1.3 Fitting a mesh

Optimized for 5000 iterations (Adam, lr 4e-4).

Mesh fitting: per-iteration, fitted and target meshes
Mesh, the fitted mesh and the target mesh.

2. Reconstructing 3D from a single view

2.1 Image to voxel grid

Training: batch 128, Adam lr 1e-3, 3000 steps. Training the model for 3000 more steps could have give better F1 score

F1@0.05 = 58.7

Threshold0.010.020.030.040.05
F14.018.935.949.258.7
F1 vs threshold for the voxel model
F1 vs. threshold for voxewl prediction.
Voxel prediction, test example 0
Test example 0.
Voxel prediction, test example 400
Test example 400.
Voxel prediction, test example 600
Test example 600.

2.2 Image to point cloud

Training: batch 32, Adam lr 4e-4, 6000 steps.

F1@0.05 = 80.2

Threshold0.010.020.030.040.05
F13.621.247.468.180.2
F1 vs threshold for the point-cloud model
F1 vs. threshold for point cloud prediction.
Point-cloud prediction, test example 0
Test example 0.
Point-cloud prediction, test example 400
Test example 400.
Point-cloud prediction, test example 600
Test example 600.

2.3 Image to mesh

Training: batch 32, Adam lr 4e-4, 6000 steps.

F1@0.05 = 73.3

Threshold0.010.020.030.040.05
F14.724.146.162.573.3
F1 vs threshold for the mesh model
F1 vs. threshold for mesh prediction.
Mesh prediction, test example 0
Test example 0.
Mesh prediction, test example 400
Test example 400.
Mesh prediction, test example 600
Test example 600.

Comparing voxels, point clouds and meshes

RepresentationF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05
Point cloud3.621.247.468.180.2
Mesh4.724.146.162.573.3
Voxel grid4.018.935.949.258.7

Point cloud > mesh > voxel grid.

2.4 Effect of the Laplacian smoothness weight (w_smooth)

I trained the image-to-mesh model twice with identical settings, changing only the smoothness weight: the default w_smooth = 0.1 and a 10× larger w_smooth = 1.0.

w_smoothF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05
0.14.65123.85345.54361.93273.080
1.04.42222.46542.51057.60868.340
F1 vs threshold for w_smooth 0.1 and 1.0
F1 vs. threshold for both settings.
w_smooth comparison, test example 0
Test example 0: Input | w_smooth = 0.1 | w_smooth = 1.0 | Ground truth.
w_smooth comparison, test example 100
Test example 100.
w_smooth comparison, test example 200
Test example 200.

Analysis. Increasing w_smooth from 0.1 to 1.0 makes the meshes visibly cleaner, fewer spikes, and more rounded blob-like surfaces. But F1 is lower. The Laplacian term pulls each vertex toward the average of its neighbors, which directly fights the sharp, thin geometry that chairs are made of, legs get shortened and armrests and seat edges pulled inward. The conclusion is that smoothness is a trade-off: a small weight might give visually worse artifacts, while a large weight gives visual tidiness at the cost of geometric accuracy.

2.5 Interpreting the model: where do the points go wrong?

F1 compresses two different failure modes into one number- predicted points that are in the wrong place (precision) and parts of the object that nothing covers (recall). To see both, I colored every point by the same distance test the metric uses (τ = 0.05), for the point-cloud model from 2.2:

The green fraction is the precision and the gray fraction is the recall, so each GIF shows which parts of the chair the score is lost on.

Error visualization, test 150, F1 75.4
Test #150 (F1 75.4)
Error visualization, test 301, F1 89.4
Test #301 (F1 89.4)
Error visualization, test 527, F1 82.5
Test #527 (F1 82.5)

Analysis: The model reliably reconstructs the bulky central mass of a chair (seat, cushions), where almost all predictions are green and almost all ground truth is gray.Recall is lost on thin, far-from-center structures: legs, backrests and armrest edges are purple. Precision is lost as a halo of orange and red points around the outline, where the model is indecisive between plausible positions instead of committing to one. Chairs with more thin structure (test #150) score noticeably lower than bulky ones (test #301), even though the overall shape is right in both.

3. Exploring other architectures and datasets

3.1 Implicit network

Architecture: The query point x ∈ (-1,1)³ is encoded with Sin/Cos positional encoding and combined with the 512-d ResNet18 image feature in a linear layer to 256 units, followed by three residual blocks (256→256, ReLU) and a linear output. At test time the network is queried on the full 32×32×32 meshgrid of (-1,1)³, and the resulting occupancy grid is converted to a mesh with marching cubes and evaluated exactly like the voxel model.

Training: binary cross-entropy against the ground-truth voxels on all 32³ grid cells each step, with occupied cells weighted 3× (pos_weight = 3); batch 32, Adam lr 2e-4; 3000 steps.

F1@0.05 = 58.140

Implicit modelF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05
Final: all cells, pos_weight 33.02315.51031.35245.85858.140
F1 vs threshold for the implicit network
F1 vs. threshold.
Implicit network prediction, test example 0
Test example 0.
Implicit network prediction, test example 200
Test example 200
Implicit network prediction, test example 500
Test example 500.

3.2 Parametric network

Architecture (AtlasNet-style): 4 surface patches, each a small MLP (514 → 256 → 256 → 128 → 3, ReLU) that maps a 2D sample (u, v) ∈ [0,1]², concatenated with the 512-d image feature, to a 3D point. Each patch learns one parametric piece of the surface, and their union covers the object.

Training: every step samples fresh random (u, v) points (250 per patch, 1000 in total) and minimizes chamfer distance to 1000 points sampled from the ground-truth mesh; batch 32, Adam lr 4e-4, 3000 steps. A

F1@0.05 = 75.321

Threshold0.010.020.030.040.05
F14.84524.52547.41764.29175.321

With only 3000 training steps the parametric network comes close to the point-cloud decoder from 2.2, which trained for 6000 steps.

F1 vs threshold for the parametric network
F1 vs. threshold.
Parametric network prediction, test example 0
Test example 0.
Parametric network prediction, test example 300
Test example 300
Parametric network prediction, test example 400
Test example 400.

3.3 Training on an extended dataset

I compared two image-to-mesh models: the chair-only model from 2.3 and the same architecture trained on all three classes (3000 steps). Both are evaluated on the extended dataset's test set with 3 classes

F1@0.05 / test classChair-only model (6k steps)3-class model (3k steps)
Chair54.4068.88
Airplane45.1991.78
Car55.4388.64

Note: the two models were not trained for the same number of steps because of limited compute resources. The chair-only model was trained for 6k steps and the 3-class model for only 3k steps, so the comparison is not step-matched, and the size of the gap should be read with that in mind. The 3-class model scores higher on every class despite having half the training.

F1 vs threshold, chair-only vs 3-class, per class
F1 vs. threshold for the chair-only (dashed) and 3-class (solid) models, overall and per class.

Each GIF shows Input RGB | chair-only model | 3-class model | Ground truth:

Airplane: chair-only vs 3-class model
Airplane (test 30).
Airplane: chair-only vs 3-class model
Airplane (test 270).
Car: chair-only vs 3-class model
Car (test 525).
Car: chair-only vs 3-class model
Car (test 615).
Chair: chair-only vs 3-class model
Chair (test 773).
Chair: chair-only vs 3-class model
Chair (test 833).

Analysis.