Assignment 2: Single View to 3D

1. Exploring Loss Functions

The geometry is directly optimized in these experiments. The animations compare the optimized representation with the ground truth.

Optimized voxel grid and ground truth
1.1 Fitting a voxel grid
Optimized point cloud and ground truth
1.2 Fitting a point cloud
Optimized mesh and ground truth
1.3 Fitting a mesh

Voxel: Binary cross-entropy with logits supervises occupancy.

Point cloud: A custom symmetric squared Chamfer loss matches the two point sets.

Mesh: Sampled-surface Chamfer loss is combined with uniform Laplacian regularization to encourage smooth geometry.

2. Reconstructing 3D from a Single View

A ResNet18 encoder produces a 512-dimensional image feature. Each decoder maps this feature to a different 3D representation. Each image below shows the input RGB, predicted geometry, and ground-truth mesh, from left to right.

2.1 Image to Voxel Grid

A fully connected layer followed by 3D transposed convolutions predicts a 32 × 32 × 32 occupancy grid. Marching cubes extracts the surface at occupancy 0.5.

Voxel reconstruction: test 000
Test example 000
Voxel reconstruction: test 002
Test example 002
Voxel reconstruction: test 005
Test example 005

2.2 Image to Point Cloud

An MLP with dimensions 512 → 1024 → 2048 → 3000 predicts 1,000 unordered 3D points, trained with Chamfer loss.

Point-cloud reconstruction: test 000
Test example 000
Point-cloud reconstruction: test 002
Test example 002
Point-cloud reconstruction: test 005
Test example 005

2.3 Image to Mesh

An MLP predicts offsets for the 2,562 vertices of a level-4 ico-sphere. The loss combines sampled-surface Chamfer distance and Laplacian smoothing.

Mesh reconstruction: test 000
Test example 000
Mesh reconstruction: test 002
Test example 002
Mesh reconstruction: test 005
Test example 005

Quantitative Comparison

Voxel F1 evaluation curve
Voxel
Point-cloud F1 evaluation curve
Point cloud
Mesh F1 evaluation curve
Mesh
RepresentationAvg F1@0.05 (/100)
Voxel grid47.580
Point cloud77.834
Mesh74.490

The reported validation averages show point cloud > mesh > voxel at threshold 0.05. Point clouds can represent geometry without the fixed connectivity of a mesh template, while connected mesh surfaces provide a continuity prior. A 32³ voxel grid can lose thin structures such as chair legs. Mesh predictions may still contain folds, and a higher F1 at a relaxed distance threshold does not guarantee better topology or fine detail.

2.4 Effect of Learning Rate

The voxel model is compared using learning rates of 1e-4, 4e-4, and 1e-3. All three available experiment checkpoints record step 6000. The same test indices, 000, 002, and 005, are shown below.

Learning RateAvg F1@0.05 (/100)
1e-439.953
4e-447.580
1e-330.298

Test Example 000

Test 000 with learning rate 1e-4
Learning rate: 1e-4
Test 000 with learning rate 4e-4
Learning rate: 4e-4
Test 000 with learning rate 1e-3
Learning rate: 1e-3

Test Example 002

Test 002 with learning rate 1e-4
Learning rate: 1e-4
Test 002 with learning rate 4e-4
Learning rate: 4e-4
Test 002 with learning rate 1e-3
Learning rate: 1e-3

Test Example 005

Test 005 with learning rate 1e-4
Learning rate: 1e-4
Test 005 with learning rate 4e-4
Learning rate: 4e-4
Test 005 with learning rate 1e-3
Learning rate: 1e-3

Among the three tested learning rates, 4e-4 achieves the highest average F1@0.05 (47.580), followed by 1e-4 (39.953) and 1e-3 (30.298). The 4e-4 run exceeds 1e-4 by 7.627 F1 points and 1e-3 by 17.282 points under the same specified training iteration budget. Thus, 4e-4 is the best-performing setting in these reported runs, not necessarily the optimal learning rate for every training configuration.

In example 005, the 1e-3 result contains small disconnected fragments, while lower learning rates retain part of the chair back. This qualitative observation is consistent with the lower average F1 of the 1e-3 run. A larger learning rate may make updates less stable, while 1e-4 may need more iterations to converge. The intermediate rate may offer a better balance between progress and stability within this training budget. These are hypotheses: F1 alone does not establish the optimization dynamics or prove underfitting or instability.

2.5 Model Interpretation: Input Occlusion

For each test image, the upper, middle, or lower third of the foreground bounding box is filled with black. The same voxel checkpoint predicts geometry from the original and masked images without retraining. F1@0.05 uses 1,000 surface samples, with fixed GT samples shared across all four variants of each example and metric seed 0.

Test ExampleInputF1@0.05 (/100)Drop from Original
000Original87.135+0.000
000Upper54.648+32.487
000Middle85.792+1.342
000Lower59.836+27.298
002Original68.980+0.000
002Upper67.322+1.659
002Middle49.166+19.814
002Lower82.386-13.406
005Original30.515+0.000
005Upper31.014-0.499
005Middle65.043-34.528
005Lower5.258+25.257

Drop = original F1 minus masked-input F1, in F1 points. Positive values indicate degradation; negative values indicate improvement. Values are computed before rounding, so displayed differences may differ by 0.001 when calculated from the rounded F1 values.

Original and upper, middle, lower occlusion results for test 000
Test example 000: original and three occlusion conditions
Original and upper, middle, lower occlusion results for test 002
Test example 002: original and three occlusion conditions
Original and upper, middle, lower occlusion results for test 005
Test example 005: original and three occlusion conditions

Example 000: Upper and lower occlusion substantially reduce F1 by 32.487 and 27.298 points, while middle occlusion changes it by only 1.342 points. For this input, the prediction is more sensitive to information in the upper and lower image bands.

Example 002: Middle occlusion produces the largest drop (19.814 points), whereas lower occlusion improves F1 from 68.980 to 82.386.

Example 005: Lower occlusion reduces F1 from 30.515 to 5.258, while middle occlusion improves it to 65.043. Thus, removing image information does not uniformly reduce the measured reconstruction quality. Suppression of misleading image cues is one possible explanation, but these three examples do not establish the cause or demonstrate a general benefit from occlusion.

Under lower occlusion, example 005 retains only 3 occupied voxels, with precision 100.0 but recall only 2.7. The low F1 shows that a small matching fragment can have high precision yet miss almost the entire target surface. Its mean absolute probability change is only about 0.017, illustrating that small probability changes near occupancy 0.5 can substantially alter the extracted surface.

3.1 Implicit Occupancy Network

The implicit decoder predicts occupancy as a function of an image feature and a 3D query coordinate: f(image feature, x, y, z).

A 32 × 32 × 32 meshgrid is generated in [-1, 1]³. Each coordinate is concatenated with the 512-dimensional feature and passed through an MLP: 515 → 256 → 256 → 128 → 1. The 32,768 queries are processed in chunks of 2048. Outputs are occupancy logits, trained with binary cross-entropy; sigmoid converts them to probabilities for evaluation.

Implicit reconstruction: test 000
Implicit network: test example 000
Implicit reconstruction: test 002
Implicit network: test example 002
Implicit reconstruction: test 005
Implicit network: test example 005
Implicit experiment F1 curve
Implicit evaluation curve; older output names still use “vox”.

The supplied curve gives an approximate F1@0.05 of 11.3, below the explicit voxel baseline. The fragmented predictions suggest several possible explanations. One possibility is a bias toward predicting empty space: if empty locations dominate the target grid, the unweighted binary cross-entropy loss may favor conservative occupancy predictions, allowing loss to decrease even when substantial parts of the chair are missed. If many occupancy probabilities remain below or near 0.5, surface extraction could retain only small fragments.

Another possibility is limited geometric learning in the simple MLP. Unlike the convolutional decoder, it has no explicit local convolutional structure, which may make it harder to learn coherent surfaces and thin chair parts from a global image feature. Training configuration may also contribute: the learning rate and iteration budget used for the voxel baseline may not suit this decoder, and the implicit model may require different settings or more training to converge. These are plausible hypotheses, not confirmed causes; checking occupancy distributions, training curves, and controlled decoder comparisons would help distinguish them.