Assignment 2: Single View to 3D

16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)

1.1 Fitting a Voxel Grid (4 points)

Left: fitted. Right: ground truth.

Marching-cubes meshes of the fitted voxel grid and ground truth

1.2 Fitting a Point Cloud (3 points)

Fitted point cloud on the left and ground truth on the right

1.3 Fitting a Mesh (3 points)

Fitted mesh on the left and ground truth on the right

2.1 Image to Voxel Grid (20 points)

Q2 models use cached ResNet18 features (--load_feat).

Voxel model is trained for 25,080 ietrations.

The starter code has a buf in the evaluation code that does not properly account for the different x/z scaling between the input and ground truth meshes. The pictures below are rendered after fixing this bug.

Left to right: input RGB, prediction, ground-truth mesh. Voxel surfaces use occupancy probability 0.3.

Test example 1: input RGB, predicted voxel surface, ground-truth mesh
Test example 2: input RGB, predicted voxel surface, ground-truth mesh
Test example 3: input RGB, predicted voxel surface, ground-truth mesh

Average F1@0.05: 66.823

Voxel model F1 score against distance threshold

2.2 Image to Point Cloud (20 points)

Point cloud and mesh models trained for 50,000 updates.

Test example 1: input RGB, predicted point cloud, ground-truth mesh
Test example 2: input RGB, predicted point cloud, ground-truth mesh
Test example 3: input RGB, predicted point cloud, ground-truth mesh

Average F1@0.05: 84.328

Point cloud model F1 score against distance threshold

2.3 Image to Mesh (20 points)

Test example 1: input RGB, predicted mesh, ground-truth mesh
Test example 2: input RGB, predicted mesh, ground-truth mesh
Test example 3: input RGB, predicted mesh, ground-truth mesh

Average F1@0.05: 75.586

Mesh model F1 score against distance threshold
RepresentationF1@0.05 (%)
Voxel66.823
Point cloud84.328
Mesh75.586

The point cloud model shows the best F1, which is also visually the best, although the points are not not well distributed on the surfaces but concentrate on the chair corners.

Voxel model shows a lot of noisy geometry that drags down the F1.

The mesh model is the worst, with very pointy faces.

2.4 Mesh Smoothness Ablation (10 points)

I varied w_smooth, keeping the other settings fixed: 10,000 updates, the same training seed, and 610 of the 6,100 training objects held out for validation (5,490 used for training). Validation scores average three surface-sampling seeds.

Smoothness weightValidation F1@0.05 (%)
0.174.958
0.572.776
170.796
369.344
1062.044

Left to right: input, predictions with weights 0.1, 0.5, 1, 3, 10, and GT.

Test example 1: input, mesh predictions at smoothness weights 0.1, 0.5, 1, 3, and 10, and ground truth
Test example 2: input, mesh predictions at smoothness weights 0.1, 0.5, 1, 3, and 10, and ground truth
Test example 3: input, mesh predictions at smoothness weights 0.1, 0.5, 1, 3, and 10, and ground truth

Weight 0.1 shows the highest validation F1 but it is visually the worst. The predicion doesn't capture the shapes well. The weight 3 looks better since it has continuous surfaces, although its F1 is lower.

2.5 Interpreting the Model (10 points)

Left to right: input, prediction-to-GT distance, GT-to-prediction distance, color scale. Blue means close; red means distance ≥ 0.05.

Warm prediction regions show misplaced geometry. Warm GT regions show geometry missing from the prediction. Both pointcloud and mesh models show the weakness of captureing the edge and outline of the shapes.

Point Cloud

Point cloud example 1: prediction and ground truth colored by distance to the other point set
Point cloud example 2: prediction and ground truth colored by distance to the other point set
Point cloud example 3: prediction and ground truth colored by distance to the other point set

Mesh

Mesh example 1: sampled prediction and ground truth colored by distance to the other point set
Mesh example 2: sampled prediction and ground truth colored by distance to the other point set
Mesh example 3: sampled prediction and ground truth colored by distance to the other point set

3.1 Implicit Network (10 points)

The model is trained with a simple mlp, whose input is a simply concatenation of image features and a 3D position.

As I mentioned in the 2.1, the starter code has a bug in the evaluation code that does not properly account for the different x/z scaling between the input and ground truth meshes. The model below is trained before the bug is fixed, therefore, the left most column shows wrong x/z scaling - the 2nd example is especially obvious. So I added a column that shows the x/z-corrected prediction, noted that it's not rescaling during evaluation rather than re-training.

Left to right: input, original prediction, x/z-corrected prediction, GT. Y stays unchanged.

Implicit network example 1: input RGB, original prediction, rescaled prediction, ground-truth mesh
Implicit network example 2: input RGB, original prediction, rescaled prediction, ground-truth mesh
Implicit network example 3: input RGB, original prediction, rescaled prediction, ground-truth mesh

Test F1@0.05: original (buggy) 24.145; x/z scale-aligned 23.322 (678 objects).

Implicit network test F1: original buggy evaluation and x/z scale-aligned evaluation, leaving y unchanged

Error Heatmaps

Input | original-to-GT | rescaled-to-GT | GT-to-rescaled | color scale. Centered surfaces, 10,000 points; red means error ≥ 0.05.

According to the heatmap, the implicit model learns a very "averaged" shape, which is roughly look correct but dimensional-wise wrong. Honestly, as a human, I feel this kind of rough-shape learing is kind of similar to human learing trace. Although the F1 is very low and error heatmap looks so bad, I still feel it predicts better than the 3 models in question 2 that have much higher F1 but shows discontinuity. The implicit decoder I used is super naive, which may make learing very inefficient. I think the model is currently underiftting, and can predict better if running for more iterations.

Implicit example 1: original and rescaled prediction error heatmaps, and GT distance to the rescaled prediction
Implicit example 2: original and rescaled prediction error heatmaps, and GT distance to the rescaled prediction
Implicit example 3: original and rescaled prediction error heatmaps, and GT distance to the rescaled prediction