16-825 Assignment 2: Single View to 3D

Models for 2.1, 2.2, 2.3, 2.4 and 3.1 were trained and evaluated with the provided pretrained ResNet18 features (--load_feat). The 3.3 three-class model is the exception: it is trained end-to-end from images.

1.1 Fitting a Voxel Grid

Optimized voxel grid
Ground truth

1.2 Fitting a Point Cloud

Optimized point cloud
Ground truth

1.3 Fitting a Mesh

Optimized mesh
Ground truth

2.1 Image to Voxel Grid

Example A: input RGB
Ground truth mesh
Predicted voxel grid
Example B: input RGB
Ground truth mesh
Predicted voxel grid
Example C: input RGB
Ground truth mesh
Predicted voxel grid

F1@0.05: 61.860

2.2 Image to Point Cloud

Example A: input RGB
Ground truth mesh
Predicted point cloud
Example B: input RGB
Ground truth mesh
Predicted point cloud
Example C: input RGB
Ground truth mesh
Predicted point cloud

F1@0.05: 76.332

2.3 Image to Mesh

The mesh model used a 50,000-step training budget with w_smooth = 0.1.

Example A: input RGB
Ground truth mesh
Predicted mesh
Example B: input RGB
Ground truth mesh
Predicted mesh
Example C: input RGB
Ground truth mesh
Predicted mesh

F1@0.05: 72.056

Comparing voxel vs. point cloud vs. mesh

RepresentationF1@0.05
Voxel grid61.860
Point cloud76.332
Mesh72.056

Point cloud decoders regress coordinates directly with no structural prior, so they can place points freely and often score highest on F1; mesh decoders are constrained to deform a fixed template, so they struggle with thin structures like chair legs; voxel grids are limited by their fixed resolution and need marching cubes to extract a surface, which can lose fine detail.

2.4 Analyse Effects of Hyperparameter Variations

Varied the mesh decoder's Laplacian smoothness weight across {0.1, 1.0, 10.0}

Input RGB
Ground truth mesh
w_smooth = 0.1
w_smooth = 1.0
w_smooth = 10.0
F1 curve, w_smooth = 0.1
F1 curve, w_smooth = 1.0
F1 curve, w_smooth = 10.0
w_smoothTraining budget (steps)F1@0.05
0.150,00072.056
1.050,00071.941
10.050,00071.056

With a 50,000-step training budget, all models were evaluated on the same 678 chairs using matching views and sampling seeds. Weight 0.1 gives the best F1 (72.056), but only slightly exceeds 1.0 (71.941); weight 10.0 scores lower (71.056).

All three GIFs still show jagged, folded-looking surfaces. Stronger smoothing does not clearly remove these artifacts, which F1 does not directly measure. One possible reason is that our Chamfer loss sums over 1,000 points, making the smoothness term relatively weak. Normalising Chamfer and retuning the weight may improve this balance.

2.5 Interpret Your Model

Error heatmap

Each predicted vertex is colored by its distance to the nearest ground-truth surface point (blue = close, red = far), to see which parts of the shape the model reconstructs well.

Errors concentrate at the tips of the mesh's spiky protrusions. The model gets the rough shape right but not surface smoothness, which is consistent with 2.4's finding that F1 doesn't reward smoothness.

Latent space interpolation

Linearly interpolating between two test images' encoded feature vectors and decoding every step, to check whether the learned feature space is a smooth, meaningful shape manifold rather than a lookup table.

Interpolating between two chairs' encoded features

Every intermediate shape is spiky like the endpoints, but the mesh reshapes continuously. The feature space seems locally smooth even though individual decodes are noisy.

3.1 Implicit Network

Test example A(Sofa): input RGB / ground truth / prediction
Test example B(Chair): input RGB / ground truth / prediction
Test example C(Chair): input RGB / ground truth / prediction
F1 score vs. distance threshold on the test set

F1@0.05: 47.858. The decoder takes the 512-d image feature and a positionally encoded (x, y, z) query and outputs occupancy, and is queried on a 32×32×32 grid followed by marching cubes. It scores below the voxel decoder (61.860): a single global feature has to specify occupancy at every location through a small MLP, which may give blurrier occupancy boundaries than a decoder that outputs the grid directly.

3.3 Extended Dataset Training

Point-cloud model trained on chair + car + plane (30k iterations, same architecture as 2.2) versus the chair-only model, both evaluated on the same 3-class test images (one fixed view per model).

Plane: input RGB
Ground truth mesh
Trained on 3 classes
Trained on chairs only
Car: input RGB
Ground truth mesh
Trained on 3 classes
Trained on chairs only
Chair: input RGB
Ground truth mesh
Trained on 3 classes
Trained on chairs only
F1@0.05PlaneCarChairAll
Trained on 3 classes94.0892.2476.7085.49
Trained on chairs only48.7861.9277.1165.35
F1 score vs. distance threshold, per class

On chairs the two models are indistinguishable (76.70 vs. 77.11), so training on three classes did not hurt the chair class or help it. The chair-only model still gets some points near planes and cars but produces a chair-like blob instead of the right shape, while the 3-class model recovers wings and car bodies clearly.

3D consistency and diversity (chairs)

Consistency: the same test chair is fed from 4 different views and the predictions are compared with each other (mean pairwise Chamfer distance, ×100, lower = more consistent).
Diversity: mean Chamfer distance between the predictions of 1000 random pairs of different chairs; the ground-truth chairs have 7.73.

ChairsCross-view inconsistency (lower is better)Diversity (GT = 7.73)Chamfer to GT (lower is better)
Trained on 3 classes2.335.243.79
Trained on chairs only2.263.693.84

Consistency and the distance to the ground truth are essentially the same for both models. The difference is diversity: both models output less varied chairs than the real ones, they drift toward an average chair, but the three-class model is noticeably closer to the ground truth (5.24 vs. 3.69).