Assignment 2 — Single View to 3D

16-825: Learning for 3D Vision

1. Exploring loss functions

1.1 Fitting a voxel grid

def voxel_loss(voxel_src, voxel_tgt):
    loss = torch.nn.functional.binary_cross_entropy_with_logits(voxel_src, voxel_tgt)
    return loss
Turntable of the optimized voxel grid beside the ground truth
Optimized grid (left) and ground truth (right). The fit is exact.

1.2 Fitting a point cloud

def chamfer_loss(point_cloud_src, point_cloud_tgt):
    dist_src = knn_points(point_cloud_src, point_cloud_tgt, K=1).dists  # b x n_src x 1
    dist_tgt = knn_points(point_cloud_tgt, point_cloud_src, K=1).dists  # b x n_tgt x 1
    loss_chamfer = dist_src.squeeze(-1).mean(dim=1) + dist_tgt.squeeze(-1).mean(dim=1)
    return loss_chamfer.mean()
Turntable of the optimized point cloud beside the ground truth
Optimized cloud (left) and ground truth (right), 5000 points each.

1.3 Fitting a mesh

def smoothness_loss(mesh_src):
    loss_laplacian = mesh_laplacian_smoothing(mesh_src, method="uniform")
    return loss_laplacian
Turntable of the optimized mesh beside the ground truth
Optimized mesh (left) and ground truth (right), deformed from an ico_sphere.

2. Reconstructing 3D from a single view

2.1 Image to voxel grid

F1@0.05 on the chair test set (678 models): 75.50

F1 score of the voxel model against distance threshold
F1 vs. distance threshold.
Input RGBPredictedGT
Test sample 0: input image, predicted voxel grid and ground truth mesh
Chair: sample 0
Input RGBPredictedGT
Test sample 400: input image, predicted voxel grid and ground truth mesh
Chair: sample 400
Input RGBPredictedGT
Test sample 600: input image, predicted voxel grid and ground truth mesh
Chair: sample 600
Input RGBPredictedGT
Test sample 200: input image, predicted voxel grid and ground truth mesh
Chair: sample 200

2.2 Image to point cloud

F1@0.05 on the chair test set (678 models): 77.48

F1 score of the point cloud model against distance threshold
F1 vs. distance threshold.
Input RGBPredictedGT
Test sample 0: input image, predicted point cloud and ground truth mesh
Chair: sample 0
Input RGBPredictedGT
Test sample 400: input image, predicted point cloud and ground truth mesh
Chair: sample 400
Input RGBPredictedGT
Test sample 600: input image, predicted point cloud and ground truth mesh
Chair: sample 600
Input RGBPredictedGT
Test sample 200: input image, predicted point cloud and ground truth mesh
Chair: sample 200

2.3 Image to mesh

F1@0.05 on the chair test set (678 models): 74.08

F1 score of the mesh model against distance threshold
F1 vs. distance threshold.
Input RGBPredictedGT
Test sample 0: input image, predicted mesh and ground truth mesh
Chair: sample 0
Input RGBPredictedGT
Test sample 400: input image, predicted mesh and ground truth mesh
Chair: sample 400
Input RGBPredictedGT
Test sample 600: input image, predicted mesh and ground truth mesh
Chair: sample 600
Input RGBPredictedGT
Test sample 200: input image, predicted mesh and ground truth mesh
Chair: sample 200

Comparing voxels, point clouds and meshes

RepresentationF1@0.05
Point cloud (2.2)77.48
Voxel grid (2.1)75.50
Mesh (2.3)74.08

The point cloud does best because its points can go anywhere, and chamfer loss is basically what F1 measures. Voxels lose thin parts like legs to the 323 grid. The mesh is lowest because it starts from one sphere, so it can't open holes between legs, and smoothing pulls thin parts inward. Still, the mesh is the only one that gives a real surface.

2.4 Effect of the smoothness weight (w_smooth)

I test the effect of the Laplacian smoothness weight on the 2.3 mesh decoder, training it with w_smooth = 0, 0.1, 1 and 10.

w_smoothF1@0.02F1@0.05
026.3975.07
0.1 (default)25.6573.23
124.4171.58
1018.1059.33

The same test chair (sample 400) at each weight:

Input RGBPredictedGT
Sample 400 predicted with w_smooth 0
w_smooth = 0
Input RGBPredictedGT
Sample 400 predicted with w_smooth 0.1
w_smooth = 0.1
Input RGBPredictedGT
Sample 400 predicted with w_smooth 1
w_smooth = 1
Input RGBPredictedGT
Sample 400 predicted with w_smooth 10
w_smooth = 10

F1@0.05 drops from 75.1 to 73.2, 71.6 and finally 59.3 since the Laplacian term keeps pulling each vertex toward its neighbours, and thin parts like the legs get squeezed away first. Turning smoothing off gives the best number, but the mesh itself looks broken. F1 only checks point distances, but not penalize folded, self-intersecting faces. I think 0.1 is the sweet spot: it cleans up the surface and costs less than 2 points of F1.

2.5 Interpreting the models

Parametric charts coloured by patch

This visualization shows the 3.2 parametric prediction from four viewpoints, next to the input image. The decoder builds the point cloud from 4 learned 2D charts, and each of the 4,000 sampled points is coloured by the chart that produced it (red, blue, green or orange).

Parametric prediction for sample 400, points coloured by chart, four views
Sample 400, input and four views. The charts specialise: blue covers the seat and front legs, orange the rear legs, and red and green share the back.
Parametric prediction for sample 0, points coloured by chart, four views
Sample 0, an armchair. Here all four charts overlap across the whole shape, so a boxy armchair is not split into parts the way a chair with separate legs and back is.

Point cloud error heatmaps

This visualization colours the 2.2 point cloud prediction by error. The top row shows each predicted point coloured by its distance to the ground truth (precision errors), and the bottom row shows each ground-truth point coloured by its distance to the prediction (missed regions). The colour scale saturates at 0.05, the F1 threshold, so dark red means the point is counted as wrong.

Colour scale from 0 to 0.05
Error heatmaps for sample 400
Sample 400 (a different random view than above). The legs are well covered, while the seat and the top edge of the back are missed most, and a few predicted points float loosely behind the back.
Error heatmaps for sample 0
Sample 0. Errors concentrate on the top edges of the armrests and back and across the seat. Most points sit between 0.03 and 0.05, which is why F1 drops from 77 at 0.05 to about 53 at 0.03.

Interpolating between two chairs

This visualization blends the ResNet features of two test images with weights 0, 0.25, 0.5, 0.75 and 1, and decodes each blend with the 3.1 implicit decoder, showing how one chair turns into the other.

Interpolation from an armchair to a simple chair
Armchair (left) to simple chair (right). The shape changes gradually: the armrests shrink, the back rises and the legs appear, with every intermediate step still a plausible chair. The decoder has learned a smooth space of chair shapes rather than memorising individual ones.

3. Exploring other architectures / datasets

3.1 Implicit network

An MLP takes the image feature and a positionally encoded 3D point and outputs its occupancy logit. It is trained on 4,096 sampled points per shape with BCE, then queried on a 323 grid over (-1,1)3 and meshed with marching cubes.

F1@0.05 on the chair test set (678 models): 76.82 (voxel decoder: 75.50)

F1 score of the implicit model against distance threshold
F1 vs. distance threshold.
Input RGBPredictedGT
Test sample 0: input image, implicit prediction and ground truth mesh
Chair: sample 0
Input RGBPredictedGT
Test sample 400: input image, implicit prediction and ground truth mesh
Chair: sample 400
Input RGBPredictedGT
Test sample 600: input image, implicit prediction and ground truth mesh
Chair: sample 600
Input RGBPredictedGT
Test sample 200: input image, implicit prediction and ground truth mesh
Chair: sample 200

The implicit network beats the voxel decoder by a little (76.82 vs. 75.50). One MLP is shared across all 3D points, so the predicted shape comes out smoother. The gain is small because I still mesh it at 323, but I could query it at a higher resolution without retraining.

3.2 Parametric network

4 per-chart MLPs map random (u, v) points in [0,1]2, combined with the image feature, to 3D points (250 each, 1,000 total), trained with chamfer loss.

F1@0.05 on the chair test set (678 models): 75.26 (point decoder: 77.48)

F1 score of the parametric model against distance threshold
F1 vs. distance threshold.
Input RGBPredictedGT
Test sample 0: input image, parametric prediction and ground truth mesh
Chair: sample 0
Input RGBPredictedGT
Test sample 400: input image, parametric prediction and ground truth mesh
Chair: sample 400
Input RGBPredictedGT
Test sample 600: input image, parametric prediction and ground truth mesh
Chair: sample 600
Input RGBPredictedGT
Test sample 200: input image, parametric prediction and ground truth mesh
Chair: sample 200

The parametric network is slightly worse than the plain point decoder (75.26 vs. 77.48). Each chart has to be a smooth surface bent from a square, so it has to stretch over thin legs and gaps, and points there miss the surface. In return, the output is more organised (each chart covers one part, see 2.5), and I can sample as many points as I want.

3.3 Extended dataset for training

The 2.2 point decoder is trained on chairs only and on chairs + cars + planes, then both are tested on the same 678 chair test views.

Trained onF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05
Chairs only5.8729.6853.4768.4477.62
Chair + car + plane4.7226.2851.1667.3777.25

The three-class model on all three test classes, F1@0.05:

ClassTest modelsF1@0.05
Plane40595.65
Car36892.25
Chair67877.58
F1 against threshold, chair-only model
F1 vs. distance threshold, trained on chairs only.
F1 against threshold on chairs, three-class model
F1 vs. distance threshold on chairs, trained on chair + car + plane.

The same chair test views, predicted by both models:

Input RGBPredictedGT
Sample 0, chair-only model
Chair sample 0: trained on chairs only
Input RGBPredictedGT
Sample 0, three-class model
Chair sample 0: trained on chair + car + plane
Input RGBPredictedGT
Sample 400, chair-only model
Chair sample 400: trained on chairs only
Input RGBPredictedGT
Sample 400, three-class model
Chair sample 400: trained on chair + car + plane
Input RGBPredictedGT
Sample 600, chair-only model
Chair sample 600: trained on chairs only
Input RGBPredictedGT
Sample 600, three-class model
Chair sample 600: trained on chair + car + plane

Car and plane predictions from the three-class model:

Input RGBPredictedGT
Car test sample, three-class model
Car (three-class model)
Input RGBPredictedGT
Plane test sample, three-class model
Plane (three-class model)

Adding cars and planes barely changes chair F1@0.05 (77.62 vs. 77.25), but the gap grows at tighter thresholds (29.7 vs. 26.3 at 0.02). So the model still gets the overall chair shape right, just less precisely. Its chairs look a bit noisier, but they never look like cars or planes, since the image makes the class obvious. I think this is because the decoder now has to cover three very different shape families and sees fewer chairs. Cars and planes score much higher (92 and 96) because their shapes are simpler, with no thin legs or holes.