Assignment 2: Single View to 3D

Contents

1. Exploring loss functions (10 points)

1.1 Fitting a voxel grid (4 points)

Optimized voxel grid (left) and ground truth (right)
Optimized and ground-truth voxel grid, rotating

1.2 Fitting a point cloud (3 points)

Optimized point cloud (left) and ground truth (right)
Optimized and ground-truth point cloud, rotating

1.3 Fitting a mesh (3 points)

Optimized mesh (left) and ground truth (right)
Optimized and ground-truth mesh, rotating

2. Reconstructing 3D from single view (80 points)

Setup. I used --load_feat for every model in sections 2 and 3. Instead of 100k iterations, I trained each model for up to 20k and kept the checkpoint with the best F1@0.05 on 10% of the training chairs held out for validation. The test set was only used for the numbers reported here. The decoder architectures are in model.py.

2.1 Image to voxel grid (20 points)

The model checkpoint for this task is checkpoint_vox.pth

My voxel model gets F1@0.05 = 69.0 on the test set.

F1 versus threshold for the voxel model
InputPredicted voxels (left) and ground truth (right)
Input image, chair 0Predicted voxels and ground truth, chair 0, rotating
Input image, chair 300Predicted voxels and ground truth, chair 300, rotating
Input image, chair 500Predicted voxels and ground truth, chair 500, rotating

2.2 Image to point cloud (20 points)

The model checkpoint for this task is checkpoint_point.pth

My point cloud model gets F1@0.05 = 77.7 on the test set.

F1 versus threshold for the point model
InputPredicted points (left) and ground truth (right)
Input image, chair 0Predicted points and ground truth, chair 0, rotating
Input image, chair 300Predicted points and ground truth, chair 300, rotating
Input image, chair 500Predicted points and ground truth, chair 500, rotating

2.3 Image to mesh (20 points)

The model checkpoint for this task is checkpoint_mesh.pth

My mesh model gets F1@0.05 = 72.3 on the test set.

F1 versus threshold for the mesh model
InputPredicted mesh (left) and ground truth (right)
Input image, chair 0Predicted mesh and ground truth, chair 0, rotating
Input image, chair 300Predicted mesh and ground truth, chair 300, rotating
Input image, chair 500Predicted mesh and ground truth, chair 500, rotating

Comparing voxels, point clouds and meshes

RepresentationTest F1@0.05
Point cloud77.7
Mesh72.3
Voxel grid69.0

Points do best, then meshes, then voxels. On validation, recall is similar for all three (about 75 to 77), so most of the gap is precision where the mesh and voxel models put surface where there is no chair.

2.4 Analyse effects of hyperparams variations (10 points)

The main hyperparameter I varied is the positive-class weight (pos_weight) in the voxel BCE loss. I also looked at the mesh smoothness weight and the initial mesh. All numbers here are validation F1@0.05.

Voxel pos_weight

Voxel loss (1,024-wide MLP decoder)Val F1PrecisionRecall
Plain BCE (pos_weight 1)57.869.853.4
pos_weight 1.566.071.164.0
pos_weight 267.365.870.4
pos_weight 457.951.767.3
InputPlain BCE, pos_weight 2, ground truth
Input image, chair 100Plain BCE, pos_weight 2 and ground truth, chair 100, rotating
Input image, chair 500Plain BCE, pos_weight 2 and ground truth, chair 500, rotating

Only about 5% of the cells are occupied, so with plain BCE the model plays it safe with high precision, low recall, and thin chairs mostly disappear. pos_weight 2 raises F1 by about 9 points, and 4 overshoots and fills in too much.

I wanted to know if pos_weight actually teaches the model something or mostly moves the decision threshold. At the optimum, thresholding a weight-w model at 0.5 is the same as thresholding a plain-BCE model at 1/(1+w). So I re-evaluated the plain-BCE checkpoint at those isovalues:

Equivalent threshold 1/(1+w)Plain BCE model at that isovaluepos_weight w model at 0.5
0.4 (w = 1.5)65.765.8
0.33 (w = 2)66.267.3
0.2 (w = 4)55.258.0

The two columns are close. The plain model already ranks the cells well, but its probabilities are too low for 0.5 and lowering the isovalue to 1/3 gets back 8.3 of the 9.4 points. This also explains the sweet spot near 2.

Mesh smoothness weight

w_smoothVal F1
0.01 (2.3 model)71.0
0.1 (starter default)69.8
1.066.3

Less smoothing gives higher F1. Laplacian smoothing pulls every vertex towards its neighbours, which makes it difficult to fit the large uneven moves the sphere needs to reach the legs and corners. The stronger weight looks tidier but still isn't a chair-shaped surface.

Initial mesh impact

Initial mesh (w_smooth 0.1)Val F1
Unit ico-sphere69.8
Chair-sized ellipsoid69.9
Chair-sized box69.7
Sphere, ellipsoid, box, ground truth (chair 300)
Meshes from the three initial shapes and ground truth, chair 300, rotating

The starting shape makes almost no difference. The decoder predicts every offset from scratch, so shrinking a too-big sphere is learned in the first few thousand steps, and after that all three are limited by a closed surface with no holes.

2.5 Interpret your model (10 points)

I used the point cloud model.

Where does it go wrong?

Error maps in both directions for four test chairs

Row 2: each predicted point coloured by its distance to the nearest ground-truth point (red hurts precision). Row 3: each ground-truth point coloured by its distance to the nearest predicted point (red hurts recall). Blue is a hit at the 0.05 threshold and red is a miss.

The predicted points are mostly fine (precision 79% to 93%) apart from the halo. But for recall the performance isn't great. On the tall chair 150 the whole upper back is missed (recall 44%), and on the armchair (0) and barrel chair (300) the outer edges are missed. So the model gets the middle of the chair right and misses whatever sticks out.

What does the image feature encode?

Decoded point clouds along a feature-space interpolation

I blended the image features of the armchair (0) and the tall chair (150) and decoded each blend. The shape changes smoothly from wide and low to narrow and tall, with no odd shapes along the way. The proportions change first, and leg-like strands only show up from about alpha 0.6, so the feature mostly encodes overall chair dimensions instead of structure.

3. Exploring other architectures / datasets

3.1 Implicit network (10 points)

The model checkpoint for this task is checkpoint_implicit.pth.

My implicit model gets F1@0.05 = 58.5 on the test set.

F1 versus threshold for the implicit model
InputImplicit prediction (left) and ground truth (right)
Input image, chair 0Implicit prediction and ground truth, chair 0, rotating
Input image, chair 300Implicit prediction and ground truth, chair 300, rotating
Input image, chair 500Implicit prediction and ground truth, chair 500, rotating

The implicit model is about 10 points below the voxel MLP (58.5 against 69.0). Its outputs are very smooth. A small MLP on raw coordinates fits smooth functions easily and sharp ones slowly, and thin legs are the sharp part. I also tried positional encoding of (x, y, z), which learned faster but ended at the same F1. The voxel MLP has 242M parameters against 0.23M here, and its per-cell output weights are probably where its extra detail comes from.

3.2 Parametric network (10 points)

The model checkpoint for this task is checkpoint_parametric.pth.

My parametric model gets F1@0.05 = 72.6 on the test set.

F1 versus threshold for the parametric model
InputPredicted points, patch surfaces, ground truth
Input image, chair 0Parametric points, patch surfaces and ground truth, chair 0, rotating
Input image, chair 300Parametric points, patch surfaces and ground truth, chair 300, rotating
Input image, chair 500Parametric points, patch surfaces and ground truth, chair 500, rotating

Patch surfaces: the decoder evaluated on a regular 24x24 (u, v) grid per patch and connected into triangles. Each colour is one patch.

It scores 5 points below the point decoder (72.6 against 77.7). I think the reason is continuity because each patch is a smooth map of a square, so it forms sheets and has a hard time following thin, separate legs. The patch surfaces show this with a few large sheets for the back and seat that often overshoot the real chair. The upside is that (u, v) is an input, so I can sample as many points as I want from a model trained with 1,000.

3.3 Extended dataset for training (10 points)

The model checkpoints for this task are checkpoints/point3c.pth (three classes) and checkpoint_point.pth (chairs only). I set use_full_dataset = True in dataset_location.py and retrained the point model from 2.2 with the same settings on split_3c.json (6,100 chairs, 3,640 planes, 3,146 cars). Then I evaluated both models on the same 678 chair test objects.

Trained onF1@0.05 on the chair test set
Chairs only77.5
Chairs, planes and cars76.5

Both from the same eval run (the chair-only model got 77.7 in its own run in 2.2).

F1 versus threshold for the three-class model on the chair test set

F1 vs threshold, three-class model on the chair test set.

Test classObjectsThree-class model F1@0.05
Chair67876.7
Car36893.4
Plane40595.7
All1,45186.3
InputTrained on chairs, trained on three classes, ground truth
Input image, chair 0Chair-only model, three-class model and ground truth, chair 0, rotating
Input image, chair 300Chair-only model, three-class model and ground truth, chair 300, rotating
Input image, chair 500Chair-only model, three-class model and ground truth, chair 500, rotating
InputThree-class model: prediction (left) and ground truth (right)
Input image, planeThree-class model prediction and ground truth, plane, rotating
Input image, carThree-class model prediction and ground truth, car, rotating

Training on three classes didn't help the chairs in my case since F1 on the chair test set is about 1 point lower and the predicted chairs look almost the same as before. The planes and cars themselves come out as the right kind of object and score above 93 on average, since they are compact shapes with less variety than chairs.

The outputs are also not more diverse. For both models, the predicted chair sizes (width, height and depth over the test set) vary about 30% less than the real ones, so both pull unusual chairs towards an average chair, and the three-class model's predictions are just as consistent from chair to chair as the chair-only model's. I think this is because the encoder is frozen (--load_feat), so the extra classes can only change the decoder, and a decoder that also has to produce planes and cars has less room left for chair detail. It also saw each chair about half as often in the same number of steps.