16-825 Assignment 2: Single View to 3D

Yi Zhao · Andrew ID yizhao5

All models are trained on the single-class (chair) R2N2 split with a frozen-free ResNet-18 encoder trained end to end, and evaluated on the full test split with the F1 score provided in eval_model.py. Predictions and ground truth are always rendered with the same camera, lights and shader, so the pictures compare geometry rather than presentation.

1. Exploring loss functions

1.1 Fitting a voxel grid

Binary cross entropy on logits between the optimised grid and the target occupancy grid. Optimising the grid directly means it leaves [0,1] immediately, so the loss is taken on logits and the isosurface is read from the sigmoid.

Final fitting loss 0.00043 · 30000 iterations, lr 4e-4

left: optimised, right: ground truth
optimised, turntable
ground truth, turntable

1.2 Fitting a point cloud

Chamfer distance written out with knn_points (squared distances, averaged over points and batch), no pytorch3d.loss helper.

Final fitting loss 0.00019 · 30000 iterations, lr 4e-4, 5000 points

left: optimised, right: ground truth
optimised, turntable
ground truth, turntable

1.3 Fitting a mesh

Chamfer on surface samples plus a uniform-weight laplacian smoothing term.

Final fitting loss 0.00032 · 30000 iterations, lr 4e-4, w_smooth 0.1, ico_sphere level 4

left: optimised, right: ground truth
optimised, turntable
ground truth, turntable

2. Reconstructing 3D from a single view

Voxel grid (Q2.1)

F1@0.05 = 70.85

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
F1 against distance threshold
thresholdprecisionrecallF1
0.014.604.514.53
0.0222.9423.1422.73
0.0343.7545.0843.69
0.0460.4860.9959.79
0.0572.6771.1070.85

Point cloud (Q2.2)

F1@0.05 = 78.89

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
F1 against distance threshold
thresholdprecisionrecallF1
0.017.213.924.69
0.0232.9323.2626.03
0.0356.8050.5352.07
0.0471.9168.7169.09
0.0580.8478.7478.89

Mesh (Q2.3), w_smooth = 2.0

F1@0.05 = 69.49

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
input RGB | prediction | ground-truth mesh
prediction
ground truth
F1 against distance threshold
thresholdprecisionrecallF1
0.014.534.674.59
0.0222.3824.6423.31
0.0342.0446.7843.90
0.0457.2161.7458.96
0.0568.3671.4169.49

The laplacian weight is 2.0 rather than the default 0.1. At 0.1 the surface folds over itself and stops looking like a chair, even though it scores higher on F1; section 2.4 shows both and explains why the metric prefers the folded mesh. The model reported here is the one a human would call a reconstruction.

2.3 Quantitative comparison

representationF1@0.05
Voxel grid (Q2.1)70.85
Point cloud (Q2.2)78.89
Mesh (Q2.3), w_smooth = 2.069.49

Point cloud (Q2.2) scores highest. The ordering follows how directly each representation can place surface samples where the evaluation measures them. The point cloud decoder predicts the evaluated quantity itself -- 3D points -- so nothing is lost between what it optimises (chamfer on those points) and what is scored. The mesh carries the extra constraint of a fixed sphere topology and a smoothness term, which costs accuracy on thin parts and on the gaps between chair legs that a genus-0 surface cannot open. The voxel grid is limited by its 32^3 resolution: a leg thinner than one voxel either disappears or becomes a blocky column, and the marching-cubes surface is then sampled to produce the points that get scored.

2.4 Effect of the smoothing weight

The mesh model is retrained with the laplacian weight w_smooth set to each of the values below, everything else fixed (30000 iterations, batch 32, lr 4e-4, ico-sphere level 4).

w_smoothF1@0.05
0.075.77
0.172.97
0.573.24
2.069.49
w_smooth = 0.0
w_smooth = 0.1
w_smooth = 0.5
w_smooth = 2.0

F1@0.05 across the sweep: w=0.0 gives 75.8, w=0.1 gives 73.0, w=0.5 gives 73.2, w=2.0 gives 69.5. The ends are what matter: the unsmoothed mesh (w=0.0) scores 75.8 and the most smoothed one (w=2.0) scores 69.5, a gap of 6.3 points, while the two middle settings are separated by less than the evaluation noise. The interesting part is that this is the opposite of what the renders say. The laplacian term pulls every vertex towards the centroid of its neighbours, which shrinks thin structure (chair legs, arm rails) and rounds off the concave joint between seat and back; both move surface samples away from the ground truth and cost F1 directly. What the term buys is regularity, not accuracy: at w_smooth = 0 the mesh is visibly crumpled, with spikes and self-crossings, but F1 scores points sampled on the surface, and a spike displaces only the handful of samples that land on it, so the metric hardly notices. At w_smooth = 2.0 the surface is clean and the chair is unmistakable, and the score is the worst of the four.

So on this task F1@0.05 is not a proxy for mesh quality: the best-scoring setting produces the worst-looking mesh. Which setting to ship depends on what the mesh is for. If it feeds a metric like this one, take the small weight; if a human or a downstream geometric operation sees it, the regularity is worth the points. The renders above are the same test object in each column, but the input view is drawn at random per epoch by the dataset, so the photographs differ between columns.

Interpreting the model (Q2.5)

1. Walking through the image feature

The two end points are real test images; every shape between them is decoded from a linear interpolation of their ResNet-18 features. The chair back and the legs deform continuously rather than snapping between memorised shapes, which is what a learned shape space -- as opposed to a lookup table -- looks like.

decoded shapes along the interpolation
the same walk, animated

2. Where the voxel model is confident

2477 of the 32768 voxels come out above p = 0.8 and 28152 below p = 0.2, leaving only 2139 (6.5%) in the uncertain band between. The occupancy field the network produces is sharp: the three panels at 0.2, 0.5 and 0.8 enclose almost the same solid, which is what a well-separated field looks like. This was not true of the first, under-trained model, where 0 voxels reached 0.8 and the 0.8 panel was blank (vox_confidence_320k.txt, kept next to this page); the next probe follows that thread, because an unconfident field and a fixed 0.5 level set interact badly.

input | p>0.2 | p>0.5 | p>0.8
voxelscount
voxels with p > 0.8 (confident occupied)2477
voxels with p < 0.2 (confident empty)28152
voxels in between (uncertain)2139
total voxels32768

3. The 0.5 isosurface, and why it stopped mattering

When the field is not yet sharp, the 0.5 level set cuts through a soft shell, and on some shapes it encloses nothing at all. On the first, under-trained voxel model (a third of the training budget) that is exactly what happened: 49 of the 678 test shapes came out empty at the default threshold and were scored F1 = 0.

under-trained (320k samples)
isovalueF1@0.05empty predictions
0.258.400 / 678
0.367.052 / 678
0.464.1615 / 678
0.553.9649 / 678
reported model (960k samples)
isovalueF1@0.05empty predictions
0.261.560 / 678
0.368.890 / 678
0.471.261 / 678
0.570.091 / 678

On the under-trained model the threshold was worth 13.1 points (54.0 at 0.5 against 67.1 at 0.3). On the reported model the same sweep is nearly flat: 70.1 at the default against a best of 71.3 at 0.4, and 1 shape comes out empty. So the miscalibration was a symptom of under-training rather than a property of the loss: a network that has not yet learned to be confident hedges, and hedging interacts badly with a fixed level set. Everything reported above uses the unmodified 0.5.

4. Reconstruction or retrieval?

For 100% of the test images the closest of the 200 sampled training chairs is nearer to the prediction than the chair the prediction was asked to reconstruct (mean chamfer 0.0616 against 0.1128). The point-cloud model, which scores far better, is also the one that does this less often (68%) -- the probe tracks how instance-specific a reconstruction is, and it ranks the two decoders the same way F1 does. The reading is not that the network stores a lookup table, which the interpolation above rules out, but that it outputs a plausible chair conditioned on the image, and the better the model the closer that plausible chair sits to the particular one it was shown.

quantityvalue
mean chamfer(prediction, its own ground truth)0.11279
mean chamfer(prediction, nearest TRAINING shape)0.06161
fraction where a training shape is closer than GT1.00000

3. Other architectures

Two of the three options are implemented: an implicit occupancy decoder (3.1) and a parametric surface decoder (3.2). The implicit decoder is shown in three variants, because the first one scored far below the convolutional decoder and a single number could not say why:

Implicit occupancy decoder, raw xyz input (Q3.1)

F1@0.05 = 34.19

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
F1 against distance threshold
thresholdprecisionrecallF1
0.012.681.902.06
0.0214.228.659.87
0.0328.6616.5919.29
0.0440.5323.6627.55
0.0549.2429.4534.19

Against the Q2 decoder for the same representation: -36.66 F1@0.05.

Implicit occupancy decoder, Fourier features (Q3.1)

F1@0.05 = 34.36

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
F1 against distance threshold
thresholdprecisionrecallF1
0.012.591.912.06
0.0214.358.9110.02
0.0328.9917.1119.67
0.0440.6424.0827.84
0.0549.4929.7434.36

Against the Q2 decoder for the same representation: -36.49 F1@0.05.

Implicit occupancy decoder, equal sample budget (Q3.1)

F1@0.05 = 67.92

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
F1 against distance threshold
thresholdprecisionrecallF1
0.014.544.314.35
0.0222.7422.0421.84
0.0343.6742.8942.08
0.0460.0158.1857.49
0.0571.6967.7667.92

Against the Q2 decoder for the same representation: -2.93 F1@0.05.

Parametric surface decoder (Q3.2)

F1@0.05 = 75.44

Averaged over all 678 test shapes.

input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
input RGB | prediction | ground-truth mesh
prediction
F1 against distance threshold
thresholdprecisionrecallF1
0.015.845.015.23
0.0228.4227.3327.42
0.0350.5951.5050.48
0.0465.8266.6265.71
0.0575.9075.7975.44

Against the Q2 decoder for the same representation: -3.46 F1@0.05.

The equal-budget control settles it, and not the way the first two runs suggested: at a matched 960k samples the implicit decoder lands within about three points of the convolutional one, against a gap of roughly thirty-five when it was starved. Almost all of what looked like an architectural deficit was training budget. This is worth stating plainly because the earlier reading was wrong: at batch 8 the implicit decoder's training loss appeared to plateau around iteration 4000, which I took as a capacity limit; run at batch 16 for long enough it keeps improving and closes nearly the whole gap. A flat loss curve on a small batch is not evidence of saturation.

The Fourier features are the other negative result. Lifting the coordinate to 6 octaves of sin/cos, the standard remedy for an MLP's low-frequency bias, changed nothing at the small budget (34.4 against 34.2). Whatever limits this decoder, it is not the frequency content of the coordinate encoding.

Notes