16-825 Assignment 2: Single View to 3D

Zijin Ye (Andrew ID: zijiny)

Losses, decoders and analysis for voxel, point cloud, mesh and implicit reconstruction on the R2N2 chair dataset.

Compute: all models were trained on a local RTX 5070 Laptop GPU (8 GB) with the full image pipeline (ResNet18 encoder fine-tuned end to end). I did not use --load_feat or CPU training for any part, including the extra-credit models. Every model uses 4000 iterations at batch size 32 (batch size 16 for the implicit model), lr 4e-4, Adam. Chair evaluation covers all 678 test images; the three-class evaluation in 3.3 covers 1,451.

1. Exploring loss functions

1.1 Fitting a voxel grid

The loss is binary cross entropy on logits (F.binary_cross_entropy_with_logits), which applies the sigmoid inside the loss for numerical stability. The optimized grid is shown on the left, ground truth on the right.

Optimized voxel grid next to ground truth voxel grid, rotating

1.2 Fitting a point cloud

Chamfer loss is written from scratch using only knn_points. For every point it takes the squared distance to the nearest neighbour in the other cloud, averages each direction, and sums the two directions.

Optimized point cloud next to ground truth point cloud, rotating

1.3 Fitting a mesh

The smoothness loss is the mean norm of the uniform Laplacian L·V, where each vertex is compared with the mean of its neighbours. It is combined with Chamfer on 5000 points sampled from both meshes (lr 4e-4, w_smooth 0.1, 10k steps, starting from an ico-sphere).

Optimized mesh next to ground truth mesh, rotating
Blobby mesh fit with no separated legs
A setting that failed (lr 1e-3, w_smooth 2.0): the strong Laplacian term stops the sphere from growing thin legs.

2. Reconstructing 3D from a single view

Every model shares a ResNet18 encoder that produces a 512-d global feature per image. Each panel below shows the input RGB | prediction | ground-truth mesh. The same three test images are used for every representation.

2.1 Image to voxel grid, F1@0.05 = 47.2

Decoder: a fully connected layer maps the feature to a 256×4³ grid, followed by three ConvTranspose3d + BatchNorm + ReLU stages (4³→8³→16³→32³) and a final 3×3×3 Conv3d that outputs occupancy logits.

Voxel example 1Voxel example 2Voxel example 3
F1 vs threshold for voxels

Upper bound: feeding the ground-truth 32³ voxels through the same evaluation gives F1@0.05 = 77.2. The voxel model therefore reaches about 61% of what a 32³ grid can represent.

2.2 Image to point cloud, F1@0.05 = 78.7

Decoder: an MLP 512 → 1024 → 2048 → 1000×3 with LeakyReLU, trained with Chamfer loss against 1000 points sampled from the GT mesh.

Point example 1Point example 2Point example 3
F1 vs threshold for point clouds

2.3 Image to mesh, F1@0.05 = 70.7

Decoder: an MLP 512 → 1024 → 2048 → 2562×3 followed by tanh. It predicts a per-vertex offset of a level-4 ico-sphere. Training uses Chamfer on sampled points plus 0.1 × Laplacian smoothness.

Mesh example 1Mesh example 2Mesh example 3
F1 vs threshold for meshes

Comparison: voxel vs point vs mesh

RepresentationF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05
Voxel 32³3.014.828.439.147.2
Point cloud3.820.444.765.578.7
Mesh4.422.843.759.670.7

2.4 Effect of the smoothness weight w_smooth

I trained the mesh model three times, identical except for w_smooth ∈ {0.01, 0.1, 1.0}. Besides F1, I measured surface quality on 100 test predictions: the mean Laplacian norm, the standard deviation of edge length, and the fraction of adjacent face pairs whose normals point in opposite directions ("folded" faces, a proxy for crumpling).

w_smoothF1@0.05 ↑Laplacian ↓Edge-length std ↓Folded faces ↓
0.0174.20.03030.07531.5%
0.1 (Q2.3)70.70.00980.04714.2%
1.069.00.00430.0233.5%
w_smooth 0.01 prediction
w_smooth = 0.01
w_smooth 0.1 prediction
w_smooth = 0.1
w_smooth 1.0 prediction
w_smooth = 1.0

Conclusion: w_smooth is a direct trade-off between the benchmark score and surface quality. Raising it 100× cuts folded faces by 9× (31.5% → 3.5%) and makes the surface 7× smoother, but costs 5.2 F1 points. F1 only measures point-to-surface distance, so it rewards the crumpled solution: folding the sphere lets vertices reach into chair parts that a smooth genus-0 surface cannot cover. The visual improvement is smaller than the numbers suggest because even the smoothest model still self-intersects. The Laplacian penalizes local roughness, not global folds. To get meshes that are both accurate and clean, a regularizer that is not local (normal consistency, edge length) or a better initial topology would help more than tuning this weight.

2.5 Interpreting the models

(a) Where do the errors happen?

Left to right: input, predicted points coloured by their distance to the GT surface (precision error), and GT points coloured by their distance to the prediction (recall error). Blue means an error near 0 and red means 0.1 or more.

Error heatmaps for the point cloud model

The recall errors (right column) are concentrated on thin, distinctive parts: the tall slotted back rest of chair 2 and the top of chair 3's back. The seat, which every chair has in about the same place, is reconstructed almost perfectly. The predicted clouds also show how the model hedges. When uncertain, it spreads points diffusely through the plausible volume, and it parks two dense clumps of points at the seat, where a point is always close to some surface. Chamfer loss makes this a cheap way to keep precision. To check that this is not a collapse, I scored the same predictions assigned to the wrong images: a constant prediction reaches only F1 55.2 and a shuffled one F1 59.5, against 78.3 for the real ones. The model clearly uses the image; the blur comes from single-view ambiguity.

Error heatmaps for the mesh model

The mesh model fails in a different way. Its precision errors (middle column, red) sit on sheets of surface folded through empty space, such as in front of the seat of chair 2 and below the seat of chair 3. The sphere has to stretch past those regions to reach the legs, which is the crumpling measured in 2.4. The recall errors again sit on the thin, unusual parts, for example the curved sled base of chair 2.

(b) Is the learned shape space smooth?

I interpolated linearly between the ResNet features of two test images (the armchair on the left and the side chair on the right) and decoded each step.

Mesh interpolation between two chairs
Point cloud interpolation between two chairs

The decoded shapes morph gradually from a wide, low armchair to a tall, narrow side chair: the back rises and the body narrows, with no sudden jumps. The dense "safe" clumps of the point model stay fixed at the seat throughout. The decoder has therefore learned a continuous shape manifold in which feature distance corresponds to geometric similarity, rather than memorizing separate shapes.

(c) How confident is the voxel model?

Columns: input, the mesh extracted at iso-levels 0.2 / 0.5 / 0.8, and the occupancy probability on the middle slice along each axis.

Voxel predictions at three iso levels and probability slices

The 0.8 column is empty for every chair: the network never reaches 0.8 occupancy on these examples. The probability slices show that even the seat and back are only moderately confident (teal), with soft edges, and thin parts are fainter still. Most cells are empty and a single view is ambiguous, so BCE rewards hedging, and thin legs and armrests fall below 0.5. They appear at iso 0.2 and vanish at 0.5, which explains the missing legs in 2.1 and much of the gap to the 77.2 upper bound. Lowering the extraction threshold or weighting occupied cells more heavily in the loss would likely recover some of them.

3.1 Implicit network

Architecture: an occupancy MLP f(feature, xyz) → logit. The image feature and the encoded query point are each projected to 256-d and added, followed by four 256-d ReLU layers and one output. At inference it is queried on a 32×32×32 meshgrid over (−1, 1)³ and reshaped into a voxel grid, so it shares the voxel loss and evaluation.

ModelF1@0.05
Implicit, raw xyz input30.8
Implicit, Fourier positional encoding (6 frequencies)36.4
Voxel CNN decoder (2.1), for reference47.2
Implicit example 1Implicit example 2Implicit example 3
Implicit prediction without positional encoding
The same test image decoded without positional encoding: a single low-frequency blob.

Analysis: my first version fed raw (x, y, z) into the MLP. It reached the same training BCE as the voxel CNN (about 0.09), yet at test time it produced a single blob (F1 30.8). An MLP on raw coordinates is biased toward very smooth functions ("spectral bias"), so it cannot represent a leg one or two voxels thick next to empty space. It hedges instead, predicting occupancy below 0.5 almost everywhere. Encoding the coordinates with sin/cos at 6 octave-spaced frequencies (as in NeRF and occupancy networks) lets the MLP represent sharp boundaries. The model then recovers the seat and back correctly and F1 rises to 36.4.

It still trails the voxel CNN (47.2) for three reasons:

The implicit form has a practical advantage the CNN lacks: resolution is chosen at query time. Training on random point subsets would remove the batch-size limit, and conditioning on local, pixel-aligned image features (as in PIFu) would likely close the gap.

3.2 Parametric network (extra credit)

Architecture (AtlasNet-style): K = 4 patches. Each patch is its own MLP fk(u, v, image feature) → (x, y, z), with layers 514 → 512 → 512 → 256 → 3, ReLU, and tanh output. During training, 250 points (u, v) are sampled uniformly from the unit square for each patch (1000 points in total), and the output is trained with the same Chamfer loss as 2.2. Same budget as 2.2: 4000 iterations on the GPU.

Parametric example 1Parametric example 2Parametric example 3

Points are coloured by the patch they come from.

F1 vs threshold for the parametric model
Model (test samples)F1@0.05
Point decoder, 2.2 (fixed 1000 points)78.7
Parametric, 1000 samples (training density)72.2
Parametric, same model, 5000 samples*79.4

* Measured on every 4th test image (170 images), with the GT also sampled at 5000 points. Denser sampling on both sides also changes the metric, so this row shows what extra samples buy you rather than a like-for-like comparison with the point decoder, which can only ever output 1000 points.

What the parametric form gives

Left to right: input, 1000 samples, 10,000 samples from the same learned maps, and each patch's regular 24×24 (u, v) grid triangulated into a mesh.

Parametric model at 1000 points, 10000 points, and as a triangulated surface

3.3 Extended dataset: training on three classes (extra credit)

Setup: the point cloud model from 2.2 was trained again, with identical architecture and hyperparameters and the same 4000 iterations at batch size 32 on the GPU. The only change was the training data: the three-class split (split_3c.json), with 12,886 training models (3,640 planes, 3,146 cars, 6,100 chairs) instead of 6,100 chairs. Because the budget is fixed, the three-class model saw each chair about half as often. I then evaluated both models on the same chair test images, and both models on the three-class test set (1,451 models).

F1@0.05Trained on chairsTrained on 3 classes
Chair test set (same 678 images)78.777.9
3-class test set, all 1,45174.286.1
  airplane (405)71.294.7
  car (368)70.191.6
  chair (678)78.478.0

Left to right: input, trained on chairs, trained on 3 classes, and ground truth. Rows show a plane, a car and a chair.

Predictions from the chair-only and three-class models on a plane, a car and a chair

To measure the consistency and diversity of the outputs, I used 50 test chairs, each seen from 4 views:

Chamfer (lower = closer)Trained on chairsTrained on 3 classesGT chairs
Diversity: between predictions for different chairs0.00380.00370.0201
View inconsistency: same chair, different views0.000790.00073–
Error: prediction vs. GT0.00390.0037–

Analysis: