Assignment 2 for 16-825 (Single View to 3D)

Kartik Sharma
Andrew ID: kartiksh

Experimental setup

1.1 Fitting a voxel grid (4 points)

I define voxel_loss as binary cross-entropy on logits (binary_cross_entropy_with_logits); the optimized grid therefore stores logits, and sigmoid(logits) gives its occupancy. Adam optimizes a randomly initialized grid for 100k iterations, until the loss reaches zero.

Optimized voxel grid next to the ground-truth voxel grid
The optimized voxel grid (left) matches the ground-truth voxel grid (right).

1.2 Fitting a point cloud (3 points)

I implement chamfer_loss from scratch using only knn_points. The loss averages the squared distance from each source point to its nearest target point, averages the squared distance from each target point to its nearest source point, and sums the two terms. Adam optimizes 5000 points, initialized from a Gaussian, toward 5000 points sampled from the ground-truth chair mesh.

Optimized point cloud next to the ground-truth point cloud
The optimized point cloud (left) covers the same surface as the ground-truth point cloud (right).

1.3 Fitting a mesh (3 points)

I define smoothness_loss as the uniform Laplacian smoothing loss (mesh_laplacian_smoothing). Adam deforms an ico_sphere(4) by minimizing the Chamfer loss between points sampled from the two meshes plus 0.1 × the Laplacian term.

Optimized mesh next to the ground-truth mesh
The deformed ico-sphere (left) reproduces the ground-truth mesh (right).

2.1 Image to voxel grid (20 points)

Decoder. ResNet18 feature → linear layer → 256-channel 4³ grid → three upsampling blocks (transposed convolution + convolution) to 32³ → one logit per voxel. Loss. Binary cross-entropy on logits with occupied voxels weighted ×2 (pos_weight = 2). Occupied voxels make up only about 3% of a chair grid, and without the weight the predicted occupancy of thin legs falls below the 0.5 threshold (see 2.4). Training. 30k iterations, batch size 32, Adam, learning rate 4e-4 with cosine decay.

Input image, predicted voxels and ground-truth mesh for test chair 10Input image, predicted voxels and ground-truth mesh for test chair 250Input image, predicted voxels and ground-truth mesh for test chair 500

Test chairs 10, 250 and 500. Columns: input RGB, predicted voxel grid, ground-truth mesh.

F1@0.05 = 83.8 (± 0.3 across views and point-sampling seeds).

F1 score against distance threshold for the voxel model
eval_vox.png: F1 score against distance threshold, produced by eval_model.py.

Best and worst test chairs (ranked by F1@0.05):

Best voxel reconstruction
Best: test chair 629, F1 100
Second-best voxel reconstruction
2nd best: test chair 569, F1 100
Worst voxel reconstruction
Worst: test chair 343, F1 18 (spindle backrest viewed edge-on)
Second-worst voxel reconstruction
2nd worst: test chair 95, F1 31

2.2 Image to point cloud (20 points)

Decoder. An MLP (512 → 1024 → 2048 → 5000 × 3) with LeakyReLU activations and a tanh output. Loss. Chamfer distance + 0.1 × a repulsion term, which penalizes predicted points that lie closer than 0.01 to any of their 4 nearest neighbours. With the Chamfer loss alone, about half of the 5000 points collapse into two clusters at the rear corners of the seat (see 2.4). Training. 15k iterations; I keep the checkpoint with the highest validation F1.

Input image, predicted point cloud and ground-truth mesh for test chair 10Input image, predicted point cloud and ground-truth mesh for test chair 250Input image, predicted point cloud and ground-truth mesh for test chair 500

Test chairs 10, 250 and 500. Columns: input RGB, predicted point cloud, ground-truth mesh.

F1@0.05 = 86.2 (± 0.2 across views and point-sampling seeds).

F1 score against distance threshold for the point cloud model
eval_point.png: F1 score against distance threshold, produced by eval_model.py.

Best and worst test chairs (ranked by F1@0.05):

Best point cloud reconstruction
Best: test chair 569, F1 99
Second-best point cloud reconstruction
2nd best: test chair 608, F1 99
Worst point cloud reconstruction
Worst: test chair 441, F1 16 (boxy upholstered chair)
Second-worst point cloud reconstruction
2nd worst: test chair 491, F1 16

2.3 Image to mesh (20 points)

Decoder. An MLP (512 → 1024 → 2048 → 2562 × 3) predicts per-vertex offsets for an ico_sphere(4) template of radius 0.5; training starts from the undeformed template. Loss. Chamfer distance + 0.3 × Laplacian smoothing + 0.03 × normal consistency + 0.5 × edge length.

Input image, predicted mesh and ground-truth mesh for test chair 10Input image, predicted mesh and ground-truth mesh for test chair 250Input image, predicted mesh and ground-truth mesh for test chair 500

Test chairs 10, 250 and 500. Columns: input RGB, predicted mesh, ground-truth mesh.

F1@0.05 = 75.8 (± 0.4 across views and point-sampling seeds).

F1 score against distance threshold for the mesh model
eval_mesh.png: F1 score against distance threshold, produced by eval_model.py.

Best and worst test chairs (ranked by F1@0.05):

Best mesh reconstruction
Best: test chair 487, F1 100
Second-best mesh reconstruction
2nd best: test chair 629, F1 100
Worst mesh reconstruction
Worst: test chair 653, F1 13 (the ground-truth model contains floating 3D text beside the chair)
Second-worst mesh reconstruction
2nd worst: test chair 491, F1 16

Comparing voxels, point clouds and meshes

RepresentationF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05P@0.05R@0.05View std
Voxel grid13.7 ± 0.140.0 ± 0.360.3 ± 0.474.6 ± 0.483.8 ± 0.381.087.75.5
Point cloud16.7 ± 0.144.5 ± 0.265.7 ± 0.378.7 ± 0.386.2 ± 0.282.391.85.0
Mesh15.5 ± 0.240.9 ± 0.356.9 ± 0.367.8 ± 0.475.8 ± 0.475.277.09.0

Test set: 678 chairs × 10 views × 3 point-sampling seeds; values in %. P and R denote precision and recall. View std is the standard deviation of one chair’s F1@0.05 across its 10 input views, averaged over chairs.

F1 against threshold for the voxel, point cloud and mesh models
Point clouds lead at every threshold. Voxels trail meshes up to a threshold of 0.02 and overtake them from 0.03.

Point clouds achieve the highest F1@0.05 (86.2). The decoder regresses the coordinates of each point without any constraint on topology or resolution, and its Chamfer loss measures the same nearest-neighbour distances that the F1 score thresholds. This flexibility also produces the highest recall (91.8): the predicted points cover every part of the chair.

Voxels follow closely at the 0.05 threshold (83.8) but score lowest at the 0.01 threshold (13.7). Each cell of the 32³ grid is about 0.031 wide, so the grid localizes the surface only to within about half a cell. This quantization error is small relative to 0.05 but exceeds 0.01.

Meshes achieve the lowest F1@0.05 (75.8) and the lowest precision (75.2). The decoder deforms a sphere template, and deformation preserves the topology of the template (genus 0, no holes). The mesh therefore cannot open the gaps in a backrest or between the legs; it spans them with spurious surfaces that lie far from the ground truth. The regularization terms that keep the surface smooth cost a further 2.6 F1 (78.5 without them; see 2.4). Meshes are also the most sensitive to viewpoint: the F1 of one chair varies by about 9 points across its views, against about 5 for voxels and point clouds.

2.4 Effect of hyper-parameters (10 points)

For each representation, I vary the loss weight that most affects its output. Each figure shows the same three test chairs reconstructed by every model in the sweep, with the test F1 of each model below its column. The code folder lists every run and its settings in ablation_runs.md.

Mesh reconstructions for five smoothness, normal-consistency and edge-loss settings
Stronger mesh regularization produces smoother surfaces at a lower F1@0.05.

Mesh regularization trades F1 for surface quality. With Laplacian smoothing alone at weight 0.1, the model reaches the highest F1@0.05 (78.5) but produces non-smooth, self-intersecting surfaces: the weighted Laplacian term is about one tenth of the Chamfer term, so it barely constrains the vertices. Adding a normal-consistency term, which penalizes adjacent faces whose normals disagree, and an edge-length term yields smooth chair surfaces. Larger weights reduce F1 further; at a Laplacian weight of 1.0 (F1 74.5) the legs begin to merge. I select the intermediate setting (0.3 / 0.03 / 0.5; F1 75.9), the smallest weights that yield smooth surfaces.

Point cloud reconstructions for four repulsion weights
The repulsion term spreads the points over the surface: F1@0.01 roughly doubles while F1@0.05 decreases slightly.

The repulsion term trades a small amount of F1@0.05 for a more uniform point distribution. Without repulsion the model reaches the highest F1@0.05 (87.5), but about half of its points collapse into two clusters, so F1@0.01 is only 8.7. The Chamfer loss does not penalize duplicate points; the repulsion term separates them and roughly doubles F1@0.01 (16.5 to 17.3). Larger weights push points off thin structures, so F1@0.05 decreases from 86.1 at weight 0.1 to 82.5 at weight 1.0. I select weight 0.1, which costs only 1.4 F1@0.05.

Voxel reconstructions with BCE positive weight 1 and 2, and 15k vs 30k training steps
Weighting occupied voxels ×2 (pos_weight = 2) recovers legs and armrests; training for 30k instead of 15k iterations sharpens them.

Weighting occupied voxels recovers thin structures. About 97% of the voxels are empty, so plain binary cross-entropy (pos_weight = 1) favours predicting empty space. The predicted occupancy of thin structures (legs, armrests) then falls below the 0.5 threshold, and they disappear from the reconstruction (F1 79.4). Weighting occupied voxels ×2 restores them (F1 82.2), and doubling the training length to 30k iterations raises F1 to 84.1.

2.5 Interpreting the model (10 points)

Does the network reconstruct or retrieve?

Single-view reconstruction networks often perform recognition and retrieval rather than reconstruction: they recognize the input and output a shape close to one in the training set (Tatarchenko et al., What Do Single-view 3D Reconstruction Networks Learn?, CVPR 2019). I test my models for this behaviour with two nearest-neighbour experiments over the 6100 training chairs, using Chamfer distance.

  1. Oracle retrieval baseline. For each test chair, I retrieve the training chair nearest to the ground-truth shape and score it as if it were the prediction.
  2. Nearest neighbour of the prediction. For each point-cloud prediction, I retrieve the training chair nearest to the predicted shape.
Per-chair F1 of each model plotted against the F1 of the closest training chair
Oracle retrieval outperforms the networks on most chairs. Each dot is one test chair; dots below the diagonal are chairs for which the nearest training chair scores higher than the prediction of the network.
Input, point-cloud prediction, closest training chair to the prediction, and ground truth for three test chairs
Columns: input image, point-cloud prediction, training chair nearest to the prediction, ground truth. The prediction for chair 10 (top row) is nearest to a straight-backed training chair, although the ground truth has a rounded backrest.

Which image regions does the network use?

I measure occlusion sensitivity (Zeiler and Fergus, ECCV 2014): a black 24 × 24 patch slides over the input image with a stride of 8 pixels, and I record how much each 3D prediction changes (change in occupancy probability for voxels, Chamfer distance for point clouds and meshes). Bright regions mark the image regions on which the reconstruction depends.

Occlusion sensitivity heatmaps over three input images for the voxel, point-cloud and mesh models
Occlusion sensitivity for test chairs 10, 250 and 500; each map is normalized to its own maximum. All models ignore the background, and the point-cloud and mesh models respond most to individual legs.

All three models ignore the background and depend only on the object. The voxel model depends on a broad region covering the seat and the centre of the backrest. The point-cloud and mesh models respond most to individual legs: occluding one front leg changes their prediction more than occluding most of the backrest. A plausible explanation is that the leg configuration varies most across chairs and therefore discriminates best between shapes.

How does the prediction vary with viewpoint?

One cantilever chair seen from all ten views, with each model's prediction for each view
One test chair from five of its 10 input views, ordered by azimuth and including the best and worst view of the voxel model. Each prediction is rendered from the same fixed camera and labelled with its F1@0.05. The voxel prediction varies with the input view; the point-cloud and mesh predictions remain almost unchanged.

The voxel prediction depends on the input view; the point-cloud and mesh predictions barely change. For this cantilever chair, the voxel model recovers the tubular base from some views and loses parts of it from others (F1 65 to 93). The point-cloud and mesh models output nearly the same shape for every view, which agrees with the retrieval behaviour above. The mesh model cannot form the closed loop of the base, because its sphere template has no holes.

Where do the errors occur?

Point cloud coloured by distance to the GT, and GT coloured by distance to the prediction, for test chair 250

Error map for test chair 250. Middle: predicted points coloured by their distance to the ground-truth surface (blue = 0, red ≥ 0.1). Right: ground-truth points coloured by their distance to the prediction (red marks missed regions).

The errors are false-positive points, not missed regions. The predicted points with large error lie at the extremities of the shape, below the seat and around the outline of the backrest. The ground-truth points are almost entirely blue, so the prediction covers the whole surface.

3.1 Implicit network (10 points)

Network. I implement an occupancy network f(image, xyz) → logit and query it at the 32³ points of a regular grid over (−1, 1)³ to obtain the full occupancy volume. The volume shares the layout of the voxel targets, so the loss (binary cross-entropy with pos_weight = 2), evaluation and rendering of 2.1 apply unchanged.

Input image, implicit-network occupancy and ground-truth mesh for test chair 10Input image, implicit-network occupancy and ground-truth mesh for test chair 250Input image, implicit-network occupancy and ground-truth mesh for test chair 500

Test chairs 10, 250 and 500. Columns: input RGB, predicted occupancy (voxels with p > 0.5), ground-truth mesh.

The implicit network reaches F1@0.05 = 84.8 ± 0.2, 1.0 above the voxel decoder of 2.1 (83.8) under the same loss and evaluation. Its F1@0.01 is 14.0, its precision 82.9 and its recall 88.0. Precision accounts for most of the gain (82.9 against 81.0): the predictions contain fewer floating artifacts. On the three chairs of 2.1, the implicit network reconstructs a complete backrest for chair 10 and resolves the armrest of chair 250. Its limitation is over-smoothing of fine structure: the slatted backrest of chair 500 becomes a solid panel.

A likely explanation. The voxel decoder upsamples a coarse 4³ feature grid into all 32,768 voxels, so shared convolution kernels couple neighbouring regions. The implicit network evaluates one image-conditioned function independently at each query point, and the positional encoding lets that function change sharply at the surface.

3.2 Parametric network (10 points)

Network. I implement a multi-patch parametric decoder in the style of AtlasNet: f(image, patch k, uv) → xyz. Ten learned patch embeddings each select one deformed unit square, and Fourier features (4 frequencies) encode uv. A 4-block residual MLP maps each sample to a 3D point, conditioned on the global image feature through FiLM in every block. After the second block, each sample also attends to the 5 × 5 ResNet18 layer-4 feature map, which supplies local image features in addition to the global feature; the network uses no camera pose. Training samples uv uniformly at every iteration; evaluation uses a fixed, low-discrepancy (scrambled Sobol) set of 500 samples per patch. The loss and schedule match 2.2.

Input image, parametric-network point cloud and ground-truth mesh for test chair 10Input image, parametric-network point cloud and ground-truth mesh for test chair 250Input image, parametric-network point cloud and ground-truth mesh for test chair 500

Test chairs 10, 250 and 500. Columns: input RGB, parametric-network prediction, ground-truth mesh.

The parametric network reaches F1@0.05 = 84.2 ± 0.2, 2.0 below the MLP point decoder of 2.2 (86.2) under the same evaluation. Its F1@0.01 is 15.4, against 16.7. Validation F1 peaks after 4k of the 15k iterations and then declines, which indicates overfitting. Qualitatively, the points of each patch form a surface-like sheet, but thin legs remain diffuse.

The local image features did not make the parametric decoder better than the MLP baseline. A likely reason is the missing camera pose: the network must learn which of the 25 coarse feature-map cells corresponds to each 3D point, and this correspondence changes with the viewpoint. Projecting each 3D point into the image with the known camera, as pixel-aligned methods such as DISN and PIFu do, would supply this correspondence directly.

3.3 Extended dataset for training (10 points)

Setup. I retrain the point-cloud model of 2.2 with the same decoder, loss and 15k-iteration budget on the 3-class training split: 3640 planes, 3146 cars and 6100 chairs. I evaluate both models on the 3-class test split. The chairs in both splits are identical to those of the single-class dataset, so the chair column compares the two models on the same test images.

Trained onChair F1@0.05Car F1@0.05Plane F1@0.05All F1@0.05Chair F1@0.01
Chairs only86.2 ± 0.282.1 ± 0.472.9 ± 0.581.4 ± 0.316.7 ± 0.1
Chairs + cars + planes87.4 ± 0.296.4 ± 0.197.6 ± 0.192.5 ± 0.117.2 ± 0.1

3-class test set (678 chairs, 368 cars, 405 planes); all views × 3 point-sampling seeds; values in %.

A plane reconstructed by the chairs-only and the 3-class modelA car reconstructed by the chairs-only and the 3-class modelTest chair 10 reconstructed by the chairs-only and the 3-class modelTest chair 250 reconstructed by the chairs-only and the 3-class model

A plane, a car and two chairs (test chairs 10 and 250 of 2.2). Columns: input RGB, model trained on chairs only, model trained on 3 classes, ground-truth mesh.

Training on three classes raises F1@0.05 from 82.1 to 96.4 on cars and from 72.9 to 97.6 on planes. The chairs-only model has learned only the shape distribution of chairs, so it maps every input to a chair-like shape: it reconstructs the plane and the car above as diffuse, chair-sized point clouds. The 3-class model has learned the shape distributions of cars and planes and reconstructs their characteristic geometry: a flat fuselage with wings, and a long, low car body. Cars and planes also score higher than chairs, probably because their shapes vary less within the class and contain fewer thin structures.

Chairs improve by 1.2 F1@0.05 (86.2 → 87.4), although chairs make up fewer than half of the 3-class training set. A plausible explanation is that the more diverse training set regularizes the image encoder; the chair reconstructions of the two models are visually similar.