16-825 Assignment 2: Single View to 3D

Liza Dahiya

1. Exploring Loss Functions (10 pts)

Each fit uses Adam (lr 4e-4) for 10,000 steps. Each GIF shows the optimized source (left, lavender) next to the ground truth (right, gray).

1.1 Fitting a Voxel Grid (4 pts)

Binary cross-entropy on a 32³ grid of logits, starting from random values. The loss fell from 0.967 to 0.067.

Fitted voxel grid vs ground truth

1.2 Fitting a Point Cloud (3 pts)

My own chamfer loss (nearest neighbours from knn_points, squared distances averaged in both directions) on 5,000 points, starting from random positions. The loss fell from 2.151 to under 0.001.

Fitted point cloud vs ground truth

1.3 Fitting a Mesh (3 pts)

A level-4 ico-sphere deformed with chamfer loss on 5,000 sampled points plus 0.1 × Laplacian smoothing. The loss fell from 1.052 to 0.001.

Fitted mesh vs ground truth

2. Reconstructing 3D from Single View (80 pts)

All models use the ResNet18 encoder trained end to end (I did not use --load_feat), with Adam (lr 4e-4, batch size 32) for 10,000 iterations, so every comparison has the same training budget. All F1 scores are over the full 678-chair test set. The examples throughout are test chairs 0, 300 and 400. Each shows the input image, the prediction (lavender) and the ground-truth mesh (gray), with a turntable GIF next to it.

2.1 Image to Voxel Grid (20 pts)

Note on voxel labels. The provided voxel labels don't line up with the meshes for many chairs: their width and depth are swapped. Even the ground-truth voxels score only about 81 F1@0.05 against their own meshes under the provided evaluation. I therefore rebuilt the labels from each chair's mesh, in the same coordinate frame the evaluation code uses (utils_vox.voxelize_mesh). These labels score about 91, and the evaluation code is unchanged. Q1.1, Q2.1 and Q3.1 all use the rebuilt labels; with the original labels, this voxel model scored 56.9.

Voxel example 0
Voxel example 0 turntable
Voxel example 300
Voxel example 300 turntable
Voxel example 400
Voxel example 400 turntable
Voxel F1 curve
F1@0.05 = 73.5

2.2 Image to Point Cloud (20 pts)

Point example 0
Point example 0 turntable
Point example 300
Point example 300 turntable
Point example 400
Point example 400 turntable
Point cloud F1 curve
F1@0.05 = 80.3

2.3 Image to Mesh (20 pts)

Mesh example 0
Mesh example 0 turntable
Mesh example 300
Mesh example 300 turntable
Mesh example 400
Mesh example 400 turntable
Mesh F1 curve
F1@0.05 = 73.8

Comparison b/w methods

RepresentationF1@0.05
Point cloud80.3
Mesh73.8
Voxel grid73.5

Point clouds score highest because chamfer loss directly optimizes the same nearest-point distances that F1 measures, and each point can go anywhere, including thin legs. The mesh is a deformed sphere, so it can't make holes: it stretches surface across the gaps between legs, and those extra faces hurt precision. Voxels are limited by the 32³ resolution. Thin legs are only one or two cells wide, and because most cells are empty, the model tends to drop parts it is unsure about.

2.4 Analysing Hyperparameter Variations (10 pts)

I varied w_smooth, the weight of the Laplacian smoothness loss for the mesh model, and kept everything else fixed. To measure surface quality, I also counted folded edges: edges where the normals of the two adjacent faces differ by more than 90°. This catches folds and very sharp creases.

w_smooth00.115
F1@0.0576.173.869.260.2
Folded edges60.5%5.3%0.7%0.2%
Mesh predictions for each w_smooth
Input | w_smooth = 0 | 0.1 | 1 | 5 | ground truth.

Raising w_smooth always lowers F1 but makes the surface cleaner. With no smoothing, the model gets the highest F1, but the mesh is a crumpled mess: it just places vertices like a point cloud. At 5 the chair turns into a smooth blob and loses its legs. So F1 alone would pick the worst-looking mesh. For a usable mesh, a weight around 1 gives my preferred trade-off.

2.5 Interpret your Model (10 pts)

Where the point-cloud model makes mistakes. A point is red if it is beyond the 0.05 threshold, using dense samples (20k points) of the GT surface. On the left, red predicted points are more than 0.05 from the GT surface. On the right, red GT points have no prediction within 0.05.

Error coloring, chair 0
Error coloring, chair 300
Error coloring, chair 400

Ordinary seats and backs are covered well, and the errors gather where a chair is unusual. On the wooden chair (400), only the feet are missed. For the wide armchair (0), the model fills the inside of the volume with points, so 27% of them are off the surface, and the armrest tops are missed. The tub chair (300) is unlike most training chairs, and 35% of its curved wall is missed.

Interpolating image features. I blended the encoder features of two test images and decoded each blend with the point decoder.

Feature interpolation armchair to chair
Input A | decoded blends from A to B | input B.

The shape changes smoothly from the wide armchair to the taller chair, with the backrest and legs appearing gradually. This shows a gradual transition between the decoded shapes along this feature-space path.

3. Exploring Other Architectures / Datasets

3.1 Implicit Network (10 pts)

The decoder takes the image feature and a 3D point in (-1,1)³ (with sin/cos positional encoding) and outputs an occupancy value through a small residual MLP. At test time I query a 32×32×32 grid of points and score it the same way as Q2.1.

Implicit example 0
Implicit example 0 turntable
Implicit example 300
Implicit example 300 turntable
Implicit example 400
Implicit example 400 turntable
Implicit F1 curve
F1@0.05 = 56.5voxel decoder from 2.1: 73.5

The implicit network does worse than the voxel decoder. Every point is predicted on its own from one global feature, so the outputs are smoother and blobbier, and the model is more cautious: 69 of 678 test chairs come out completely empty, versus 2 for the voxel decoder. One advantage is that resolution becomes a test-time choice. Querying the same model on a 128³ grid raises F1 to 63.9.

3.2 Parametric Network (10 pts)

This is an AtlasNet-style decoder: 25 small MLPs ("patches"). Each maps the image feature plus a random 2D point in [0,1]² to a 3D point. It is trained with chamfer loss on 1,000 points, the same as Q2.2.

Parametric example 0
Parametric example 0 turntable
Parametric example 300
Parametric example 300 turntable
Parametric example 400
Parametric example 400 turntable
Parametric F1 curve
F1@0.05 = 76.0point decoder from 2.2: 80.3

The parametric decoder spreads its points more evenly over the surface, but scores a bit lower than the plain MLP decoder. Each patch is a continuous surface, so with only 1,000 random samples some thin parts like legs get few points. Its advantage is that it can output any number of points: sampling 10,000 gives a much denser cloud from the same model.

3.3 Extended Dataset for Training (10 pts)

I trained the point-cloud model from 2.2 on all three classes (chair, car, airplane) with the same settings, and compared it with the chair-only model on the same test images.

Trained onChairCarAirplaneAll 3 classes
Chairs only80.066.667.973.2
Chair + car + airplane80.493.596.388.1
F1 vs threshold per class
F1 vs threshold for each test class.

Each image: input | chair-only model | 3-class model | ground truth.

Chair example
Chair example
Car example
Airplane example

On chairs, training on three classes makes almost no difference (80.0 vs 80.4). The chair-only model has never seen cars or planes and outputs a chair-shaped cloud for them, so it scores poorly there. The 3-class model does very well on cars and planes, even better than on chairs, because those shapes vary much less within their class than chairs do. Qualitatively, the 3-class model's chair predictions are a little more generic and less varied from chair to chair. But it never confused a chair with a car or a plane.