Assignment 2: Single View to 3D

Name: Sumanth Yegurla · Andrew ID: syegurla

Setup used for all single-view models (Sections 2 and 3): trained on the single-class chair split (6,100 training models), one V100 GPU on PSC Bridges-2, Adam with learning rate 4e-4, batch size 32, 100,000 iterations, checkpoint every 2,000 iterations (the evaluated checkpoint is from iteration 98,000). Evaluation uses eval_model.py on all 678 test chairs.

Use of --load_feat: all models were trained and evaluated with --load_feat, i.e. on the provided pretrained ResNet-18 image features rather than training the image encoder end to end.

I started this assignment on my laptop (an M2 Pro), and training there had to run on the CPU because Apple's GPU backend doesn't support 3D transposed convolutions or PyTorch3D's nearest-neighbor search. Using the precomputed features made that workable. I kept the same setting when I moved the final runs to PSC, so everything on this page uses the same inputs.

1. Exploring Loss Functions

Each fit optimizes one shape directly (no network) to match the first chair of the training set, for 100,000 iterations with learning rate 4e-4. Turntables show the optimized shape (left, blue) next to the target (right, yellow).

1.1 Fitting a Voxel Grid

Binary cross-entropy between the optimized grid and the 32³ target occupancy, computed on logits (binary_cross_entropy_with_logits); the surface is extracted from the sigmoid of the grid with marching cubes.

Optimized voxel grid (left) next to the target voxel grid (right), 360-degree turntable

1.2 Fitting a Point Cloud

Chamfer distance implemented from scratch with knn_points: for every point, the squared distance to its nearest neighbor in the other cloud, averaged in both directions and summed. 5,000 points.

Optimized point cloud (left) next to the target point cloud (right), 360-degree turntable

1.3 Fitting a Mesh

Starting from a level-4 icosphere, the per-vertex offsets are optimized with chamfer distance between 5,000 points sampled from each surface, plus a uniform Laplacian smoothing term (w_chamfer = 1.0, w_smooth = 0.1), implemented directly from the mesh's edge list: for each vertex, the distance to the average of its neighbors.

Optimized mesh (left) next to the target mesh (right), 360-degree turntable

Growing a chair out of a sphere is harder than it looks. To match the chair, the mesh has to grow four long, thin legs, which means some vertices travel a long way, and the smoothing term pushes back against thin spikes. It takes a lot of iterations: after 5,000 the mesh was still spiky and folded, and only by 100,000 had it split into four legs. A sphere also can't change its topology, so a mesh fit this way could never have real holes, like the gaps in a slatted backrest. The voxel grid and the point cloud don't have either problem, and both match the target closely.

2. Reconstructing 3D from a Single View

For each model: three test chairs, each shown as the input image, a turntable of the prediction and a turntable of the ground-truth mesh (same camera and lighting for both), with the F1 curve over distance thresholds alongside.

2.1 Image to Voxel Grid

F1@0.05 = 68.1 · Decoder: a linear layer maps the 512-d feature to a 256×2×2×2 volume, then four 3D transposed convolutions (with batch norm and ReLU between them) upsample it to 32×32×32 occupancy logits. Loss: binary cross-entropy on logits.

Input image, test chair 0
Input (test chair 0)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 200
Input (test chair 200)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 600
Input (test chair 600)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Voxel model: F1 against distance threshold

2.2 Image to Point Cloud

F1@0.05 = 74.7 · Decoder: an MLP (512 → 1024 → 2048 → 1000×3) that outputs 1,000 points directly. Loss: chamfer distance to 1,000 points sampled from the ground-truth surface.

Input image, test chair 0
Input (test chair 0)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 200
Input (test chair 200)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 600
Input (test chair 600)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Point cloud model: F1 against distance threshold

2.3 Image to Mesh

F1@0.05 = 69.3 · Decoder: an MLP (512 → 1024 → 2048 → 2562×3) that predicts an offset for every vertex of a level-4 icosphere; the last layer is initialized near zero so training starts from the sphere. Loss: chamfer distance on 1,000 surface samples plus Laplacian smoothing (w_smooth = 0.1).

Input image, test chair 0
Input (test chair 0)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 200
Input (test chair 200)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 600
Input (test chair 600)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Mesh model: F1 against distance threshold

Comparing the Three Representations

RepresentationF1@0.05
Voxel grid (2.1)68.1
Point cloud (2.2)74.7
Mesh (2.3)69.3

The point cloud comes out on top (74.7), with the mesh (69.3) and the voxel grid (68.1) close to each other. I think this mostly comes down to how directly each representation matches the metric. The point decoder outputs surface points and is trained with chamfer distance, which uses the same nearest-point distances that F1 is computed from, so it's optimizing almost exactly what gets scored.

The voxel grid is held back by its 32³ resolution. Any leg or slat thinner than a voxel either vanishes or becomes a blocky column (you can see this on test chair 200), and the surface only gets recovered afterwards through marching cubes.

The mesh has a different problem. It's one deformed sphere with a fixed number of vertices, so growing long, thin legs is hard, and it can't make holes at all, like the openings in a framed backrest or the loops formed by a folding chair's crossed legs. It ends up stretching surface across those gaps, which puts points where the real chair has none. And since F1 only looks at how close sampled points are to the true surface, it barely tells a crumpled mesh apart from a clean one (more on that in 2.4).

2.4 Effect of a Hyperparameter: Smoothness Weight

The mesh model is retrained with the Laplacian smoothness weight w_smooth set to 0.0, 0.1 (the model in 2.3) and 1.0, with everything else unchanged.

w_smoothF1@0.05
0.071.2
0.170.4
1.067.3

All three models were evaluated in the same run. The w_smooth = 0.1 model is the same checkpoint as in 2.3, where it scored 69.3: each evaluation draws random input views and random surface samples, so scores vary by about ±1 point between runs.

The same three test chairs and input images, predicted by the model trained with each weight:

Input image, test chair 0
Input (test chair 0)
w_smooth = 0.0; w_smooth = 0.1; w_smooth = 1.0; Ground truth (360-degree turntables, rotating together)
w_smooth = 0.0w_smooth = 0.1w_smooth = 1.0Ground truth
Input image, test chair 200
Input (test chair 200)
w_smooth = 0.0; w_smooth = 0.1; w_smooth = 1.0; Ground truth (360-degree turntables, rotating together)
w_smooth = 0.0w_smooth = 0.1w_smooth = 1.0Ground truth
Input image, test chair 600
Input (test chair 600)
w_smooth = 0.0; w_smooth = 0.1; w_smooth = 1.0; Ground truth (360-degree turntables, rotating together)
w_smooth = 0.0w_smooth = 0.1w_smooth = 1.0Ground truth

The numbers and the pictures tell opposite stories here. F1 drops steadily as the weight goes up (71.2 → 70.4 → 67.3), but the meshes look better and better. With no smoothing, the surface is crumpled, full of small folds and faces that cross through each other. At 1.0 it's smooth and clean, but the thin parts shrink: legs taper off to points and edges get pulled inward.

That makes sense given what the Laplacian term does. It pulls each vertex toward the average of its neighbors, which irons out folds but also drags thin structures inward, away from the true surface. F1 only cares about how far sampled points are from the ground truth, and a small fold barely moves any of them, so the crumpled mesh ends up scoring higher than the clean one.

2.5 Interpreting the Model

Each predicted point of the point-cloud model is colored by its distance to the nearest point on the ground-truth surface, using one shared color scale (blue = on the surface, red = 0.1 or more away).

Input, prediction colored by error, and ground truth for four test chairs

The error maps make it pretty clear where the model can be trusted. Big, well-lit parts of ordinary chairs, like backrests and seats, are mostly blue. The errors pile up on thin or unusual parts, such as legs and slanted frames. The folding chair (test chair 200) is a good example: the model gives it the legs of a normal upright chair. When the input doesn't look typical, it seems to fall back on an average chair shape.

Test chair 400 is the worst case. Its input is almost completely black (mean brightness 2.2 out of 255), and the prediction goes wrong in two ways. The four legs collapse into a single column of points, and the whole chair ends up at the wrong height. Its vertical extent (−0.38 to 0.28) is nearly the same as for the other chairs I looked at (about −0.35 to 0.28), while the real chair is taller (−0.24 to 0.48). With so little to see, the model basically guesses a chair of typical size.

Test chair 400: dark input, brightened input, error map and ground truth
Test chair 400: error-colored prediction (left) and ground truth (right), 360-degree turntable
Prediction (left) and ground truth (right), rendered with the same camera, so the vertical offset between them is real.

To check whether the darkness was really the cause, I fed the same model brightened versions of the image. I computed their features with the same pretrained ResNet-18 and preprocessing that produced the provided features; recomputing the feature of the original image reproduces the provided one exactly (cosine similarity 1.000), so the comparison is fair.

Test chair 400 at 1x, 2x, 4x and 8x brightness with the resulting error maps and an error-vs-brightness plot

Vertical extent of the predicted points at each brightness (y is up; bottom and top are the 2nd and 98th percentiles):

InputBottomMiddleTop
Original (dark)−0.38−0.080.28
2× brighter−0.33−0.020.32
4× brighter−0.280.020.35
8× brighter−0.260.000.31
True chair−0.240.080.48

At 4× brightness the mean error drops from 0.100 to 0.035, F1 jumps from 16.2 to 78.9, and the bottom and middle of the prediction get close to the true values (−0.28 vs −0.24, and 0.02 vs 0.08). The top never gets past 0.35, though (the real top is at 0.48), so even with a clear image the model underestimates this unusually tall backrest, which looks like the same pull toward typical chairs. Going to 8× makes the error go up a little again (0.038). My guess is that the colors get so saturated they stop looking like the training images. This is only one chair, so I'd read it as a case study rather than a result about the whole test set.

3. Exploring Other Architectures / Datasets

3.1 Implicit Network

F1@0.05 = 48.0 · An MLP (4 layers, 256 hidden units) takes the 512-d image feature together with a 3D coordinate and predicts an occupancy logit for that point. The coordinate is passed through a sinusoidal positional encoding (6 frequencies, 39 dimensions). At train and test time it is queried on a 32×32×32 grid over [−1, 1]³, giving a volume trained with the same binary cross-entropy as the voxel model and evaluated the same way (marching cubes at probability 0.5).

Input image, test chair 0
Input (test chair 0)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 200
Input (test chair 200)
Prediction (nearly empty at the 0.5 cutoff); Ground truth (360-degree turntables, rotating together)
Prediction (nearly empty at the 0.5 cutoff)Ground truth
Input image, test chair 600
Input (test chair 600)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Implicit model: F1 against distance threshold

Some predictions are nearly empty. Drawing the same predicted occupancy at lower probability cutoffs shows the chair is present but below 0.5:

Implicit model predictions for three test chairs drawn at cutoffs 0.5, 0.3 and 0.15, next to the ground truth

The implicit model scores 48.0, a lot lower than the voxel model (68.1), even though both predict occupancy on the same 32³ grid with the same loss. Positional encoding turned out to be essential. In an early short run without it, the network learned to predict empty space everywhere (F1 around 3). A plain MLP on raw xyz coordinates has a hard time representing a sharp boundary between inside and outside, and since most of the grid is empty anyway, predicting "empty" everywhere was the easy way out. With the encoding it learns real chair shapes, scoring 95.6 on test chair 600.

The main problem that's left is confidence. For test chairs 200 and 400, the highest probability anywhere in the grid is only about 0.55, and just 15–26 grid points make it past 0.5. The model does know roughly where the chair is, since points inside the true chair get about 6× the probability of points outside it (0.13 vs 0.02). It just never commits. Because evaluation uses a fixed 0.5 cutoff, those predictions turn into nearly empty shapes that score close to zero. Dropping the cutoff to 0.3 brings back a recognizable chair (figure above), while for confident predictions like test chair 600 the cutoff hardly matters.

3.2 Parametric Network

F1@0.05 = 72.1 · An MLP (4 layers, 256 hidden units) maps the 512-d image feature together with a 2D point (u, v) ∈ [−1, 1]² to a 3D point. On every forward pass 1,000 new (u, v) points are sampled uniformly, so the network learns a continuous map from the square onto the object's surface. Loss: chamfer distance, as for the point cloud.

Input image, test chair 0
Input (test chair 0)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 200
Input (test chair 200)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, test chair 600
Input (test chair 600)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Parametric model: F1 against distance threshold

The parametric model gets 72.1, close to the point-cloud model's 74.7 even though it has about 30 times fewer parameters (0.26M vs 8.8M). That's because a single small network is shared by every output point, whereas the point decoder needs a separate output for each one. The catch is that it has to learn one continuous map from the square onto the chair's surface. In the examples above, that spreads the points more evenly over the surface, while the point decoder tends to bunch points into dense clusters (compare with the turntables in 2.2). But one square has to be stretched and folded to cover separate parts like the four legs, which is probably why it ends up a little behind. A nice side effect of sampling (u, v) fresh on every forward pass is that it can output as many points as you want at test time.

3.3 Extended Dataset for Training

The point-cloud model from 2.2 is retrained, with identical settings, on the three-class split (split_3c.json: chairs, cars and airplanes). Both models are evaluated on the same 678 chair test samples, with the same input views. dataset_location.py selects the three-class dataset and split_3c.json when the environment variable R2N2_FULL=1 is set (single-class by default), and R2N2_SPLIT=split_03001627.json evaluates on the chair test split of the same data.

ModelTest setF1@0.05
Trained on chairs only (2.2)678 test chairs74.7
Trained on three classesthe same 678 test chairs76.2
Trained on three classesfull three-class test set (1,451)86.1

The same chairs, predicted by each model from the same input image:

Input image, test chair 0
Input (test chair 0)
Trained on chairs; Trained on 3 classes; Ground truth (360-degree turntables, rotating together)
Trained on chairsTrained on 3 classesGround truth
Input image, test chair 200
Input (test chair 200)
Trained on chairs; Trained on 3 classes; Ground truth (360-degree turntables, rotating together)
Trained on chairsTrained on 3 classesGround truth
Input image, test chair 600
Input (test chair 600)
Trained on chairs; Trained on 3 classes; Ground truth (360-degree turntables, rotating together)
Trained on chairsTrained on 3 classesGround truth

The three-class model on the other classes:

Input image, airplane
Input (airplane)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, car
Input (car)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Input image, chair from the three-class test set
Input (chair)
Prediction; Ground truth (360-degree turntables, rotating together)
PredictionGround truth
Three-class model on the chair test set: F1 against distance threshold
Three-class model, chair test set
Three-class model on the three-class test set: F1 against distance threshold
Three-class model, three-class test set

Adding cars and airplanes to training didn't hurt the chairs. On the same 678 test chairs, the three-class model scores 76.2 and the chair-only model 74.7. That difference is within the ±1–2 points that scores move between evaluation runs, so I'd call it a tie rather than an improvement.

On the full three-class test set the model scores 86.1. Breaking that down by class, the 773 airplanes and cars average 94.6, while the 678 chairs average 76.4. So chairs are by far the hardest class. Their shapes vary a lot more (thin legs, slats, open backs), while cars and airplanes are more compact and look more alike.

Looking at the turntables, the two models predict almost the same shape for each chair. The three-class model's points are a bit more scattered and its legs a little less crisp, and neither one gets the slanted legs of the folding chair (test chair 200). The outputs also stay just as varied from chair to chair: the wide armchair still looks clearly different from the narrow upright chairs. So at least in these examples, sharing the model with two other classes didn't make its chair predictions more uniform.