16-825 Assignment 2: Single View to 3D

Leo Xu

Compute note (required by the assignment). No GPU is available on my laptop (Apple M1, macOS 12; MPS is not usable with this PyTorch build). I trained and evaluated every learned model with frozen ResNet18 features (--load_feat) on --device cpu, as the course announcement allows. I did not fine-tune the image encoder — only the 3D decoder is learned.

1. Exploring loss functions

For this part I am not training a network. I start from a random source shape and try to match one ground-truth chair using only the loss. If the left render looks like the right one, the loss is doing its job.

1.1 Fitting a voxel grid

I implemented voxel_loss as PyTorch binary_cross_entropy_with_logits. I treat the source occupancy as logits and the target as a 0/1 grid from voxelize_xyz. I used logits instead of sigmoid-then-BCE so the fit stays stable, and so I can reuse the same loss later when I train the voxel decoder.

The default lr=4e-4 was too slow for this toy problem. I bumped it to --lr 0.05 for 8000 steps, and the loss went from about 0.97 → ~0. Because every voxel can flip on its own, this fit can recover thin legs — something my learned 32³ decoder later fails at.

Optimized voxels vs ground-truth voxels
My optimized voxel grid (left) vs ground-truth voxels (right). Seat, back, and legs line up.

After fitting, the two chairs look basically the same to me: occupied where the GT is occupied, empty elsewhere. That was enough to convince me BCE-with-logits is a reasonable occupancy loss.

1.2 Fitting a point cloud

I wrote chamfer_loss as bidirectional nearest-neighbor squared L2, using pairwise distances (torch.cdist) rather than a packaged Chamfer helper. For each source point I find the closest target point, then I do the same in reverse and average both sides. If I only used one direction, the source could collapse onto a few target points and ignore the legs; the reverse term is what forces coverage.

Optimized point cloud vs ground truth
My optimized point cloud (left) vs ground-truth samples (right).

I get the same silhouette, back curve, and four legs. The cloud is a bit noisier than the GT samples, which I expected — Chamfer only cares about nearest-neighbor distance, not whether points clump together.

1.3 Fitting a mesh

I defined smoothness_loss as uniform Laplacian smoothing (mesh_laplacian_smoothing in PyTorch3D). Together with Chamfer on sampled surface points, it deforms an ico_sphere(4) toward the chair without letting faces crumple. When I tried thinking about Chamfer alone, vertices would bunch up; the Laplacian is what keeps the mesh usable.

I ran this on CPU with lr=0.01, 2000 steps, and 1500 sample points. Loss went from about 1.05 → 0.001.

Optimized mesh vs ground-truth mesh
My optimized mesh (left) vs ground-truth mesh (right).

The sphere turns into a chair-like surface with a rounded back and legs, but it stays watertight (genus 0). I cannot open a hole under the seat the way a real chair mesh can. That same topology limit shows up again when I train the mesh decoder in Q2.3.

2. Reconstructing 3D from a single view

Here I decode a frozen 512-D ResNet18 embedding of one RGB image into voxels, a point cloud, or a mesh. Unless I say otherwise, I trained for 4000 iterations, batch size 4, Adam lr=4e-4, on CPU with --load_feat. I report mean F1 over the 678 chair test models. Each eval figure is input RGB | my prediction | ground-truth mesh.

2.1 Image to voxel grid

I map the 512-D feature to a 256×2×2×2 volume with a linear layer, then upsample with four conv-transpose3d blocks to a 32³ occupancy logit grid. I train with the Q1.1 BCE-with-logits loss against voxelized GT points. At eval I run sigmoid, marching cubes at 0.5, sample 1000 surface points, and compute F1 against GT mesh samples.

F1@0.05 = 29.4

Voxel F1 vs threshold plot
F1 vs distance threshold for my voxel model (eval_vox.png). F1@0.05 is the rightmost point.

Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:

If a predicted GIF looks empty, occupancy never crossed the usual 0.5 marching-cubes cutoff. I measured those chairs: voxel example 50 peaks at about 0.50 and example 100 at about 0.47, so 0.5 is just barely too high for display. F1@0.05 is still computed at 0.5 (the standard occupancy threshold). For the rotating GIFs I draw the voxel isosurface at 0.3 so you can see the weak occupancy that is there. Other students’ voxel F1 is often ~50 rather than my 29.4, so their occupancy more often clears 0.5 and does not look blank.

Example 0 RGB
Example 0 input RGB — bulky armchair
Example 0 predicted voxels rotating
Predicted voxels (rotate)
Example 0 GT mesh rotating
Ground-truth mesh (rotate)
Example 50 RGB
Example 50 input RGB — thin bench
Example 50 predicted voxels rotating
Predicted voxels (rotate). Display isosurface 0.3; occupancy peaked just under 0.5.
Example 50 GT mesh rotating
Ground-truth mesh (rotate)
Example 100 RGB
Example 100 input RGB — thin-legged chair
Example 100 predicted voxels rotating
Predicted voxels (rotate). Display isosurface 0.3; occupancy peaked around 0.47.
Example 100 GT mesh rotating
Ground-truth mesh (rotate)

The F1 curve is almost linear in the threshold and still only 29.4 at 0.05, so even a generous distance cannot hide missing geometry. Thick armchairs become a coarse blob (example 0). Thin furniture is under-confident: at the official 0.5 cutoff those meshes were empty, which is why F1 is low; at 0.3 you can see a coarse occupancy blob that does not match thin legs well. To me this looks like a coarse “chair mass” prior, not a view-accurate occupancy field — a resolution problem, not a wrong pose.

2.2 Image to point cloud

I use an MLP 512 → 1024 → 1024 → n_points×3, then tanh so points stay in [-1, 1]. Default is n_points=1000. I train with the Q1.2 Chamfer loss against 1000 points sampled from the GT mesh.

F1@0.05 = 73.7

Point F1 vs threshold plot
F1 vs threshold for my 1000-point decoder (eval_point.png).

Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:

Example 0 RGB
Example 0 input RGB
Example 0 predicted points rotating
Predicted point cloud (rotate)
Example 0 GT mesh rotating
Ground-truth mesh (rotate)
Example 50 RGB
Example 50 input RGB
Example 50 predicted points rotating
Predicted point cloud (rotate). Elongated like the GT bench.
Example 50 GT mesh rotating
Ground-truth mesh (rotate)
Example 100 RGB
Example 100 input RGB
Example 100 predicted points rotating
Predicted point cloud (rotate). Seat/back are there; legs are a spray of points.
Example 100 GT mesh rotating
Ground-truth mesh (rotate)

The same three images that broke my voxel model now look like plausible chairs. There is no topology constraint, so I can stretch into a bench or keep an armchair compact without a 32³ bin size. What still fails is local: legs and thin rails are noisy, and the cloud is more a filled volume than a clean surface. F1 is still high because a lot of points land near the GT surface.

2.3 Image to mesh

I predict a 3D offset per vertex of a shared ico_sphere(4) (~2562 vertices). I initialize the last layer near zero so training starts from the sphere. The loss is w_chamfer · Chamfer + w_smooth · Laplacian with w_chamfer=1, w_smooth=0.1. I sample 1000 points from both the predicted mesh and the GT mesh.

F1@0.05 = 66.0

Mesh F1 vs threshold plot
F1 vs threshold for my mesh decoder (eval_mesh.png).

Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:

Example 0 RGB
Example 0 input RGB
Example 0 predicted mesh rotating
Predicted mesh (rotate)
Example 0 GT mesh rotating
Ground-truth mesh (rotate)
Example 50 RGB
Example 50 input RGB
Example 50 predicted mesh rotating
Predicted mesh (rotate). Short folded sheet, not a long slatted bench.
Example 50 GT mesh rotating
Ground-truth mesh (rotate)
Example 100 RGB
Example 100 input RGB
Example 100 predicted mesh rotating
Predicted mesh (rotate). Legs look like spiky sphere vertices.
Example 100 GT mesh rotating
Ground-truth mesh (rotate)

Every prediction is still a deformed sphere: watertight, no hole under the seat, and “legs” that stay connected sheets or spikes. Laplacian smoothing stops the mesh from collapsing, but it also fights the high-frequency offsets I would need for slats. F1 is much better than voxels because a surface actually exists near the chair, but worse than free points because the topology is wrong.

Quantitative comparison: voxels vs points vs meshes

I trained all three with the same frozen features, iteration budget, and chair test set.

RepresentationF1@0.05What I see
Voxels 32³29.4Many empty or blocky occupancies; the isosurface is far from GT or missing
Mesh (ico-sphere 4 + Laplacian)66.0A surface exists near the chair, but genus-0 cannot match holes / thin legs
Point cloud (1000 pts)73.7Points can sit on seat, back, and legs independently; F1 only cares about proximity

F1@0.05 just asks whether my predicted samples lie close to the GT samples (and the other way around). Here is how I read the ranking:

So I think the ranking is about the representation, not “I undertrained the mesh.” Q2.4 backs that up: even my weak 250-point decoder (F1 55) still beats 32³ voxels (29).

2.4 Hyperparameter: number of predicted points

I varied n_points for the image-to-point decoder and kept everything else the same: MLP depth/width (except the last layer), 4000 iterations, batch size 4, Chamfer, frozen features, CPU. I wanted to isolate how densely I am allowed to cover the surface, not mix in optimizer or encoder changes.

n_pointsF1@0.05 (test)Qualitative
25055.0Correct pose, sparse cloud, missing thin legs
1000 (Q2.2)73.7Denser seat/back, legs more often present
F1 plot n_points=250
F1 plot, n_points = 250. F1@0.05 ≈ 55.
F1 plot n_points=1000
F1 plot, n_points = 1000. F1@0.05 ≈ 73.7.
Same example with 250 points
Same test view with 250 points: a small clump, little structure.
Same example with 1000 points
Same test view with 1000 points: volume of the armchair is filled.

F1@0.05 is really a coverage score. With only 250 points I cannot put a sample on every thin part of a chair even if placement is perfect, so recall drops and F1 falls by about 19 points. The pair above is the same RGB: 250 points look like a sparse blob to me, while 1000 points actually look like furniture.

For this decoder, point count matters a lot up to at least 1000 — more points buy recall on legs and rims. Adding points is still cheaper than switching to voxels: my 250 unconstrained points (F1 55) already beat a 32³ occupancy grid (F1 29) and land next to the parametric UV fold (F1 54, Q3.2). So Q2.1 is limited by discretization, and a weak point model is limited by sampling density — not because “points are magic.” I would not expect F1 to keep climbing linearly past a few thousand points; extra samples would mostly densify the seat, which is already close.

2.5 Interpreting the model

Looking only at final RGB → 3D renders does not tell me how the decoder uses the 512-D feature. Because the ResNet is frozen, I am really learning a function f(feature) → 3D. I made three visualizations to poke at that: interpolation, occupancy slices, and point error. (The purple “Input RGB” strips are empty on purpose — these probes used stored features, not pixels.)

1. Linear interpolation in feature space

I take two test-chair embeddings z₀, z₁ and decode (1−α) z₀ + α z₁ for α ∈ {0, 0.2, …, 1} with my Q2.2 point decoder.

Feature interpolation of two chairs
Decoded clouds along a straight line in ResNet feature space.

Intermediate α do not smear two unrelated blobs together. The cloud stays chair-like and changes smoothly (extent / thickness). That tells me the decoder learned a locally linear family of chairs over the feature manifold: nearby ImageNet embeddings map to nearby chairs. If it were just memorizing training IDs, I would expect a jump or a collapse in the middle. I do not see that, so it behaves more like a smooth shape generator.

2. Occupancy slices of the voxel decoder

For one test image I take mid-plane slices of σ(logits) along z, y, and x. Bright means predicted occupied.

Voxel occupancy slices
32³ occupancy slices. Seat/back light up; legs do not.

This is not random noise or a solid cube. The L-shaped occupancy is a coarse chair: filled seat and back, almost no legs. Together with F1 ≈ 29 and the empty marching-cubes examples in Q2.1, I think the failure is missing thin structure on a coarse grid, not a wrong global pose. If I only looked at the isosurface I might think the network learned nothing; the slices show it did learn a low-frequency occupancy prior.

3. Point-to-GT residual

I color each predicted point by L2 distance to its nearest GT sample (blue = close, red = far).

Predicted points colored by GT error
Most of the cloud is blue (near the surface); red outliers sit on the periphery.

Errors sit on extremities (legs, corners, underside), not on the seat bulk. Same story as the voxel slices: a strong coarse prior and weaker high-frequency detail, but enough samples that F1 still looks good. So a high F1 does not mean I recovered a perfect silhouette — it means most points are on the right object.

3. Other architectures / datasets

The assignment asked for at least one of 3.1 / 3.2 / 3.3. I implemented all three for extra credit.

3.1 Implicit occupancy network

I followed Occupancy Networks: an MLP takes concat(image feature, xyz) and outputs one occupancy logit per query. During train and eval I query a 32×32×32 meshgrid in [-1, 1]³ and use the same BCE-with-logits voxel loss. Occupancy is a function of location, so in principle I could query a finer grid at test time without changing weights.

F1@0.05 ≈ 1.2 after 3000 CPU steps.

Implicit F1 plot
F1 vs threshold. Even at 0.05, F1 is ~1.2 — the occupancy field basically failed.

The predicted GIFs below are blank on purpose. Same three test chairs as Q2.1, same marching-cubes cutoff of 0.5 that eval uses. Predicted occupancy never crosses that threshold (max ≈ 0.24 / 0.04 / 0.10 on examples 0 / 50 / 100), so cubify returns an empty mesh and there is nothing to rotate. I am not lowering the cutoff for display: a weaker isosurface is still not a chair, and F1@0.05 is computed at 0.5 anyway, which is why it stays ≈ 1.2.

Example 0 RGB
Example 0 input RGB
Example 0 predicted implicit occupancy (blank)
Predicted occupancy mesh (blank). Peak occupancy ≈ 0.24 < 0.5.
Example 0 GT mesh rotating
Ground-truth mesh (rotate)
Example 50 RGB
Example 50 input RGB
Example 50 predicted implicit occupancy (blank)
Predicted occupancy mesh (blank). Peak occupancy ≈ 0.04 < 0.5.
Example 50 GT mesh rotating
Ground-truth mesh (rotate)
Example 100 RGB
Example 100 input RGB
Example 100 predicted implicit occupancy (blank)
Predicted occupancy mesh (blank). Peak occupancy ≈ 0.10 < 0.5.
Example 100 GT mesh rotating
Ground-truth mesh (rotate)

This is not a missing figure. The decoder ran; occupancy just stayed too small to extract a surface. Most GT voxels are empty, so BCE-with-logits is happy to push every logit negative. A pointwise MLP also has no 3D locality (unlike my conv-transpose voxel decoder), and with a frozen image vector it can ignore the photo and still get a low loss by predicting “empty everywhere.” Same frozen features on the voxel CNN still fill thick chairs (F1 29.4), so the failure is the implicit MLP, not the encoder. 3000 CPU steps is also a weak setting for a coordinate network.

I am reporting this as a negative result — I did implement the implicit interface (feature + xyz → occupancy on a 32³ grid in [-1, 1]³), but this simple MLP is not a drop-in upgrade over the voxel CNN here.

3.2 Parametric network (AtlasNet / FoldingNet)

I followed AtlasNet / FoldingNet: an MLP maps concat(image feature, uv) to xyz, with uv in [0, 1]². At train time I sample random uv; at eval I use a regular UV grid so the output is one folded surface patch. Loss is Chamfer vs GT samples, same as my free point decoder.

F1@0.05 ≈ 54.0

Parametric F1 plot
F1 vs threshold for my parametric UV decoder.
Example 0 RGB
Example 0 input RGB
Param pred 0
Predicted parametric surface (rotate)
Param GT 0
Ground-truth mesh (rotate)
Example 50 RGB
Example 50 input RGB
Param pred 50
Predicted parametric surface (rotate). Compact fold, not a long bench.
Param GT 50
Ground-truth mesh (rotate)
Example 100 RGB
Example 100 input RGB
Param pred 100
Predicted parametric surface (rotate). Legs are under-covered.
Param GT 100
Ground-truth mesh (rotate)

One UV square is just a disk. Folding it can make a seat-and-back sheet, but I cannot cover a chair with holes (under the seat, between legs) as freely as 1000 unconstrained points (F1 73.7). The fold looks more “surface-like” than my Q2.2 cloud, which is the AtlasNet idea, but recall on legs is worse. F1 ≈ 54 sits next to my 250-point MLP (55.0): both are limited surface samplings, not a dense free cloud. Multiple charts would be the natural next step; a single patch is the minimal parametric function the assignment asked for.

3.3 Extended dataset (chair / car / plane)

I pointed dataset_location.py at the full dataset and split_3c.json (plane 02691156, car 02958343, chair 03001627). I trained the point-cloud decoder with the same recipe as Q2.2 (frozen features, 1000 points, Chamfer, 4000 steps, batch 4, CPU) and saved checkpoint_point_3c.pth. I did not overwrite the one-class chair checkpoint checkpoint_point.pth.

I evaluated both models on the same 678 chair test models from the 3-class split (--synset_id 03001627), following the hint: keep the chair test set fixed and only change what the decoder saw during training.

Training dataChair test F1@0.05F1 plot
One class (chair only)73.3eval_point_1c_chair.png
Three classes (plane, car, chair)71.9eval_point_3c_chair.png
One-class chair F1 plot
One-class model, chair test only. F1@0.05 = 73.3.
Three-class chair F1 plot
Three-class model, same chair test. F1@0.05 = 71.9.

Quantitative comparison

Chair F1 drops by about 1.4 points (73.3 → 71.9). The drop is real but small. My take:

(73.3 on this chair split vs 73.7 in Q2.2 is the same one-class weights; the 0.4 gap is just eval bookkeeping.)

Qualitative comparison: consistency vs diversity

Each block is the same RGB. I show rotating one-class vs three-class predictions.

Armchair RGB
Armchair input RGB
Armchair one-class rotating
One-class prediction (rotate)
Armchair three-class rotating
Three-class prediction (rotate)
Bench RGB
Bench input RGB
Bench one-class rotating
One-class (rotate): more elongated
Bench three-class rotating
Three-class (rotate): chunkier blob
Office chair RGB
Office chair input RGB
Office chair one-class rotating
One-class (rotate): tighter seat
Office chair three-class rotating
Three-class (rotate): more scatter

On consistency: one-class outputs stay tighter on the object (especially bench length and the office-chair seat). Three-class clouds have more stray points off the surface. I read that as the decoder being a bit less sure where a chair’s surface is, because the same 512-D space also has to map to cars and planes.

On diversity: I did not see useful extra chair variety (for example new leg styles borrowed from planes or cars). The 3-class model just looks like a noisier chair prior, not a richer family of chairs. Training saw more categories, but on chair test images that shows up as a weaker chair mapping, not as more interesting modes.

Bottom line: with frozen features and a shared point decoder, multi-class training mostly dilutes the chair prior. If I had fine-tuned the ResNet, extra classes might help the embedding separate furniture from vehicles; here they cannot. In this setting, one-class training is the better chair reconstructor on both F1 and visual consistency.