16-825 Assignment 2: Single View to 3D
Leo Xu
--load_feat) on --device cpu, as the course announcement allows.
I did not fine-tune the image encoder — only the 3D decoder is learned.
1. Exploring loss functions
For this part I am not training a network. I start from a random source shape and try to match one ground-truth chair using only the loss. If the left render looks like the right one, the loss is doing its job.
1.1 Fitting a voxel grid
I implemented voxel_loss as PyTorch binary_cross_entropy_with_logits.
I treat the source occupancy as logits and the target as a 0/1 grid from voxelize_xyz.
I used logits instead of sigmoid-then-BCE so the fit stays stable, and so I can reuse the same loss later
when I train the voxel decoder.
The default lr=4e-4 was too slow for this toy problem. I bumped it to --lr 0.05
for 8000 steps, and the loss went from about 0.97 → ~0. Because every voxel can flip on
its own, this fit can recover thin legs — something my learned 32³ decoder later fails at.
After fitting, the two chairs look basically the same to me: occupied where the GT is occupied, empty elsewhere. That was enough to convince me BCE-with-logits is a reasonable occupancy loss.
1.2 Fitting a point cloud
I wrote chamfer_loss as bidirectional nearest-neighbor squared L2, using pairwise distances
(torch.cdist) rather than a packaged Chamfer helper. For each source point I find the closest
target point, then I do the same in reverse and average both sides. If I only used one direction, the source
could collapse onto a few target points and ignore the legs; the reverse term is what forces coverage.
I get the same silhouette, back curve, and four legs. The cloud is a bit noisier than the GT samples, which I expected — Chamfer only cares about nearest-neighbor distance, not whether points clump together.
1.3 Fitting a mesh
I defined smoothness_loss as uniform Laplacian smoothing
(mesh_laplacian_smoothing in PyTorch3D). Together with Chamfer on sampled surface points,
it deforms an ico_sphere(4) toward the chair without letting faces crumple.
When I tried thinking about Chamfer alone, vertices would bunch up; the Laplacian is what keeps the mesh usable.
I ran this on CPU with lr=0.01, 2000 steps, and 1500 sample points.
Loss went from about 1.05 → 0.001.
The sphere turns into a chair-like surface with a rounded back and legs, but it stays watertight (genus 0). I cannot open a hole under the seat the way a real chair mesh can. That same topology limit shows up again when I train the mesh decoder in Q2.3.
2. Reconstructing 3D from a single view
Here I decode a frozen 512-D ResNet18 embedding of one RGB image into voxels, a point cloud, or a mesh.
Unless I say otherwise, I trained for 4000 iterations, batch size 4, Adam lr=4e-4, on CPU with
--load_feat. I report mean F1 over the 678 chair test models.
Each eval figure is input RGB | my prediction | ground-truth mesh.
2.1 Image to voxel grid
I map the 512-D feature to a 256×2×2×2 volume with a linear layer, then upsample with four conv-transpose3d blocks to a 32³ occupancy logit grid. I train with the Q1.1 BCE-with-logits loss against voxelized GT points. At eval I run sigmoid, marching cubes at 0.5, sample 1000 surface points, and compute F1 against GT mesh samples.
F1@0.05 = 29.4
eval_vox.png). F1@0.05 is the rightmost point.Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:
If a predicted GIF looks empty, occupancy never crossed the usual 0.5 marching-cubes cutoff. I measured those chairs: voxel example 50 peaks at about 0.50 and example 100 at about 0.47, so 0.5 is just barely too high for display. F1@0.05 is still computed at 0.5 (the standard occupancy threshold). For the rotating GIFs I draw the voxel isosurface at 0.3 so you can see the weak occupancy that is there. Other students’ voxel F1 is often ~50 rather than my 29.4, so their occupancy more often clears 0.5 and does not look blank.
The F1 curve is almost linear in the threshold and still only 29.4 at 0.05, so even a generous distance cannot hide missing geometry. Thick armchairs become a coarse blob (example 0). Thin furniture is under-confident: at the official 0.5 cutoff those meshes were empty, which is why F1 is low; at 0.3 you can see a coarse occupancy blob that does not match thin legs well. To me this looks like a coarse “chair mass” prior, not a view-accurate occupancy field — a resolution problem, not a wrong pose.
2.2 Image to point cloud
I use an MLP 512 → 1024 → 1024 → n_points×3, then tanh so points stay in [-1, 1].
Default is n_points=1000. I train with the Q1.2 Chamfer loss against 1000 points sampled
from the GT mesh.
F1@0.05 = 73.7
eval_point.png).Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:
The same three images that broke my voxel model now look like plausible chairs. There is no topology constraint, so I can stretch into a bench or keep an armchair compact without a 32³ bin size. What still fails is local: legs and thin rails are noisy, and the cloud is more a filled volume than a clean surface. F1 is still high because a lot of points land near the GT surface.
2.3 Image to mesh
I predict a 3D offset per vertex of a shared ico_sphere(4) (~2562 vertices).
I initialize the last layer near zero so training starts from the sphere.
The loss is w_chamfer · Chamfer + w_smooth · Laplacian with w_chamfer=1,
w_smooth=0.1. I sample 1000 points from both the predicted mesh and the GT mesh.
F1@0.05 = 66.0
eval_mesh.png).Three test examples. Input RGB is a still; prediction and ground truth are 360° turntables:
Every prediction is still a deformed sphere: watertight, no hole under the seat, and “legs” that stay connected sheets or spikes. Laplacian smoothing stops the mesh from collapsing, but it also fights the high-frequency offsets I would need for slats. F1 is much better than voxels because a surface actually exists near the chair, but worse than free points because the topology is wrong.
Quantitative comparison: voxels vs points vs meshes
I trained all three with the same frozen features, iteration budget, and chair test set.
| Representation | F1@0.05 | What I see |
|---|---|---|
| Voxels 32³ | 29.4 | Many empty or blocky occupancies; the isosurface is far from GT or missing |
| Mesh (ico-sphere 4 + Laplacian) | 66.0 | A surface exists near the chair, but genus-0 cannot match holes / thin legs |
| Point cloud (1000 pts) | 73.7 | Points can sit on seat, back, and legs independently; F1 only cares about proximity |
F1@0.05 just asks whether my predicted samples lie close to the GT samples (and the other way around). Here is how I read the ranking:
- Voxels lose. A 32³ grid in a unit box has cells around 6 cm. Chair legs are thinner than that, so occupancy averages toward empty. Marching cubes then either disappears (examples 50, 100) or outputs a staircase blob (example 0). A bigger network would not fix a grid that cannot store the geometry.
- Meshes sit in the middle. I always get a connected surface, which helps recall, but an ico-sphere cannot change genus. Pulling sphere vertices into four legs without tearing is hard, so I get a lumpy “chair-ish” blob. I need Laplacian smoothing to avoid crumpled faces, and that also damps thin structure.
- Points win. There is no voxel bin and no face connectivity, so the MLP can drop 1000 independent xyz’s wherever the image suggests. Chamfer does not require a manifold, so a noisy but well-located cloud still scores well. The images that were empty in voxel space become elongated benches or compact armchairs as point clouds.
So I think the ranking is about the representation, not “I undertrained the mesh.” Q2.4 backs that up: even my weak 250-point decoder (F1 55) still beats 32³ voxels (29).
2.4 Hyperparameter: number of predicted points
I varied n_points for the image-to-point decoder and kept everything else the same:
MLP depth/width (except the last layer), 4000 iterations, batch size 4, Chamfer, frozen features, CPU.
I wanted to isolate how densely I am allowed to cover the surface, not mix in optimizer or encoder changes.
| n_points | F1@0.05 (test) | Qualitative |
|---|---|---|
| 250 | 55.0 | Correct pose, sparse cloud, missing thin legs |
| 1000 (Q2.2) | 73.7 | Denser seat/back, legs more often present |
F1@0.05 is really a coverage score. With only 250 points I cannot put a sample on every thin part of a chair even if placement is perfect, so recall drops and F1 falls by about 19 points. The pair above is the same RGB: 250 points look like a sparse blob to me, while 1000 points actually look like furniture.
For this decoder, point count matters a lot up to at least 1000 — more points buy recall on legs and rims. Adding points is still cheaper than switching to voxels: my 250 unconstrained points (F1 55) already beat a 32³ occupancy grid (F1 29) and land next to the parametric UV fold (F1 54, Q3.2). So Q2.1 is limited by discretization, and a weak point model is limited by sampling density — not because “points are magic.” I would not expect F1 to keep climbing linearly past a few thousand points; extra samples would mostly densify the seat, which is already close.
2.5 Interpreting the model
Looking only at final RGB → 3D renders does not tell me how the decoder uses the 512-D feature. Because the ResNet is frozen, I am really learning a function f(feature) → 3D. I made three visualizations to poke at that: interpolation, occupancy slices, and point error. (The purple “Input RGB” strips are empty on purpose — these probes used stored features, not pixels.)
1. Linear interpolation in feature space
I take two test-chair embeddings z₀, z₁ and decode
(1−α) z₀ + α z₁ for α ∈ {0, 0.2, …, 1} with my Q2.2 point decoder.
Intermediate α do not smear two unrelated blobs together. The cloud stays chair-like and changes smoothly (extent / thickness). That tells me the decoder learned a locally linear family of chairs over the feature manifold: nearby ImageNet embeddings map to nearby chairs. If it were just memorizing training IDs, I would expect a jump or a collapse in the middle. I do not see that, so it behaves more like a smooth shape generator.
2. Occupancy slices of the voxel decoder
For one test image I take mid-plane slices of σ(logits) along z, y, and x. Bright means predicted occupied.
This is not random noise or a solid cube. The L-shaped occupancy is a coarse chair: filled seat and back, almost no legs. Together with F1 ≈ 29 and the empty marching-cubes examples in Q2.1, I think the failure is missing thin structure on a coarse grid, not a wrong global pose. If I only looked at the isosurface I might think the network learned nothing; the slices show it did learn a low-frequency occupancy prior.
3. Point-to-GT residual
I color each predicted point by L2 distance to its nearest GT sample (blue = close, red = far).
Errors sit on extremities (legs, corners, underside), not on the seat bulk. Same story as the voxel slices: a strong coarse prior and weaker high-frequency detail, but enough samples that F1 still looks good. So a high F1 does not mean I recovered a perfect silhouette — it means most points are on the right object.
3. Other architectures / datasets
The assignment asked for at least one of 3.1 / 3.2 / 3.3. I implemented all three for extra credit.
3.1 Implicit occupancy network
I followed Occupancy Networks: an MLP takes concat(image feature, xyz) and outputs one
occupancy logit per query. During train and eval I query a 32×32×32 meshgrid in [-1, 1]³
and use the same BCE-with-logits voxel loss. Occupancy is a function of location, so in principle I could
query a finer grid at test time without changing weights.
F1@0.05 ≈ 1.2 after 3000 CPU steps.
The predicted GIFs below are blank on purpose. Same three test chairs as Q2.1, same marching-cubes cutoff of 0.5 that eval uses. Predicted occupancy never crosses that threshold (max ≈ 0.24 / 0.04 / 0.10 on examples 0 / 50 / 100), so cubify returns an empty mesh and there is nothing to rotate. I am not lowering the cutoff for display: a weaker isosurface is still not a chair, and F1@0.05 is computed at 0.5 anyway, which is why it stays ≈ 1.2.
This is not a missing figure. The decoder ran; occupancy just stayed too small to extract a surface. Most GT voxels are empty, so BCE-with-logits is happy to push every logit negative. A pointwise MLP also has no 3D locality (unlike my conv-transpose voxel decoder), and with a frozen image vector it can ignore the photo and still get a low loss by predicting “empty everywhere.” Same frozen features on the voxel CNN still fill thick chairs (F1 29.4), so the failure is the implicit MLP, not the encoder. 3000 CPU steps is also a weak setting for a coordinate network.
I am reporting this as a negative result — I did implement the implicit interface (feature + xyz → occupancy on a 32³ grid in [-1, 1]³), but this simple MLP is not a drop-in upgrade over the voxel CNN here.
3.2 Parametric network (AtlasNet / FoldingNet)
I followed AtlasNet / FoldingNet: an MLP maps concat(image feature, uv) to xyz, with
uv in [0, 1]². At train time I sample random uv; at eval I use a regular UV grid so the output is one
folded surface patch. Loss is Chamfer vs GT samples, same as my free point decoder.
F1@0.05 ≈ 54.0
One UV square is just a disk. Folding it can make a seat-and-back sheet, but I cannot cover a chair with holes (under the seat, between legs) as freely as 1000 unconstrained points (F1 73.7). The fold looks more “surface-like” than my Q2.2 cloud, which is the AtlasNet idea, but recall on legs is worse. F1 ≈ 54 sits next to my 250-point MLP (55.0): both are limited surface samplings, not a dense free cloud. Multiple charts would be the natural next step; a single patch is the minimal parametric function the assignment asked for.
3.3 Extended dataset (chair / car / plane)
I pointed dataset_location.py at the full dataset and split_3c.json
(plane 02691156, car 02958343, chair 03001627).
I trained the point-cloud decoder with the same recipe as Q2.2 (frozen features, 1000 points,
Chamfer, 4000 steps, batch 4, CPU) and saved checkpoint_point_3c.pth.
I did not overwrite the one-class chair checkpoint checkpoint_point.pth.
I evaluated both models on the same 678 chair test models from the 3-class split
(--synset_id 03001627), following the hint: keep the chair test set fixed and only change
what the decoder saw during training.
| Training data | Chair test F1@0.05 | F1 plot |
|---|---|---|
| One class (chair only) | 73.3 | eval_point_1c_chair.png |
| Three classes (plane, car, chair) | 71.9 | eval_point_3c_chair.png |
Quantitative comparison
Chair F1 drops by about 1.4 points (73.3 → 71.9). The drop is real but small. My take:
- The encoder is frozen, so extra classes cannot improve the image embedding. They only add competing 3D targets for one MLP. This is not the usual “more data helps the backbone” story.
- I kept 4000 steps. The 3-class loader has ~128k train views vs ~61k chairs, so each step is less likely to be a chair and I get fewer chair-specific updates.
- Chairs are still about 47% of 3-class train views, so the chair prior is not erased — that is why I only lose 1.4 points instead of collapsing.
(73.3 on this chair split vs 73.7 in Q2.2 is the same one-class weights; the 0.4 gap is just eval bookkeeping.)
Qualitative comparison: consistency vs diversity
Each block is the same RGB. I show rotating one-class vs three-class predictions.
On consistency: one-class outputs stay tighter on the object (especially bench length and the office-chair seat). Three-class clouds have more stray points off the surface. I read that as the decoder being a bit less sure where a chair’s surface is, because the same 512-D space also has to map to cars and planes.
On diversity: I did not see useful extra chair variety (for example new leg styles borrowed from planes or cars). The 3-class model just looks like a noisier chair prior, not a richer family of chairs. Training saw more categories, but on chair test images that shows up as a weaker chair mapping, not as more interesting modes.
Bottom line: with frozen features and a shared point decoder, multi-class training mostly dilutes the chair prior. If I had fine-tuned the ResNet, extra classes might help the embedding separate furniture from vehicles; here they cannot. In this setting, one-class training is the better chair reconstructor on both F1 and visual consistency.