16‑825 · Assignment 02 · Learning for 3D Vision
Voxel grids, point clouds and meshes reconstructed from one RGB image of an R2N2/ShapeNet chair — five decoder families evaluated on a common 678-object held-out test set.
| Method | § | F1@0.05 | Encoder |
|---|---|---|---|
| Point cloud (CNN decoder) | 2.2 | 73.8 | fine-tuned |
| Voxel grid (CNN decoder) | 2.1 | 73.1 | fine-tuned |
| Mesh (CNN decoder, ico_sphere) | 2.3 | 69.4 | fine-tuned |
| Parametric network (AtlasNet-style) | 3.2 | 69.4 | fine-tuned |
| Voxel grid, trained on 3 classes | 3.3 | 69.3 | fine-tuned |
| Implicit occupancy network | 3.1 | 30.8 | fine-tuned |
All methods now use a jointly fine-tuned ResNet18 encoder rather than frozen precomputed features (§2.6). F1 alone is a misleading ranking — see §2.3.
No network yet: a free-standing voxel grid / point cloud / mesh is optimized directly (via Adam) to match a target shape, to sanity-check the loss functions in losses.py before using them to train a real network in Part 2.


The source point cloud was initialized as 0.1 * randn(...) rather than raw randn(...): the target has std ≈ 0.17, so a unit-scale Gaussian start is ~6× too spread out and does not converge within a few thousand Adam steps.

At 3000 iterations this fit looked spiky and self-intersecting, and this section originally blamed the genus-0 ico_sphere's topology. That was wrong: a chamfer-loss diagnostic shows the fit simply isn't converged at 3000 steps (0.0083) and keeps improving to a plateau around 12000–15000 (≈0.00013, 60× lower) — where the mesh above is clean, with four separated legs. Too few steps, not a topology limit. This meant the Part 2 mesh decoder (originally also 3000 iterations) was likely undertrained too, prompting the retraining below.
All three models share a ResNet18 encoder, jointly fine-tuned with the decoder rather than frozen behind precomputed features (§2.6). Trained at batch size 32 on the single-class (chair) R2N2 train split (6100 objects), evaluated on the held-out 678-object test split.
Originally reported at 3000, then 15000, then 50000 frozen-encoder iterations (§1.3). A §2.4 failure case raised the question of whether the frozen encoder was the bottleneck; §2.6 confirms it was. 2.1–2.3 below now report fine-tuned-encoder numbers (15000 iterations each) as primary; frozen numbers are kept in §2.6 for comparison.
F1@0.05 = 73.1 (frozen encoder: 69.3 at 50000 iters — see §2.6) — mean over all 678 test objects, 0 skipped, 15000 iterations.




C and D both fail on thin, high-frequency structure (a thin curved barrel wall; many thin slats and legs) rather than typical blocky chairs. Marching cubes on both breaks into scattered disconnected fragments instead of one coherent shape. At 32³ resolution, thin geometry occupies only a few voxels, so occupancy probability there sits close to the 0.5 threshold and flips unstably; blocky shapes have enough volume to stay confidently above threshold even with noise. This is a systematic weakness of low-resolution voxels on thin structures, not sample-specific.
F1@0.05 = 73.8 (frozen encoder: 69.8 — see §2.6) — mean over all 678 test objects, 0 skipped, 1000 points, 15000 iterations.




F1@0.05 = 69.4 (frozen encoder: 62.1 — see §2.6, the largest fine-tuning gain of the three representations) — mean over all 678 test objects, 0 skipped, 15000 iterations. ico_sphere(level 4), 2562 verts, default w_smooth=0.1.




| Representation | F1@0.05 | Valid samples |
|---|---|---|
| Voxel grid | 73.1 | 678 / 678 |
| Point cloud | 73.8 | 678 / 678 |
| Mesh | 69.4 | 678 / 678 |
All three improved with the fine-tuned encoder and now sit even closer together (69.4–73.8, §2.6). Mesh gained the most (+7.3) but remains lowest, consistent with it being the only representation with a hard topological constraint (fixed genus-0 template). Visual quality still doesn't track F1: mesh predictions are better structured than before but still show self-intersecting folds — F1@0.05 doesn't penalize that, and w_smooth=0.1 still isn't strong enough to prevent it (§2.4).
Varied hyperparameter: the mesh model's Laplacian smoothing weight w_smooth (0.02 vs. default 0.1 vs. 5.0), all else identical — fine-tuned encoder, 15000 iterations, batch 32.
| w_smooth | 0.02 | 0.1 (§2.3) | 5.0 |
|---|---|---|---|
| F1@0.05 (fine-tuned) | 71.1 | 69.4 | 61.1 |
| F1@0.05 (frozen, §2.6) | 66.7 | 62.1 | 55.9 |




The smoothness/fidelity trade-off persists and the ranking (low > default > high) is unchanged by fine-tuning, though every setting improved (+4.4 to +5.2). Low smoothing (0.02) still scores highest (71.1) despite a more chaotic, self-intersecting fold pattern — F1@0.05 is nearly blind to self-intersection. High smoothing (5.0) still collapses toward a smooth, non-self-intersecting blob lacking clear chair structure, scoring lowest (61.1): the Laplacian penalty resists the displacements needed to form legs/backrest regardless of encoder quality. A genus-0 template struggling with disconnected chair topology remains the likely shared root cause across all three settings.
Method: linearly interpolate the (frozen, 50000-iteration) encoder features of two test-set chairs (7 steps, α=0…1), decode each step, render with a 360° rotation. A smooth, meaningful feature→shape mapping should morph continuously between the two chairs rather than jump discontinuously.

The shape morphs smoothly with no abrupt jumps, even though individual frames are visually degenerate (self-intersecting folds, per §2.3). The decoder learned a genuinely continuous feature→deformation mapping; the problem is the quality of what it maps to (low w_smooth, genus-0 topology), not a lack of structure in the mapping itself.
Every model up through 2.5 (and 3.1–3.3) was originally trained with --load_feat: model.py skips building an encoder and feeds the decoder precomputed, frozen ImageNet-pretrained ResNet18 features (model.py:13-16, 74-78) — the optimizer only ever sees decoder parameters (train_model.py:94). Originally just a speed shortcut, this also meant the encoder was never adapted to the task. A §2.4 barrel-chair failure (prediction collapsed to a generic four-legged shape instead of the unusual wide/short target) raised the question of whether frozen, generic features were part of the cause. To test it, vox/point/mesh were each retrained without --load_feat — the same pretrained ResNet18 now gets real gradients jointly with the decoder — for 15000 iterations each (vs. 50000 frozen).
| Frozen (50000 iters) | Fine-tuned (15000 iters) | Δ | |
|---|---|---|---|
| Voxel | 69.3 | 73.1 | +3.8 |
| Point cloud | 69.8 | 73.8 | +4.0 |
| Mesh | 62.1 | 69.4 | +7.3 |
Voxel (§2.1)




Point cloud (§2.2)




Mesh (§2.3)




Every representation improved, mesh the most (+7.3), despite using under a third of the training iterations — 2.1–2.3 now report these as primary. The qualitative gap for mesh is bigger than the F1 gain suggests: the frozen prediction (Fig 2.6-I) is a structureless blob, while the fine-tuned one (Fig 2.6-J, same object) shows distinct arms, seat, backrest and legs. Voxel/point cloud (Fig 2.6-A–H) sharpen more subtly. This supports the hypothesis from §2.4's barrel-chair failure: frozen ImageNet features are generic, never asked to encode the cues that separate an atypical shape from the training distribution's average chair; letting gradients reach the encoder lets it specialize. §2.4, §3.1 and §3.3 are all now updated too (mixed results — see those sections for where fine-tuning didn't fully close the gap).
All three sub-parts attempted (only one is required). 3.1, 3.2, 3.3 have all been re-run with a fine-tuned encoder (§2.6); frozen-encoder numbers are noted inline. Trained for at least 15000 iterations, evaluated with the same F1@0.05 metric — directly comparable to 2.1 and 2.2.
A coordinate-conditioned MLP f(image_feature, xyz) → occupancy_logit (Occupancy-Networks-style), no positional encoding of the coordinate. Trained with balanced occupied/empty point sampling from the same 32³ voxel grids used in 2.1, evaluated by querying a 32³ grid over (-1,1)³ (as specified) and reusing 2.1's marching-cubes + F1 pipeline unchanged. Also re-run with a fine-tuned encoder, 15000 iterations.
F1@0.05 = 30.8 (frozen encoder: 31.0 at 50000 iters, 12.0 at 2000 iters) — mean over all 678 test objects, 0 skipped.



Unlike every other representation (§2.6), fine-tuning left implicit essentially unchanged (31.0→30.8, within noise, despite 50000 frozen vs. 15000 fine-tuned iterations) and still far below the CNN voxel decoder (73.1, §2.1). The bottleneck isn't encoder feature quality: a coordinate MLP without positional encoding has a well-known low-frequency bias (the motivation for Fourier features in NeRF) — it fits smooth occupancy fields but struggles with sharp boundaries, so the shape stays a rounded blob lacking crisp legs/seat/backrest edges regardless of input feature quality. The CNN decoder's transposed-convolution upsampling bakes in spatial structure this MLP has to learn from scratch, point by point.
An AtlasNet-style single-chart generator g(image_feature, uv) → xyz: every step samples fresh uv ∼ Uniform[0,1]², maps to a 3D point, trained with 2.2's chamfer_loss against 1000 GT surface points. Eval uses a fixed seeded uv set, scored with 2.2's F1 pipeline. Like 2.1–2.3, re-run with a fine-tuned encoder (§2.6) rather than frozen features.
F1@0.05 = 69.4 (frozen encoder: 58.9 at 2000 iters, 60.9 at 15000 iters) — mean over all 678 test objects, 15000 iterations.



Fine-tuning gave parametric the largest jump of any 3.1/3.2 model (60.9→69.4, +8.5), closing most of the gap to 2.2's fine-tuned point decoder (73.8). Predicted points still cover somewhat less of the chair's silhouette — sparser, more clustered — the structural cost of a single continuous 2D→3D chart rather than independently-predicted points. That constraint is AtlasNet's whole point for surface-like shapes, but costs coverage on a topologically complex chair; better features narrow the gap but don't eliminate it.
Trained a voxel model on the 3-class extended dataset (chair + car + airplane, 12886 train objects, split_3c.json), same architecture as 2.1, compared against the chair-only (6100 train objects) voxel model — both evaluated on the same 678-object held-out chair test set. Also re-run with a fine-tuned encoder, 15000 iterations.
| Trained on | Encoder | F1@0.05 (chair test set) | Valid predictions |
|---|---|---|---|
| Chair only (2.1) | fine-tuned | 73.1 | 678 / 678 |
| Chair + car + airplane | fine-tuned | 69.3 | 662 / 678 (98%) |
| Chair only (2.1) | frozen | 69.3 | 678 / 678 |
| Chair + car + airplane | frozen | 67.1 | 678 / 678 |


At 3000 iterations, splitting capacity across three categories roughly halved F1 (45.7→27.4) and more than doubled the empty-prediction rate. By 50000 frozen-encoder iterations that gap had nearly closed (69.3 vs. 67.1, both 0 skipped). Fine-tuning raises both further (73.1 vs. 69.3) but reopens a small reliability gap: the 3-class model now skips 16/678 empty predictions where the chair-only model skips none — sharing decoder capacity across categories still costs some robustness on hard chair cases even with a good encoder and ample training. One shared decoder is close to sufficient for chair-level quality while also covering cars and airplanes, but not perfectly so.
R2N2 / ShapeNet chairs · PyTorch3D · single NVIDIA GPU. Part 1 fits are per-object direct optimization (12000–15000 iterations, seconds each). Part 2 and Extra Credit networks were first trained at 50000 iterations with --load_feat (frozen precomputed features, disclosed here per the assignment README's requirement — this flag defaults to False and is meant only as a fallback for limited GPU access), then re-trained without it (jointly fine-tuned encoder, §2.6) once frozen features turned out to measurably hurt quality.