16-825: Learning for 3D Vision · Kashu Yamazaki
Each turntable shows the optimized source on the left and the target on the right.
voxel_loss is binary cross-entropy on logits
(F.binary_cross_entropy_with_logits), so the optimized grid is unconstrained and
the rendered occupancy is its sigmoid, meshed with marching cubes at 0.5. 10k Adam steps at
lr 1e-2 drive the loss to ~0.
chamfer_loss finds each point's nearest neighbour in the other cloud with
knn_points (K = 1) in both directions, and sums the two mean squared
distances, averaged over the batch.
smoothness_loss is the uniform Laplacian smoothing term
(mesh_laplacian_smoothing): each vertex is pulled toward the mean of its
neighbours. It is added to the chamfer loss on points sampled from both meshes with weight 0.1.
The source is a level-4 ico-sphere, so the fit cannot open the holes between the legs and
instead stretches thin webs across them.
All models train end to end from RGB images (no --load_feat) with the ResNet18
encoder fine-tuned: batch size 32, Adam at lr 4e-4, 10k iterations (~5 epochs), on an RTX 3080.
Each example below shows the input image, the prediction and the ground-truth mesh. The test
loader picks a random view per chair, so F1 moves by about ±0.5 between evaluation runs.
The decoder is a linear layer to a 256×4×4×4 volume followed by three stride-2 3D transposed convolutions (with BatchNorm and ReLU) to 323 and a final 3×3×3 convolution to one occupancy logit per voxel.



F1@0.05 = 51.3 (precision 65.8, recall 46.8).
An MLP (512 → 1024 → 2048 → 1000×3, ReLU) regresses the 1000 point coordinates directly, trained with the chamfer loss against 1000 points sampled from the GT mesh.



F1@0.05 = 80.3.
The same MLP shape outputs a 3D offset for each of the 2562 vertices of a level-4 ico-sphere. Loss: chamfer on 1000 sampled points + 0.1 × Laplacian smoothing.



F1@0.05 = 72.5.
| Representation | F1@0.05 |
|---|---|
| Point cloud | 80.3 |
| Mesh | 72.5 |
| Voxels (plain BCE) | 51.3 |
| Voxels (BCE, pos_weight 3; see 2.4) | 62.3 |
Points win because they are trained on what is measured. F1 is a nearest-neighbour test between point sets, and chamfer is its soft version, so the point decoder optimizes almost exactly the metric. Nothing ties the points together, so they can go wherever the surface is, including thin legs.
Meshes pay for their topology. The mesh is still trained with chamfer, but on points sampled from a deformed sphere. A sphere cannot open the gaps between legs, slats and armrests, so it stretches long thin triangles across them. Those faces put sampled points in empty space, which costs precision. The same stretched faces produce the spikes, which the 0.1 smoothness weight only partly suppresses.
Voxels pay twice. A 323 grid has cells 1/32 ≈ 0.03 wide in a unit box, already over half the 0.05 threshold, so even the ground-truth voxels only reach F1 ≈ 80.7 under this evaluation. On top of that, BCE with ~8% occupied cells makes the network cautious, and anything below 0.5 disappears at the marching-cubes threshold. 44 of 678 test chairs come back completely empty; the folding chair in the 2.4 figure is one.
The voxel failures above are an imbalance problem, so I varied pos_weight in
binary_cross_entropy_with_logits, which scales the loss on occupied voxels. All
other settings match 2.1.
| pos_weight | Precision | Recall | F1@0.05 | Empty predictions |
|---|---|---|---|---|
| 1 (2.1 baseline) | 65.8 | 46.8 | 51.3 | 44 / 678 |
| 3 | 58.6 | 68.5 | 62.3 | 2 / 678 |
| 10 | 33.5 | 48.4 | 38.8 | 0 / 678 |
At weight 1 the network leaves out whatever it is unsure of: legs, seat edges, and sometimes the whole chair. Weight 3 roughly balances the two errors; empty predictions almost disappear and recall rises by 22 points for a 7-point drop in precision. Weight 10 over-corrects: every uncertain voxel is filled, so chairs turn into thick blobs. Precision collapses, and recall drops as well. The bloated outer surface sits away from the true surface, and the evaluation re-centres on a centroid that the extra mass moves. The weight that matches the occupancy ratio (≈12) is too high. A much smaller weight (3, about its square root) works best here, because what matters is where the 0.5 marching-cubes threshold lands, not how well the model ranks voxels.
Where the point decoder is wrong. On the left, each predicted point is coloured by its distance to the GT surface (precision errors). On the right, each GT point is coloured by its distance to the prediction (recall errors). The colour map saturates at 0.05, so bright yellow marks points that the F1 metric counts as misses.



Three kinds of error show up. (1) Hedging. Stray points float above the backrest and around the outline. When the model is unsure how tall or wide the chair is, it spreads a few points over the possibilities, and each of those costs precision. (2) Whole parts slightly misplaced. In the third chair the seat is bright on both sides: the predicted seat is a correct-looking slab at a slightly wrong height, so a whole part lands just past 0.05. From one view, part placement along the viewing direction is ambiguous, and a single wrong offset fails a whole block of points at once. (3) Uncommon shapes. On the slatted side chair the backrest and legs are mostly dark. The boxy armchair is bright almost everywhere, worst on the armrests and outer sides. The decoder is good at the typical side-chair layout that dominates the training set and regresses toward it on shapes that differ from it.
What the latent space encodes. I encoded two test images with the ResNet18 encoder of the mesh model and decoded linear blends of the two 512-d features.
The morph is smooth: the wide, low armchair narrows, the back rises and the legs come out, with no sudden jumps or broken shapes in between. So the encoder maps images to a continuous shape space, not a lookup of memorized chairs, and decoding the midpoint gives a plausible in-between chair.
The decoder is an occupancy network: for every query point p ∈ (−1, 1)3 of a 323 grid it predicts an occupancy logit from the image feature and p. It is an MLP on concat(f, p) (hidden size 128, three hidden layers). The first layer is split into Wff + Wpp so that the feature term is computed once per image, not once per point. Points are laid out in the same (z, y, x) order as the voxel targets, so training (BCE) and evaluation (sigmoid → marching cubes) are exactly as in 2.1.



| Decoder | pos_weight | Precision | Recall | F1@0.05 | Empty |
|---|---|---|---|---|---|
| Implicit | 1 | 56.7 | 38.8 | 43.5 | 125 / 678 |
| Implicit | 3 | 58.7 | 67.2 | 61.9 | 17 / 678 |
| 3D conv (2.1 / 2.4) | 1 | 65.8 | 46.8 | 51.3 | 44 / 678 |
| 3D conv (2.1 / 2.4) | 3 | 58.6 | 68.5 | 62.3 | 2 / 678 |
With plain BCE the implicit network under-fills even more than the convolutional decoder, with 125 empty chairs against 44. Each query point is classified on its own, with no transposed convolutions to spread evidence between neighbouring cells, so a part the network is unsure of drops below 0.5 everywhere at once. With the class weight from 2.4 the two decoders are level (61.9 vs 62.3 F1), even though the implicit MLP has 99k decoder parameters against 11.2M for the convolutional one. The implicit network still leaves more chairs empty (17 vs 2). It is also not tied to 323: in principle the same network can be queried on a finer grid at test time (not tested here), while the convolutional decoder's output resolution is fixed by its architecture.