16-825 · Learning for 3D Vision

Assignment 2: Single View to 3D

1. Exploring Loss Functions

Each GIF shows the optimized source (left) next to the ground-truth target (right), rendered from 360°.

1.1 Fitting a Voxel Grid

Optimized voxel grid beside the ground-truth voxel grid
Optimized voxel grid (left) and ground truth (right). Command: python fit_data.py --type vox

Loss

$$\mathcal{L}_{\text{vox}} = -\frac{1}{N}\sum_{i=1}^{N}\Big[\, y_i \log \sigma(x_i) + (1 - y_i)\log\big(1 - \sigma(x_i)\big) \Big]$$

where \(x_i\) is the predicted logit of voxel \(i\), \(y_i \in \{0, 1\}\) its target occupancy, \(\sigma\) the sigmoid, and \(N = 32^3\) the number of voxels.

1.2 Fitting a Point Cloud

Optimized point cloud beside the ground-truth point cloud
Optimized point cloud (left) and ground truth (right). Command: python fit_data.py --type point

Loss

$$\mathcal{L}_{\text{chamfer}} = \frac{1}{|S|}\sum_{p \in S}\min_{q \in T}\lVert p - q \rVert_2^2 + \frac{1}{|T|}\sum_{q \in T}\min_{p \in S}\lVert p - q \rVert_2^2$$

where \(S\) is the predicted point cloud, \(T\) the target point cloud, and \(|S|\), \(|T|\) their numbers of points.

1.3 Fitting a Mesh

Optimized mesh beside the ground-truth mesh
Optimized mesh deformed from an icosphere (left) and ground truth (right). Command: python fit_data.py --type mesh

Loss

$$\mathcal{L} = w_{\text{chamfer}}\,\mathcal{L}_{\text{chamfer}} + w_{\text{smooth}}\,\mathcal{L}_{\text{smooth}}, \qquad \mathcal{L}_{\text{smooth}} = \frac{1}{|V|}\sum_{i \in V}\Big\lVert v_i - \frac{1}{|N(i)|}\sum_{j \in N(i)} v_j \Big\rVert_2$$

where \(\mathcal{L}_{\text{chamfer}}\) is the 1.2 loss on points sampled from the predicted and target mesh surfaces, \(V\) the set of mesh vertices, \(v_i\) the position of vertex \(i\), \(N(i)\) the vertices that share an edge with it, \(w_{\text{chamfer}} = 1.0\) and \(w_{\text{smooth}} = 0.1\).

2. Reconstructing 3D from a Single View

Each example shows the input RGB image, the predicted 3D output, and the ground-truth mesh for a sample from the test set.

2.1 Image to Voxel Grid

Example 1 · test sample 240 · F1@0.05 = 87.5

Input RGB, predicted voxels and ground-truth mesh for test sample 240
Show 360° view — left: predicted voxels, right: ground-truth mesh
360 degree view of the predicted voxels beside the ground-truth mesh for test sample 240
Left: predicted voxels. Right: ground-truth mesh. 36 views, 10° apart.

Example 2 · test sample 660 · F1@0.05 = 75.8

Input RGB, predicted voxels and ground-truth mesh for test sample 660
Show 360° view — left: predicted voxels, right: ground-truth mesh
360 degree view of the predicted voxels beside the ground-truth mesh for test sample 660
Left: predicted voxels. Right: ground-truth mesh. 36 views, 10° apart.

Example 3 · test sample 150 · F1@0.05 = 55.3

Input RGB, predicted voxels and ground-truth mesh for test sample 150
Show 360° view — left: predicted voxels, right: ground-truth mesh
360 degree view of the predicted voxels beside the ground-truth mesh for test sample 150
Left: predicted voxels. Right: ground-truth mesh. 36 views, 10° apart.

Each example uses the chair’s most front-facing input view; the prediction (occupancy > 0.5) and ground truth are rendered from the input camera.

F1@0.05: 53.4

Average F1 score over the test set at thresholds from 0.01 to 0.05
Average F1 over the 678 test chairs. Command: python eval_model.py --type vox --load_checkpoint
Show setup — decoder, loss and training

FC to 256×4³, three ConvTranspose3d (+BatchNorm, ReLU) up to 32³, then a 3×3×3 Conv3d to one logit per voxel. BCE-with-logits loss; Adam (lr 4e-4, batch 32), 10,000 iterations, encoder fine-tuned on a GPU; --load_feat not used.

2.2 Image to Point Cloud

Example 1 · test sample 240 · F1@0.05 = 92.0

Input RGB, predicted point cloud and ground-truth mesh for test sample 240
Show 360° view — left: predicted point cloud, right: ground-truth mesh
360 degree view of the predicted point cloud beside the ground-truth mesh for test sample 240
Left: predicted point cloud. Right: ground-truth mesh. 36 views, 10° apart.

Example 2 · test sample 660 · F1@0.05 = 97.3

Input RGB, predicted point cloud and ground-truth mesh for test sample 660
Show 360° view — left: predicted point cloud, right: ground-truth mesh
360 degree view of the predicted point cloud beside the ground-truth mesh for test sample 660
Left: predicted point cloud. Right: ground-truth mesh. 36 views, 10° apart.

Example 3 · test sample 150 · F1@0.05 = 47.9

Input RGB, predicted point cloud and ground-truth mesh for test sample 150
Show 360° view — left: predicted point cloud, right: ground-truth mesh
360 degree view of the predicted point cloud beside the ground-truth mesh for test sample 150
Left: predicted point cloud. Right: ground-truth mesh. 36 views, 10° apart.

Each example uses the chair’s most front-facing input view; the prediction (1,000 points) and ground truth are rendered from the input camera.

F1@0.05: 80.0

Average F1 score over the test set at thresholds from 0.01 to 0.05
Average F1 over the 678 test chairs. Command: python eval_model.py --type point --load_checkpoint
Show setup — decoder, loss and training

MLP 512 → 1024 → 2048 → 1000×3 (ReLU between layers). Chamfer loss against 1,000 points sampled from the GT mesh; Adam (lr 4e-4, batch 32), 10,000 iterations, encoder fine-tuned on a GPU; --load_feat not used.

2.3 Image to Mesh

Example 1 · test sample 240 · F1@0.05 = 97.1

Input RGB, predicted mesh and ground-truth mesh for test sample 240
Show 360° view — left: predicted mesh, right: ground-truth mesh
360 degree view of the predicted mesh beside the ground-truth mesh for test sample 240
Left: predicted mesh. Right: ground-truth mesh. 36 views, 10° apart.

Example 2 · test sample 660 · F1@0.05 = 94.7

Input RGB, predicted mesh and ground-truth mesh for test sample 660
Show 360° view — left: predicted mesh, right: ground-truth mesh
360 degree view of the predicted mesh beside the ground-truth mesh for test sample 660
Left: predicted mesh. Right: ground-truth mesh. 36 views, 10° apart.

Example 3 · test sample 150 · F1@0.05 = 45.7

Input RGB, predicted mesh and ground-truth mesh for test sample 150
Show 360° view — left: predicted mesh, right: ground-truth mesh
360 degree view of the predicted mesh beside the ground-truth mesh for test sample 150
Left: predicted mesh. Right: ground-truth mesh. 36 views, 10° apart.

Each example uses the chair’s most front-facing input view; the prediction and ground truth are rendered from the input camera.

F1@0.05: 72.7

Average F1 score over the test set at thresholds from 0.01 to 0.05
Average F1 over the 678 test chairs. Command: python eval_model.py --type mesh --load_checkpoint
Show setup — decoder, loss and training

Level-4 icosphere (2,562 vertices). MLP 512 → 1024 → 2048 → 2562×3 with tanh predicts per-vertex offsets. Loss: chamfer (1,000 samples, w_chamfer = 1) + Laplacian smoothing (w_smooth = 0.1); Adam (lr 4e-4, batch 32), 10,000 iterations, encoder fine-tuned on a GPU; --load_feat not used.

Comparing voxels, point clouds and meshes

RepresentationF1@0.05
Voxel grid53.4
Point cloud80.0
Mesh72.7
  • Point cloud (best): the chamfer loss directly optimizes what F1 measures, and free points can cover thin, disconnected parts such as legs.
  • Mesh: Also using chamfer loss as point cloud, but the fixed sphere topology can’t open holes, so gaps between legs become folded surfaces that miss the ground truth.
  • Voxel (worst): at 32³ one voxel is ~0.031 wide, close to the 0.05 threshold; also thin parts rarely exceed 0.5 occupancy and vanish.

2.4 Effects of Hyperparameters

Varied: the occupancy threshold that turns the voxel model’s probabilities into a surface (--vox_thresh, default 0.5). Same checkpoint and the same input views for every setting (--seed 0).

ThresholdF1@0.05Empty predictions
0.139.00
0.257.10
0.365.03
0.461.67
0.5 (default)53.025
Voxel predictions for three chairs at thresholds 0.1, 0.3 and 0.5 beside the ground truth
The same three test chairs thresholded at 0.1, 0.3 and 0.5, rendered from the input view.

Analysis

t=0.3 is the best statistically and visually, largely raising F1@0.05 from t=0.1 to t=0.3, and mostly tying with t=0.5 on each sample without retraining. About 96% of voxels are empty, so BCE leaves the model under-confident on occupied ones. At t=0.5 thin parts (legs) drop out and 25 predictions are empty in overall test dataset; at 0.1 the shape inflates and precision falls. Thin chairs gain most (sample 150: 64 → 83), while solid chairs are slightly better at 0.5 (samples 240, 660). The threshold was swept on the test set, so 65.0 is slightly optimistic.

2.5 Interpreting the Model

1. View consistency: does the prediction depend on the input view?

Line chart of mean F1 at 0.05 versus input view azimuth for the voxel, point and mesh models
Mean F1@0.05 per 45° bin of input azimuth, over 57 test chairs × all 10 views each.
Ground-truth meshes beside their ground-truth voxel grids for three test chairs whose voxels are rotated 90 degrees
Rotated ground-truth voxels. Examples where the dataset’s ground-truth voxels are rotated relative to the mesh: the mesh’s long axis runs left to right, the voxel grid’s runs front to back. Both are rendered from the same camera.

What this shows

Point and mesh are view-invariant (their curves are flat), so they infer the shape rather than memorizing a viewpoint. The voxel model depends much more on the view: its F1 changes with the input angle and drops to 43 for front-facing inputs. A likely cause is its training targets: some ground-truth voxel grids in the original dataset are rotated relative to their meshes (see Rotated ground-truth voxels above), so the model learns an inconsistent orientation.

2. Latent interpolation: is the feature space smooth?

Point and mesh shapes decoded from blends of two chairs' image features
Decoding (1−α)·zA + α·zB between a slim side chair (A) and an armchair (B).

What this shows

The shapes change gradually (the chair widens, the seat deepens, the back shortens) with no jumps or broken intermediates. The interpolated latent is the 512-d output of the ResNet18 encoder’s global average-pooling layer, so the encoder maps chairs to a continuous shape space rather than to isolated templates.

3. Exploring Other Architectures / Datasets

3.1 Implicit Network

Example 1 · test sample 240 · F1@0.05 = 93.8 (2.1 decoder 87.5)

Input RGB, implicit-decoder occupancy and ground-truth mesh for test sample 240
Show 360° view — left: predicted occupancy (implicit decoder), right: ground-truth mesh
360 degree view of the implicit-decoder prediction beside the ground-truth mesh for test sample 240
Left: predicted occupancy (implicit decoder). Right: ground-truth mesh. 36 views, 10° apart.

Example 2 · test sample 660 · F1@0.05 = 88.5 (2.1 decoder 75.8)

Input RGB, implicit-decoder occupancy and ground-truth mesh for test sample 660
Show 360° view — left: predicted occupancy (implicit decoder), right: ground-truth mesh
360 degree view of the implicit-decoder prediction beside the ground-truth mesh for test sample 660
Left: predicted occupancy (implicit decoder). Right: ground-truth mesh. 36 views, 10° apart.

Example 3 · test sample 150 · F1@0.05 = 57.9 (2.1 decoder 55.3)

Input RGB, implicit-decoder occupancy and ground-truth mesh for test sample 150
Show 360° view — left: predicted occupancy (implicit decoder), right: ground-truth mesh
360 degree view of the implicit-decoder prediction beside the ground-truth mesh for test sample 150
Left: predicted occupancy (implicit decoder). Right: ground-truth mesh. 36 views, 10° apart.

Same chairs and input views as 2.1. Each image: input RGB | predicted occupancy (> 0.5) | ground-truth mesh, rendered from the input camera.

F1@0.05: 52.3 (occupancy > 0.5) · 60.9 (occupancy > 0.3)

Average F1 score of the implicit decoder at thresholds from 0.01 to 0.05
Average F1 over the 678 test chairs (occupancy > 0.5). Command: python eval_model.py --type vox --vox_dec implicit --exp _implicit --load_checkpoint
Show setup — decoder, loss and training

An MLP predicts the occupancy logit of a 3D point p from [p; z]. It is queried at all points of a 32³ grid in (−1, 1)³ and reshaped to 32³, so the BCE loss and evaluation from 2.1 are reused. Layers: Linear(3, 256) on p plus Linear(512, 256) on z, two 256-wide hidden layers, then 256 → 1 (ReLU between). Training as in 2.1.

Result

On the same input views, it matches 2.1’s transposed-conv decoder at threshold 0.5 (52.3 vs 53.0; 2.1’s model re-evaluated on the views used in 2.4) with 42× fewer parameters (0.26 M vs 11.2 M), but loses more thin parts (60.9 vs 65.0 at 0.3).

3.2 Parametric Network

Example 1 · test sample 240 · F1@0.05 = 91.8 (2.2 decoder 92.0)

Input RGB, parametric-decoder point cloud and ground-truth mesh for test sample 240
Show 360° view — left: predicted point cloud (parametric decoder), right: ground-truth mesh
360 degree view of the parametric-decoder point cloud beside the ground-truth mesh for test sample 240
Left: predicted point cloud (parametric decoder). Right: ground-truth mesh. 36 views, 10° apart.

Example 2 · test sample 660 · F1@0.05 = 91.4 (2.2 decoder 97.3)

Input RGB, parametric-decoder point cloud and ground-truth mesh for test sample 660
Show 360° view — left: predicted point cloud (parametric decoder), right: ground-truth mesh
360 degree view of the parametric-decoder point cloud beside the ground-truth mesh for test sample 660
Left: predicted point cloud (parametric decoder). Right: ground-truth mesh. 36 views, 10° apart.

Example 3 · test sample 150 · F1@0.05 = 52.1 (2.2 decoder 47.9)

Input RGB, parametric-decoder point cloud and ground-truth mesh for test sample 150
Show 360° view — left: predicted point cloud (parametric decoder), right: ground-truth mesh
360 degree view of the parametric-decoder point cloud beside the ground-truth mesh for test sample 150
Left: predicted point cloud (parametric decoder). Right: ground-truth mesh. 36 views, 10° apart.

Same chairs and input views as 2.2. Each image: input RGB | predicted point cloud (1,000 points) | ground-truth mesh, rendered from the input camera.

F1@0.05: 75.8

Average F1 score of the parametric decoder at thresholds from 0.01 to 0.05
Average F1 over the 678 test chairs. Command: python eval_model.py --type point --point_dec parametric --exp _param --load_checkpoint
Parametric-decoder predictions for three chairs, with each point colored by the patch that produced it
Parametric-decoder predictions colored by patch (4 patches × 250 points), for the same three chairs and input views as the examples, rendered from three azimuths.
Show setup — decoder, loss and training

AtlasNet-style decoder with 4 patches. Each patch is an MLP that maps a 2D sample (u, v) ∈ [0, 1]² and z to a 3D point (Linear(2, 256) on (u, v) plus Linear(512, 256) on z, two 256-wide hidden layers, then 256 → 3). Each patch outputs 250 points, giving 1,000 in total; (u, v) are resampled every training step and fixed at evaluation. Chamfer loss and training as in 2.2.

Result

On the same input views, it scores 75.8 vs 79.3 for 2.2’s fully connected decoder (2.2’s model re-evaluated on the views used in 2.4), with 8× fewer parameters (1.06 M vs 8.8 M). Colored by patches, each of the four patches spreads over different parts (e.g. red covers left 2 legs, green covers right 2 legs and upper backrest), but the patches overlap heavily, so no patch specializes in a single part.