16-825 Assignment 2: Single View to 3D

Learning for 3D Vision · Abhikhya Tripathy (abhikhyt)

1. Exploring Loss Functions

Each GIF shows the Optimized shape on the left and the ground truth on the right.

1.1 Fitting a voxel grid

Details: voxel_loss is binary cross-entropy between the source grid and the binary target grid. The source grid is Optimized as unconstrained logits (binary_cross_entropy_with_logits) and passed through a sigmoid before marching cubes at 0.5 for the render.

Optimized voxel grid (left) and ground-truth voxel grid (right).
Optimized voxel grid (left) and ground-truth voxel grid (right).

1.2 Fitting a point cloud

Details: chamfer_loss is written with knn_points only: for every source point the squared distance to its nearest target point, and for every target point the squared distance to its nearest source point, each averaged over the points and summed.

Optimized point cloud (left) and ground-truth point cloud (right).
Optimized point cloud (left) and ground-truth point cloud (right).

1.3 Fitting a mesh

Details: The mesh is fitted with the Chamfer loss on points sampled from both surfaces plus smoothness_loss, a uniform Laplacian smoothing term (weight 0.1) that keeps each vertex close to the average of its neighbours.

Optimized mesh starting from <code>ico_sphere(4)</code> (left) and ground-truth mesh (right).
Optimized mesh starting from ico_sphere(4) (left) (2,562 vertices) and ground-truth mesh (right).

I also tried two other initial meshes for the same target:

Starting from <code>ico_sphere(2)</code> (162 vertices): too few vertices to form the legs.
Starting from ico_sphere(2) (162 vertices): too few vertices to form the legs.
Starting from a torus (2,048 vertices).
Starting from a torus (2,048 vertices): perhaps not a very good starting point because of its shape and topology - the ring is too wide and it has a hole the chair doesn't have.

2. Reconstructing 3D from a Single View

Setup details for every model on this webpage. --load_feat was not used. Checkpoints were saved every 10,000 iterations and all results below use the 90,000-iteration checkpoint. Each example is a turntable GIF with the input RGB on the left. F1 scores are averages over the 678 test chairs.

2.1 Image to voxel grid

Decoder: a linear layer to a 256×4×4×4 volume, three transposed 3D convolutions (each doubling the resolution, with batch norm and ReLU) up to 32×32×32, and a 1×1×1 convolution that outputs occupancy logits. Trained with the binary cross-entropy loss from 1.1. Predicted and ground-truth voxels are shown as marching-cubes surfaces in the frame of the ground-truth mesh.

Test shape 0: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh
Test shape 0: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh
Test shape 200: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh
Test shape 200: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh
Test shape 400: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh
Test shape 400: input RGB, predicted voxels, ground-truth voxels, ground-truth mesh

F1@0.05 = 76.0

F1 score of the voxel model at thresholds 0.01 to 0.05.
F1 score of the voxel model at thresholds 0.01 to 0.05.

2.2 Image to point cloud

Decoder: an MLP (512 → 1024 → 2048 → 3×1000) with a tanh on the output, giving 1,000 points. Trained with the Chamfer loss from 1.2 against 1,000 points sampled from the ground-truth mesh.

Test shape 0: input RGB, predicted point cloud, ground-truth mesh
Test shape 0: input RGB, predicted point cloud, ground-truth mesh
Test shape 200: input RGB, predicted point cloud, ground-truth mesh
Test shape 200: input RGB, predicted point cloud, ground-truth mesh
Test shape 400: input RGB, predicted point cloud, ground-truth mesh
Test shape 400: input RGB, predicted point cloud, ground-truth mesh

F1@0.05 = 76.7

F1 score of the point cloud model at thresholds 0.01 to 0.05.
F1 score of the point cloud model at thresholds 0.01 to 0.05.

2.3 Image to mesh

Decoder: an MLP (512 → 1024 → 2048 → 3×2562) that predicts one offset per vertex of an ico_sphere(4) template. Trained with the Chamfer loss on 1,000 points sampled from each surface plus 0.1 × the Laplacian smoothness loss from 1.3.

Test shape 0: input RGB, predicted mesh, ground-truth mesh
Test shape 0: input RGB, predicted mesh, ground-truth mesh
Test shape 200: input RGB, predicted mesh, ground-truth mesh
Test shape 200: input RGB, predicted mesh, ground-truth mesh
Test shape 400: input RGB, predicted mesh, ground-truth mesh
Test shape 400: input RGB, predicted mesh, ground-truth mesh

F1@0.05 = 72.7

F1 score of the mesh model at thresholds 0.01 to 0.05.
F1 score of the mesh model at thresholds 0.01 to 0.05.

Comparison of the three representations

RepresentationF1@0.05
Point cloud (1,000 points)76.7
Voxel grid (32³)76.0
Mesh (ico_sphere(4))72.7

The point cloud scores highest, the voxel grid is close behind and the mesh is lowest. Point cloud is trained with the Chamfer loss which is a quantity that's very close to the F1 score (how many predicted points are near the ground truth and vice versa). The points also don't need to sit on a grid or follow any connectivity so they can sit on thin structures and create gaps wherever the shape needs that. So, the point cloud scores highest. Voxel grids are next because occupancy can handle varied topologies, including holes. However, the grid we used is coarse, so thin parts such as legs are often missing or fragmented at the 0.5 threshold. The mesh scores the lowest because the network deforms vertices of a sphere, so the output is always a single closed surface with no holes. Gaps get created by stretched faces and points sampled on such faces count as false positives. The smoothing loss also counters sharp edges and corners that thin structures might need.

2.4 Effect of the initial mesh (hyperparameter variation)

I varied the template mesh that the mesh decoder deforms, keeping everything else fixed (same losses and weights, 1,000 sampled points, 90,000 iterations). This changes two things: the number of vertices, and the topology.

TemplateVerticesTopologyF1@0.05
ico_sphere(2)162sphere (no holes)70.7
ico_sphere(4) (baseline in 2.3)2,562sphere (no holes)72.7
ico_sphere(5)10,242sphere (no holes)74.2
Torus (r = 0.2, R = 0.5)2,048one hole73.9
<code>ico_sphere(2)</code>
ico_sphere(2)
<code>ico_sphere(4)</code>
ico_sphere(4)
<code>ico_sphere(5)</code>
ico_sphere(5)
Torus
Torus

The same test image (test shape 200) given to each of the four models: input RGB, predicted mesh, ground-truth mesh. We observe the following:

2.5 Interpreting the models

Two experiments, each run on the same test images for every model so the results can be compared.

(a) Latent interpolation: is the shape space continuous?

The 512-d image features of two test images are blended linearly and every blend is decoded (inspired by Achlioptas et al., 2018). A smooth morph means the decoder has learned a continuous space of chairs. t is the blending amount between the two input images, from 0 to 1: at t = 0 the shape is decoded purely from image A's feature, at t = 1 purely from image B's, and so on.

Point cloud model: image A, decoded blend, image B.
Point cloud model: image A, decoded blend, image B.
Mesh model: image A, decoded blend, image B
Mesh model.
Voxel model: image A, decoded blend, image B
Voxel model.
Voxel model, six blends from image A (left) to image B (right).
Voxel model, six blends from image A (left) to image B (right).

The point cloud and the mesh morph smoothly from the armchair to the thin chair: the arms shrink and the legs grow, with a plausible chair at every step. The voxel model does not. Its intermediate predictions break into scattered fragments, because the blended features give occupancies close to the 0.5 threshold. So the voxel decoder is confident at the two ends but not in between. It likely does not learn a continuous shape space for chairs.

(b) Checkerboard occlusion: how much does the model rely on seeing the full image?

Half of the input image is blacked out with a checkerboard of 8, 16 or 32 pixel cells, and compared with the clean image and with a completely black image (which shows what the model predicts when it sees nothing).

Voxel model.
Voxel model.
Point cloud model.
Point cloud model.
Mesh model.
Mesh model.
Observations:

3. Exploring Other Architectures / Datasets

3.1 Implicit network

The implicit decoder takes the image feature and a 3D coordinate and outputs the occupancy of that single location. The coordinate and the 512-d feature are each projected to 256 dimensions and added, followed by a three-layer MLP that outputs one logit. To produce a full prediction it is evaluated on a 32×32×32 grid of coordinates in (-1, 1)³, so it is trained and evaluated exactly like the voxel model in 2.1 (same loss, same ground truth). The only difference is that one small network is shared across all locations instead of generating the whole grid at once.

Test shape 0: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh
Test shape 0: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh
Test shape 200: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh
Test shape 200: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh
Test shape 400: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh
Test shape 400: input RGB, predicted occupancy, ground-truth voxels, ground-truth mesh

F1@0.05 = 74.2 (voxel decoder in 2.1: 76.0)

F1 score of the implicit model at thresholds 0.01 to 0.05.
F1 score of the implicit model at thresholds 0.01 to 0.05.
Latent interpolation with the implicit decoder (compare with the voxel model in 2.5b).
Latent interpolation with the implicit decoder (compare with the voxel model in 2.5a).

3.2 Parametric network

The parametric decoder maps a 2D point sampled from the unit square, together with the image feature, to a 3D point (a single-patch AtlasNet). The 2D point and the feature are each projected to 512 dimensions and added, followed by a three-layer MLP with a tanh output. During training 1,000 random 2D points are sampled per image and the output is trained with the Chamfer loss, exactly like the point model in 2.2.

3.3 Extended dataset for training

The point cloud model of 2.2 was trained on the three-class dataset (chairs, cars and planes; 12,886 training shapes) with the same settings and number of iterations, and compared with the model trained on chairs only.

F1@0.05 of the point modelChairs (678)Cars (368)Planes (405)All three (1,451)
Trained on chairs only77.066.064.970.8
Trained on three classes77.792.795.786.5

Scores vary by a few tenths between evaluation runs, because each run uses a randomly chosen view of every test shape.

F1 score of the three-class model on the chair test set.
F1 score of the three-class model on the chair test set.

Same test image for both models

Each GIF shows the input RGB, the predicted point cloud and the ground-truth mesh.

Plane: trained on three classes
Plane: trained on three classes
Plane: trained on chairs only
Plane: trained on chairs only
Car: trained on three classes
Car: trained on three classes
Car: trained on chairs only
Car: trained on chairs only
Chair: trained on three classes
Chair: trained on three classes
Chair: trained on chairs only
Chair: trained on chairs only