Project 2: Single View to 3D

By Teja Kanthamneni · 16-825 Learning for 3D Vision

Fitting 3D representations and reconstructing chairs from one RGB image.

1. Exploring Loss Functions

These experiments fit a 3D representation directly to a target. Each pair shows the optimized result and ground truth.

1.1 Fitting a voxel grid

Binary cross entropy loss is used to predict the occupancy value of each voxel.

1.2 Fitting a point cloud

Symmetric Chamfer distance loss is used to match each predicted point to its nearest target point and each target point to its nearest prediction.

1.3 Fitting a mesh

Combination of Chamfer distance and Laplacian smoothing make up the loss for the predicted mesh.

2. Reconstructing 3D from a Single View

2.1 Image to voxel grid

The voxel decoder consists of several nn.ConvTranspose3d layers. At a distance threshold of 0.05, the mean F1 score is 80.51%.

Test example 0

Test example 338

Test example 677

Voxel reconstruction mean F1 versus distance threshold
Voxel reconstruction: mean F1 versus threshold.

2.2 Image to point cloud

The point cloud decoder consists of several widening MLP layers to predict 10000 points. At 0.05, the mean F1 score is 76.81%.

Test example 0

Test example 338

Test example 677

Point cloud reconstruction mean F1 versus distance threshold
Point cloud reconstruction: mean F1 versus threshold.

2.3 Image to mesh

The mesh decoder aslo uses several widening MLP layers to predict a deformation of a mesh. At 0.05, the mean F1 score is 71.38% for smoothness weight 0.1.

Test example 0

Test example 338

Test example 677

Mesh reconstruction mean F1 versus distance threshold
Mesh reconstruction: mean F1 versus threshold.

Representation comparison

RepresentationF1 at 0.05What the representation can capture
Voxel grid80.51%Occupancy and topology without a fixed mesh template; limited by grid resolution.
Point cloud76.81%Flexible surface samples; no explicit faces or continuous surface.
Mesh (smoothness 0.1)71.38%Connected surface; the starting topology and smoothness term restrict thin or separate parts.

F1@0.05 counts an evaluation point as matched when its nearest point on the other shape is less than 0.05 away in 3D reference coordinates. In these runs, the voxel model likely scores highest because it predicts occupancy throughout a 3D grid without a fixed surface topology. The point cloud can place points freely on chair parts, but its finite samples leave gaps. The mesh starts from a connected sphere and is deformed into the target shape. The nature of this process makes it difficult to capture smooth surfaces, even with the smoothness loss term. Additionally the smoothness loss can also be working against the model by marking harder to capture edges, corners, and then components.

2.4 Effect of the Mesh Smoothness Weight

The mesh experiments vary the weight on the smoothness loss w_smooth. F1@0.05 decreases as the weight increases across these five runs. A stronger penalty can suppress narrow legs and other fine chair details that help surface overlap. While w_smooth = 0 performed the best, the surfaces, as expected, were quite jagged. There may be some middle ground between 0 and 0.1 for the smoothness weight, but given that with w_smooth = 0.1 the predicted mesh is still not very smooth, it is likely that wouldn't result in much improvement.

Smoothness weightMean F1 at 0.05
074.24%
0.171.38%
0.569.75%
168.15%
561.50%

Mean scores are from each run's saved evaluation CSV, using 1,000 sampled points.

F1 curves

Example predictions

2.5 Interpreting the Models

Each caption reads colored shape → comparison target. For every visible vertex or point on the colored shape, we measure the 3D Euclidean distance to the nearest location on the target. A mesh target uses its triangle surfaces and a point-cloud target uses its nearest discrete point. A voxel-grid target is first converted to a triangle boundary surface; faces shared by two occupied voxels are removed. It is then treated like a mesh target. Mesh and voxel colors are computed at vertices and interpolated across faces, while point-cloud colors are assigned to individual points.

ColorDistance to the comparison target
Green0: the colored geometry lies on the target.
Yellow0.05 units away.
Red0.10 units away or farther.

Colors blend between those values and stop changing beyond 0.10. Intuitively, red on a prediction captures incorrect geometry in the prediction, while red on the ground truth captures geometry the prediction misses. All views use test sample 0, view 0.

Voxel model

Point cloud model

Mesh model

Overall, this visualization technique is good at capturing areas of the ground truth mesh that the models struggled to capture and the excesses created by the model that don't exist in the ground truth. While this information could be determined without this gradient texture, highlighting the differences makes them significantly easier to determine.

3. Exploring Other Architectures

3.2 Parametric point cloud decoder

This parametric point cloud model learns 25 different mappings that morph points from 2D planes into the object's 3D shape. We sample points on each of these planes during training and evaluation. This configuration allows us to sample different numbers of points at inference, unlike the original point cloud model, which is limited to the fixed number of points it was trained to predict. At 1,000 sampled points, its mean F1 at 0.05 is 75.76%, which is comparable to that of the original point cloud model. At 2,500, 5,000, and 10,000 sampled points, mean F1 at 0.05 increases to 76.86%, 78.83%, and 79.02%, respectively. However, this parametric model took about twice as long to train as the original.

Test example 0000

Input RGB chair for the 1,000-point run, example 0000
Input RGB (1,000-point run)
Ground truth rotating chair mesh, example 0000
Ground truth mesh
Parametric point cloud prediction with 1,000 queried points, example 0000
1,000 queried points
Parametric point cloud prediction with 2,500 queried points, example 0000
2,500 queried points
Parametric point cloud prediction with 5,000 queried points, example 0000
5,000 queried points
Parametric point cloud prediction with 10,000 queried points, example 0000
10,000 queried points

Test example 0338

Input RGB chair for the 1,000-point run, example 0338
Input RGB (1,000-point run)
Ground truth rotating chair mesh, example 0338
Ground truth mesh
Parametric point cloud prediction with 1,000 queried points, example 0338
1,000 queried points
Parametric point cloud prediction with 2,500 queried points, example 0338
2,500 queried points
Parametric point cloud prediction with 5,000 queried points, example 0338
5,000 queried points
Parametric point cloud prediction with 10,000 queried points, example 0338
10,000 queried points

Test example 0677

Input RGB chair for the 1,000-point run, example 0677
Input RGB (1,000-point run)
Ground truth rotating chair mesh, example 0677
Ground truth mesh
Parametric point cloud prediction with 1,000 queried points, example 0677
1,000 queried points
Parametric point cloud prediction with 2,500 queried points, example 0677
2,500 queried points
Parametric point cloud prediction with 5,000 queried points, example 0677
5,000 queried points
Parametric point cloud prediction with 10,000 queried points, example 0677
10,000 queried points
Parametric point cloud mean F1 versus distance threshold for 1,000, 2,500, 5,000, and 10,000 queried points
Mean F1 versus distance threshold for all four query counts.