Single-View 3D Reconstruction

16-825 Assignment 2 · Ashutosh Panda · Andrew ID: apanda2

We study three representations for reconstructing a 3D object from one RGB image: voxel occupancy, a point cloud, and a deformable mesh. We first fit each representation directly to a target, then train image-conditioned models and evaluate their reconstructions.

Q1 · Exploring loss functions

We optimized source shapes to match target shapes using binary cross-entropy for voxels, a bidirectional Chamfer distance for point clouds, and Chamfer distance plus a mesh smoothness penalty for meshes. Each representation was fit until its loss reached zero at the displayed precision; the figures show the fitted result beside its target.

Fitted voxel shape next to its target
Voxel fitting: fitted occupancy and target.
Fitted point cloud next to its target
Point-cloud fitting: fitted points follow the target structure.
Fitted mesh next to its target
Mesh fitting: the fitted surface captures the target's main shape.

Q2 · Reconstructing 3D from one image

Each model uses a ResNet-18 image encoder with a representation-specific decoder. The voxel model predicts a 32³ occupancy grid; the point model predicts 1,000 points; and the mesh model deforms an icosphere template. Runs were continued from 5,000 to 11,000 total updates using seed 42 and batch size 32. Evaluation used 678 chair test shapes.

RepresentationF1@0.05EvaluatedSkipped
Voxel63.955677 / 6781 empty predicted surface
Point cloud80.476678 / 6780
Mesh (`w_smooth=0.1`)72.515678 / 6780

The point-cloud model has the highest F1, followed by mesh and voxel. Chamfer supervision directly aligns sampled surface points, while the voxel model represents shape on a finite grid and must convert its predicted occupancy to a surface for evaluation. That grid can lose fine detail and occasionally fail to produce a surface at the 0.5 threshold. The mesh decoder has a fixed template topology and smoothness regularization, which can limit how well it captures detailed or unusual chair geometry. These are plausible explanations for this run's ranking, not universal claims about the representations.

Voxel F1 as a function of distance threshold
Voxel F1 curve. One empty predicted surface was excluded from the reported mean.
Point-cloud F1 as a function of distance threshold
Point-cloud F1 curve.
Mesh F1 as a function of distance threshold
Mesh F1 curve (`w_smooth=0.1`).

Test examples

Each panel shows the input image, the model prediction, and its ground-truth mesh (the point-cloud panel also shows target points).

Voxel

Voxel reconstruction example 1
Voxel reconstruction example 2
Voxel reconstruction example 3

Point cloud

Point-cloud reconstruction example 1
Point-cloud reconstruction example 2
Point-cloud reconstruction example 3

Mesh

Mesh reconstruction example 1
Mesh reconstruction example 2
Mesh reconstruction example 3

Q2.4 · Effect of mesh smoothness

We compare the same mesh model trained for 11,000 updates with smoothness weights 0.1 and 0.5. Increasing the weight reduces F1 from 72.515 to 70.582 (−1.933 points). This is consistent with a regularization tradeoff: stronger smoothness can suppress detail as it penalizes local surface variation. The saved examples remain visibly imperfect for both settings, so the score change is clearer than the visual smoothness difference in these three cases.

F1 curve for mesh smoothness weight 0.1
Baseline: `w_smooth=0.1`, F1@0.05 72.515.
F1 curve for mesh smoothness weight 0.5
Higher smoothness: `w_smooth=0.5`, F1@0.05 70.582.
Mesh smoothness 0.5 example 1
Mesh smoothness 0.5 example 2
Mesh smoothness 0.5 example 3

Q2.5 · Input-occlusion sensitivity

We occlude each cell of a 4×4 image grid and measure the mean absolute change in predicted voxel occupancy. The heatmap highlights image patches whose removal changes the prediction most. This is a sensitivity visualization: it indicates which image regions affect the output, but does not establish that a region is causally or semantically important.

Input image, predicted occupancy projection, and patch-occlusion sensitivity map
The occlusion figure was carried forward from the earlier voxel run; it was not regenerated after the 11,000-update continuation.

Q3.3 · Training on one class versus three

We compare voxel models trained on chairs only and on chairs, cars, and airplanes. Evaluation uses the same 678 chair test IDs and fixed input view 0. The two models both produced surfaces for 661 of the same shapes; the comparison below uses only that paired valid set.

ModelF1@0.05 on paired setScored / 678Empty-surface skips
One class (chairs)64.8546771
Three classes57.02666216

On the 661 shared valid shapes, three-class training is 7.828 F1 points lower (paired 95% CI for three-class minus one-class: [−9.546, −6.110]); three-class training scores higher on 256 of the 661 shapes. In these runs, adding classes did not improve chair reconstruction and instead reduced the paired mean. The broader data distribution may make chair reconstruction harder at a fixed training budget, but this experiment does not isolate the cause. Skips are reported separately because the models did not produce evaluable surfaces for every test shape.

Single-class voxel F1 curve
Chair-only model. Whole valid-set F1@0.05: 63.818.
Three-class voxel F1 curve
Three-class model evaluated on chairs. Whole valid-set F1@0.05: 56.962.

The curve captions show each run's mean over its own valid predictions; the paired numbers above are the fair same-shape comparison.

Matched chair examples

Rows in the two sets use the same chair IDs and input views.

Single-class example 1
Single-class · example 1
Three-class example 1
Three-class · example 1
Single-class example 2
Single-class · example 2
Three-class example 2
Three-class · example 2
Single-class example 3
Single-class · example 3
Three-class example 3
Three-class · example 3

Notes and limitations