Assignment 2: Single-View to 3D Reconstruction

Name: Junchi Fan
Course: 16-825 Learning for 3D Vision

This project studies five representations for reconstructing a 3D chair from one RGB image: voxels, point clouds, deformable meshes, an implicit occupancy function, and a parametric surface. All reported F1 scores were computed on the complete 678-model chair test split with 1,000 sampled surface points per example.

1. Fitting a Single 3D Example

These experiments verify the losses independently of the image encoder. The source representation is optimized directly to match one target chair.

1.1 Fitting a Voxel Grid

I represent the prediction as unconstrained logits and use numerically stable binary cross entropy with logits. Sigmoid is applied only for thresholding and visualization. After 5,000 optimization steps, the loss decreased from 0.965 to 0.258 and the thresholded voxel accuracy reached 100%.

Optimized and target voxel grids

1.2 Fitting a Point Cloud

I implemented symmetric squared-L2 Chamfer distance from the two directed nearest-neighbor terms. PyTorch3D KNN is used only for nearest-neighbor search; the Chamfer reduction is assembled explicitly. With 5,000 points, the loss decreased from 2.1210 to 0.000117.

Optimized and target point clouds

1.3 Fitting a Mesh

The source is an icosphere whose vertex offsets are optimized with Chamfer distance plus Laplacian and adjacent-face normal regularization. The total loss decreased from 1.0497 to 0.000441 in the direct fitting experiment. The fixed sphere topology limits its ability to exactly reproduce disconnected legs, openings, and thin supports.

Optimized and target meshes

2. Reconstructing 3D from a Single View

All three primary models use an ImageNet-pretrained ResNet18 encoder. RGB images are normalized with the ImageNet statistics, and the resulting 512-dimensional feature is passed to a representation-specific decoder.

2.1 Image to Voxel Grid

The voxel decoder reshapes a linear projection of the image feature into a small 3D feature block and upsamples it with four transposed-convolution stages to produce a \(32^3\) logit grid. Because occupied voxels are sparse, I used a positive-class BCE weight of 3. This prevented thin chairs from collapsing to an empty prediction.

Full-test F1@0.05: 71.5313

Voxel F1 curve

2.2 Image to Point Cloud

The point decoder is an MLP that directly predicts 1,000 bounded xyz coordinates. It is trained using the symmetric Chamfer loss from Section 1.2. The lack of topology gives each point freedom to cover thin structures, although it can also produce local point clusters.

Full-test F1@0.05: 75.1925

Point-cloud F1 curve

2.3 Image to Mesh

The mesh decoder predicts bounded offsets for every vertex of a level-4 icosphere. Training combines surface Chamfer distance with Laplacian and adjacent-face normal consistency. Stronger regularization reduces foldovers but also limits coverage of thin and disconnected chair components.

Full-test F1@0.05: 60.5360

Mesh F1 curve

Quantitative Comparison

Representation F1@0.05 Main strength Main limitation
Point cloud 75.1925 Flexible coverage of thin parts No connectivity; local clustering
Voxel grid 71.5313 Stable occupancy learning \(32^3\) discretization blurs thin geometry
Mesh 60.5360 Continuous connected surface Fixed spherical topology causes folds and cannot create holes

The point cloud performs best because F1 is computed from sampled surface distances and independent points can move directly toward those surfaces. Voxels are robust but quantized. Meshes provide an explicit surface, yet the fixed topology and smoothness constraint trade geometric coverage for regularity.

2.4 Hyperparameter Analysis: Mesh Regularization

I varied the mesh smoothness formulation while keeping the encoder, decoder, sampling count, and test protocol fixed.

Setting F1@0.05 Observation
\(w_{smooth}=0.1\), Laplacian only 68.4820 Better surface coverage, but severe spikes and triangle foldovers
\(w_{smooth}=1.0\), Laplacian + normal consistency 60.5360 Smoother connected body and fewer foldovers, but lower fine-part recall

This experiment shows that Chamfer/F1 alone can reward geometrically invalid meshes: folded triangles may place sampled points close to the target. Increasing smoothness improves surface quality but lowers F1 because the spherical topology cannot smoothly split into separate legs and armrests. The useful conclusion is that mesh evaluation should combine point-distance metrics with topology-aware or visual checks.

2.5 Interpreting the Learned Models

The three primary decoders expose different inductive biases on the same input images:

The threshold curves also reveal that all methods improve rapidly as the distance tolerance increases from 0.01 to 0.05. This means their coarse global geometry is substantially better than their fine-scale alignment.

3. Exploring Other Architectures

3.1 Implicit Occupancy Network

I created a normalized \(32\times32\times32\) coordinate grid in \((-1,1)^3\). For every coordinate, a shared MLP receives the coordinate concatenated with a projected image feature and predicts an occupancy logit. Queries are processed in chunks to limit memory. This coordinate-conditioned decoder is compact and resolution-independent in principle, but the simple MLP tends to learn a smooth category-average volume.

Full-test F1@0.05: 43.7611

Implicit-network F1 curve

Compared with the convolutional voxel decoder, the implicit model has a weaker local spatial prior. A deeper network, positional encoding, or multi-scale image features would likely improve its ability to recover thin chair parts.

3.2 Parametric Surface Network

I sample 1,000 points from a regular 2D square parameter domain. A shared MLP maps each \((u,v)\) coordinate, conditioned on the image feature, to a 3D point. The ordered sampling visibly preserves the continuous 2D parameterization.

Full-test F1@0.05: 53.9258

Parametric-network F1 curve

The parametric model is more structured than the independent point decoder, but one 2D patch must stretch and fold to represent multiple chair components. Multiple learned patches would better represent disconnected legs, cushions, and armrests.

Final Summary

Method F1@0.05
Direct point decoder 75.1925
Convolutional voxel decoder 71.5313
Regularized deformable mesh 60.5360
Parametric 2D-to-3D decoder 53.9258
Implicit coordinate decoder 43.7611

The experiments demonstrate that representation choice changes both optimization behavior and failure mode. Flexible explicit points maximize the point-based metric, while structured representations provide occupancy or connectivity at the cost of geometric freedom.