Name: Junchi Fan
Course: 16-825 Learning for 3D Vision
This project studies five representations for reconstructing a 3D chair from one RGB image: voxels, point clouds, deformable meshes, an implicit occupancy function, and a parametric surface. All reported F1 scores were computed on the complete 678-model chair test split with 1,000 sampled surface points per example.
These experiments verify the losses independently of the image encoder. The source representation is optimized directly to match one target chair.
I represent the prediction as unconstrained logits and use numerically stable binary cross entropy with logits. Sigmoid is applied only for thresholding and visualization. After 5,000 optimization steps, the loss decreased from 0.965 to 0.258 and the thresholded voxel accuracy reached 100%.
I implemented symmetric squared-L2 Chamfer distance from the two directed nearest-neighbor terms. PyTorch3D KNN is used only for nearest-neighbor search; the Chamfer reduction is assembled explicitly. With 5,000 points, the loss decreased from 2.1210 to 0.000117.
The source is an icosphere whose vertex offsets are optimized with Chamfer distance plus Laplacian and adjacent-face normal regularization. The total loss decreased from 1.0497 to 0.000441 in the direct fitting experiment. The fixed sphere topology limits its ability to exactly reproduce disconnected legs, openings, and thin supports.
All three primary models use an ImageNet-pretrained ResNet18 encoder. RGB images are normalized with the ImageNet statistics, and the resulting 512-dimensional feature is passed to a representation-specific decoder.
The voxel decoder reshapes a linear projection of the image feature into a small 3D feature block and upsamples it with four transposed-convolution stages to produce a \(32^3\) logit grid. Because occupied voxels are sparse, I used a positive-class BCE weight of 3. This prevented thin chairs from collapsing to an empty prediction.
Full-test F1@0.05: 71.5313

The point decoder is an MLP that directly predicts 1,000 bounded xyz coordinates. It is trained using the symmetric Chamfer loss from Section 1.2. The lack of topology gives each point freedom to cover thin structures, although it can also produce local point clusters.
Full-test F1@0.05: 75.1925

The mesh decoder predicts bounded offsets for every vertex of a level-4 icosphere. Training combines surface Chamfer distance with Laplacian and adjacent-face normal consistency. Stronger regularization reduces foldovers but also limits coverage of thin and disconnected chair components.
Full-test F1@0.05: 60.5360

| Representation | F1@0.05 | Main strength | Main limitation |
|---|---|---|---|
| Point cloud | 75.1925 | Flexible coverage of thin parts | No connectivity; local clustering |
| Voxel grid | 71.5313 | Stable occupancy learning | \(32^3\) discretization blurs thin geometry |
| Mesh | 60.5360 | Continuous connected surface | Fixed spherical topology causes folds and cannot create holes |
The point cloud performs best because F1 is computed from sampled surface distances and independent points can move directly toward those surfaces. Voxels are robust but quantized. Meshes provide an explicit surface, yet the fixed topology and smoothness constraint trade geometric coverage for regularity.
I varied the mesh smoothness formulation while keeping the encoder, decoder, sampling count, and test protocol fixed.
| Setting | F1@0.05 | Observation |
|---|---|---|
| \(w_{smooth}=0.1\), Laplacian only | 68.4820 | Better surface coverage, but severe spikes and triangle foldovers |
| \(w_{smooth}=1.0\), Laplacian + normal consistency | 60.5360 | Smoother connected body and fewer foldovers, but lower fine-part recall |
This experiment shows that Chamfer/F1 alone can reward geometrically invalid meshes: folded triangles may place sampled points close to the target. Increasing smoothness improves surface quality but lowers F1 because the spherical topology cannot smoothly split into separate legs and armrests. The useful conclusion is that mesh evaluation should combine point-distance metrics with topology-aware or visual checks.
The three primary decoders expose different inductive biases on the same input images:
The threshold curves also reveal that all methods improve rapidly as the distance tolerance increases from 0.01 to 0.05. This means their coarse global geometry is substantially better than their fine-scale alignment.
I created a normalized \(32\times32\times32\) coordinate grid in \((-1,1)^3\). For every coordinate, a shared MLP receives the coordinate concatenated with a projected image feature and predicts an occupancy logit. Queries are processed in chunks to limit memory. This coordinate-conditioned decoder is compact and resolution-independent in principle, but the simple MLP tends to learn a smooth category-average volume.
Full-test F1@0.05: 43.7611

Compared with the convolutional voxel decoder, the implicit model has a weaker local spatial prior. A deeper network, positional encoding, or multi-scale image features would likely improve its ability to recover thin chair parts.
I sample 1,000 points from a regular 2D square parameter domain. A shared MLP maps each \((u,v)\) coordinate, conditioned on the image feature, to a 3D point. The ordered sampling visibly preserves the continuous 2D parameterization.
Full-test F1@0.05: 53.9258

The parametric model is more structured than the independent point decoder, but one 2D patch must stretch and fold to represent multiple chair components. Multiple learned patches would better represent disconnected legs, cushions, and armrests.
| Method | F1@0.05 |
|---|---|
| Direct point decoder | 75.1925 |
| Convolutional voxel decoder | 71.5313 |
| Regularized deformable mesh | 60.5360 |
| Parametric 2D-to-3D decoder | 53.9258 |
| Implicit coordinate decoder | 43.7611 |
The experiments demonstrate that representation choice changes both optimization behavior and failure mode. Flexible explicit points maximize the point-based metric, while structured representations provide occupancy or connectivity at the cost of geometric freedom.