Assignment 2 - Chan WangSing

All models use a pretrained ResNet18 image encoder (512-D feature), batch size 8 and learning rate 4e-4. All F1 scores are on the 678 test chairs. The evaluation is seeded, so every model sees the same camera view of each test chair and the example figures show the same three chairs throughout.

1. Exploring Loss Functions

1.1 Fitting a Voxel Grid

I implemented the voxel loss as binary cross-entropy with logits between the optimized 32x32x32 grid and the target occupancy grid. The source grid starts from random values.

1.2 Fitting a Point Cloud

I implemented the Chamfer loss from scratch using knn_points: for every source point I find its nearest target point and for every target point its nearest source point, and sum the mean squared distances of both directions.

1.3 Fitting a Mesh

I deformed an ico_sphere(4) by optimizing a per-vertex offset. The loss is the Chamfer loss on points sampled from both meshes plus 0.1 times the Laplacian smoothing loss (mesh_laplacian_smoothing), which keeps neighbouring vertices together.

2. Reconstructing 3D from Single View

2.1 Image to Voxel Grid

The decoder maps the 512-D feature with a linear layer to a 128x4x4x4 volume, then three 3D transposed convolutions (kernel 4, stride 2, each followed by BatchNorm and ReLU) upsample it to 8, 16 and 32 voxels per side, and a final 1x1x1 convolution outputs one occupancy logit per voxel.

Only about 4.4% of the voxels in a chair are occupied, so plain BCE mostly predicted empty grids. I used BCE with pos_weight = 4 on the occupied voxels. A weight of 10 overpredicted occupancy (about 13% of voxels filled), so I kept 4. The model was trained for 10k iterations. For evaluation I apply a sigmoid and marching cubes (isovalue 0.5) only to sample points for the F1 metric; the figures show the predicted voxels directly.

F1@0.05 = 52.2

Input RGB, predicted voxels and ground-truth mesh for three test chairs:

2.2 Image to Point Cloud

The decoder is an MLP (512 → 1024 → 1024 → 1000x3, ReLU between layers, no activation at the end so coordinates can be negative) whose output is reshaped to 1000 points. It is trained with the Chamfer loss against 1000 points sampled from the ground-truth mesh for 18k iterations.

F1@0.05 = 77.2

2.3 Image to Mesh

The decoder uses the same MLP to predict an offset for each of the 2562 vertices of an ico_sphere(4). The loss is Chamfer on 1000 sampled points plus w_smooth times the Laplacian smoothing loss. With the default w_smooth = 0.1 the mesh collapsed into a few huge triangles over a crumpled core (see 2.4), so the final model uses w_smooth = 3. I also initialised the last layer to zero so training starts from the clean sphere. Trained for 10k iterations.

F1@0.05 = 65.4

Comparing voxels, point clouds and meshes

RepresentationF1@0.05
Point cloud77.2
Mesh65.4
Voxel grid52.2

The point cloud scores highest because every point can move freely and the Chamfer training loss measures almost the same thing as F1. The mesh has to stay one smooth closed surface with the topology of a sphere, so it cannot open holes or form thin separate legs, and the smoothness term trades some accuracy for a valid surface. The voxel grid is limited by its 32³ resolution: thin parts are lost and marching cubes adds quantization error. As a reference, a perfect prediction scores 99.8, while the ground-truth voxels themselves only reach 81.2 through the same voxel evaluation.

2.4 Analysing the Effect of w_smooth

I trained the mesh model three times with different smoothing weights, keeping all other settings fixed (10k iterations each), and evaluated them on the same chairs.

w_smoothF1@0.05Predicted mesh
0.168.9spiky, self-intersecting: a few huge triangles over a crumpled core
365.4smooth chair-like surface with seat, backrest and leg stubs
1058.6smoothest, but a rounded blob that looks similar for every chair

w_smooth = 0.1:

w_smooth = 3:

w_smooth = 10:

F1 curves for w_smooth = 0.1, 3 and 10.

F1 drops steadily as the smoothing weight increases, because the Laplacian term pulls every vertex towards the average of its neighbours and the mesh can no longer reach sharp corners or thin parts. However, the highest F1 belongs to the worst-looking mesh. F1 is computed on points sampled from the surface in proportion to triangle area, so the w_smooth = 0.1 model covers the chair with a few huge triangles (about 90% of the surface area is in 5% of the triangles) and hides the rest in a crumpled clump that is almost never sampled. With too much smoothing (10) the mesh stays close to a sphere and loses what makes each chair different. w_smooth therefore trades F1 accuracy against surface quality, and a mesh should be judged by both, not by F1 alone. A weight of 3 gave the best balance.

2.5 Interpreting the Model

I analysed the point-cloud model (my best model) with three visualizations.

Error heatmaps

The middle column colours every ground-truth point by the distance to the nearest predicted point (what the model missed); the right column colours every predicted point by the distance to the nearest ground-truth point (points in the wrong place). Red is above the F1 threshold of 0.05.

The missed parts are the tops of the backrests, the leg tips and the thin metal frame of the lounge chair. The wrong predicted points are mostly below the seat: the model scatters points where legs might be, hedging between plausible shapes because that lowers the expected Chamfer loss.

Where the model misses the chair

Over every 5th test chair, I normalised each chair's height to 0 (floor) to 1 (top) and measured the percentage of ground-truth points with no prediction within 0.05. The result is U-shaped: about 40% of the bottom 10% (legs) and 43% of the top 10% (backrest tops) are missed, compared with 15-19% in the middle. The model reconstructs the seat and body well but misses the extremities, which vary the most between chairs.

Viewpoint test

I fed the same chair from four different views. Averaged over 23 chairs x 4 views, the Chamfer distance between predictions of the same chair from different views is 0.031, between predictions of different chairs 0.060, and between the real different chairs 0.147. The predictions are consistent across viewpoints, so the model predicts in the object's own frame rather than the camera's. But predicted chairs differ from each other only about 40% as much as real chairs do: the model is pulled strongly towards an average chair and partly recognises a typical shape instead of reconstructing every detail (as discussed by Tatarchenko et al., "What Do Single-view 3D Reconstruction Networks Learn?", CVPR 2019).

3. Exploring Other Architectures / Datasets

3.1 Implicit Network

The implicit decoder is an MLP (input → 256 → 256 → 256 → 1, ReLU) that takes the 512-D image feature together with a 3D query point and outputs an occupancy logit. I query it at every point of a 32x32x32 grid in (-1, 1)³ and reshape the answers into a 32³ grid, so it uses the same weighted BCE loss and the same evaluation as the voxel model. The query point is passed through a positional encoding (the point itself plus sin and cos at 6 frequencies, 39 values), which lets the MLP represent sharp inside/outside boundaries. Trained for 10k iterations.

F1@0.05 = 55.1 (voxel decoder: 52.2)

A short comparison of implicit decoder variants (3,000 iterations each):

DecoderF1@0.05
Voxel decoder (Q2.1)51.6
Implicit, raw xyz42.4
Implicit, 512 hidden units47.5
Implicit + positional encoding48.1
Implicit + positional encoding, 512 hidden units44.3

With raw xyz the network underfit and sometimes predicted an empty chair, because MLPs are biased towards smooth functions of their input. The positional encoding fixed this at no extra cost. After the full 10k iterations the implicit network outperforms the voxel decoder, and unlike it, it can be queried at any resolution.

3.2 Parametric Network

Following AtlasNet, the parametric decoder learns to fold a flat 2D sheet onto the chair. For each image I sample random 2D points (u, v) in [0, 1]², and a small MLP (514 → 512 → 256 → 128 → 3) maps the image feature and (u, v) to a 3D point. I use 10 patches, each with its own MLP and 100 points, giving 1000 points trained with the Chamfer loss for 18k iterations.

F1@0.05 = 74.5 (point MLP: 77.2)

A short comparison (3,000 iterations each):

DecoderF1@0.05
Point MLP (Q2.2)73.9
Parametric, 1 patch61.2
Parametric, 10 patches68.1

Each patch is a smooth function of (u, v), so its points must lie on a continuous surface. This produces more structured output (with a single patch the flat backrest and seat are clearly visible) but slightly lower F1, because the point MLP can place every point independently and hedge more freely, which Chamfer and F1 reward. More patches cover more of the chair. Because the decoder maps any (u, v) to 3D, the same trained model can be sampled at any resolution without retraining. With 10,000 points instead of 1000 it gives a much denser surface (F1 83.6, not directly comparable to the 1000-point models because more predicted points raise recall):

3.3 Extended Dataset for Training

I trained the point-cloud model on the extended dataset (chair, car and plane) with the same settings as Q2.2 (18k iterations) and evaluated both models on the same 678 test chairs. Chairs are only 47% of the 3-class training set, so the 3-class model saw about half as many chairs.

Measured on chairsTrained on chairsTrained on 3 classes
F1@0.0577.276.6
Consistency: same chair, different views0.0310.031
Diversity: different chairs, predicted0.0600.059
Real chairs' diversity (reference)0.1470.146

Predictions of the 3-class model for the same three chairs:

Left to right: misses by height for the 1-class model, for the 3-class model, and the 3-class F1 curve.

Training on three classes barely changed the chair reconstructions: F1 dropped by only 0.6 even though the model saw half as many chairs, the predictions are equally consistent across viewpoints, and they are just as un-diverse (0.06 against 0.15 for real chairs). The average-chair tendency therefore comes from the architecture (one global feature decoded by an MLP) rather than from training on a single class. The small loss appears at the extremities, mainly the backrest tops (about 47% missed compared with 43%). Chairs, cars and planes look very different, so the pretrained features separate them easily and the decoder learns a separate typical shape per class without the classes competing.