Assignment 2: Single View to 3D
1. Exploring Loss Functions
Each animation shows the optimized shape (blue, left) next to the target (gray, right), fitted for 100k steps.
1.1 Fitting a voxel grid
Loss: binary cross-entropy with logits. Final loss: 7.09 × 10⁻²
1.2 Fitting a point cloud
Loss: chamfer distance, implemented with knn_points (mean squared nearest-neighbour distance in both directions). Final loss: 1.88 × 10⁻⁴
1.3 Fitting a mesh
Loss: chamfer distance on sampled points + 0.1 × Laplacian smoothing. Final loss: 5.25 × 10⁻⁴
2.1 Image to Voxel Grid
Decoder: MLP 512 → 1024 (ReLU, dropout 0.3) → 32³ occupancy logits. Loss: BCE with (pos_weight = 5). The encoder is fine-tuned (lr 1e-4 for 50k steps, then lr 3e-5 for 50k more). The surface is extracted with marching cubes at threshold 0.8 (Tried different threshold, but 0.8 worked best., explained in 2.4 and 3.1).
F1@0.05 = 70.05 (precision 73.57, recall 69.01)
With frozen features, three different decoders (transposed convolutions from 2³, transposed convolutions from 4³, and an MLP) all plateaued at F1 45-48. Fine-tuning the encoder broke this plateau:
| Voxel model | F1@0.05 |
|---|---|
| frozen features | 45–48 |
| fine-tuned, 50k steps, lr 1e-4 (threshold 0.5) | 66.6 |
| + 50k steps, lr 3e-5 (threshold 0.5) | 68.7 |
| + threshold 0.8 | 70.1 |
| ground-truth voxels through the same evaluation (upper bound) | ~79 |
2.2 Image to Point Cloud
Decoder: MLP 512 → 1024 → 2048 → 1000×3 (ReLU). Loss: chamfer distance. Frozen features, lr 1e-4, batch size 32, 10k steps.
F1@0.05 = 76.62 (precision 77.75, recall 77.50)
2.3 Image to Mesh
Decoder: MLP 512 → 1024 → 2048 → V×3 (BatchNorm + ReLU). Loss: chamfer + 0.1 × Laplacian smoothing. Frozen features, lr 4e-4, batch size 32, 10k steps.
F1@0.05 = 70.08 (precision 66.68, recall 74.96)
Comparison: voxel vs. point cloud vs. mesh
| Representation | Encoder | Precision | Recall | F1@0.05 |
|---|---|---|---|---|
| Voxel | fine-tuned | 73.57 | 69.01 | 70.05 |
| Point cloud | frozen | 77.75 | 77.50 | 76.62 |
| Mesh | frozen | 66.68 | 74.96 | 70.08 |
My Objservation: With the same frozen features the ranking is point (76.6) > mesh (70.1) >> voxel (45-48); fine-tuning lifts voxels to 70. The precision/recall split explains why:
- Points (precision ≈ recall): every point moves freely, with no resolution or topology constraint, maybe that is why the error is low.
- Voxels (precision > recall): it is mostly due to missing geometry. A 32³ cell is about 3 cm wide, so thin legs and bars disappear, and the voxel-to-surface conversion is lossy and just bad overall
- Mesh (recall > precision): I guess the extra geometry explains this. A deformed sphere has no holes, so it covers the chair but stretches "webbing" between the legs and under the seat, which makes our structure look pointy.
2.4 Effect of Hyperparameters: Number of Points
I varied n_points for the point-cloud model (frozen features, same decoder,
loss, learning rate and 10k steps); only the number of predicted points changes.
| n_points | Precision | Recall | F1@0.05 |
|---|---|---|---|
| 500 | 71.97 | 68.39 | 69.30 |
| 1000 | 77.75 | 77.50 | 76.62 |
| 5000 | 85.54 | 89.53 | 85.62 |
F1 rises with the number of points, and recall rises most (68 → 78 → 90): with just 500 points the thin parts (legs, bars) get few or no points whci his why the loss is high, while with 5000 the model can afford points everywhere on the surface.
One thing is that the eval_model.py samples as many ground-truth points as predicted points, so with
5000 points both shapes are denser and it is easier for every point to find a neighbour within 0.05.
So part of this gain is therefore a density effect of the metric, not only a better model.
Another hyperparameter that mattered a lot is the occupancy threshold for voxel-type outputs (see 3.1): with the same checkpoint, raising it from 0.5 to 0.8 improved the implicit model from 63.1 to 76.1 and the voxel model from 68.7 to 70.1.
2.5 Interpreting the Model: Where Does It Fail?
Here I have tried to visualize the precision and recall using 2 colors on the GT and predicted sample. I split the F-score into its two parts and show them on the shape:
- Middle: prediction colored by distance to the ground truth. Red = extra or wrong points (precision).
- Right: ground truth colored by distance to the prediction. Red = parts the model missed (recall).
- Yellow = within 0.05, red = 0.05 or more (the same threshold as F1@0.05).
Observations: As we can see, most of the ground truth is yellow, so the model covers the main body of each chair, but the red regions show what it misses. In Sample 0 (a wide armchair) the red is spread over the armrests and sides of the back, while in Sample 250 it is concentrated on the unusual pedestal base and the top of the backrest, and in Sample 500 on the top of the tall backrest. The predictions are mostly yellow with a few red points scattered around the edges, so most predicted points lie close to the true surface and the precision errors come from a few stray points. The model therefore gets the overall chair shape right but misses the extremities and unusual parts
3.1 Implicit Network
Decoder: MLP on [image feature (512) ; (x, y, z)] → occupancy logit
(515 → 512 → 1024 → 512 → 256 → 1, ReLU). Queries come from a 32×32×32 meshgrid.
Loss: BCE with pos_weight = 5.
The encoder is fine-tuned: steps 0–65k with uniformly sampled queries, steps 65k-100k with
boundary sampling (half the queries drawn near the surface). Threshold 0.8.
F1@0.05 = 76.1 (precision 73.69, recall 79.16)
| Implicit model | F1@0.05 |
|---|---|
| frozen features | 45–50 |
| fine-tuned, 40k steps | 59.1 |
| + boundary sampling (threshold 0.5) | 63.1 |
| threshold 0.6 / 0.7 / 0.8 | 68.3 / 72.6 / 76.1 |
With pos_weight = 5 the model predicts "occupied" even when unsure,
so probabilities just outside the true surface exceed 0.5 and the extracted surface is too thick.
3.2 Parametric Network
For each of N random 2D samples (u, v) ∈ [0, 1]², the MLP maps [image feature (512) ; u ; v] → 1024 → 512 → 256 → 3 → tanh to one 3D point. N = 1000. Loss: chamfer. Frozen features.
F1@0.05 = 69.86 (precision 74.28, recall 67.50)
With the same frozen features, the single-patch parametric decoder scores below the point decoder (69.9 vs. 76.6) This matches AtlasNet, where a single patch was slightly worse than a point baseline on chairs and many patches were clearly better. An advntage of the parametric decoder is that it is resolution-independent: the same trained model can be used to output any number of points without retraining, which the point decoder cannot.
3.3 Extended Dataset for Training
I trained the point-cloud model from 2.2 (identical architecture and settings) on three classes (chair, car, plane: 12,886 training models, of which 6,100 chairs) and compared it with the chair-only model on the same chair test set.
| Model | Test set | Precision | Recall | F1@0.05 |
|---|---|---|---|---|
| chair-only (2.2) | chairs (678) | 77.75 | 77.50 | 76.62 |
| three-class | chairs (678) | 78.19 | 77.45 | 76.57 |
| three-class | planes (405) | 96.39 | 95.54 | 95.84 |
| three-class | cars (368) | 92.48 | 93.66 | 92.89 |
Same chairs, two models
Three-class model on planes and cars
Quantitatively, chair reconstruction is unchanged (76.62 vs. 76.57), even though chairs are only about 47% of the training data, so each chair is seen roughly half as often in the same 10k steps: the extra classes did not take capacity away from chairs. Planes and cars are much easier (95.8 / 92.9): they have little variety within the class(cars are box-like, planes share one layout), compact/smooth geometry without thin legs or holes, and the class is obvious from the image. Chairs vary enormously (stools, armchairs, office chairs) and have thin parts. Moreover, the difference between chair vs cars and planes is alot feature wise, therefore, the model remains the same. Qualitatively also, the model performance is similar to the previous 1 class training. There's no visible difference. Maybe, not sure, the model is able to distinguish bit better in multiclass training and therefore, the boundaries or the overall structure of the object might seem to be more well defined.