Assignment 2: Single View to 3D

Shobhit Arora · (shobhit2)
16-825: Learning for 3D Vision

1. Exploring Loss Functions

Each animation shows the optimized shape (blue, left) next to the target (gray, right), fitted for 100k steps.

1.1 Fitting a voxel grid

Loss: binary cross-entropy with logits. Final loss: 7.09 × 10⁻²

Fitted voxel grid next to the target voxel grid
Fitted voxels (left) vs. target voxels (right).

1.2 Fitting a point cloud

Loss: chamfer distance, implemented with knn_points (mean squared nearest-neighbour distance in both directions). Final loss: 1.88 × 10⁻⁴

Fitted point cloud next to the target point cloud
Fitted points (left) vs. target points (right).

1.3 Fitting a mesh

Loss: chamfer distance on sampled points + 0.1 × Laplacian smoothing. Final loss: 5.25 × 10⁻⁴

Fitted mesh next to the target mesh
Fitted mesh (left) vs. target mesh (right).

2.1 Image to Voxel Grid

Decoder: MLP 512 → 1024 (ReLU, dropout 0.3) → 32³ occupancy logits. Loss: BCE with (pos_weight = 5). The encoder is fine-tuned (lr 1e-4 for 50k steps, then lr 3e-5 for 50k more). The surface is extracted with marching cubes at threshold 0.8 (Tried different threshold, but 0.8 worked best., explained in 2.4 and 3.1).

F1@0.05 = 70.05 (precision 73.57, recall 69.01)

F1 vs. distance threshold (voxels).
F1 vs. distance threshold (voxels).

With frozen features, three different decoders (transposed convolutions from 2³, transposed convolutions from 4³, and an MLP) all plateaued at F1 45-48. Fine-tuning the encoder broke this plateau:

Voxel modelF1@0.05
frozen features 45–48
fine-tuned, 50k steps, lr 1e-4 (threshold 0.5)66.6
+ 50k steps, lr 3e-5 (threshold 0.5)68.7
+ threshold 0.870.1
ground-truth voxels through the same evaluation (upper bound)~79

2.2 Image to Point Cloud

Decoder: MLP 512 → 1024 → 2048 → 1000×3 (ReLU). Loss: chamfer distance. Frozen features, lr 1e-4, batch size 32, 10k steps.

F1@0.05 = 76.62 (precision 77.75, recall 77.50)

F1 vs. distance threshold (points).
F1 vs. distance threshold (points).

2.3 Image to Mesh

Decoder: MLP 512 → 1024 → 2048 → V×3 (BatchNorm + ReLU). Loss: chamfer + 0.1 × Laplacian smoothing. Frozen features, lr 4e-4, batch size 32, 10k steps.

F1@0.05 = 70.08 (precision 66.68, recall 74.96)

F1 vs. distance threshold (mesh).
F1 vs. distance threshold (mesh).

Comparison: voxel vs. point cloud vs. mesh

RepresentationEncoderPrecisionRecallF1@0.05
Voxelfine-tuned73.5769.0170.05
Point cloudfrozen77.7577.5076.62
Meshfrozen66.6874.9670.08

My Objservation: With the same frozen features the ranking is point (76.6) > mesh (70.1) >> voxel (45-48); fine-tuning lifts voxels to 70. The precision/recall split explains why:

2.4 Effect of Hyperparameters: Number of Points

I varied n_points for the point-cloud model (frozen features, same decoder, loss, learning rate and 10k steps); only the number of predicted points changes.

n_pointsPrecisionRecallF1@0.05
50071.9768.3969.30
100077.7577.5076.62
500085.5489.5385.62

F1 rises with the number of points, and recall rises most (68 → 78 → 90): with just 500 points the thin parts (legs, bars) get few or no points whci his why the loss is high, while with 5000 the model can afford points everywhere on the surface.

One thing is that the eval_model.py samples as many ground-truth points as predicted points, so with 5000 points both shapes are denser and it is easier for every point to find a neighbour within 0.05. So part of this gain is therefore a density effect of the metric, not only a better model.

Another hyperparameter that mattered a lot is the occupancy threshold for voxel-type outputs (see 3.1): with the same checkpoint, raising it from 0.5 to 0.8 improved the implicit model from 63.1 to 76.1 and the voxel model from 68.7 to 70.1.

2.5 Interpreting the Model: Where Does It Fail?

Here I have tried to visualize the precision and recall using 2 colors on the GT and predicted sample. I split the F-score into its two parts and show them on the shape:

Input, error-colored prediction and error-colored ground truth, sample 0
Input | prediction colored by error | ground truth colored by error (sample 0).
Input, error-colored prediction and error-colored ground truth, sample 250
Sample 250.
Input, error-colored prediction and error-colored ground truth, sample 500
Sample 500.

Observations: As we can see, most of the ground truth is yellow, so the model covers the main body of each chair, but the red regions show what it misses. In Sample 0 (a wide armchair) the red is spread over the armrests and sides of the back, while in Sample 250 it is concentrated on the unusual pedestal base and the top of the backrest, and in Sample 500 on the top of the tall backrest. The predictions are mostly yellow with a few red points scattered around the edges, so most predicted points lie close to the true surface and the precision errors come from a few stray points. The model therefore gets the overall chair shape right but misses the extremities and unusual parts

3.1 Implicit Network

Decoder: MLP on [image feature (512) ; (x, y, z)] → occupancy logit (515 → 512 → 1024 → 512 → 256 → 1, ReLU). Queries come from a 32×32×32 meshgrid. Loss: BCE with pos_weight = 5. The encoder is fine-tuned: steps 0–65k with uniformly sampled queries, steps 65k-100k with boundary sampling (half the queries drawn near the surface). Threshold 0.8.

F1@0.05 = 76.1 (precision 73.69, recall 79.16)

F1 vs. distance threshold (implicit).
F1 vs. distance threshold (implicit).
Implicit modelF1@0.05
frozen features45–50
fine-tuned, 40k steps59.1
+ boundary sampling (threshold 0.5)63.1
threshold 0.6 / 0.7 / 0.868.3 / 72.6 / 76.1

With pos_weight = 5 the model predicts "occupied" even when unsure, so probabilities just outside the true surface exceed 0.5 and the extracted surface is too thick.

3.2 Parametric Network

For each of N random 2D samples (u, v) ∈ [0, 1]², the MLP maps [image feature (512) ; u ; v] → 1024 → 512 → 256 → 3 → tanh to one 3D point. N = 1000. Loss: chamfer. Frozen features.

F1@0.05 = 69.86 (precision 74.28, recall 67.50)

F1 vs. distance threshold (parametric).
F1 vs. distance threshold (parametric).

With the same frozen features, the single-patch parametric decoder scores below the point decoder (69.9 vs. 76.6) This matches AtlasNet, where a single patch was slightly worse than a point baseline on chairs and many patches were clearly better. An advntage of the parametric decoder is that it is resolution-independent: the same trained model can be used to output any number of points without retraining, which the point decoder cannot.

3.3 Extended Dataset for Training

I trained the point-cloud model from 2.2 (identical architecture and settings) on three classes (chair, car, plane: 12,886 training models, of which 6,100 chairs) and compared it with the chair-only model on the same chair test set.

ModelTest setPrecisionRecallF1@0.05
chair-only (2.2)chairs (678)77.7577.5076.62
three-classchairs (678)78.1977.4576.57
three-classplanes (405)96.3995.5495.84
three-classcars (368)92.4893.6692.89

Same chairs, two models

Three-class model on planes and cars

Quantitatively, chair reconstruction is unchanged (76.62 vs. 76.57), even though chairs are only about 47% of the training data, so each chair is seen roughly half as often in the same 10k steps: the extra classes did not take capacity away from chairs. Planes and cars are much easier (95.8 / 92.9): they have little variety within the class(cars are box-like, planes share one layout), compact/smooth geometry without thin legs or holes, and the class is obvious from the image. Chairs vary enormously (stools, armchairs, office chairs) and have thin parts. Moreover, the difference between chair vs cars and planes is alot feature wise, therefore, the model remains the same. Qualitatively also, the model performance is similar to the previous 1 class training. There's no visible difference. Maybe, not sure, the model is able to distinguish bit better in multiclass training and therefore, the boundaries or the overall structure of the object might seem to be more well defined.