Assignment 2: Single View to 3D

Name: Yannick Daniel Gibson
Andrew ID: ydg

1.1. Fitting a voxel grid (4 points)

Voxel grid fitted using binary cross-entropy loss.

Optimized voxel grid and ground truth
Synchronized optimized and ground truth voxel grid rotations

1.2. Fitting a point cloud (3 points)

5,000 points initialized from N(0, 1) are fitted with symmetric squared Chamfer loss for 1,500 Adam steps.

Optimized point cloud and ground truth
Synchronized optimized and ground truth point cloud rotations

1.3. Fitting a mesh (3 points)

Level-4 sphere fitted with Chamfer + 0.1 x Laplacian loss, shown at five stages over 1,500 Adam steps.

Optimized mesh and ground truth
Synchronized optimized and ground truth mesh rotations
Fitting progress (5 stages)
Five optimization stages from the initial sphere to the fitted chair mesh

2.1. Image to voxel grid (20 points)

A fine-tuned ResNet18 and 512-1024-2048-32768 MLP predict a 32x32x32 voxel grid using weighted BCE + Dice loss (4,000 Adam steps, batch 32, validation-selected threshold 0.75), achieving 73.88% F1@0.05 on 678 test chairs.

Test example 1
Test example 1: input RGB and synchronized predicted voxel and ground truth mesh rotations
Test example 2
Test example 2: input RGB and synchronized predicted voxel and ground truth mesh rotations
Test example 3: sled-base chair
Test example 3: input RGB and synchronized predicted voxel and ground truth mesh rotations
Test-set F1
Mean test-set F1 score across distance thresholds

2.2. Image to point cloud (20 points)

ResNet18 with a 512-1024-2048-3000 MLP predicts 1,000 points using squared Chamfer loss (3,000 Adam steps, batch 32), achieving 80.13% F1@0.05 on 678 test chairs; Chamfer distance does not guarantee equal point spacing, so point clusters may occur.

Test example 1
Test example 1: input RGB and synchronized predicted point cloud and ground truth mesh rotations
Test example 2
Test example 2: input RGB and synchronized predicted point cloud and ground truth mesh rotations
Test example 3
Test example 3: input RGB and synchronized predicted point cloud and ground truth mesh rotations
Test-set F1
Mean point-cloud reconstruction F1 across distance thresholds on 678 test chairs
Training loss and validation F1 by epoch
Training Chamfer loss and held-out validation F1 over 16.13 epochs, using separate y-axes

2.3. Image to mesh (20 points)

ResNet18 with a 512-1024-2048-7686 MLP deforms ico_sphere(4) using Chamfer + 0.1 x Laplacian loss (3,000 Adam steps, batch 32, CPU); points achieve the best F1@0.05 by placing samples freely, while coarse voxels lose thin details and fixed-topology meshes can fold.

678 test chairs, 1,000 surface samples each; training settings differ and voxel samples are centered before evaluation
RepresentationF1@0.05
32x32x32 voxels73.88%
1,000 points80.13%
2,562-vertex mesh72.11%
F1 across representations
Test-set F1 curves comparing voxel, point cloud, and mesh reconstruction
Test example 1
Test example 1: input RGB and synchronized predicted mesh and ground truth mesh rotations
Test example 2
Test example 2: input RGB and synchronized predicted mesh and ground truth mesh rotations
Test example 3
Test example 3: input RGB and synchronized predicted mesh and ground truth mesh rotations
Test-set F1
Mean mesh reconstruction F1 across distance thresholds on 678 test chairs

2.4. Analyse effects of hyperparameter variations (10 points)

With matched Chamfer + 0.1 x uniform Laplacian training (3,000 Adam steps, batch 32, CPU), ico_sphere(2) cuts decoder parameters by 80.3% for a 1.92-point F1@0.05 drop, while the finer model's visible folds show that higher surface F1 does not guarantee cleaner mesh geometry.

678 test chairs, 1,000 sampled points per mesh
TemplateVerticesFacesDecoder parametersF1@0.05
ico_sphere(2)1623203,620,32670.20%
ico_sphere(4)2,5625,12018,373,12672.11%
Test-set F1 by mesh resolution
Test-set F1 curves for 162-vertex and 2562-vertex mesh templates, reaching 70.20% and 72.11% at distance threshold 0.05
Test example 1
Test example 1: input RGB, 162-vertex prediction, 2562-vertex prediction, and ground truth mesh in one synchronized animation
Test example 2
Test example 2: input RGB, 162-vertex prediction, 2562-vertex prediction, and ground truth mesh in one synchronized animation
Test example 3
Test example 3: input RGB, 162-vertex prediction, 2562-vertex prediction, and ground truth mesh in one synchronized animation

2.5. Interpret your model (10 points)

Sliding 24x24 black masks (stride 12) over Task 2.2 inputs highlights seat and leg sensitivity in examples 2-3, while the largest shape change improves F1 in example 1, showing that sensitivity does not necessarily imply reduced accuracy.

Occlusion sensitivity | Color: RMS Chamfer change from the original prediction, averaged over overlapping windows on a shared scale; cyan: the mask causing the largest change.
Three test images, heatmaps of 3D prediction changes under black image patches, and the most disruptive masked inputs
F1@0.05 with identical ground-truth surface samples; masks selected by prediction change
ExampleOriginal F1Masked F1Change (points)
179.30%81.15%+1.85
292.53%79.23%-13.30
391.39%82.24%-9.15
Test example 1: effect on reconstructed shape
Example 1: synchronized original prediction, prediction after the selected mask, and ground truth mesh
Test example 2: effect on reconstructed shape
Example 2: synchronized original prediction, prediction after the selected mask, and ground truth mesh
Test example 3: effect on reconstructed shape
Example 3: synchronized original prediction, prediction after the selected mask, and ground truth mesh

3.1. Implicit network (10 points)

A frozen Task 2.1 ResNet18 conditions a three-layer, 256-wide XYZ occupancy MLP trained with weighted BCE and selected by validation IoU, achieving 65.66% test F1@0.05 from a 32x32x32 grid at occupancy threshold 0.5.

Added after assignment update: I added this F1 evaluation after grading using the original checkpoint; I had not pulled the September 29 assignment update before submitting.

Test example 1
Test example 1: input RGB and synchronized implicit occupancy surface and ground truth mesh rotations
Test example 2
Test example 2: input RGB and synchronized implicit occupancy surface and ground truth mesh rotations
Test example 3: sled-base chair
Test example 3: input RGB and synchronized implicit occupancy surface and ground truth mesh rotations

3.2. Parametric network (10 points)

A frozen Task 2.2 ResNet18 conditions four UV-to-XYZ MLPs with three 128-wide ReLU hidden layers, trained for 6,000 Adam steps with symmetric squared Chamfer loss, achieving 71.41% test F1@0.05 on surfaces extracted using a 32x32 grid per patch.

Added after assignment update: I added this F1 evaluation after grading using the original checkpoint; I had not pulled the September 29 assignment update before submitting.

Test example 1
Test example 1: input RGB and synchronized predicted parametric surface patches and ground truth mesh rotations
Test example 2
Test example 2: input RGB and synchronized predicted parametric surface patches and ground truth mesh rotations
Test example 3: sled-base chair
Test example 3: input RGB and synchronized predicted parametric surface patches and ground truth mesh rotations

3.3. Extended dataset for training (10 points)

With identical frozen ImageNet ResNet18 encoders and 1,000-point decoders trained for 3,000 Adam steps on MPS, three-class training improves chair F1@0.05 by 1.26 points and produces clearer car and plane shapes, suggesting broader shape coverage without sacrificing chair accuracy, although thin chair details remain approximate.

Same test images and surface samples; batch 32, squared Chamfer loss, validation-selected checkpoints
Training classesChair F1 (678)Car F1 (368)Plane F1 (405)
Chairs only73.78%63.46%58.22%
Chairs, cars, planes75.04%91.77%95.13%
Test-set F1 by category
Matched test-set F1 curves for chair-only and three-class point-cloud models on chairs, cars, and planes
Chair test example 1
Chair example 1: input RGB, chair-only prediction, three-class prediction, and ground truth mesh in one synchronized animation
Chair test example 2
Chair example 2: input RGB, chair-only prediction, three-class prediction, and ground truth mesh in one synchronized animation
Chair test example 3: sled-base chair
Chair example 3: input RGB, chair-only prediction, three-class prediction, and ground truth mesh in one synchronized animation
Car test example
Car example: input RGB, chair-only prediction, three-class prediction, and ground truth mesh in one synchronized animation
Plane test example
Plane example: input RGB, chair-only prediction, three-class prediction, and ground truth mesh in one synchronized animation