16-825: Learning for 3D Vision

Assignment 2 - Fall 2026

Karthik Srinivasan (ksriniv2)

I did not use --load_feat, all models train on raw images.

Compute: everything on my laptop (RTX 3060 Laptop GPU, 6 GB), except 3.3 which was trained on an RTX 5090.

1. Exploring Loss Functions

1.1 Fitting a Voxel Grid

BCE loss on the voxel grid. Left: optimized, right: ground truth.

Voxel Fit

1.2 Fitting a Point Cloud

Chamfer loss. Left: optimized, right: ground truth.

Point Fit

1.3 Fitting a Mesh

Chamfer + Laplacian smoothing, starting from an ico-sphere. Left: optimized, right: ground truth.

Mesh Fit

2. Reconstructing 3D from Single View

2.1 Image to Voxel Grid

Linear layer + 3D transposed convs up to 32^3. Left to right: input, prediction, ground truth.

F1@0.05 = 55.1

Vox F1

Vox 1 Vox 2 Vox 3

2.2 Image to Point Cloud

MLP that outputs 1000 points. Left to right: input, prediction, ground truth.

F1@0.05 = 81.0

Point F1

Point 1 Point 2 Point 3

2.3 Image to Mesh

MLP that moves the vertices of an ico-sphere. Left to right: input, prediction, ground truth.

F1@0.05 = 71.7

Mesh F1

Mesh 1 Mesh 2 Mesh 3

Comparison: point cloud (81.0) > mesh (71.7) > voxel (55.1)

The point cloud does best because each point can go anywhere, and chamfer loss is close to what F1 measures. The mesh has to stay a deformed sphere, so it can't open the gaps between legs or slats, and the smoothness term shrinks thin parts. Voxels do worst: at 32^3 a chair leg is thinner than one voxel, and since most of the grid is empty the model tends to predict thin parts as empty.

2.4 Analysing Hyperparameter Variations

I changed n_points for the point cloud model (500, 1000, 3000) and trained each for 4000 iterations with everything else the same.

n_points F1@0.05
500 72.7
1000 78.8
3000 85.1
n = 500 n = 1000 n = 3000

F1 goes up with more points. With 500 points the legs get very few points, so a lot of the ground truth has nothing predicted within 0.05 and recall suffers. More points cover the surface better, which helps both precision and recall. The downside is size and speed: the last layer has 2048 x 3n weights (18.4M at n = 3000), and evaluation took 91 s instead of 27 s at n = 500.

2.5 Interpreting the Model

I made two visualizations: error maps for the point cloud model and an occlusion test for the mesh model.

Error maps. In the middle column each predicted point is colored by its distance to the closest ground truth point, so red there means the model put points where there is no chair. In the right column each ground truth point is colored by its distance to the closest predicted point, so red there means the model missed that part. Blue is close, red is 0.1 or more.

Error Maps

The seat and back are blue in both columns, so the model gets the main body right. Almost all the red is in the right column, on the legs (#150, #450) and the rim of the barrel chair (#300). So the model mostly leaves parts out rather than adding wrong ones, and it struggles with thin or unusual shapes.

Occlusion sensitivity. This follows Zeiler & Fergus, Visualizing and Understanding Convolutional Networks (ECCV 2014). I slide a 24x24 black patch (same color as the background) over the input with stride 12, run the mesh model on each occluded image, and record how much F1@0.05 drops. Red means covering that area hurts the prediction the most.

Occlusion

Covering the background does nothing, so the model is using the chair and not the background. The seat and the middle of the chair matter most (#0, #300, #450), while covering legs or armrest tips barely changes F1. That lines up with the error maps, since the legs are mostly missed anyway. The thin chair #150 (F1 34.9) has a weak, scattered map with a max drop of 1.6: the model doesn't get much from this image in the first place, so hiding parts of it doesn't change much.

3. Exploring Other Architectures / Datasets

3.1 Implicit Network

MLP that takes the image feature and an (x, y, z) point and gives the occupancy, queried on a 32^3 grid. Left to right: input, prediction, ground truth.

F1@0.05 = 39.6 (vs 55.1 for the voxel decoder)

Implicit F1

Implicit 1 Implicit 2 Implicit 3

3.2 Parametric Network

AtlasNet style: an MLP maps random (u, v) samples on 8 patches to 3D points. Left to right: input, prediction, ground truth.

F1@0.05 = 75.8 (vs 81.0 for the point decoder)

Parametric F1

Parametric 1 Parametric 2 Parametric 3

3.3 Extended Dataset for Training

I set use_full_dataset = True in dataset_location.py, which switches to the extended dataset and split_3c.json. I used the point cloud model since it did best in Q2. Every model was trained with the Q2.2 settings (batch size 32, lr 4e-4, 10000 iterations, 1000 points). The 3-class model is one model trained on all three classes together. To test it on each class separately, and to train the car-only and plane-only models, I made per-class split files from split_3c.json. The chair split in split_3c.json is identical to the single-class chair split, so both chair models are tested on the same chairs.

The dataset has 10 rendered views per object. Training draws a random view each time an object is loaded, and evaluation uses one random view per test object.

ClassTrain objectsTrain imagesTest objects (= test images used)
Chair610061000678
Car314631460368
Plane364036400405
All 3128861288601451

F1@0.05 on each class's test set:

Test classTrained on that class onlyTrained on 3 classes
Chair80.578.7
Car92.392.7
Plane96.296.0

The chair-only number is a new eval of the Q2.2 checkpoint, run alongside the other evals. It is 80.5 here instead of 81.0 because a different random view of each test object was used.

Same input image through both models. Each row: input | trained on 1 class | trained on 3 classes | ground truth.

ChairCarPlane

Conclusion and inference

Training on three classes only hurts chairs, and only a little: F1 drops by 1.8 points (80.5 to 78.7). Car and plane barely change (+0.5 and -0.2), which is about the size of the noise from the random views. Two reasons chairs suffer most:

For the same image, the two models give almost the same point cloud, and I can't see a clear difference in the renders. In the samples I looked at, the 3-class model never mixes classes: it doesn't put wings on a chair or chair legs on a car. So the encoder tells the classes apart easily, and the three classes don't interfere much. Within a class, both models output a similar "average" shape and miss unusual details, for example the barrel chair (last chair row) comes out as a blob in both. Adding more classes did not make the outputs within a class any more varied.

Addendum: single-class models on other classes

I was curious whether a model trained on one category can do anything zero-shot on another one (even though I was pretty sure it would not work). It needed almost no extra code, so I ran each single-class model on all three test sets.

F1@0.05, rows are the class the model was trained on, columns are the test set:

Trained on \ Tested onChairCarPlane
Chair80.571.166.8
Car42.192.369.9
Plane41.658.296.2

Car-only model on chairs. Left to right: input, prediction, ground truth.

Car on chair 1 Car on chair 2 Car on chair 3

Every model drops a lot on classes it never saw (80 to 96 on its own class, 42 to 71 on the others). The chair-only model holds up best on the other classes (71.1 on cars, 66.8 on planes), probably because chairs vary the most, so it learned a wider range of shapes. The car and plane models do worst on chairs (42.1 and 41.6). In the renders above the car-only model gives a flat cloud of points in the middle, with no back or legs. So there is no real zero-shot transfer: the model only rebuilds shapes it saw in training, and each class has to be in the training data.