Assignment 2: Single View to 3D

1. Exploring loss functions

Optimized shape on left, ground truth on right

1.1 Fitting a voxel grid

voxel fit

1.2 Fitting a point cloud

point cloud fit

1.3 Fitting a mesh

mesh fit

2. Reconstructing 3D from single view

Didn't use --load_feat or cpu training, left to right has input rgb, prediction, ground truth, ground truth mesh

2.1 Image to voxel grid

vox 1

vox 2

vox 3

F1@0.05: 0.70

vox f1

2.2 Image to point cloud

point 1

point 2

point 3

F1@0.05: 0.80

point f1

2.3 Image to mesh

mesh 1

mesh 2

mesh 3

F1@0.05: 0.74

mesh f1

Point clouds score highest (0.80) > meshes (0.74) > voxels (0.70). The point decoder can put every point anywhere and is trained directly on chamfer distance, which is close to what F1 measures. The mesh is a deformed sphere so it can't open holes like the gap in a chair back or between legs, and the smoothing term works against thin parts. Voxels are limited by the grid where one cell is about 3% of the chair so thin legs disappear or break into floating pieces.

2.4 Analyse effects of hyperparams variations

I varied n_points for the point cloud model to be 500, 1000, or 5000

n_points = 500, F1@0.05: 0.74

500 points

500 points f1

n_points = 1000, F1@0.05: 0.80

1000 points

n_points = 5000, F1@0.05: 0.88

5000 points

5000 points f1

F1 goes up with more points because more of the surface gets covered, so recall improves and thin parts like legs get some points. However evaluate() samples the same number of ground truth points as n_points, so the denser models are also compared against a denser ground truth. This could be causing part of the increase in score from 0.74 to 0.88. Also, training time didn't really change because data loading was the bottleneck.

2.5 Interpret your model

error 1

error 2

error 3

For the point cloud model I colored every point by its error from blue at 0 to red at 0.05 as the F1 threshold. The middle image shows prediction colored by distance to the nearest ground truth point, so red points are the ones making precision worse. The right image shows ground truth colored by distance to the nearest predicted point, so the red areas are the ones the model missed and affected recall.

On the first and third chairs the top of the backrest is the reddest part on the ground truth side while the seat is bluer and greener, so the model gets the seat right but misses the ends of the chair. The round chair is seen from the side so the model can't see the seat and falls back to a generic chair shape, which I think is why most of its ground truth is red; the same chair scores up to 0.87 from the front but can go down to 0.47 from the side. On the prediction side the red points are scattered off the surface. The predicted points also clump, and 48% of them have another predicted point within 0.005, and the clumps mostly sit around the seat and are blue. I think chamfer loss lets a lot of points collapse onto the middle of the chair, which is accurate but wastes points that could cover the backrest. I think this is also why more points helped in 2.4.

3. Exploring other architectures / datasets

3.3 Extended dataset for training

I trained the point cloud model on chairs, cars and planes with the same settings as 2.2

Chair test set, trained on one class vs three classes:

chair one class

chair three classes

chair one class

chair three classes

Trained on chairs only, F1@0.05 on chairs: 0.80

Trained on three classes, F1@0.05 on chairs: 0.79

three class chair f1

Planes and cars from the three class model:

plane 1

plane 2

car 1

car 2

On three class test set F1@0.05 is 0.87 (planes 0.96, cars 0.93, chairs 0.79)

three class f1

The chair F1 is about the same although chairs are only 47% of the three class training set so the model saw about half as many chair images. The chair predictions from both models also look very similar so the chair results seem to be limited by the point decoder and not by how much chair data there is. Planes come out as a blob along the body with barely any wings but still score 0.96 probably since a plane is thin and compact so most points land within 0.05. Cars also came out somewhat blob like, though the normal car looks more distributed, with mostly the right footprint. Overall the outputs mostly change by class and there isn't much variety within a class though the car outputs seem to be more diverse like if it's a tractor or normal car.