Assignment 2

All Part 2 models use the --load_feat argument.

1. Exploring loss functions

1.1 Fitting a voxel grid

Left: optimized voxel grid, right: ground truth.

1.2 Fitting a point cloud

Left: optimized point cloud, right: ground truth.

1.3 Fitting a mesh

Left: optimized mesh, right: ground truth.

2. Reconstructing 3D from single view

Each example: input RGB | prediction | ground truth mesh.

2.1 Image to voxel grid

Pix2Vox-A decoder. F1@0.05 = 67.8



2.2 Image to point cloud

PSGN vanilla MLP, 1000 points. F1@0.05 = 75.6



2.3 Image to mesh

Sphere FC (ico_sphere level 4), w_smooth = 0.1. F1@0.05 = 70.9



F1@0.05: point 75.6 > mesh 70.9 > voxel 67.8.

Since points have no topology, they are free to be rearranged in any configuration. This freedom allows them to go to any part of the chair, and so they can cover thinner parts as well (supported by the second term of the chamfer loss) and achieve a higher F1 score. The mesh is deformed from a sphere and cannot create holes. Additionally, since I used Sphere FC, the decoder has no knowledge of the topology, leading to a jagged look. Finally, voxels have a low resolution, and cannot represent thin objects well.

2.4 Analyse effects of hyperparams variations

Mesh: w_smooth

w_smoothF1@0.05
0.1 (baseline)70.9
2.066.3

w_smooth = 2.0:



With w_smooth = 0.1, the mesh has sharp edges and visible legs, but was also spiky. With w_smooth = 2.0, the spikes go away, but the shape becomes a round blob. The seat and the back merge, and the legs become thin tips. The smoothness term removes the detail that the chamfer loss needs, so F1 decreases.

Voxel: pos_weight in BCE

pos_weightF1@0.05
1 (baseline)67.8
264.0

pos_weight = 2:



Approximately 4.5% of the voxels are occupied, so BCE gives a high reward for empty space. pos_weight = 2 gives occupied voxels more weight. The shapes cover more of the chair, but they are still blobs, and the legs are not recovered. A lower threshold on the baseline model (see 2.5) also adds voxels, but does not increase F1, and so I expect only a small effect from pos_weight.

Mesh: initial sphere scale 0.5

initial scaleF1@0.05mean offset length
1.0 (baseline)70.90.87 to 0.95
0.571.10.38 to 0.44

Initial scale 0.5:



This network predicts the offsets for the vertices, and since the chairs fit in approximately ±0.5, a sphere of radius 0.5 should require less deformation (and so, smaller offsets) to recover the chairs. However, the F1 seems to not have changed as much, and the mesh remains spiky. The initial scale may not be as important for this decoder.

Point: increased weight of the target --> source term in chamfer loss

The chamfer loss has two terms. The source --> target term pulls each predicted point to the chair. The target --> source term makes sure that each part of the chair has a predicted point near it (coverage). This second term is multiplied by w_coverage.

w_coverageprecisionrecallF1@0.05points with no other point within 0.01
1 (baseline)77.277.075.948.9%
272.084.176.971.6%

w_coverage = 2:



With w_coverage = 2, the points spread more, and there are fewer clumps. More points are in the space between the parts of the chair, so recall increases and precision decreases.

2.5 Interpret your model

Mesh: size of the vertex offsets

The ico_sphere has radius 1, but the chairs fit in approximately ±0.5. For samples 0, 200 and 400, all offsets are longer than 0.5:

samplelargest offset
01.26
2001.18
4001.36

If we add a tanh to the offsets of the trained model (no retraining), the offsets become smaller. The sphere does not shrink enough, and the chair becomes a blob (F1 decreases by 10 to 21). Top: no tanh, bottom: tanh.

3. Exploring other architectures / datasets

3.1 Implicit network

F1@0.05 = 49.1