1.1. Fitting a voxel grid (4 points)

Caption for result 1.

1.2. Fitting a point cloud (3 points)

Caption for result 1.

1.3. Fitting a mesh (3 points)

Caption for result 1.

2.1. Image to voxel grid (20 points)

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot

Results

--load_feat flag was used. F1@0.05 score was 39.67.

2.2. Image to point cloud (20 points)

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot

Results

--load_feat flag was used. F1@0.05 score was 71.5

2.3. Image to mesh (20 points)

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot

Results

--load_feat flag was used. F1@0.05 score was 67.6.

Voxel, Point Cloud, Mesh Comparisons

Point clouds (71.5) > Mesh (67.6) > Voxel (39.7)

My intuition is that the order follows the ease of learning. Point clouds are the most unstructured data, with just x,y,z values for each point, and meshes are harder to learn in comparison because they encode more information. Voxels don’t compare as easily, but looking at some of the prediction, it looks like it was difficult for the model to get over the threshold in the marching cubes algorithm to confidently predict more voxels.

2.4. Analyse effects of hyperparams variations (10 points)

2.4.1 Varying initial mesh

2.4.1.1 Using torus

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot
Results

--load_feat flag was used. Using torus, F1@0.05 score was 70.6. Which is interesting because the F1@0.05 score is higher, but the generated mesh looks worse because of the torus hole being carried over as big gaps in the final chair meshes.

2.4.1.2 Using mesh of chair

The following chair mesh was used as the starting mesh. Caption for result 1.

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot
Results

--load_feat flag was used. Using chair mesh, F1@0.05 score was 70.2. The F1@0.05 marginally improved, but no visible qualitative gains in reconstruction quality compared to sphere. It is worth noting that even when the starting shape is very close to the starting mesh, as in 600_mesh_chair.gif, the shape generated was still much more deformed.

2.4.2 Weighted loss

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot
Results

--load_feat flag was used. F1@0.05 score was 55.0. Penalizing a false negative voxel 3 times more than false positive voxel. The F1@0.05 score was +15 points higher. The model more aggressively filled in voxels. There are more false positives, but the shapes look much more complete.

2.5. Interpret your model (10 points)

I want to see if there is a pattern in where our models fail to predict features. Generating an error heatmap can highlight where the errors are. The heatmap shows that in these samples, common zone of error is in the top and near the feet of the chairs.

Caption for result 1. Caption for result 1. Caption for result 1.

To see if this is true across the different models we’ve learned, I will plot the mean distance of erroneous points and GT over normalized height of chairs.

Caption for result 1.

There is some truth to the claim that the common zone of error is in the top and near the feet of the chairs.

3.1 Implicit network (10 points)

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot

Results

--load_feat flag was used. F1@0.05 score was 31.3. One of the benchmark examples produced an empty output.

3.2 Parametric network (10 points)

Visualizations

Caption for result 1. Caption for result 1. Caption for result 1.

F1 plot

Results

--load_feat flag was used. F1@0.05 score was 74.3. Best F1@0.05 score.

3.3 Extended dataset for training (10 points)

Visualizations

Multiclass model on chair eval

Multiclass model on chair eval

Multiclass model on multiclass eval

Caption for result 1.
F1 plot

Chair model on multiclass eval (plane)

Caption for result 1.

Chair model on multiclass eval (car)

Caption for result 1.
F1 plot

Results

--load_feat flag was used. F1@0.05 score 70.6 evaluated just on chair data. Performed slightly worse F1 score and shows no qualitative improvements in the renderings compared to the model trained on just chairs.

The multiclass model did show comprable F1@0.05 scores on the multiclass dataset (plane: 91.84, car 85.83, chair 70.81), but the generated point clouds were not qualitatively better. The point clouds generated by the multiclass models tended to cluster too much in the center of the cloud, and points near the surface were very sparse. I believe the variance in samples led to the model preferring to learn points closer to the average, middle of the cloud. The scores for the plane and car had higher means. This is likely because their shapes are “easier” with more mass in the center of the clouds, rewarding the model’s tendacy to shoot to the center.

The chair model predicted chair-like clusters in each of the multiclass evals. This suggested that the model wasn’t predicting, but rather doing something akin to nearest neighbor retrieval1


  1. M. Tatarchenko, S. R. Richter, R. Ranftl, Z. Li, V. Koltun, T. Brox. “What Do Single-view 3D Reconstruction Networks Learn?” CVPR 2019. arXiv:1905.03678↩︎