Srikar Gouru Assignment 2

Task 1: Exploring loss functions

Task 1.1: Fitting a voxel grid

Shown below are the ground truth voxel grid (left), as well as the fitted voxel grid (right) after optimizing over the loss function.

GT 1.1 Pred 1.1

Task 1.2: Fitting a point cloud

Shown below are the ground truth point cloud (left), as well as the fitted point cloud (right) after optimizing over the loss function.

GT 1.2 Pred 1.2

Task 1.3: Fitting a mesh

Shown below are the ground truth mesh (left), as well as the fitted mesh (right) after optimizing over the loss function.

GT 1.3 Pred 1.3

Task 2: Reconstructing 3D from single view

Task 2.1: Image to voxel grid

Since voxels represent 3D imagery with a pre-defined structure, a Convolutional Neural Network (CNN) is the perfect candidate. In model.py, I created a 3D CNN with 4 convolutional layers, and with ReLUs in between. I also added in a sigmoid at the evaluation point, since that’s what the logits-based loss function uses by default. Below are a few examples. For each example, on the left is the original image, in the middle is the ground truth, and on the right is the predicted 3D structure.

2.1 Raw 1 2.1 GT 1 2.1 Pred 1

2.1 Raw 2 2.1 GT 2 2.1 Pred 2

2.1 Raw 3 2.1 GT 3 2.1 Pred 3

The average F1@0.05 score was 35.937. The F1 plot is shown below.

Voxel grid eval

Task 2.2: Image to point cloud

Unlike voxels, point clouds don’t have a predefined structure and can be in any order. Thus, I used a simple deep network with 3 linear layers, of 512, 1024, and 2048 nodes in each hidden layer. Each linear layer is also separated by a LeakyReLU, with a Tanh function used at the output layer for normalization. Below are a few examples. For each example, on the left is the original image, in the middle is the ground truth, and on the right is the predicted 3D structure.

2.2 Raw 1 2.2 GT 1 2.2 Pred 1

2.2 Raw 2 2.2 GT 2 2.2 Pred 2

2.2 Raw 3 2.2 GT 3 2.2 Pred 3

The average F1@0.05 score was 80.084. The F1 plot is shown below.

Point cloud eval

Task 2.3: Image to mesh

Similar to point clouds, meshes don’t have a predefined structure and can be in any order. Thus, I used the exact same DNN structure that I described for point clouds. Below are a few examples. For each example, on the left is the original image, in the middle is the ground truth, and on the right is the predicted 3D structure.

2.3 Raw 1 2.3 GT 1 2.3 Pred 1

2.3 Raw 2 2.3 GT 2 2.3 Pred 2

2.3 Raw 3 2.3 GT 3 2.3 Pred 3

The average F1@0.05 score was 70.897. The F1 plot is shown below.

Mesh eval