Assignment 2

1. Exploring loss functions

1.1 Fitting a voxel grid


Target Voxel

Fitted Voxel

1.2 Fitting a point cloud


Target Pointcloud

Fitted Pointcloud

1.3 Fitting a mesh


Target Pointcloud

Fitted Pointcloud

2. Reconstructing 3D from single view

Pretrained ResNet features were used in the training of these models

2.1 Image to voxel grid

The fitted voxels look slightly different in overall appearance from the ground truth because they were implemented using marching cubes to convert the estimated voxels to a surface, instead of directly outputting a mesh.


Voxel Evaluation F1 plot

Target Image

Ground Truth Voxel

Fitted Voxel

Target Image

Ground Truth Voxel

Fitted Voxel

Target Image

Ground Truth Voxel

Fitted Voxel

2.2 Image to point cloud


Pointcloud Evaluation F1 plot

Target Image

Ground Truth Pointcloud

Fitted Pointcloud

Target Image

Ground Truth Pointcloud

Fitted Pointcloud

Target Image

Ground Truth Pointcloud

Fitted Pointcloud

2.3 Image to mesh

The initialization used for this was not a sphere, but a chair mesh. The mesh was picked up from the dataset ground truth itself, based on manual inspection of what type of chairs were present in the dataset. A very generic chair with 4 legs, a back, and arms was picked to be the starting mesh.


Mesh Evaluation F1 plot

Target Image

Ground Truth Mesh

Fitted Mesh

Target Image

Ground Truth Mesh

Fitted Mesh

Target Image

Ground Truth Mesh

Fitted Mesh

Analysis

The pointcloud model achieved the highest F1 score, followed by the mesh and voxel models. This is probably because the evaluation metric is computed using sampled point to point distances, which is exactly what the point cloud trains on. Points can be placed directly without having to worry about connectivity. Meshes on the other hand have some additional constraints because of their geometry. Voxels are extremely limited because of the resolution that is used, and they are also converted to a surface using marching cubes before evaluation.

2.4 Analyse effects of hyperparams variations

The hyperparameter that was changed was the initial mesh. 2 different initializations were used. For the first one (whose plots are also above) a chair mesh was used as the initialization. For the second one, the sphere was used to start it off. I observed that the final F1 scores of the chairs which were initialized with a generic chair mesh were a lot higher as compared to the meshes obtained after moulding a sphere. This is pretty expected, as the vertices have to move a lot less if the initial geometry matches the final expected output. Even without training, the chair acheived an F1 score of around 50, while the sphere was much lower.

The final outputs are also pretty different. For chairs that were moulded from the sphere, the back is still a bit rounded in some examples. Also, if the chair has an open back, a sphere cannot be moulded into that shape. It is also harder to draw out legs from a blob than from a preexisting chair.


Mesh Evaluation F1 plot (Chair Initialization)

Mesh Evaluation F1 plot (Sphere Initialization)

Ground Truth Mesh

Fitted Mesh with Chair Initialization

Fitted Mesh with Sphere Initialization

Ground Truth Mesh

Fitted Mesh with Chair Initialization

Fitted Mesh with Sphere Initialization

Ground Truth Mesh

Fitted Mesh with Chair Initialization

Fitted Mesh with Sphere Initialization

2.5 Interpret your model

I used two different methods to try and interpret my models.

Different Iso values

For the voxel representation, I varied the ISO value that was used to get the final output. The surface varying with the changing iso value means that the model is uncertain about those areas. As we can see below, the model is mostly confident in the seat and back regions of most chairs as they are pretty consistent across multiple inputs. Where most of the change is visible is the hands and the legs of the chair which are varying drastically with a change in the ISO value.


Iso = 0.1

Iso = 0.2

Iso = 0.3

Iso = 0.1

Iso = 0.2

Iso = 0.3

Iso = 0.1

Iso = 0.2

Iso = 0.3

Coloring points based on chamfer distance

Green -> Closer to gt, Red -> Furthest from gt

This visualization is also similar in what it represents. The green dots are the ones closes to the ground truth, while the red/yellow dots are further away. This can also be a slight sampling bias because less points are sampled from legs. But overall, we can observe that the model is able to get the seat and the back of the chair right in most cases, while it messes up the most on legs and arms. For chairs that are very generic however, (4 skinny legs, a back) it gets the legs pretty right, whereas with the couch, the predicted pointcloud is kind of a blob at the base of the seat.


Ground truth

Pointcloud

Ground truth

Pointcloud

Ground truth

Pointcloud

3. Exploring other architectures/datasets

3.1 Implicit network

The implicit network produced lower quality reconstructions and a lower F1 score compared to the parametric network. Unlike the direct voxel decoder, the implicit model has to learn a continuous occupancy function over 3D space, which probably makes reconstruction a lot more challenging. The predicted surfaces are poorer and also incomplete for some images, and the model is not able to capture spiky features (like legs) very well.


Implicit Network Evaluation F1 plot

Target Image

Ground Truth Voxels

Fitted Voxels

Target Image

Ground Truth Voxels

Fitted Voxels

Target Image

Ground Truth Voxels

Fitted Voxels

3.2 Parametric network

The parametric network in comparison was better at reconstruction, and also achieved a higher F1 score. This provides a direct mapping from 2D image points to the corresponding 3D coordinates, and since it is directly working with points, rather than predicting an occupancy field, it produced sharper reconstructions, and was able to capture legs etc as well.


Parametric Network Evaluation F1 plot

Target Image

Ground Truth Pointcloud

Fitted Pointcloud

Target Image

Ground Truth Pointcloud

Fitted Pointcloud

Target Image

Ground Truth Pointcloud

Fitted Pointcloud