16-825 Assignment 2: Single View to 3D

1. Exploring Loss Functions

1.1. Fitting a Voxel Grid (5 points)

Ground Truth	Optimized

1.2. Fitting a Point Cloud (10 points)

Ground Truth	Optimized

1.3. Fitting a Mesh (5 points)

Ground Truth	Optimized

2. Reconstructing 3D from Single View

2.1. Image to Voxel Grid (15 points)

Input RGB	Ground Truth Voxel	Predicted Voxel

2.2. Image to Point Cloud (15 points)

Input RGB	Ground Truth Mesh	Ground Truth Cloud	Predicted Cloud

2.3. Image to Mesh (15 points)

Input RGB	Ground Truth Mesh	Predicted Mesh

2.4. Quantitative Comparisons (10 points)

Average test F1 score at 0.05 threshold for voxelgrid, pointcloud and the mesh network.

Voxel F1 Curve	Point Cloud F1 Curve	Mesh F1 Curve

2.5. Analyze Effects of Hyperparameter Variations (10 points)

Analysis of hyperparameter effects on model performance.

n points	500	1000	2000
Point Cloud Examples
Mesh Examples
Point Cloud Evaluation Curves
Mesh Evaluation Curves
f1@0.05	Point: 58.59% Mesh: 59.71%	Point: 74.86% Mesh: 66.81%	Point: 50.47% Mesh: 72.47%

Analysis: Here we obsever that in case of point n_points = 1000 have the highest F1@0.05 score. The reason although I am not sure might be because higher number of points might lead to model confused which points belongs where creating some adversarial atttack in the training. However any increase in number of points should increase the F1@0.05 score. since more points means better estimation of the geometry. This is the case for meshes where this is very evident to notice.

2.6. Interpret Your Model (15 points)

Simply seeing final predictions and numerical evaluations is not always insightful. Can you create some visualizations that help highlight what your learned model does? Be creative and think of what visualizations would help you gain insights.

We model the failure cases of the model by visualizing the ground truth and prediction for each type of model.

Example 1

Label	Voxel Example	Mesh Example	Point Cloud Example
Target
Prediction

Example 2

Label	Voxel Example	Mesh Example	Point Cloud Example
Target
Prediction

Example 3

Label	Voxel Example	Mesh Example	Point Cloud Example
Target
Prediction

In all the example tthe voxels performs better as in it s able to capture the detailed structure of the object. The mesh has the most outliers and visually looks the workst since the outliers can be attibuted dur to the low weights of lalcaian smoother loss. However despite loooking worse it does seem to be able to capture the overall shape of the object. Point cloud have the higher f1 score than the mesh and voxel since it is able to capture the overall shape of the object in more relaxed manner. In example 1 we can notice that only point cloud is able to capture the legs of the object. Similarly in example 2 we can notice that only voxel is able to capture the legs of the object. On the other hand meshes fails to capture details and gets lower F1 score by minimizing the loss by having outliers

3. Exploring Some Recent Architectures

3.2. Parametric Surface Reconstruction

Input RGB	Ground Truth Parametric	Predicted Parametric

3.2. Parametric Evaluation

Parametric F1 Curve	F1@0.05 Score
	64.33%