3D Vision Assignment 2: Single View to 3D

Q1: Exploring Loss Functions

Voxel

Ground Truth
Optimized

Point Cloud

Ground Truth
Optimized

Mesh

Ground Truth
Optimized

Q2: Reconstructing 3D from single view

Voxel

Example 1

Input
Ground Truth
Predicted

Example 2

Input
Ground Truth
Predicted

Example 3

Input
Ground Truth
Predicted

F1@0.05 Score: 27.275
F1 Plot

Point Cloud

Example 1

Input
Ground Truth
Predicted

Example 2

Input
Ground Truth
Predicted

Example 3

Input
Ground Truth
Predicted

F1@0.05 Score: 79.895
F1 Plot

Mesh

Example 1

Input
Ground Truth
Predicted

Example 2

Input
Ground Truth
Predicted

Example 3

Input
Ground Truth
Predicted

F1@0.05 Score: 68.432
F1 Plot

Hyperparameter Variations

n = 1000

Input
Ground Truth
Predicted
Input
Ground Truth
Predicted

n = 3000

Input
Ground Truth
Predicted
Input
Ground Truth
Predicted

n = 5000

Input
Ground Truth
Predicted
Input
Ground Truth
Predicted

I tried varying the number of points in the point cloud to see if it would create a denser and more complete representation of the object geometry. The default version at n = 1000 points creates a sparse but reasonable representation of the object, but ideally more points would end up covering more of the surface. We see this in the ground truth where the point cloud is denser, but the prediction still seems a bit too sparse and spread out. However, some of the harder-to-define features, like the chair legs, start coming out more with higher numbers of points; this is visible in n = 3000 and yields a slightly higher F1@0.05 score (83.302 compared to 79.895 for n = 1000). I also hoped that it would help with recovering the gap in the chair geometry for Example 2, but it still seemed to struggle with that despite the higher number of points.

Interpretations

I was curious as to how the icosphere deformation worked to create the final mesh, since the prediction ended up being very spiky and disorganized. I tried rendering the ground truth and predicted mesh normals and their directions to visualize the deformation. The side with an outward-facing normal is colored blue while the other side is colored red. This is a common visualization in 3D modeling to determine the stability and connectivity of mesh faces and vertices. The ground-truth mesh has expected normal directions according to the orientations of their faces; the predicted mesh similarly has a significant portion of its normals be outward-facing, meaning that more of the outer surface of the icosphere is still being deformed to face outwards.

Visualizations

Input
Ground Truth
Predicted

Q3: Exploring other datasets

Point Cloud

Example 1

Input
Ground Truth
Predicted

Example 2

Input
Ground Truth
Predicted

Example 3

Input
Ground Truth
Predicted

F1@0.05 Score: 83.623
F1 Plot

The F1 score for the chairs was slightly higher for three classes than for one class, comparable to the F1 score seen in hyperparameter variations for the point cloud. The change in prediction quality for the chair shapes was not large; as shown in the renders, the points still form the approximate correct shape but are still spread out significantly. However, the F1 scores for other classes are significantly higher (mid-high 90s), implying that their predictions are quite accurate. It could be that the variation in training data allowed the model to learn more diverse geometric information that it could then use to predict the varying geometries of objects across each category.

Collaboration

I briefly discussed the Q2 voxel decoder structure with Jacob Lessing (jlessing).