16-825 Assignment 2: Single View to 3D

Name: Wen Wu
Andrew ID: wengleiw

1. Exploring Loss Functions

1.1 Fitting a Voxel Grid

Optimized and ground-truth voxel grids

1.2 Fitting a Point Cloud

Optimized and ground-truth point clouds

1.3 Fitting a Mesh

Optimized and ground-truth meshes

2. Reconstructing 3D from a Single View

2.1 Image to Voxel Grid

F1@0.05: 75.237

Voxel F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth voxel grid, predicted voxel grid.

Example 1 (test index 0):

Voxel reconstruction example 1

Example 2 (test index 100):

Voxel reconstruction example 2

Example 3 (test index 200):

Voxel reconstruction example 3

Example 4 (test index 300):

Voxel reconstruction example 4

Example 5 (test index 400):

Voxel reconstruction example 5

Example 6 (test index 500):

Voxel reconstruction example 6

Example 7 (test index 600):

Voxel reconstruction example 7

2.2 Image to Point Cloud

F1@0.05: 77.477

Point-cloud F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth point cloud, predicted point cloud.

Example 1 (test index 0):

Point-cloud reconstruction example 1

Example 2 (test index 100):

Point-cloud reconstruction example 2

Example 3 (test index 200):

Point-cloud reconstruction example 3

Example 4 (test index 300):

Point-cloud reconstruction example 4

Example 5 (test index 400):

Point-cloud reconstruction example 5

Example 6 (test index 500):

Point-cloud reconstruction example 6

Example 7 (test index 600):

Point-cloud reconstruction example 7

2.3 Image to Mesh

F1@0.05: 75.315

Mesh F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, predicted mesh.

Example 1 (test index 0):

Mesh reconstruction example 1

Example 2 (test index 100):

Mesh reconstruction example 2

Example 3 (test index 200):

Mesh reconstruction example 3

Example 4 (test index 300):

Mesh reconstruction example 4

Example 5 (test index 400):

Mesh reconstruction example 5

Example 6 (test index 500):

Mesh reconstruction example 6

Example 7 (test index 600):

Mesh reconstruction example 7

Comparison of 3D Reconstructions

Representation F1@0.05
Voxel grid 75.237
Point cloud 77.477
Mesh 75.315

The three representations have similar F1 scores, but the point cloud has the highest score. Voxel grids divide 3D space into discrete cells to represent shapes, and meshes are limited by their capacity for deformation. These are limitations of the representations. Points in a point cloud can move freely in 3D space, making it easier for the model to match the ground truth.

2.4 Analyze Effects of Hyperparameter Variations

Hyperparameter varied: n_points

Setting F1@0.05
1000 77.477
200 59.524

Hyperparameter-study point-cloud F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth point cloud, predicted point cloud.

Example 1 (test index 0):

Hyperparameter-study point-cloud reconstruction example 1

Example 2 (test index 100):

Hyperparameter-study point-cloud reconstruction example 2

Example 3 (test index 200):

Hyperparameter-study point-cloud reconstruction example 3

Example 4 (test index 300):

Hyperparameter-study point-cloud reconstruction example 4

Example 5 (test index 400):

Hyperparameter-study point-cloud reconstruction example 5

Example 6 (test index 500):

Hyperparameter-study point-cloud reconstruction example 6

Example 7 (test index 600):

Hyperparameter-study point-cloud reconstruction example 7

I varied n_points for the point cloud from 1000 to 200. Even though the point cloud is sparser, the original object remains recognizable. Given enough points, changing the number of points has little effect on the visible reconstruction quality. However, there is a significant drop in F1 score. I suspect that with fewer points, the model can still arrange them into a visually recognizable shape, but surface coverage is reduced and alignment with the ground truth is poor.

2.5 Interpret Your Model

F1@0.05: 72.849

Model-interpretation point-cloud F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth point cloud, predicted point cloud.

Example 1 (test index 0):

Model-interpretation point-cloud reconstruction example 1

Example 2 (test index 100):

Model-interpretation point-cloud reconstruction example 2

Example 3 (test index 200):

Model-interpretation point-cloud reconstruction example 3

Example 4 (test index 300):

Model-interpretation point-cloud reconstruction example 4

Example 5 (test index 400):

Model-interpretation point-cloud reconstruction example 5

Example 6 (test index 500):

Model-interpretation point-cloud reconstruction example 6

Example 7 (test index 600):

Model-interpretation point-cloud reconstruction example 7

I tested my model's robustness to occlusion by adding an occlusion of a fixed size in the center of each image. The images show that the model can still reconstruct the point cloud under occlusion. There are two interpretations. First, the model is robust to occlusion. Second, the model may have learned a prior over chair shapes, allowing it to recognize a chair without seeing every pixel. If two chairs differ in the occluded region, our model will likely produce the same point clouds for them. However, the F1 score shows some decrease in performance.

3. Extension: Other Architectures / Datasets

3.1 Implicit Network

F1@0.05: 33.733

Implicit-network F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth voxel grid, predicted implicit voxel grid.

Example 1 (test index 0):

Implicit reconstruction example 1

Example 2 (test index 300):

Implicit reconstruction example 2

Example 3 (test index 400):

Implicit reconstruction example 3

Example 4 (test index 500):

Implicit reconstruction example 4

Example 5 (test index 600):

Implicit reconstruction example 5

Compared with Section 2.1, reconstruction quality decreases considerably.

3.2 Parametric Network

F1@0.05: 73.737

Parametric-network F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth point cloud, predicted point cloud.

Example 1 (test index 0):

Parametric reconstruction example 1

Example 2 (test index 100):

Parametric reconstruction example 2

Example 3 (test index 200):

Parametric reconstruction example 3

Example 4 (test index 300):

Parametric reconstruction example 4

Example 5 (test index 400):

Parametric reconstruction example 5

Example 6 (test index 500):

Parametric reconstruction example 6

Example 7 (test index 600):

Parametric reconstruction example 7

Compared with Section 2.2, reconstruction quality decreases somewhat. This decrease is visible in the first four examples.

3.3 Extended Dataset for Training

Training data F1@0.05
Single-class chairs - single-class dataset 77.477
Single-class chairs - extended dataset 77.404
Three classes: chair, car, plane - extended dataset 86.492

Single-dataset F1 plot

Extended-dataset F1 plot

Each example shows, from left to right: input RGB, ground-truth mesh, ground-truth point cloud, predicted point cloud.

Single-class Example 1 (test index 0):

Single-dataset reconstruction example 1

Single-class Example 2 (test index 100):

Single-dataset reconstruction example 2

Single-class Example 3 (test index 200):

Single-dataset reconstruction example 3

Single-class Example 4 (test index 300):

Single-dataset reconstruction example 4

Single-class Example 5 (test index 400):

Single-dataset reconstruction example 5

Single-class Example 6 (test index 500):

Single-dataset reconstruction example 6

Single-class Example 7 (test index 600):

Single-dataset reconstruction example 7

Three-classes Example 1 (test index 0):

Extended-dataset reconstruction example 1

Three-classes Example 2 (test index 100):

Extended-dataset reconstruction example 2

Three-classes Example 3 (test index 200):

Extended-dataset reconstruction example 3

Three-classes Example 4 (test index 300):

Extended-dataset reconstruction example 4

Three-classes Example 5 (test index 400):

Extended-dataset reconstruction example 5

Three-classes Example 6 (test index 500):

Extended-dataset reconstruction example 6

Three-classes Example 7 (test index 600):

Extended-dataset reconstruction example 7

Three-classes Example 8 (test index 700):

Extended-dataset reconstruction example 8

Three-classes Example 9 (test index 800):

Extended-dataset reconstruction example 9

Three-classes Example 10 (test index 900):

Extended-dataset reconstruction example 10

Three-classes Example 11 (test index 1000):

Extended-dataset reconstruction example 11

Three-classes Example 12 (test index 1100):

Extended-dataset reconstruction example 12

Three-classes Example 13 (test index 1200):

Extended-dataset reconstruction example 13

Three-classes Example 14 (test index 1300):

Extended-dataset reconstruction example 14

Three-classes Example 15 (test index 1400):

Extended-dataset reconstruction example 15

The model was trained on the extended dataset and evaluated on both the single-class and three-class datasets. The images show perfect point-cloud reconstructions on the three-class dataset. For the single-class images, there is no visible degradation in the point clouds compared with the results in Section 2.2, particularly Example 4. The small difference is also reflected in the F1 scores.