Assignment 2: Single View to 3D

Name: Chenzhe Zhang

Course: 16-825 Learning for 3D Vision

1.1 Fitting a Voxel Grid

Optimized representation and ground truth

1.2 Fitting a Point Cloud

Optimized representation and ground truth

1.3 Fitting a Mesh

Optimized representation and ground truth

For Section 2.1-2.3, average F1 scores are computed over all 678 test examples.

2.1 Image to Voxel Grid

Average F1@0.05: 64.138

Test example 0: input RGB, prediction and ground truth mesh

Test example 1: input RGB, prediction and ground truth mesh

Test example 2: input RGB, prediction and ground truth mesh

Average F1 versus distance threshold

2.2 Image to Point Cloud

Average F1@0.05: 77.030

Test example 0: input RGB, prediction and ground truth mesh

Test example 1: input RGB, prediction and ground truth mesh

Test example 2: input RGB, prediction and ground truth mesh

Average F1 versus distance threshold

2.3 Image to Mesh

Average F1@0.05: 68.904

Test example 0: input RGB, prediction and ground truth mesh

Test example 1: input RGB, prediction and ground truth mesh

Test example 2: input RGB, prediction and ground truth mesh

Average F1 versus distance threshold

Comparison of the Three Representations

Representation Average F1@0.05
Voxel grid 64.138
Point cloud 77.030
Mesh 68.904

The point-cloud model achieves the highest F1 score in our experiments. Points can move freely to match the target surface. The voxel model struggles with thin structures at a resolution of 32×32×32. The mesh model is limited by its fixed sphere connectivity and shows folds and distortions.

2.4 Effect of Point Count

I compared models with 500, 1000, and 2000 output points.

Points Average F1@0.05
500 72.054
1000 77.030
2000 79.728

500 points

500-point prediction

1000 points

1000-point prediction

2000 points

2000-point prediction

Increasing the number of output points improved the average F1@0.05 from 72.054 with 500 points to 77.030 with 1000 points and 79.728 with 2000 points. More points can cover the target surface more densely, which may explain the improvement. However, the gain from 1000 to 2000 points was smaller than the gain from 500 to 1000 points, suggesting diminishing returns in this experiment.

2.5 Interpret the Model

I masked the upper, middle, and lower regions of an input image and compared the predictions of the trained 1000-point model.

Occlusion comparison

Masking the upper region reduced F1@0.05 from 60.021 to 54.040, while masking the middle region increased it to 74.988. This suggests that upper-region cues are useful for this example, but removing image information does not always reduce accuracy.

3.1 Implicit Network

I implemented an MLP that predicts occupancy from image features and a 3D coordinate. I evaluated it on a 32 × 32 × 32 grid in [-1, 1]³, using an occupancy threshold of 0.5. The model was trained from scratch for 5,000 steps.

Average F1@0.05: 48.122, evaluated on 678 test examples.

Test example 0

Test example 1

Test example 2

F1 versus distance threshold

The implicit model has a lower F1 score than the voxel model (48.122 vs. 64.138). The predicted shapes are rough and often miss chair legs. Its decoder has about 0.107 million parameters, while the voxel decoder has 11.031 million. The smaller network may explain some of the missing details. This does not mean that all implicit networks perform worse.