16-825 Learning for 3D Vision

Assignment 2: Single View to 3D

Andrew ID: pburujwa

Name: Parth Burujwale


Implementation Details

Due to limited GPU resources, I used the provided pretrained ResNet18 image features using the --load_feat argument. The reconstruction decoders were trained and evaluated on CPU.


1. Exploring Loss Functions

1.1 Fitting a Voxel Grid

Voxel fitting result

1.2 Fitting a Point Cloud

Point cloud fitting result

1.3 Fitting a Mesh

Mesh fitting result

2. Reconstructing 3D from a Single View

2.1 Image to Voxel Grid

Qualitative Results

Example 1

Voxel Reconstruction Example 1

Example 2

Voxel Reconstruction Example 2

Example 3

Voxel Reconstruction Example 3

Quantitative Result

F1@0.05 = 44.136

Voxel F1 Plot

2.2 Image to Point Cloud

Qualitative Results

Example 1

Point Cloud Example 1

Example 2

Point Cloud Example 2

Example 3

Point Cloud Example 3

Quantitative Result

F1@0.05 = 73.738

Point Cloud F1 Plot

2.3 Image to Mesh

Qualitative Results

Example 1

Mesh Example 1

Example 2

Mesh Example 2

Example 3

Mesh Example 3

Quantitative Result

F1@0.05 = 68.792

Mesh F1 Plot

Comparison of Representations

Representation F1@0.05
Voxel 44.136
Point Cloud 73.738
Mesh 68.792

The point cloud model had the best F1 score because it directly predicts 3D points and does not have to follow a fixed shape structure. The mesh model also predicts in 3D, but it is limited by the shape and connectivity of the starting icosphere. The voxel model had the lowest F1 score because it represents the object on a fixed 32 × 32 × 32 grid, which makes it harder to capture small details.


2.4 Effects of Hyperparameter Variation

I analyzed the effect of changing the subdivision level of the initial icosphere used by the mesh reconstruction model.

Icosphere Level F1@0.05
2 64.994
3 67.272
4 68.792

Icosphere Level 2

Icosphere Level 2

Icosphere Level 3

Icosphere Level 3

Icosphere Level 4

Icosphere Level 4

The results show how increasing the number of vertices changes reconstruction performance. A higher subdivision level gives the decoder more geometric freedom but also creates a more difficult prediction problem because substantially more vertex offsets must be estimated.


2.5 Model Interpretation

To better understand the learned representation, I performed interpolation between the pretrained ResNet18 image features of two different chair images.

The feature vectors of the two images were linearly interpolated using different values of alpha. Each intermediate feature was then passed through the trained point cloud decoder.

Feature Space Interpolation

The predicted point clouds change gradually as alpha moves from 0 to 1. This suggests that the model learned a continuous representation of object geometry in feature space. Intermediate feature vectors produce intermediate 3D structures instead of abrupt changes between the two endpoint objects.


3. Exploring Other Architectures and Datasets

3.1 Implicit Network

I implemented an implicit reconstruction network that predicts occupancy as a function of the image feature and a queried 3D coordinate.

A 32 × 32 × 32 grid of coordinates was generated in the normalized coordinate space from -1 to 1.

The implicit function:

Image Feature + (x, y, z) → Occupancy

Quantitative Result

F1@0.05 = 47.055

Implicit Network F1 Plot

The implicit model achieved a higher F1 score than the standard voxel decoder despite predicting occupancy independently as a function of spatial coordinates. Separately encoding the image feature and 3D coordinates helped the model preserve the spatial information required for reconstruction.


3.2 Parametric Network

The model represents the mapping:

Image Feature + (u, v) → (x, y, z)

The image feature and sampled 2D coordinates are independently encoded and combined before predicting the corresponding 3D position. Chamfer distance was used to compare the generated surface points with points sampled from the ground truth mesh.

Quantitative Result

F1@0.05 = 60.632

Parametric Network F1 Plot

3.3 Extended Dataset Training

For this experiment, I trained the point cloud reconstruction model on the extended dataset containing three object categories:

Chair, Car, and Airplane

The same point cloud architecture was used so that the effect of increasing the diversity of the training data could be analyzed.

Quantitative Results

Training Dataset F1@0.05
Chair Only 73.738
Chair + Car + Airplane 72.312

The chair only score was obtained on the chair test dataset, while the three class score was obtained on the complete three class test dataset. Therefore, the values provide a broad comparison rather than a direct evaluation using identical test samples.

The three class model maintained reconstruction performance very close to the chair only model even though it was required to learn substantially more diverse geometries. This suggests that the point cloud decoder was able to learn a representation that generalized across multiple object categories without a large decrease in reconstruction quality.

Qualitative Results

Chair

Three Class Chair

Car

Three Class Car

Airplane

Three Class Airplane