16-825 Learning for 3D Vision

Assignment 2: Single-View to 3D

Voxel grid
76.43
F1@0.05
Point cloud
76.64
F1@0.05 · best
Mesh
74.22
F1@0.05
Implicit
43.5
F1@0.05

1. Exploring loss functions

For this section, all images follow the order: [Prediction | GT]

1.1Fitting a voxel grid

Fitted voxel grid next to ground truth

1.2Fitting a point cloud

Fitted point cloud next to ground truth

1.3Fitting a mesh

Fitted mesh next to ground truth

2. Reconstructing 3D from single view

For this section, if there is no labels on the image, and the image is about comparing among input image, ground truth 3D voxel/point cloud/mesh, predicted 3D voxel/point cloud/mesh, the order follows: [Input Image | GT | Predicted].

2.1Image to voxel grid

Average F1@0.05 76.43
F1 versus threshold for voxel predictions
F1 plot
Visuals of 3 examples
Voxel example 75
Voxel example 85
Voxel example 347

2.2Image to point cloud

Average F1@0.05 76.64
F1 versus threshold for point cloud predictions
F1 plot
Visuals of 3 examples
Point cloud example 338
Point cloud example 530
Point cloud example 560

2.3Image to mesh

Average F1@0.05 74.22
F1 versus threshold for mesh predictions
F1 plot
Visuals of 3 examples
Mesh example 83
Mesh example 231
Mesh example 488

From the results, it can be concluded that point cloud has the highest F1@0.05 score. This is because the point clouds has no ties, they are separated in the 3D space, and we only need to consider the chamfer loss which is how close it is from each point to the closest ground truth surface, therefore, leaving a gap has nearly no effect on the F1 score of the point cloud predcitions. Voxels has a slight lower F1@score compared to point cloud, depending on the resolution of the 3D space, this score can be different. Because the training set and the testing set involves a lot of chairs, with chair legs so thin that they are hardly represented by voxels, resulting the reduce of the F1 score. Mesh has the lowest F1 score, this is because we are stretching a ico-sphere into a chair-like shape, and the loss function (laplacian smoothing loss) penalizes unsmooth structures, but chairs has sharp edges. That results the lowest F1 score for meshes.

2.4Analyse effects of hyperparams variations

In train_model.py, I noticed this loss function for mesh:

loss = args.w_chamfer * loss_reg + args.w_smooth * loss_smooth

In my analysis of the F1 score in the previous section, I mentioned that smoothness is a factor that reduce F1 score for meshes because chairs has shape edges. I wonder if I can change w_smooth so I can visualize the F1 score change specifically for meshes, therefore, I explored w_smooth = [0.01, 0.1, 1.0, 10.0] to see what will it effect. By training and evaluating for 4 different times, I obtained the result below:

w_smoothAverage F1@0.05
0.0174.22
0.173.96
1.072.50
10.064.03
Below are the meshes predicted for comparison
Mesh prediction with w_smooth 0.01
Prediction result obtained for training with w_smooth = 0.01
Mesh prediction with w_smooth 0.1
Prediction result obtained for training with w_smooth = 0.1
Mesh prediction with w_smooth 1.0
Prediction result obtained for training with w_smooth = 1.0
Mesh prediction with w_smooth 10.0
Prediction result obtained for training with w_smooth = 10.0

The result does show my hypothesis, the predicted 3D meshes all became smoother with less sharp edges, but the corresponding F1 score also decreases as w_smooth increase.

2.5Interpret your model

2.5.1 F1 score investigation

The first visualization I want to create is an overall F1 score analysis of the evaluation process, because in the code we were provided we can only see the average F1 score, and I want to gain more information from the distribution of the F1 scores for the testing set. Therefore, I plotted 3 histograms for showing the distribuition.

Histogram of F1 scores for voxel predictions
Histogram for voxel predictions
Histogram of F1 scores for point cloud predictions
Histogram for point cloud predictions
Histogram of F1 scores for mesh predictions
Histogram for mesh predictions

And I also wanted to see the results with the best F1 score, the result with the lowest F1 score and the result with median F1 score, so I extracted 5 top-scored prediction results, 5 closest-to-median scored prediction results, and 5 bottom-scored prediction results. Doing so helped me understand if the F1 score is affected by the shape of 2D object.

Voxel predictions with highest, median and lowest F1
Visualization of the predicted vox with highest, median, lowest F1 score
Point cloud predictions with highest, median and lowest F1
Visualization of the predicted point cloud with highest, median, lowest F1 score
Mesh predictions with highest, median and lowest F1
Visualization of the predicted mesh with highest, median, lowest F1 score

From the result above, there are several takeaways:

  1. Point cloud does yield the best F1 score over the dataset provided, as nearly 100 samples from the testing set achieves nearly full F1 score. Even though voxels has a overall higher F1 score compared to mesh, it has fewer high-score predicted results.
  2. Vox tend to have a better F1 score on an object that has no holes, where point cloud is good at predicting objects with holes and fine details. Some of the result doesn't make too much sense for me as I think the predicted result is actually simliar to the GT result but still receives a low score.

2.5.2 Azimuth investigation

From my understanding of the dataset, each object in the test set comes with 10 rendered views, taken from different, randomly chosen camera azimuths, but the evaluation only uses one randomly selected view per object. I wondered whether the input viewpoint affects the prediction. To test this, I took one test object (chair #85) and predicted it from all 10 of its views using the voxel, point cloud and mesh models. Below are the results I obtained:

Voxel predictions of chair 85 from 10 azimuths
Voxel predictions from different camera azimuths
Point cloud predictions of chair 85 from 10 azimuths
Point cloud predictions from different camera azimuths
Mesh predictions of chair 85 from 10 azimuths
Mesh predictions from different camera azimuths

The result shows that different camera azimuth do have an effect on the predicted result F1 scores. My hypothesis is that if the image shows fine details about the object, the predicted result should have a higher F1 score, but it seems like for camera azimuth 244 all 3 predicted result has the nearly highest F1 score, which doesn't agree with my hypothesis.

3. Exploring other architectures / datasets

For this section, I chose 3.1 as the part I want to explore.

3.1Implicit network

Average F1@0.05 43.5
F1 versus threshold for implicit network predictions
F1 plot
Visuals of 3 examples (left is GT and right is predicted)
Implicit network example 366
Implicit network example 545
Implicit network example 672

The result has a worse F1 score and a worse result compared to the methods I tried in part 2, I think 2 main factors caused this result:

  1. I trained the implicit network with --load_feat, while the other models were trained without. Because of this, only implicit network was trained with frozen ImageNet features that doesn't describe 3D shape well.
  2. The same as the voxel problem, a result of a meshgrid sized 32x32x32 cannot describe thin features like chair legs well. For the implicit network, the leg positions become more uncertain so that it can lead to empty result being produced (refer to the first example shown above).