1. Exploring Loss Functions¶

1.1 Fitting a voxel grid¶

Trained on the subset dataset

Optimized Voxel Grid
optim_vxl_grid

Ground-Truth Voxel Grid
gt_vxl_grid

1.2 Fitting a point cloud¶

Trained on the full dataset

Optimized Point Clouds
optim_pc

Ground-Truth Point clouds
gt_pc

1.3 Fitting a mesh¶

Trained on the full dataset

Optimized Mesh
optim_mesh

Ground-Truth Mesh
gt_mesh

2. Reconstructing 3D from single view¶

2.1 Image to voxel grid¶

avg F1@0.05: 66.213

comments: I had to reduce the isovalue from 0.5 to 0.3 otherwise the sometimes the mes was completely empty. The model is not confident in its prediction when ground truth meshes are really small

Input images:
vox_in_1 vox_in_2 vox_in_3

Groud Truth voxels
vox_in_1_mesh vox_in_2_mesh vox_in_3_mesh

Predicted voxels
vox_in_1_mesh_pred vox_in_2_mesh_pred vox_in_3_mesh_pred

eval_vox

2.2 Image to Point Clouds¶

avg F1@0.05: 74.761

comments: A lot of point clouds are concentrated at the joint near the hind legs, backrest and the seat. This seems to be a local minima based on how the points are initialized.

Sample 2 (F1@0.05: 89.15) got an input image from the side, which I believe made the model uncertain about how much empty space is there in the back rest, I am surprised that this got such a high F1 score.

Sample 3 (F1@0.05: 60.19) got an image of the back of a closed designer chair, it was not able to identify that the cair was round (ig it should have picked that up based on how the light gradually changes, but I'll never know) thus, making the pridiction boxier than it should be.

Input images:
point_in_1 point_in_2 point_in_3

Ground Truth point clouds:
point_1_gt point_2_gt point_3_gt

Predicted point clouds:
point_pred_1 point_pred_2 point_pred_3

eval_point

2.3 Image to Mesh¶

avg F1@0.05: 68.7

comments: The predictions give decent shapes but are terrible in their smoothening. I think this will improve if I increase the weight for smoothening.

Input images:
mesh_in_1 mesh_in_2 mesh_in_3

Ground Truth meshes:
mesh_gt_1 mesh_gt_2 mesh_gt_3

Predicted meshes:
mesh_pred_1 mesh_pred_2 mesh_pred_3

eval_mesh

2.4 Analyze the effects of hyparameter variation¶

I chose to vary w_smooth, as I found the meshes to be very jaggered in the defaul w_smooth = 0.1, I increased the value to w_smooth = 0.5, w_smooth = 1.0 and w_smooth = 2.0

comments: So w_smooth plays a big role in ironing out the meshes. But it also brings to light how the model failed to learn the basic shape the third sample, as the shape matches the shape of the first sample.

Another odd behaviour is that the avg F1 score keeps decreasing with increasing w_smooth. Perhpas the model is overfitting to the training data with the smoothening term.

It is also interesting to see that the model is predicting two horns for every one of the chairs.

avg F1@0.05: 65.649

Ground Truth meshes:
mesh_gt_1 mesh_gt_2 mesh_gt_3

Predicted meshes:
mesh_pred_1 mesh_pred_2 mesh_pred_3

Predicted meshes with w_smooth = 0.5:
mesh_pred_1_05 mesh_pred_2_05 mesh_pred_3_05

avg F1@0.05: 64.430
Predicted meshes with w_smooth = 1.0:
mesh_pred_1_1 mesh_pred_2_1 mesh_pred_3_1

avg F1@0.05: 60.438
Predicted meshes with w_smooth = 2.0:
mesh_pred_1_2 mesh_pred_2_2 mesh_pred_3_2

2.5 Visualization of what the model is learning¶

I want to undersatnd how the point cloud model is working, so I experimented with it. since the model only has discrete outputs for the coordinates, I thought it would be interesting to see how the model is learns with a a lot of itererations early on.

I saved the model after 1, 3, 4, 7 and 20 iterations. This makes it clear why the there are huge pockets in the joints mentioned earlier. The initialization of the model prooduces a lot of points which are in the center, and very few points which are farther away. As the model learns to push these points onto the surface, bringing them closer to the actual mesh, the bulk at the point clouds get deposited at the pockets.

No description has been provided for this image
Iter 1
No description has been provided for this image
Iter 3
No description has been provided for this image
Iter 4
No description has been provided for this image
Iter 7
No description has been provided for this image
Iter 20
No description has been provided for this image
Iter 500

Exploring other Datasets¶

3.3 Using 3 class dataset¶

Two models were trained for this comparison. One was trained only on the chairs dataset. The other was trained on the full dataset.

For each of the model, the F1@0.05 scores were computed on the full dataset and the chairs only dataset.

Model 1: Trained only on the chairs dataset
Full dataset avg F1@0.05: 70.042
Chairs only dataset avg F1@0.05: 74.761

Model 2: Trained on the full dataset
Full dataset avg F1@0.05: 86.58
Chairs only dataset avg F1@0.05: 73.17

Using the full dataset, The original network which was only trained on the chairs dataset

Looking at the final scores, it seems like that the model learns to generalize better with full dataset, preventing overfitting to the chairs. But looking at the outputs, it seems to produce the same output for each category instead of learning the true shape of each object.

F1@0.05 on the chairs-only test set: chairs-only model ≈ 74.7, 3-class model 73.17. On the 3-class test set the 3-class model gets 84.00.

SampleInputGround truthTrained on chairsTrained on 3 classesComments
0
(plane)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image Both models have similar predictions for plane sample 0 and 100.
100
(plane)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image The chair model also produces the pockets of point clouds at the wing joints.
500
(car)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image Both model predict similar shapes for the cars.
700
(monster truck)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image The model has overfit to the normal car shape and has produced a generic car shape instead of creating a monster truck like shape.
1100
(chair)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image Some similarity in the shape, but is missing the armrests entirely. Seems to have overfit to a chair of similar size.
1200
(chair)
No description has been provided for this image No description has been provided for this image No description has been provided for this image No description has been provided for this image A larger chair, and approximates the size of the chair, but doens't capture the long hind legs / rocking chair like structure.