Homework2 learning for 3D

Contact Information:

• Name: Shibo Zhao
• Email: shiboz@andrew.cmu.edu

ipython nbconvert --to HTML your_notebook.ipynb

1.1. Fitting a voxel grid (5 points)

Run the file python fit_data.py --type 'voxel', to fit the source voxel to the target voxel.

Visualize the optimized voxel grid along-side the ground truth voxel grid using the tools learnt in previous section.

Our Source Prediction Results

Source: 3D voxel grid	Ground Truth: 3D voxel grid

1.2. Fitting a point cloud (10 points)

Run the file python fit_data.py --type 'point', to fit the source point cloud to the target point cloud.

Visualize the optimized point cloud along-side the ground truth point cloud using the tools learnt in previous section.

Our Source Prediction Results

Source: point cloud	Ground Truth: point cloud

1.3. Fitting a mesh (5 points)

Run the file python fit_data.py --type 'mesh', to fit the source mesh to the target mesh.

Visualize the optimized mesh along-side the ground truth mesh using the tools learnt in previous section.

Our Source Prediction Results

Source: mesh	Ground Truth: mesh

2. Reconstructing 3D from single view

This section will involve training a single view to 3D pipeline for voxels, point clouds and meshes. Refer to the save_freq argument in train_model.py to save the model checkpoint quicker/slower.

2.1. Image to voxel grid (15 points)

After trained, visualize the input RGB, ground truth voxel grid and predicted voxel in eval_model.py file using: python eval_model.py --type 'vox' --load_checkpoint

You need to add the respective visualization code in eval_model.py

On your webpage, you should include visuals of any three examples in the test set. For each example show the input RGB, render of the predicted 3D voxel grid and a render of the ground truth mesh.

RGB	3D voxel grid	mesh groundtruth

RGB	3D voxel grid	mesh groundtruth

RGB	3D voxel grid	mesh groundtruth

2.2. Image to point cloud (15 points)

On your webpage, you should include visuals of any three examples in the test set. For each example show the input RGB, render of the predicted 3D point cloud and a render of the ground truth mesh.

RGB	3D point cloud	mesh groundtruth

RGB	3D point cloud	mesh groundtruth

RGB	3D point cloud	mesh groundtruth

2.3. Image to mesh (15 points)

On your webpage, you should include visuals of any three examples in the test set. For each example show the input RGB, render of the predicted mesh and a render of the ground truth mesh.

RGB	3D predicted mesh	mesh groundtruth

RGB	3D predicted mesh	mesh groundtruth

RGB	3D predicted mesh	mesh groundtruth

2.4. Quantitative comparisions(10 points)

Quantitatively compare the F1 score of 3D reconstruction for meshes vs pointcloud vs voxelgrids. Provide an intutive explaination justifying the comparision.

For evaluating you can run: python eval_model.py --type voxel|mesh|point --load_checkpoint

We compare the

Images	voxelgrids	3D pointcloud	meshes
	F1 score=81.324	F1 score= 98.385	F1 score=90.100

From the quantative side,we found that 3D pointcloud> meshes>voxelgrids on F1 score.

For F1 score, we have to calculate L1 and L2 distances between each pred point and its nearest GT points. The model of 3D point cloud are more easily having higher F1 score since it can easily find the nearest GT.

However, the modal of meshes and voxel grids may not able to find the nearest GT or find the incorrect neaest GT. Therefore, these modal have lower F1 score

2.5. Analyse effects of hyperparms variations (10 points)

Analyse the results, by varying an hyperparameter of your choice. Try to be unique and conclusive in your analysis.

We evalute the difference between the batch size=2 and batch size=8 and train the its pointcloud model.

Quantitative comparisions

Images	batch size =2	batch size =8
	F1 score=71.035	F1 score= 81.324

From the quantative comparison,we found that the modal with batch size=8 can get the higher F1 soore than the modal with batch size=2

Qualitive comparisions

Images	batch size =2	batch size =8

From the qualitive comparison,we found that the modal of batch size=8 can get better point cloud reconstruction. Genrerally speaking, the larger the batch size, the more accurate the descending direction is. Therefore, we may get the better 3d reconstruction result.

2.6. Interpret your model (15 points)

Simply seeing final predictions and numerical evaluations is not always insightful. Can you create some visualizations that help highlight what your learned model does? Be creative and think of what visualizations would help you gain insights. There is no `right' answer - although reading some papers to get inspiration might give you ideas.

Loss Curve (Blue: loss of voxel model Orange: The loss of Pointcloud model) |
:-------------------------:

We compare the loss/train curve of voxel modal(Blue) and loss/train curve of pointcloud model(orange). The loss of voxel is bigger than the loss of point cloud, which makes sense since the point cloud is more easily find its neighbour. Meanwhile, We found that both loss/train curves of voxel modal and point cloud model converges in the end.