16-825 HW2

Nicholas Leone (nleone)

Problem 1

Problem 1.1

Left is the original voxel (target), and the right is the fitted voxel (source).

Problem 1.2

Left is the original pointcloud (target), and the right is the fitted pointcloud (source).

Problem 1.3

Left is the original mesh (target), and the right is the fitted mesh (source).

Problem 2

Left is the input image, center is the ground truth, right is the predicted voxel/pointcloud/mesh

Problem 2.1

Problem 2.2

Problem 2.3

Problem 2.4

Note: I tried training the vox network with different architecture and hyper parameters, but always faced some tests having an empty mesh. I ran out of time and could not fully train a voxel network. As such, the f1 scores are lower than the other models, and will be difficult to compare the other two networks that are fully trained.

Point cloud performed the best, most likely because the point cloud network doesn't have to worry about connectivity, so having a few points out of place won't heavily impact the network's learning and output. The mesh network performed the second best. Unlike the point cloud, the mesh network has to be mindful about the connectivity as it handles vertices and faces, so having a few out of place vertices heavily impacts the network's learning and performance. In the visualizations, we can see a few spikes in the network's outputs. The voxel network performed the worst. I struggled trying to train the network, as no matter how much I trained, I always had the network fail to generate outputs for some test cases. I ended up having to skip those test cases and provide an f1 score of 0. In the end, I only trained the model for 20,000 iterations, while the other two had around 40000 to 50000 iterations. Despite the poor training, it seems that the network's output doesn't have spikes like the mesh's. If properly trained, I think the voxel network may generate smoother models than the mesh network.

Problem 2.5

I trained the point network using 500, 1000 (original), and 5000 points. Note: Due to time, the networks with 500 and 5000 points were only trained on 10000 iterations, while the 1000 poitns (original) was trained for 40000 iterations. All three were trained with a batch size of 16.

Despite the different iterations used for training, it is evident based on the f1 scores that having more points used in training yield improved results. Especially with the 5000 point network be trained with less iterations than the 1000 point network, it outperformed the original network by 0.05 points.

Problem 2.6

By placing the voxel, point, and mesh visualizes side-by-side, one can see that all three networks learn a generic shape of a chair. This can especially seen in the voxel visualizations, whose model hasn't been fully trained. The voxel network struggles to generate 3D models of untraditional charis, such as the one in the first row. Even the mesh network has a hard time handling the first chair. In the rest of the rows, the chairs are more traditional. The point and mesh network can generate shapes that match the proportions of the chair. The voxel network seems to prioritize the "head" of the chair rather than the legs. The under-trained network fails to generate legs for the third and fourth chair. All three models also fail to properly model any holes in the third chair. This is can most likely be remidied by including more points in training. This example shows how all three models prioritize learning the heads and legs of the chair, since they are characteristics shared amongst most chairs, rather than specific features such as holes, furnishings, etc.

Problem 3

I ran out of time and did not attempt this problem. I started the assignment too late and underestimated how long the assignment will take. I have learned my lesson and will start earlier for future assignments.