Pretrained ResNet features were used in the training of these models
The fitted voxels look slightly different in overall appearance from the ground truth because they were implemented using marching cubes to convert the estimated voxels to a surface, instead of directly outputting a mesh.
The initialization used for this was not a sphere, but a chair mesh. The mesh was picked up from the dataset ground truth itself, based on manual inspection of what type of chairs were present in the dataset. A very generic chair with 4 legs, a back, and arms was picked to be the starting mesh.
The pointcloud model achieved the highest F1 score, followed by the mesh and voxel models. This is probably because the evaluation metric is computed using sampled point to point distances, which is exactly what the point cloud trains on. Points can be placed directly without having to worry about connectivity. Meshes on the other hand have some additional constraints because of their geometry. Voxels are extremely limited because of the resolution that is used, and they are also converted to a surface using marching cubes before evaluation.
The hyperparameter that was changed was the initial mesh. 2 different initializations were used. For the first one (whose plots are also above) a chair mesh was used as the initialization. For the second one, the sphere was used to start it off. I observed that the final F1 scores of the chairs which were initialized with a generic chair mesh were a lot higher as compared to the meshes obtained after moulding a sphere. This is pretty expected, as the vertices have to move a lot less if the initial geometry matches the final expected output. Even without training, the chair acheived an F1 score of around 50, while the sphere was much lower.
The final outputs are also pretty different. For chairs that were moulded from the sphere, the back is still a bit rounded in some examples. Also, if the chair has an open back, a sphere cannot be moulded into that shape. It is also harder to draw out legs from a blob than from a preexisting chair.
I used two different methods to try and interpret my models.
For the voxel representation, I varied the ISO value that was used to get the final output. The surface varying with the changing iso value means that the model is uncertain about those areas. As we can see below, the model is mostly confident in the seat and back regions of most chairs as they are pretty consistent across multiple inputs. Where most of the change is visible is the hands and the legs of the chair which are varying drastically with a change in the ISO value.
Green -> Closer to gt, Red -> Furthest from gt
This visualization is also similar in what it represents. The green dots are the ones closes to the ground truth, while the red/yellow dots are further away. This can also be a slight sampling bias because less points are sampled from legs. But overall, we can observe that the model is able to get the seat and the back of the chair right in most cases, while it messes up the most on legs and arms. For chairs that are very generic however, (4 skinny legs, a back) it gets the legs pretty right, whereas with the couch, the predicted pointcloud is kind of a blob at the base of the seat.
The implicit network produced lower quality reconstructions and a lower F1 score compared to the parametric network. Unlike the direct voxel decoder, the implicit model has to learn a continuous occupancy function over 3D space, which probably makes reconstruction a lot more challenging. The predicted surfaces are poorer and also incomplete for some images, and the model is not able to capture spiky features (like legs) very well.
The parametric network in comparison was better at reconstruction, and also achieved a higher F1 score. This provides a direct mapping from 2D image points to the corresponding 3D coordinates, and since it is directly working with points, rather than predicting an occupancy field, it produced sharper reconstructions, and was able to capture legs etc as well.