16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)
Left: fitted. Right: ground truth.
Q2 models use cached ResNet18 features (--load_feat).
Voxel model is trained for 25,080 ietrations.
The starter code has a buf in the evaluation code that does not properly account for the different x/z scaling between the input and ground truth meshes. The pictures below are rendered after fixing this bug.
Left to right: input RGB, prediction, ground-truth mesh. Voxel surfaces use occupancy probability 0.3.


Average F1@0.05: 66.823
Point cloud and mesh models trained for 50,000 updates.


Average F1@0.05: 84.328


Average F1@0.05: 75.586
| Representation | F1@0.05 (%) |
|---|---|
| Voxel | 66.823 |
| Point cloud | 84.328 |
| Mesh | 75.586 |
The point cloud model shows the best F1, which is also visually the best, although the points are not not well distributed on the surfaces but concentrate on the chair corners.
Voxel model shows a lot of noisy geometry that drags down the F1.
The mesh model is the worst, with very pointy faces.
I varied w_smooth, keeping the other settings fixed: 10,000 updates, the same training seed, and 610 of the 6,100 training objects held out for validation (5,490 used for training). Validation scores average three surface-sampling seeds.
| Smoothness weight | Validation F1@0.05 (%) |
|---|---|
| 0.1 | 74.958 |
| 0.5 | 72.776 |
| 1 | 70.796 |
| 3 | 69.344 |
| 10 | 62.044 |
Left to right: input, predictions with weights 0.1, 0.5, 1, 3, 10, and GT.


Weight 0.1 shows the highest validation F1 but it is visually the worst. The predicion doesn't capture the shapes well. The weight 3 looks better since it has continuous surfaces, although its F1 is lower.
Left to right: input, prediction-to-GT distance, GT-to-prediction distance, color scale. Blue means close; red means distance ≥ 0.05.
Warm prediction regions show misplaced geometry. Warm GT regions show geometry missing from the prediction. Both pointcloud and mesh models show the weakness of captureing the edge and outline of the shapes.




The model is trained with a simple mlp, whose input is a simply concatenation of image features and a 3D position.
As I mentioned in the 2.1, the starter code has a bug in the evaluation code that does not properly account for the different x/z scaling between the input and ground truth meshes. The model below is trained before the bug is fixed, therefore, the left most column shows wrong x/z scaling - the 2nd example is especially obvious. So I added a column that shows the x/z-corrected prediction, noted that it's not rescaling during evaluation rather than re-training.
Left to right: input, original prediction, x/z-corrected prediction, GT. Y stays unchanged.


Test F1@0.05: original (buggy) 24.145; x/z scale-aligned 23.322 (678 objects).
Input | original-to-GT | rescaled-to-GT | GT-to-rescaled | color scale. Centered surfaces, 10,000 points; red means error ≥ 0.05.
According to the heatmap, the implicit model learns a very "averaged" shape, which is roughly look correct but dimensional-wise wrong. Honestly, as a human, I feel this kind of rough-shape learing is kind of similar to human learing trace. Although the F1 is very low and error heatmap looks so bad, I still feel it predicts better than the 3 models in question 2 that have much higher F1 but shows discontinuity. The implicit decoder I used is super naive, which may make learing very inefficient. I think the model is currently underiftting, and can predict better if running for more iterations.

