Assignment 2 — Learning 3D

1.1. Fitting a voxel grid

Optimized
360-degree rotation of the optimized voxel grid
Ground truth
360-degree rotation of the ground-truth voxel grid

1.2. Fitting a point cloud

Optimized
360-degree rotation of the optimized point cloud
Ground truth
360-degree rotation of the ground-truth point cloud

1.3. Fitting a mesh

Optimized
360-degree rotation of the optimized mesh
Ground truth
360-degree rotation of the ground-truth mesh

For 2.1–2.3, training and evaluation use cached ResNet18 image features (--load_feat). Scores are averaged over 678 test objects. Each example shows ten matching views around 360°.

2.1. Image to voxel grid

Decoder: MLP with two hidden layers of width 512 and ReLU, followed by a linear output layer for 32 × 32 × 32 voxel logits. Loss: binary cross-entropy with logits.

F1@0.05: 40.43

Example 1

Input RGB
Input RGB for image to voxel grid, example 1
Trained prediction
360-degree rotation of the predicted vox, example 1
Ground truth
360-degree rotation of the ground-truth mesh, example 1

Example 2

Input RGB
Input RGB for image to voxel grid, example 2
Trained prediction
360-degree rotation of the predicted vox, example 2
Ground truth
360-degree rotation of the ground-truth mesh, example 2

Example 3

Input RGB
Input RGB for image to voxel grid, example 3
Trained prediction
360-degree rotation of the predicted vox, example 3
Ground truth
360-degree rotation of the ground-truth mesh, example 3
F1 score versus distance threshold
F1 curve for the vox MLP decoder

2.2. Image to point cloud

Decoder: MLP with three hidden layers of width 512 and ReLU, followed by a linear output layer for 1,000 XYZ points. Loss: symmetric squared Chamfer distance.

F1@0.05: 78.65

Example 1

Input RGB
Input RGB for image to point cloud, example 1
Trained prediction
360-degree rotation of the predicted point, example 1
Ground truth
360-degree rotation of the ground-truth mesh, example 1

Example 2

Input RGB
Input RGB for image to point cloud, example 2
Trained prediction
360-degree rotation of the predicted point, example 2
Ground truth
360-degree rotation of the ground-truth mesh, example 2

Example 3

Input RGB
Input RGB for image to point cloud, example 3
Trained prediction
360-degree rotation of the predicted point, example 3
Ground truth
360-degree rotation of the ground-truth mesh, example 3
F1 score versus distance threshold
F1 curve for the point MLP decoder

2.3. Image to mesh

Decoder: MLP with two hidden layers of width 512 and ReLU, predicting vertex offsets for a level-4 sphere with 2,562 vertices. Loss: Chamfer distance plus Laplacian smoothing, with weights 1.0 and 0.1.

F1@0.05: 71.25

Example 1

Input RGB
Input RGB for image to mesh, example 1
Trained prediction
360-degree rotation of the predicted mesh, example 1
Ground truth
360-degree rotation of the ground-truth mesh, example 1

Example 2

Input RGB
Input RGB for image to mesh, example 2
Trained prediction
360-degree rotation of the predicted mesh, example 2
Ground truth
360-degree rotation of the ground-truth mesh, example 2

Example 3

Input RGB
Input RGB for image to mesh, example 3
Trained prediction
360-degree rotation of the predicted mesh, example 3
Ground truth
360-degree rotation of the ground-truth mesh, example 3
F1 score versus distance threshold
F1 curve for the mesh MLP decoder

2.4. Point-count ablation

Varying n_points across 1,000, 2,500, and 5,000. Each run uses a two-hidden-layer MLP with width 512 and ReLU, cached ResNet18 features (--load_feat), and Chamfer loss. Ground-truth surface sampling remains at 1,000 points. Scores use all predicted points and are averaged over 678 test objects.

Predicted pointsF1@0.05
1,00078.51
2,50083.30
5,00084.84

Example 1

1,000 points
360-degree rotation of the 1,000-point prediction, example 1
2,500 points
360-degree rotation of the 2,500-point prediction, example 1
5,000 points
360-degree rotation of the 5,000-point prediction, example 1
Ground truth
360-degree rotation of the ground-truth mesh, example 1

Example 2

1,000 points
360-degree rotation of the 1,000-point prediction, example 2
2,500 points
360-degree rotation of the 2,500-point prediction, example 2
5,000 points
360-degree rotation of the 5,000-point prediction, example 2
Ground truth
360-degree rotation of the ground-truth mesh, example 2

Example 3

1,000 points
360-degree rotation of the 1,000-point prediction, example 3
2,500 points
360-degree rotation of the 2,500-point prediction, example 3
5,000 points
360-degree rotation of the 5,000-point prediction, example 3
Ground truth
360-degree rotation of the ground-truth mesh, example 3

Conclusion: After seeing how sparse the original 1,000-point predictions were, I expected increasing the point count to recover finer detail. However, the 2,500- and 5,000-point outputs appeared to make the existing coarse shapes denser without clearly recovering additional surface geometry in the visualized examples. Although F1@0.05 increased from 78.51 to 84.84, denser sampling can improve recall by giving more ground-truth samples a predicted point within the matching threshold. This suggests that part of the score increase reflects denser coverage of the coarse shape, rather than a clear improvement in fine reconstruction detail.

2.5. View consistency

The test object is Example 1 from 2.1. Each RGB view is predicted independently using the same trained voxel MLP with two hidden layers of width 512 and cached ResNet18 features (--load_feat). All prediction GIFs use the same ten render angles, camera distance, elevation, and scale. The ground-truth mesh is shared across all inputs.

Input view 06 (29°)
RGB view 06 of Q2.1 Example 1
Predicted voxels
360-degree rotation of the voxel prediction from input view 06
Input view 05 (110°)
RGB view 05 of Q2.1 Example 1
Predicted voxels
360-degree rotation of the voxel prediction from input view 05
Input view 09 (164°)
RGB view 09 of Q2.1 Example 1
Predicted voxels
360-degree rotation of the voxel prediction from input view 09
Input view 00 (212°)
RGB view 00 of Q2.1 Example 1
Predicted voxels
360-degree rotation of the voxel prediction from input view 00
Shared ground truth
360-degree rotation of the ground-truth mesh shared by all four views

Analysis: I wanted to check view consistency by investigating whether the learned decoder predicts similar 3D structure from different views of the same object. Overall, the predictions for this example appear reasonably consistent across the four input views. The seat, backrest, and armrests have a similar arrangement, and the reconstructed chair keeps the same orientation even as the input viewpoint changes.

There are small differences in the width and thickness of the predicted surfaces, but the main structure remains similar. All four predictions miss the chair legs, even though legs are visible in the input images. They also have smoother, more rounded surfaces than the rectangular ground-truth mesh. This shows that view consistency and reconstruction accuracy are separate: the model can produce a stable coarse shape while repeating the same missing details across views. These observations are qualitative and limited to this one test object.

3.1. Implicit occupancy network

A shared MLP takes an L2-normalized 512-dimensional image feature and an XYZ location and predicts one occupancy logit. The decoder has 3 hidden layers of width 256 with ReLU. All 32 × 32 × 32 locations in the normalized cube (−1, 1)3 are queried for every image during training and evaluation. The ResNet18 image encoder and decoder are trained jointly from RGB using binary cross-entropy with logits.

F1@0.05: 41.24 / 100, averaged over all 678 test objects. F1 compares 1,000 sampled points on each predicted and ground-truth surface. Each example shows ten matching views around 360°.

Example 1

Input RGB
Input RGB for implicit reconstruction, example 1
Implicit prediction
360-degree rotation of the implicit prediction, example 1
Ground truth
360-degree rotation of the ground-truth mesh, example 1

Example 2

Input RGB
Input RGB for implicit reconstruction, example 2
Implicit prediction
360-degree rotation of the implicit prediction, example 2
Ground truth
360-degree rotation of the ground-truth mesh, example 2

Example 3

Input RGB
Input RGB for implicit reconstruction, example 3
Implicit prediction
360-degree rotation of the implicit prediction, example 3
Ground truth
360-degree rotation of the ground-truth mesh, example 3
Mean test F1 score versus distance threshold
Mean test F1 curve for the implicit occupancy decoder