Assignment 2: Single View to 3D

Author: Po-Ting Ko

Andrew ID: potingk

Note: all models in Q2 and Q3 were trained and evaluated with --load_feat (the provided precomputed ResNet-18 features).

1. Exploring loss functions

Each fit optimizes the shape directly (no network) to match one training chair from the R2N2 [1] / ShapeNet [2] data. Left: fitted source, right: target.

1.1 Fitting a voxel grid

Loss: binary cross-entropy on logits, F.binary_cross_entropy_with_logits(voxel_src, voxel_tgt).

Voxel fit

1.2 Fitting a point cloud

Loss: chamfer distance, implemented with knn_points only:

L(P, Q) = (1/|P|) Σp∈P minq∈Q ‖p − q‖2 + (1/|Q|) Σq∈Q minp∈P ‖p − q‖2

The first term pulls every predicted point onto the target; the second makes every part of the target have a predicted point nearby.

Point cloud fit

1.3 Fitting a mesh

Loss: w_chamfer · chamfer + w_smooth · smoothness, with chamfer between points sampled from both meshes and PyTorch3D's mesh_laplacian_smoothing (uniform weights) as the smoothness term. Source: ico_sphere(4).

Mesh fit

2. Reconstructing 3D from single view

All decoders take the 512-dimensional feature of the input image from a pretrained, frozen ResNet-18 [9] and are trained with PyTorch3D [8].

2.1 Image to voxel grid

Input RGB, predicted voxel grid (marching cubes at 0.5) and ground-truth mesh, rotating together:

Voxel predictions

Decoder

Linear(512 → 256·4³) + ReLU, reshaped to a 256 × 4³ grid, then three ConvTranspose3d (kernel 4, stride 2) + BatchNorm + ReLU blocks (4 → 8 → 16 → 32) and a Conv3d to one logit per voxel (11.2M parameters). Loss: BCE. Adam, lr 4e-4, batch 32, 18k iterations (chosen on a held-out validation split).

Evaluation

F1@0.05 = 62.18

F1 vs threshold, voxels

Decoder F1@0.05 (10k iterations)
Single linear layer 512 → 32³ 45.17
Conv decoder (above) 57.19

2.2 Image to point cloud

Input RGB, predicted point cloud and ground-truth mesh:

Point cloud predictions

Decoder

MLP 512 → 1024 → 2048 → 1000·3 with ReLU, reshaped to 1000 points. Loss: chamfer (1.2) [3]. 14k iterations (chosen on val).

Evaluation

F1@0.05 = 76.26

F1 vs threshold, points

2.3 Image to mesh

Input RGB, predicted mesh and ground-truth mesh:

Mesh predictions

Decoder

MLP 512 → 1024 → 2048 → 2562·3 predicting one offset per vertex of ico_sphere(4); faces stay fixed. The last layer is initialized to zero, so training starts from the undeformed sphere. Loss: chamfer + 0.1 · Laplacian smoothness (the default weights). 6k iterations (chosen on val).

Evaluation

F1@0.05 = 70.41

F1 vs threshold, mesh

Similar mesh predictions

Comparing voxels, point clouds and meshes

F1@0.05, all with 1000 evaluation points; "average chair" = the same prediction for every test image:

Representation F1@0.05 Average chair Gain from the image
Voxels (2.1) 62.18 57.60 +4.6
Mesh (2.3) 70.41 60.38 +10.0
Point cloud (2.2) 76.26 54.49 +21.8

2.4 Effect of hyperparameters

I varied one hyperparameter for points and one for meshes, each answering a question raised above.

Number of points (2.2: does a denser cloud give more detail?)

n_points F1@0.05 F1@0.02 Recall@0.05 Precision@0.05 Spacing / ground truth Gain over average chair
1000 76.26 22.4 77.4 81.3 0.44 +21.8
2000 82.03 26.5 84.4 84.2 0.22 –
5000 85.76 33.1 90.4 85.5 0.16 +8.3

(F1@0.02, recall, precision and spacing against the same 10k ground-truth points per chair; spacing = median distance to the nearest other point, divided by the same for the same number of ground-truth points.)

n_points comparison

F1 vs threshold and point spacing by n_points

(a) F1 against the same 10k ground-truth points per chair. (b–d) Distance from each point to its nearest other point, for the prediction and for the same number of points sampled from the ground truth: the predicted points are much closer together, more so with more points.

Smoothness weight and mesh regularizers (2.3: can the folded surface be fixed?)

All runs start from the 2.3 model and change one thing. Surface quality: normal inconsistency (mean 1 − cos of the angle between adjacent faces; 0 = smooth).

w_smooth sweep

w_smooth F1@0.05 Normal inconsistency
0 72.13 1.223
0.1 (2.3) 70.41 0.290
1 68.22 0.104
10 58.26 0.021

(Only the ratio w_smooth / w_chamfer matters with Adam, so this also covers w_chamfer.)

Edge and normal regularizers

Model F1@0.05 Normal inconsistency
2.3 model 70.41 0.290
+ edge length + normal consistency losses 68.98 0.064
+ finer template ico_sphere(5) 72.02 0.072

2.5 Interpreting the models

Two questions, one for each failure seen above.

Why do thin parts disappear from the voxel predictions? The same prediction drawn at three probability thresholds:

Voxel confidence

The folding chair almost vanishes at p > 0.5 but its frame and legs are clearly there at p > 0.2. Over the test set, the mean predicted probability on occupied ground-truth voxels grows with how thick the part is (number of occupied neighbours out of 26):

Confidence vs thickness

Occupied neighbours 2 (thin rod) 6 8 (thin board) 17 (surface of a solid part) 26 (inside)
Mean predicted probability 0.15 0.17 0.25 0.39 0.53

The model does locate thin parts; it just gives them low probability. Thin parts are a tiny share of the occupied voxels (and only 4.3% of voxels are occupied), so BCE gains little from them and pushes them toward "empty".

Why do point and mesh predictions look alike for different chairs? Each predicted point is colored by its output index (by where that index lands on average), and each mesh vertex by its starting direction on the template sphere:

Template correspondence

The same colors land on the same parts of every chair: for points, teal at the top of the backrest, pink at the legs; for the mesh, the green part of the sphere always becomes the backrest and the purple part the legs. Both decoders have learned one template chair with a fixed mapping from outputs to chair parts and deform it only slightly per image (median standard deviation of a point's position across test chairs: 0.07, for a chair about 1.0 tall).

What does each model do when it knows nothing? Given the mean training feature, the voxel decoder predicts an empty grid (BCE pushes uncertain voxels to the 96% "empty" prior), while the point and mesh decoders predict a generic average chair (chamfer pushes uncertain points to the middle of all plausible positions). This is the same behaviour as above at a smaller scale: voxel models drop what they are unsure about, point and mesh models blur it into an average.

3. Exploring other architectures / datasets

3.1 Implicit network

Input RGB, the 2.1 conv decoder, the implicit decoder queried at 32³ and at 128³ (same model), and the ground-truth mesh (all voxel models at isovalue 0.3, see below):

Implicit network predictions

Model

An occupancy network [6]: f(feature, x, y, z) → occupancy logit, with (x, y, z) ∈ [−1, 1)³. The query point goes through a positional encoding [7] (sin/cos at 5 frequencies), is combined with the image feature in the first layer, and passes through a 4-layer MLP of width 256 (0.35M parameters).

Evaluation

Threshold. At the default isovalue 0.5 the implicit model scores only 43.0, with 101 empty predictions. Its probabilities are flatter than the conv decoder's (its most confident voxel is about 0.69 for a typical chair, vs. 0.96), so a 0.5 threshold removes most of the shape. Both models do best at isovalue 0.3 on the validation set, so both are compared at 0.3:

Model (isovalue 0.3) Parameters F1@0.05
Conv voxel decoder (2.1) 11.2M 63.89
Implicit, queried at 32³ 0.35M 62.38
Implicit, queried at 128³ (same model) 0.35M 62.64

F1 vs threshold, implicit

3.2 Parametric network

Input RGB, the 2.2 MLP, the parametric model's points and mesh (colored by patch), and the ground-truth mesh:

Parametric network predictions

Model

AtlasNet [4]: 8 MLPs g_k(feature, u, v) → (x, y, z), each folding the unit square onto part of the surface. Each patch: [feature, u, v] → 512 → 256 → 128 → 3 with ReLU (3.4M parameters in total).

Evaluation

F1@0.05 = 72.85 (2.2 MLP: 76.26)

F1 vs threshold, parametric

Against the same 10k ground-truth points per chair:

Model F1@0.01 F1@0.02 F1@0.05 Spacing / ground truth
MLP, 1000 points (2.2) 4.6 22.4 78.1 0.44
Parametric, 1000 points 5.1 24.3 74.5 0.70
MLP, 5000 points (separate model, 2.4) 7.2 33.1 86.1 0.16
Parametric, 5000 points (same model) 13.9 42.1 81.0 0.70

3.3 Extended dataset for training

Input RGB, the 2.2 model trained on chairs only, the same model trained on chairs, cars and planes, and the ground-truth mesh. Three test chairs (the 2.2 examples), then a plane and a car; row labels give F1 for the chair-only / 3-class model:

One class vs three classes

Setup

dataset_location.py selects the 3-class dataset when R2N2_FULL=1 is set. I trained the 2.2 point-cloud model (1000 points, same settings and the same number of iterations) on chairs, cars and planes. The chair train/test lists of the 3-class split are identical to the chair-only split, so both models are compared on exactly the same 678 test chairs (same views).

Evaluation

Test set Trained on chairs Trained on 3 classes
Chairs (678) 76.15 77.06
Cars (368) 61.11 93.02
Planes (405) 42.02 95.72
Diversity (prediction pair distance / ground-truth pair distance) Trained on chairs Trained on 3 classes
Chairs 0.43 0.42
Cars 0.70 0.49
Planes 0.69 0.31

(F1@0.05. Diversity: for 500 random pairs of test shapes of one class, the distance between the two predictions divided by the distance between the two ground truths; lower = closer to a single template.)

References

  1. C. B. Choy, D. Xu, J. Gwak, K. Chen, S. Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV 2016.
  2. A. X. Chang et al. ShapeNet: An Information-Rich 3D Model Repository. arXiv:1512.03012, 2015.
  3. H. Fan, H. Su, L. Guibas. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. CVPR 2017.
  4. T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, M. Aubry. AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. CVPR 2018 (arXiv:1802.05384).
  5. N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, Y.-G. Jiang. Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. ECCV 2018.
  6. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, A. Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR 2019 (arXiv:1812.03828).
  7. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV 2020.
  8. N. Ravi et al. Accelerating 3D Deep Learning with PyTorch3D. arXiv:2007.08501, 2020.
  9. K. He, X. Zhang, S. Ren, J. Sun. Deep Residual Learning for Image Recognition. CVPR 2016.