Assignment 2: Single View to 3D

16-825 Learning for 3D Vision | Andrew ID: lnayak

1. Exploring loss functions

1.1. Fitting a voxel grid

Fitted vox beside ground truth

1.2. Fitting a point cloud

Fitted point beside ground truth

1.3. Fitting a mesh

Fitted mesh beside ground truth

2. Reconstructing 3D from single view

The models are trained on RGB images with batch size 16. The voxel model uses 30,000 steps, the point model 15,000, and the mesh model 3,000. Evaluation uses 678 held-out chairs, with one fixed image per chair. Precomputed image features are not used. The voxel loss gives occupied cells 2 times the weight of empty cells. F1 uses 2,048 target surface samples and native mesh coordinates. Mesh and voxel predictions use 2,048 samples; point decoders use their output points. This differs from the starter evaluator's default sampling and voxel alignment.

2.1. Image to voxel grid

F1@0.05: 52.69%.

Input RGB, vox prediction and ground truth mesh Input RGB, vox prediction and ground truth mesh Input RGB, vox prediction and ground truth meshF1 versus distance threshold for vox

2.2. Image to point cloud

F1@0.05: 84.55%.

Input RGB, point prediction and ground truth mesh Input RGB, point prediction and ground truth mesh Input RGB, point prediction and ground truth meshF1 versus distance threshold for point

The decoder predicts 4,096 points. Its loss is Chamfer distance plus 0.1 times a repulsion term. For each point, the term penalizes its four nearest neighbors when they are closer than 0.01. This spreads points that would otherwise collect in tight clusters. The chair surfaces are easier to see, but thin supports and some chair backs are still inaccurate. Section 2.4 compares the same model with and without this term.

2.3. Image to mesh

F1@0.05: 78.67%.

Input RGB, mesh prediction and ground truth mesh Input RGB, mesh prediction and ground truth mesh Input RGB, mesh prediction and ground truth meshF1 versus distance threshold for mesh
RepresentationF1@0.01F1@0.02F1@0.03F1@0.04F1@0.05P@0.05R@0.05
Voxel grid4.7319.2533.1043.9952.6950.1057.32
Point cloud12.2440.1363.0476.5984.5581.2489.76
Mesh8.8534.4655.7569.6578.6773.5585.65

The point cloud has the highest F1 score. Point clouds can place points on separate chair parts without connecting them. The 32 by 32 by 32 voxel grid misses thin parts. The mesh keeps the connections of its starting sphere, so holes and separate parts are harder to recover. Its predictions have folds and overlapping faces, which surface F1 does not measure. Training lengths also differ, so this is not a comparison of representation alone.

2.4. Hyperparameter comparisons

Number of points

This comparison uses a separate 1,000-point model trained for 3,000 steps, matching the 256-point run. It is not the longer-trained model shown in section 2.2.

Predicted pointsPrecision@0.05 (%)Recall@0.05 (%)F1@0.05 (%)
25678.3961.8368.44
100082.1480.4880.28

Changing the output from 256 to 1,000 points increases F1 by 11.84 percentage points. Precision changes by +3.75 points and recall by +18.65 points. The split, seed, batch size, learning rate and training steps are unchanged. More points give denser coverage of the surface. The larger output also needs a larger final layer and uses more target points during training.

256 predicted points:

Input RGB, point_256 prediction and ground truth mesh Input RGB, point_256 prediction and ground truth mesh Input RGB, point_256 prediction and ground truth mesh

1,000 predicted points, with the same training budget:

Input RGB, point_baseline prediction and ground truth mesh Input RGB, point_baseline prediction and ground truth mesh Input RGB, point_baseline prediction and ground truth mesh

As a sampling check, points are also drawn from each ground-truth mesh, separately from the 2,048 evaluation points. Ground-truth samples score 94.57% with 256 points and 99.86% with 1,000 points. Even with the correct shape, denser sampling changes F1. These scores are a reference, not an upper bound.

F1 comparison for 256 and 1,000 predicted points

Point repulsion

Both runs predict 4,096 points and use the same decoder, seed, batch size and 15,000 training steps. Only the repulsion weight changes.

Repulsion weightF1@0.01 (%)F1@0.05 (%)
05.7886.08
0.112.2484.55

Without repulsion, 59.4 to 72.9% of predicted points have four neighbors within 0.002 in the six inputs below. With repulsion, that range is 0.0 to 0.0%. The points cover the surfaces more evenly and the stricter F1 score improves, while F1@0.05 decreases. Repulsion reduces clustering but does not fix every shape error.

Input, Chamfer only, Chamfer plus repulsion, and ground truth points Input, Chamfer only, Chamfer plus repulsion, and ground truth points Input, Chamfer only, Chamfer plus repulsion, and ground truth points Input, Chamfer only, Chamfer plus repulsion, and ground truth points Input, Chamfer only, Chamfer plus repulsion, and ground truth points Input, Chamfer only, Chamfer plus repulsion, and ground truth points Effect of repulsion at different distance thresholds

2.5. Interpreting the point-cloud model

Each image is divided into 16 regions. One region at a time is covered with white, then the predicted point cloud is compared with the original prediction using Chamfer distance. Brighter regions change the output more. The white mask is artificial, so this shows sensitivity to covering a region, not an attention map.

Input image and occlusion sensitivity Prediction before and after covering the most sensitive region Input image and occlusion sensitivity Prediction before and after covering the most sensitive region Input image and occlusion sensitivity Prediction before and after covering the most sensitive region

The next image blends the encoded features of two test images and decodes each blend. The intermediate point clouds show the transition between the two predictions.

Predictions along an interpolation between two encoded test images

3. Other architectures and datasets

3.1. Implicit network

F1@0.05: 51.02%.

Input RGB, implicit prediction and ground truth mesh Input RGB, implicit prediction and ground truth mesh Input RGB, implicit prediction and ground truth meshF1 versus distance threshold for implicit

The decoder predicts occupancy from an image feature and a 3D query coordinate. It is trained for 30,000 steps, with occupied cells weighted 2 times as much as empty cells. It is queried on a 32 by 32 by 32 grid in [-1, 1] cubed. Sine and cosine features at four frequencies help the coordinate network represent small details. Its F1@0.05 is -1.67 percentage points relative to the voxel decoder. Unlike a convolutional voxel decoder, the same coordinate function is used at every location. The examples still simplify some thin chair parts; coordinate features alone do not remove reconstruction errors. The displayed surface is still extracted from a 32-cubed grid, so the visualization retains that resolution limit.

3.2. Parametric network

F1@0.05: 78.92%.

Input RGB, parametric prediction and ground truth mesh Input RGB, parametric prediction and ground truth mesh Input RGB, parametric prediction and ground truth meshF1 versus distance threshold for parametric

The decoder maps sampled 2D coordinates to 3D points, conditioned on the image feature. It is trained for 15,000 steps. Four learned patch identities let it use separate surface patches. Its F1@0.05 is -1.92 percentage points relative to a 1,000-point direct decoder (80.84%). Both use 15,000 steps and Chamfer loss only. This comparison does not use the 4,096-point model from section 2.2. The 2D coordinates encourage a surface parameterization, but the patches can still overlap and do not guarantee a closed mesh.

3.3. Training on three classes

The point-cloud model is trained on chairs, cars and planes using split_3c.json. F1@0.05 over all 1451 test objects is 87.14%.

Input RGB, point_full prediction and ground truth mesh Input RGB, point_full prediction and ground truth mesh Input RGB, point_full prediction and ground truth mesh F1 over the three-class test set
Test classObjectsF1@0.05 (%)
Plane40595.64
Car36893.55
Chair67878.58
Training dataEvaluation dataF1@0.05 (%)
Chairs onlyThe same 678 test chairs80.28
Chairs, cars and planesThe same 678 test chairs78.58

On the same chair test set, three-class training changes F1 by -1.70 percentage points. Both models use the same architecture and batch size, with 3,000 training steps. The chair-only score is from the shorter run used in section 2.4. The three-class model learns a wider range of shapes, but sees fewer chair examples within the same training budget. This comparison therefore measures shared training at a fixed step budget, not performance after equal numbers of chair updates.

Input RGB, point_full_chair prediction and ground truth mesh Input RGB, point_full_chair prediction and ground truth mesh Input RGB, point_full_chair prediction and ground truth mesh