Assignment 2: Single View to 3D

Fitting voxels, point clouds and meshes with 3D losses, then predicting each representation (plus an implicit and a parametric decoder) from a single RGB image of a ShapeNet chair.

All models use the ResNet18 image encoder trained end to end; I never used --load_feat. Hyperparameters were always chosen on 300 training chairs held out for validation, never on the test set. Every test chair is shown from one view picked with a fixed seed, so all models see the same images. Each render below turns 360° in 60 frames.

1. Exploring loss functions

1.1 Fitting a voxel grid

Optimized voxel grid
Ground-truth voxel grid
360-degree render of the optimized voxel grid
360-degree render of the ground-truth voxel grid

Raw 32³ voxels: every cell with occupancy above 0.5 is drawn as a cube.

Loss

voxel_loss = F.binary_cross_entropy_with_logits(voxel_src, voxel_tgt)

Evaluation

I also scored the grids against the ground-truth mesh with the Q2 metric (compute_sampling_metrics, F1@0.05), sampling points from the marching-cubes surface of each grid at occupancy 0.5:

ComparisonMetric1,000 points (Q2 default)10,000 points
Fitted grid vs. GT voxel gridIoU1.000
Fitted surface vs. GT meshF1@0.0591.096.9

Voxel alignment fix

The provided R2N2 binvox grids are rotated 90° about the up axis relative to the ShapeNet meshes. The loader now maps binvox (x, y, z) to mesh (z, y, −x) before aligning bounding boxes, which raises voxel-to-mesh F1@0.05 from 0.695 to 0.999.

1.2 Fitting a point cloud

Optimized point cloud
Ground-truth point cloud
360-degree render of the optimized point cloud
360-degree render of the ground-truth point cloud

5,000 points each. The source starts as Gaussian noise; the target is sampled from the ground-truth mesh.

Loss

The squared distance from each point to its nearest neighbor in the other cloud (knn_points, K = 1), averaged in both directions and summed:

chamfer_loss = meanp∈S minq∈T ‖p − q‖² + meanq∈T minp∈S ‖p − q‖²

Evaluation

After 30,000 Adam steps (lr 4e-4) the fitted cloud scores F1@0.05 = 100.0 against 5,000 points sampled from the mesh. 10,000 steps were not enough: a few Gaussian samples start about 4 units away, and Adam moves each point by roughly the learning rate per step.

1.3 Fitting a mesh

Optimized mesh
Ground-truth mesh
360-degree render of the optimized mesh
360-degree render of the ground-truth mesh

The source is a level-4 ico-sphere (2,562 vertices) deformed by a learned offset per vertex.

Loss

loss = w_chamfer · chamfer_loss + w_smooth · smoothness_loss   (w_chamfer = 1.0, w_smooth = 0.1)

The chamfer term compares 5,000 points sampled from each mesh at every step. smoothness_loss is the uniform Laplacian: the distance from each vertex to the mean of its edge neighbors, averaged over all vertices.

Evaluation

The fit reaches F1@0.05 = 100.0 with both 1,000 and 10,000 points. The surface is still bumpier than the ground truth: the leg tips are uneven and the chair's sharp edges and flat faces are rounded off.

Comparing the three representations

VoxelPoint cloudMesh
LossBCEChamferChamfer + Laplacian
StrengthRegular structure, simplest lossLightweight and preciseContinuous surface, renders directly
WeaknessResolution limited by the gridNo surface, only scattered pointsTopology fixed by the initial shape; needs regularization

2. Reconstructing 3D from a single view

All Q2 decoders take the 512-d ResNet18 feature of one 137×137 view and train for 10,000 steps (unless noted) with batch size 32 and Adam at lr 4e-4, showing each chair from one random view out of its 10. The three test chairs below (an armchair, a folding chair and a wooden chair) and their input views are the same in every section.

2.1 Image to voxel grid

Input
Baselinepos_weight 1
Finalpos_weight 2
Ground truth
Input image of test[0]
test[0]
Baseline voxel prediction for test[0]
F1@0.05 79.2
Final voxel prediction for test[0]
F1@0.05 65.6
Ground-truth mesh of test[0]
Input image of test[200]
test[200]
Baseline voxel prediction for test[200]
F1@0.05 20.0
Final voxel prediction for test[200]
F1@0.05 62.0
Ground-truth mesh of test[200]
Input image of test[400]
test[400]
Baseline voxel prediction for test[400]
F1@0.05 92.3
Final voxel prediction for test[400]
F1@0.05 54.0
Ground-truth mesh of test[400]

Predictions are drawn as the marching-cubes surface at occupancy 0.5. F1@0.05 is F1@0.05 for that chair.

Decoder

A linear layer lifts the feature to a 128×4×4×4 volume. Three ConvTranspose3d layers (kernel 4, stride 2, BatchNorm + ReLU) upsample it to 8³, 16³ and 32³ with 64, 32 and 16 channels, and a 3×3×3 Conv3d outputs one logit per voxel.

Weighting occupied voxels

A chair fills only a small part of the grid, so plain BCE (the baseline) learns to say “empty” wherever it is unsure: its test precision (69.3) is far above its recall (54.5), it misses thin parts, and it reduces the folding chair to a few specks. I added a pos_weight to voxel_loss that scales the loss of occupied cells (the default 1 leaves 1.1 unchanged) and chose it on the held-out chairs:

pos_weight12345
Validation F1@0.0558.562.759.457.653.0
Precision67.760.254.252.147.1
Recall55.166.867.366.162.4

The final model uses weight 2, retrained on all 6,100 training chairs. Section 2.4 analyzes the sweep.

Evaluation

Model (678 test chairs)F1@0.05PrecisionRecall
Baseline, pos_weight 158.069.354.5
Final, pos_weight 262.161.464.8
F1 against distance threshold, baseline voxel model
Baseline
F1 against distance threshold, final voxel model
Final

Change to the evaluation

The provided voxel branch of evaluate() (Mem2Ref, a −π rotation about y, and re-centering) does not put the grid in the mesh frame; scored that way, even the ground-truth voxels only reach F1@0.05 = 53.4. I replaced it with the 1.1 transform: take the marching-cubes surface of sigmoid(logits) in [−1, 1]³ and scale it by 0.5, with no re-centering.

2.2 Image to point cloud

Input
Predicted point cloud
Ground truth
Input image of test[0]
test[0]
Predicted point cloud for test[0]
F1@0.05 84.8
Ground-truth mesh of test[0]
Input image of test[200]
test[200]
Predicted point cloud for test[200]
F1@0.05 91.1
Ground-truth mesh of test[200]
Input image of test[400]
test[400]
Predicted point cloud for test[400]
F1@0.05 97.2
Ground-truth mesh of test[400]

Decoder

An MLP maps the feature straight to 1,000 × 3 coordinates: three hidden layers of width 1,024 with ReLU and a linear output. The loss is chamfer_loss from 1.2 against 1,000 points sampled from the ground-truth surface at every step.

Choosing the training length

One 30,000-step run on the 5,800 non-held-out chairs, with checkpoints scored on the held-out ones:

Steps10,00020,00030,000
Training chamfer loss0.00300.00200.0019
Validation F1@0.0579.477.977.4
Precision80.580.180.2
Recall80.477.876.4

Training loss keeps falling after 10k steps while validation F1@0.05 drops, almost entirely through recall: the decoder starts fitting the training chairs instead of covering new ones. The final model is retrained on all 6,100 chairs for 10,000 steps.

Evaluation

Test F1@0.05 over 678 chairs: 80.1 (precision 81.2, recall 81.3).

F1 against distance threshold, point cloud model

The clouds capture each chair's layout: the armchair's box, the folding chair's backward lean, and the wooden chair's seat, back and legs. They are noisy, though: points form a band around the surface instead of lying on it, and thin legs are only sketched. F1@0.05 accepts any point within 0.05 of the surface (about a tenth of a chair's width), so this spread costs little, and the point decoder scores far above the voxel model (62.1).

2.3 Image to mesh

Input
Chamfer + Laplacianw_smooth 0.01
Bsphere radius 0.5
RegularizedB + C
Ground truth
Input image of test[0]
test[0]
Chamfer and Laplacian mesh prediction for test[0]
F1@0.05 79.5
B mesh prediction for test[0]
F1@0.05 64.7
Regularized mesh prediction for test[0]
F1@0.05 67.3
Ground-truth mesh of test[0]
Input image of test[200]
test[200]
Chamfer and Laplacian mesh prediction for test[200]
F1@0.05 75.9
B mesh prediction for test[200]
F1@0.05 75.0
Regularized mesh prediction for test[200]
F1@0.05 70.2
Ground-truth mesh of test[200]
Input image of test[400]
test[400]
Chamfer and Laplacian mesh prediction for test[400]
F1@0.05 82.9
B mesh prediction for test[400]
F1@0.05 88.9
Regularized mesh prediction for test[400]
F1@0.05 52.3
Ground-truth mesh of test[400]

Decoder

The decoder deforms a level-4 ico-sphere (2,562 vertices, 5,120 faces): an MLP with three hidden layers of width 1,024 maps the feature to an offset per vertex. The loss is chamfer_loss between 1,000 points sampled from the predicted and the ground-truth surfaces, plus w_smooth times the Laplacian smoothness_loss from 1.3.

Model selection

I first chose w_smooth, then tried three changes on top of the best value (0.01):

VariantValidation F1@0.05PrecisionRecall
w_smooth 0.0173.170.377.1
w_smooth 0.1 (starter default)72.369.076.6
w_smooth 167.265.570.0
B: radius 0.575.073.077.8
B + C: radius 0.5, edge + normal67.963.075.0
D + C: ellipsoid, edge + normal67.865.172.1

I retrained three models on all 6,100 chairs: plain Chamfer + Laplacian, B (best on validation) and the regularized B + C (chosen over D + C, which scored within 0.1).

Evaluation

Model (678 test chairs)F1@0.05PrecisionRecall
Chamfer + Laplacian (w_smooth 0.01)74.972.178.6
B: radius 0.573.871.976.7
Regularized (B + C)68.463.575.7
F1 against distance threshold, Chamfer and Laplacian mesh model
Chamfer + Laplacian
F1 against distance threshold, B mesh model
B
F1 against distance threshold, regularized mesh model
Regularized

Voxels vs. point clouds vs. meshes

Representation (final model)Test F1@0.05PrecisionRecall
Voxel grid (2.1, pos_weight 2)62.161.464.8
Point cloud (2.2)80.181.281.3
Mesh (2.3, Chamfer + Laplacian)74.972.178.6

2.4 Effects of hyperparameter variations

Every setting is trained on 5,800 chairs and scored on the same 300 held-out chairs.

Weight of occupied voxels (image to voxel)

Validation F1@0.05, precision and recall of the voxel model against pos_weight from 1 to 5

Raising pos_weight moves along a precision/recall trade-off (numbers in the 2.1 table):

Input
w = 1
w = 2
w = 3
w = 4
w = 5
Input image of the folding chair, test[200]
test[200]
Voxel prediction for test[200] with pos_weight 1
F1@0.05 9.4
Voxel prediction for test[200] with pos_weight 2
F1@0.05 57.6
Voxel prediction for test[200] with pos_weight 3
F1@0.05 73.3
Voxel prediction for test[200] with pos_weight 4
F1@0.05 70.2
Voxel prediction for test[200] with pos_weight 5
F1@0.05 64.1
Input image of the wooden chair, test[400]
test[400]
Voxel prediction for test[400] with pos_weight 1
F1@0.05 82.2
Voxel prediction for test[400] with pos_weight 2
F1@0.05 89.5
Voxel prediction for test[400] with pos_weight 3
F1@0.05 69.4
Voxel prediction for test[400] with pos_weight 4
F1@0.05 79.6
Voxel prediction for test[400] with pos_weight 5
F1@0.05 54.7

The folding chair goes from a few specks at weight 1 to a complete chair at 3, then swells into a blob at 5. The wooden chair is complete at 2, and higher weights only thicken it. The right weight depends on the chair: thin chairs need a high one and solid chairs a low one, and one global weight can only pick the best average. Single chairs are also noisy (the wooden chair's F1@0.05 is not monotonic), which is why the choice uses 300 of them.

Training length (image to point cloud)

From the 2.2 table: training loss falls from 0.0030 at 10k steps to 0.0019 at 30k, but validation F1@0.05 drops from 79.4 to 77.4, almost entirely through recall (80.4 to 76.4) while precision stays near 80. Longer training overfits.

Smoothing, initial shape and regularizers (image to mesh)

w_smooth 0.01
w_smooth 0.1
w_smooth 1
B
B + C
D + C
Mesh prediction for test[200] with w_smooth 0.01
test[200] · F1@0.05 71.6
Mesh prediction for test[200] with w_smooth 0.1
F1@0.05 80.3
Mesh prediction for test[200] with w_smooth 1
F1@0.05 83.4
Mesh prediction for test[200] with B
F1@0.05 85.6
Mesh prediction for test[200] with B + C
F1@0.05 68.4
Mesh prediction for test[200] with D + C
F1@0.05 77.2
Mesh prediction for test[400] with w_smooth 0.01
test[400] · F1@0.05 88.6
Mesh prediction for test[400] with w_smooth 0.1
F1@0.05 40.4
Mesh prediction for test[400] with w_smooth 1
F1@0.05 90.1
Mesh prediction for test[400] with B
F1@0.05 84.7
Mesh prediction for test[400] with B + C
F1@0.05 61.6
Mesh prediction for test[400] with D + C
F1@0.05 63.4

Validation numbers are in the 2.3 table.

Per-chair F1@0.05 is noisy: w_smooth 0.1 scores 40.4 on test[400] but 72.3 on average. Every choice above uses the 300-chair averages.

2.5 Interpreting the point cloud model

Interpolating between two chairs in feature space

α = 0folding chair
α = 0.17
α = 0.33
α = 0.50
α = 0.67
α = 0.83
α = 1armchair
Decoded point cloud at alpha 0
Decoded point cloud at alpha 0.17
Decoded point cloud at alpha 0.33
Decoded point cloud at alpha 0.50
Decoded point cloud at alpha 0.67
Decoded point cloud at alpha 0.83
Decoded point cloud at alpha 1

I encoded the folding chair (test[200]) and the armchair (test[0]), mixed the two 512-d features linearly in 7 steps, and decoded each mix. The thin, tilted folding chair thickens and straightens step by step into the armchair's box, with no jump between two memorized shapes: the decoder has learned a continuous map from features to plausible chairs.

The feature encodes the object, not the view

View 0
View 3
View 6
View 9
Folding chair, input view 0
Folding chair, input view 3
Folding chair, input view 6
Folding chair, input view 9
Prediction from view 0
Prediction from view 3
Prediction from view 6
Prediction from view 9

All four views of the folding chair give the same tilted chair. To measure this, I predicted 50 test chairs (every 13th) from the same four views and averaged chamfer_loss over pairs:

PairMean chamfer
Same chair, different views (predictions)0.0014
Different chairs, same view (predictions)0.0044
Different chairs (ground truth)0.0201

error analysis

test[0]prediction
test[0]ground truth
test[200]prediction
test[200]ground truth
test[400]prediction
test[400]ground truth
Error heatmap of the predicted points, test[0]
89.9% within 0.05
Error heatmap of the ground-truth points, test[0]
85.2% covered
Error heatmap of the predicted points, test[200]
92.1% within 0.05
Error heatmap of the ground-truth points, test[200]
91.3% covered
Error heatmap of the predicted points, test[400]
96.5% within 0.05
Error heatmap of the ground-truth points, test[400]
99.2% covered

Color scale: light blue for distance 0, dark blue for distance 0.1 or more

Predicted points are colored by their distance to the ground-truth surface (precision error), and ground-truth points by their distance to the nearest prediction (recall error).

3. Exploring other architectures

3.1 Implicit network

Input
2.1 voxel decoderpos_weight 2
Implicit, step 2global feature, 32³
Implicit, step 4final, 32³
Implicit, step 4final, 128³
Ground truth
Input image of test[0]
test[0]
2.1 voxel prediction for test[0]
F1@0.05 65.6
Step 2 implicit prediction at 32 cubed for test[0]
F1@0.05 41.8
Final implicit prediction at 32 cubed for test[0]
F1@0.05 67.5
Final implicit prediction at 128 cubed for test[0]
F1@0.05 71.9
Ground-truth mesh of test[0]
Input image of test[200]
test[200]
2.1 voxel prediction for test[200]
F1@0.05 62.0
Step 2 implicit prediction at 32 cubed for test[200]
F1@0.05 57.1
Final implicit prediction at 32 cubed for test[200]
F1@0.05 73.5
Final implicit prediction at 128 cubed for test[200]
F1@0.05 70.3
Ground-truth mesh of test[200]
Input image of test[400]
test[400]
2.1 voxel prediction for test[400]
F1@0.05 54.0
Step 2 implicit prediction at 32 cubed for test[400]
F1@0.05 56.6
Final implicit prediction at 32 cubed for test[400]
F1@0.05 86.0
Final implicit prediction at 128 cubed for test[400]
F1@0.05 85.6
Ground-truth mesh of test[400]

Implicit surfaces are extracted with marching cubes at each model's chosen occupancy threshold (0.3 for step 2, 0.7 for step 4). The steps are explained below.

Decoder

The decoder is a function f(c, p) from the 512-d image feature c and a 3D location p in (−1, 1)³ to an occupancy logit:

Each training step queries 2,048 cell centers per chair, drawn uniformly (ONet found uniform sampling best; here the ground truth only exists at cell centers). The loss is plain BCE. A full 32³ query per chair would not fit in 4 GB at batch size 32.

Choosing the occupancy threshold

ONet picks the extraction threshold on validation data (0.2 for single-view reconstruction) rather than using 0.5:

Threshold0.20.30.40.5
Validation F1@0.0550.958.355.643.6
Precision45.856.964.459.2
Recall58.661.152.638.4

The threshold plays the role of pos_weight in 2.1. A chair occupies only 4.6% of the cells on average (median 3.4%), so plain BCE learns low occupancies and a 0.5 cut under-fills: 149 of the 678 test chairs come out completely empty at 0.5, but only 9 at 0.3.

How the decoder evolved

Each step changes one thing and keeps the same encoder, training length and protocol (train on 5,800 chairs, choose the threshold on the 300 held-out, report the test set):

StepChangeThresholdVal F1@0.05Test F1@0.05Test P / REmpty test predictions
1Global-feature decoder, uniform sampling0.543.640.057.2 / 35.1149
2Threshold chosen on validation0.358.356.454.5 / 60.19
3+ attended local image features0.449.0–––
4Step 2 + Fourier features + balanced sampling0.763.565.762.3 / 71.10

For reference, the 2.1 voxel decoders score 58.0 (plain BCE) and 62.1 (pos_weight 2) on the test set, so only step 4 beats them.

Step 2 → the global feature looked like the bottleneck. With a tuned threshold the implicit decoder (56.4) lands next to the 2.1 decoder trained with the same plain BCE (58.0). Both read the same 512-d feature, which suggested the limit was the global code — the problem Convolutional Occupancy Networks [2] solves with local features. Its renders are smooth blobs, and the folding chair's thin frame is reduced to fragments.

Step 3: local features (did not help). The natural single-image version of local features is pixel-aligned, as in PIFu [3]: project each query point into the image and read the feature map there. The R2N2 cameras do not line up with these meshes, though. Projecting ground-truth surfaces through the provided camera parameters, under every axis convention I tried, put at best 81% of the points on the chair's silhouette, with chair-dependent misalignment; the renders were most likely made from differently normalized ShapeNet models. Instead, each query point cross-attends to the ResNet18 layer3 and layer4 maps (106 positions with learned position embeddings) and learns the correspondence itself. This overfit: training loss ended lower than step 2's (0.087 vs. 0.090) but validation F1@0.05 was 9 points worse. Learned attention has no built-in link between a 3D location and an image location, so the extra capacity went into fitting the training images. I stopped it before the final retrain, so it has no test score.

Step 4: Fourier features and balanced sampling. Both target weaknesses that the global decoder exposed:

Together they raise test F1@0.05 from 56.4 to 65.7, past the weighted voxel decoder. The difference already shows when overfitting a single chair: step 2 still predicted an empty grid after 400 steps, while step 4 reached IoU 1.0 after 300. The two changes were added together, so the table does not separate their contributions. In the renders, the folding chair goes from fragments to a connected back, seat and legs, and every test chair gets a non-empty prediction.

Evaluation

Model (678 test chairs)F1@0.05PrecisionRecallEmpty
2.1 voxel decoder, pos_weight 158.069.354.517
2.1 voxel decoder, pos_weight 262.161.464.80
Implicit step 2, threshold 0.356.454.560.19
Implicit step 4, threshold 0.765.762.371.10
F1 against distance threshold, implicit step 2 at occupancy 0.3
Step 2, threshold 0.3
F1 against distance threshold, implicit step 4 at occupancy 0.7
Step 4 (final), threshold 0.7

Querying at higher resolution

A continuous decoder can be queried on any grid, which a voxel decoder cannot. Test F1@0.05 over all 678 chairs:

Query gridStep 2Step 4
32³ (training grid)56.565.7
64³56.766.2
128³56.766.3

References

  1. [1] Occupancy Networks: Learning 3D Reconstruction in Function Space. Mescheder et al., CVPR 2019. arXiv:1812.03828
    Represents a shape as the decision boundary of a network that maps a 3D point and an image (or shape) code to an occupancy probability, so a mesh can be extracted at any resolution. Used here: the residual fully connected decoder, uniform point sampling, and choosing the extraction threshold on validation data.
  2. [2] Convolutional Occupancy Networks. Peng et al., ECCV 2020. arXiv:2003.04618
    Replaces ONet's single global code with convolutional features on planes or a 3D grid that each query point interpolates, so the decoder sees local detail. Used here: adding the projected code before every block, and the motivation for local features in step 3.
  3. [3] PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. Saito et al., ICCV 2019. arXiv:1905.05172
    Projects each 3D query point into the image with the known camera and conditions the implicit decoder on the image feature at that pixel and the point's depth. Used here: the intended step-3 design, which the misaligned R2N2 cameras ruled out.
  4. [4] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Mildenhall et al., ECCV 2020. arXiv:2003.08934
    Fits a scene with a coordinate MLP whose inputs first pass through sines and cosines at octave frequencies (positional encoding). Used here: the step-4 encoding of the query point with 6 frequencies.
  5. [5] Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Tancik et al., NeurIPS 2020. arXiv:2006.10739
    Shows that MLPs on raw coordinates are biased toward low frequencies and that a Fourier feature mapping removes this bias; frequencies set too high cause noisy artifacts. Used here: why step 4 helps thin parts, and why its 128³ surfaces ripple.

3.2 Parametric network

Input
Atlas meshone color per patch
Predicted points1,000 points
Ground truth
Input image of test[0]
test[0]
AtlasNet mesh for test[0], one color per patch
AtlasNet point cloud for test[0], one color per patch
F1@0.05 78.5
Ground-truth mesh of test[0]
Input image of test[200]
test[200]
AtlasNet mesh for test[200], one color per patch
AtlasNet point cloud for test[200], one color per patch
F1@0.05 90.6
Ground-truth mesh of test[200]
Input image of test[400]
test[400]
AtlasNet mesh for test[400], one color per patch
AtlasNet point cloud for test[400], one color per patch
F1@0.05 93.2
Ground-truth mesh of test[400]

Decoder

The decoder follows AtlasNet [6]: the surface is the union of K learned parametrizations. Each patch is its own MLP fk(c, u) that maps a point u of the unit square, concatenated with the image feature c, to a 3D point (widths 1024, 512, 256, 128, ReLU, tanh output). The paper's sphere variant samples u on the unit sphere instead, giving one closed patch.

The model outputs 1,000 points, 1000/K per patch, with fresh random u every step, and trains with chamfer_loss against 1,000 ground-truth surface points; it is scored like 2.2. For the mesh renders, each square patch is sampled on a 32×32 grid whose connectivity is carried to 3D (the sphere uses level-4 ico-sphere faces).

Number of patches

1 patch
5 patches
25 patches
1 sphere
Ground truth
1-patch model for test[400]
val F1@0.05 71.1
5-patch model for test[400]
val F1@0.05 74.2
25-patch model for test[400]
val F1@0.05 75.1
Sphere model for test[400]
val F1@0.05 73.5
Ground-truth mesh of test[400]
test[400]
Variant1 patch5 patches25 patches1 sphere
Validation F1@0.0571.174.275.173.5
Precision76.877.177.473.8
Recall68.272.874.374.7

The final model uses 25 patches, the paper's choice for single-view reconstruction, retrained on all 6,100 chairs.

Evaluation

Test F1@0.05 over 678 chairs: 76.5 (precision 78.5, recall 75.9), no empty predictions.

F1 against distance threshold, AtlasNet model
Final model (678 test chairs)F1@0.05Output
2.1 voxel decoder62.132³ grid
3.1 implicit decoder (step 4)65.7occupancy at any point
2.3 deformed ico-sphere74.9one closed mesh
3.2 AtlasNet, 25 patches76.5points and a patch mesh
2.2 point decoder80.11,000 points

References

  1. [6] AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. Groueix et al., CVPR 2018. arXiv:1802.05384
    Represents a surface as the union of K learned parametrizations: each MLP maps points of a unit square (or one sphere) plus a shape code to 3D, trained with the Chamfer distance, so points or a mesh can be generated at any resolution. Used here: the decoder, the patch counts and the sphere variant.
  2. [7] Deep Geometric Prior for Surface Reconstruction. Williams et al., CVPR 2019. arXiv:1811.10943
    Reconstructs a surface from a single noisy point cloud, with no training set, by fitting one small MLP per local chart with an optimal-transport loss and a consistency loss between overlapping charts. Used here: a contrast that explains the seams between AtlasNet patches.

3.3 Extended dataset for training

I retrained the 2.2 point decoder on the extended dataset (split_3c.json): the 3D-R2N2 [8] renderings of three ShapeNet [9] classes.

SplitChairsCarsPlanesTotal
Train6,1003,1463,64012,886
Test6783684051,451

Quantitative comparison

Test F1@0.05 (precision / recall)Chairs (678)Cars (368)Planes (405)All (1,451)
Trained on chairs80.1 (81.2 / 81.3)68.5 (75.6 / 64.6)68.0 (63.9 / 79.3)–
Trained on 3 classes79.9 (81.4 / 80.5)93.5 (93.9 / 93.7)96.7 (97.1 / 96.5)88.5 (89.5 / 88.5)
F1 against threshold on the test chairs, chair-only model
Chairs, trained on chairs
F1 against threshold on the test chairs, three-class model
Chairs, trained on 3 classes
F1 against threshold on the test cars, three-class model
Cars, trained on 3 classes
F1 against threshold on the test planes, three-class model
Planes, trained on 3 classes

Qualitative comparison

Input
Trained on chairs
Trained on 3 classes
Ground truth
Input image of chair test[0]
chair test[0]
Chair-only model's points for chair test[0]
F1@0.05 85.0
Three-class model's points for chair test[0]
F1@0.05 86.6
Ground-truth mesh of chair test[0]
Input image of chair test[200]
chair test[200]
Chair-only model's points for chair test[200]
F1@0.05 91.2
Three-class model's points for chair test[200]
F1@0.05 92.6
Ground-truth mesh of chair test[200]
Input image of chair test[400]
chair test[400]
Chair-only model's points for chair test[400]
F1@0.05 97.6
Three-class model's points for chair test[400]
F1@0.05 95.8
Ground-truth mesh of chair test[400]

Q2's three test chairs. Views are drawn with a fixed seed over the 3-class test set, so they can differ from the Q2 views.

The two models predict almost the same shapes (the armchair's box, the folding chair's slanted back and seat with a thin spread of leg points). The 3-class model gives slightly sharper tall backs, as in the third chair.

3D consistency and diversity

Measured as in 2.5, on 50 test shapes per class seen from views 0, 3, 6 and 9: the mean chamfer_loss between predictions of the same shape from different views (consistency, lower is better), between predictions of different shapes from the same view (diversity), and between the ground truths of different shapes.

ClassModelSame shape, other viewOther shape, same viewGT of other shapesDiversity / consistency
Chairtrained on chairs1.364.4420.203.3×
trained on 3 classes1.265.2720.074.2×
Cartrained on chairs2.252.673.901.2×
trained on 3 classes0.500.923.861.8×
Planetrained on chairs1.592.315.321.4×
trained on 3 classes0.230.675.352.9×

Mean chamfer_loss ×10−3.

One network can cover three classes without hurting chair accuracy, and the extra classes make its chair predictions a little more varied and more view-invariant. A decoder trained on one class can only produce that class, whatever the input. This supports the choice in 3D-R2N2 [8] to train one network on all 13 classes.

References

  1. [8] 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. Choy et al., ECCV 2016. arXiv:1604.00449
    Predicts a 32³ voxel grid from one or more views with a 3D convolutional LSTM, trained as one network on 13 ShapeNet classes. Used here: its renderings, voxels and splits are the dataset of this assignment.
  2. [9] ShapeNet: An Information-Rich 3D Model Repository. Chang et al., arXiv 2015. arXiv:1512.03012
    A large repository of 3D CAD models organized by category. Used here: the ground-truth meshes of chairs (03001627), cars (02958343) and planes (02691156).