All outputs on this page are produced by running python main.py in this folder, which writes them to outputs/. python main.py --train retrains the five models first.
Left: optimized voxel grid. Right: ground truth voxel grid.
Left: optimized point cloud. Right: ground truth point cloud.
Left: optimized mesh. Right: ground truth mesh.
Left to right: input RGB, predicted voxel grid, ground truth mesh.
F1@0.05 = 72.82. The decoder is a 3D convolutional network that upsamples the image feature to a 32 x 32 x 32 grid, trained with binary cross entropy plus a dice loss.
Left to right: input RGB, predicted point cloud, ground truth mesh.
F1@0.05 = 77.11. The decoder is an MLP that outputs 1000 points, trained with the chamfer loss. This model was trained with --load_feat, on the provided ResNet18 features.
Left to right: input RGB, predicted mesh, ground truth mesh.
F1@0.05 = 72.63. The decoder is an MLP that moves every vertex of an ico sphere, trained with the chamfer loss plus a smoothness loss.
Comparison. Point cloud (77.11) is clearly ahead, and the voxel grid (72.82) and the mesh (72.63) are about tied. The evaluation picks a random view of each chair, so a 0.2 difference between the last two is within run-to-run noise.
Hyperparameter: vox_pos_weight w, the weight on occupied voxels in the voxel BCE loss. Only about 4% of the voxels are occupied. Four voxel models were trained the same way except for w in {1, 2, 3, 6}: the smaller voxel decoder, plain weighted BCE without dice, 10000 iterations. Each is scored on the full test set at 9 marching cubes thresholds t.
Weighted BCE is minimised by p_w = w p / (w p + 1 - p), which only adds log(w) to the logit. Thresholding p_w at t is the same as thresholding the unweighted probability at p* = t / (t + w (1 - t)), so changing w should only move the threshold.
So pos_weight mostly moves the decision threshold. F1@0.05 is best when a voxel counts as occupied at about 1/3 confidence, because a missing leg costs more recall than a slightly thick surface costs precision. w = 2 is best at t = 0.5 because it maps 0.5 to p* = 1/3. w = 1 peaks about 3 F1 lower than w >= 2 even at its best threshold, which suggests up-weighting occupied voxels helps a little with thin parts, but each w is a single run, so this gap may be noise.
Error maps. Middle: the prediction colored by the distance from each point to the ground truth, which shows precision errors. Right: the ground truth colored by the distance to the prediction, which shows the parts the model missed. Dark is close and yellow is 0.1 or more away, with 0.05 being the F1 threshold. The mesh covers almost all of the ground truth but its stretched faces sit away from the surface, so its errors are mostly precision errors. The voxel model fails on thin structures, such as the legs of #200, which come out too thick.
Occlusion sensitivity. A 24x24 black patch slides over the input with stride 8, and the heat map shows how much the prediction changes (1-IoU for voxels, chamfer distance for points and meshes).Last two columns show the prediction and the prediction with the most sensitive patch removed. The heat sits on the chair rather than the background, mostly on the seat and the back, so the models use the object itself. Note that removing one patch can change the shape a lot, for example the mesh for #400 loses its backrest.
Left to right: input RGB, predicted occupancy, ground truth mesh.
F1@0.05 = 67.10. The decoder takes a 3D point in (-1,1)^3 with 4 frequencies of sin/cos Fourier features and predicts its occupancy with a 4 layer MLP of width 128. We have the image feature is added at every layer. It is queried on a 32x32x32 grid and trained with the same loss as the voxel model for 10000 iterations. We have the second figure shows the occupancy on the middle slice of the grid and the query points predicted occupied.
Notice that the scores lower than the voxel decoder (72.82). It has about 0.4M parameters against 11.7M for the voxel decoder and was trained for half as long. Its shapes are smoother, with rounded blobs for the armchairs, but it does recover thin parts such as the legs of the bar stool (#255).
Left to right: input RGB, predicted point cloud, ground truth mesh.
F1@0.05 = 73.71. The decoder maps samples from 8 unit squares to 3D points. Each square has a learned 8d embedding, and a 3 layer MLP of width 256 maps (u,v,embedding) to xyz with the image feature added at each layer. We are sampling random (u, v), whcih we draw every time step. IT is trained with chamfer for 4000 iterations. Our second figure evaluates a regular 20 x 20 grid on every square, colored by square.
Notice the scores lower than the point cloud decoder (77.11). with about about 0.7M parameters against 8.8M, and each square has to map to a connected piece of surface. We notice overlap and bunch up instead of each covering one part of the chair, which suggests 4000 iterations were not enough for them to spread out.