16-825 · Learning for 3D Vision
Pixel coordinates are mapped to [-1, 1] with 2x / (W - 1) - 1, which matches the align_corners=True convention used by sample_images_at_xy. Each NDC point is placed at depth 1 and unprojected with camera.unproject_points. The ray origin is the camera center and the direction is the normalized vector from the origin to the unprojected point.
64 depths are taken with torch.linspace between near and far, and each sample point is origin + z * direction.
The weights are Ti (1 - exp(-σi Δti)), where Ti is the cumulative product of exp(-σj Δtj) over the earlier samples and starts at 1. Color and depth are both the weighted sum of the per sample values. The depth map below is divided by its maximum.
Random rays are sampled with torch.randperm over all pixels and the loss is the mean squared error between the rendered and ground truth colors. After 1000 epochs:
| x | y | z | |
|---|---|---|---|
| Box center | 0.25 | 0.25 | 0.00 |
| Box side lengths | 2.01 | 1.50 | 1.50 |








Positions go through the harmonic embedding (6 frequencies, 39 dims) and a 6 layer MLP with 128 hidden units, with the embedding concatenated again at layer 3. A linear head followed by ReLU gives the density. A second linear layer gives a 128 dim feature, which goes through a 64 unit hidden layer and a sigmoid to give the color. The model here has no view dependence. It was trained with the provided nerf_lego.yaml (250 epochs, 1024 rays per step, 128 samples per ray, lr 5e-4 decayed by 0.8 every 50 epochs).
PSNR is measured over the 200 test views of the lego dataset at the training resolution. My first 400 x 400 run collapsed to zero density in the first epoch, because the ReLU density head went negative everywhere and gave no gradient. A rerun with a new random initialization trained normally and is the one shown here.
For view dependence, the ray direction goes through its own harmonic embedding (2 frequencies, 15 dims) and is concatenated to the 128 dim feature right before the color head. The density head only sees position, so the geometry still has to agree across views. Only the last two layers of the color branch can change with direction. This is turned on with view_dependence: True in nerf_materials.yaml and nerf_materials_highres.yaml.
| PSNR (dB) | Train | Val | Test |
|---|---|---|---|
| 128, view dependent | 24.13 | 23.95 | 23.51 |
| 128, position only | 23.43 | 23.31 | 22.90 |
| 400, view dependent | 24.48 | 24.32 | 23.86 |
| 400, position only | 23.63 | 23.53 | 23.20 |
With view dependence the highlights on the shiny spheres move as the camera moves. Without it, the highlights are baked into the spheres as fixed bright patches, and the model has to average the specular color over the training views. View dependence raises test PSNR by 0.61 dB at 128 x 128 and 0.66 dB at 400 x 400.
If direction enters the network early or has many frequencies, the network can explain each training image with a direction specific color instead of correct geometry, for example by painting a floater that is only visible from one training camera. It then fits the training views well and does worse on new views. I kept direction out of the density branch, gave it only 2 frequencies and let only a small color head use it, so the extra capacity is limited to smooth changes in color. The gap between train and test PSNR grows from 0.53 to 0.62 dB at 128 x 128 and from 0.43 to 0.62 dB at 400 x 400 when view dependence is added, so the network does overfit a little more to the training views, but the test PSNR still improves.
Two copies of the Q3 network are trained together, a coarse one and a fine one. For each ray, the coarse network is evaluated at 64 evenly spaced depths between 2 and 6 and its rendering weights are computed. The weights of the inner samples (plus 1e-5) are normalized into a piecewise constant PDF over the bins between sample midpoints, and N_fine new depths are drawn from it by inverting the CDF. During training these use uniform random numbers, and at test time evenly spaced ones. The weights are detached so the fine samples do not send gradient into the coarse network. The fine network is evaluated at the sorted union of the 64 coarse and N_fine fine depths, and its render is the output. The loss is the sum of the MSE of the coarse and the fine render, so the coarse network also learns to predict where the surface is. The config is nerf_lego_coarse_fine.yaml, with renderer volume_coarse_fine and implicit function nerf_coarse_fine.
| Lego, 128 x 128 | Network evaluations per ray | Train speed (it/s) | Render time per image (s) | Test PSNR (dB) |
|---|---|---|---|---|
| Single network, 128 uniform | 128 | 48.6 | 0.143 | 24.08 |
| Coarse 64 + fine 64 | 64 + 128 | 32.7 | 0.213 | 23.57 |
| Coarse 64 + fine 128 | 64 + 192 | 26.2 | 0.284 | 24.45 |
The fine network is evaluated at both the coarse and the fine depths, so it sees 64 + N_fine points per ray. All runs use 250 epochs with the same learning rate schedule on a V100. The single network row uses its checkpoint at epoch 200, the last one saved for that run.
With 64 + 128 samples, coarse and fine sampling gives the best test PSNR, 0.37 dB above the single network. The fine samples cluster where the coarse weights are large, near the surface, instead of being spread evenly over [2, 6], where most samples fall in empty space. The cost is speed: two networks are evaluated per ray, so training runs at 26.2 it/s instead of 48.6 (1.85 times slower) and rendering takes 0.284 s per image instead of 0.143 (about 2 times slower). It also doubles the parameters.
With only 64 fine samples, the run is still 1.5 times slower to train than the single network and scores 0.51 dB lower. In that setting the fine network sees 128 points per ray, the same count as the single network, but only half of them are evenly spaced, and in 250 epochs this did not beat plain uniform sampling. In my runs the extra quality needed the larger fine budget, at roughly twice the compute.
The distance network uses the same harmonic embedding and a 6 layer, 128 unit MLP followed by a linear layer with no activation, since a signed distance can be negative. The color head is a 2 layer MLP on the 128 dim distance feature, followed by a sigmoid. The eikonal loss is mean((||∇d|| - 1)2) on random points in [-4, 4]3. The density is α Ψβ(-d), where Ψβ is the CDF of a zero mean Laplace distribution with scale β.
alpha is the density deep inside the object, where Ψ goes to 1. At the surface the density is alpha / 2, and it falls off to 0 outside. beta sets how wide this transition is: the density goes from alpha to 0 over a band of a few beta around the zero level set. As beta goes to 0 the density becomes a step function at the surface, and a large beta gives a soft, foggy shell.
With a high beta, points well outside the surface still have noticeable density, so the rendering can match the images with a blurry shell and has no reason to make the zero level set sharp or exact. The learned surface comes out smooth and inflated and thin parts are lost. With beta 0.5 the mesh is a thick slab with floaters and the color render is blurred (test PSNR 18.75 dB). A low beta pushes the density toward an indicator of the inside, so the rendering weight peaks right at the zero crossing and the SDF is pulled toward a sharp surface at the right place.
High beta. The density, and so the gradient of the loss, is non zero over a wide band around the current surface, so samples far from it still get a signal, and the gradients are smoother. With a low beta only samples within a tiny distance of the zero level set get any gradient, so a poorly initialized SDF gets almost no signal. My beta 0.02 run never formed a surface: marching cubes found no zero crossing at any of the 24 render checkpoints, and test PSNR was 13.18 dB.
Low beta, as long as training converges. With a small beta the color comes from a thin layer at the zero level set, so the SDF has to put its zero crossing exactly where the images show the surface. With a high beta the color is blended over a thick shell, so many SDFs give similar images and the surface is not well constrained.
| beta | 0.02 | 0.05 | 0.1 | 0.5 |
|---|---|---|---|---|
| Test PSNR (dB) | 13.18 | 22.05 | 21.24 | 18.75 |



Best settings: alpha 10, beta 0.05, eikonal weight 0.02, 250 epochs, lr 5e-4. beta 0.05 is the smallest value in my sweep that still trains from a random initialization. 0.02 fails to find the surface, and 0.1 and 0.5 give blobbier geometry and lower PSNR. The eikonal weight of 0.02 keeps the field close to a true distance function, so the beta band has the same width everywhere, while the photometric loss still dominates.
Isotropic covariances are s2 I. Anisotropic ones are R S ST RT, with R from quaternion_to_matrix. The 2D covariance is J W Σ WT JT, where W is the camera rotation camera.R and J comes from _compute_jacobian. Gaussians are sorted by view space depth and the ones with negative depth are dropped. Transmittance is a cumulative product of 1 - alpha, and depth and mask use the same blending as color, with the depth value or 1 in place of the color. All 4 unit tests pass.
The loss is L1 between the rendered and ground truth image, with Adam and isotropic Gaussians initialized from the provided point cloud. Trained for 1000 iterations.
| Parameter | Opacities | Scales | Colours | Means |
|---|---|---|---|---|
| Learning rate | 0.05 | 0.005 | 0.02 | 0.0005 |
Mean over the held out views: PSNR 29.22, SSIM 0.936
The colour is evaluated from all 16 SH coefficients per channel (degree 3) along the normalized direction from the camera center to each Gaussian, then shifted by 0.5 and clamped at 0, as in the original implementation.
In each pair the left image is DC only and the right image uses the full SH.
In frame 3 the seat fabric gets darker and more saturated with SH where it faces away from the camera, while the DC render is one flat bright green. The gold carvings on the frame get brighter highlights and darker shadows, so they look metallic instead of matte brown.
In frame 10 the seat cushion is seen at a grazing angle and shows a much stronger light and dark pattern in the brocade with SH. The DC render washes this out to an average green. The white wood of the arms also gets a shaded side.
In frame 24 the beige underside of the seat looks almost the same in both, since it is a diffuse surface whose colour hardly changes with direction. The streaks around the legs are the same in both renders, because SH only changes the colour of each Gaussian and not its shape. The green back is slightly darker with SH.
The Q6 setup on the materials scene: 10000 isotropic Gaussians from init_type="random", the same learning rates, L1 loss and 1000 iterations.
The Gaussians are anisotropic, with the quaternions trained at lr 0.001. The random means are spread 3 times wider (std 0.6 instead of 0.2), because the materials spheres lie well outside the default ball of radius about 0.2 around the origin. The initial opacity is sigmoid(-2) = 0.12 instead of sigmoid(8) = 0.9997, so the random cloud does not start as an opaque blob that hides the Gaussians behind it.
The loss is 0.8 L1 + 0.2 (1 - SSIM), with an 11 x 11 Gaussian window SSIM, as in the paper. The means lr starts at 0.002 and decays exponentially to 0.00002 over training. The colours lr is 0.01 and the other rates are the same as in Q6. Training runs for 4000 iterations instead of 1000.
A simplified density control runs every 500 iterations up to iteration 3000. Gaussians with opacity below 0.01 are removed, and the 10% with the largest accumulated gradient on their means are cloned. Each clone is offset by a sample from its Gaussian, and both copies have their scale divided by 1.6, up to 16000 Gaussians.
| Iterations | PSNR | SSIM | |
|---|---|---|---|
| Baseline (Q6 setup) | 1000 | 18.43 | 0.679 |
| Improved without density control | 4000 | 23.52 | 0.867 |
| Improved (all changes) | 4000 | 24.12 | 0.886 |
The baseline stays a cloud of blurry blobs around the origin. Its random means start in a ball of std 0.2, while the spheres spread well beyond that ball, and the means lr of 0.0005 is too small to move them that far in 1000 steps. The fully opaque initial Gaussians also block the ones behind them, so most Gaussians never get a useful gradient. The improved version recovers all spheres with their colours and the reflections on the metallic ones, raising PSNR from 18.43 to 24.12 and SSIM from 0.679 to 0.886.
Most of the gain comes from the wider initialization, low initial opacity, higher decaying means lr, anisotropy, SSIM loss and longer training. These alone reach 23.52 PSNR. Density control adds 0.59 dB and 0.019 SSIM. Pruning removed about 1500 near transparent Gaussians at iteration 1000 (10997 before, 9500 after pruning, 10450 after cloning), and cloning grew the set to 13572 Gaussians by iteration 3000, adding Gaussians where the means gradient was largest.