16-825 · Learning for 3D Vision

Assignment 3: Neural Volume Rendering and Gaussian Splatting

Question 1.3

Ray sampling

Pixel coordinates are mapped to [-1, 1] with 2x / (W - 1) - 1, which matches the align_corners=True convention used by sample_images_at_xy. Each NDC point is placed at depth 1 and unprojected with camera.unproject_points. The ray origin is the camera center and the direction is the normalized vector from the origin to the unprojected point.

xy grid
xy_grid from vis_grid
Ray directions
Ray directions from vis_rays

Question 1.4

Point sampling

64 depths are taken with torch.linspace between near and far, and each sample point is origin + z * direction.

Sample points
Sample points along all rays of the first camera

Question 1.5

Volume rendering

The weights are Ti (1 - exp(-σi Δti)), where Ti is the cumulative product of exp(-σj Δtj) over the earlier samples and starts at 1. Color and depth are both the weighted sum of the per sample values. The depth map below is divided by its maximum.

Spiral rendering of the box
Spiral rendering of the box
Depth of the box
Depth of the box from the third camera

Question 2

Optimizing a basic implicit volume

Random rays are sampled with torch.randperm over all pixels and the loss is the mean squared error between the rendered and ground truth colors. After 1000 epochs:

xyz
Box center0.250.250.00
Box side lengths2.011.501.50
Before training, view 0
Before, view 0
Before training, view 1
Before, view 1
Before training, view 2
Before, view 2
Before training, view 3
Before, view 3
After training, view 0
After, view 0
After training, view 1
After, view 1
After training, view 2
After, view 2
After training, view 3
After, view 3
Spiral rendering of the optimized box
Spiral rendering of the optimized box

Question 3

Neural radiance field

Positions go through the harmonic embedding (6 frequencies, 39 dims) and a 6 layer MLP with 128 hidden units, with the embedding concatenated again at layer 3. A linear head followed by ReLU gives the density. A second linear layer gives a 128 dim feature, which goes through a 64 unit hidden layer and a sigmoid to give the color. The model here has no view dependence. It was trained with the provided nerf_lego.yaml (250 epochs, 1024 rays per step, 128 samples per ray, lr 5e-4 decayed by 0.8 every 50 epochs).

Lego at 128 x 128
128 x 128, test PSNR 24.08 dB
Lego at 400 x 400
400 x 400 (nerf_lego_highres.yaml), test PSNR 24.65 dB

PSNR is measured over the 200 test views of the lego dataset at the training resolution. My first 400 x 400 run collapsed to zero density in the first epoch, because the ReLU density head went negative everywhere and gave no gradient. A rerun with a new random initialization trained normally and is the one shown here.

Question 4.1

View dependence

For view dependence, the ray direction goes through its own harmonic embedding (2 frequencies, 15 dims) and is concatenated to the 128 dim feature right before the color head. The density head only sees position, so the geometry still has to agree across views. Only the last two layers of the color branch can change with direction. This is turned on with view_dependence: True in nerf_materials.yaml and nerf_materials_highres.yaml.

Materials 400 with view dependence
400 x 400, with view dependence
Materials 400 without view dependence
400 x 400, without view dependence
Materials 128 with view dependence
128 x 128, with view dependence
Materials 128 without view dependence
128 x 128, without view dependence
PSNR (dB)TrainValTest
128, view dependent24.1323.9523.51
128, position only23.4323.3122.90
400, view dependent24.4824.3223.86
400, position only23.6323.5323.20

With view dependence the highlights on the shiny spheres move as the camera moves. Without it, the highlights are baked into the spheres as fixed bright patches, and the model has to average the specular color over the training views. View dependence raises test PSNR by 0.61 dB at 128 x 128 and 0.66 dB at 400 x 400.

Trade off with generalization

If direction enters the network early or has many frequencies, the network can explain each training image with a direction specific color instead of correct geometry, for example by painting a floater that is only visible from one training camera. It then fits the training views well and does worse on new views. I kept direction out of the density branch, gave it only 2 frequencies and let only a small color head use it, so the extra capacity is limited to smooth changes in color. The gap between train and test PSNR grows from 0.53 to 0.62 dB at 128 x 128 and from 0.43 to 0.62 dB at 400 x 400 when view dependence is added, so the network does overfit a little more to the training views, but the test PSNR still improves.

Question 4.2

Coarse and fine sampling

Two copies of the Q3 network are trained together, a coarse one and a fine one. For each ray, the coarse network is evaluated at 64 evenly spaced depths between 2 and 6 and its rendering weights are computed. The weights of the inner samples (plus 1e-5) are normalized into a piecewise constant PDF over the bins between sample midpoints, and N_fine new depths are drawn from it by inverting the CDF. During training these use uniform random numbers, and at test time evenly spaced ones. The weights are detached so the fine samples do not send gradient into the coarse network. The fine network is evaluated at the sorted union of the 64 coarse and N_fine fine depths, and its render is the output. The loss is the sum of the MSE of the coarse and the fine render, so the coarse network also learns to predict where the surface is. The config is nerf_lego_coarse_fine.yaml, with renderer volume_coarse_fine and implicit function nerf_coarse_fine.

Coarse 64 and fine 128
Coarse 64 + fine 128, test PSNR 24.45 dB
Single network
Single network, 128 uniform samples (Q3), test PSNR 24.08 dB
Lego, 128 x 128Network evaluations per rayTrain speed (it/s)Render time per image (s)Test PSNR (dB)
Single network, 128 uniform12848.60.14324.08
Coarse 64 + fine 6464 + 12832.70.21323.57
Coarse 64 + fine 12864 + 19226.20.28424.45

The fine network is evaluated at both the coarse and the fine depths, so it sees 64 + N_fine points per ray. All runs use 250 epochs with the same learning rate schedule on a V100. The single network row uses its checkpoint at epoch 200, the last one saved for that run.

Speed and quality trade offs

With 64 + 128 samples, coarse and fine sampling gives the best test PSNR, 0.37 dB above the single network. The fine samples cluster where the coarse weights are large, near the surface, instead of being spread evenly over [2, 6], where most samples fall in empty space. The cost is speed: two networks are evaluated per ray, so training runs at 26.2 it/s instead of 48.6 (1.85 times slower) and rendering takes 0.284 s per image instead of 0.143 (about 2 times slower). It also doubles the parameters.

With only 64 fine samples, the run is still 1.5 times slower to train than the single network and scores 0.51 dB lower. In that setting the fine network sees 128 points per ray, the same count as the single network, but only half of them are evenly spaced, and in 250 epochs this did not beat plain uniform sampling. In my runs the extra quality needed the larger fine budget, at roughly twice the compute.

Question 4.3

VolSDF

The distance network uses the same harmonic embedding and a 6 layer, 128 unit MLP followed by a linear layer with no activation, since a signed distance can be negative. The color head is a 2 layer MLP on the 128 dim distance feature, followed by a sigmoid. The eikonal loss is mean((||∇d|| - 1)2) on random points in [-4, 4]3. The density is α Ψβ(-d), where Ψβ is the CDF of a zero mean Laplace distribution with scale β.

VolSDF geometry
Geometry, alpha 10, beta 0.05
VolSDF color
Color, alpha 10, beta 0.05, test PSNR 22.05 dB

What alpha and beta do

alpha is the density deep inside the object, where Ψ goes to 1. At the surface the density is alpha / 2, and it falls off to 0 outside. beta sets how wide this transition is: the density goes from alpha to 0 over a band of a few beta around the zero level set. As beta goes to 0 the density becomes a step function at the surface, and a large beta gives a soft, foggy shell.

How does high beta bias the SDF, and low beta?

With a high beta, points well outside the surface still have noticeable density, so the rendering can match the images with a blurry shell and has no reason to make the zero level set sharp or exact. The learned surface comes out smooth and inflated and thin parts are lost. With beta 0.5 the mesh is a thick slab with floaters and the color render is blurred (test PSNR 18.75 dB). A low beta pushes the density toward an indicator of the inside, so the rendering weight peaks right at the zero crossing and the SDF is pulled toward a sharp surface at the right place.

Is it easier to train with low or high beta?

High beta. The density, and so the gradient of the loss, is non zero over a wide band around the current surface, so samples far from it still get a signal, and the gradients are smoother. With a low beta only samples within a tiny distance of the zero level set get any gradient, so a poorly initialized SDF gets almost no signal. My beta 0.02 run never formed a surface: marching cubes found no zero crossing at any of the 24 render checkpoints, and test PSNR was 13.18 dB.

Is the surface more accurate with high or low beta?

Low beta, as long as training converges. With a small beta the color comes from a thin layer at the zero level set, so the SDF has to put its zero crossing exactly where the images show the surface. With a high beta the color is blended over a thick shell, so many SDFs give similar images and the surface is not well constrained.

Effect of beta

beta0.020.050.10.5
Test PSNR (dB)13.1822.0521.2418.75
beta 0.1 geometry
beta 0.1 geometry
beta 0.5 geometry
beta 0.5 geometry
beta 0.02 color
beta 0.02 color (no surface found)

Best settings: alpha 10, beta 0.05, eikonal weight 0.02, 250 epochs, lr 5e-4. beta 0.05 is the smallest value in my sweep that still trains from a random initialization. 0.02 fails to find the surface, and 0.1 and 0.5 give blobbier geometry and lower PSNR. The eikonal weight of 0.02 keeps the field close to a true distance function, so the beta band has the same width everywhere, while the photometric loss still dominates.

Question 5

3D Gaussian rasterization

Isotropic covariances are s2 I. Anisotropic ones are R S ST RT, with R from quaternion_to_matrix. The 2D covariance is J W Σ WT JT, where W is the camera rotation camera.R and J comes from _compute_jacobian. Gaussians are sorted by view space depth and the ones with negative depth are dropped. Transmittance is a cumulative product of 1 - alpha, and depth and mask use the same blending as color, with the depth value or 1 in place of the color. All 4 unit tests pass.

Pre-trained chair render
Pre-trained chair rendered with the DC color only: RGB, depth and silhouette

Question 6

Training 3D Gaussians

The loss is L1 between the rendered and ground truth image, with Adam and isotropic Gaussians initialized from the provided point cloud. Trained for 1000 iterations.

ParameterOpacitiesScalesColoursMeans
Learning rate0.050.0050.020.0005

Mean over the held out views: PSNR 29.22, SSIM 0.936

Training progress
Training progress, top: renders, bottom: ground truth
Final renders
Final renders of the training views

Question 7.1

Rendering with spherical harmonics

The colour is evaluated from all 16 SH coefficients per channel (degree 3) along the normalized direction from the camera center to each Gaussian, then shifted by 0.5 and clamped at 0, as in the original implementation.

Render with spherical harmonics
With spherical harmonics (Q7.1)
Render with DC color only
DC only (Q5)

Side by side comparison

In each pair the left image is DC only and the right image uses the full SH.

Frame 3, DC only and full SH
Frame 3, left: DC only, right: full SH
Frame 10, DC only and full SH
Frame 10, left: DC only, right: full SH
Frame 24, DC only and full SH
Frame 24, left: DC only, right: full SH

In frame 3 the seat fabric gets darker and more saturated with SH where it faces away from the camera, while the DC render is one flat bright green. The gold carvings on the frame get brighter highlights and darker shadows, so they look metallic instead of matte brown.

In frame 10 the seat cushion is seen at a grazing angle and shows a much stronger light and dark pattern in the brocade with SH. The DC render washes this out to an average green. The white wood of the arms also gets a shaded side.

In frame 24 the beige underside of the seat looks almost the same in both, since it is a diffuse surface whose colour hardly changes with direction. The streaks around the legs are the same in both renders, because SH only changes the colour of each Gaussian and not its shape. The green back is slightly darker with SH.

Question 7.2

Training on a harder scene

Baseline

The Q6 setup on the materials scene: 10000 isotropic Gaussians from init_type="random", the same learning rates, L1 loss and 1000 iterations.

Changes for the improved version

The Gaussians are anisotropic, with the quaternions trained at lr 0.001. The random means are spread 3 times wider (std 0.6 instead of 0.2), because the materials spheres lie well outside the default ball of radius about 0.2 around the origin. The initial opacity is sigmoid(-2) = 0.12 instead of sigmoid(8) = 0.9997, so the random cloud does not start as an opaque blob that hides the Gaussians behind it.

The loss is 0.8 L1 + 0.2 (1 - SSIM), with an 11 x 11 Gaussian window SSIM, as in the paper. The means lr starts at 0.002 and decays exponentially to 0.00002 over training. The colours lr is 0.01 and the other rates are the same as in Q6. Training runs for 4000 iterations instead of 1000.

A simplified density control runs every 500 iterations up to iteration 3000. Gaussians with opacity below 0.01 are removed, and the 10% with the largest accumulated gradient on their means are cloned. Each clone is offset by a sample from its Gaussian, and both copies have their scale divided by 1.6, up to 16000 Gaussians.

Results

IterationsPSNRSSIM
Baseline (Q6 setup)100018.430.679
Improved without density control400023.520.867
Improved (all changes)400024.120.886
Baseline training progress
Baseline training progress
Improved training progress
Improved training progress
Baseline final renders
Baseline final renders
Improved final renders
Improved final renders

The baseline stays a cloud of blurry blobs around the origin. Its random means start in a ball of std 0.2, while the spheres spread well beyond that ball, and the means lr of 0.0005 is too small to move them that far in 1000 steps. The fully opaque initial Gaussians also block the ones behind them, so most Gaussians never get a useful gradient. The improved version recovers all spheres with their colours and the reflections on the metallic ones, raising PSNR from 18.43 to 24.12 and SSIM from 0.679 to 0.886.

Most of the gain comes from the wider initialization, low initial opacity, higher decaying means lr, anisotropy, SSIM loss and longer training. These alone reach 23.52 PSNR. Density control adds 0.59 dB and 0.019 SSIM. Pruning removed about 1500 near transparent Gaussians at iteration 1000 (10997 before, 9500 after pruning, 10450 after cloning), and cloning grew the set to 13572 Gaussians by iteration 3000, adding Gaussians where the means gradient was largest.