Contents
A. Neural Volume Rendering
All Part A code is in nerf/. Q1 and Q2 render at 256×256, and Q3 and
Q4.3 at 128×128 (the starter's settings).
1.3 Ray Sampling
vis_grid of the pixel grid
vis_rays of the ray directionsget_pixels_from_image makes a grid of pixel coordinates in NDC (−1 to 1
across the image). get_rays_from_pixels unprojects each pixel to depth 1 and
sends a ray from the camera centre through it. I
normalise the ray directions so distances along a ray are real world distances. Both images
are pixel-identical to the TA's grid.png and rays.png.
Pixel centres I sample the centre of each pixel, x = 2(i + 0.5)/W − 1, instead of
linspace(-1, 1, W), which lands on the pixel edges. Only the pixel-centre version
matches the TA's images exactly, so I think the TA did the same.
1.4 Point Sampling
StratifiedRaysampler puts P evenly spaced depths between near and far on
every ray, turns them into 3D points o + z·d, and stores the depths
in sample_lengths. The TODO asks for uniform samples, so there's no random jitter.
I only plotted every 4th ray because pytorch3d's CPU point renderer is slow. It matches
the TA's sample_points.png.
1.5 Volume Rendering
part_1.gif: colour
depth.png. Both outputs match the TA's
(the GIF to within one grey level).
2. Optimizing a Basic Implicit Volume
2.1 Random ray sampling and 2.2 Loss
get_random_pixels_from_image shuffles all the pixel coordinates each iteration and
keeps the first n_pixels, so a batch never repeats a pixel. The loss is the MSE
between the rendered and ground-truth colours.
| Quantity | Reported (2 d.p.) | Printed by train_box |
|---|---|---|
| Box center | (0.25, 0.25, 0.00) | (0.2502, 0.2506, −0.0006) |
| Box side lengths | (2.01, 1.50, 1.50) | (2.0051, 1.5035, 1.5034) |
2.3 Visualization
part_2.gif
My GIF matches the TA's. Below are the four training views before and after training.








3. Optimizing a Neural Radiance Field (NeRF)
part_3.gif, epoch 249 of 250
NeuralRadianceField)
- Positional encoding with
HarmonicEmbedding(3, 6): sines and cosines of the position at 6 frequencies, so the MLP can fit sharp edges instead of blurring them. - Trunk: the starter's
MLPWithInputSkips, 6 layers of 128 units with ReLU, with the encoding fed back in at layer 3. - Density head: Linear + ReLU, so σ ≥ 0.
- Colour head: Linear + ReLU + Linear + Sigmoid, so RGB stays in [0, 1]. No view dependence here (that's 4.1).
nerf_lego config: 128 samples per ray, 250 epochs,
Adam (lr 5e-4) and an MSE loss. The one change: I added
training.checkpoint_interval=3 training.render_interval=83 (also for VolSDF and
the NeRF extensions), so the final GIF comes from the last epoch (249) instead of epoch 240. I can't tell my render apart from
the TA's at this resolution.
Density bias starts at 0.5 At the start, the raw density is almost the same everywhere, so the ReLU is either on everywhere or off everywhere. With PyTorch's default init, many seeds started at zero density everywhere: a black render with no gradients, so nothing trains. A density bias of 0.5 starts training from a light fog that always has gradients. I kept the ReLU because the README asks for it.
4.1 View Dependence
Spirals (epoch 249) on the materials scene. The first three use
nerf_materials.yaml and only differ in the colour head. The fourth uses
nerf_materials_highres.yaml (400×400).
Implementation
implicit_function.view_dependence (off by default), feeds the encoded
viewing direction into the colour head only. The density still depends on position alone;
otherwise each training camera could fit its own geometry, which shows up as floaters in new
views.
A small direction branch A very flexible colour branch can fit the training images even with wrong geometry, by painting each view's colour onto whatever its rays hit. So I only use Ldir = 2 direction frequencies (NeRF uses 4) and one small layer. The Ldir = 6 run tests a more flexible version.
Results
My eval_heldout.py scores each final checkpoint on all 100 training and 200 test
views. The train−test gap is a rough measure of overfitting.
| Run | Colour head input | Params | Train PSNR | Test PSNR | Gap | Train step | Render / view |
|---|---|---|---|---|---|---|---|
| No view dependence | f (128) | 101,252 | 25.03 dB | 24.62 dB | 0.41 dB | 17.0 ms | 0.126 s |
| Ldir = 2 (main result) | f, γ(d) (128 + 15) | 102,212 | 25.86 dB | 25.20 dB | 0.66 dB | 17.6 ms | 0.136 s |
| Ldir = 6 | f, γ(d) (128 + 39) | 103,748 | 26.00 dB | 25.38 dB | 0.61 dB | 17.7 ms | 0.138 s |
| Ldir = 2, 400×400* | f, γ(d) (128 + 15) | 102,212 | 24.92 dB* | 24.30 dB* | 0.62 dB* | 17.5 ms | 1.326 s |
* Every fourth view (25 training, 50 test) at 400×400, so not comparable with the rows above.
View dependence vs. generalisation
The trade-off Without view dependence, a highlight moving across a sphere gets averaged into a dull glow, which is equally wrong on training and test views. With it, the colour can follow the angle, which helps both, but the network can also fit a training pixel with a colour that's only right from that one camera. With only 100 cameras, some of that doesn't carry over to new views. I think that's the wider gap, and the start of the overfitting the README warns about. My guess is that a more flexible branch or fewer views would eventually push test PSNR below the baseline, but I didn't test that. I kept Ldir = 2 as the main result: I chose it before the runs, and it gets most of the gain with the least flexibility.
400×400 It spreads the same training rays over far more pixels and is scored against sharper images, hence the footnote. Rendering costs about 10× more, in line with the extra pixels.
4.2 Coarse/Fine Sampling
Both on lego with Q3's settings, spirals from epoch 249.
Implementation
- Coarse pass (
sampler.type: stratified_jitter): 64 depths per ray, jittered within their bins during training. - Importance sampling (
sample_pdf): I treat the coarse weights as a probability distribution over the bins and draw 128 new depths from it (evenly spaced at test time, so renders don't flicker). - Fine pass: a second NeRF renders all the coarse and new depths together; its colour is the output.
- Loss = MSE(coarse) + MSE(fine), so the coarse network keeps improving.
nerf_lego_single256.yaml). I also tried coarse/fine at 32 + 64, which costs the
same as Q3.
Results
Same eval_heldout.py as in 4.1. "Coarse" is the coarse network's own render.
| Config | Queries per ray | Params | Train PSNR | Test PSNR | Coarse test PSNR | Train step | Render / view |
|---|---|---|---|---|---|---|---|
Q3 (nerf_lego, 128 fixed samples) | 128 | 101,252 | 26.65 dB | 25.99 dB | – | 17.0 ms | 0.126 s |
| Coarse/fine 32 + 64 | 32 + 96 = 128 | 202,504 | 25.80 dB | 25.24 dB | 23.05 dB | 17.8 ms | 0.126 s |
| One network, 256 stratified | 256 | 101,252 | 26.84 dB | 26.25 dB | – | 32.8 ms | 0.252 s |
| Coarse/fine 64 + 128 | 64 + 192 = 256 | 202,504 | 26.54 dB | 26.00 dB | 24.81 dB | 33.8 ms | 0.253 s |
Speed / quality trade-off
Quality Coarse/fine did not beat the single network at equal compute, and at Q3's cost, 32 + 64 came out below Q3. On lego at 128×128, more uniform samples for one network was the better use of a fixed budget. Importance sampling does help inside each model (the fine render beats its own coarse render), but spending a quarter of the queries on a coarse network whose colour gets thrown away still came out behind.
Why I think it lost (I didn't test these separately)
- Small scene, low resolution. With 256 uniform samples, neighbouring samples are already closer than a pixel's width at the object, so there's little left for importance sampling to fix. It should pay off more with bigger scenes or NeRF's 800×800 images.
- Fewer points where it counts. The fine pass composites 192 points instead of 256, and the sparse coarse samples can miss thin parts.
- The leftover error isn't sampling. All four models miss the studs and the base-plate texture, so I think the network itself is the limit.
- One unseeded run each, so part of the gap could be noise.
4.3 VolSDF
My best result uses α = 10, β = 0.02, with everything else from
volsdf_surface.yaml (eikonal weight 0.02). The geometry GIF is the marching-cubes
mesh of the surface d = 0.
Implementation
- Distance and colour (
NeuralSurface): Q3's trunk outputs a signed distance (no final activation, since it goes negative inside) and a feature, which a small MLP with a Sigmoid turns into colour. - Eikonal loss: mean (‖∇d‖ − 1)2 at random points, which pushes the gradient length toward 1, like a true SDF.
- SDF to density (VolSDF Sec. 3.1): σ(x) = α
Ψβ(−d(x)), where Ψβ is the CDF of a
zero-mean Laplace distribution with scale β:
σ = α · ½ e−d/β (outside, d > 0), σ = α (1 − ½ ed/β) (inside, d ≤ 0)In words: the density is α/2 right on the surface, climbs to α inside, and fades to 0 outside over a distance of a few β.
What α and β do
β is how thick the surface is: the density ramps from 0 to α across a band a few β wide. Small β gives a hard, sharp edge; large β a soft, fuzzy one.
Together, they set how much light a ray loses before it reaches d = 0: the bigger αβ, the further in front of the zero level set the rendered surface sits.
Questions
2. Is an SDF easier to train with low or high β? High β. The density only changes, and so only sends gradient back to d, within a few β of the surface. With a big β, many samples per ray get gradient, even far from the current surface, so the surface can move a long way. With a small β only a sample or two do, and a ray can even skip the surface between samples. That's why VolSDF learns β, starting from a large value.
3. Is an accurate surface more likely with high or low β? Low β, if training works. The density then acts like a solid with a crisp edge at d = 0, so the zero level set gets pinned to the edges in the images, and thin parts become opaque. So in practice: start with a high β and lower it.
Hyperparameter comparison
Four full runs that only differ in α or β. The "vs TA" columns compare my GIF with the TA's (PSNR), not with the ground truth. Final loss is MSE + 0.02 × eikonal.
| Setting | Geometry | Colour | Final loss | Colour vs TA | Geometry vs TA | Observations |
|---|---|---|---|---|---|---|
| default α = 10, β = 0.05 |
![]() |
![]() |
0.00502 | 33.6 dB | 22.3 dB | Cleanest colour; blobby base plate |
| α = 50 (β = 0.05) |
![]() |
![]() |
0.00369 | 31.6 dB | 21.2 dB | Lowest loss, but holes and floaters in the d = 0 mesh; rays turn opaque before d = 0 |
| β = 0.1 (α = 10) |
![]() |
![]() |
0.00571 | 32.6 dB | 22.1 dB | Blurriest colour; over-smoothed mesh |
| β = 0.02 (α = 10), chosen |
![]() |
![]() |
0.00448 | 32.1 dB | 23.9 dB | Most detailed mesh, squarest base plate; faint haze in frames 0, 1 and 19 |
β = 0.1 is the blurriest, because a lot of light is gone before rays reach d = 0 (question 1). α = 50 has the lowest image loss but the worst mesh: rays go opaque before d = 0, so density outside the surface explains the images, leaving holes and floaters. A lower image loss doesn't mean better geometry. The default has the cleanest colour but a blobby base plate.
The small-β risk from question 2 didn't hurt here. My guess (not tested) is that the surface band still spans a few samples, and the sphere init gives every ray a surface to cross from the start.
B. 3D Gaussian Splatting
All Part B code is in GS/. Every function works on all N Gaussians at once (no
Python loops over Gaussians).
5. 3D Gaussian Rasterization
5.1 Projecting 3D Gaussians to 2D
2D covariance. Perspective projection isn't linear, so EWA splatting linearises it at the Gaussian's centre: Σ' = J W Σ WT JT, where W rotates world vectors into the camera frame and J is the projection's Jacobian.
W = camera.R, not its transpose I use
W = camera.R because the unit tests expect it and it reproduces the course's
reference renders. Strictly, RT is correct, since pytorch3d uses row vectors. In the crops below, W = R leaves
thin white "hairs" on the silhouette, e.g. at the left armrest, while RT is clean. My
guess is that the wrong rotation tilts long, thin Gaussians off their surface. It doesn't matter
for Q6's round Gaussians, and in 7.2 I switch to RT.
5.2 Evaluating 2D Gaussians
The "power" is P(x) = −½ (x − μ)T Σ'−1
(x − μ), so exp(P) is 1 at the centre and fades out along the ellipse. I expand it into
dx2, dx·dy and dy2 terms to avoid a 2×2 matrix per Gaussian
per pixel. python unit_test_gaussians.py prints
[4/4] Tests Passed.
5.3 Filtering and sorting
A Gaussian's depth is the z of its mean in camera space. I sort front to back and keep only positive depths. The README says to drop depth < 0; I also drop depth = 0, since the projection divides by z.
5.4 Alphas and transmittance
αi(x) = oi exp(Pi(x)) is Gaussian i's opacity
at pixel x, fading away from its centre. Ti = Πj<i(1 −
αj) is the light left after passing every Gaussian in front of i. One
cumprod gives all the T's, and each mini-batch's last T seeds the next one, so
batching matches one big splat.
5.5 Splatting
python render.py --no_sh: colour | depth | silhouette, 32 views at 256 pxrender.py uses spherical harmonics by default after Q7.1, I added a
--no_sh flag to get this DC-only render. Frame 3 is byte-identical to the
course's part_7_1_without_sh.png. The red/orange specks in the depth panel
come from a starter quirk outside the code we edit (how transmittance is passed between
mini-batches). The reference images have them too, so I left it alone.
6. Training 3D Gaussian Representations
| Parameter | Learning rate | Why |
|---|---|---|
| Opacities (logits, init 8.0) | 0.05 | Adam moves each value by about lr per step, and a logit has to move several units to make a Gaussian transparent. Same as official 3DGS. |
| Scales (log radius) | 0.005 | 0.005 in log space is a 0.5% size change per step: enough to grow or shrink over hundreds of steps without blowing up. Same as official. |
| Colours (RGB, random init) | 0.02 | Random colours need to cross most of [0, 1] in the first ~50 of 1000 iterations (the official rate assumes 30k). |
| Means (world units) | 0.0005 | The point cloud is already a good start: at most ~0.5 units of drift over 1000 steps (the truck spans ~5). Close to the official 1.6e-4 × scene extent. |
make_trainable turns on gradients for the means, scales, colours and opacities. The
quaternions stay frozen, since rotating a sphere does nothing. setup_optimizer makes
one Adam optimiser with a learning rate per parameter type. Each iteration renders a random
training view and uses an L1 loss. I picked the rates from the reasoning in the table and the
3DGS defaults; I didn't sweep them, so this is the only set.
| Iterations | Mean PSNR (held-out views) | Mean SSIM (held-out views) | Time (A10G) |
|---|---|---|---|
| 1000 | 29.125 dB | 0.933 | 4.2 min |
part_6_training_progress.gif: top row renders, bottom row ground truth, every 20 iterations
part_6_training_final_renders.gif (160×90, shown at 3×)The Gaussians start as randomly coloured blobs, the truck is clear by iteration 100, and the
final renders look close to the ground truth. I also changed the hard-coded .cuda()
calls in train.py to .to(args.device) so --device cpu
works.
7.1 Rendering Using Spherical Harmonics
python render.py (full spherical harmonics): colour | depth | silhouette
python render.py --no_sh (DC colours only), for comparisoncalculate_gaussian_directions), and colours_from_spherical_harmonics
evaluates the basis with the standard constants in one einsum.
Clamping SH colours to [0, 1] The official 3DGS code only clamps at 0, but clamping at both ends matched the course's reference much better (otherwise the gold trim comes out too bright), so I do that. Frame 3 of the GPU render above is byte-identical to
part_7_1_with_sh.png.
Side-by-side comparisons
Each figure shows one frame without SH (left) and with SH (right), with the red box zoomed 4× below.
7.2 Training on a Harder Scene
The materials scene with random initial means, scored on the starter's 100 held-out views. I added six changes to my Q6 setup one at a time, and each one improved both PSNR and SSIM.
Best method
My best run (final_wT) gets 24.994 dB PSNR / 0.906 SSIM after
5000 iterations, using all six changes below. I report it as my best even though its last change
(W = RT) differs from my Q5 code, because RT is the correct rotation. The
W = R version is the final row in the ablation.
| Parameter | Learning rate (best run) | Q6 / baseline |
|---|---|---|
| Opacities (logits) | 0.05 | 0.05 |
| Scales (log, 3 per Gaussian) | 0.005 | 0.005 |
| Colours (RGB) | 0.02 | 0.02 |
| Means | 0.005 → 0.00005, exponential decay over the 5000 iterations | 0.0005, constant |
| Quaternions | 0.001 | frozen (isotropic) |
| Run | Iterations | Mean PSNR | Mean SSIM | Gaussians at the end | Training time (A10G) |
|---|---|---|---|---|---|
| Baseline (Q6 setup) | 1000 | 18.398 dB | 0.676 | 10,000 | 10.7 min (GPU shared) |
Best (final_wT) | 5000 | 24.994 dB | 0.906 | 16,000 | 31.4 min |
Other settings of the best run: Adam, 10,000 Gaussians at the start, all Gaussians in one splat, seed 0.
part_7_2_training_progress.gif: top row renders of four
training views, bottom row ground truth, every 100 iterations (0–4900; shrunk to
512×256 with every second frame kept)
part_7_2_training_final_renders.gif (128×128, shown
at 2×)Baseline: the Q6 setup with random initialisation
Q6's recipe unchanged (10,000 isotropic Gaussians, the Q6 learning rates above,
L1 loss, 1000 iterations), with Gaussians(init_type="random").
part_7_2_training_progress.gif, every 20 iterations
(0–980; shrunk to 512×256)Comparison and ablation
Each row adds one change to the row above.
| Run | Change | Iterations | Gaussians | PSNR | SSIM | Views improved (PSNR / SSIM) | Train time |
|---|---|---|---|---|---|---|---|
| baseline | Q6 setup, init_type="random" | 1000 | 10,000 | 18.398 dB | 0.676 | – | 10.7 min* |
| + init | random init covering the scene | 1000 | 10,000 | 21.901 dB | 0.767 | 100 / 100 | 7.2 min* |
| + long_lr | 5000 iterations, means lr 5e-3 → 5e-5 | 5000 | 10,000 | 22.768 dB | 0.830 | 100 / 100 | 39.2 min* |
| + ssim | 0.8·L1 + 0.2·(1 − SSIM) | 5000 | 10,000 | 22.949 dB | 0.848 | 76 / 100 | 39.5 min* |
| + aniso | anisotropic Gaussians | 5000 | 10,000 | 23.185 dB | 0.855 | 87 / 84 | 20.7 min |
+ densify (final) | adaptive density control, cap 16k | 5000 | 16,000 | 24.619 dB | 0.897 | 100 / 100 | 31.4 min |
+ W = RT (final_wT) | geometrically correct W in the 2D covariance | 5000 | 16,000 | 24.994 dB | 0.906 | 95 / 99 | 31.4 min |
* These runs shared the GPU with the NeRF extension jobs, so their times are inflated. The last three ran alone.
Overall, the best method beats the baseline by 6.60 dB PSNR and 0.231 SSIM. In its progress GIF, the haze clears to show every sphere by about iteration 300, and they sharpen once densification starts. The two almost-black frames come right after the opacity resets (change 5).
final (W = R) | final_wT (W = RT) | ground
truthModifications relative to Q6
GS/train_harder_scene.py; with no flags you get the Q6
baseline, and model.py is unchanged.
- A random init that covers the scene (
--init bounded), as 3DGS does for synthetic scenes: means spread uniformly over a cube around the whole scene, each Gaussian sized to just overlap its neighbours, and opacity 0.1 instead of nearly 1. Every sphere now has Gaussians nearby, and they all get gradient. By far the biggest gain. - Longer training with a decaying position learning rate
(
--num_itrs 5000 --lr_means 0.005 --lr_means_final 0.00005). The means need big moves early and fine settling later, so their rate starts high and decays 100× over training, like in 3DGS. - L1 + SSIM loss (
--lambda_ssim 0.2), as in 3DGS. SSIM rewards the sharp edges that L1 lets blur. - Anisotropic Gaussians (
--anisotropic): three scales and a learned rotation each, using Σ = R S ST RT from 5.1. Flat Gaussians cover a sphere's surface better than round ones. - Simplified adaptive density control (
--densify). The spheres need lots of small Gaussians, but most random ones start in empty space. Every 100 iterations early in training, I delete Gaussians that are nearly transparent or huge, and duplicate the ones the gradient keeps pushing around on screen (small ones get copied, big ones split in two). I also reset opacities to near zero twice so floaters fade. The count is capped at 16,000. - W = RT in the 2D covariance (
--w_transpose). In 5.1 I kept W =camera.Rto match the tests, but RT is correct. With W = R, each camera sees a differently rotated version of every Gaussian, so no single shape fits all views; I think that's why this helps.model.pystill uses W = R, so Q5–Q7.1 aren't affected.
.cuda() calls here, as in Q6.
Why at most 16,000 Gaussians My rasterizer evaluates every Gaussian at every pixel, so memory grows with Gaussians × pixels, and 16,000 was about what fit on the GPU.
Caveats The hyperparameters are 3DGS's, with its 30k-iteration schedule squeezed into 5k and a higher starting position learning rate. Each setup ran once, and I picked the best of seven runs on the same 100 views I report, which flatters it a little. The reported SSIM is skimage's default, not the one in the loss. I didn't try learned spherical harmonics, which would probably help with these glossy materials.





