Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

16-825 Learning for 3D Vision

Name: Alicia Yang  |  Andrew ID: aliciay

Contents

  1. A. Neural Volume Rendering: 1.3 · 1.4 · 1.5 · 2 · 3 · 4.1 · 4.2 · 4.3
  2. B. 3D Gaussian Splatting: 5.1–5.5 · 6 · 7.1 · 7.2

A. Neural Volume Rendering

All Part A code is in nerf/. Q1 and Q2 render at 256×256, and Q3 and Q4.3 at 128×128 (the starter's settings).

1.3 Ray Sampling

Pixel grid in NDC, visualised as colours
vis_grid of the pixel grid
Ray directions visualised as colours
vis_rays of the ray directions
get_pixels_from_image makes a grid of pixel coordinates in NDC (−1 to 1 across the image). get_rays_from_pixels unprojects each pixel to depth 1 and sends a ray from the camera centre through it. I normalise the ray directions so distances along a ray are real world distances. Both images are pixel-identical to the TA's grid.png and rays.png.

Pixel centres I sample the centre of each pixel, x = 2(i + 0.5)/W − 1, instead of linspace(-1, 1, W), which lands on the pixel edges. Only the pixel-centre version matches the TA's images exactly, so I think the TA did the same.

1.4 Point Sampling

Sample points along the rays of the first camera
Sample points of the first camera (every 4th ray in x and y, all 64 samples per ray)
StratifiedRaysampler puts P evenly spaced depths between near and far on every ray, turns them into 3D points o + z·d, and stores the depths in sample_lengths. The TODO asks for uniform samples, so there's no random jitter. I only plotted every 4th ray because pytorch3d's CPU point renderer is slow. It matches the TA's sample_points.png.

1.5 Volume Rendering

Spiral rendering of the SDF box
part_1.gif: colour
Depth map of the box, normalised by its maximum
Depth of camera 2, divided by its maximum
For one ray with samples at distances t1 < … < tP (Δi = ti+1 − ti; the starter sets the last one to 1e10):
Ti = exp(−Σj<i σjΔj),    wi = Ti (1 − exp(−σiΔi)),    C = Σi wi ci,    D = Σi wi ti
In words: Ti is how much light reaches sample i without being absorbed (T1 = 1), and wi is the chance that the ray stops there. The colour is the weighted sum of the sample colours, and the depth is the expected stopping distance. I don't divide the depth by Σw, so rays that miss the box get depth ≈ 0 and look dark, like in the TA's depth.png. Both outputs match the TA's (the GIF to within one grey level).

2. Optimizing a Basic Implicit Volume

2.1 Random ray sampling and 2.2 Loss

get_random_pixels_from_image shuffles all the pixel coordinates each iteration and keeps the first n_pixels, so a batch never repeats a pixel. The loss is the MSE between the rendered and ground-truth colours.

QuantityReported (2 d.p.)Printed by train_box
Box center(0.25, 0.25, 0.00)(0.2502, 0.2506, −0.0006)
Box side lengths(2.01, 1.50, 1.50)(2.0051, 1.5035, 1.5034)
The x side, 2.0051, is right on the rounding boundary, so it rounds up to 2.01. The true box is most likely 2.00 × 1.50 × 1.50, centred at (0.25, 0.25, 0).

2.3 Visualization

Spiral rendering of the optimised box
Mine: part_2.gif
TA reference spiral rendering of the box
TA reference

My GIF matches the TA's. Below are the four training views before and after training.

View 0 before training
View 0, before
View 1 before training
View 1, before
View 2 before training
View 2, before
View 3 before training
View 3, before
View 0 after training
View 0, after
View 1 after training
View 1, after
View 2 after training
View 2, after
View 3 after training
View 3, after

3. Optimizing a Neural Radiance Field (NeRF)

Spiral rendering of my NeRF on the lego scene
Mine: part_3.gif, epoch 249 of 250
TA reference NeRF rendering
TA reference
Network (NeuralRadianceField)
  1. Positional encoding with HarmonicEmbedding(3, 6): sines and cosines of the position at 6 frequencies, so the MLP can fit sharp edges instead of blurring them.
  2. Trunk: the starter's MLPWithInputSkips, 6 layers of 128 units with ReLU, with the encoding fed back in at layer 3.
  3. Density head: Linear + ReLU, so σ ≥ 0.
  4. Colour head: Linear + ReLU + Linear + Sigmoid, so RGB stays in [0, 1]. No view dependence here (that's 4.1).
I trained with the starter's nerf_lego config: 128 samples per ray, 250 epochs, Adam (lr 5e-4) and an MSE loss. The one change: I added training.checkpoint_interval=3 training.render_interval=83 (also for VolSDF and the NeRF extensions), so the final GIF comes from the last epoch (249) instead of epoch 240. I can't tell my render apart from the TA's at this resolution.

Density bias starts at 0.5 At the start, the raw density is almost the same everywhere, so the ReLU is either on everywhere or off everywhere. With PyTorch's default init, many seeds started at zero density everywhere: a black render with no gradients, so nothing trains. A density bias of 0.5 starts training from a light fog that always has gradients. I kept the ReLU because the README asks for it.

4.1 View Dependence

Materials scene, NeRF without view dependence
No view dependence (baseline)
Materials scene, view-dependent NeRF with L_dir = 2
View dependence, Ldir = 2 (main result)
Materials scene, view-dependent NeRF with L_dir = 6
More view dependence, Ldir = 6
Materials scene at 400 by 400, view-dependent NeRF with L_dir = 2
Ldir = 2 at 400×400

Spirals (epoch 249) on the materials scene. The first three use nerf_materials.yaml and only differ in the colour head. The fourth uses nerf_materials_highres.yaml (400×400).

Implementation

A new flag, implicit_function.view_dependence (off by default), feeds the encoded viewing direction into the colour head only. The density still depends on position alone; otherwise each training camera could fit its own geometry, which shows up as floaters in new views.

A small direction branch A very flexible colour branch can fit the training images even with wrong geometry, by painting each view's colour onto whatever its rays hit. So I only use Ldir = 2 direction frequencies (NeRF uses 4) and one small layer. The Ldir = 6 run tests a more flexible version.

Results

My eval_heldout.py scores each final checkpoint on all 100 training and 200 test views. The train−test gap is a rough measure of overfitting.

RunColour head inputParamsTrain PSNRTest PSNRGapTrain stepRender / view
No view dependencef (128)101,25225.03 dB24.62 dB0.41 dB17.0 ms0.126 s
Ldir = 2 (main result)f, γ(d) (128 + 15)102,21225.86 dB25.20 dB0.66 dB17.6 ms0.136 s
Ldir = 6f, γ(d) (128 + 39)103,74826.00 dB25.38 dB0.61 dB17.7 ms0.138 s
Ldir = 2, 400×400*f, γ(d) (128 + 15)102,21224.92 dB*24.30 dB*0.62 dB*17.5 ms1.326 s

* Every fourth view (25 training, 50 test) at 400×400, so not comparable with the rows above.

Held-out test view 266: ground truth and the three models, with error maps
Held-out test view 266 (2×): ground truth | no view dependence | Ldir = 2 | Ldir = 6; bottom row: 4× |error|. PSNR 25.02 / 25.26 / 25.75 dB.

View dependence vs. generalisation

What happened View dependence helped: test PSNR went up with Ldir = 2 and a bit more with Ldir = 6, at almost no extra cost. But the train−test gap got wider, because the training views improved more. Visually it's subtle at 128 px: all three models blur the glossy spheres.

The trade-off Without view dependence, a highlight moving across a sphere gets averaged into a dull glow, which is equally wrong on training and test views. With it, the colour can follow the angle, which helps both, but the network can also fit a training pixel with a colour that's only right from that one camera. With only 100 cameras, some of that doesn't carry over to new views. I think that's the wider gap, and the start of the overfitting the README warns about. My guess is that a more flexible branch or fewer views would eventually push test PSNR below the baseline, but I didn't test that. I kept Ldir = 2 as the main result: I chose it before the runs, and it gets most of the gain with the least flexibility.

400×400 It spreads the same training rays over far more pixels and is scored against sharper images, hence the footnote. Rendering costs about 10× more, in line with the extra pixels.

4.2 Coarse/Fine Sampling

Lego, coarse/fine NeRF with 64 coarse and 128 fine samples
Coarse/fine, 64 + 128 samples (fine render)
Lego, single NeRF with 256 stratified samples
One network, 256 stratified samples (same compute)

Both on lego with Q3's settings, spirals from epoch 249.

Implementation

Coarse/fine uses two networks. The coarse one renders each ray with a few evenly spread samples, which roughly shows where the surface is. Then extra samples go where the coarse pass found stuff, and the fine network renders the ray with all of them.
  1. Coarse pass (sampler.type: stratified_jitter): 64 depths per ray, jittered within their bins during training.
  2. Importance sampling (sample_pdf): I treat the coarse weights as a probability distribution over the bins and draw 128 new depths from it (evenly spaced at test time, so renders don't flicker).
  3. Fine pass: a second NeRF renders all the coarse and new depths together; its colour is the output.
  4. Loss = MSE(coarse) + MSE(fine), so the coarse network keeps improving.
What to compare against 64 + 128 is the NeRF paper's setting and costs 256 network queries per ray, twice Q3's 128. So my fair baseline is one network with 256 jittered samples (nerf_lego_single256.yaml). I also tried coarse/fine at 32 + 64, which costs the same as Q3.

Results

Same eval_heldout.py as in 4.1. "Coarse" is the coarse network's own render.

ConfigQueries per rayParamsTrain PSNRTest PSNRCoarse test PSNRTrain stepRender / view
Q3 (nerf_lego, 128 fixed samples)128101,25226.65 dB25.99 dB–17.0 ms0.126 s
Coarse/fine 32 + 6432 + 96 = 128202,50425.80 dB25.24 dB23.05 dB17.8 ms0.126 s
One network, 256 stratified256101,25226.84 dB26.25 dB–32.8 ms0.252 s
Coarse/fine 64 + 12864 + 192 = 256202,50426.54 dB26.00 dB24.81 dB33.8 ms0.253 s
Held-out test view 399: ground truth and the four models, with error maps
Held-out test view 399 (2×): ground truth | Q3 | one network, 256 | coarse/fine 64 + 128 | coarse/fine 32 + 64; bottom row: 4× |error|. PSNR 26.25 / 26.34 / 26.05 / 25.47 dB.

Speed / quality trade-off

Speed Coarse/fine doesn't make a ray cheaper; the cost follows the number of network queries. At 256 queries per ray both methods take about the same time, roughly twice Q3, and coarse/fine has twice the parameters.

Quality Coarse/fine did not beat the single network at equal compute, and at Q3's cost, 32 + 64 came out below Q3. On lego at 128×128, more uniform samples for one network was the better use of a fixed budget. Importance sampling does help inside each model (the fine render beats its own coarse render), but spending a quarter of the queries on a coarse network whose colour gets thrown away still came out behind.

Why I think it lost (I didn't test these separately)
  • Small scene, low resolution. With 256 uniform samples, neighbouring samples are already closer than a pixel's width at the object, so there's little left for importance sampling to fix. It should pay off more with bigger scenes or NeRF's 800×800 images.
  • Fewer points where it counts. The fine pass composites 192 points instead of 256, and the sparse coarse samples can miss thin parts.
  • The leftover error isn't sampling. All four models miss the studs and the base-plate texture, so I think the network itself is the limit.
  • One unseeded run each, so part of the gap could be noise.

4.3 VolSDF

My VolSDF geometry, beta 0.02
Mine (β = 0.02): geometry
My VolSDF colour rendering, beta 0.02
Mine (β = 0.02): colour
TA reference VolSDF geometry
TA reference: geometry
TA reference VolSDF colour rendering
TA reference: colour

My best result uses α = 10, β = 0.02, with everything else from volsdf_surface.yaml (eikonal weight 0.02). The geometry GIF is the marching-cubes mesh of the surface d = 0.

Implementation

  • Distance and colour (NeuralSurface): Q3's trunk outputs a signed distance (no final activation, since it goes negative inside) and a feature, which a small MLP with a Sigmoid turns into colour.
  • Eikonal loss: mean (‖∇d‖ − 1)2 at random points, which pushes the gradient length toward 1, like a true SDF.
  • SDF to density (VolSDF Sec. 3.1): σ(x) = α Ψβ(−d(x)), where Ψβ is the CDF of a zero-mean Laplace distribution with scale β:
    σ = α · ½ e−d/β (outside, d > 0),     σ = α (1 − ½ ed/β) (inside, d ≤ 0)
    In words: the density is α/2 right on the surface, climbs to α inside, and fades to 0 outside over a distance of a few β.
Starting from a sphere With PyTorch's default init, the untrained distance is almost constant, so there's no surface at all. I use the SAL/IDR geometric init (Atzmon & Lipman 2020) instead, which starts the SDF as a rough unit sphere. In a short same-seed test, the default init got stuck with the SDF's sign flipped, while the sphere init already showed a bulldozer. That was one short run, so I don't know whether the default would have recovered.

What α and β do

α is how dense the solid is: it's the density deep inside the object. Bigger α means more opaque.

β is how thick the surface is: the density ramps from 0 to α across a band a few β wide. Small β gives a hard, sharp edge; large β a soft, fuzzy one.

Together, they set how much light a ray loses before it reaches d = 0: the bigger αβ, the further in front of the zero level set the rendered surface sits.

Questions

1. How does high β bias the learned SDF? Low β? High β spreads density out in front of the surface, so the rendered surface sits in front of d = 0. To match the images, the optimiser pulls the zero level set inward and smooths it, so the shape comes out shrunken and over-smoothed, and thin parts can vanish. Low β makes the density nearly a step at d = 0, so there's little bias, but the SDF only gets a learning signal right next to the surface.

2. Is an SDF easier to train with low or high β? High β. The density only changes, and so only sends gradient back to d, within a few β of the surface. With a big β, many samples per ray get gradient, even far from the current surface, so the surface can move a long way. With a small β only a sample or two do, and a ray can even skip the surface between samples. That's why VolSDF learns β, starting from a large value.

3. Is an accurate surface more likely with high or low β? Low β, if training works. The density then acts like a solid with a crisp edge at d = 0, so the zero level set gets pinned to the edges in the images, and thin parts become opaque. So in practice: start with a high β and lower it.

Hyperparameter comparison

Four full runs that only differ in α or β. The "vs TA" columns compare my GIF with the TA's (PSNR), not with the ground truth. Final loss is MSE + 0.02 × eikonal.

SettingGeometryColourFinal lossColour vs TAGeometry vs TAObservations
default
α = 10, β = 0.05
Geometry, default setting Colour, default setting 0.0050233.6 dB22.3 dB Cleanest colour; blobby base plate
α = 50
(β = 0.05)
Geometry, alpha 50 Colour, alpha 50 0.0036931.6 dB21.2 dB Lowest loss, but holes and floaters in the d = 0 mesh; rays turn opaque before d = 0
β = 0.1
(α = 10)
Geometry, beta 0.1 Colour, beta 0.1 0.0057132.6 dB22.1 dB Blurriest colour; over-smoothed mesh
β = 0.02
(α = 10), chosen
Geometry, beta 0.02 Colour, beta 0.02 0.0044832.1 dB23.9 dB Most detailed mesh, squarest base plate; faint haze in frames 0, 1 and 19
Why β = 0.02 works best here β = 0.02 gave the most detailed mesh and the best geometry match with the TA. Smaller β means a thinner surface, so the rendered surface lines up with d = 0 and the shape gets pinned down better (question 3). The price is slightly worse colour and a faint haze in a few frames, but I care most about the surface here.

β = 0.1 is the blurriest, because a lot of light is gone before rays reach d = 0 (question 1). α = 50 has the lowest image loss but the worst mesh: rays go opaque before d = 0, so density outside the surface explains the images, leaving holes and floaters. A lower image loss doesn't mean better geometry. The default has the cleanest colour but a blobby base plate.

The small-β risk from question 2 didn't hurt here. My guess (not tested) is that the surface band still spans a few samples, and the sphere init gives every ray a surface to cross from the start.

B. 3D Gaussian Splatting

All Part B code is in GS/. Every function works on all N Gaussians at once (no Python loops over Gaussians).

5. 3D Gaussian Rasterization

5.1 Projecting 3D Gaussians to 2D

3D covariance. A Gaussian's shape is a rotation R (from its quaternion) and three axis scales S:
Σ = R S ST RT = (RS)(RS)T,
which is always a valid covariance, so 3DGS learns rotations and scales instead of Σ. Isotropic Gaussians just use Σ = s2I.

2D covariance. Perspective projection isn't linear, so EWA splatting linearises it at the Gaussian's centre: Σ' = J W Σ WT JT, where W rotates world vectors into the camera frame and J is the projection's Jacobian.

W = camera.R, not its transpose I use W = camera.R because the unit tests expect it and it reproduces the course's reference renders. Strictly, RT is correct, since pytorch3d uses row vectors. In the crops below, W = R leaves thin white "hairs" on the silhouette, e.g. at the left armrest, while RT is clean. My guess is that the wrong rotation tilts long, thin Gaussians off their surface. It doesn't matter for Q6's round Gaussians, and in 7.2 I switch to RT.
Zoomed crops of view 3: course reference, W = R, W = R transposed
2× crops of view 3 (CPU render, DC colours): course reference | W = camera.R (my code) | W = camera.RT

5.2 Evaluating 2D Gaussians

The "power" is P(x) = −½ (x − μ)T Σ'−1 (x − μ), so exp(P) is 1 at the centre and fades out along the ellipse. I expand it into dx2, dx·dy and dy2 terms to avoid a 2×2 matrix per Gaussian per pixel. python unit_test_gaussians.py prints [4/4] Tests Passed.

5.3 Filtering and sorting

A Gaussian's depth is the z of its mean in camera space. I sort front to back and keep only positive depths. The README says to drop depth < 0; I also drop depth = 0, since the projection divides by z.

5.4 Alphas and transmittance

αi(x) = oi exp(Pi(x)) is Gaussian i's opacity at pixel x, fading away from its centre. Ti = Πj<i(1 − αj) is the light left after passing every Gaussian in front of i. One cumprod gives all the T's, and each mini-batch's last T seeds the next one, so batching matches one big splat.

5.5 Splatting

Chair rendered with DC colours: colour, depth and mask
python render.py --no_sh: colour | depth | silhouette, 32 views at 256 px
This is front-to-back alpha compositing. The weight wi = αiTi is how visible Gaussian i is at a pixel, and the same weights give all three maps:
C = Σi ci wi,    D = Σi zi wi,    M = Σi wi = 1 − Tfinal
Since render.py uses spherical harmonics by default after Q7.1, I added a --no_sh flag to get this DC-only render. Frame 3 is byte-identical to the course's part_7_1_without_sh.png. The red/orange specks in the depth panel come from a starter quirk outside the code we edit (how transmittance is passed between mini-batches). The reference images have them too, so I left it alone.

6. Training 3D Gaussian Representations

ParameterLearning rateWhy
Opacities (logits, init 8.0)0.05 Adam moves each value by about lr per step, and a logit has to move several units to make a Gaussian transparent. Same as official 3DGS.
Scales (log radius)0.005 0.005 in log space is a 0.5% size change per step: enough to grow or shrink over hundreds of steps without blowing up. Same as official.
Colours (RGB, random init)0.02 Random colours need to cross most of [0, 1] in the first ~50 of 1000 iterations (the official rate assumes 30k).
Means (world units)0.0005 The point cloud is already a good start: at most ~0.5 units of drift over 1000 steps (the truck spans ~5). Close to the official 1.6e-4 × scene extent.
make_trainable turns on gradients for the means, scales, colours and opacities. The quaternions stay frozen, since rotating a sphere does nothing. setup_optimizer makes one Adam optimiser with a learning rate per parameter type. Each iteration renders a random training view and uses an L1 loss. I picked the rates from the reasoning in the table and the 3DGS defaults; I didn't sweep them, so this is the only set.
IterationsMean PSNR (held-out views)Mean SSIM (held-out views)Time (A10G)
100029.125 dB0.9334.2 min
Training progress: renders on top, ground truth below
part_6_training_progress.gif: top row renders, bottom row ground truth, every 20 iterations
Final renders of the trained Gaussians along a camera path
part_6_training_final_renders.gif (160×90, shown at 3×)

The Gaussians start as randomly coloured blobs, the truck is clear by iteration 100, and the final renders look close to the ground truth. I also changed the hard-coded .cuda() calls in train.py to .to(args.device) so --device cpu works.

7.1 Rendering Using Spherical Harmonics

Chair rendered with spherical harmonics: colour, depth and mask
Q7.1, python render.py (full spherical harmonics): colour | depth | silhouette
Chair rendered with DC colours only: colour, depth and mask
Q5.5, python render.py --no_sh (DC colours only), for comparison
In 3DGS, a Gaussian's colour can depend on the viewing direction d. It's a sum of 16 spherical-harmonic (SH) basis functions up to degree 3, with separate weights per colour channel:
c(d) = clamp( Σl=0..3 Σm=−l..l klm Ylm(d) + 0.5, 0, 1 )
In words: degree 0 is the constant "DC" colour that Q5 used, and the higher degrees let the colour shift with the viewing angle. d is the unit vector from the camera to the Gaussian (calculate_gaussian_directions), and colours_from_spherical_harmonics evaluates the basis with the standard constants in one einsum.

Clamping SH colours to [0, 1] The official 3DGS code only clamps at 0, but clamping at both ends matched the course's reference much better (otherwise the gold trim comes out too bright), so I do that. Frame 3 of the GPU render above is byte-identical to part_7_1_with_sh.png.

Side-by-side comparisons

Each figure shows one frame without SH (left) and with SH (right), with the red box zoomed 4× below.

Frame 14 without and with spherical harmonics
Frame 14: front three-quarter view of backrest and seat
This frame changes the most. With SH, the green satin on the backrest gets about a third darker while the gold medallion stays just as bright, so the pattern stands out much more. The lighter band across the top of the DC backrest also goes away.
Frame 3 without and with spherical harmonics
Frame 3: the view of the course's reference images
Same effect, just smaller. With SH, the light lime fabric turns a deeper green while the gold sprigs barely change, so the pattern looks crisp instead of washed out.
Frame 10 without and with spherical harmonics
Frame 10: looking steeply down onto the seat
Here the seat changes the most. With SH, its green gets darker and the gold threads actually get brighter, so a pattern that's faint without SH stands out. The right side of the backrest also turns dark green while the left side stays close to the DC colour, which looks like a sheen that depends on the angle.
Why the DC-only render looks washed out Every SH basis function except the constant one averages to zero over the sphere, so the DC colour is basically the Gaussian's colour averaged over all viewing directions. Satin and gold thread look different from different angles, so DC flattens them into one lighter, lower-contrast colour. The difference is biggest from the upholstered front and smallest from the plain beige underside. My guess (not tested) is that the underside just isn't very view-dependent.

7.2 Training on a Harder Scene

The materials scene with random initial means, scored on the starter's 100 held-out views. I added six changes to my Q6 setup one at a time, and each one improved both PSNR and SSIM.

Best method

My best run (final_wT) gets 24.994 dB PSNR / 0.906 SSIM after 5000 iterations, using all six changes below. I report it as my best even though its last change (W = RT) differs from my Q5 code, because RT is the correct rotation. The W = R version is the final row in the ablation.

ParameterLearning rate (best run)Q6 / baseline
Opacities (logits)0.050.05
Scales (log, 3 per Gaussian)0.0050.005
Colours (RGB)0.020.02
Means0.005 → 0.00005, exponential decay over the 5000 iterations0.0005, constant
Quaternions0.001frozen (isotropic)
RunIterationsMean PSNRMean SSIMGaussians at the endTraining time (A10G)
Baseline (Q6 setup)100018.398 dB0.67610,00010.7 min (GPU shared)
Best (final_wT)500024.994 dB0.90616,00031.4 min

Other settings of the best run: Adam, 10,000 Gaussians at the start, all Gaussians in one splat, seed 0.

Training progress of the best method: renders on top, ground truth below
Best method, part_7_2_training_progress.gif: top row renders of four training views, bottom row ground truth, every 100 iterations (0–4900; shrunk to 512×256 with every second frame kept)
Final renders of the best method over the training views
Best method, part_7_2_training_final_renders.gif (128×128, shown at 2×)

Baseline: the Q6 setup with random initialisation

Q6's recipe unchanged (10,000 isotropic Gaussians, the Q6 learning rates above, L1 loss, 1000 iterations), with Gaussians(init_type="random").

Training progress of the baseline: renders on top, ground truth below
Baseline, part_7_2_training_progress.gif, every 20 iterations (0–980; shrunk to 512×256)
Why the baseline fails The random init is the problem. All the Gaussians start as a tiny, nearly opaque clump near the origin, but the spheres are spread much further out. With Q6's small position learning rate, they can't move far enough in 1000 iterations to reach the outer spheres, and the clump's opaque front hides the Gaussians behind it from the gradient. So the renders end up as a speckled clump plus a few faint blobs (left column below).

Comparison and ablation

Each row adds one change to the row above.

RunChangeIterationsGaussiansPSNRSSIMViews improved (PSNR / SSIM)Train time
baselineQ6 setup, init_type="random"100010,00018.398 dB0.676–10.7 min*
+ initrandom init covering the scene100010,00021.901 dB0.767100 / 1007.2 min*
+ long_lr5000 iterations, means lr 5e-3 → 5e-5500010,00022.768 dB0.830100 / 10039.2 min*
+ ssim0.8·L1 + 0.2·(1 − SSIM)500010,00022.949 dB0.84876 / 10039.5 min*
+ anisoanisotropic Gaussians500010,00023.185 dB0.85587 / 8420.7 min
+ densify (final)adaptive density control, cap 16k500016,00024.619 dB0.897100 / 10031.4 min
+ W = RT (final_wT)geometrically correct W in the 2D covariance500016,00024.994 dB0.90695 / 9931.4 min

* These runs shared the GPU with the NeRF extension jobs, so their times are inflated. The last three ran alone.

Overall, the best method beats the baseline by 6.60 dB PSNR and 0.231 SSIM. In its progress GIF, the haze clears to show every sphere by about iteration 300, and they sharpen once densification starts. The two almost-black frames come right after the opacity resets (change 5).

Held-out views 0, 3 and 4 for five runs of the ablation and the ground truth
Held-out views 0, 3 and 4 (2×): baseline | + init | + aniso (also long_lr and SSIM) | final (W = R) | final_wT (W = RT) | ground truth

Modifications relative to Q6

All changes are flags in GS/train_harder_scene.py; with no flags you get the Q6 baseline, and model.py is unchanged.
  1. A random init that covers the scene (--init bounded), as 3DGS does for synthetic scenes: means spread uniformly over a cube around the whole scene, each Gaussian sized to just overlap its neighbours, and opacity 0.1 instead of nearly 1. Every sphere now has Gaussians nearby, and they all get gradient. By far the biggest gain.
  2. Longer training with a decaying position learning rate (--num_itrs 5000 --lr_means 0.005 --lr_means_final 0.00005). The means need big moves early and fine settling later, so their rate starts high and decays 100× over training, like in 3DGS.
  3. L1 + SSIM loss (--lambda_ssim 0.2), as in 3DGS. SSIM rewards the sharp edges that L1 lets blur.
  4. Anisotropic Gaussians (--anisotropic): three scales and a learned rotation each, using Σ = R S ST RT from 5.1. Flat Gaussians cover a sphere's surface better than round ones.
  5. Simplified adaptive density control (--densify). The spheres need lots of small Gaussians, but most random ones start in empty space. Every 100 iterations early in training, I delete Gaussians that are nearly transparent or huge, and duplicate the ones the gradient keeps pushing around on screen (small ones get copied, big ones split in two). I also reset opacities to near zero twice so floaters fade. The count is capped at 16,000.
  6. W = RT in the 2D covariance (--w_transpose). In 5.1 I kept W = camera.R to match the tests, but RT is correct. With W = R, each camera sees a differently rotated version of every Gaussian, so no single shape fits all views; I think that's why this helps. model.py still uses W = R, so Q5–Q7.1 aren't affected.
I also replaced the hard-coded .cuda() calls here, as in Q6.

Why at most 16,000 Gaussians My rasterizer evaluates every Gaussian at every pixel, so memory grows with Gaussians × pixels, and 16,000 was about what fit on the GPU.

Caveats The hyperparameters are 3DGS's, with its 30k-iteration schedule squeezed into 5k and a higher starting position learning rate. Each setup ran once, and I picked the best of seven runs on the same 100 views I report, which flatters it a little. The reported SSIM is skimage's default, not the one in the loss. I didn't try learned spherical harmonics, which would probably help with these glossy materials.