Assignment 3 - Chan WangSing

Everything was run on an RTX 5090. Part A is in nerf/ and Part B in GS/.

A. Neural Volume Rendering

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel centres are mapped to NDC in [-1, 1], placed on the z = 1 plane and unprojected to world space with camera.unproject_points. Every ray starts at the camera centre and points through its unprojected pixel, normalised to unit length.

xy grid
ray directions

1.4 Point Sampling

The stratified sampler places 64 points uniformly between the near and far depth (0 and 5) on every ray, p = o + t d. The point cloud below is all sample points of the first camera's rays.

1.5 Volume Rendering

For each sample, alpha is 1 - exp(-σ Δt) and the transmittance is the cumulative product of 1 - alpha over the earlier samples (1 for the first sample). The weights are T * alpha, the colour is the weighted sum of the sample colours, and the depth is the weighted sum of the sample distances. The depth map is normalised by its maximum.

part_1.gif
depth (camera 2)

2. Optimizing a Basic Implicit Volume

2.1 Random Ray Sampling

get_random_pixels_from_image shuffles the full pixel grid with torch.randperm and keeps the first 4096 pixels, so every iteration trains on a random subset of rays without repeats.

2.2 Loss and Training

The loss is the mean squared error between the rendered colours and the ground-truth colours at the sampled pixels. After 1000 epochs:

xyz
Box center0.250.250.00
Box side lengths2.011.501.50

2.3 Visualization

3. Optimizing a Neural Radiance Field (NeRF)

Each sample point is encoded with 6 harmonic frequencies (39-D with the raw xyz) and passed through a 6-layer MLP with 128 hidden units and ReLU, where the encoding is concatenated again at layer 3 as a skip connection. Two heads read the 128-D feature: a linear layer with ReLU for the density, and a 128 → 64 → 3 MLP with a sigmoid for the colour. The model has no view dependence here. I trained it with the MSE loss for 250 epochs on the 128x128 lego images, with 1024 rays per iteration, 128 samples per ray between depth 2 and 6, and learning rate 5e-4 decayed by 0.8 every 50 epochs.

4. NeRF Extras

4.1 View Dependence

The viewing direction of each ray is copied to all of its sample points, encoded with 2 harmonic frequencies (15-D) and concatenated to the 128-D feature, but only after the density has been computed. The density therefore depends on position only and only the small colour head sees the direction. A flag view_dependence in the config turns this on, so the lego model from Q3 is unchanged. I trained on the materials scene at 128x128 with and without view dependence, and at 400x400 with it (nerf_materials.yaml and nerf_materials_highres.yaml, 250 epochs each).

no view dependence (128)
view dependence (128)
view dependence (400)

PSNR on 20 training views and 20 held-out test views (epoch-200 checkpoints):

ModelTrain PSNRTest PSNRGap
No view dependence, 12823.2921.252.04
View dependence, 12823.6621.402.26
View dependence, 40024.3921.872.52

With view dependence the highlights on the metallic spheres move across the surface as the camera orbits, while without it the spheres look flat and matte, since a single colour per point has to average the highlight over all views. Both train and test PSNR improve, but the train improvement is larger and the gap between them grows. The extra freedom lets the network explain parts of the training images with the viewing direction instead of the geometry, which is a form of overfitting. With only 100 training views, the directions in between are interpolated by the colour head, so too much view dependence gives worse novel views. That is why I followed the NeRF paper: the direction enters only after the density (the geometry has to be consistent across all views), it uses fewer frequencies than the position (smooth changes with angle) and it goes through a small head. More view dependence fits shiny materials better at the cost of generalisation.

B. 3D Gaussian Splatting

5. 3D Gaussian Rasterization

The 3D covariance is R S Sᵀ Rᵀ with R from the quaternion and S the diagonal of scales (just s² I for isotropic Gaussians, whose quaternions are never used). It is projected to 2D with J W Σ Wᵀ Jᵀ, using the camera rotation as W and the perspective Jacobian J, and the means are projected with transform_points_screen. Gaussians are sorted by view-space depth and those with depth below 0 are removed. Alpha is opacity * exp(power), the transmittance is the cumulative product of 1 - alpha, and the colour, depth and mask are the sums of colour, depth and 1 weighted by alpha * T. All 4 unit tests pass.

6. Training 3D Gaussian Representations

The 10,000 isotropic Gaussians are initialised from the truck point cloud. I trained them with an L1 loss for 1000 iterations using these learning rates (Adam):

ParameterLearning rate
Opacities (pre-sigmoid)0.05
Scales (log)0.005
Colours0.01
Means0.001

The opacities start at 8 before the sigmoid, so they need a large rate to change at all, while the means need a small one so the Gaussians don't drift off the truck. I also tried means at 0.0005 and 0.002 and colours at 0.02 and 0.05, which were all slightly worse.

PSNR = 29.981, SSIM = 0.942

Training progress (top: renders, bottom: ground truth).

Final renders of the training views.

7. Extensions

7.1 Rendering Using Spherical Harmonics

The chair Gaussians store degree-3 spherical harmonics, 16 coefficients per colour channel. For each Gaussian I compute the unit direction from the camera centre to its mean, evaluate the 16 SH basis functions at that direction and take their weighted sum with the coefficients, plus 0.5 and clamped at 0. With only the first (DC) coefficient this gives exactly the colours used in 5.5.

5.5: DC colour only
7.1: full spherical harmonics

Side-by-side RGB renders of the same frame (left: DC only, right: spherical harmonics):

Frame 3: with SH the fabric is a deeper green with darker shading in the folds and the gold trim looks brighter and more metallic. With DC only the chair looks flat and washed out, like matte paint.

Frame 12: the same effect on the back, with more contrast in the fabric pattern and highlights on the carved wood. This is the largest difference of the three frames (mean 6.1 out of 255).

Frame 20: the underside is matte beige and the two renders are almost identical (mean difference 1.8). Its colour hardly depends on the viewing direction, so the higher SH terms add almost nothing there.

7.2 Training On a Harder Scene

Baseline. The same setup as Q6 (isotropic Gaussians, the Q6 learning rates, L1 loss, 1000 iterations) with 10,000 Gaussians at random init on the materials scene: PSNR = 18.293, SSIM = 0.694.

Baseline training progress.

The random init puts all Gaussians in a ball of standard deviation 0.2 around the origin, while the objects spread over roughly ±1.5. With a means learning rate of 0.001 most Gaussians never reach the outer objects, which stay blurry smears. Training twice as long barely helps (19.00).

Improved. PSNR = 22.944, SSIM = 0.851 after 2000 iterations. The changes are:

Improved training progress.

baseline final renders
improved final renders

Experiments, each adding one change and trained for 2000 iterations (single runs, so differences under about 0.5 dB are within run-to-run noise from the random init):

SettingPSNRSSIM
Baseline setup, 2000 iterations19.000.709
+ wider init (std 0.8)22.670.828
+ means lr 0.005 and lr decay22.680.831
+ SSIM loss22.720.841
+ anisotropic22.760.843
+ init std 0.5 instead of 0.8 (final)22.990.854
final, init std 1.2 instead22.250.817
final with 4.5x larger initial scales21.670.803
final with 15,000 Gaussians22.820.847

Larger initial scales made the Gaussians overlap and blur the result, and 15,000 Gaussians gave a similar score for about 50% more memory and time, so I kept 10,000. The remaining errors are mostly the dark caps of the objects, which blend into the black background, and grainy edges where the Gaussians are still small.