Volume rendering, NeRF, and Gaussian splatting

16-825 Learning for 3D Vision · Assignment 3
Joshua Jacob · JJACOB2

1. Differentiable volume rendering

1.3. Camera rays

Pixel locations are mapped into normalized device coordinates. Unprojecting those coordinates gives a point on each camera ray; subtracting the camera center and normalizing gives its world-space direction.

Image coordinates visualized as RGB values.
Image coordinates visualized as RGB values.
World-space ray directions from the first camera.
World-space ray directions from the first camera.

1.4. Sampling points along a ray

Each sample position is the ray origin plus its distance times the unit ray direction. Sampling remains inside the configured near and far bounds.

Sampled points for the first camera.
Sampled points for the first camera.

1.5. Emission–absorption rendering

A segment with density σ and length Δ has opacity α = 1 − exp(−σΔ). Its transmittance is the product of (1 − α) over earlier segments, so the first segment has transmittance one. The renderer sums RGB values weighted by αT. Replacing RGB by sample distance gives the depth map.

Rendered box from a camera orbit.
Rendered box from a camera orbit.
Accumulated depth, normalized by the maximum depth value for display.
Accumulated depth, normalized by the maximum depth value for display.

2. Optimizing a basic implicit volume

Random pixel subsets limit memory during training. Mean squared RGB error updates the box center and side lengths from the four supplied posed views.

ParameterResult
Center (x, y, z)(0.25, 0.25, 0.00)
Side lengths (x, y, z)(2.01, 1.50, 1.50)
Epochs / updates1000 / 4000
Final training MSE0.0000009
Training time (minutes)0.35
Optimized box rendered from a camera orbit.
Optimized box rendered from a camera orbit.

The optimized box is wider along x and displaced toward positive x and y. Its orbit retains the colored faces of the supplied target. The rendered edges remain soft because the SDF-to-opacity mapping uses a finite-width transition.

3. Optimizing a neural radiance field

The MLP encodes 3D position with sine and cosine features. Its density branch uses ReLU and its color branch uses sigmoid. Training minimizes RGB mean squared error on sampled rays from the supplied Lego training split. The checkpoint selected by validation PSNR is evaluated on held-out test images. This experiment uses position-dependent color.

ModelTest PSNR (dB)Test SSIMEpochsSelected epochUpdatesTraining (min)Test rendering (s)Parameters
Uniform sampling24.170.881225025025,0002.954.72101,252
Coarse + fine24.490.888125025025,0003.747.91202,504
Lego reconstruction with the baseline neural radiance field.
Lego reconstruction with the baseline neural radiance field.
Held-out Lego example 1: ground truth on the left, baseline prediction on the right.
Held-out Lego example 1: ground truth on the left, baseline prediction on the right.
Held-out Lego example 2: ground truth on the left, baseline prediction on the right.
Held-out Lego example 2: ground truth on the left, baseline prediction on the right.
Held-out Lego example 3: ground truth on the left, baseline prediction on the right.
Held-out Lego example 3: ground truth on the left, baseline prediction on the right.

The orbit reconstructs the yellow bulldozer body, raised arm, tracks, and gray base from front, rear, and side views. At 128-pixel resolution, track edges, windows, and small mechanical parts remain soft, and the arm has a few irregular edges.

The supplied split contains 100 training, 100 validation, and 200 test views. Both runs use seed 1, 128 × 128 images, and batches of 1,024 rays.

PSNR is computed from image RGB error; SSIM compares local image structure. The reported values are means over held-out views. Baseline configuration and full measured results record the run settings.

4.2. Coarse/fine sampling

The coarse network first estimates density along each ray. Its rendering weights define a probability distribution for additional samples. Inverse-CDF sampling places more points in intervals that contribute to the rendered color. A separate fine network evaluates the combined, depth-sorted sample locations. Both networks receive RGB supervision. The baseline queries 128 uniform samples. The hierarchical model queries 64 coarse locations, adds 64 importance samples, and evaluates the fine network at all 128 sorted locations: 192 network queries per ray in total.

Lego reconstruction from the fine network.
Lego reconstruction from the fine network.
The same held-out example 1: ground truth on the left, fine-network prediction on the right.
The same held-out example 1: ground truth on the left, fine-network prediction on the right.
The same held-out example 2: ground truth on the left, fine-network prediction on the right.
The same held-out example 2: ground truth on the left, fine-network prediction on the right.
The same held-out example 3: ground truth on the left, fine-network prediction on the right.
The same held-out example 3: ground truth on the left, fine-network prediction on the right.

The coarse/fine orbit keeps the same bulldozer layout and resolves some small edges more distinctly. The gain is modest: studs and the underside remain smooth, and dark spots persist beneath the platform. Importance sampling improves the measured reconstruction without removing every visible artifact.

Measured training loss and validation PSNR for the two Lego runs.
Measured training loss and validation PSNR for the two Lego runs.

Relative to the baseline, coarse/fine sampling changes test PSNR by +0.31 dB and SSIM by +0.0068. Measured optimization time is 1.27× the baseline; held-out rendering time is 1.68×. These are measurements from the recorded runs on the same desktop, including their stated configurations and epoch counts. The GPU was shared with Gaussian training and other desktop workloads during part of these runs. These wall times are observations under that load, not a controlled or equal-compute speed benchmark. Training timings exclude validation and final rendering.

Concentrating samples near occupied regions can resolve detail that uniform sampling misses. The second network and its extra evaluations cost memory and time. Early coarse predictions can also direct samples poorly, so retaining uniform samples gives the fine network coverage of the whole ray. Coarse/fine configuration.

5. 3D Gaussian rasterization

Each Gaussian covariance is constructed from its scale and rotation. A perspective-projection Jacobian maps the covariance into image space. Gaussians behind the camera are discarded; the remainder are sorted front to back. Their projected Gaussian power, opacity, and accumulated transmittance determine each pixel contribution. The same weights accumulate color, depth, and silhouette.

Pretrained chair with view-independent color. Each frame shows RGB, depth, and silhouette.
Pretrained chair with view-independent color. Each frame shows RGB, depth, and silhouette.

6. Training a Gaussian representation

The supplied truck point cloud initializes the centers of isotropic Gaussians. Training updates centers, scales, opacities, and RGB colors through the differentiable renderer using L1 image loss. Held-out camera views are reserved for evaluation.

Setting / metricValue
Iterations1000
Gaussians10000
Seed3
Training time (minutes)5.18
Held-out views5
Mean test PSNR (dB)29.35
Mean test SSIM0.9398
ParameterLearning rate
opacities0.05
scales0.005
colours0.01
means0.0001
Truck optimization: predicted views above the matching ground-truth images.
Truck optimization: predicted views above the matching ground-truth images.
Final truck representation across the supplied training camera views. Quantitative metrics use separate held-out views.
Final truck representation across the supplied training camera views. Quantitative metrics use separate held-out views.

7.1. Rendering with spherical harmonics

The pretrained chair includes spherical harmonic coefficients. Evaluating their directional basis at each camera-to-Gaussian direction produces view-dependent RGB values. The DC-only baseline and the full spherical harmonic render use the same geometry, opacity, camera poses, and rasterizer.

Pretrained chair with spherical harmonic color; RGB, depth, and silhouette. Compare with the DC-only orbit in Section 5.
Pretrained chair with spherical harmonic color; RGB, depth, and silhouette. Compare with the DC-only orbit in Section 5.
Matched camera frame 008: DC-only and full spherical harmonics. In this oblique view, SH darkens the green backrest and increases the contrast of its gold floral pattern. The curved legs and arm supports remain crisp and share the same silhouette in both renders; the difference comes from directional color. Mean absolute foreground RGB difference: 0.0621 on the 0–1 scale.
Matched camera frame 008: DC-only and full spherical harmonics. In this oblique view, SH darkens the green backrest and increases the contrast of its gold floral pattern. The curved legs and arm supports remain crisp and share the same silhouette in both renders; the difference comes from directional color. Mean absolute foreground RGB difference: 0.0621 on the 0–1 scale.
Matched camera frame 016: DC-only and full spherical harmonics. In the front view, SH produces a darker, richer green backrest and makes the gold floral motif more distinct. The seat becomes brighter green. The silhouette and carved wooden frame retain the same geometry. Mean absolute foreground RGB difference: 0.0920 on the 0–1 scale.
Matched camera frame 016: DC-only and full spherical harmonics. In the front view, SH produces a darker, richer green backrest and makes the gold floral motif more distinct. The seat becomes brighter green. The silhouette and carved wooden frame retain the same geometry. Mean absolute foreground RGB difference: 0.0920 on the 0–1 scale.
Matched camera frame 024: DC-only and full spherical harmonics. In this rear oblique view, the large beige rear panel changes little. SH strengthens the contrast in the narrow green side upholstery and subtly changes the highlights on the pale frame. The shape and silhouette remain the same. Mean absolute foreground RGB difference: 0.0220 on the 0–1 scale.
Matched camera frame 024: DC-only and full spherical harmonics. In this rear oblique view, the large beige rear panel changes little. SH strengthens the contrast in the narrow green side upholstery and subtly changes the highlights on the pale frame. The shape and silhouette remain the same. Mean absolute foreground RGB difference: 0.0220 on the 0–1 scale.

The paired images isolate the effect of the directional color coefficients. No held-out ground-truth images are supplied for this chair, so a visible difference alone does not establish a quantitative accuracy improvement.

Reproduction and checks

EnvironmentVersion / device
deviceNVIDIA GeForce RTX 5090
python3.11.16
torch2.8.0+cu128
cuda12.8
pytorch3d0.7.9

The implementation, training, evaluation, rendering, report figures, and archive generation ran on the desktop. The report verifier decodes every GIF frame, checks all local links and required outputs, and verifies finite metrics. The Gaussian projection regression checks three rotated anisotropic cases against autograd and finite-difference Jacobians. Renderer checks cover mini-batch equivalence, finite gradients, and DC-only spherical harmonic agreement. The unchanged supplied fixtures pass two of four cases: the two cases that depend on projected covariance encode the opposite rotation convention. PyTorch3D uses X_cam = X_world R + T for row vectors, so the covariance transform uses W = Rᵀ. The chair renders use this corrected convention.

python build_report.py
python verify_artifacts.py
python package_submission.py

Setup and complete commands · Environment · Measured results · Artifact verification

Starter code, datasets, and assignment instructions: learning3d/assignment3. The NeRF implementation follows the sampling procedure described by Mildenhall et al. Gaussian projection and compositing follow Kerbl et al. The spherical harmonic basis follows the reference linked in the assignment. These implementation references are separate from the experiments reported above.