Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Ashutosh Panda · CMU · Fall 2026

This report presents the implementation and results for differentiable volume rendering, NeRF, and 3D Gaussian splatting. View-dependent NeRF (4.1) and spherical harmonics (7.1) are the selected extensions. VolSDF (4.3) and harder-scene training (7.2) are additional experiments.

1. Differentiable Volume Rendering

1.3 Ray sampling

Pixel coordinates are generated in [−1, 1] and unprojected from camera NDC into world coordinates. Ray origins are the camera center, and directions are normalized vectors toward the unprojected points.

Pixel coordinate grid.
Pixel coordinate grid.
Absolute ray directions visualized as RGB.
Absolute ray directions visualized as RGB.

1.4 Point sampling

For each ray, I uniformly sample distances between the near and far bounds and compute sample positions as xᵢ = o + tᵢd. The sampled positions and distances are stored in the RayBundle.

Sample points from the first camera.
Sample points from the first camera.

1.5 Volume rendering

I compute opacity from density and segment length, then use exclusive transmittance to obtain per-sample weights. Weighted sums produce RGB and depth. The first sample has transmittance 1.

αᵢ = 1 − exp(−σᵢ Δtᵢ)
Tᵢ = exp(−Σⱼ<ᵢ σⱼ Δtⱼ)
wᵢ = Tᵢ αᵢ
Color = Σᵢ wᵢ cᵢ
Depth = Σᵢ wᵢ tᵢ

The analytic box is rendered at 256 × 256 with 64 samples per ray, near bound 0, and far bound 5.

Rendered analytic box.
Rendered analytic box.

Download GIF

Depth map normalized by its maximum value; shown in grayscale.
Depth map normalized by its maximum value; shown in grayscale.

2. Optimizing a Basic Implicit Volume

I randomly sample image pixels and minimize mean squared error between rendered RGB values and the corresponding target colors. Adam optimizes the box center and side lengths through the differentiable renderer.

Training uses four posed images, 1,000 epochs, learning rate 0.0005, and 4,096 rays per step.

ParameterLearned value, rounded to two decimals
Center(0.25, 0.25, 0.00)
Side lengths(2.01, 1.50, 1.50)

The first side length is 2.005093 before rounding. Unrounded parameters.

Before optimization, camera 1.
Before optimization, camera 1.
After optimization, camera 1.
After optimization, camera 1.
Spiral rendering of the optimized box.
Spiral rendering of the optimized box.

Download GIF

The optimized shape is wider and offset relative to its initialization. Across the four target images, the saved final PNGs have mean RGB MSE 2.76 × 10⁻⁶, including quantization.

3. Optimizing a Neural Radiance Field

The position-only NeRF uses a six-layer position MLP with width 128 and a skip connection at layer index 3. Six harmonic frequency bands encode each 3D point into 39 channels. Density uses a ReLU output, while the color head (hidden width 64) uses sigmoid to keep RGB in [0, 1].

SettingValue
Dataset / image sizeLEGO / 128 × 128
Training250 epochs, Adam
Initial learning rate0.0005
Ray sampling1,024 rays/step, 128 points/ray, depths 2–6
LR scheduleDecay by a factor of 0.8 per 50 epochs
Validation PSNR24.5527 dB
Validation SSIM0.8936
Novel-view spiral of the LEGO NeRF.
Novel-view spiral of the LEGO NeRF.

Download GIF

The bulldozer, wheels, articulated arm, and platform are recognizable around the orbit. Thin details and high-frequency textures remain soft at this resolution. Full configuration.

4. NeRF Extensions

4.1 View dependence (selected extension)

Density depends only on position. The color branch additionally receives harmonic embeddings of the normalized viewing direction (two frequency bands). This allows color at a fixed location to vary with viewpoint, helping represent reflections and highlights.

Both materials runs use 250 epochs, 128 × 128 images, initial learning rate 0.0001, 1,024 rays per step, and 128 samples per ray between depths 1 and 7.

Materials: position-only baseline.
Materials: position-only baseline.

Download GIF

Materials: view-dependent NeRF.
Materials: view-dependent NeRF.

Download GIF

Materials modelValidation PSNRValidation SSIM
Position only23.3615 dB0.8589
View dependent24.3253 dB0.8749
Change+0.9638 dB+0.0161

Trade-off: direction conditioning can model specular appearance instead of averaging it across views, but extra capacity can also memorize training-ray appearances. The compact direction branch and direction-independent geometry limit this risk. The validation improvement supports this design on the materials split; it does not establish that stronger view dependence always generalizes better.

Baseline configuration · View-dependent configuration.

4.3 VolSDF (additional experiment)

A neural surface predicts signed distance and per-point color. I convert signed distance to density with a Laplace CDF and add an Eikonal loss to encourage unit-magnitude SDF gradients. The distance network learns a residual around an initial sphere.

σ(s) = α Ψβ(−s)
L_eikonal = mean[(‖∇s‖₂ − 1)²]
VolSDF color rendering of LEGO.
VolSDF color rendering of LEGO.

Download GIF

Extracted zero-level geometry.
Extracted zero-level geometry.

Download GIF

This run uses α = 10, β = 0.05, Eikonal weight 0.02, and 250 epochs. It reaches 20.6421 dB PSNR / 0.7507 SSIM. Geometry is recognizable, but the color result has blur and ghosting. Only this configuration was evaluated; no hyperparameter sweep is claimed. Configuration.

Meaning of α and β: α scales the density and therefore the strength of absorption. β controls the width of the transition from outside to inside. At the zero-level surface, density is α/2.

1. How do high and low β bias the SDF? A high β spreads density into a broad band around the surface. The network can shift the zero-level surface to compensate for opacity outside it, so the rendered and extracted boundaries need not agree closely. A low β ties opacity more closely to the zero level, favoring a sharp boundary but increasing sensitivity to sampling.

2. Which is easier to train? A moderately high β is generally easier early in training because useful density gradients reach points farther from the current surface. With very low β, uniform samples can miss the narrow transition. Extremely high β can still leave geometry poorly constrained.

3. Which is more likely to recover an accurate surface? A low β can better align the visible boundary with the zero-level surface, provided sampling resolves the narrow transition and optimization converges. Starting smoother and reducing β later is a useful future experiment, but that schedule was not evaluated here.

5. 3D Gaussian Rasterization

I construct 3D covariance matrices from rotation and squared scale, project them with a perspective Jacobian, and add 0.3 to the projected covariance diagonal for a minimum pixel footprint. Gaussian means are projected into screen coordinates. Gaussians behind the camera are filtered out, and the remaining ones are sorted front to back.

For each pixel, the Gaussian falloff times opacity gives alpha. An exclusive product of transparency gives transmittance. Color, depth, and silhouette use the resulting compositing weights. Transmittance is carried across Gaussian mini-batches.

αᵢ(x) = oᵢ exp[−½(x−μᵢ)ᵀ Σᵢ⁻¹ (x−μᵢ)]
Tᵢ(x) = Πⱼ<ᵢ [1−αⱼ(x)]
Color(x) = Σᵢ Tᵢ(x) αᵢ(x) cᵢ

All 4/4 supplied Gaussian unit tests pass.

Pretrained chair: RGB, depth, and silhouette using DC color only.
Pretrained chair: RGB, depth, and silhouette using DC color only.

Download GIF

The chair is recognizable, with streaking around elongated footprints at oblique angles. A separate post-run camera-Jacobian check identified a covariance-convention inconsistency despite the reference tests passing. These figures show the current run and have not been regenerated under a revised convention.

6. Training 3D Gaussian Representations

The toy truck uses 10,000 isotropic Gaussians initialized from the provided point cloud. Means, log-scales, opacity logits, and colors are optimized with Adam and RGB L1 loss. Quaternions are fixed in isotropic mode.

Optimized parameterLearning rate
Means0.00016
Log-scales0.005
Colors0.01
Opacity logits0.01
QuaternionsNot optimized (isotropic)

Training uses 1,000 iterations, 256 Gaussians per splat, and seed 1. Held-out metrics are 24.9622 dB PSNR and 0.8590 SSIM, evaluated on five test views. Metrics and settings.

Truck training progression: predictions above and ground truth below.
Truck training progression: predictions above and ground truth below.

Download GIF

Final truck renderings from the training cameras.
Final truck renderings from the training cameras.

Download GIF

The model captures broad blue appearance but does not recover sharp cabin and wheel details. The large black background helps whole-image scores; a separate object-crop diagnostic gives about 18.09 dB PSNR. The relatively large initial splats are a plausible contributor to blur. Smaller initialization followed by retraining would be an appropriate next comparison.

7. Gaussian Extensions

7.1 Spherical harmonics (selected extension)

I evaluate real spherical harmonics through degree three, using 16 coefficients per RGB channel and the normalized camera-to-Gaussian direction. The pretrained coefficients are loaded from the PLY; new SH coefficients are not trained.

Pretrained chair rendered with spherical-harmonic color.
Pretrained chair rendered with spherical-harmonic color.

Download GIF

The DC-only GIF is in Section 5. The comparisons below use the exact same saved camera/frame for both methods. Depth and silhouette are unchanged across the two rendering modes.

Frame 000: SH darkens the green upholstery and increases contrast in the gold pattern. The outline is unchanged.
Frame 000: SH darkens the green upholstery and increases contrast in the gold pattern. The outline is unchanged.
Frame 008: SH produces darker upholstery and stronger contrast in the central ornament. Both versions retain stretched edge footprints; changing color does not fix projection artifacts.
Frame 008: SH produces darker upholstery and stronger contrast in the central ornament. Both versions retain stretched edge footprints; changing color does not fix projection artifacts.
Frame 016: the DC-only image is brighter and more uniform. SH introduces stronger directional shading on the backrest and seat while preserving geometry.
Frame 016: the DC-only image is brighter and more uniform. SH introduces stronger directional shading on the backrest and seat while preserving geometry.

7.2 Training on a harder scene (additional experiment)

The materials baseline reuses the truck’s isotropic representation, learning rates, and L1 loss, with random means instead of point-cloud initialization. The modified method changes initialization, opacity, Gaussian shape, loss, schedule, and training duration.

MethodIterationsValidation PSNRValidation SSIM
Isotropic baseline1,00017.9657 dB0.6426
Modified anisotropic method5,00022.9757 dB0.8434
Materials baseline training progression: predictions above, ground truth below.
Materials baseline training progression: predictions above, ground truth below.

Download GIF

Materials modified training progression: predictions above, ground truth below.
Materials modified training progression: predictions above, ground truth below.

Download GIF

Final materials baseline views.
Final materials baseline views.

Download GIF

Final modified materials views.
Final modified materials views.

Download GIF

ChangeBaselineModified method
Gaussian shapeIsotropicAnisotropic
Mean initializationNormal, standard deviation 0.2Initial means × 3 (standard deviation 0.6)
Opacity logit8 (nearly opaque)−2 (opacity ≈ 0.119)
Scale initializationRandom, roughly below 0.01Log-scale −3.5 (scale ≈ 0.0302)
LossL10.8 L1 + 0.2 (1 − SSIM)
ScheduleConstant learning ratesExponential decay to approximately 10%
Iterations1,0005,000

Initial rates match Section 6, with quaternion learning rate 0.001 added for the anisotropic run. No adaptive densification is used. Lower initial opacity allows several Gaussians to contribute, broader initialization improves coverage, and anisotropic footprints add flexibility. The combined approach improves PSNR by 5.0100 dB and SSIM by 0.2007; objects become clearer, but edges remain soft.

This is not an isolated ablation: the modified model also trains five times longer. The full gain cannot be attributed to anisotropy alone. The anisotropic result shares the covariance-convention limitation described in Section 5.

Baseline metrics and settings · Modified metrics and settings.

Summary of Measurements

ExperimentTrainingPSNR (dB)SSIM
LEGO NeRF250 epochs24.55270.8936
Materials NeRF, position only250 epochs23.36150.8589
Materials NeRF, view dependent250 epochs24.32530.8749
LEGO VolSDF250 epochs20.64210.7507
Truck Gaussians1,000 iterations24.96220.8590
Materials Gaussians, baseline1,000 iterations17.96570.6426
Materials Gaussians, modified5,000 iterations22.97570.8434

Download measurements as CSV. Runs use seed 1. NeRF/materials images are 128 × 128; truck images are 160 × 90. Progress GIFs are resized for the webpage (640 × 320, or 512 × 256 for the modified materials run); all saved snapshots are retained. Recorded metrics are unchanged.

References

Mildenhall et al., NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (2020).

Yariv et al., Volume Rendering of Neural Implicit Surfaces (2021).

Kerbl et al., 3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023).