Project 3: Volume Rendering, NeRF, and 3D Gaussian Splatting
By Teja Kanthamneni · 16-825 Learning for 3D Vision
Differentiable rendering and scene reconstruction from posed images.
Part A · Neural volume rendering
1. Differentiable Volume Rendering
The renderer generates camera rays, samples positions along each ray, and composites density and color using emission–absorption weights. The same weights are used to render depth.
1.3 Ray sampling
Image-plane pixel gridWorld-space camera rays
Pixel coordinates span [−1, 1] in normalized device coordinates. Unprojecting the image-plane points and subtracting the camera center gives normalized world-space ray directions. The smooth grid and ray color gradients show that neighboring pixels produce smoothly varying rays.
1.4 Point sampling
Points sampled along the rays
We sample 64 evenly spaced distances between the near and far planes and evaluate points as o + td. The visualization shows points at which the renderer queries density and color.
1.5 Volume rendering
Rendered volumeDepth normalized by its maximum · purple: 0, yellow: 1
For each ray segment, we compute opacity as α = 1 − exp(−σΔt) and transmittance as the product of the preceding segments’ remaining transparency. The rendering weights are w = Tα. Summing weighted colors produces the RGB image, and summing the weighted sample distances produces depth.
2. Optimizing a Basic Implicit Volume
A random subset of image pixels is used at each optimization step. Mean squared error between predicted and target colors supervises the box position and side lengths. I used Adam with a learning rate of 0.0005 for 1,000 epochs, sampling 4,096 rays per training image and 64 points per ray.
2.2 Learned box parameters
Parameter
Optimized value (rounded to 0.01)
Center (x, y, z)
(0.25, 0.25, 0.00)
Side lengths (x, y, z)
(2.01, 1.50, 1.50)
2.3 Optimized volume
Optimized box rendering
Optimization moves the center from (0, 0, 0) to approximately (0.25, 0.25, 0) and increases the x side length from 1.50 to 2.01. The y and z side lengths remain close to 1.50. The final orbit recovers the elongated box and rainbow faces seen in the reference. Across the training views, the larger silhouette and its shifted location are consistent with these recovered parameters. The softly blurred edges reflect the smooth SDF-to-opacity transition and finite ray sampling.
Before and after optimization · four training views
View 1
Before optimizationAfter optimization
View 2
Before optimizationAfter optimization
View 3
Before optimizationAfter optimization
View 4
Before optimizationAfter optimization
3. Optimizing a Neural Radiance Field
Here we replace the analytic box with an MLP that maps position to nonnegative density and RGB color. The spatial network has six linear layers with width 128 and a skip connection at layer index 3. Position is encoded with six harmonic frequencies per coordinate. ReLU enforces nonnegative density, and sigmoid bounds RGB values to [0, 1]. The results in this section use position alone, without view dependence.
Here, density continues to depend only on position. For color, I concatenated the spatial feature vector with the viewing direction encoded using two harmonic frequencies per coordinate, then apply a 64-unit hidden layer with ReLU and an RGB output layer with sigmoid. This lets appearance vary with direction while sharing geometry across all views.
The position-only model captures the arrangement and overall colors of the objects, but their appearance is comparatively smooth and muted. With view dependence, the glossy objects show stronger localized highlights and changes in brightness as the camera moves. Both models recover a similar overall layout.
View dependence increases the model’s ability to represent specular appearance, but also increases the risk of overfitting to individual training views. A flexible color network can explain away errors in geometry by changing color with direction, producing unstable appearance at unseen cameras. Keeping density independent of direction and using a small direction-conditioned color head limits this flexibility.
Pretrained 3D Gaussians are projected to elliptical footprints on the image plane, sorted by depth, and blended using opacity and front-to-back transmittance. This rendering uses the view-independent color components.
Rendered chair
RGB rendering (left), depth (center), and silhouette mask (right).
The RGB, depth, and silhouette views remain aligned as the camera moves. The pretrained representation preserves the chair’s upholstery pattern and frame shape, while poorly covered views show elongated splats and incomplete detail.
6. Training 3D Gaussian Representations
The truck experiment initializes Gaussian centers from the provided point cloud and optimizes isotropic scales, positions, colors, and opacities using posed training images.
6.1 Training configuration
Parameter group
Learning rate used
Positions
0.0001
Log scales
0.005
Colors
0.01
Opacity logits
0.05
Training detail
Value
Optimizer
Adam
Image loss
L1 (mean absolute error)
Training iterations
1,000
Rotation is not optimized for isotropic Gaussians.
6.2 Training results
Training progress: predictions (top) and ground truth (bottom).Final truck renderings from multiple camera views.
Evaluation split
PSNR ↑
SSIM ↑
Held-out truck views
29.376 dB
0.938
7. Gaussian Splatting Extensions
7.1 Rendering using spherical harmonics
I evaluated the pretrained spherical harmonic coefficients in the direction from the camera to each Gaussian to obtain its RGB color. Using only the degree-zero (DC) component assigns a fixed color to each Gaussian. Adding the higher-order components allows that color to vary with the viewing direction.
Q5 baseline · DC color onlyQ7.1 · full spherical harmonic color
Both animations follow the same 32-view camera trajectory. The chair’s shape and upholstery pattern are consistent across the two results, while the full spherical harmonics change the shading and color contrast. The following comparisons crop only the RGB panel from the same numbered frame in each run.
Frame 000
Frame 000 · DC onlyFrame 000 · spherical harmonics
With spherical harmonics, a shadow appears on the seat, making it a darker green and the gold floral pattern stand out more. The DC-only result appears brighter and flatter overall.
Frame 008
Frame 008 · DC onlyFrame 008 · spherical harmonics
In this frame, the green cushions shows more greater contrast with the frame when spherical harmonics are enabled. The difference is subtler than in frame 000. Both images retain stretched splats around the exposed frame and legs. Changing the color representation does not repair these geometric artifacts.
Frame 028
Frame 028 · DC onlyFrame 028 · spherical harmonics
In this frame, the spherical-harmonic rendering shows more variation in brightness across the seat cushion. Compared with the DC-only result, the cushion is darker toward the back and brighter toward the front. This illustrates how spherical harmonics capture changes in shading with viewing direction.
7.2 Training on a harder scene
I trained a Gaussian representation of the materials scene from randomly initialized centers, rather than initializing from the accurate point cloud used for the truck. All runs use 10,000 Gaussians, 128 × 128 images, a black background, and Adam with its default momentum coefficients and epsilon 10−15. Each iteration renders one training view. I evaluated the mean PSNR and SSIM over the same 100 validation views.
Experimental settings
The baseline reuses Q6’s isotropic representation, learning rates, L1 image loss, and 1,000-iteration budget. Trial 1 changes the random initialization and position optimization, and trains for 5,000 iterations. Trial 2 retains Trial 1’s initialization strategy and training budget, but uses anisotropic Gaussians and a combined L1/SSIM objective.
Training and initialization settings. Improved runs use exponential position-learning-rate decay. The other rates remain constant.
Setting
Baseline
Trial 1
Trial 2
Gaussian type
Isotropic
Isotropic
Anisotropic
Training iterations
1,000
5,000
5,000
Position learning rate
0.0001, constant
0.001 → 0.0001
0.001 → 0.0001
Log-scale learning rate
0.005
0.005
0.005
Color learning rate
0.01
0.01
0.01
Opacity-logit learning rate
0.05
0.05
0.05
Quaternion learning rate
Not optimized
Not optimized
0.001
Initial center standard deviation
0.2
0.5
0.5
Initial scale range
Approximately (0, 0.01)
[0.02, 0.05]
[0.02, 0.05] per axis
Initial opacity
Approximately 0.9997
0.1
0.1
Image loss
L1
L1
0.8 L1 + 0.2 (1 − SSIM)
The initialization changes address the difference between fitting an accurate point cloud and finding scene geometry from random points. A wider distribution places Gaussians across a larger region. Larger initial footprints give more pixels a contribution. Reducing the initial opacity avoids starting with nearly opaque, sigmoid-saturated parameters. A larger early position learning rate allows centers to move farther, while decay supports smaller late-stage adjustments.
Trial 2 allows independent scales along three axes and optimizes a quaternion for orientation, giving each Gaussian an ellipsoidal footprint. Its differentiable SSIM loss operates on RGB images in [0, 1], while L1 also penalizes color error.
Validation results
Higher is better for both metrics. Trial 2 has the best PSNRm while Trial 1 has the best SSIM.
Run
Mean PSNR ↑
Mean SSIM ↑
Baseline
17.838 dB
0.657
Trial 1
23.189 dB
0.848
Trial 2
23.482 dB
0.846
Trial 1 improves PSNR by 5.351 dB and SSIM by 0.191 over the baseline. This is a combined effect of initialization, position learning-rate changes, and the longer training budget. Trial 2 adds another 0.293 dB of PSNR, but SSIM decreases by 0.002. Therfore, adding anisotropy and the structural loss provides a modest pixel-error improvement, rather than a consistent gain across both metrics.
Training progress: baseline versus Trial 2
Each animation shows predictions above the corresponding ground-truth views. The baseline animation spans 1,000 updates, while Trial 2 spans 5,000, so their playback positions do not represent equal iteration counts.
Baseline · isotropic Gaussians and L1 lossTrial 2 · anisotropic Gaussians and L1 + SSIM loss
The baseline leaves several objects as diffuse bright regions and gives the central objects incomplete, speckled coverage. Trial 2 recovers a more recognizable arrangement and more coherent object boundaries and colors. Fine surface detail and highlights remain blurred or uneven relative to the ground truth, so the reconstruction is still visibly approximate.
Final renderings: baseline versus Trial 2
Baseline · final renderingsTrial 2 · final renderings
The improved setup recovers substantially more of the scene than the Q6-style baseline under random initialization. Remaining limitations include the fixed Gaussian allocation, finite optimization budget, and view-independent RGB appearance in a scene containing reflective materials.