Project 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

By Teja Kanthamneni · 16-825 Learning for 3D Vision

Differentiable rendering and scene reconstruction from posed images.

Part A · Neural volume rendering

1. Differentiable Volume Rendering

The renderer generates camera rays, samples positions along each ray, and composites density and color using emission–absorption weights. The same weights are used to render depth.

1.3 Ray sampling

Pixel coordinates span [−1, 1] in normalized device coordinates. Unprojecting the image-plane points and subtracting the camera center gives normalized world-space ray directions. The smooth grid and ray color gradients show that neighboring pixels produce smoothly varying rays.

1.4 Point sampling

We sample 64 evenly spaced distances between the near and far planes and evaluate points as o + td. The visualization shows points at which the renderer queries density and color.

1.5 Volume rendering

For each ray segment, we compute opacity as α = 1 − exp(−σΔt) and transmittance as the product of the preceding segments’ remaining transparency. The rendering weights are w = Tα. Summing weighted colors produces the RGB image, and summing the weighted sample distances produces depth.

2. Optimizing a Basic Implicit Volume

A random subset of image pixels is used at each optimization step. Mean squared error between predicted and target colors supervises the box position and side lengths. I used Adam with a learning rate of 0.0005 for 1,000 epochs, sampling 4,096 rays per training image and 64 points per ray.

2.2 Learned box parameters

ParameterOptimized value (rounded to 0.01)
Center (x, y, z)(0.25, 0.25, 0.00)
Side lengths (x, y, z)(2.01, 1.50, 1.50)

2.3 Optimized volume

Optimization moves the center from (0, 0, 0) to approximately (0.25, 0.25, 0) and increases the x side length from 1.50 to 2.01. The y and z side lengths remain close to 1.50. The final orbit recovers the elongated box and rainbow faces seen in the reference. Across the training views, the larger silhouette and its shifted location are consistent with these recovered parameters. The softly blurred edges reflect the smooth SDF-to-opacity transition and finite ray sampling.

Before and after optimization · four training views

View 1

View 2

View 3

View 4

3. Optimizing a Neural Radiance Field

Here we replace the analytic box with an MLP that maps position to nonnegative density and RGB color. The spatial network has six linear layers with width 128 and a skip connection at layer index 3. Position is encoded with six harmonic frequencies per coordinate. ReLU enforces nonnegative density, and sigmoid bounds RGB values to [0, 1]. The results in this section use position alone, without view dependence.

Lego scene results

Training settingValue
Optimizer / lossAdam / RGB mean squared error
Epochs250
Initial learning rate0.0005
Learning-rate decayExponential: LR = initial LR × 0.8epoch / 50
Rays per training image1,024
Samples per ray / depth range128 / [2, 6]

4. NeRF Extension

4.1 View dependence

Here, density continues to depend only on position. For color, I concatenated the spatial feature vector with the viewing direction encoded using two harmonic frequencies per coordinate, then apply a 64-unit hidden layer with ReLU and an RGB output layer with sigmoid. This lets appearance vary with direction while sharing geometry across all views.

Materials: view dependence off versus on

The position-only model captures the arrangement and overall colors of the objects, but their appearance is comparatively smooth and muted. With view dependence, the glossy objects show stronger localized highlights and changes in brightness as the camera moves. Both models recover a similar overall layout.

View dependence increases the model’s ability to represent specular appearance, but also increases the risk of overfitting to individual training views. A flexible color network can explain away errors in geometry by changing color with direction, producing unstable appearance at unseen cameras. Keeping density independent of direction and using a small direction-conditioned color head limits this flexibility.

Additional view-dependent renders

Part B · 3D Gaussian splatting

5. 3D Gaussian Rasterization

Pretrained 3D Gaussians are projected to elliptical footprints on the image plane, sorted by depth, and blended using opacity and front-to-back transmittance. This rendering uses the view-independent color components.

Rendered chair

The RGB, depth, and silhouette views remain aligned as the camera moves. The pretrained representation preserves the chair’s upholstery pattern and frame shape, while poorly covered views show elongated splats and incomplete detail.

6. Training 3D Gaussian Representations

The truck experiment initializes Gaussian centers from the provided point cloud and optimizes isotropic scales, positions, colors, and opacities using posed training images.

6.1 Training configuration

Parameter groupLearning rate used
Positions0.0001
Log scales0.005
Colors0.01
Opacity logits0.05
Training detailValue
OptimizerAdam
Image lossL1 (mean absolute error)
Training iterations1,000

Rotation is not optimized for isotropic Gaussians.

6.2 Training results

Evaluation splitPSNR ↑SSIM ↑
Held-out truck views29.376 dB0.938

7. Gaussian Splatting Extensions

7.1 Rendering using spherical harmonics

I evaluated the pretrained spherical harmonic coefficients in the direction from the camera to each Gaussian to obtain its RGB color. Using only the degree-zero (DC) component assigns a fixed color to each Gaussian. Adding the higher-order components allows that color to vary with the viewing direction.

Both animations follow the same 32-view camera trajectory. The chair’s shape and upholstery pattern are consistent across the two results, while the full spherical harmonics change the shading and color contrast. The following comparisons crop only the RGB panel from the same numbered frame in each run.

Frame 000

With spherical harmonics, a shadow appears on the seat, making it a darker green and the gold floral pattern stand out more. The DC-only result appears brighter and flatter overall.

Frame 008

In this frame, the green cushions shows more greater contrast with the frame when spherical harmonics are enabled. The difference is subtler than in frame 000. Both images retain stretched splats around the exposed frame and legs. Changing the color representation does not repair these geometric artifacts.

Frame 028

In this frame, the spherical-harmonic rendering shows more variation in brightness across the seat cushion. Compared with the DC-only result, the cushion is darker toward the back and brighter toward the front. This illustrates how spherical harmonics capture changes in shading with viewing direction.

7.2 Training on a harder scene

I trained a Gaussian representation of the materials scene from randomly initialized centers, rather than initializing from the accurate point cloud used for the truck. All runs use 10,000 Gaussians, 128 × 128 images, a black background, and Adam with its default momentum coefficients and epsilon 10−15. Each iteration renders one training view. I evaluated the mean PSNR and SSIM over the same 100 validation views.

Experimental settings

The baseline reuses Q6’s isotropic representation, learning rates, L1 image loss, and 1,000-iteration budget. Trial 1 changes the random initialization and position optimization, and trains for 5,000 iterations. Trial 2 retains Trial 1’s initialization strategy and training budget, but uses anisotropic Gaussians and a combined L1/SSIM objective.

Training and initialization settings. Improved runs use exponential position-learning-rate decay. The other rates remain constant.
SettingBaselineTrial 1Trial 2
Gaussian typeIsotropicIsotropicAnisotropic
Training iterations1,0005,0005,000
Position learning rate0.0001, constant0.001 → 0.00010.001 → 0.0001
Log-scale learning rate0.0050.0050.005
Color learning rate0.010.010.01
Opacity-logit learning rate0.050.050.05
Quaternion learning rateNot optimizedNot optimized0.001
Initial center standard deviation0.20.50.5
Initial scale rangeApproximately (0, 0.01)[0.02, 0.05][0.02, 0.05] per axis
Initial opacityApproximately 0.99970.10.1
Image lossL1L10.8 L1 + 0.2 (1 − SSIM)

The initialization changes address the difference between fitting an accurate point cloud and finding scene geometry from random points. A wider distribution places Gaussians across a larger region. Larger initial footprints give more pixels a contribution. Reducing the initial opacity avoids starting with nearly opaque, sigmoid-saturated parameters. A larger early position learning rate allows centers to move farther, while decay supports smaller late-stage adjustments.

Trial 2 allows independent scales along three axes and optimizes a quaternion for orientation, giving each Gaussian an ellipsoidal footprint. Its differentiable SSIM loss operates on RGB images in [0, 1], while L1 also penalizes color error.

Validation results

Higher is better for both metrics. Trial 2 has the best PSNRm while Trial 1 has the best SSIM.
RunMean PSNR ↑Mean SSIM ↑
Baseline17.838 dB0.657
Trial 123.189 dB0.848
Trial 223.482 dB0.846

Trial 1 improves PSNR by 5.351 dB and SSIM by 0.191 over the baseline. This is a combined effect of initialization, position learning-rate changes, and the longer training budget. Trial 2 adds another 0.293 dB of PSNR, but SSIM decreases by 0.002. Therfore, adding anisotropy and the structural loss provides a modest pixel-error improvement, rather than a consistent gain across both metrics.

Training progress: baseline versus Trial 2

Each animation shows predictions above the corresponding ground-truth views. The baseline animation spans 1,000 updates, while Trial 2 spans 5,000, so their playback positions do not represent equal iteration counts.

The baseline leaves several objects as diffuse bright regions and gives the central objects incomplete, speckled coverage. Trial 2 recovers a more recognizable arrangement and more coherent object boundaries and colors. Fine surface detail and highlights remain blurred or uneven relative to the ground truth, so the reconstruction is still visibly approximate.

Final renderings: baseline versus Trial 2

The improved setup recovers substantially more of the scene than the Q6-style baseline under random initialization. Remaining limitations include the fixed Gaussian allocation, finite optimization budget, and view-independent RGB appearance in a scene containing reflective materials.