16-825 Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Andrew ID: skorada

1. Differentiable volume rendering

1.3 Ray sampling

I generated one NDC coordinate for every image pixel, unprojected those points through the camera, and formed normalized world-space ray directions from the camera center.

Pixel grid

Normalized image-coordinate grid

Camera rays

World-space camera rays

1.4 Point sampling

I sampled uniformly spaced depths between the near and far planes and evaluated each point as origin + depth * direction.

Stratified samples along camera rays

1.5 Volume rendering

For each ray I computed alpha from density and segment length, accumulated exclusive transmittance, and used the resulting weights to aggregate both color and depth.

Rendered box

Spiral volume rendering of a box

Normalized depth

Normalized depth map

2. Optimizing a basic implicit volume

I trained the box with random ray batches and mean-squared color error. The learned center was (0.25, 0.25, 0.00) and its side lengths were (2.01, 1.50, 1.50), rounded to two decimals.

Before training

Box before optimization

After training

Box after optimization
Spiral rendering of the optimized box

3. Neural Radiance Field

The NeRF uses harmonic position encoding followed by an MLP that predicts nonnegative density and sigmoid-bounded RGB. It was optimized from posed RGB views using mean-squared error.

Novel-view rendering from the trained NeRF

4. NeRF extensions

4.1 View dependence

I separated density from appearance and conditioned the color head on the viewing direction using a two-frequency harmonic embedding. Position still determines density, while the direction branch lets specular appearance change across views. The result below uses the materials scene.

View-dependent NeRF on the materials scene

View conditioning represents reflections more faithfully, but it also increases model capacity: the network can explain training images with direction-specific colors instead of consistent geometry. Limiting the direction encoding to fewer frequencies helps retain generalization to unseen views.

4.3 VolSDF

I represented geometry with a neural signed-distance field, added an Eikonal penalty that encourages gradient norm one, and mapped signed distance to density with the VolSDF cumulative-Laplace formulation. The color MLP is evaluated at the same sampled positions.

Extracted geometry

Geometry extracted from the learned SDF

Rendered color

Color rendering from VolSDF

Alpha scales the maximum density and therefore the amount of opacity a surface can accumulate. Beta controls the width of the transition around the zero level set. A low beta produces a sharper, more surface-like density; a high beta spreads density across a thicker region and biases learning toward smoother, less precisely localized surfaces. High beta is generally easier to optimize because more samples receive useful gradients, while low beta is more likely to yield an accurate surface once training succeeds.

I ran a 25-epoch ablation over the Eikonal weight, alpha, and beta. Raising alpha from 10 to 20 gave the best validation PSNR, 20.65 dB. A high beta produced a much denser, noisier extracted mesh and the lowest planarity score.

SettingEikonal weightAlphaBetaVal. PSNRMesh vertices
Baseline0.02100.0520.051,884
Low Eikonal0.005100.0520.432,003
High alpha0.02200.0520.651,886
Low beta0.02100.0220.122,280
High beta0.02100.1019.7222,919

Baseline

Baseline VolSDF geometry

High alpha

High-alpha VolSDF geometry

Low Eikonal weight

Low-Eikonal-weight VolSDF geometry

High beta

High-beta VolSDF geometry

5. 3D Gaussian rasterization

I projected each 3D covariance into the image plane with the camera Jacobian, evaluated the resulting 2D Gaussians per pixel, depth-sorted visible splats, and composited color, depth, and silhouette front to back.

Rendering of pretrained 3D Gaussians

6. Training a 3D Gaussian representation

I trained isotropic Gaussians for 1,000 iterations with an RGB MSE loss plus mask MSE. Adam learning rates were 0.05 for opacity, 0.0005 for scale, 0.01 for color, and 0.0002 for position. Held-out performance was 29.399 dB PSNR and 0.936 SSIM.

Training progress

3D Gaussian training progress and ground truth

Final training-view renders

Final renders of the optimized Gaussians

7.1 Spherical-harmonic rendering

I evaluated all degree-three spherical-harmonic coefficients against the normalized camera-to-Gaussian direction, producing a different RGB value for each Gaussian at each view. The view-dependent version reached 29.586 dB PSNR and 0.938 SSIM, a small improvement over the DC-only model.

DC component only

Gaussian rendering using only DC color

All SH coefficients

View-dependent Gaussian rendering using spherical harmonics

Matched-frame comparisons

In the first view, spherical harmonics recover stronger local highlights and more varied surface color; DC-only rendering averages these effects into flatter tones.

Frame 0, DC onlyFrame 0, spherical harmonics

At the side view, the SH image changes the highlight pattern with the camera direction and preserves more contrast on oblique surfaces.

Frame 8, DC onlyFrame 8, spherical harmonics

From the opposite view, the direction-dependent colors move with the viewpoint rather than remaining baked into each Gaussian, producing a more convincing appearance.

Frame 16, DC onlyFrame 16, spherical harmonics