Assignment 3: Volume Rendering, NeRF, and 3D Gaussian Splatting

Guo Yu · 16-825 Learning for 3D Vision

Parts 1–6 and 7.1. NeRF extensions: view dependence, coarse/fine sampling, and VolSDF.

1. Differentiable Volume Rendering

1.3 Ray Sampling

Pixel coordinates are generated in normalized device coordinates and unprojected into world space. Each ray starts at the camera center and follows the normalized direction toward its unprojected pixel.

Add result: data/part_1_grid.png
Pixel grid in normalized device coordinates.
Add result: data/part_1_rays.png
World-space camera rays.

1.4 Point Sampling

Points are sampled between the near and far bounds along each ray: x(t) = o + td.

Add result: data/part_1_samples.png
Sample points along the rays from the first camera.

1.5 Volume Rendering

Colors and depths are accumulated using emission–absorption weights. Transmittance accounts for absorption by preceding samples.

αᵢ = 1 − exp(−σᵢΔtᵢ),   Tᵢ = exp(−∑ⱼ<ᵢ σⱼΔtⱼ),   wᵢ = Tᵢαᵢ

Add result: data/part_1.gif
Spiral rendering of the box.
Add result: data/part_1_depth.gif
Rendered depth, normalized by its maximum value.

2. Optimizing a Basic Implicit Volume

2.1 Random Ray Sampling

A subset of image pixels is sampled at each iteration to reduce the memory required for differentiable rendering.

2.2 Loss and Training

The box center and side lengths are optimized using mean squared error between rendered RGB values and ground-truth colors.

ParameterOptimized value (rounded to 0.01)
Center (x, y, z)(0.25, 0.25, -0.00)
Side lengths (x, y, z)(2.01, 1.50, 1.50)

2.3 Visualization

Add result: data/part_2.gif
Spiral rendering of the optimized box.

3. Optimizing a Neural Radiance Field

An MLP maps encoded 3D positions to density and RGB color. Density is constrained to be nonnegative using ReLU; color is mapped to [0, 1] using sigmoid. The field is optimized with image reconstruction supervision.

[TODO: report the actual network architecture, positional encoding, training epochs, and image resolution.]

Add result: data/part_3.gif
Novel-view spiral rendering of the Lego scene.

4. NeRF Extensions

4.1 View Dependence

Color prediction additionally depends on viewing direction, while density depends only on position. This separates scene geometry from directional appearance.

Add result: data/part_4_1.gif
View-dependent NeRF rendering of the materials scene.

Trade-off. Directional conditioning can represent specular highlights and reflections. Excessive directional capacity can also fit training views without learning appearance that interpolates reliably to unseen cameras. Restricting direction conditioning to the color branch and controlling its encoding and network capacity can improve generalization.

View dependence allows NeRF to represent appearance changes across viewing directions, such as specular highlights and reflections in the materials scene. However, increasing directional capacity can encourage the model to memorize training views rather than learn appearance that generalizes to unseen viewpoints. Conditioning only the color branch on viewing direction, while keeping density position-dependent, helps separate directional appearance from geometry. A moderate directional encoding provides a practical balance between modeling these effects and maintaining novel-view consistency.

4.2 Coarse/Fine Sampling

A coarse pass estimates where samples contribute along each ray. Additional samples are drawn from the resulting weight distribution; the fine pass renders the combined, depth-sorted samples.

Add result: data/part_4_2_coarse.gif
Coarse-pass rendering.
Add result: data/part_4_2_fine.gif
Fine-pass rendering.

Speed / quality trade-off. Importance sampling concentrates evaluations near contributing regions and can recover detail more efficiently than uniform sampling alone. Two network passes and resampling add computation and memory; total runtime depends on both sample counts and network sizes. A poor coarse estimate can miss regions that should receive fine samples.

Coarse-to-fine sampling uses the coarse network’s rendering weights to allocate additional samples to regions likely to contribute to the final image. This can improve sampling efficiency and recover finer details, but requires an additional network pass and resampling. In my experiments, iteration time increased from approximately 9 seconds with the baseline to 11 seconds with coarse-to-fine sampling, an overhead of about 22%. The rendered results looked similar, with no substantial visual improvement. Under the current settings, coarse-to-fine sampling therefore introduced a moderate computational cost for limited visible benefit.

4.3 VolSDF

A neural field predicts signed distance and color. Signed distance is converted to volume density using a scaled Laplace CDF, and the resulting field is rendered with the volume renderer.

α and β. With negative signed distance inside the surface, σ(x) = α · Fβ(−s(x)). The parameter α controls the density scale; β controls the width of the transition around the zero level set.

High β. A broad density transition distributes contributions across a thicker region around the surface. This can provide gradients over a wider range and make early optimization easier, but it can also introduce geometric bias and soften the recovered surface.

Low β. A narrow transition concentrates rendering near the surface and can support more accurate localization once geometry is reasonably estimated. It may make optimization harder because useful gradients are more localized. A lower β therefore does not by itself guarantee a more accurate learned surface.

Eikonal loss. The regularizer encourages the spatial gradient of the predicted distance to have unit norm, helping the field behave like a signed distance function.

LEikonal = Ex [(‖∇s(x)‖₂ − 1)²]

Add result: data/part_4_3_geometry.gif
VolSDF geometry reconstruction of the Lego scene.
Add result: data/part_4_3.gif
VolSDF color rendering of the Lego scene.

alpha: 10.0 beta: 0.05 eikonal_weight: 0.02 num_epochs: 100 batch_size: 1024 lr: 0.0005

5. 3D Gaussian Rasterization

5.1 Projecting 3D Gaussians

Each Gaussian’s scale and rotation define its 3D covariance. The center is projected into image space, and the covariance is transformed using the camera rotation and perspective-projection Jacobian.

5.2 Evaluating 2D Gaussians

The projected covariance defines the quadratic Gaussian exponent at each pixel.

5.3 Filtering and Sorting

Gaussian centers are transformed into camera space. Gaussians behind the camera are removed, and the remaining Gaussians are sorted from near to far.

5.4 Alpha and Transmittance

Pixel alpha is the Gaussian opacity multiplied by its unnormalized Gaussian response. Exclusive cumulative products of (1 − α) produce front-to-back transmittance.

5.5 Splatting Results

RGB, depth, and mask are accumulated with weights αᵢTᵢ. Remaining transmittance contributes the background color.

Add result: data/part_5_render.gif
Pretrained chair rendered with view-independent color. Each frame shows RGB, depth, and mask.

6. Training 3D Gaussian Representations

6.1 Parameters and Optimizer

Isotropic Gaussians are initialized from the provided truck point cloud. Their means, scales, opacities, and colors are optimized with Adam. Rotation is not optimized for isotropic Gaussians.

ParameterLearning rate
Means0.0001
Scales0.005
Opacities0.01
Colors0.0025

6.2 Training and Evaluation

Training minimizes L1 loss between rendered images and the corresponding ground-truth images. Gradient checkpointing is used for the Gaussian batches to reduce the memory required for backward propagation.

Setting / metricValue
Training iterations1000
Gaussians per splat128
Held-out PSNR (dB)25.881
Held-out SSIM0.882
Add result: data/part_6_training_progress.gif
Training progress: predicted images on the top row and ground-truth images on the bottom row.
Add result: data/part_6_training_final_renders.gif
Final novel-view renderings of the trained truck representation.

7.1 Rendering Using Spherical Harmonics

The pretrained spherical harmonic coefficients are evaluated using normalized camera-to-Gaussian directions. Degree 0–3 basis functions produce view-dependent RGB colors; geometry and opacity are unchanged.

Add result: data/part_5_render.gif
View-independent color (Part 5.5).
Add result: data/part_7_1_render.gif
View-dependent color with spherical harmonics.

Same-View Comparison 1

Add result: data/part_7_1_view1_dc.png
View 1: DC component only (RGB crop).
Add result: data/part_7_1_view1_sh.png
View 1: full spherical harmonics (same camera, RGB crop).

Same-View Comparison 2

Add result: data/part_7_1_view2_dc.png
View 2: DC component only (RGB crop).
Add result: data/part_7_1_view2_sh.png
View 2: full spherical harmonics (same camera, RGB crop).

Same-View Comparison 3

Add result: data/part_7_1_view3_dc.png
View 3: DC component only (RGB crop).
Add result: data/part_7_1_view3_sh.png
View 3: full spherical harmonics (same camera, RGB crop).

The rendering with spherical harmonics shows slightly more natural shading, particularly in the darker regions of the chair. Compared with the DC-only rendering, it captures more subtle variations in appearance across viewing directions, although the overall difference is modest. This improvement comes from view-dependent color modeling rather than changes to geometry or explicit shadow computation.