Learning for 3D Vision · Fall 2026

Assignment 3: Volume Rendering,
NeRF & Gaussian Splatting

Name: Sumanth Yegurla · Andrew ID: syegurla

Completed experiments. This report includes volume rendering, NeRF, VolSDF, Gaussian splatting, and all three additional experiments: view dependence, coarse/fine sampling, and a harder-scene Gaussian comparison. Results and limitations are reported below.

Setup. The full training runs used a Tesla V100-SXM2-32GB on PSC Bridges-2, PyTorch 2.4.1, PyTorch3D, and NumPy 1.26.4. I first ran small execution checks, then trained NeRF and VolSDF for 250 epochs each and the truck Gaussians for 1,000 iterations. Follow-ups include 500-epoch Lego NeRF at 256 × 256, a four-setting VolSDF sweep, and 3,000-iteration Materials Gaussian comparisons.

Output provenance. Images in nerf/images/ are my generated results. Instructor examples are labeled explicitly. The original NeRF and VolSDF GIFs were last rendered at epoch index 240 (of 0–249); their original checkpoints were saved at epoch 200. The bonus implementation now saves a final-epoch checkpoint and render.

A. Neural volume rendering

1. Differentiable Volume Rendering

1.1–1.2 Pipeline

A camera defines each ray’s origin and direction. The sampler places 3D points along the ray, the implicit function returns density and color at these points, and the renderer integrates their contributions into an image. Keeping these operations differentiable allows image-space error to update scene parameters.

1.3 Ray sampling

I generate a pixel grid in [−1, 1], unproject points at depth 1 from NDC into world coordinates, and set every ray origin to the camera center. Each direction is the normalized vector from the camera center to its unprojected point.

Normalized image-plane pixel grid
Pixel grid in normalized coordinates.
Visualization of camera ray directions
Camera rays.

1.4 Point sampling

For each ray, I use 64 evenly spaced distances between 0 and 5 and compute x(t) = o + t d. The resulting positions and distances are stored in the ray bundle. The visualization uses a regular subset of rays for readability.

Point samples along a subset of camera rays
Sampled 3D points.

1.5 Volume rendering and depth

I compute alpha from density and segment length, use an exclusive cumulative product for transmittance, and sum the weighted colors. The same weights produce depth by replacing color with the distance along the ray.

αi = 1 − exp(−σi Δti)   ·   Ti = ∏j<i(1 − αj)   ·   wi = Ti αi
C = ∑i wi ci   ·   D = ∑i wi ti
Twenty-view rendering of the analytic colored box
Analytic box: 20 views, 256 × 256.
Box depth map normalized by its maximum depth
Depth, normalized by its maximum value.

2. Optimizing a Basic Implicit Volume

2.1 Random ray sampling

I randomly select 4,096 pixel coordinates without replacement per training image. This reduces the memory needed to optimize the volume compared with rendering all 256 × 256 pixels.

2.2 Loss and training

I optimize the box center and side lengths using mean squared RGB error and Adam at a learning rate of 0.0005. Training uses four posed images over 1,000 outer epochs, with one update per image. The final displayed loss was approximately 0.000001.

Recovered parameters, rounded to two decimal places
ParameterInitial valueOptimized value
Center(0.00, 0.00, 0.00)(0.25, 0.25, 0.00)
Side lengths(1.50, 1.50, 1.50)(2.01, 1.50, 1.50)

The logged unrounded first side length was 2.005124; its two-decimal value is 2.01.

2.3 Before / after and orbit visualization

Box before optimization, training camera zero
Before — camera 0
Box after optimization, training camera zero
After — camera 0
Box before optimization, training camera one
Before — camera 1
Box after optimization, training camera one
After — camera 1
Box before optimization, training camera two
Before — camera 2
Box after optimization, training camera two
After — camera 2
Box before optimization, training camera three
Before — camera 3
Box after optimization, training camera three
After — camera 3
Orbit of my optimized box
My optimized box.
Instructor reference orbit of the optimized box
Instructor reference, supplied with the assignment.

The fitted volume changes from a centered cube to an offset rectangular box. The recovered parameters are close to a center of (0.25, 0.25, 0) and dimensions (2, 1.5, 1.5). The reference is included for comparison of overall shape, placement, and color pattern.

3. Optimizing a Neural Radiance Field

The base network maps position to density and RGB, using six-frequency positional encoding, a six-layer MLP of width 128 with an input skip at layer index 3, ReLU for density, and sigmoid for color. The base model does not use viewing direction.

For Lego, I use 1,024 rays per batch, 128 samples per ray over distances 2–6, Adam with initial learning rate 0.0005, and 500 epochs at 256 × 256 resolution for the final run. The learning rate is multiplied by 0.8epoch/100. The training objective is RGB MSE. This run starts from scratch, uses rendering chunks of 4,096 rays, and saves orbit renders every 50 epochs and at the final epoch.

Original 128-pixel Lego NeRF orbit
Original: 128 × 128, 250 training epochs; orbit saved at epoch index 240.
Final 256-pixel Lego NeRF orbit after 500 epochs
Final: native 256 × 256, 500 epochs; orbit saved at epoch index 499.

The final model recovers the yellow body, dark tracks, raised arm, and supporting platform. At matched display size, the platform texture and outlines of the arm and tracks are more clearly resolved than in the original run, although some thin structures remain imperfect. Both training resolution and training duration changed, so this is not an isolated resolution ablation.

The final validation scores are 24.981 dB PSNR and 0.8681 SSIM. The measured training-plus-render time was 1,032.95 seconds; the complete run took 18 minutes 26 seconds, including setup and evaluation. Raw Lego validation metrics.

4.1 View Dependence

Implemented method. Density depends only on position. A separate RGB head combines the position-network features with a two-frequency encoding of normalized viewing direction. This allows appearance to change between views without making geometry view-dependent.

The comparison uses the Materials scene for both models, 250 epochs, 128 × 128 images, 1,024 rays per batch, 128 samples per ray over distances 1–7, and initial learning rate 0.0001. The view-dependent color head has width 64. Both runs use seed 1.

Materials scene reconstructed by position-only NeRF
Position-only model.
Materials scene reconstructed by view-dependent NeRF
Position + viewing direction.
Materials validation comparison — measured results from metrics.json
MethodPSNR ↑SSIM ↑Training + render time
Position only23.259 dB0.8436488.98 s
View-dependent color24.043 dB0.8620569.71 s

Results and generalization trade-off

Direction conditioning improves validation PSNR by 0.784 dB and SSIM by 0.0185, with approximately 16.5% more measured training-and-render time. The orbit retains the same arrangement of objects in both models; the differences are mainly in their surface brightness and color rather than a large change in silhouette. The view-dependent result has more distinct surface shading on several objects, although the 128 × 128 output limits fine-detail inspection.

Direction conditioning can represent moving highlights and other non-Lambertian appearance, which a position-only model must average across views. The additional flexibility can also fit view-specific training effects that fail to generalize to unseen cameras. Keeping density independent of direction limits one way the model could explain away geometric errors. The color-head architecture also changes, so this is a practical model comparison rather than a parameter-count-matched ablation.

The held-out validation metrics favor the direction-conditioned model in this run. They do not establish that extra view dependence always generalizes better: this comparison uses one seed and one scene. No repeated-run uncertainty estimate was measured.

4.2 Coarse/Fine Sampling

Implemented method. The coarse network evaluates 128 uniformly spaced samples. Its detached rendering weights define a piecewise-constant probability density over depth. Inverse-CDF sampling adds 128 samples, which are merged and sorted with the original positions. A separately parameterized fine network evaluates all 256 positions.

I minimize the sum of coarse and fine RGB MSE. Training uses random PDF samples, while evaluation uses deterministic quantiles. Both networks use the view-dependent architecture from 4.1. The single-network 4.1 result is the baseline, keeping dataset, epochs, and ray batch size fixed.

Single-network view-dependent Materials NeRF baseline
Same baseline as the view-dependent result in 4.1.
Materials scene reconstructed with coarse and fine NeRF networks
Importance sampling guided by coarse opacity weights.
Sampling comparison — validation metrics and measured run time
MethodNetwork evaluations/rayPSNR ↑SSIM ↑Time
Uniform, one network12824.043 dB0.8620569.71 s
Hierarchical, two networks128 + 25624.118 dB0.86511,573.02 s

Speed / quality trade-off

The fine pass concentrates samples near depths that contribute to the coarse image. This can improve the integration of thin or localized surfaces. The chosen configuration costs more: it evaluates 384 points across two networks instead of 128 in one, adds PDF sampling and sorting, and stores activations for both networks during training. Benefits therefore need to be evaluated against actual runtime and memory use. A poor coarse distribution can also undersample a missed surface; retaining the original uniform samples provides some coverage.

Measured outcome. Coarse/fine sampling adds 0.075 dB PSNR and 0.0030 SSIM over the single-network view-dependent model, while taking 2.76× as long in the recorded training-and-render interval. The matched orbit frames look very similar; this is a small refinement rather than a dramatic visual improvement. For this scene and budget, the single-network view-dependent model offers the better time/quality trade-off. The hierarchical model has the highest measured quality of the three NeRF variants, but its small advantage has not been tested across multiple seeds. No peak-memory benchmark was recorded.

The timings above come from the same timer in each NeRF run and include periodic rendering, but exclude final validation. They are distinct from total elapsed times.

4.3 VolSDF

I predict signed distance with a six-layer, width-128 MLP and color with a two-layer, width-128 MLP, both using six-frequency positional encoding. Density is obtained from the Laplace CDF of the negated signed distance. RGB MSE is combined with an Eikonal penalty of weight 0.02.

Training uses 250 epochs, 1,024 rays per batch, 128 samples per ray over distances 2–6, 128 × 128 images, and initial learning rate 0.0005. The learning-rate multiplier is 0.8epoch/50. Four fresh runs use seed 1, identical budgets, rendering chunks of 4,096 rays, and final-epoch outputs. The selected color-reconstruction setting is α = 50, β = 0.02.

σ(d) = α · { ½ exp(−d/β), if d > 0;   1 − ½ exp(d/β), if d ≤ 0 }
L = MSE(C, Cgt) + 0.02 · E[(‖∇d‖2 − 1)²]
Selected VolSDF color reconstruction
Selected color rendering: α=50, β=0.02.
Selected VolSDF geometry with remaining extraneous surfaces
Corresponding zero-level geometry; artifacts remain.
Controlled VolSDF sweep — validation split, 250 epochs per run
αβPSNR ↑SSIM ↑Training + render timeEvidence
100.0221.1180.6713546.08 sMetrics
100.0521.2160.7675563.34 sMetrics
100.120.7300.7342507.27 sMetrics
500.0223.1420.8194503.04 sMetrics

At fixed α=10, simply reducing β does not improve both metrics: β=0.02 is worse than β=0.05, especially in SSIM. At fixed β=0.02, raising α to 50 improves PSNR by 2.024 dB and SSIM by 0.1482. The higher interior density makes the volume more opaque over a shorter distance, while the narrow transition can localize appearance near the boundary. The selected model exceeds the default α=10, β=0.05 by 1.926 dB and 0.0519 SSIM.

The selected color views have more distinct tracks and platform texture. However, all four extracted meshes contain extensive extraneous surfaces and grid-like artifacts. I select α=50, β=0.02 for its validation color scores, not as evidence of an accurate recovered surface. These are single-seed runs selected on validation, not an independent test-set comparison.

Geometry across all four settings

VolSDF geometry alpha 10 beta 0.02
α=10, β=0.02
VolSDF geometry alpha 10 beta 0.05
α=10, β=0.05
VolSDF geometry alpha 10 beta 0.1
α=10, β=0.1
VolSDF geometry alpha 50 beta 0.02
α=50, β=0.02
Selected VolSDF prediction and ground truth on validation view zero
Held-out view 0: prediction left, ground truth right.

What do α and β control?

α controls the density scale. Increasing it makes the interior more opaque over a shorter ray segment. β controls the transition width around the surface. With the convention d < 0 inside, density approaches α inside and zero outside; at the surface it is α/2.

How does high versus low β bias the learned SDF?

A large β spreads opacity across a wider band around the zero-level set, allowing a diffuse volume to explain the image and weakening localization of a sharp boundary. A small β concentrates the transition near the surface, favoring sharper geometry, but making the image more sensitive to surface position and to whether the sampling captures that narrow transition.

Which is easier to train?

A larger β is generally easier early in training because more samples lie in the transition region and receive useful gradients. With a very small β, many samples are in nearly saturated regions, and coarse ray sampling may miss the informative band.

Which is more likely to yield an accurate surface?

A smaller β can localize the surface more precisely once the geometry is reasonably initialized and sampling is adequate. It is not automatically better: optimization instability or insufficient sampling can outweigh the benefit. A broad-to-narrow β schedule is a possible follow-up, but was not used in this run.

B. 3D Gaussian splatting

5. 3D Gaussian Rasterization

5.1–5.2 Projection and Gaussian evaluation

For anisotropic Gaussians, I construct the world covariance from rotation and scale as Σ = R S ST RT; the isotropic case reduces to s²I. The camera transform and projection Jacobian map the covariance to the image plane. I project means into screen coordinates and evaluate the quadratic exponent −½(x − μ)TΣ−1(x − μ). The supplied unit tests passed 4/4.

5.3–5.5 Sorting, compositing, and rendering

I filter out Gaussians behind the camera, sort the remaining Gaussians by increasing depth, and compute alpha as opacity times the projected Gaussian value. Exclusive transmittance weights blend color, depth, and silhouette. The final 256 × 256 renders use batches of 256 Gaussians, carrying the remaining transmittance into the next batch.

Pretrained chair Gaussians rendered as RGB, depth, and silhouette
View-independent Gaussian rendering: RGB, depth, and mask.

Native-resolution final renders: the chair was re-rendered at 256 × 256 for both DC-only and spherical-harmonic colors, using the supplied pretrained Gaussians without retraining. Each RGB/depth/mask panel now contains a native 256 × 256 render; these replace the earlier 64 × 64 previews. Both 32-view render passes together took 43 minutes 48 seconds.

Covariance-convention diagnostic

The default implementation uses W=R in J W Σ Wᵀ Jᵀ and passes the four supplied tests. PyTorch3D transforms row-vector points as xview=xworldR+T; the corresponding column-vector covariance rotation is therefore Rᵀ. I added an opt-in --transpose_view_rotation path and checked its covariance against an independently differentiated projection Jacobian. The default remains unchanged for test compatibility. This is a separate diagnostic, not an undocumented replacement. PyTorch3D camera convention.

Frame eight with default covariance rotation and SH
Frame 8, SH: W=R.
Frame eight with transposed covariance rotation and SH
Same frame and SH: W=Rᵀ.

The tilted view shows substantially cleaner legs and carved edges with the transposed rotation; the stretched streaks in the default rendering largely disappear. Frontal views change much less. No Gaussian parameters were retrained. These observations support the convention diagnosis, but do not replace the assignment's supplied-test results.

Full diagnostic chair orbits: transposed rotation

Transposed-covariance chair orbit with DC colors
W=Rᵀ, DC colors.
Transposed-covariance chair orbit with spherical harmonics
W=Rᵀ, spherical harmonics.

6. Training 3D Gaussian Representations

I initialize isotropic Gaussians from the provided truck point cloud and optimize their means, scales, colors, and opacities using L1 image loss. Quaternion optimization is unnecessary for isotropic Gaussians. Training uses Adam for 1,000 iterations, a black background, and 2,048 Gaussians per splatting batch.

Learning rates used in the completed truck run
ParameterLearning rate
Means0.00016
Scales0.005
Colors0.0025
Opacities0.05
Truck evaluation on five held-out views
RunIterationsPSNR ↑SSIM ↑
Execution check217.1060.792
Full training1,00029.0870.936

The completed full run took 21 minutes 23 seconds, including rendering all 197 training views and evaluation. It improved both held-out metrics over the two-iteration execution check. The last displayed training loss was 0.010.

Truck training progress with predicted views above ground truth
Top: predicted views. Bottom: ground-truth views.
Final trained truck Gaussian renders
Final renders after 1,000 training iterations.

7.1 Rendering Using Spherical Harmonics

I evaluate the supplied degree-3 spherical harmonics using the normalized direction from the camera center to each Gaussian. All 16 coefficients per color channel contribute to a direction-dependent color, compared with the constant DC color used in section 5. Both runs render the same 32 camera views from the same pretrained chair representation.

Chair render using only DC colors
Section 5 baseline.
Chair render using spherical harmonic colors
View-dependent colors.

Matched RGB comparisons

The slots below show only the left RGB panel of each saved RGB/depth/mask composite. The view index is matched between methods.

DC-only RGB chair, view zero
View 0 — DC only
Spherical harmonic RGB chair, view zero
View 0 — spherical harmonics

View 0. The DC-only backrest is relatively bright and evenly green. SH darkens its side regions and makes the lighter central upholstery pattern stand out more clearly. The silhouette stays the same because only the color evaluation changes.

DC-only RGB chair, view eight
View 8 — DC only
Spherical harmonic RGB chair, view eight
View 8 — spherical harmonics

View 8. In this rotated view, the SH image has a darker seat and stronger contrast between its pale center and green edge. The upright upholstered panel also has stronger directional shading. The geometry and camera are identical, so these changes come from the direction-conditioned appearance. Visible streaks around the legs and thin structures remain in this rotated view; higher resolution does not remove those artifacts.

DC-only RGB chair, view sixteen
View 16 — DC only
Spherical harmonic RGB chair, view sixteen
View 16 — spherical harmonics

View 16. Near the returned frontal orientation, the SH backrest again shows darker green regions around a brighter central pattern; the armrests remain bright in both renderings. SH changes appearance rather than repairing geometry or silhouette artifacts. These observations show a shading difference, but without matched ground truth I cannot claim that every SH pixel is more accurate.

7.2 Training on a Harder Scene

Final comparison. All new runs use 10,000 randomly initialized Gaussians, Materials at 128 × 128, seed 1, L1 loss, 2,048 Gaussians per splat, and 3,000 iterations. The baseline retains the truck optimizer and isotropic representation; the original 1,000-iteration Q6-matched baseline is included in the main table and displayed below. The 3,000-iteration baseline controls for the longer training budget of the improved runs.

Selected method: scene-covering initialization, anisotropic Gaussians, revised color/mean learning rates, exponential mean learning-rate decay, and transposed view rotation for covariance projection. No SSIM loss or densification is used.

Materials Gaussian validation — required baseline and equal-budget follow-ups
MethodIterationsPSNR ↑SSIM ↑Total measured timeEvidence
Required Q6-matched isotropic baseline1,00018.0020.66001,283.40 sMetrics
Isotropic baseline, longer training3,00018.6670.69053,485.00 sMetrics
Improved recipe, W=R3,00022.8800.82063,486.98 sMetrics
Improved recipe, W=Rᵀ (selected)3,00023.1180.83573,478.94 sMetrics

Training progress and final renders

Three-thousand-iteration baseline training progress
Baseline: predictions above ground truth.
Selected improved Materials training progress
Selected method: same visualization cameras and training budget.
Final baseline Materials renders
Baseline, 3,000 iterations.
Improved Materials renders using original rotation convention
Improved recipe, W=R.
Selected improved Materials renders using transposed rotation
Selected recipe, W=Rᵀ.

Modifications relative to the truck recipe

Measured outcome. The selected method improves over the equal-length baseline by 4.452 dB PSNR and 0.1451 SSIM. The improved recipe already gains 4.213 dB with W=R; changing only the rotation convention adds a further 0.238 dB and 0.0151 SSIM. Longer baseline training alone raises PSNR from 18.002 to 18.667, much less than the recipe change.

The baseline leaves diffuse outer blobs and broken central objects. The improved models recover separated objects across the scene; the transposed version reduces jagged outlines. Fine reflections and some surfaces remain soft. These are single-seed validation results, and the initialization/learning-rate/representation changes were bundled, so their individual effects cannot be separated. The measured times include visualization and evaluation; small timing differences should not be treated as speed improvements.

Materials baseline validation prediction beside ground truth
Baseline, held-out view 0: prediction / ground truth.
Selected Materials validation prediction beside ground truth
Selected method, same held-out view: prediction / ground truth.

Required 1,000-iteration baseline GIFs and initial anisotropy-only experiment

Required baseline. The Materials scene uses 10,000 randomly initialized isotropic Gaussians, the truck learning rates, L1 loss, and 1,000 iterations. The means are sampled from a centered Gaussian distribution with standard deviation 0.2, rather than a reconstructed point cloud. Training images are 128 × 128.

Candidate improvement. The second run uses anisotropic Gaussians with three scale components and trainable quaternion rotations (learning rate 0.001). Dataset, random seed 1, loss, other learning rates, 1,000-iteration budget, and splat batch size 2,048 are held fixed. No densification, SSIM loss, or learning-rate schedule is used in this comparison.

Anisotropic Gaussians can stretch and rotate to fit surfaces with fewer primitives than spheres, but introduce additional parameters and optimization difficulty. Random initialization also makes this problem harder than the truck task: many Gaussians must move before they can explain scene content. The completed comparison shows a small PSNR gain but no consistent improvement across both quality metrics.

Materials isotropic Gaussian baseline training progress
Predicted views above ground truth.
Materials anisotropic Gaussian training progress
Matched visualization cameras.
Final Materials isotropic Gaussian renders
Baseline after 1,000 iterations.
Final Materials anisotropic Gaussian renders
Candidate improvement after 1,000 iterations.
Materials Gaussian comparison — validation split
MethodIterationsPSNR ↑SSIM ↑Total runtime
Isotropic baseline1,00018.002 dB0.66001,283.40 s
Anisotropic candidate1,00018.031 dB0.65761,240.14 s

Results and limitations

Anisotropy improves PSNR by only 0.029 dB, while SSIM decreases by 0.0024. It is the best of these two runs by PSNR, but the isotropic baseline is better by SSIM; neither dominates. The measured total runtime is about 3.4% lower for the anisotropic run, but this single-run difference is not enough to establish a speed advantage.

The final views expose the main failure: both models have blurry outer blobs and a noisy, incomplete cluster near the center rather than clean, separated objects. The anisotropic run adds elongated streaks in some views without resolving the overall reconstruction. Increasing the representation’s flexibility alone was insufficient under the same 1,000-iteration budget and tightly concentrated random initialization.

Modification summary. Relative to the baseline, the PSNR-best run replaces one isotropic scale with three trainable scales and enables quaternion optimization at learning rate 0.001. All other settings remained fixed in this initial experiment. Its failure motivated the wider initialization, learning-rate changes, and longer controlled runs presented above. Density control was not used in either phase.

Both rows use the Materials validation split. Their timer includes final visualization and evaluation; it differs from the NeRF training-and-render timer. The required baseline metrics are linked in the main comparison table.