Learning for 3D Vision · Assignment 3

Volume rendering meets Gaussian splatting.

An implementation study spanning world-space ray construction, differentiable emission–absorption rendering, learned neural fields, signed-distance surfaces, and a vectorized 3D Gaussian rasterizer.

7 tasks completed 3 NeRF bonus extensions 2 Gaussian extensions PyTorch · PyTorch3D
Leo XuResults & implementation report
Overview

Two representations, one rendering story.

The first half builds an emission–absorption volume renderer and uses it for inverse rendering, NeRF, hierarchical sampling, and VolSDF. The second half implements the projection, sorting, compositing, training, and view-dependent appearance components of 3D Gaussian splatting.

Part A · Tasks 1–4

Implicit fields

Scene properties are queried continuously at sampled 3D positions. Differentiability connects pixel loss back to geometry, density, color, and neural-field parameters.

Part B · Tasks 5–7

Explicit Gaussians

A finite collection of oriented 3D Gaussians is projected to screen space and alpha-composited, providing an explicit and efficiently optimizable representation.

Task 01

Differentiable volume rendering

I implemented the full path from camera pixels to world-space sample points, followed by emission–absorption compositing of color and depth.

1.3 · Ray sampling

Pixels into world-space rays

get_pixels_from_image creates an NDC grid in [−1, 1]. get_rays_from_pixels unprojects each point through the camera, subtracts the camera center, and normalizes the resulting direction. The common camera origin is expanded to match the ray batch.

Normalized device coordinate pixel grid
NDC pixel grid: uniform coverage over the image plane.
World-space camera rays
World-space rays: origins coincide at the camera and directions fan through the image.
1.4 · Point sampling

Stratified samples along each ray

The raysampler linearly spaces depths between the configured near and far planes. Each point is computed as origin + depth × direction; the point and its depth remain paired inside the ray bundle.

Points sampled along camera rays
Samples occupy regular depth intervals along every ray frustum.
1.5 · Volume rendering

Front-to-back emission–absorption

αᵢ = 1 − exp(−σᵢ Δtᵢ)    Tᵢ = ∏ⱼ<ᵢ (1 − αⱼ)    wᵢ = Tᵢ αᵢ
C(r) = Σᵢ wᵢ cᵢ    D(r) = Σᵢ wᵢ tᵢ

I compute exclusive transmittance with a shifted cumulative product, then use the same weights to aggregate emitted RGB and sample depths. Chunked evaluation controls memory without changing the rendering equation.

Spiral volume rendering of a box
Rendered RGB spiral around the analytic SDF box.
Normalized depth rendering
Normalized depth: nearer visible faces appear darker and farther faces brighter.
Task 02

Optimizing an implicit box

Random ray minibatches make inverse rendering tractable. Image-space supervision recovers the box center and side lengths without direct 3D labels.

2.1 · Random rays

Memory-aware sampling

I uniformly sample pixel indices without replacement, convert those selected pixels to NDC coordinates, generate their camera rays, and sample matching ground-truth colors. This reduces each optimization step from a full image to 4,096 informative rays.

2.2 · Objective

Image-space MSE

The predicted RGB for each sampled ray is compared against the corresponding ground-truth pixel with mean squared error. Gradients pass through volume weights and the box SDF to its trainable center and dimensions.

1,000epochs
4,096rays / step
(0.25, 0.25, −0.00)learned center
(2.01, 1.50, 1.51)side lengths

Before optimization

Box view before training 0
Box view before training 1
Box view before training 2
Box view before training 3

After optimization

Box view after training 0
Box view after training 1
Box view after training 2
Box view after training 3
Spiral rendering of optimized box
Novel-view spiral of the recovered box. The rounded values closely match the hidden scene geometry.
Task 03

Optimizing a NeRF

The analytic volume is replaced with an MLP that learns density and color as continuous functions of 3D position.

Architecture

Position-encoded neural field

Six-frequency harmonic encoding maps each 3D sample to a high-dimensional basis. A six-layer, 128-unit MLP predicts non-negative density through ReLU and RGB through sigmoid. A skip connection reintroduces the encoded position at layer three to preserve high-frequency information.

Training

Multi-view image supervision

The Lego NeRF trains for 250 epochs at 128² resolution with 1,024 rays per batch, 128 samples per ray, Adam at 5×10⁻⁴, and step decay by 0.8 every 50 epochs. The objective is RGB MSE.

NeRF spiral rendering of Lego bulldozer
The learned radiance field reconstructs recognizable bulldozer geometry, color, and consistent novel views.
Task 04

NeRF extras

All three optional extensions were implemented: view-dependent emission, hierarchical coarse/fine sampling, and a learned signed-distance surface with volumetric rendering.

4.1 · View dependence

Separating geometry from directional appearance

Density and a learned feature vector come from position alone. A color head combines that feature with a harmonic encoding of the normalized ray direction, allowing specular and direction-dependent appearance without letting viewing direction alter geometry.

Materials NeRF without view dependence
Position-only baseline on the materials scene.
Materials NeRF with view dependence
View-dependent model using a separate directional color branch.
Trade-off. Directional conditioning improves highlights and appearance changes across views, but adds parameters and can encode training-camera-specific effects. Keeping density independent of direction limits geometric overfitting; a smaller direction branch and lower learning rate further improve generalization.
4.2 · Coarse / fine sampling

Spend samples where matter exists

A 64-sample coarse pass estimates ray weights. Inverse-CDF sampling then draws 128 additional depths from this distribution, and the merged depths are sorted for a fine rendering. Compared with uniform 128-sample rendering, this concentrates computation near occupied regions and object boundaries.

Hierarchical coarse and fine NeRF rendering
Hierarchical Lego rendering after 250 epochs.
Trade-off. Importance sampling generally sharpens geometry for a fixed sampling budget, but two network evaluations and PDF sampling increase training time and implementation complexity. Here it uses 64 coarse + 128 fine samples per ray.
4.3 · VolSDF

A neural surface rendered as density

A six-layer distance MLP predicts signed distance, while a two-layer color head predicts RGB. The Eikonal term encourages unit gradient magnitude, making the learned scalar field behave like an SDF. I used α = 10, β = 0.05, an Eikonal weight of 0.02, 128 samples per ray, and 250 epochs.

VolSDF color rendering
Color predicted by the neural surface model.
VolSDF geometry rendering
Geometry visualization from the learned signed-distance field.
density(s) = α · LaplaceCDF(−s / β)
  • α controls density scale. Larger α makes occupied regions more opaque and strengthens termination near the surface.
  • β controls boundary softness. High β spreads density over a wide band, yielding smoother gradients but a thicker, biased surface. Low β approaches a sharp step around zero level-set.
  • Training versus accuracy. High β is easier early in training because gradients act over a wider region. Low β can recover a more precise surface, but optimization becomes more sensitive because useful gradients are concentrated near the surface.
Task 05

3D Gaussian rasterization

I implemented a vectorized rasterization pipeline that projects oriented 3D Gaussians, evaluates them on pixels, sorts them by depth, and composites color, depth, and silhouettes.

5.1–5.2

Projection & evaluation

Quaternion rotation and axis scales form each 3D covariance. The camera Jacobian maps it into a stabilized 2D covariance; projected means locate the splats. Pixel power is the vectorized Mahalanobis term −½(x−μ)ᵀΣ⁻¹(x−μ).

5.3–5.4

Ordering & visibility

Gaussians behind the camera are filtered, and remaining splats are sorted front-to-back by camera-space depth. Opacity times the Gaussian response gives α; an exclusive cumulative product gives transmittance.

5.5

Splatting

Color, depth, and mask use the common weight Tα. Mini-batches carry final transmittance forward, preserving global compositing order. Background contributes only through remaining transmittance (1−mask).

All four provided Gaussian unit tests pass. The final animation displays RGB, colored depth, and silhouette from left to right.
Chair rendered with 3D Gaussian splatting, depth, and mask
Pretrained chair represented by 283,873 anisotropic Gaussians.
Task 06

Training a Gaussian scene

Starting from a supplied point cloud, I optimized 10,000 isotropic Gaussians to reproduce multi-view images of a toy truck.

1,000iterations
29.152held-out PSNR
0.940held-out SSIM
L1training loss
Parameter groupLearning rateReasoning
Opacity logits0.05Fast visibility adjustment.
Log scales0.005Controlled footprint changes.
Colors0.0025Stable appearance fitting.
Means0.00016Conservative geometric motion.
Gaussian truck training progress
Training progression: predictions above, corresponding ground truth below.
Final Gaussian truck renders
Final renderings across the full training-view orbit.

The model learns opacity, scale, color, and means with Adam. Rotations are fixed because rotating an isotropic Gaussian leaves its covariance unchanged. The high held-out scores and stable orbit indicate strong view consistency.

Task 07

Gaussian extensions

Both extensions were completed: degree-three spherical-harmonic rendering and harder-scene training from random initialization.

7.1 · Spherical harmonics

View-dependent Gaussian appearance

I evaluate all 16 real SH basis terms through degree three for each RGB channel. Directions are normalized world-space vectors from the camera center to each Gaussian. The original degree-zero path remains available for a controlled comparison.

Chair using only degree-zero color
Task 5: fixed DC color per Gaussian.
Chair using degree-three spherical harmonic color
Task 7.1: view-dependent degree-3 SH color.
DC and SH comparison at frame 0
Front view: SH strengthens contrast in the green fabric and preserves richer patterned detail.
DC and SH comparison at frame 8
Side/top view: directional color changes are clearest on the wood trim and textured upholstery.
DC and SH comparison at frame 16
Rear view: geometry and silhouette agree, while SH changes local brightness and material response.
The geometry, camera, opacity, and covariance are identical in each pair. Therefore the observed changes—directional highlights, contrast, and local color—come specifically from higher-order SH coefficients.
7.2 · Harder scene

Random initialization on Materials

The baseline transfers the Task 6 recipe directly: 10,000 random isotropic Gaussians, L1 loss, and the same learning rates. The improved model targets the failure modes visible in the baseline—poor initial scene coverage, opaque blobs, limited shape flexibility, and blurred structural detail.

MethodIterationsRepresentationLossPSNRSSIM
Task 6 baseline1,00010k isotropicL118.226 dB0.651
Improved3,00010k anisotropic0.8 L1 + 0.2 DSSIM22.106 dB0.824
+3.880 dBPSNR gain
+0.173SSIM gain
3×training duration
10× decayfinal learning rates

Improved learning-rate schedule

Each parameter group uses its own initial learning rate and decays exponentially over 3,000 iterations to one tenth of that value.

Parameter groupInitial learning rateFinal learning rate
Opacity logits0.0250.0025
Log scales0.0030.0003
Colors0.010.001
Means0.0010.0001
Quaternions0.0010.0001
Baseline harder scene training progress
Baseline progress: the scene layout begins to emerge, but large diffuse blobs persist.
Improved harder scene training progress
Improved progress: individual objects become sharper and spatially separated.
Baseline harder scene final renderings
Baseline final orbit: recognizable colors, but weak boundaries and floaters.
Improved harder scene final renderings
Improved final orbit: stronger object structure, cleaner separation, and better consistency.
Initialization

Better early gradients

I spread random means over a region three times wider, initialized opacity at 0.1 instead of near one, and set initial scale to 0.03. Wider coverage reduces the chance that scene regions receive no splats; lower opacity prevents early occlusion from saturated blobs.

Representation

Anisotropic shape

Three independent scales plus trainable quaternions let Gaussians elongate and rotate along object surfaces. This represents boundaries with fewer primitives than spheres and reduces the soft, circular artifacts visible in the baseline.

Objective

L1 + structural similarity

DSSIM adds local luminance, contrast, and structural supervision to per-pixel L1. This directly targets the blurred organization of the baseline and aligns with the large SSIM improvement.

Optimization

Longer, scheduled training

The improved run uses 3,000 iterations, faster initial color/position learning, quaternion optimization, and exponential decay to one tenth of each initial rate. Large early steps organize the scene; smaller late steps refine it.