Assignment 03 · Learning for 3D Vision

Learning
to render.

From continuous neural fields to explicit Gaussian primitives. A study of differentiable rendering, appearance, and geometry.

03Scene representations
28.444 dBTruck held-out PSNR
+2.172 dBMaterials validation gain
01

Differentiable volume rendering

From camera pixels to an emission–absorption image.

I generated normalized pixel coordinates, unprojected them into world space, and normalized directions from the camera center. Uniform distances produce sample points along each ray.

p(t) = o + td   ·   αᵢ = 1 − exp(−σᵢΔtᵢ)   ·   wᵢ = Tᵢαᵢ

Transmittance uses an exclusive accumulated optical depth, so the first sample has T = 1. Weighted sums produce RGB and depth; depth is divided by its maximum for display.

Pixel grid
Pixel gridNormalized image coordinates.
Camera rays
Camera raysAbsolute components of unit ray directions.
Point samples
Point samplesUniform samples in the camera viewing volume.
Rendered box
Rendered boxSpiral sequence from the completed renderer.
Normalized depth
Normalized depthViridis maps the normalized weighted depth.
02

Recovering a basic volume

Learning box geometry from posed RGB images.

I randomly selected pixels without replacement and minimized RGB mean squared error to optimize the box center and side lengths. Training ran for 1,000 epochs; the last displayed loss was approximately 10⁻⁶.

Parameterxyz
Center0.250.250.00
Side lengths2.011.501.50
Optimized box
Optimized boxFinal spiral rendering.
Before optimization
Before optimizationOne of the input camera views before fitting.
03

Neural radiance field

Replacing an analytic box with a learned scene function.

A position-only MLP maps harmonic encodings of 3D samples to density and RGB. Six harmonic frequencies give 39 input features. The six-layer position network uses 128 hidden units and a skip connection at layer index 3. ReLU ensures nonnegative density; Sigmoid bounds RGB.

128 × 128 images250 epochs1,024 rays / step128 samples / rayAdam · 5 × 10⁻⁴
Lego · position-only NeRF
Lego · position-only NeRFTraining uses RGB MSE with learning-rate decay.
04

Beyond a basic NeRF

View-dependent appearance, hierarchical sampling, and signed-distance geometry.

4.1 · View dependence

I concatenate a low-frequency direction encoding with position features only in the shallow color head. Density remains independent of viewing direction. This allows highlights and reflections to change with the camera without making geometry direction-dependent.

Additional directional capacity can fit reflective appearance, but can also memorize training views. A small direction encoding and shallow head limit this flexibility. The materials sequence below demonstrates view-dependent rendering; without a matched held-out ablation, it does not establish a quantitative generalization gain.

Materials · 128 × 128250 epochsDirection frequencies: 2Learning rate: 10⁻⁴
Materials · view-dependent NeRF
Materials · view-dependent NeRF

4.2 · Coarse / fine sampling

Two independent networks are supervised by the sum of their RGB MSE losses. The coarse network evaluates 64 samples. Its detached interior weights define a PDF for 64 additional samples; the fine network evaluates the sorted union of 128 points. Training jitters coarse distances; evaluation uses deterministic samples.

Importance sampling concentrates evaluations near contributing surfaces. The trade-off is a second network and 192 total network sample evaluations per ray, compared with 128 for the original single network. It can improve sample placement, but speed and quality gains require measurement rather than being guaranteed.

Uniform baseline
Uniform baseline
Hierarchical sampling
Hierarchical sampling

4.3 · VolSDF

Separate encoded MLPs predict signed distance and color. Signed distance has no final activation; color uses Sigmoid. I convert distance to density using a scaled Laplace CDF and add an Eikonal penalty E[(‖∇f‖ − 1)²].

σ(s) = α · { ½ exp(−s/β), s ≥ 0;   1 − ½ exp(s/β), s < 0 }
SettingValue
Training250 epochs · Adam · 0.0005
Density parametersα = 10 · β = 0.05
Eikonal weight0.1
Initialization1,000 steps fitting a unit-sphere SDF
Distance / color MLP6 layers × 128 / 2 layers × 128
Eikonal sampling bounds[−4, 4]³
VolSDF · color
VolSDF · color
VolSDF · zero-level surface
VolSDF · zero-level surface

The first random initialization produced extensive unwanted surfaces. Sphere pretraining and stronger Eikonal regularization reduced these artifacts. The final mesh remains simplified and the RGB result is softer than the density NeRF, so the surface reconstruction is not fully resolved.

Initial VolSDF failure
Initial color
Initial color
Initial surface
Initial surfaceUnwanted geometry extends well beyond the object.

Understanding α and β

α controls density magnitude. Larger α makes the interior more opaque. β controls transition width. High β spreads density across a wider band and can bias image formation toward diffuse, less localized surfaces. Low β approximates a sharper inside/outside transition.

Training: a moderately high β generally offers a wider region of useful gradients and is easier to optimize with finite samples. Very low β needs samples close to the surface and can make optimization unstable. Accuracy: a lower β can localize a surface more precisely after convergence, but only if sampling and optimization resolve that narrow transition; low β alone does not guarantee better geometry.

05

3D Gaussian rasterization

Project, sort, evaluate, and composite.

I construct 3D covariance matrices from scales and rotations, project means into screen coordinates, and propagate covariance through the camera transform and projection Jacobian. A 0.3 diagonal floor limits subpixel footprints. Isotropic covariance reduces to s²I.

Camera-space depth filters Gaussians behind the camera and orders the remainder front to back. At each pixel, opacity multiplies the exponential quadratic form; exclusive transmittance gives compositing weights. The same weights aggregate RGB, depth, and silhouette. Mini-batches carry transmittance including the final Gaussian; residual transmittance blends the background once.

Pᵢ(x) = −½(x − μᵢ)ᵀΣᵢ⁻¹(x − μᵢ)   ·   C(x) = Σᵢ cᵢ αᵢ(x) Tᵢ(x)
4 / 4 supplied unit tests passedVectorized PyTorch
Pretrained chair · DC colors
Pretrained chair · DC colorsRGB, depth, and silhouette across 32 camera views.
06

Training Gaussian representations

Fitting a toy truck from a point-cloud initialization.

I initialize 10,000 isotropic Gaussians from the supplied point cloud and optimize positions, log scales, opacity logits, and RGB using L1 image loss. Rotations remain fixed because isotropic Gaussians are rotation invariant.

IterationsHeld-out PSNRHeld-out SSIM
1,00028.444 dB0.929
ParameterLearning rate
Means0.0001
Log scales0.005
Opacity logits0.01
Colors0.01
Truck · optimization progress
Truck · optimization progressPredictions above; ground truth below.
Truck · final training views
Truck · final training views
07.1

Appearance with spherical harmonics

The same geometry, viewed with a richer color model.

I load all 16 real SH coefficients per color channel through degree 3 and evaluate them along normalized camera-to-Gaussian directions. The weighted basis sum plus 0.5 produces RGB, clamped below at zero. Coefficients are pretrained; this experiment performs inference only.

DC-only rendering
DC-only rendering
Full SH rendering
Full SH rendering

Matched-view RGB comparisons

Frame 6: DC only
DC onlyFrame 006
Frame 6: Degree-3 SH
Degree-3 SHFrame 006

The SH rendering subtly changes the green upholstery shading and edge contrast; the DC version has a more uniform appearance.

Frame 12: DC only
DC onlyFrame 012
Frame 12: Degree-3 SH
Degree-3 SHFrame 012

The oblique view shows small direction-dependent tonal changes along the seat and frame. Silhouette and geometry remain the same.

Frame 24: DC only
DC onlyFrame 024
Frame 24: Degree-3 SH
Degree-3 SHFrame 024

From this view, differences are mainly in local shading rather than object shape. SH preserves the geometry while adjusting RGB with direction.

07.2

A harder scene: materials

Random initialization, multiple objects, and anisotropic refinement.

The baseline transfers the truck settings to 10,000 randomly initialized isotropic Gaussians. The improved experiment keeps 10,000 Gaussians and L1 loss, but uses anisotropic scales and trainable rotations, broader random scene coverage, lower initial opacity, and exponential learning-rate decay.

MethodIterationsValidation PSNR ↑Validation SSIM ↑
Isotropic baseline1,00019.524 dB0.669
Anisotropic experiment1,00021.696 dB0.786
ChangeSame budget+2.172 dB+0.117
Same iteration budget+2.172 dB

Higher validation PSNR after changing initialization and optimization together.

Training progress

Baseline · isotropic
Baseline · isotropicPredictions above; ground truth below.
Improved · anisotropic
Improved · anisotropicPredictions above; ground truth below.
Baseline · final views
Baseline · final views
Improved · final views
Improved · final views
SettingBaselineImproved
Random positionsGaussian cloud near origin, std 0.2Uniform ball, radius 1.5
Initial scaleRandom, approximately 0–0.010.03 per axis
Initial opacitysigmoid(8) ≈ 10.1
ShapeIsotropicAnisotropic; trainable rotations
Means LR0.00010.001 → 0.0001
Scales LR0.0050.003 → 0.0003
Opacity / color LR0.010.01 → 0.001
Rotation LRFixed0.001 → 0.0001
LossL1L1
Scale boundsNo extra bounds0.003–0.3
Color boundsNo extra boundsClamped to [0, 1]

Broader initialization places Gaussians near more objects, while larger initial footprints provide gradients over a wider image region. Lower initial opacity reduces early occlusion. Anisotropy provides more flexible surface coverage, and learning-rate decay allows larger early updates followed by refinement.

The combined method improves both reported validation metrics at the same iteration budget. Because several settings changed together and the baseline was not explicitly seeded in the starter loop, this is a combined-method comparison rather than a controlled attribution of the gain to anisotropy alone. Remaining errors include fine detail and view-dependent reflections; the training model uses RGB colors rather than learned SH.