Assignment 3: Volume Rendering, NeRF and Gaussian Splatting
1. Differentiable volume rendering
Pixel coordinates span [-1, 1] and follow the PyTorch3D NDC axis convention. Points at camera depth 1 are unprojected to world space; the camera center is the ray origin and the normalized displacement is the ray direction. Uniform distances between the configured near and far planes generate samples as x(t) = o + td.
For each sample, alpha = 1 - exp(-sigma * delta). Transmittance is exp(-sum of preceding sigma * delta), with T_0 = 1. Color and depth are weighted sums using weights T * alpha. The depth visualization is divided by its maximum value as requested. Inference runs without gradients and uses ray chunks to limit memory.
Pixel coordinates
Ray directions
Sample points
Normalized depth
Rendered box2. Optimizing a basic implicit volume
Each iteration randomly samples 4096 pixels without replacement and minimizes RGB mean squared error. Adam uses learning rate 0.0005 for 1000 epochs, with the four provided camera views and the original train_box configuration.
- Final box center (rounded to 0.01): (0.25, 0.25, 0.00).
- Final side lengths (rounded to 0.01): (2.01, 1.50, 1.50).
- Final sampled training MSE: 1.0787062e-06.
- Full-image MSE across all four training views: 7.8300664e-07.
Compared with the TA GIF, the optimized box has a similar rectangular shape, rainbow colours, and soft silhouette.
Optimized box3. Optimizing a neural radiance field
The position-only MLP uses the provided harmonic positional encoding with six frequency bands. The trunk has six linear layers, width 128, and concatenates the position encoding at layer index 3. A ReLU density head produces nonnegative density. A two-layer color head (128 to 64 to 3) uses sigmoid to produce RGB in [0, 1]. Color does not depend on viewing direction in this part. Both heads receive gradients from RGB mean squared error.
Training uses the original Lego setup: 128 by 128 images, 250 epochs, 1024 rays per iteration, 128 samples per ray, near/far distances 2 and 6, Adam learning rate 0.0005 with the provided decay schedule. Random seed is 1. The rendering chunk size is reduced from 32768 to 2048 rays to fit the 8 GB laptop GPU; the training ray batch is unchanged.
- Completed epochs: 250.
- Held-out validation views: 100.
- Validation MSE: 0.003708.
- Validation PSNR: 24.31 dB, computed as
- -10 log10 of mean image MSE, RGB range [0, 1].
Novel-view spiral
Validation comparison: target left, prediction right ## 4.2 Coarse/fine sampling (selected NeRF extension)
Two distinct position-only NeRFs are optimized jointly. The coarse network evaluates 64 uniform points per ray. Detached coarse interior rendering weights define a piecewise-constant PDF on midpoint bins; inverse-CDF sampling draws 64 additional points. The fine network evaluates the sorted union of the 64 coarse and 64 importance samples. A small positive PDF floor handles empty rays. Training samples are randomized, while evaluation uses deterministic quantiles. The objective is coarse RGB MSE plus fine RGB MSE, allowing both networks to learn rather than letting sampling gradients couple them.
Both networks initialize from the completed Part 3 checkpoint, then train jointly for 100 further epochs at 128 by 128 resolution, 1024 rays per batch, and the same Adam learning rate and decay schedule. This is a warm-start extension experiment, not a comparison at equal total training compute.
| Model | Validation MSE | PSNR (dB) | Mean seconds/image |
|---|
| Part 3, uniform 128 samples | 0.003708 | 24.31 | 0.1584 |
| Part 4.2, coarse 64 + fine 128 evaluations | 0.003111 | 25.07 | 0.2364 |
The evaluation uses the same 100 held-out views for both models. Timings are wall-clock GPU-synchronized measurements on this laptop, after all training jobs finished. The coarse/fine result improves PSNR by 0.76 dB, while rendering takes about 49% longer in this measurement. Importance sampling puts more fine evaluations near estimated surfaces, while retaining uniform samples to cover empty or poorly estimated regions. Two network evaluations and sorting add work; improving sample placement does not guarantee either a speedup or a quality improvement at every training budget. Coarse errors can under-sample a real surface, and added samples can otherwise be spent on already well-covered geometry.
Coarse/fine spiral
Coarse/fine validation: target left, prediction right5. 3D Gaussian rasterization
Isotropic covariance is s squared times the identity; anisotropic covariance uses quaternion rotation and diagonal squared scales. The supplied camera Jacobian projects covariance into screen space, with the provided 0.3-pixel diagonal regularization. Projected means are in screen pixels. Gaussians behind the camera are removed and remaining Gaussians are sorted by depth. Pixel alpha is opacity times the exponential Gaussian power. Exclusive front-to-back transmittance weights produce RGB, weighted depth, and mask.
Chunk boundaries carry transmittance AFTER the final Gaussian, including that Gaussian's alpha. The accumulated RGB is already premultiplied by opacity, so background blending adds (1 - mask) times the background colour. These two corrections make chunked and one-shot integration consistent.
To accelerate inference, 16 by 16 screen tiles conservatively cull Gaussians outside the alpha >= 1/255 ellipse already used by the supplied renderer. Depth order and the original alpha/transmittance thresholds are preserved. Official covariance/projection/Gaussian tests pass 4/4. Additional tests verify tiled and dense image, depth, mask, and parameter-gradient agreement.
DC-only chair RGB, depth, mask6. Training 3D Gaussian representations
The truck point cloud initializes 10,000 isotropic Gaussians. Means, log scales, colours and opacity logits are optimized with Adam and standard RGB L1 loss. Quaternions remain fixed because isotropic rotations do not change covariance. Training uses the original 160 by 90 images, all 197 training views and the provided five held-out test views. Seed is 1.
| Parameter | Learning rate |
|---|
| Means | 0.0001 |
| Log scales | 0.005 |
| Colours | 0.01 |
| Opacity logits | 0.01 |
- Training iterations: 1000.
- Held-out mean PSNR: 28.501 dB.
- Held-out mean SSIM: 0.9303.
- Gaussian batch cap per tile: 512.
Tiled culling and activation checkpointing reduce memory use for the 8 GB GPU. Checkpoint recomputation preserves gradients, including transmittance between chunks. Periodic optimizer/parameter checkpoints and final Gaussian parameters are saved. PSNR uses data range 1.0; SSIM uses RGB channels and data range 1.0, averaged across the held-out test views.
Truck training progress: prediction top, target bottom
Truck final training-view renders7.1 Spherical harmonics (selected Gaussian extension)
The 16 real degree-0-through-3 spherical harmonic basis functions are evaluated using normalized world-space directions from camera center to each Gaussian. The loaded 48 coefficients reshape to 16 bases by three colour channels. The weighted basis sum is shifted by 0.5 and clamped below at zero, following the pretrained representation's convention.
The DC-only and full-SH renders use identical geometry, cameras, image size, depth ordering, and background. Only colour evaluation changes. The basis constants and ordering follow the original implementation's SH utility.
DC-only chair
Degree-3 SH chair
View 4: DC left, SH rightIn view 4, SH gives the seat a darker green center and stronger shading under the backrest. The decorative gold upholstery pattern remains aligned because the geometry and texture coefficients are unchanged.
View 12: DC left, SH rightIn view 12, the SH seat again has deeper shadows and more contrast, with subtler brightness changes along the upholstered back and wooden frame. DC-only appearance is comparatively flat across these views.
View 24: DC left, SH rightIn this extreme view the seat edge changes tone slightly, while the stretched geometry artifacts remain in both renders. SH changes view-dependent colour and cannot repair missing or inaccurate geometry.
Extension selection assessment
7.1 is a small inference extension using coefficients and data already provided, so it is suitable for this environment. 7.2 requires the larger materials dataset, random initialization, additional tuning, and potentially much longer training or density-control work. It is not a convenient additional option on this 8 GB laptop, so 7.1 alone is selected, satisfying the requirement to choose at least one. For Part 4, only 4.2 is selected; 4.1 and 4.3 are unselected alternatives.