The pipeline connects camera rays, point sampling, an implicit scene, and volume rendering.
I implement ray sampling, point sampling, and compositing to render an implicit scene.
Camera rays map image pixels to viewing directions in world space.
Uniform samples along each ray provide the points used for volume rendering.
Volume rendering combines sampled colors and densities into color and normalized depth images.
Random pixel samples provide batches of camera rays for training.
Minimizing color mean squared error (MSE) recovers the box center (0.25, 0.25, 0.00) and side lengths (2.01, 1.50, 1.50).
The optimized box reaches a training-view peak signal-to-noise ratio (PSNR) of 61.27 dB and structural similarity (SSIM) of 0.99998.
A 6-layer, 128-unit multilayer perceptron (MLP) with positional encoding learns the Lego scene for novel-view rendering.
| Model | PSNR (dB) | SSIM |
|---|---|---|
| Lego NeRF | 25.57 | 0.9000 |
View-dependent color changes with camera direction, allowing it to capture reflections that position-only color cannot, though this flexibility can also overfit; with matched training on 100 materials images at 400 x 400, view dependence changes held-out PSNR by +1.06 dB.
| Model | PSNR (dB) | SSIM |
|---|---|---|
| Position only | 24.27 | 0.8408 |
| View dependent | 25.33 | 0.8561 |
Coarse/fine sampling combines 64 uniform and 128 importance samples per ray, compared with 192 uniform samples.
| Model | PSNR (dB) | SSIM |
|---|---|---|
| Uniform | 26.11 | 0.9179 |
| Coarse and fine | 26.11 | 0.9185 |
Coarse/fine sampling makes the red rear lights and roof-mounted warning beacon easier to distinguish, with a more vivid red color.
This small quality gain costs more computation: the coarse network evaluates 64 samples and the fine network evaluates all 192 merged samples, versus 192 evaluations for uniform sampling; the second network also increases memory use.
A neural signed distance function (SDF) defines the surface through VolSDF density, with alpha=80 and beta=0.01 selected by validation PSNR among the seven tested settings.
Alpha scales density, making the interior more opaque as it increases, while beta controls the width of the density transition around the surface; the selected small beta favors sharper boundaries, and alpha=80 gave the best validation result.
| Alpha | Beta | Validation PSNR (dB, 4 views) | Test PSNR (dB, 200 views) | Test SSIM |
|---|---|---|---|---|
| 10 | 0.05 | 21.93 | 21.69 | 0.7787 |
| 10 | 0.1 | 21.43 | 21.15 | 0.7583 |
| 40 | 0.05 | 22.71 | 22.62 | 0.8104 |
| 40 | 0.02 | 22.91 | 22.83 | 0.8219 |
| 120 | 0.5 | 13.67 | 12.62 | 0.3824 |
| 120 | 0.01 | 22.82 | 22.81 | 0.8291 |
| 80 | 0.01 | 22.97 | 22.96 | 0.8368 |
| Question | Interpretation |
|---|---|
| How does beta bias the learned SDF? | High beta spreads density across a wider region around the zero level set, allowing blurred boundaries and loss of fine geometric detail; low beta concentrates the transition near the zero level set, favoring sharper surfaces. |
| Is training easier with high or low beta, and why? | A moderately high beta is generally easier to train because more samples along each ray receive useful gradients; low beta confines these gradients to a narrow region that coarse sampling can miss. |
| Which is more likely to recover an accurate surface, and why? | Low beta generally favors more accurate geometry because the rendered density boundary lies closer to the SDF zero level set, reducing surface blurring; this requires sufficient sampling and successful optimization, so smaller beta is not always better in practice. |


Covariance projection uses J R^T Sigma R J^T because PyTorch3D transforms row-vector points as x_view = x_world R + T, so the corresponding column-vector rotation is R^T.
I trained isotropic Gaussians for 1,600 iterations with L1 loss and constant learning rates of 0.0005 for means, 0.005 for scales, 0.01 for colors, and 0.05 for opacities; rotations remain fixed.
| Scene | PSNR (dB) | SSIM | Foreground PSNR (dB) |
|---|---|---|---|
| Truck | 30.09 | 0.9450 | 21.01 |
Degree-3 spherical harmonics (SH) model view-dependent color, compared at matching viewpoints with the constant-color baseline using only the direct current (DC) term; SH shows more noticeable detail in the chair's material.
The materials baseline reuses the Task 6.2 optimizer and full-image L1 training with random initialization, while the improved model uses broader initialization, anisotropic covariance, a combined loss, sampled patches, adaptive splitting and pruning, and longer training.
The baseline remains diffuse during training, while the improved run gradually resolves separate objects and clearer boundaries, though fine reflections remain imperfect.
Held-out PSNR increases from 17.75 to 21.34 dB (+3.59 dB), and SSIM from 0.6357 to 0.8077 (+0.1720); these gains reflect the combined changes and longer training, rather than an isolated test of any one modification.
| Method | PSNR (dB) | SSIM | Foreground PSNR (dB) |
|---|---|---|---|
| baseline | 17.75 | 0.6357 | 12.62 |
| improved | 21.34 | 0.8077 | 16.30 |
| Method | Iterations | Gaussians | Test views |
|---|---|---|---|
| baseline | 1600 | 10000 | 200 |
| improved | 8000 | 18197 | 200 |
| Setting | Baseline | Improved |
|---|---|---|
| Image resolution | 128 x 128 | 128 x 128 |
| Training supervision | Full image (16,384 pixels per step) | 2,048 sampled pixels per step in 8 x 8 patches |
| Covariance | Isotropic | Anisotropic with learned rotations |
| Initial positions | Zero-mean Gaussian, standard deviation 0.2 | Zero-mean Gaussian, standard deviation 0.8 |
| Initial scale | uniform random (0, 0.01) | root mean squared distance to 3 nearest points |
| Initial opacity | 0.999665 | 0.1 |
| Initial colors | uniform random [0, 1) | gray 0.5 |
| Loss | 1 L1 + 0 DSSIM | 0.8 L1 + 0.2 DSSIM + 0.1 MSE |
| Final learning-rate factor | 1.0 | 0.1 |
| Adaptive density | Disabled | Split high-gradient Gaussians along their longest axis; prune low-opacity Gaussians |
| Mean learning rate | 0.0005 | 0.002 |
| Scale learning rate | 0.005 | 0.005 |
| Color learning rate | 0.01 | 0.01 |
| Opacity learning rate | 0.05 | 0.03 |
| Rotation learning rate | Not optimized | 0.001 |