# Assignment 3: volume rendering, NeRF, and Gaussian splatting

Open `index.html` for the completed report. `ASSIGNMENT.md` preserves the course
instructions. The implementation covers A1–A3, A4.2 (coarse/fine sampling),
B5–B6, and B7.1 (spherical harmonic rendering).

## Environment

All implementation checks, experiments, figures, and archive generation run on
the desktop reached by `ssh desktop`. The checkout is
`~/Desktop/Projects/L3D/proj3`; the existing conda environment is `l3d-proj2`.
`environment.json` records the actual Python, CUDA, PyTorch, PyTorch3D, GPU, and
starter revision. The RTX 5090 uses PyTorch 2.8.0 with CUDA 12.8 and PyTorch3D
built for compute capability 12.0.

```sh
source ~/miniconda3/etc/profile.d/conda.sh
conda activate l3d-proj2
cd ~/Desktop/Projects/L3D/proj3
pip install -r requirements.txt
```

For a fresh machine, follow the Assignment 1 PyTorch/PyTorch3D environment
instructions before installing these small additional dependencies. The lockfile
records installed package versions; its PyTorch CUDA wheels require the
`https://download.pytorch.org/whl/cu128` index. Build the pinned PyTorch3D
source after PyTorch, with `--no-build-isolation` and the local CUDA toolkit.

## Data

The source ZIP excludes datasets. From a fresh copy of the course repository,
the box images and GS chair/truck assets are already present. To restore them
alongside an extracted submission, download the original starter repository and
copy its `nerf/data` and `GS/data` folders into the corresponding directories.

```sh
git clone https://github.com/learning3d/assignment3 /tmp/l3d-assignment3-starter
git -C /tmp/l3d-assignment3-starter checkout f80077292be0f80cbb2a3d285a7f6b0af14c56ff
cp -R /tmp/l3d-assignment3-starter/nerf/data nerf/
cp -R /tmp/l3d-assignment3-starter/GS/data GS/
```

The Lego dataset consists of `lego.pth`, `lego.png`, and `lego_license.txt` from
`https://dl.fbaipublicfiles.com/pytorch3d_nerf_data`. The dataset loader downloads
missing files. These experiments retain the supplied train/validation/test
splits. The materials dataset is unnecessary for the two selected extensions.

## Neural volume rendering

Run these commands from `nerf/`:

```sh
python -m pytest -q ../tests
python volume_rendering_main.py --config-name=box
python volume_rendering_main.py --config-name=train_box
python volume_rendering_main.py --config-name=nerf_lego
python volume_rendering_main.py --config-name=nerf_lego_hierarchical
```

The box renderer writes `images/part_1.gif`, the grid/ray/sample-point images,
and the normalized depth image. Box fitting writes `images/part_2.gif` and
`runs/box/summary.json`, including the recovered center and side lengths.

Lego settings are recorded in `configs/nerf_lego.yaml` and
`configs/nerf_lego_hierarchical.yaml`, then copied into each run directory.
The baseline uses 128 uniform stratified samples per ray. The hierarchical
model uses 64 coarse samples, draws 64 additional samples from the coarse
weight distribution, and evaluates a separate fine network on all 128 points.
Its loss supervises both coarse and fine colors.

Both configurations train for 250 epochs with seed 1, 128 × 128 images, and
1,024 rays per update. Validation PSNR selects the best checkpoint. Testing
uses the supplied held-out views; PSNR and SSIM are averaged per image.
`summary.json`, `test_metrics.json`, and `history.jsonl` contain measured
results. Training time excludes validation and final rendering. Evaluation time
includes the complete held-out pass and saving its example images.

`last.pth` stores optimizer, scheduler, and random states for resuming an
interrupted training run. `best.pth` stores the validation-selected model and
configuration. To evaluate the included checkpoints without training:

```sh
python volume_rendering_main.py --config-name=nerf_lego type=evaluate_nerf
python volume_rendering_main.py --config-name=nerf_lego_hierarchical type=evaluate_nerf
```

The submission includes only `best.pth`; starting training from
the submission begins a new run. Set `training.run_dir` and
`training.output_gif` to fresh paths when retaining the included results.

## Gaussian splatting

Run these commands from `GS/`:

```sh
python validate_projection.py
python validate_renderer.py
python train.py --out_path ./output --num_itrs 1000 --gaussians_per_splat 512
python render.py --out_path ./output --gaussians_per_splat 512
python render.py --out_path ./output --gaussians_per_splat 512 --use_sh
python finalize_outputs.py
```

The supplied `unit_test_gaussians.py` remains unchanged for comparison. Its
3D-covariance and projected-mean cases pass; the 2D-covariance and Gaussian-power
fixtures use the opposite camera rotation convention and disagree
with the corrected implementation. Running it reports 2/4 rather than 4/4.
PyTorch3D defines the row-vector transform as `X_cam = X_world R + T`, so a
column-vector covariance transforms with `W = R.T`. The independent projection
regression compares anisotropic covariances at three rotated cameras with
both autograd and central finite-difference projection Jacobians. The renderer
regression checks mini-batch equivalence, finite gradients, and DC-only SH
agreement. See the [official camera transform documentation](https://pytorch3d.readthedocs.io/en/latest/modules/renderer/cameras.html#pytorch3d.renderer.cameras.get_world_to_view_transform).

The truck uses the supplied point initialization, isotropic Gaussians, and L1 image loss.
`output/metrics.json` records the learning rate for each parameter, iteration
count, training time, held-out per-view PSNR/SSIM, and their means. Training
writes both progress and final-view GIFs. DC-only and spherical harmonic chair
renders use identical camera poses. `finalize_outputs.py` creates the matched
RGB comparison panels and checks the required animations.

`output/trained_gaussians_inference.pt` contains the optimized parameters for
inference. `python render_checkpoint.py` renders the inference checkpoint independently.
`output/trained_gaussians.pt` additionally contains optimizer state
for `train.py --resume`; this larger resume checkpoint is excluded from the
submission. Gaussian mini-batching controls the peak memory use. Smaller
`--gaussians_per_splat` values reduce temporary rasterizer memory.

## Report and archives

After the experiment outputs exist, run from the project root:

```sh
python build_report.py
python verify_artifacts.py
python package_submission.py
```

The report uses relative links and opens offline. Its figures are copied into
`assets/`; plots and explanations use saved experiment measurements. Existing
packaged assets allow report rebuilding without rerendering; fresh original
renderer outputs take precedence when present.
Verification decodes every animation frame, checks required deliverables,
rejects nonfinite metrics, and verifies local report links.

The packager writes `jjacob2_code_proj3.zip` and `proj3_website.zip` into the
parent directory, plus a SHA-256 manifest. The code archive includes source,
configuration, tests, inference checkpoints, measured results, and report
assets. It excludes datasets, environments, caches, logs, Git metadata,
intermediate checkpoints, and internal coordination files. The website archive
contains only the report and its linked files. Neither script submits to Canvas
or publishes to the course server.

## Sources

Starter code and data: [learning3d/assignment3](https://github.com/learning3d/assignment3),
revision `f80077292be0f80cbb2a3d285a7f6b0af14c56ff`.
The assignment links the [NeRF paper](https://arxiv.org/abs/2003.08934),
the [3D Gaussian Splatting paper](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/),
and the spherical harmonic basis reference. PyTorch3D supplies camera transforms.

License notices: `LICENSE-PyTorch3D` and `LICENSE-Lego.txt`.
