# Assignment 4: multi-view transformers

Joshua Jacob · JJACOB2 · 16-825 Learning for 3D Vision

Open `index.html` for the report. Its artifact-status section states which
required results are present. `assets/results.json` contains measured results;
`assets/requirement_status.json` records missing deliverables explicitly.
`ASSIGNMENT.md` preserves the original starter overview, and `docs/` contains
the original task definitions and coordinate contract.

## Main entry point

Run `python main.py --help` for the common entry point. Its commands cover
`check-data`, `visualize-data`, `geometry-figures`, `lvsm-train`, `lvsm-eval`,
`vggt-train`, `vggt-eval`, `test`, `report`, `verify` and `package`. Subsequent
arguments are forwarded to that program; for example:

```sh
python main.py check-data --split train --max-samples 4
python main.py lvsm-eval --checkpoint outputs/lvsm/best.pt \
  --num-source-views 1 2 4 --conditioning correct shuffled none
python main.py report
python main.py verify --require-complete
```

## Environment and data

Run commands from this directory. Install the PyTorch build appropriate to the
machine, followed by the assignment dependencies:

```sh
python -m pip install -r requirements.txt
python -m pip install -e . --no-deps
```

CUDA is supported, and the supplied fast-run scripts also support Apple MPS.
For MPS, set `PYTORCH_ENABLE_MPS_FALLBACK=1`; Q1 preprocessing runs on CPU.
Use a CPU environment when no supported accelerator is available.
The VGGT encoder downloads frozen pretrained DINOv2 weights through PyTorch Hub.
The DINOv2 source revision is pinned in `models/vggt/encoder.py`. Even when
evaluating the included weights, a fresh machine needs network access once to
fetch that encoder source; `--encoder-repo` can instead select a local checkout.

The dataset is excluded from the submission. Restore the official data with:

```sh
python -m pip install huggingface_hub
hf download learning3dvision/ScanNetpp_iphone ScanNetpp_A4.zip \
  --repo-type dataset --local-dir data/scannetpp
unzip data/scannetpp/ScanNetpp_A4.zip -d data/scannetpp
```

Expected directories are `data/scannetpp/processed` and
`data/scannetpp/prepared_eval`. The official scene split has 224 training,
28 validation and 28 test scenes. The training loader applies the implemented
geometry; prepared evaluation uses the supplied independent geometry loader.

## Part 1: geometry

`a4data/geometry.py` implements resize intrinsics, optical-axis depth
unprojection, first-camera canonicalization, world-space rays and Plücker
coordinates. Independent convention tests include a known rotated camera,
half-pixel scaling, rigid-transform invariance and full-pipeline reprojection.

```sh
python -m unittest discover -s tests -p test_geometry.py -v
python scripts/check_data.py --config configs/scannetpp/iphone/data.json \
  --split train --max-samples 4
python scripts/visualize_data.py --stage unprojection --num-scenes 5 \
  --max-points 10000 --output outputs/unprojection
python scripts/visualize_data.py --stage canonicalization --num-scenes 5 \
  --max-points 10000 --output outputs/canonicalization
```

Save the data-check output to `outputs/geometry_check.txt`. Report figures may
be placed at `assets/geometry/unprojection.png` and
`assets/geometry/canonicalization.png`. The interactive viewers are copied
into report assets when the report is built.

## Part 2: Mini-LVSM

`models/lvsm/model.py` implements patchification, target-specific token
assembly and image decoding; the loss and training step are implemented in
the corresponding modules. The model never receives target RGB as input.

```sh
LVSM_EPOCHS=100 LVSM_OUTPUT=outputs/lvsm_retrain \
  LVSM_OVERFIT_OUTPUT=outputs/lvsm_overfit_retrain bash scripts/run_lvsm.sh
```

The script overfits one fixed batch, trains across the official training split,
then evaluates all nine combinations of 1/2/4 source views and
correct/shuffled/zero target rays. `configs/lvsm_fast.json` reduces model
capacity relative to the supplied baseline. The 100-epoch schedule matches the
course baseline; the report records how many epochs actually completed. Environment
variables `LVSM_EPOCHS`, `LVSM_OVERFIT_STEPS`, `LVSM_CONFIG`, `LVSM_OUTPUT`,
`LVSM_OVERFIT_OUTPUT` and `LVSM_PYTHON` select run settings and destinations.
Existing normal-training logs are preserved; choose a new output directory for
a new experiment. The standalone trainer supports `--resume`.

Actual settings, recorded epochs, PSNR/SSIM, curves and visualizations are read
from each output directory. Overfit metrics are training-batch measurements;
normal-run metrics use held-out validation scenes. The source-count ablation
changes the target partition as specified by the provided evaluator.

## Part 3: Mini-VGGT

`models/vggt/model.py`, `heads.py` and `losses.py` implement alternating
attention, camera and depth/confidence heads, and the camera/depth objectives.

```sh
python -m models.vggt.train --config configs/vggt_fast.json --device mps \
  --overfit --epochs 20 --output outputs/vggt_overfit_retrain
python -m models.vggt.train --config configs/vggt_fast.json --device mps \
  --epochs 40 --stop-after-epoch 20 --output outputs/vggt_retrain
```

These fresh output directories preserve the measurements in the submission.
Change `--device mps` to `cuda` or `cpu` on other machines. From a fresh checkout
without existing outputs, `VGGT_EPOCHS=40 bash scripts/run_vggt.sh` also runs the
complete training/evaluation sequence.

The fast configuration keeps eight views, a 32-frame sampling range and
224-pixel images. It uses a smaller frozen DINOv2 encoder and fewer trainable
attention blocks. The normal run retains a 40-epoch cosine schedule, with a
checkpoint-safe stop at epoch 20; the report records the number actually
completed. This is shorter than the supplied 1,000-epoch schedule.
`VGGT_DEVICE`, `VGGT_CONFIG`, `VGGT_EPOCHS`,
`VGGT_OVERFIT_EPOCHS` and `VGGT_PYTHON` control execution. Choose fresh output
paths when starting a new experiment; use the standalone trainer's
`--resume outputs/vggt_train/last.pt` to continue an interrupted run.

The final evaluation commands in the pipeline script evaluate every validation/test
scene using eight views from the first 32 prepared frames, image size 224
and seed 825. No evaluation batch limit is used. The JSON reports identify
the number of scenes actually evaluated, depth metrics and pose AUCs.
`visualization/index.html` compares predicted geometry with ground truth.
Screenshots exported to the same directory are picked up by the report.

The measured normal run resumed from its saved epoch-7 optimizer/scheduler
state. The unsaved part of epoch 8 was discarded; the last logged step was 40,
but the exact number of additional unlogged updates is unknown. The supplied
resume loader reseeds sampling and data order, so an uninterrupted fresh run
does not have the identical loader RNG trajectory. After the restart, the MPS
model computes the unchanged differentiable loss on CPU; autograd transfers
gradients back to the model. CPU/MPS loss values and representative gradients
were checked on three real training clips before switching. The final source
uses this CPU-loss path for MPS training. The saved configuration retains the
original 40-epoch learning-rate schedule throughout.

## Evaluate the included checkpoints

From an extracted code archive with the dataset restored, run:

```sh
python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt \
  --num-source-views 1 2 4 --conditioning correct shuffled none
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt \
  --split val --eval-root data/scannetpp/prepared_eval \
  --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 \
  --report outputs/vggt_train/metrics.json
python -m models.vggt.eval --checkpoint outputs/vggt_train/best.pt \
  --split test --eval-root data/scannetpp/prepared_eval \
  --num-views 8 --eval-num-frames 32 --image-size 224 --seed 825 \
  --report outputs/vggt_train/test_metrics.json
```

To evaluate retrained weights, substitute the fresh checkpoint paths above.
LVSM chooses CUDA, MPS or CPU automatically. VGGT chooses CUDA or CPU by default;
add `--device mps` and set `PYTORCH_ENABLE_MPS_FALLBACK=1` to use Apple MPS.

## Report, verification and archives

```sh
python build_report.py
python verify_artifacts.py
python package_submission.py
```

`report_config.json` can override run paths. `report_notes.json` can supply
observations keyed by `q1`, `lvsm` and `vggt`. The report never fills absent
measurements with invented values. `verify_artifacts.py` validates local HTML
links, decodes image frames and checks that saved numeric values are finite.
Add `--require-complete` to make missing required artifacts an error.

The public webpage keeps every required figure, screenshot and metric while
omitting the optional large interactive viewers to meet the course's 20 MB
website limit. Original viewers remain available locally under `outputs/`
and in the code archive under `assets/outputs/`. To link them in a local-only
report, set `include_interactive_viewers` to `true` in `report_config.json`.

Packaging writes `jjacob2_code_proj4.zip`, `proj4_website.zip`, a SHA-256
manifest and a submission audit in the parent directory. The code archive
contains implementations, configurations, tests, measured results, the report
and compact normal-run `best.pt` inference checkpoints when available. The
packager retains model weights, configuration, selected epoch and evaluation
metadata while stripping optimizer/scheduler state; local originals remain
unchanged. Evaluate these checkpoints with the commands in the model task
documents. Resuming optimization requires the original local training
checkpoint. Datasets, environment caches, internal coordination files and the
supplied TA visualization are excluded. The website archive contains the
report and its linked assets. Packaging does not upload, publish or submit
to Canvas.

## Sources

Starter code and assignment: [learning3d/assignment4_f26](https://github.com/learning3d/assignment4_f26).
Data: the course's prepared [ScanNet++ iPhone subset](https://huggingface.co/datasets/learning3dvision/ScanNetpp_iphone).
Frozen encoder: [DINOv2](https://github.com/facebookresearch/dinov2).

The frozen DINOv2 weights included in the VGGT inference checkpoint retain
their Apache 2.0 license; see `LICENSE-DINOv2`.
