Assignment 4 - Fall 2026
Karthik Srinivasan (ksriniv2)
Compute: RTX 5090
check_data.py output)







check_data.py output)| Sample | Views | Resolution | Valid depth fraction | Max point reprojection error (px) | Max ray reprojection error (px) | Centroid distance |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 9.54e-05 | 9.76e-05 | 1.000 |
| 5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.08e-04 | 9.65e-05 | 1.000 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 1.23e-04 | 1.11e-04 | 1.000 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.11e-04 | 9.92e-05 | 1.000 |
Both reprojection errors are about 1e-4 px (requirement: below 0.01 px) and the centroid distance is 1.000 for every sample.
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 9.5367431640625e-05,
"max_ray_reprojection_error_px": 9.761195542523637e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.0001078959321603179,
"max_ray_reprojection_error_px": 9.650505671743304e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011056454241042957,
"max_ray_reprojection_error_px": 9.921078162733465e-05,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization
World-space ray directions
Plücker moment
The Plücker moment is used to tell the model where each camera is. It is the cross product of the ray origin o (the camera center) and the world-space unit direction d:
m = o × d
The direction alone cannot identify a ray, since cameras at different positions can look the same way. Together, [d, m] identify exactly one line in 3D. In Mini-LVSM, source images get these six channels stacked with RGB, and the target view gets them alone to say which view to draw.
Final overfit result: validation PSNR 30.97 dB, SSIM 0.871 (24 images; train PSNR 25.41 dB).
100 epochs, batch size 4, AdamW (learning rate 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps, gradient clipping 1.0, bf16 autocast, MSE loss. Model: patch size 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm. Trained with 2 source views per sample on 224x224 ScanNet++ images, about 9.3 s per epoch (15.3 min total).

Source views S in {1, 2, 4}. Conditioning: correct uses the true target Plücker rays, shuffled uses the rays of a different target view, none zeroes the target rays.
| Source views | Conditioning | PSNR | SSIM |
|---|---|---|---|
| 1 | correct | 15.19 | 0.558 |
| 1 | shuffled | 11.79 | 0.487 |
| 1 | none | 12.47 | 0.527 |
| 2 | correct | 16.13 | 0.560 |
| 2 | shuffled | 11.49 | 0.472 |
| 2 | none | 12.45 | 0.527 |
| 4 | correct | 16.87 | 0.576 |
| 4 | shuffled | 11.59 | 0.473 |
| 4 | none | 12.46 | 0.537 |
Training (validation batch during training at epoch 95, 2 source views, correct rays; columns are source views, ground truth target, prediction):

Evaluation (validation, 2 source views, correct conditioning):

Effect of the number of source views. With correct conditioning, PSNR rises from 15.19 dB (S=1) to 16.13 dB (S=2) to 16.87 dB (S=4), and SSIM from 0.558 to 0.560 to 0.576. More source views cover more of the target view, so the model has to guess less. The model was trained only with S=2, yet S=4 still improves, so it generalizes to more views than it saw.
Effect of the ray-conditioning ablation. Correct rays give 16.13 dB at S=2. Zeroing the target rays drops it to 12.45 dB, and shuffled rays drop it to 11.49 dB. The model therefore uses the target rays to decide which view to render. Shuffled is worse than none because the model renders the wrong viewpoint, while with no rays it falls back to an average-looking image. The effect is the same at S=1 and S=4 (12.47 and 12.46 dB for none; 11.79 and 11.59 dB for shuffled), so extra source views do not make up for missing target rays.
Loss function: how do PSNR, SSIM and visual sharpness change when a loss term (L1, SSIM or perceptual) is added alongside the MSE loss?
Same Mini-LVSM and training setup as 2.2 (S=2, 100 epochs); only the loss changed. Added terms: L1 (weight 1.0), SSIM loss (weight 0.5), perceptual loss (weight 0.1, MSE between VGG16 relu3_3 features). Each variant was trained once. Metrics are on the validation set with the best checkpoint and correct rays.
| Loss | S=1 | S=2 | S=4 | |||
|---|---|---|---|---|---|---|
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |
| MSE (baseline) | 15.21 | 0.559 | 16.16 | 0.560 | 16.90 | 0.576 |
| MSE + L1 | 15.35 | 0.560 | 16.29 | 0.563 | 17.11 | 0.579 |
| MSE + SSIM | 14.74 | 0.577 | 15.38 | 0.581 | 15.72 | 0.595 |
| MSE + perceptual | 14.14 | 0.257 | 14.57 | 0.254 | 14.76 | 0.254 |
Validation, S=2. Columns: source view 1, source view 2, ground truth, prediction. Rows are four target views.
MSE (baseline)

MSE + L1

MSE + SSIM

MSE + perceptual

L1 is the only term that raises PSNR, by 0.1 to 0.2 dB, and it looks the same as MSE: blurry with visible patch blockiness. The SSIM term raises SSIM by about 0.02 and removes the blockiness, but PSNR drops by 0.8 to 1.2 dB and the output is still blurry. The perceptual term lowers both metrics and adds a fine checkerboard texture over a blurry base, which is texture but not real sharpness. None of the losses gives sharp predictions, so the blur is more likely limited by the model and training length than by the loss. Limitations: one run per loss, one weight per term, and no L1 + SSIM combination (that run did not finish).
One scene, 40 epochs of 25 steps, learning rate 1e-3. Final overfit metrics on that scene: depth AbsRel 0.038, delta1 0.954, mean rotation error 0.26 degrees, mean translation direction error 0.47 degrees, joint AUC@30 0.994. Camera loss falls from 0.34 to 0.0026.

Ground truth (left) vs. prediction (right), scene 036bce3393_frame_004470:
All hyperparameters are the defaults from configs/vggt.json; none were changed.
The best checkpoint by validation loss is evaluated below.

| Split | Depth AbsRel | Depth δ1 | Rot AUC@5 | @15 | @30 | Trans AUC@5 | @15 | @30 | Joint AUC@5 | @15 | @30 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Val | 0.157 | 0.788 | 0.193 | 0.524 | 0.717 | 0.020 | 0.095 | 0.209 | 0.007 | 0.070 | 0.184 |
| Test | 0.157 | 0.792 | 0.217 | 0.568 | 0.754 | 0.018 | 0.084 | 0.195 | 0.006 | 0.064 | 0.178 |
Each image shows ground truth (left) and prediction (right) in canonical first-view coordinates, with the RGB and depth of view 0.
Scene 1: 0a76e06478
Scene 2: 1b9692f0c7
Scene 3: 210f741378
Scene 4: 251443268c
Unfreezing DINO: how does training the DINO encoder affect performance compared with keeping it frozen?
| Split | Encoder | Depth AbsRel | Depth δ1 | Rot err (deg) | Trans err (deg) | Rot AUC@30 | Trans AUC@30 | Joint AUC@30 |
|---|---|---|---|---|---|---|---|---|
| Val | Frozen | 0.173 | 0.754 | 17.1 | 47.9 | 0.605 | 0.138 | 0.120 |
| Val | Unfrozen | 0.158 | 0.795 | 18.0 | 45.8 | 0.599 | 0.139 | 0.112 |
| Test | Frozen | 0.185 | 0.726 | 11.8 | 45.9 | 0.647 | 0.121 | 0.111 |
| Test | Unfrozen | 0.171 | 0.772 | 10.8 | 46.8 | 0.681 | 0.122 | 0.104 |
Final training loss: 1.527 (frozen) and 1.412 (unfrozen). Best validation loss: 1.035 (frozen) and 0.994 (unfrozen).
Frozen:

Unfrozen:
