16-825: Learning for 3D Vision

Assignment 4 - Fall 2026

Karthik Srinivasan (ksriniv2)

Compute: RTX 5090

Contents

Part 1. Multi-view Data

1. Data Preprocessing

1.2. Depth Unprojection

unprojection 1
unprojection 2
unprojection 3
unprojection 4

1.3. Camera and Depth Canonicalization

canonicalization 1
canonicalization 2
canonicalization 3
canonicalization 4

1.7. Final check (check_data.py output)

SampleViewsResolutionValid depth fractionMax point reprojection error (px)Max ray reprojection error (px)Centroid distance
e050c15a8d_frame_00708082240.9789.54e-059.76e-051.000
5f99900f09_frame_00282082240.9901.08e-049.65e-051.000
47b37eb6f9_frame_00272082240.9611.23e-041.11e-041.000
4c5c60fa76_frame_00669082240.7521.11e-049.92e-051.000

Both reprojection errors are about 1e-4 px (requirement: below 0.01 px) and the centroid distance is 1.000 for every sample.

Raw check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 9.5367431640625e-05,
      "max_ray_reprojection_error_px": 9.761195542523637e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.0001078959321603179,
      "max_ray_reprojection_error_px": 9.650505671743304e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00011056454241042957,
      "max_ray_reprojection_error_px": 9.921078162733465e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Explanations

Canonicalization

World-space ray directions

Plücker moment

The Plücker moment is used to tell the model where each camera is. It is the cross product of the ray origin o (the camera center) and the world-space unit direction d:

m = o × d

The direction alone cannot identify a ray, since cameras at different positions can look the same way. Together, [d, m] identify exactly one line in 3D. In Mini-LVSM, source images get these six channels stacked with RGB, and the target view gets them alone to say which view to draw.

Part 2. Mini-LVSM

2.1. Overfitting

Final overfit result: validation PSNR 30.97 dB, SSIM 0.871 (24 images; train PSNR 25.41 dB).

Source view 1
Source view 2
Ground truth
Prediction
LVSM overfit visualization

2.2. Training and Evaluation

Training settings

100 epochs, batch size 4, AdamW (learning rate 2e-4, weight decay 0.05, betas 0.9/0.95), 200 warmup steps, gradient clipping 1.0, bf16 autocast, MSE loss. Model: patch size 8, width 384, 8 layers, head dim 64, MLP ratio 4, QK-norm. Trained with 2 source views per sample on 224x224 ScanNet++ images, about 9.3 s per epoch (15.3 min total).

Training curves

LVSM training curves

Validation PSNR / SSIM per ablation setting

Source views S in {1, 2, 4}. Conditioning: correct uses the true target Plücker rays, shuffled uses the rays of a different target view, none zeroes the target rays.

Source viewsConditioningPSNRSSIM
1correct15.190.558
1shuffled11.790.487
1none12.470.527
2correct16.130.560
2shuffled11.490.472
2none12.450.527
4correct16.870.576
4shuffled11.590.473
4none12.460.537

Visualizations (source views, ground truth, prediction)

Training (validation batch during training at epoch 95, 2 source views, correct rays; columns are source views, ground truth target, prediction):

LVSM train visualization

Evaluation (validation, 2 source views, correct conditioning):

LVSM eval visualization

Discussion

Effect of the number of source views. With correct conditioning, PSNR rises from 15.19 dB (S=1) to 16.13 dB (S=2) to 16.87 dB (S=4), and SSIM from 0.558 to 0.560 to 0.576. More source views cover more of the target view, so the model has to guess less. The model was trained only with S=2, yet S=4 still improves, so it generalizes to more views than it saw.

Effect of the ray-conditioning ablation. Correct rays give 16.13 dB at S=2. Zeroing the target rays drops it to 12.45 dB, and shuffled rays drop it to 11.49 dB. The model therefore uses the target rays to decide which view to render. Shuffled is worse than none because the model renders the wrong viewpoint, while with no rays it falls back to an average-looking image. The effect is the same at S=1 and S=4 (12.47 and 12.46 dB for none; 11.79 and 11.59 dB for shuffled), so extra source views do not make up for missing target rays.

2.3. Bonus: Analysis and Exploration

Question

Loss function: how do PSNR, SSIM and visual sharpness change when a loss term (L1, SSIM or perceptual) is added alongside the MSE loss?

Experimental settings

Same Mini-LVSM and training setup as 2.2 (S=2, 100 epochs); only the loss changed. Added terms: L1 (weight 1.0), SSIM loss (weight 0.5), perceptual loss (weight 0.1, MSE between VGG16 relu3_3 features). Each variant was trained once. Metrics are on the validation set with the best checkpoint and correct rays.

Results

LossS=1S=2S=4
PSNRSSIMPSNRSSIMPSNRSSIM
MSE (baseline)15.210.55916.160.56016.900.576
MSE + L115.350.56016.290.56317.110.579
MSE + SSIM14.740.57715.380.58115.720.595
MSE + perceptual14.140.25714.570.25414.760.254

Validation, S=2. Columns: source view 1, source view 2, ground truth, prediction. Rows are four target views.

MSE (baseline)

MSE

MSE + L1

MSE + L1

MSE + SSIM

MSE + SSIM

MSE + perceptual

MSE + perceptual

Conclusions, explanations and limitations

L1 is the only term that raises PSNR, by 0.1 to 0.2 dB, and it looks the same as MSE: blurry with visible patch blockiness. The SSIM term raises SSIM by about 0.02 and removes the blockiness, but PSNR drops by 0.8 to 1.2 dB and the output is still blurry. The perceptual term lowers both metrics and adds a fine checkerboard texture over a blurry base, which is texture but not real sharpness. None of the losses gives sharp predictions, so the blur is more likely limited by the model and training length than by the loss. Limitations: one run per loss, one weight per term, and no L1 + SSIM combination (that run did not finish).

Part 3. Mini-VGGT

3.1. Overfitting

One scene, 40 epochs of 25 steps, learning rate 1e-3. Final overfit metrics on that scene: depth AbsRel 0.038, delta1 0.954, mean rotation error 0.26 degrees, mean translation direction error 0.47 degrees, joint AUC@30 0.994. Camera loss falls from 0.34 to 0.0026.

VGGT overfit loss curve

Ground truth (left) vs. prediction (right), scene 036bce3393_frame_004470:

Overfit GT vs prediction

3.2. Training and Evaluation

Training settings

All hyperparameters are the defaults from configs/vggt.json; none were changed.

The best checkpoint by validation loss is evaluated below.

Loss curve (total, camera, depth)

VGGT training loss curve

Validation and test metrics

SplitDepth AbsRelDepth δ1Rot AUC@5@15@30Trans AUC@5@15@30Joint AUC@5@15@30
Val0.1570.7880.1930.5240.7170.0200.0950.2090.0070.0700.184
Test0.1570.7920.2170.5680.7540.0180.0840.1950.0060.0640.178

GT vs. predicted geometry (four different test scenes)

Each image shows ground truth (left) and prediction (right) in canonical first-view coordinates, with the RGB and depth of view 0.

Scene 1: 0a76e06478

Scene 1

Scene 2: 1b9692f0c7

Scene 2

Scene 3: 210f741378

Scene 3

Scene 4: 251443268c

Scene 4

Discussion of errors on held-out scenes

3.3. Bonus: Analysis and Exploration

Question

Unfreezing DINO: how does training the DINO encoder affect performance compared with keeping it frozen?

Experimental settings

Results

SplitEncoderDepth AbsRelDepth δ1Rot err (deg)Trans err (deg)Rot AUC@30Trans AUC@30Joint AUC@30
ValFrozen0.1730.75417.147.90.6050.1380.120
ValUnfrozen0.1580.79518.045.80.5990.1390.112
TestFrozen0.1850.72611.845.90.6470.1210.111
TestUnfrozen0.1710.77210.846.80.6810.1220.104

Final training loss: 1.527 (frozen) and 1.412 (unfrozen). Best validation loss: 1.035 (frozen) and 0.994 (unfrozen).

Frozen:

Frozen loss curve

Unfrozen:

Unfrozen loss curve

Conclusions, explanations and limitations