16-825 Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

1.1–1.2 Unprojection (original world frame, meters)

Command: python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000. Each view's depth is unprojected with its resized intrinsics and original camera-to-world pose, so all views should form one consistent point cloud in the ScanNet++ world frame.

Unprojected depth and camera poses in the original world frame.
Unprojected depth and camera poses in the original world frame.

1.3 Canonicalization

Command: python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000. After canonicalization camera 0 sits at the origin with identity pose, and the valid-point centroid is 1 unit from the origin.

Canonicalized geometry in the first camera reference frame.
Canonicalized geometry in the first camera reference frame.

Canonical first-view coordinates. First camera center (X, Y, Z): (0.0000, 0.0000, 0.0000) normalized units. Scene scale: 1.867 m. Valid-point centroid distance: 1.000 normalized units. The red frustum marks the first sampled camera; black depth pixels indicate missing measurements.

1.7 Final check

Command: python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4. All four samples have 8 views at 224×224, reprojection errors around 10−4 px (requirement: < 0.01 px) and a centroid distance of 1.

SampleValid depthMax point reproj. error (px)Max ray reproj. error (px)Centroid distance
e050c15a8d_frame_0070800.9781.10e-41.08e-41.000
5f99900f09_frame_0028200.9901.23e-41.11e-41.000
47b37eb6f9_frame_0027200.9619.88e-58.90e-51.000
4c5c60fa76_frame_0066900.7521.13e-41.02e-41.000
Full check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {"sequence_name": "e050c15a8d_frame_007080", "views": 8, "resolution": 224,
     "valid_depth_fraction": 0.9779127240180969,
     "max_reprojection_error_px": 0.00011009880108758807,
     "max_ray_reprojection_error_px": 0.00010764275066321716,
     "centroid_distance": 1.0},
    {"sequence_name": "5f99900f09_frame_002820", "views": 8, "resolution": 224,
     "valid_depth_fraction": 0.9903116226196289,
     "max_reprojection_error_px": 0.00012302029062993824,
     "max_ray_reprojection_error_px": 0.00011108566104667261,
     "centroid_distance": 0.9999999403953552},
    {"sequence_name": "47b37eb6f9_frame_002720", "views": 8, "resolution": 224,
     "valid_depth_fraction": 0.9609947800636292,
     "max_reprojection_error_px": 9.881925507215783e-05,
     "max_ray_reprojection_error_px": 8.897326188161969e-05,
     "centroid_distance": 1.0},
    {"sequence_name": "4c5c60fa76_frame_006690", "views": 8, "resolution": 224,
     "valid_depth_fraction": 0.7516716122627258,
     "max_reprojection_error_px": 0.00011280165199423209,
     "max_ray_reprojection_error_px": 0.00010225349979009479,
     "centroid_distance": 0.9999999403953552}
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224], "depth": [2, 8, 1, 224, 224], "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3], "frame_numbers": [2, 8], "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4], "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224], "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224], "plucker": [2, 8, 6, 224, 224]
  }
}

Explanations

Canonicalization. I express all camera poses relative to the first sampled camera using T′i = Tworld←0−1Tworld←i, computed in float64. This makes camera 0 the identity. I then unproject the valid depth pixels in this reference frame and compute s = ‖mean(valid points)‖. Dividing translations, depth and points by the same scale preserves their geometry and places the valid-point centroid one unit from the origin. Rotations and intrinsics stay unchanged, and I do not subtract the centroid. This gives every sample a consistent reference frame and scale, reducing the ambiguity in supervision.

World-space ray directions. For pixel (u, v), K−1[u, v, 1]T gives a camera-space direction with z-coordinate 1. Multiplying this vector by optical-axis z-depth produces the camera-space point. To construct a ray, I rotate the direction with the camera-to-world rotation and normalize it: d = normalize(RK−1[u, v, 1]T). The ray origin is the camera center, o = t. Translation changes the origin but does not change the direction.

Plücker moment. I encode each ray as [d, m], where m = o × d. Any point p = o + λd on the same line gives the same moment because p × d = o × d. The encoding therefore represents an oriented line independently of the chosen point along it. For a unit direction, ‖m‖ is the line’s distance from the world origin. Together, direction and moment describe the viewing line in the canonical frame.

2.1 Mini-LVSM Overfitting

Command: python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit (2000 steps on one fixed batch of 4 samples, 2 source + 6 target views each; 768 s on an NVIDIA A10G).

Final overfit PSNR on the fixed batch: 31.06 dB (SSIM 0.873, MSE 8.4e-4). The epoch-average training PSNR over all 2000 steps was 25.48 dB.

LVSM overfit visualization
val_epoch000.png after overfitting. Columns: source 1, source 2, ground-truth target, prediction.

2.2 Mini-LVSM Training and Evaluation

Training settings

Training curves

LVSM training curves
Training and validation MSE, PSNR and SSIM per epoch (from outputs/lvsm/log.jsonl). The final checkpoint is also the best: validation PSNR 16.14 dB, SSIM 0.560.

Training and validation PSNR continue to improve near epoch 100, suggesting that the model may benefit from additional training. Validation PSNR is slightly higher than training PSNR. One possible reason is that training metrics average predictions made throughout an epoch, whereas validation uses the updated weights at its end. The sampling procedures also differ: training uses random windows, while validation uses fixed views from the first 32 frames.

Training visualizations

Epoch 0
Epoch 0
Epoch 50
Epoch 50
Epoch 95
Epoch 95

Columns in every LVSM visualization: source views, ground-truth target, prediction.

Ablation: number of source views × target-ray conditioning

Command: python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none (28 validation scenes, 8 views from the first 32 frames, seed 825).

Source viewscorrectshufflednoneTarget images
PSNR ↑SSIM ↑PSNR ↑SSIM ↑PSNR ↑SSIM ↑
S = 115.200.55811.800.48712.470.529196
S = 2 (trained)16.140.56011.480.47212.450.528168
S = 416.860.57511.600.47412.470.538112

Evaluation visualizations

S=1 correct
S = 1, correct rays
S=2 correct
S = 2, correct rays
S=4 correct
S = 4, correct rays
S=2 correct
S = 2, correct
S=2 shuffled
S = 2, shuffled
S=2 none
S = 2, none
All nine ablation images
S=1 shuffled
S = 1, shuffled
S=1 none
S = 1, none
S=4 shuffled
S = 4, shuffled
S=4 none
S = 4, none

Discussion

Number of source views. With correct target rays, PSNR increases from 15.20 to 16.14 to 16.86 dB as the number of sources increases from 1 to 2 to 4. SSIM also increases, from 0.558 to 0.560 to 0.575. Additional views can provide coverage of regions that are occluded in a single source. Although the model was trained with S = 2, its attention layers accept different source-token counts at evaluation. However, this is not a comparison on a fixed target set: increasing S reduces the number of evaluated targets from 196 to 168 to 112, which may also affect the scores.

Ray conditioning. Shuffling or removing target rays reduces PSNR by roughly 3–5 dB relative to correct conditioning. Under either ablation, adding source views provides little benefit. This supports the conclusion that the model uses target rays to choose the rendered viewpoint. Zero rays produce predictions with little spatial detail; shuffled rays provide a different target camera and can produce structure that is misaligned with the ground truth. The lower scores for shuffled rays are consistent with MSE penalizing misplaced structure more strongly than a smooth prediction.

Limitations. Predictions capture coarse room layout and color but remain blurry and show 8×8 patch artifacts. Fine textures are poorly reconstructed. The model has only 14.6M parameters and was trained for 5,600 steps using MSE alone. Limited training and the averaging encouraged by MSE are plausible contributors, although these experiments do not isolate their individual effects.

2.3 Bonus: Training with a Random Number of Source Views

Question

The baseline is trained only with S = 2 source views but evaluated with 1, 2 and 4. Does training with a random number of source views each step make the model better at handling different source counts at test time?

Setup

Controlled comparison: the only change is that each training step samples S uniformly from {1, 2, 3, 4} (torch RNG, same seed), so each step has T = 8 − S targets. Model, data, batch size, 100 epochs, optimizer, schedule and validation (S = 2) are identical to Q2.2. Config: configs/lvsm_random_s.json ("source_views_range": [1, 4]), implemented in train_one_epoch. I evaluated both runs' best.pt (epoch 99 for both) with the same ablation command as Q2.2. Training took 29.3 s/epoch vs. 28.2 s/epoch for the baseline.

Source viewscorrect rays: PSNR ↑ / SSIM ↑shuffled: PSNRnone: PSNR
Fixed S = 2Random SΔ PSNRFixedRandomFixedRandom
S = 115.20 / 0.55815.28 / 0.559+0.0811.8011.7612.4712.49
S = 216.14 / 0.56016.22 / 0.560+0.0811.4811.4312.4512.46
S = 416.86 / 0.57517.09 / 0.575+0.2311.6011.4712.4712.48
Fixed vs random source-view training
Left: validation PSNR (S = 2) during training for both runs. Right: evaluation PSNR with correct rays (note the y-axis starts at 14.5 dB, so the gaps look larger than they are).
Fixed S=2, S=4 eval
Fixed S = 2 training, evaluated with S = 4
Random S, S=4 eval
Random S training, evaluated with S = 4

Findings

Conclusion and limitations

Randomizing the number of source views during training is a cheap change (about 4% slower) that gives a small, consistent improvement, largest at S = 4, a source count absent from baseline training. However, the effects are small (0.1–0.2 dB), and each configuration was trained once with one seed, so part of the difference could be run-to-run noise. Several seeds would be needed to confirm it. The source counts were also limited to 1–4, and as in Q2.2, the number of target images differs between S settings.

3.1 Mini-VGGT Overfitting

Command: python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit

I trained on one fixed eight-view clip for 40 epochs with 25 steps per epoch (1,000 updates), using a learning rate of 1e-3. The fixed-clip evaluation loss decreased from 5.483 before training to −0.166. Depth AbsRel decreased from 0.619 to 0.0365, while δ1 increased from 0.208 to 0.956. Mean rotation and translation-direction errors fell to 0.344° and 0.442°, respectively. These metrics indicate that the model can fit the clip. The screenshot below compares ground-truth and predicted geometry.

The depth objective includes a confidence term, c|e| − 0.2 log(c), so a negative loss is possible. It should be interpreted alongside depth and camera accuracy rather than as a nonnegative reconstruction error.

Mini-VGGT total, camera and depth loss curves
Total, camera and depth losses during fixed-clip overfitting.
Ground-truth and predicted geometry after fixed-clip overfitting.
Ground-truth and predicted geometry after fixed-clip overfitting.

3.2 Mini-VGGT Training and Evaluation

Training settings

I used a frozen DINOv2 ViT-B/14 encoder with eight frame/global attention pairs, width 384 and six heads. Each training sample contains eight 224×224 views sampled from a 32-frame window. Training used all 224 scenes, a batch size of 16 and 1,000 epochs (14 updates per epoch). AdamW used a peak learning rate of 6e-4 and weight decay 0.01, with 10 warmup epochs followed by cosine decay to 5% of the peak. The total objective weights camera loss by 5 and depth loss by 1. The run used bf16 precision on one NVIDIA A10G. Summed epoch times in the training log total approximately 5.83 hours.

Loss curve

Mini-VGGT total, camera and depth loss curves
Total, camera and depth losses during full training.

Metrics (8 views from the first 32 frames, 224px, seed 825)

SplitAbsRel ↓δ1 ↑ Rotation AUC ↑Translation AUC ↑Joint pose AUC ↑
@5°@15°@30°@5°@15°@30°@5°@15°@30°
Val0.1570.7960.2200.5580.7250.0130.0810.1960.0040.0660.178
Test0.1640.7780.2340.5620.7430.0160.0940.2090.0040.0670.187

Evaluated outputs/vggt_train/best.pt (epoch 929, lowest validation loss) on the full splits (28 scenes each, 784 view pairs, no degenerate pairs). Numbers from outputs/vggt_train/metrics.json (val) and outputs/vggt_train/test_metrics.json (test).

GT vs. predicted geometry (four held-out scenes)

Scene 1: 09c1414f1b

Ground truth, scene 09c1414f1b
Ground truth.
Predicted cameras + depth, scene 09c1414f1b
Predicted cameras + depth.

Scene 2: 21d970d8de

Ground truth, scene 21d970d8de
Ground truth.
Predicted cameras + depth, scene 21d970d8de
Predicted cameras + depth.

Scene 3: 280b83fcf3

Ground truth, scene 280b83fcf3
Ground truth.
Predicted cameras + depth, scene 280b83fcf3
Predicted cameras + depth.

Scene 4: 2b1dc6d6a5

Ground truth, scene 2b1dc6d6a5
Ground truth.
Predicted cameras + depth, scene 2b1dc6d6a5
Predicted cameras + depth.

Each row compares GT (left) and prediction (right) for the same scene. Click an image to inspect it at full size. Screenshots come from outputs/vggt_train/visualization/index.html.

Discussion

Held-out depth accuracy is similar across splits: AbsRel is 0.157 on validation and 0.164 on test, while δ1 is 0.796 and 0.778. Test depth is slightly worse, but the gap is small in this evaluation. Rotation is substantially more accurate than translation direction: at 30°, validation AUC is 0.725 for rotation and 0.196 for translation. Joint pose AUC is only 0.178, indicating that translation remains a major limitation. Test pose AUC is slightly higher at 0.187, so the test split is not uniformly harder across tasks.

These aggregate metrics do not identify which surfaces or scenes cause the largest errors. Compared with GT, the dining-room prediction smooths the table and chair boundaries, while the hallway prediction loses some of the sharper wall structure. The office reconstruction is more diffuse around the chair and desk. These examples show that coarse layout is recovered more reliably than fine geometry; they do not isolate the causes of the errors.

3.3 Bonus: Number of Input Views at Evaluation

Question

Mini-VGGT was trained with exactly 8 views. How do accuracy, runtime and memory change if the same model is given fewer or more views at test time?

Setup

I evaluated the Q3.2 checkpoint (outputs/vggt_train/best.pt, no retraining) on the full validation split with V = 2, 4, 8, 16 and 32 views. Everything else matches the Q3.2 evaluation: views come from the first 32 prepared frames with seed 825, at 224px, and metrics use the same evaluate function. Runtime is the average forward pass per scene (batch size 1, bf16, after two warm-up scenes) on one NVIDIA A10G. Memory is the peak allocated GPU memory during those forward passes. Script: scripts/vggt_num_views.py.

ViewsAbsRel ↓δ1 ↑Rot AUC@30 ↑Trans AUC@30 ↑Pose AUC@30 ↑Pose pairsms / scenePeak GPU memory
20.1220.8520.7320.1760.1602842.70.74 GB
40.1350.8490.7270.2130.19216841.00.76 GB
8 (trained)0.1570.7960.7260.1960.17878443.80.81 GB
160.1860.7470.7110.1790.164336058.30.89 GB
320.1950.7140.7070.1600.14513888113.81.05 GB
Accuracy, runtime and memory vs. number of views
Depth metrics, camera AUC@30°, and runtime/memory against the number of input views (log scale). The V = 8 row reproduces the Q3.2 validation numbers; tiny differences come from bf16 rounding at a different batch size.

Findings

Conclusion and limitations

A model trained with a fixed number of views does not automatically get better with more views: per-view depth degrades, and camera accuracy is best near the training count. Training with a variable or larger view count is a possible follow-up, but this evaluation does not test whether it would improve accuracy. Limitations: one checkpoint, validation split only, evaluation-time changes only (no retraining), and every setting draws views from the same 32-frame window. That means V = 32 uses every frame, with many nearly identical neighboring views, and the number of pose pairs varies widely between settings.

3.3 Bonus (second exploration): Point-Map Prediction and Multi-Task Synergy

Questions

  1. Does a direct point-map head produce more accurate 3D points than unprojecting predicted depth with predicted cameras?
  2. Does learning cameras and depth jointly, or adding point maps as a third task, help each task compared with training it alone?

Setup

I trained three additional models with the Q3.2 configuration (frozen DINOv2 ViT-B/14, 8 views, sampling range 32, batch size 16, 1,000 epochs, same optimizer and schedule). Each run changes only the loss configuration:

RunLoss weights (camera / depth / points)ConfigBest epoch
Joint (Q3.2)5 / 1 / –configs/vggt.json929
Joint + points5 / 1 / 1configs/vggt_points.json912
Camera only5 / 0 / –configs/vggt_camera_only.json715
Depth only0 / 1 / –configs/vggt_depth_only.json839

The point head mirrors the depth head: a linear patch decoder to a 3-channel (x, y, z) map in the canonical first-camera frame, followed by an RGB-conditioned residual CNN, with no output activation. The point loss is the trimmed mean of the per-pixel Euclidean error on GT-valid pixels, plus the depth gradient loss averaged over the three coordinates. In the single-task runs, the other head still exists but receives no gradient, so its metrics are not meaningful and are shown as “–”. Each best.pt is selected by the lowest validation loss of that run's own objective and evaluated exactly as in Q3.2. 3D point error is the mean Euclidean distance to the GT point map over GT-valid pixels, in normalized units and in meters (scripts/vggt_points_eval.py). The joint + points run took about 23.1 s per epoch versus 20.7 s for the single-task runs.

Multi-task comparison

RunValidationTest
AbsRel ↓δ1 ↑Rot@30 ↑Trans@30 ↑Pose@30 ↑ AbsRel ↓δ1 ↑Rot@30 ↑Trans@30 ↑Pose@30 ↑
Joint (Q3.2)0.1570.7960.7250.1960.178 0.1640.7780.7430.2090.187
Joint + points0.1360.8390.6840.1660.150 0.1520.8060.7180.1970.183
Camera only––0.7100.1890.169 ––0.7150.1640.145
Depth only0.1520.806––– 0.1750.766–––
AUC at 5° / 15° / 30° (test)
RunRotation AUCTranslation AUCJoint pose AUC
Joint (Q3.2)0.234 / 0.562 / 0.7430.016 / 0.094 / 0.2090.004 / 0.067 / 0.187
Joint + points0.202 / 0.523 / 0.7180.008 / 0.073 / 0.1970.002 / 0.055 / 0.183
Camera only0.207 / 0.516 / 0.7150.010 / 0.065 / 0.1640.001 / 0.038 / 0.145

3D point accuracy: direct point map vs. cameras + depth

ModelPoints fromVal error (normalized / m)Test error (normalized / m)
Joint (Q3.2)predicted cameras + predicted depth0.289 / 0.5800.245 / 0.480
Joint + pointspredicted cameras + predicted depth0.264 / 0.5340.231 / 0.451
Joint + pointsdirect point head0.275 / 0.5600.239 / 0.463
Joint (Q3.2)GT cameras + predicted depth0.157 / 0.3370.153 / 0.315
Joint + pointsGT cameras + predicted depth0.137 / 0.2980.147 / 0.294

The GT-camera rows are diagnostic only: they use ground-truth cameras to isolate the error that comes from depth.

Ground truth: scene 09c1414f1b
Ground truth — scene 09c1414f1b.
Predicted cameras + depth: scene 09c1414f1b
Predicted cameras + depth — scene 09c1414f1b.
Direct point head: scene 09c1414f1b
Direct point head — scene 09c1414f1b.

All three screenshots show the same dining-room scene. Both prediction paths recover the broad layout, but the point clouds are more diffuse than GT and fine furniture structure is less distinct. This example illustrates reconstruction differences; the split-wide metrics above provide the quantitative comparison. Click an image to inspect it at full size.

Findings

Conclusion and limitations

Under this training budget, joint camera and depth learning appears to benefit camera estimation, while its effect on depth is mixed across splits. A direct point-map head achieves 3D accuracy similar to unprojection, and adding it as an auxiliary task improves depth at some cost to camera accuracy. Limitations: each configuration was trained once with one seed, each split has only 28 scenes, the point loss weight (1.0) was not tuned, and each run's best checkpoint was selected by a different objective. Differences of a few thousandths may therefore fall within run-to-run variation.