

➜ assignment4_f26 git:(main) ✗ uv run python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 1.0
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.777836385183036e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00011736395390471444,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
The dataset picked an arbitrary origin for each of the samples. The neural network needs to choose an origin and a scale for its output, and that choice must be determined by the input, or otherwise the network output could be among infinitely many options and the loss won’t be well-defined. When we canonicalize the data, we transform all the camera frames (also depth data, and points) to be in the first camera’s frame, and rescale it so the distance from camera 0 to the valid-point centroid is 1. This makes it so the network can learn it’s always predicting a consistent thing: normalized scale in the first camera’s frame.
After canonicalization, we compute, for each pixel in each sample, a world-space ray direction: the unit vector from each camera view’s origin through that pixel (u, v), rotated to be in the canonical world frame.
The Plucker moment is a well-formed way of expressing these rays. The naive way of expressing a ray is by specifying the origin and its direction, d, but each ray has infinitely many of such descriptions (just pick any other point along the ray to be the origin). We instead use the moment m = o x d, because regardless of what point we choose as o the moment is the same. Then (d, m) uniquely describes the line.
overfit PSNR: 30.92 dB
columns: source views, ground
truth, prediction

| Source views (S) | Conditioning | PSNR (dB) | SSIM | MSE loss | Target images |
|---|---|---|---|---|---|
| 1 | correct | 15.19 | 0.5581 | 0.03516 | 196 |
| 1 | shuffled | 11.80 | 0.4872 | 0.07280 | 196 |
| 1 | none | 12.48 | 0.5281 | 0.05865 | 196 |
| 2 | correct | 16.13 | 0.5604 | 0.02942 | 168 |
| 2 | shuffled | 11.50 | 0.4724 | 0.07776 | 168 |
| 2 | none | 12.45 | 0.5266 | 0.05897 | 168 |
| 4 | correct | 16.88 | 0.5759 | 0.02460 | 112 |
| 4 | shuffled | 11.59 | 0.4733 | 0.07837 | 112 |
| 4 | none | 12.47 | 0.5364 | 0.05866 | 112 |
Prediction accuracy increases as the number of source views increases (PSNR goes up from 15.19 with S=1 to 16.88 with S=4). More sources better define the scene, meaning less must be infered by the network. Notably, the network was trained with S=2, so the fact that S=4 is better means the network not only generalizes to more views, but takes advantage of them.
Prediction accuracy is significantly higher when the network is given the correct target ray-conditioning rather than a shuffled one or none (with S=4, correct conditioning has PSNR 16.88, much greater than the shuffled 11.59 and none 12.47). Correct is better than no conditioning (none) which is better than confidently wrong (shuffled). This means the model is actually learning camera geometry rather than just giving some average scene.
columns: source views,
ground truth, prediction
columns: source views,
ground truth, prediction
columns: source views,
ground truth, prediction
Trained with S = 2 sources (6 targets), 8 views sampled from range of 32, image size 224 px, batch size 4, and 100 epochs.
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"batch_size": 4,
"eval_num_frames": 32
},
"model": {
"patch_size": 8,
"d_model": 384,
"n_layers": 8,
"head_dim": 64,
"mlp_ratio": 4.0,
"use_qk_norm": true
},
"loss": {},
"training": {
"epochs": 100,
"learning_rate": 0.0002,
"weight_decay": 0.05,
"betas": [0.9, 0.95],
"warmup_steps": 200,
"grad_clip": 1.0,
"amp": true,
"vis_every": 5,
"output_dir": "outputs/lvsm"
},
"num_source_views": 2
}
Question: Does increasing the number of views in training to 4 increase novel-view prediction at test time?
Settings: Use identical settings to Q2.2 except
change "num_source_views" in the config from 2 to 4.
| Test S | PSNR (train S=2) | PSNR (train S=4) |
|---|---|---|
| 1 | 15.19 | 15.30 |
| 2 | 16.13 | 16.37 |
| 4 | 16.88 | 17.42 |

columns: source views,
ground truth, prediction
Training with 4 views rather than 2 consistently increases test time performance across the ablation over test samples. The improvement increases with larger Test S. Notably, the eval with Test S = 4 went from a PSNR of 16.88 to 17.42. This makes sense as training with more views teaches the network how to better take advantage of increased input view information. This increased performance was at the cost of longer training time (10s per epoch vs. previously only 8s)
The S=4 run has fewer target views per step, confounding the effect on training of number of views and target views.

| Ground truth | Prediction |
|---|---|
![]() |
![]() |

| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel | 0.146 | 0.154 |
| Depth δ1 | 0.824 | 0.800 |
| Rotation AUC@5 deg | 0.223 | 0.216 |
| Rotation AUC@15 deg | 0.525 | 0.549 |
| Rotation AUC@30 deg | 0.690 | 0.743 |
| Translation AUC@5 deg | 0.016 | 0.020 |
| Translation AUC@15 deg | 0.080 | 0.086 |
| Translation AUC@30 deg | 0.200 | 0.196 |
| Joint Pose AUC@5 deg | 0.009 | 0.005 |
| Joint Pose AUC@15 deg | 0.066 | 0.057 |
| Joint Pose AUC@30 deg | 0.173 | 0.174 |
| Scene | Ground truth | Prediction |
|---|---|---|
| 1 | ![]() |
![]() |
| 2 | ![]() |
![]() |
| 3 | ![]() |
![]() |
| 4 | ![]() |
![]() |
This run used 8 views, batch size 16, 5.0 weight on camera loss, 1.0 weight on depth loss and 1000 training epochs.
{
"data_config": "configs/scannetpp/iphone/data.json",
"data_overrides": {
"num_views": 8,
"sampling_range": 32,
"eval_num_frames": 32,
"batch_size": 16,
"num_workers": 8
},
"model": {
"embed_dim": 384,
"num_heads": 6,
"num_layers": 8,
"predict_points": false,
"encoder_name": "dinov2_vitb14",
"image_size": 224
},
"loss": {
"camera": 5.0,
"depth": 1.0
},
"training": {
"epochs": 1000,
"learning_rate": 0.0006,
"weight_decay": 0.01,
"output_dir": "outputs/vggt",
"overfit_scenes": 1,
"overfit_steps_per_epoch": 25,
"min_lr_ratio": 0.05,
"log_every": 10,
"warmup_epochs": 10,
"overfit_epochs": 40,
"overfit_learning_rate": 0.001
}
}
Quantitatively, The model predicts depth and camera rotation moderately well, but seriously struggles predicting translation. On the held-out test set, the absolute relative depth metric of 15.4% means on average the depth estimate was off by about that much, which seems pretty good. Rotation AUC@30 deg on the test set is a relatively high 0.743 meaning the model predicts camera rotation pretty well too. Translation AUC@30 deg in the test set is a paltry 0.196, so the model really struggles here.
Qualitatively the scene visualization shows many key details are blurred which reflects the error in the depth measurement. In scene one and two the chairs are blended in to the background, and in scene 3 and 4 there’s an overall noisiness to the prediction. Further, many of the walls are doubled and offset, reflecting the error in the camera translation prediction.
Question: Does increasing the number of views at evaluation time improve depth and camera pose accuracy?
Settings: Use the same base model trained in 3.2 and
identical evaluation settings to Q3.2 except vary
--num-views to be 2, 4, 8, 16.
| Eval views | Depth AbsRel | Depth δ1 | Rotation AUC@30 deg | Translation AUC@30 deg | Joint Pose AUC@30 deg | Wall time (s) |
|---|---|---|---|---|---|---|
| 2 | 0.108 | 0.891 | 0.725 | 0.273 | 0.249 | 3.01 |
| 4 | 0.127 | 0.858 | 0.656 | 0.172 | 0.141 | 3.37 |
| 8 | 0.146 | 0.824 | 0.690 | 0.200 | 0.173 | 4.26 |
| 16 | 0.167 | 0.786 | 0.681 | 0.182 | 0.161 | 6.13 |
| Eval views | Ground truth | Prediction |
|---|---|---|
| 2 | ![]() |
![]() |
| 4 | ![]() |
![]() |
| 8 | ![]() |
![]() |
| 16 | ![]() |
![]() |
Suprisingly, more views don’t seem to help. With more eval views, the depth error is worse (increases from 0.108 @ V=2 to 0.167 @ V=16), the rotation AUC is worse (decreases from 0.725 @ V=2 to 0.681 @ V=16) and the translation AUC is worse (decreases from 0.273 @ V=2 to 0.182 @ V=16). Its possible V=2 did the best, because tith more cameras, the model has to learn their placement which is extra degrees of freedom making it a much harder problem.
The 2-view metrics come from much fewer pairs than the 16-view metrics, so they are noisier.