16-825 Assignment 4: Multi-View Transformers

Andrew ID: liyuange

1. Multi-View Data Preprocessing

I implemented the five geometry functions in a4data/geometry.py, keeping the supplied dataloader and visualization code. This part uses the provided RGB-D measurements and cameras; it does not need training.

1.1. Resizing camera intrinsics

The loader handles the center crop first. I scale the focal lengths for the new image size and apply the half-pixel correction to the principal point. This matches the supplied align_corners=False resize.

sx = W_out / W_in, sy = H_out / H_in
fx_new = sx * fx, fy_new = sy * fy
cx_new = sx * (cx + 0.5) - 0.5
cy_new = sy * (cy + 0.5) - 0.5

1.2. Depth unprojection

For an integer pixel center p = (u, v, 1), I compute X_camera = depth * K^-1 * p, then X_world = R * X_camera + t. The poses are camera-to-world. Depth is optical-axis z-depth, so I do not normalize the backprojected direction before multiplying by depth.

These screenshots use five training scenes, eight views per scene, 224px images and seed 825. The provided viewer displays up to 10,000 valid points per view. Red marks the first sampled camera; blue marks the others. Coordinates here are in the original world frame, in meters.

unprojection viewer for 036bce3393_frame_004470, showing colored points, cameras, RGB, depth and coordinate readouts
036bce3393_frame_004470. Original world frame, in meters. First camera center: (3.3463, 4.6765, 1.4680). Click to enlarge.
Four more training scenes
unprojection viewer for 076c822ecc_frame_002890, showing colored points, cameras, RGB, depth and coordinate readouts
076c822ecc_frame_002890. Original world frame, in meters. First camera center: (1.5398, 2.7673, 1.5862). Click to enlarge.
unprojection viewer for 079a326597_frame_007430, showing colored points, cameras, RGB, depth and coordinate readouts
079a326597_frame_007430. Original world frame, in meters. First camera center: (3.7559, 2.0824, 1.5325). Click to enlarge.
unprojection viewer for 07f5b601ee_frame_003630, showing colored points, cameras, RGB, depth and coordinate readouts
07f5b601ee_frame_003630. Original world frame, in meters. First camera center: (3.0944, 3.4889, 1.4076). Click to enlarge.
unprojection viewer for 07ff1c45bb_frame_000460, showing colored points, cameras, RGB, depth and coordinate readouts
07ff1c45bb_frame_000460. Original world frame, in meters. First camera center: (1.3404, 2.1707, 1.5162). Click to enlarge.

1.3. Camera and depth canonicalization

I express the cameras in camera 0's frame using T0^-1 * Ti. I compute this transform in float64, then cast back to float32. After unprojection in that frame, the provided compute_scene_scale gives s = ||mean(valid points)||. I divide camera translations, depth and points by the same scale.

I do not subtract the centroid. Camera 0 stays at the origin with an identity pose, while the valid-point centroid is one unit away. The original first pose and scale are kept so the original world coordinates can be recovered.

The five scenes and sampled frames match the unprojection results above. Each screenshot shows the first camera center and centroid distance in the right-hand panel. The point clouds still have gaps from missing depth and occlusion; normalization does not fill these in.

canonicalization viewer for 036bce3393_frame_004470, showing colored points, cameras, RGB, depth and coordinate readouts
036bce3393_frame_004470. First camera center: (0, 0, 0). Valid-point centroid distance: 1.000000. Scale divisor: 1.867 m. Click to enlarge.
Four more training scenes
canonicalization viewer for 076c822ecc_frame_002890, showing colored points, cameras, RGB, depth and coordinate readouts
076c822ecc_frame_002890. First camera center: (0, 0, 0). Valid-point centroid distance: 1.000000. Scale divisor: 2.830 m. Click to enlarge.
canonicalization viewer for 079a326597_frame_007430, showing colored points, cameras, RGB, depth and coordinate readouts
079a326597_frame_007430. First camera center: (0, 0, 0). Valid-point centroid distance: 1.000000. Scale divisor: 2.080 m. Click to enlarge.
canonicalization viewer for 07f5b601ee_frame_003630, showing colored points, cameras, RGB, depth and coordinate readouts
07f5b601ee_frame_003630. First camera center: (0, 0, 0). Valid-point centroid distance: 1.000000. Scale divisor: 0.924 m. Click to enlarge.
canonicalization viewer for 07ff1c45bb_frame_000460, showing colored points, cameras, RGB, depth and coordinate readouts
07ff1c45bb_frame_000460. First camera center: (0, 0, 0). Valid-point centroid distance: 1.000000. Scale divisor: 1.966 m. Click to enlarge.

1.4. Camera rays

Each ray starts at the camera center o = t. Its unit direction in the canonical world frame is d = normalize(R * K^-1 * p). I rotate the direction but do not translate it. Rays are computed for every pixel, including pixels with missing depth.

1.5. Plücker coordinates

I concatenate the direction and moment in the required order: [d_x, d_y, d_z, m_x, m_y, m_z], where m = o × d. The moment describes where the ray's line lies relative to the origin. Moving the origin along the same line leaves this moment unchanged. It is perpendicular to the direction, and rays from camera 0 have zero moments.

1.6. Provided dataloader

The training loader randomly orders eight frames from a sampling window of up to 32 frames. The first selected frame defines camera 0, so it is not necessarily the earliest frame. The shared defaults use 224px and seed 825.

Validation and test use the prepared loader with deterministic view selection and frame 0 first. That loader bypasses my Q1 functions and applies the shared scale after sampling, so it cannot replace the training-data check below. Its shared default prefix is 128 prepared frames; the model sections specify their own evaluation settings.

1.7. Final check

I ran the required check on four training samples using the default batch size 2 and four workers:

python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 --report outputs/q1/check_data.json

All four samples pass. The largest point reprojection error is 0.00012303 pixels and the largest ray reprojection error is 0.00011109 pixels, both below 0.01 pixels. Centroid distances are approximately 1, with a maximum difference of 6 × 10−8. Invalid depth and point entries stay zero.

Training sampleValid depthMax point error (px)Max ray error (px)Centroid distance
e050c15a8d_frame_00708097.79%0.000112790.000109041.00000000
5f99900f09_frame_00282099.03%0.000123020.000111090.99999994
47b37eb6f9_frame_00272096.10%0.000098820.000088971.00000000
4c5c60fa76_frame_00669075.17%0.000109250.000102571.00000000

The RGB batch shape is (2, 8, 3, 224, 224), depth is (2, 8, 1, 224, 224), and Plücker coordinates are (2, 8, 6, 224, 224). Seven small analytic tests also pass, covering resize alignment, rotated-camera unprojection, canonicalization, masks, unit rays and the Plücker sign/order.

Full check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-View Synthesis with Mini-LVSM

2.0. Implementation

I completed patchify/unpatchify, token assembly, the forward pass, MSE loss and training step in the supplied LVSM files. Patches are flattened in row, column, channel order. Each target gets its own sequence: all source tokens followed by that target's ray-query tokens. Sources and queries share the supplied 2D position embedding.

Source RGB is shifted to [−1, 1] and concatenated with its six Plücker channels. Targets provide rays only, not RGB. After the transformer, I decode only the target-query tokens with the RGB head and sigmoid. Training uses BF16 autocast; the RGB MSE is computed in float32 over every pixel, without a depth mask.

2.1. Fixed-batch overfitting

I ran the prescribed 2,000 updates on one fixed batch: four samples, two source views and six targets per sample. Fixed-batch MSE fell from 0.071160 at initialization to 0.000859. Final overfit PSNR, measured on that same batch, is 30.95 dB. This is a debugging result, not held-out performance. Full training started from a fresh model after this check.

Fixed-batch overfit after 2,000 updates. Columns: source 1, source 2, ground truth, prediction.
Fixed-batch overfit after 2,000 updates. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.

2.2. Full training and evaluation

I used the supplied baseline config unchanged: patch size 8, width 384, eight transformer blocks, six attention heads, and MLP ratio 4. Training uses all 224 training scenes for 100 epochs, with batch size 4: 56 updates per epoch and 5,600 updates total. Each clip has eight 224 × 224 views drawn from a 32-frame window; seed is 825. Two views are sources and the other six are targets.

The optimizer is AdamW with learning rate 0.0002, weight decay 0.05, betas (0.9, 0.95), gradient clipping at 1, 200 warmup steps and the supplied cosine decay to 10% of the peak rate. The baseline ran on GPU 0. Best validation PSNR selected the checkpoint from epoch 100; all 100 epochs were completed.

Baseline training and validation curves. Raw epoch measurements, without smoothing; MSE uses a log axis.
Baseline training and validation curves. Raw epoch measurements, without smoothing; MSE uses a log axis. Click to enlarge.

Here is a fixed training-split example rendered with the best full-training checkpoint:

Training-split scene 0. Columns: source 1, source 2, ground truth, prediction.
Training-split scene 0. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
Validation previews saved during training
Validation preview after epoch 1. Columns: source 1, source 2, ground truth, prediction.
Validation preview after epoch 1. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
Validation preview after epoch 96. Columns: source 1, source 2, ground truth, prediction.
Validation preview after epoch 96. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.

I evaluated the best checkpoint on all 28 validation scenes, using eight views from the first 32 prepared frames, 224px and seed 825. All nine source-count/conditioning combinations use the provided evaluator. No validation batches were omitted.

SourcesTarget raysPSNR (dB)SSIMTarget images
1correct15.190.5582196
1shuffled11.800.4876196
1none12.460.5258196
2correct16.130.5603168
2shuffled11.490.4724168
2none12.430.5260168
4correct16.890.5759112
4shuffled11.590.4732112
4none12.440.5369112

With two sources, shuffled rays change PSNR by -4.65 dB relative to correct rays; removing rays changes it by -3.70 dB. With correct rays, the one-, two- and four-source scores are 15.19, 16.13 and 16.89 dB. More sources supply additional appearance and coverage, but the model was trained with two sources. The supplied source partitions also change which views remain targets, so these averages are not an identical-target comparison.

Shuffling gives the model a different requested viewpoint while keeping the RGB target unchanged. Zero rays remove the query's camera information. Neither changes the source images. Both ray ablations reduce PSNR at two sources, which supports the model using target-camera information. The images below show the resulting errors rather than only the average scores.

The held-out images are still heavily blurred, with visible patch boundaries. The baseline gets some room colors and large bright regions, but furniture layout and thin structures are often inaccurate. Passing the fixed-batch check does not mean strong generalization. These are the results of the full supplied 100-epoch schedule; whether substantially longer training would fix them needs a separate experiment.

All nine evaluation visualizations

1 source view(s)

S=1, correct target rays. Columns: 1 source image(s), ground truth, prediction.
S=1, correct target rays. Columns: 1 source image(s), ground truth, prediction. Click to enlarge.
S=1, shuffled target rays. Columns: 1 source image(s), ground truth, prediction.
S=1, shuffled target rays. Columns: 1 source image(s), ground truth, prediction. Click to enlarge.
S=1, none target rays. Columns: 1 source image(s), ground truth, prediction.
S=1, none target rays. Columns: 1 source image(s), ground truth, prediction. Click to enlarge.

2 source view(s)

S=2, correct target rays. Columns: 2 source image(s), ground truth, prediction.
S=2, correct target rays. Columns: 2 source image(s), ground truth, prediction. Click to enlarge.
S=2, shuffled target rays. Columns: 2 source image(s), ground truth, prediction.
S=2, shuffled target rays. Columns: 2 source image(s), ground truth, prediction. Click to enlarge.
S=2, none target rays. Columns: 2 source image(s), ground truth, prediction.
S=2, none target rays. Columns: 2 source image(s), ground truth, prediction. Click to enlarge.

4 source view(s)

S=4, correct target rays. Columns: 4 source image(s), ground truth, prediction.
S=4, correct target rays. Columns: 4 source image(s), ground truth, prediction. Click to enlarge.
S=4, shuffled target rays. Columns: 4 source image(s), ground truth, prediction.
S=4, shuffled target rays. Columns: 4 source image(s), ground truth, prediction. Click to enlarge.
S=4, none target rays. Columns: 4 source image(s), ground truth, prediction.
S=4, none target rays. Columns: 4 source image(s), ground truth, prediction. Click to enlarge.
Three held-out scene examples with correct rays
Validation scene index 0, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction.
Validation scene index 0, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
Validation scene index 13, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction.
Validation scene index 13, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
Validation scene index 27, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction.
Validation scene index 27, baseline best checkpoint. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.

2.3. Bonus: does a shallower model help?

I compared the eight-layer baseline with four transformer layers. For normal training, only layer count and the output directory change: data, seed, source count, image/patch size, optimizer and the 100-epoch schedule are the same. The four-layer model ran independently on GPU 1. Its initial 2,000-step diagnostic reached only 26.28 dB, so I repeated that fixed-batch check with 6,000 updates and reached 35.00 dB before full training. Neither normal-training run was initialized from overfit weights.

ModelParametersEpochs / updatesLogged train + val timePSNR (dB)SSIM
8 layers14.63 M100 / 5,60022.8 min16.130.5603
4 layers7.54 M100 / 5,60017.2 min15.970.5573
Capacity comparison on the same validation clips: raw PSNR and SSIM by epoch.
Capacity comparison on the same validation clips: raw PSNR and SSIM by epoch. Click to enlarge.

At the normal two-source/correct-ray evaluation, four layers change PSNR by -0.16 dB and SSIM by -0.0030. Each model uses its own best validation-PSNR checkpoint; the four-layer checkpoint is from epoch 100. The smaller model has fewer parameters and less transformer computation per update. This is an equal-epoch/equal-data comparison, not equal compute. Runtime includes validation, uses different GPUs and was measured during concurrent runs; GPU 1 also had an unrelated memory allocation. It is not a clean hardware-speed benchmark.

FP32 weights plus Adam's two moment tensors take roughly 175.6 MB for eight layers and 90.5 MB for four layers, excluding gradients and activations. This is tensor-storage accounting, not a measured peak GPU-memory benchmark. Fewer layers also store fewer block activations, but token counts and per-block attention shapes stay the same, so total memory does not simply halve.

At this training budget, eight layers do better on both held-out metrics. The extra capacity helps here, although the smaller model is cheaper per update. This is one seed on the supplied training subset. Capacity and optimization interact, and validation averages do not guarantee sharp details in every room. The matched examples below compare image quality on the same targets without substituting overfit results for generalization.

Matched eight-layer / four-layer predictions on three validation scenes
8 layers, same validation scene index 0. Columns: source 1, source 2, ground truth, prediction.
8 layers, same validation scene index 0. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
4 layers, same validation scene index 0. Columns: source 1, source 2, ground truth, prediction.
4 layers, same validation scene index 0. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
8 layers, same validation scene index 13. Columns: source 1, source 2, ground truth, prediction.
8 layers, same validation scene index 13. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
4 layers, same validation scene index 13. Columns: source 1, source 2, ground truth, prediction.
4 layers, same validation scene index 13. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
8 layers, same validation scene index 27. Columns: source 1, source 2, ground truth, prediction.
8 layers, same validation scene index 27. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.
4 layers, same validation scene index 27. Columns: source 1, source 2, ground truth, prediction.
4 layers, same validation scene index 27. Columns: source 1, source 2, ground truth, prediction. Click to enlarge.

3. Multi-View Geometry with Mini-VGGT

3.0. Implementation

I completed attention aggregation, camera/depth prediction and the losses in the three supplied VGGT files. Frame blocks attend within each view; global blocks attend over all views. I repeat the 2D positions for global attention, then concatenate the final frame/global features. The DINOv2 ViT-B/14 encoder stays frozen.

The camera head predicts world-to-camera translation, an XYZW quaternion and vertical/horizontal FoV. Only FoV gets a ReLU; camera loss compares the raw encoding directly to GT. The depth head unpatchifies its features, adds RGB refinement, and uses exp for depth and 1 + exp for confidence.

Camera loss combines translation L1 (clamped at 100), quaternion L1 and half the FoV L1. Depth loss uses the provided 98% trimmed regression/confidence terms plus masked residual gradients at strides 1, 2, 4 and 8. Total loss is 5 × camera + depth. Evaluation still uses all GT-valid pixels, without trimming or a predicted-confidence mask.

3.1. Fixed-clip overfitting

I used one fixed eight-view training clip for the prescribed 40 epochs × 25 updates = 1,000 updates, with learning rate 0.001. Total loss fell from 5.4835 to -0.1612. On the fitted clip, depth AbsRel is 0.0372, δ1 is 0.9548, and joint pose AUC@15° is 0.9905. These are training-clip results, not held-out metrics.

The depth objective includes −0.2 log(confidence), so total/depth loss can be negative. The depth and pose metrics are useful checks alongside that objective.

Overfit total, camera and depth losses; the component losses are unweighted.
Overfit total, camera and depth losses; the component losses are unweighted. Click to enlarge.
Fixed training clip: ground-truth geometry.
Fixed training clip: ground-truth geometry. Click to enlarge.
Same fixed clip: predicted geometry after overfitting, in the same coordinates and viewpoint.
Same fixed clip: predicted geometry after overfitting, in the same coordinates and viewpoint. Click to enlarge.

3.2. Full training and evaluation

I trained from a fresh initialization for all 1,000 epochs over the 224 training scenes: 14 updates per epoch, 14,000 updates total. DINO uses the provided pretrained weights but remains frozen. The model has eight frame/global pairs (16 blocks), width 384 and six heads; the camera head has four cross-view blocks.

The supplied config is unchanged: eight 224 × 224 views, sampling range 32, seed 825, AdamW learning rate 0.0006, weight decay 0.01, gradient clipping 1, ten warmup epochs and cosine decay to 5% of the peak rate. Training uses BF16 autocast. Two RTX 4090s run DDP with the provided loader's global batch size 16: eight samples per GPU, not 16 per GPU.

Each epoch validates all 28 validation scenes. Minimum validation loss selects the best checkpoint, from epoch 945. No overfit weights initialize normal training. The required evaluations use that checkpoint on all 28 validation and all 28 test scenes, with eight views from the first 32 prepared frames, 224px and seed 825. There is no max-batch limit.

Normal-training total, camera and depth losses for training and validation over all 1,000 epochs.
Normal-training total, camera and depth losses for training and validation over all 1,000 epochs. Click to enlarge.
MetricValidationTest
Depth AbsRel ↓0.14060.1716
Depth δ1 ↑0.84130.7776
Rotation AUC@5° ↑0.17450.1875
Rotation AUC@15° ↑0.47540.4561
Rotation AUC@30° ↑0.66800.6616
Translation AUC@5° ↑0.00360.0038
Translation AUC@15° ↑0.06250.0386
Translation AUC@30° ↑0.18580.1282
Joint pose AUC@5° ↑0.00100.0008
Joint pose AUC@15° ↑0.04550.0204
Joint pose AUC@30° ↑0.15740.1046

AUC values are in [0, 1]. Rotation AUC includes all camera pairs. Translation/joint AUC excludes undefined GT translation directions and allows the supplied translation-sign ambiguity. The reports contain 10,347,720 validation and 10,517,881 test depth pixels.

The broad room shape is recognizable, but the held-out geometry is much less precise than the overfit example. In the dining room, chair backs and legs merge into a smooth depth blob. The corridor has rounded wall boundaries and overlapping surfaces instead of a clean doorway; the office chair and desk also lose thin details. Camera positions are often too clustered or follow the wrong direction, so combining the views produces thick, misregistered surfaces even when individual depth maps look reasonable. Mean translation-direction error is about 43 degrees on both splits, and joint pose AUC@15 degrees is only 0.0455 on validation and 0.0204 on test. These are not accurate camera reconstructions. The test depth scores are also worse than validation, so fitting one clip well does not establish generalization. Black regions in the displayed depth maps use the GT missing-depth mask; they do not indicate that the model learned to detect missing measurements.

Below are four different validation scenes. Both GT and prediction use the same display ranges and camera viewpoint. I do not post-align the predictions or overwrite the predicted first-camera pose.

09c1414f1b_frame_000000

Scene 1: GT cameras, depth and points.
Scene 1: GT cameras, depth and points. Click to enlarge.
Scene 1: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment.
Scene 1: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment. Click to enlarge.
Scene 2: 21d970d8de_frame_000000

21d970d8de_frame_000000

Scene 2: GT cameras, depth and points.
Scene 2: GT cameras, depth and points. Click to enlarge.
Scene 2: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment.
Scene 2: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment. Click to enlarge.
Scene 3: 280b83fcf3_frame_000000

280b83fcf3_frame_000000

Scene 3: GT cameras, depth and points.
Scene 3: GT cameras, depth and points. Click to enlarge.
Scene 3: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment.
Scene 3: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment. Click to enlarge.
Scene 4: 2b1dc6d6a5_frame_000000

2b1dc6d6a5_frame_000000

Scene 4: GT cameras, depth and points.
Scene 4: GT cameras, depth and points. Click to enlarge.
Scene 4: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment.
Scene 4: predicted cameras, depth and points. Same axes and viewpoint as GT; no alignment. Click to enlarge.

3.3. Bonus: input-view count at evaluation

I kept the trained checkpoint fixed and supplied the first 2, 4 or 8 RGB views from each prepared eight-view validation clip. To make the accuracy comparison controlled, all settings score the same first two views, using the original eight-view canonicalization and scale. This is a matched-view context study, not the standard eight-view evaluation above.

All 28 validation scenes participate. The table reports common-view depth/pose metrics and batch-one inference cost on NVIDIA GeForce RTX 4090. Timing is GPU-synchronized forward time only, after five warmup forwards per setting; it excludes loading and metric computation. Memory is peak PyTorch-allocated forward memory, including resident tensors, not whole-device memory or a training-memory benchmark.

Input viewsAbsRel ↓δ1 ↑Rot. AUC@15°Trans. AUC@15°Joint AUC@15°Forward msPeak MiB
20.17910.68230.50240.07140.069029.35715.6
40.14730.81220.52860.06430.061929.35732.7
80.13710.84870.54520.09760.069029.22769.0
First-view depth in four validation scenes. Columns: RGB, GT depth, 2-view, 4-view and 8-view predictions. Colors are shared within each row; missing GT depth is black.
First-view depth in four validation scenes. Columns: RGB, GT depth, 2-view, 4-view and 8-view predictions. Colors are shared within each row; missing GT depth is black. Click to enlarge.

Common-view depth AbsRel changes from 0.1791 to 0.1473 to 0.1371 as context increases. Joint pose AUC@15° is 0.0690, 0.0619 and 0.0690; the camera result does not follow the same steady trend. In the depth grid, extra views help the dining-room depth scale, but chair legs and wall edges remain blurred.

Mean forward times are 29.35, 29.35 and 29.22 ms. They are essentially flat in this batch-one measurement; the small difference is not evidence that eight views are faster. GPU underutilization and host/kernel-launch overhead are possible explanations, but I did not profile them. Peak allocated memory grows from 715.6 to 769.0 MiB (7.5%). Resident model weights are shared across settings, so total memory need not grow in proportion to the view count.

More views increase encoder work and global-attention sequence length, even though this benchmark does not show a latency increase. The model was trained with eight views, so two/four views are a distribution change. This is one checkpoint and one seed; the matched-view scores are not directly comparable to the full eight-view metrics above, and the first two views provide only 28 camera pairs across the split.