Assignment 4: Multi-View Transformers

16-825 Learning for 3D Vision · Fall 2026 · Jinxi He (ginh)

I use the ScanNet++ iPhone subset to implement multi-view geometry, novel-view synthesis with Mini-LVSM, and camera/depth prediction with Mini-VGGT. The Q3 bonus compares 2, 4, and 8 input views. All experiments use 224 training, 28 validation, and 28 test scenes, 224 × 224 images, and seed 825.

Use the menus to compare scenes and settings. Click a figure to enlarge it.

Q1. Multi-View Data Preprocessing (20 points)

1.1–1.2. Intrinsics and depth unprojection

For a resize scale s, I update each axis of the intrinsics with f′ = sf and c′ = s(c + 0.5) − 0.5, matching the image resampling convention. Depth is measured along the camera's optical axis and converted from millimeters to meters. For pixel p = [u, v, 1]T:

Xcamera = z K−1 p
Xworld = R Xcamera + t

[R | t] is the camera-to-world pose. Pixels with missing depth are masked out.

1.3. Canonicalization

I transform each camera pose into camera 0's frame with T0−1Ti, then unproject the depth in that frame. I divide translations, points, and depths by the distance to the valid-point centroid. Camera 0 stays at (0, 0, 0) with identity rotation, and the centroid is 1 unit away. The centroid is not subtracted.

Each pair below shows the same eight views before and after normalization. Camera 0's position and the centroid distance are printed below the plots. To keep outliers from shrinking the room, the main plots use the 0.5th–99.5th percentile range on each axis, expanded to include all cameras. The insets show the full point-cloud extent.

1.4–1.5. World-space rays and Plücker coordinates

Each ray starts at the camera center o = t and has direction d = normalize(R K−1p). Rotation puts the direction in world coordinates; translation sets the origin. The six-channel Plücker representation is:

Plücker(o, d) = [d, m],   m = o × d

The moment describes the line's offset from the origin, distinguishing parallel rays through different camera centers. Moving the origin along the ray leaves it unchanged: (o + αd) × d = o × d.

1.6–1.7. Dataloader and geometry checks

The training loader samples frames in random order from a temporal window; the first selected view is camera 0. Validation and test use fixed views from the prepared frames. The checker passes on four training samples with eight views each. Both point and ray reprojection errors are below 0.01 pixels.

Q1 geometry verification
CheckMeasured
Training scenes / checked clips224 / 4
Views / image size8 / 224 × 224
Maximum point reprojection error0.0001230203 px
Maximum ray reprojection error0.0001110857 px
Valid depth fraction0.75167–0.99031
Valid-point centroid distance0.99999994–1.0
RGB / depth batch shapes(2, 8, 3, 224, 224) / (2, 8, 1, 224, 224)
Plücker batch shape(2, 8, 6, 224, 224)
Complete check_data.py output

Open the original output

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011009880108758807,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Q2. Novel-View Synthesis with Mini-LVSM (40 points)

2.0. Model and training objective

Source tokens contain RGB scaled to [−1, 1] and six Plücker channels; target queries contain only rays. Each target attends to all source tokens and its own queries, with shared 2D positional embeddings. A sigmoid RGB head decodes the query outputs into image patches. The loss is MSE over all target RGB pixels, including those without valid depth.

2.1. Fixed-batch overfit (10 points)

I fit a fixed batch of four training scenes for 2,000 updates. On the same 24 target images, the model reaches 30.9009 dB PSNR, 0.87018 SSIM, and MSE 0.00086950. It captures the viewpoints and colors, though fine details remain blurred and some patch texture is visible.

Source 1Source 2Target GTPrediction
Mini-LVSM fixed-batch overfit, two source columns, ground truth and prediction
Four target views from the first scene in the fixed training batch.

2.2. Full training and evaluation (30 points)

I train for 100 epochs (5,600 updates) on all 224 training scenes. The final epoch has the best validation PSNR with two sources. Training and validation take 1,545.6 seconds on an RTX 6000 Ada, excluding output overhead; the separate overfit run takes 454 seconds.

Q2 training settings
SettingMini-LVSM baseline
ArchitecturePatch size 8; width 384; 8 transformer blocks; head dimension 64; MLP ratio 4; Q/K normalization
Parameters14,633,408, trained from scratch
Data8 views: 2 sources + 6 targets; sampling range 32; 224 × 224; batch size 4; seed 825
OptimizerAdamW; learning rate 2 × 10⁻⁴; weight decay 0.05; β = (0.9, 0.95)
Schedule100 epochs; 200 warmup steps; cosine decay; gradient clipping 1.0; BF16 autocast
ValidationAll 28 scenes; fixed 8-view selection from the first 32 prepared frames
Mini-LVSM training loss, validation PSNR and SSIM curves
Training and validation curves. Training uses random clips; validation uses fixed frames from held-out scenes.

Training visualizations

Each row below shows one target view from the first validation sample. Even at epoch 96, predictions capture mostly room colors and rough layout; edges and textures remain blurred.

Source-view and target-ray ablations

Using the best checkpoint, I evaluate 1, 2, and 4 source views with correct, shuffled, or zero target rays. Shuffling replaces each target's rays with another target's rays. Weights stay fixed, and PSNR/SSIM are averaged over all target images.

Q2: all nine validation settings
SourcesTarget raysPSNR ↑ (dB)SSIM ↑Targets
1Correct15.20070.5579196
1Shuffled11.79940.4870196
1None (zeros)12.47400.5286196
2Correct16.13710.5600168
2Shuffled11.47410.4718168
2None (zeros)12.44650.5269168
4Correct16.85870.5749112
4Shuffled11.57260.4729112
4None (zeros)12.46520.5369112

With correct rays, PSNR increases from 15.20 to 16.14 to 16.86 dB for 1, 2, and 4 sources, even though the model was trained with two. The target sets also change (196 / 168 / 112 images), so these scores do not measure the effect of source count alone.

At two sources, shuffled rays reduce PSNR by 4.66 dB and zero rays by 3.69 dB. Incorrect rays hurt more than missing rays at every source count. With zero rays, all targets in a sample have identical queries and produce the same image.

The gap between overfit and held-out quality is large: the model memorizes one batch but produces blurry images on unseen scenes. I did not run the Q2.3 bonus.

Q3. Multi-View Geometry with Mini-VGGT (40 points)

3.0. Model and losses

A frozen DINOv2 ViT-B/14 encodes the RGB images. Eight frame/global attention pairs (width 384, six heads) mix features within each view and then across views. The camera head predicts world-to-camera translation, an XYZW quaternion, and field of view. Decoded patches are refined with RGB to predict depth; a separate projection predicts confidence. I unproject the predicted depth using the predicted cameras to obtain points.

The loss is 5 × camera loss + depth loss. Camera loss combines translation L1 (clamped at 100 per element), raw quaternion L1, and 0.5 × FoV L1. The depth head uses exp(raw depth) and confidence 1 + exp(raw confidence). Depth loss trims the regression and confidence terms separately at the 98th percentile, then adds masked gradient losses at strides 1, 2, 4, and 8. The −0.2 log(confidence) term allows depth and total losses to become negative.

3.1. Single-scene overfit (10 points)

I fit one eight-view training clip for 40 epochs × 25 updates = 1,000 steps, using learning rate 10⁻³, weight decay 0.01, no warmup, and cosine decay to 5% of the initial rate. DINO stays frozen. Epoch 40 has the lowest loss.

Q3 overfit losses
LossBefore training (epoch 0)After 1,000 updates
Total5.4835−0.177082
Camera0.3423890.002675
Depth3.77154−0.190456
Mini-VGGT single-scene overfit total, camera and depth loss curves
Total, camera, and depth loss on the fixed training clip.
Q3 overfit metrics
MetricFixed training clip
Depth AbsRel ↓0.035243
Depth δ1 ↑0.957640
Mean rotation / translation-direction error ↓0.1757° / 0.2876°
Joint pose AUC @ 5° / 15° / 30° ↑0.992857 / 0.997619 / 0.998810
Mini-VGGT overfit GT and predicted geometry, with RGB and depth maps
GT (left) and prediction (right) after 1,000 updates, with the same bounds and viewpoint. RGB and depth maps are shown below.

The camera trajectory and room shape are close to GT, but chair and table boundaries remain smoother. Camera 0 and scene scale are predicted without fitting a pose or scale alignment to GT. These scores measure fit to the training clip.

3.2. Full training and held-out evaluation (30 points)

Full training uses 1,000 epochs / 14,000 updates on 224 scenes. I select epoch 979 by validation loss. On two RTX 6000 Ada GPUs, training and viewer generation take about 2 hours 44 minutes; training/validation compute accounts for 7,310.96 seconds.

Q3 training settings
SettingMini-VGGT baseline
Encoder / transformerFrozen DINOv2 ViT-B/14; eight frame/global attention pairs; width 384; six heads
Parameters144,253,394 total; 57,672,914 trainable
Data8 views; sampling range 32; 224 × 224; seed 825; global batch 16 (8 per GPU)
OptimizerAdamW; learning rate 6 × 10⁻⁴; weight decay 0.01
Schedule1,000 epochs; 10 warmup epochs; cosine decay to 5%; BF16 autocast
SelectionMinimum validation loss; best epoch 979 (distributed validation loss 0.838922)
EvaluationFull validation and test splits; 8 views from the first 32 prepared frames; 224px; seed 825
Mini-VGGT normal-training and validation loss curves for total, camera and depth
Training and validation losses across 1,000 epochs.
Q3 full-split metrics (best checkpoint)
MetricValidationTest
Depth AbsRel ↓0.1377300.167025
Depth δ1 ↑0.8535220.790986
Rotation AUC @ 5° ↑0.1716840.171684
Rotation AUC @ 15° ↑0.4763610.479507
Rotation AUC @ 30° ↑0.6694300.678869
Translation AUC @ 5° ↑0.0068880.003827
Translation AUC @ 15° ↑0.0603740.046514
Translation AUC @ 30° ↑0.1781460.154932
Joint pose AUC @ 5° ↑0.0012760.001020
Joint pose AUC @ 15° ↑0.0346090.034184
Joint pose AUC @ 30° ↑0.1484690.134099

AbsRel is mean |prediction − GT| / GT over valid depth pixels. δ1 is the fraction with max(prediction / GT, GT / prediction) < 1.25. Pose AUC uses relative camera pairs; translation direction allows sign ambiguity, and joint pose uses the larger rotation or translation error. AUC and δ1 are reported as fractions. Each split has 28 scenes and 784 nondegenerate camera pairs, with 10,347,720 / 10,517,881 valid depth pixels on validation/test.

Re-evaluating the best checkpoint gives validation loss 0.837495, close to the distributed value of 0.838922. Batch grouping affects the trimmed loss, and BF16 rounding can slightly change pose AUC.

Test δ1 is 6.25 percentage points below validation. Translation direction is the larger pose error: 42.35° / 40.41° on validation/test, versus rotation errors of 12.53° / 11.20°, which limits joint pose AUC. Training uses random clips and validation uses fixed frames, so their losses reflect different samples.

Four held-out scene comparisons

The first four validation scenes are shown below. Each GT/prediction pair uses the same eight frames, viewpoint, axis bounds, aspect ratio, and depth colors. I use the GT-valid mask for depth maps and point clouds, with no pose/scale alignment or confidence filtering.

Camera errors and smooth depth can both cause surfaces from different views to misalign. These plots do not separate their contributions.

Q3.3. Bonus: How Many Input Views? (+10 points)

Setup

I compare 2, 4, and 8 input views using the epoch-979 checkpoint, which was trained with eight views. For each of the 28 validation scenes, smaller inputs are prefixes of the same eight selected frames. Weights, resolution, frame selection, and seed stay fixed.

I score the same first two views and camera pair at every input count: 2,588,890 valid depth pixels and 28 pose pairs. This keeps the targets fixed while adding input views.

Normalization and evaluated views

The loader normalizes each selected set by its centroid distance, so its scale sV depends on the input count. I multiply predicted depth and translation by sV / s8 to compare them with the eight-view reference GT. This is a unit conversion using the dataset scales, with no fitted alignment. Scaling predictions and targets together leaves relative depth errors and pose angles unchanged. The model receives only RGB, and results are not filtered by confidence.

The result JSON also includes metrics over all input views. Those use different target pixels and camera pairs for each input count, so the table below uses the common first two views.

Bonus view-count comparison of accuracy, runtime and memory
Accuracy on the same two views, plus forward runtime and GPU memory for 2, 4, and 8 inputs.
Bonus: identical target pixels and pose pairs
Common-view metric2 inputs4 inputs8 inputs
Depth AbsRel ↓0.1268240.1368220.133291
Depth δ1 ↑0.8408290.8428780.871791
Rotation AUC @ 5° ↑0.2000000.2000000.192857
Rotation AUC @ 15° ↑0.4904760.4761900.488095
Rotation AUC @ 30° ↑0.6678570.6607140.658333
Translation AUC @ 5° ↑0.0357140.0000000.007143
Translation AUC @ 15° ↑0.1190480.0928570.092857
Translation AUC @ 30° ↑0.2154760.2166670.209524
Joint pose AUC @ 5° ↑0.0000000.0000000.000000
Joint pose AUC @ 15° ↑0.0714290.0714290.066667
Joint pose AUC @ 30° ↑0.1797620.1892860.184524

Going from two to eight inputs raises δ1 by 3.10 percentage points, but worsens AbsRel from 0.126824 to 0.133291. The 90th-percentile relative error improves (0.3018 → 0.2546), while the 99th percentile worsens (0.6380 → 0.8893). Errors above 0.5 increase from 2.49% to 4.35%: more pixels fall within the δ1 threshold, but large errors become worse. At 30°, four inputs give the best joint pose AUC and two give the best rotation AUC.

Runtime and memory

I time BF16 forward passes on a shared RTX 6000 Ada with batch size one and RGB already on the GPU. Each input count gets five warmups per scene and 40 timed passes (four scenes × ten repeats), in randomized order. Timing excludes loading, transfer, and metric computation.

Bonus runtime and PyTorch tensor memory
InputsForward median [IQR], msPeak allocated, MiBIncremental peak, MiB
226.701 [26.539, 27.132]589.1218.83
426.689 [26.573, 26.917]605.0834.80
826.573 [26.461, 26.758]640.3370.04

The timing IQRs overlap, so there is no clear speed difference on this shared GPU. From two to eight views, peak allocated memory rises by 8.7% and the increase above resident model/input memory grows 3.72×. These are PyTorch tensor allocations; they exclude reserved memory, CUDA context, and other processes.

Variation across scenes

I bootstrap the 28 scenes 10,000 times with replacement (seed 825). Each scene has equal weight here, whereas the main depth scores weight every valid pixel equally. All 95% confidence intervals for the changes include zero.

Bonus paired scene analysis
ComparisonMetricMean change95% paired bootstrap CIScenes improved
4 minus 2 inputsAbsRel+0.010369[-0.014640, +0.038634]14/28
4 minus 2 inputsδ1+0.000659[-0.049558, +0.060139]12/28
8 minus 2 inputsAbsRel+0.006474[-0.021444, +0.039140]15/28
8 minus 2 inputsδ1+0.029580[-0.013627, +0.097112]15/28

Eight inputs improve AbsRel in 15/28 scenes. In the worst case, 88627b561e, AbsRel rises from 0.4247 to 0.7688. More views do not help consistently across scenes.

Depth, error maps, and geometry

For the first four validation scenes, I show reference-view depth and relative error, followed by point clouds from the same two views. Each comparison shares the GT-valid mask, viewpoint, bounds, and color limits. Depth colors clip at the GT 95th percentile and error colors at 0.5; metrics use unclipped values.

Extra views sometimes help depth, but the model was trained only with eight inputs. Using fewer views changes both the attention inputs and the selected-view normalization, which may explain some of the uneven results. This experiment covers one checkpoint, one seed, and 28 common pose pairs; it does not separate these effects.

Scoring all eight views gives AbsRel 0.137732 and δ1 0.853530, close to the Q3.2 baseline. The small difference is consistent with BF16 evaluation at batch size one instead of 16.

Result records

Full-precision metrics, run settings, and bonus measurements are linked below.