Assignment 4: Multi-view learning

lnayak

1. Multi-view data preprocessing

Depth is optical-axis z-depth. I unproject each pixel with the inverse intrinsics and transform it with the camera-to-world pose. Canonicalization expresses all cameras and points in camera 0's frame, then divides translations, depth and points by the valid-point centroid distance. Camera 0 stays at the origin; the centroid is not subtracted.

Ray directions are rotated into the world frame and normalized. The Plücker encoding is [d, o × d], where o is the camera center and d is the unit world-space ray direction. The moment o × d records the ray's offset from the origin and stays the same when o moves along the ray. Both stages below use the same five training scenes and frame selections.

036bce3393_frame_004470: original world coordinates
036bce3393_frame_004470: original world coordinates
Canonical coordinates; camera 0 and centroid distance are shown
Canonical coordinates; camera 0 and centroid distance are shown
076c822ecc_frame_002890: original world coordinates
076c822ecc_frame_002890: original world coordinates
Canonical coordinates; camera 0 and centroid distance are shown
Canonical coordinates; camera 0 and centroid distance are shown
079a326597_frame_007430: original world coordinates
079a326597_frame_007430: original world coordinates
Canonical coordinates; camera 0 and centroid distance are shown
Canonical coordinates; camera 0 and centroid distance are shown
07f5b601ee_frame_003630: original world coordinates
07f5b601ee_frame_003630: original world coordinates
Canonical coordinates; camera 0 and centroid distance are shown
Canonical coordinates; camera 0 and centroid distance are shown
07ff1c45bb_frame_000460: original world coordinates
07ff1c45bb_frame_000460: original world coordinates
Canonical coordinates; camera 0 and centroid distance are shown
Canonical coordinates; camera 0 and centroid distance are shown
check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011141905270051211,
      "max_ray_reprojection_error_px": 0.00010764275066321716,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 0.00010235906665911898,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010924567322945222,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

2. Novel-view synthesis with Mini-LVSM

Overfitting

After 2,000 updates on one fixed batch, final overfit PSNR was 30.97 dB. This reached the assignment's approximate 30 dB target.

Overfit: source views, ground truth, prediction
Overfit: source views, ground truth, prediction

The overfit predictions reproduce the table, chairs, wall colors and target viewpoints. Edges are softer than GT, and a grid pattern remains visible across the wall and furniture.

Training and validation

Training settingValue
Training300 epochs (100 initial + 200 continued)
OptimizerAdamW, learning rate 0.0002, weight decay 0.05
Batch size4
Views / sampling window2 source + 6 target views / 32 frames
Resolution224 × 224
TransformerWidth 384, 8 layers
Patch size8
PrecisionBF16 autocast on CUDA
Warmup200 updates
Seed825

Both LVSM models used 100 initial epochs followed by 200 continued epochs. The provided resume path restores the model and optimizer, extends the cosine schedule to 300 epochs and advances it to the saved step. The same continuation procedure was used for both models.

At 100 epochs, baseline training and validation PSNR were 15.61 and 16.14 dB. At 300 epochs they were 16.22 and 16.81 dB. The blurry initial predictions motivated the longer run.

Training and validation loss and PSNR: 100 initial + 200 continued epochs
Training and validation loss and PSNR: 100 initial + 200 continued epochs
Last saved training validation image, epoch 295: sources, ground truth, prediction
Last saved training validation image, epoch 295: sources, ground truth, prediction
Source viewsTarget raysPSNR (dB)SSIM
1correct15.69890.5776
1shuffled11.68280.4818
1none12.43230.5269
2correct16.83610.5833
2shuffled11.29160.4612
2none12.42770.5254
4correct17.89880.6007
4shuffled11.31720.4577
4none12.45810.5362
1 source views, correct target rays; sources, ground truth, prediction
1 source views, correct target rays; sources, ground truth, prediction
1 source views, shuffled target rays; sources, ground truth, prediction
1 source views, shuffled target rays; sources, ground truth, prediction
1 source views, none target rays; sources, ground truth, prediction
1 source views, none target rays; sources, ground truth, prediction
2 source views, correct target rays; sources, ground truth, prediction
2 source views, correct target rays; sources, ground truth, prediction
2 source views, shuffled target rays; sources, ground truth, prediction
2 source views, shuffled target rays; sources, ground truth, prediction
2 source views, none target rays; sources, ground truth, prediction
2 source views, none target rays; sources, ground truth, prediction
4 source views, correct target rays; sources, ground truth, prediction
4 source views, correct target rays; sources, ground truth, prediction
4 source views, shuffled target rays; sources, ground truth, prediction
4 source views, shuffled target rays; sources, ground truth, prediction
4 source views, none target rays; sources, ground truth, prediction
4 source views, none target rays; sources, ground truth, prediction

Correct rays recover coarse window, wall and furniture placement, but predictions remain blurry and show a patch grid. Four source views make the bright lamp region clearer in this scene, while thin chair legs and shelf details are still missing. Shuffled rays retain the scene's broad appearance but place it in the wrong target view. Zero rays produce the same blurry image for every target: with fixed sources, all target-query inputs are then identical.

With correct target rays, PSNR for 1, 2 and 4 source views was 15.70, 16.84 and 17.90 dB. The 4-view minus 1-view difference was +2.20 dB. The target set changes with the source count, so this comparison also changes which images are scored.

At 1 source views, correct rays changed PSNR by +4.02 dB relative to shuffled rays and +3.27 dB relative to zero rays. At 2 source views, correct rays changed PSNR by +5.54 dB relative to shuffled rays and +4.41 dB relative to zero rays. At 4 source views, correct rays changed PSNR by +6.58 dB relative to shuffled rays and +5.44 dB relative to zero rays.

Bonus: model capacity

How does reducing transformer depth affect novel-view synthesis? I compare the baseline and smaller model after 300 epochs using the same validation source counts and correct target rays.

Training settingValue
Training300 epochs (100 initial + 200 continued)
OptimizerAdamW, learning rate 0.0002, weight decay 0.05
Batch size4
Views / sampling window2 source + 6 target views / 32 frames
Resolution224 × 224
TransformerWidth 384, 4 layers
Patch size8
PrecisionBF16 autocast on CUDA
Warmup200 updates
Seed825
Changed settingBaselineSmall model
TransformerWidth 384, 8 layersWidth 384, 4 layers
ModelTrainable parametersForward ms / scenePeak allocated MiBTraining + validation hours
baseline1463340850.3325439.36911.1451
small753913625.0872412.66210.7701

Timing used NVIDIA RTX 6000 Ada Generation, FP32; autocast and TF32 disabled; eval mode; no_grad, one shared scene, two source views and six target views at 224 pixels. Each measurement used five warmup passes and 20 timed forwards. Forward time excludes loading, transfers and metrics. Peak allocated memory includes model, inputs and output; logged epoch time includes training and validation.

The smaller model took 0.498 times the baseline forward time and changed peak allocated memory by -26.7 MiB. This timing covers one scene on one device.

Source viewsBaseline PSNRSmall PSNRDifference (dB)Baseline SSIMSmall SSIM
115.698915.6055-0.09340.57760.5741
216.836116.7377-0.09840.58330.5802
417.898817.6036-0.29520.60070.5973
Small-model training and validation curves: 100 initial + 200 continued epochs
Small-model training and validation curves: 100 initial + 200 continued epochs
Small model, 1 sources: source views, ground truth, prediction
Small model, 1 sources: source views, ground truth, prediction
Small model, 2 sources: source views, ground truth, prediction
Small model, 2 sources: source views, ground truth, prediction
Small model, 4 sources: source views, ground truth, prediction
Small model, 4 sources: source views, ground truth, prediction

The smaller model captures broad window, wall and floor colors. With four source views, the bright lamp region is more visible in some predictions, but chair legs, shelf contents and textures remain blurred. A patch grid is still visible. More source images do not resolve the missing fine detail in this example.

The smaller model changed PSNR by -0.09, -0.10, -0.30 dB at 1, 2 and 4 source views. It scored higher in 0 of the three settings. Reduced capacity may limit reconstruction detail, while fewer layers can also change optimization. A single run per model cannot separate these effects or establish variation across random seeds.

Bonus: loss function

Does changing the reconstruction loss improve image quality? These experiments all use 100 epochs, the 8-layer model, AdamW at 0.0002, batch size 4 and seed 825. The baseline in these tables is the saved 100-epoch MSE run. The 300-epoch model above is not used as the control here. Checkpoints are selected by validation PSNR.

The perceptual term averages L1 feature differences at VGG16 relu1_2, relu2_2, relu3_3 and relu4_3. VGG16 is pretrained on ImageNet and stays frozen. All reported image losses are MSE, regardless of the training objective.

Training lossSource viewsPSNRSSIM
MSE115.19280.5583
MSE216.14460.5605
MSE416.87790.5749
MSE + 0.1 L1115.22890.5584
MSE + 0.1 L1216.24580.5622
MSE + 0.1 L1416.99080.5781
MSE + 0.2 (1 - SSIM)114.78500.5764
MSE + 0.2 (1 - SSIM)215.44110.5799
MSE + 0.2 (1 - SSIM)415.83960.5935
MSE + 0.05 perceptual114.68760.5648
MSE + 0.05 perceptual215.29340.5650
MSE + 0.05 perceptual415.60710.5763
Matched sources and target for the four loss functions, all trained for 100 epochs
Matched sources and target for the four loss functions, all trained for 100 epochs

At two source views, the changes from MSE were MSE + 0.1 L1: +0.10 dB PSNR and +0.0016 SSIM. MSE + 0.2 (1 - SSIM): -0.70 dB PSNR and +0.0194 SSIM. MSE + 0.05 perceptual: -0.85 dB PSNR and +0.0045 SSIM.

Training lossPredicted edge magnitudeGT edge magnitudeEdge error
MSE0.00600.01820.0208
MSE + 0.1 L10.00620.01820.0209
MSE + 0.2 (1 - SSIM)0.00280.01820.0187
MSE + 0.05 perceptual0.00550.01820.0202

Edge magnitude averages absolute horizontal and vertical RGB differences over every target image. Edge error measures the difference from the GT image gradients. The table quantifies changes in sharpness, while the matched images show their appearance. Higher edge magnitude can reflect detail or patch-grid artifacts, so it is not an overall quality score. This tests one weight per auxiliary loss and one random seed.

Bonus: training source-view count

I compare two fixed source views with a source count sampled uniformly from 1 to 4 at each training step. Each batch still has eight views; the remaining views are targets. The sampled counts are saved with the run and can be resumed.

Training source countUpdates
11448
21361
31391
41400
Source viewsBaseline PSNRRandom sources PSNRBaseline SSIMRandom sources SSIM
115.192815.23300.55830.5585
216.144616.17530.56050.5593
416.877917.01950.57490.5740

Random-source training changed PSNR by +0.04, +0.03, +0.14 dB at 1, 2 and 4 evaluation sources. It exposes the model to all tested source counts, but also changes the number of supervised targets per update. This experiment does not separate those effects.

Bonus: camera encoding

I replace the six Plücker channels with 16 camera values: the 3 by 3 camera-to-world rotation, three normalized translations and fx/W, fy/H, cx/W, cy/H. These values are repeated across pixels before patch projection. Source tokens still contain RGB, while target tokens contain only the camera encoding. The transformer and training schedule are unchanged; the input projections have more parameters.

Source viewsBaseline PSNRRaw cameras PSNRBaseline SSIMRaw cameras SSIM
115.192815.15170.55830.5582
216.144615.99330.56050.5575
416.877916.68260.57490.5708

Raw camera parameters changed PSNR by -0.04, -0.15, -0.20 dB at 1, 2 and 4 sources. Unlike Plücker rays, the repeated camera vector does not explicitly describe each pixel's viewing ray. The network must learn that mapping using the positional embedding. This comparison also changes projection size.

Fixed-source Plücker baseline, random-source training and raw-camera encoding at the same target
Fixed-source Plücker baseline, random-source training and raw-camera encoding at the same target
Train and validation PSNR for the six matched 100-epoch experiments
Train and validation PSNR for the six matched 100-epoch experiments

3. Multi-view geometry with Mini-VGGT

Overfitting

Training settingValue
Training40 epochs, 1,000 updates
OptimizerAdamW, learning rate 0.001, weight decay 0.01
Batch size1 per update from 1 fixed clip
Views / sampling window8 views / 32 frames
Resolution224 × 224
TransformerWidth 384, 8 frame/global pairs (16 blocks)
Patch size14
PrecisionBF16 autocast
Warmup0 epochs
Seed825
EncoderFrozen DINOv2 ViT-B/14
Loss weightsCamera 5, depth 1
GPUs1
Overfit total, camera and depth loss
Overfit total, camera and depth loss
Saved overfit clip 036bce3393_frame_004470: GT and prediction
Saved overfit clip 036bce3393_frame_004470: GT and prediction

Full-resolution image

The overfit room layout and camera path closely match GT. Some surfaces appear thicker, with scattered points around the upper wall edges.

Overfit total loss changed from 5.4830 to -0.1688. The render uses the best saved checkpoint at epoch 40; this fixed training clip does not measure generalization.

The depth objective includes a negative log-confidence term, so depth and total loss can fall below zero.

Training and held-out evaluation

Training settingValue
Training1,000 epochs, 14,000 updates
OptimizerAdamW, learning rate 0.0006, weight decay 0.01
Batch size16 total (4 per GPU)
Views / sampling window8 views / 32 frames
Resolution224 × 224
TransformerWidth 384, 8 frame/global pairs (16 blocks)
Patch size14
PrecisionBF16 autocast
Warmup10 epochs
Seed825
EncoderFrozen DINOv2 ViT-B/14
Loss weightsCamera 5, depth 1
GPUs4
Total, camera and depth loss over 1,000 epochs
Total, camera and depth loss over 1,000 epochs

Evaluation uses eight views from the first 32 prepared frames, 224-pixel images and seed 825, across all 28 validation and 28 test scenes. The table reports AUC in [0, 1]. Translation and joint-pose AUC exclude pairs with undefined ground-truth translation direction.

MetricValidationTest
Depth AbsRel0.13640.1655
Depth δ10.82920.7710
Rotation AUC 5°0.19440.1809
Rotation AUC 15°0.51830.4929
Rotation AUC 30°0.70720.7116
Translation AUC 5°0.01150.0156
Translation AUC 15°0.08440.0759
Translation AUC 30°0.19460.1844
Joint pose AUC 5°0.00230.0038
Joint pose AUC 15°0.06400.0506
Joint pose AUC 30°0.16920.1663

Validation depth AbsRel was 0.1364, compared with 0.1655 on test. Depth δ1 was 0.8292 and 0.7710. These depth and pose errors can both affect the reconstructed cloud. The four examples below show individual scenes and do not represent the full error distribution.

Translation-direction accuracy is the main pose weakness: validation AUC at 15 degrees is 0.0844, versus 0.5183 for rotation. Camera errors can misalign points from different views even when individual depth maps are reasonable. The held-out clouds retain broad structure, but are less consistent than the overfit example.

GT and predicted geometry use the same camera, bounds and GT-valid pixels in each comparison. Predictions are neither aligned nor rescaled to fit GT; bounds include outliers.

09c1414f1b_frame_000000: GT and predicted geometry. The broad room shape is retained, but walls and furniture appear thicker.
09c1414f1b_frame_000000: GT and predicted geometry. The broad room shape is retained, but walls and furniture appear thicker.

Full-resolution image

21d970d8de_frame_000000: GT and predicted geometry. The main white surface is close to GT, while the upper structure is compressed and noisy.
21d970d8de_frame_000000: GT and predicted geometry. The main white surface is close to GT, while the upper structure is compressed and noisy.

Full-resolution image

280b83fcf3_frame_000000: GT and predicted geometry. The tan surfaces bend and extend beyond the GT structure.
280b83fcf3_frame_000000: GT and predicted geometry. The tan surfaces bend and extend beyond the GT structure.

Full-resolution image

2b1dc6d6a5_frame_000000: GT and predicted geometry. Several surfaces are doubled, and the predicted camera path has a different spread.
2b1dc6d6a5_frame_000000: GT and predicted geometry. Several surfaces are doubled, and the predicted camera path has a different spread.

Full-resolution image

Bonus: number of input views

I evaluate the same checkpoint with 2, 4, 8, 16 and 32 input views on both complete held-out splits. The frame selections are nested prefixes. Common-view metrics always score the first two views and their camera pair. Both predictions and GT receive the dataset-derived conversion to the two-view canonical scale; no alignment is fitted. Aggregate scores include every input view and pair, so their evaluated content changes with view count.

Validation

Common-view metric2 inputs4 inputs8 inputs16 inputs32 inputs
Depth AbsRel0.12430.13330.13200.17120.1859
Depth δ10.81930.83040.83300.80240.7932
Rotation AUC 5°0.27140.18570.18570.18570.1643
Rotation AUC 15°0.58330.55000.55000.50950.4738
Rotation AUC 30°0.72380.71430.73100.69880.6738
Translation AUC 5°0.00000.00000.00710.00710.0071
Translation AUC 15°0.12860.09290.10480.12140.1262
Translation AUC 30°0.29520.22260.25480.25120.2702
Joint pose AUC 5°0.00000.00000.00000.00000.0000
Joint pose AUC 15°0.11190.06430.07380.07620.0667
Joint pose AUC 30°0.26190.19050.22140.21550.2214
Input viewsAggregate AbsRelAggregate δ1Aggregate joint AUC 30°Forward msPeak allocated MiB
20.12430.81930.261929.5512706.6514
40.13130.83720.158727.1984724.3076
80.13640.82910.169734.2974762.5625
160.17530.79120.139430.5714842.0952
320.18770.76930.134936.9486986.7690

On val, using 32 instead of 8 inputs changed common-view AbsRel by +0.0539, δ1 by -0.0397 and joint-pose AUC at 30 degrees by -0.0000.

Test

Common-view metric2 inputs4 inputs8 inputs16 inputs32 inputs
Depth AbsRel0.11950.13530.15690.16770.1737
Depth δ10.85800.83700.77580.74880.7467
Rotation AUC 5°0.25000.25000.23570.25710.2500
Rotation AUC 15°0.49050.51190.54050.50950.5167
Rotation AUC 30°0.65950.70120.70830.69880.7000
Translation AUC 5°0.00000.00710.04290.00000.0071
Translation AUC 15°0.04520.11900.12140.08810.0857
Translation AUC 30°0.15120.25120.25120.26190.2190
Joint pose AUC 5°0.00000.00710.02140.00000.0000
Joint pose AUC 15°0.00950.04290.07380.03570.0476
Joint pose AUC 30°0.10360.18210.19050.20120.1679
Input viewsAggregate AbsRelAggregate δ1Aggregate joint AUC 30°Forward msPeak allocated MiB
20.11950.85800.103626.8002706.6514
40.14720.81580.174827.9886724.3076
80.16550.77100.165734.1045762.5625
160.18060.75300.157228.2735842.0952
320.18720.74860.157437.0324986.7690

On test, using 32 instead of 8 inputs changed common-view AbsRel by +0.0169, δ1 by -0.0291 and joint-pose AUC at 30 degrees by -0.0226.

2 input views: GT and predicted geometry for the same validation scene
2 input views: GT and predicted geometry for the same validation scene

Full-resolution image

4 input views: GT and predicted geometry for the same validation scene
4 input views: GT and predicted geometry for the same validation scene

Full-resolution image

8 input views: GT and predicted geometry for the same validation scene
8 input views: GT and predicted geometry for the same validation scene

Full-resolution image

16 input views: GT and predicted geometry for the same validation scene
16 input views: GT and predicted geometry for the same validation scene

Full-resolution image

32 input views: GT and predicted geometry for the same validation scene
32 input views: GT and predicted geometry for the same validation scene

Full-resolution image

Timing used NVIDIA RTX 6000 Ada Generation, bf16 autocast and batch size 1, with five warmup passes and 20 measured passes. It includes the encoder and prediction heads, but excludes loading, transfers, metrics and rendering. Memory is peak allocated CUDA memory, including the model and inputs. Each split's timing uses its first scene.

The checkpoint was trained with eight views. Other counts change its input distribution as well as the available context. Extra views need not improve every output, and these single-scene timings do not measure uncertainty or general throughput.

Bonus: direct point-map prediction

I compare direct XYZ prediction with reconstruction from cameras and depth in the same joint model at epoch 350. The point head projects patch tokens into signed XYZ maps and adds an RGB-guided convolutional residual. Its loss is mean absolute XYZ error on GT-valid pixels, using the supplied 98 percent trimming. Camera, depth and point weights are 5, 1 and 1.

This is an intermediate-checkpoint study: 350 epochs from the original 1,000-epoch schedule, not a completed 1,000-epoch run or a 350-epoch cosine schedule. Training uses frozen DINOv2 ViT-B/14, eight frame/global pairs of width 384, eight views from a 32-frame window, global batch size 16, AdamW at 0.0006 and seed 825.

SplitReconstructionXYZ L1Point L2Point L2 (m)
valCamera + depth0.15660.32280.6586
valDirect points0.15390.31680.6464
testCamera + depth0.15240.30510.5953
testDirect points0.15320.30470.5931

Evaluation uses the same GT-valid pixels without trimming, confidence filtering or fitted alignment. L1 averages absolute XYZ error; L2 is Euclidean distance. Conversion to meters uses the dataset's scene scale.

On val, direct prediction changed mean point distance by -0.0121 m relative to camera/depth reconstruction. Lower is better.

On test, direct prediction changed mean point distance by -0.0023 m relative to camera/depth reconstruction. Lower is better.

Epoch 350: ground truth, reconstructed camera/depth points and direct point predictions
Epoch 350: ground truth, reconstructed camera/depth points and direct point predictions

Full-resolution image

Direct prediction avoids combining separate camera and depth errors, but does not enforce ray consistency. These intermediate checkpoints show current behavior, not the final performance after convergence.

Bonus: multi-task synergy

Does joint supervision help camera estimation? I compare camera-only training with joint camera/depth/point training at exactly 350 epochs. Both use the same architecture, seed, batch size, sampling and learning-rate schedule. The inactive depth head is frozen in the camera-only model. I compare camera metrics only, using fixed-epoch checkpoints rather than separate best-checkpoint selection.

SplitSupervisionRotation AUC 5°Rotation AUC 15°Rotation AUC 30°Translation AUC 5°Translation AUC 15°Translation AUC 30°Joint pose AUC 5°Joint pose AUC 15°Joint pose AUC 30°
valCamera only0.15050.42090.60530.00590.04330.11050.00380.03140.0948
valCamera + depth + points0.14260.41730.61390.00360.04120.11920.00100.02960.1042
testCamera only0.14130.43170.64320.00260.03220.11480.00100.02020.0958
testCamera + depth + points0.15790.44630.65880.00560.03390.11360.00100.01630.0934

On val, joint supervision changed rotation AUC at 15 degrees by -0.0037 and joint-pose AUC at 30 degrees by +0.0094.

On test, joint supervision changed rotation AUC at 15 degrees by +0.0145 and joint-pose AUC at 30 degrees by -0.0025.

Epoch 350: GT and predicted camera-center trajectories with common bounds and no alignment
Epoch 350: GT and predicted camera-center trajectories with common bounds and no alignment

This tests transfer to the camera task. It does not establish an effect on depth-only or point-only training. Shared geometry supervision may help camera features, while competing losses can also hurt them. The comparison uses one seed; runs used RTX A6000 and RTX 6000 Ada GPUs, so numerical differences may also contribute.

Bonus: unfreezing DINO

Both models start from the same baseline checkpoint, selected at epoch 781. Each receives 100 additional epochs with a fresh AdamW optimizer, learning rate 0.00002, five warmup epochs, global batch size 16 and the original camera/depth loss. One keeps DINO frozen; the other updates its used parameters. This is a paired fine-tuning experiment, not a comparison of training the encoder from scratch.

SplitEncoderDepth AbsRelDepth δ1Rotation AUC 5°Rotation AUC 15°Rotation AUC 30°Translation AUC 5°Translation AUC 15°Translation AUC 30°Joint pose AUC 5°Joint pose AUC 15°Joint pose AUC 30°
valFrozen0.13740.82770.19290.51800.70750.01120.08440.19380.00230.06390.1686
valUnfrozen0.14630.81650.18600.52480.70770.01710.08950.20860.00480.07240.1886
testFrozen0.16690.77080.18140.49360.71180.01580.07600.18400.00380.05030.1656
testUnfrozen0.17330.78160.18370.51730.72160.00840.05740.16370.00000.02980.1426

On val, unfreezing changed AbsRel by +0.0089, δ1 by -0.0112 and joint-pose AUC at 30 degrees by +0.0201.

On test, unfreezing changed AbsRel by +0.0064, δ1 by +0.0108 and joint-pose AUC at 30 degrees by -0.0230.

Training and validation losses during the matched 100-epoch fine-tuning runs
Training and validation losses during the matched 100-epoch fine-tuning runs
Same validation scene: GT, frozen encoder after fine-tuning, unfrozen encoder after fine-tuning
Same validation scene: GT, frozen encoder after fine-tuning, unfrozen encoder after fine-tuning

Full-resolution image

The saved frozen encoder is unchanged from initialization, while the unfrozen encoder has changed weights. Updating the encoder can adapt image features to this data, but can also overfit a small training set. These results depend on the chosen starting checkpoint and fine-tuning rate.