CMU 16-825 · Learning for 3D Vision · Fall 2026

Learning geometry from multiple views

Assignment 4 · Hongyi (Sam) Dong

Part 1From depth pixels to camera rays

I resized the camera intrinsics to the image grid, scaling each focal length along its own axis and shifting the principal point with the pixel-center rule c′ = s(c + 0.5) − 0.5. For a pixel p = [u, v, 1]ᵀ with valid z-depth z, the camera-space point is zK⁻¹p. Applying the camera-to-world pose puts the colored points from all eight views into one room.

Unprojection · points and cameras in meters
Unprojection · points and cameras in meters
Canonicalization · first camera at the origin, centroid distance 1.000
Canonicalization · first camera at the origin, centroid distance 1.000

I expressed every pose relative to the first camera as T₀⁻¹Tᵢ, then divided depth, point coordinates and camera translations by the shared scene scale. In this clip the scale is 1.867 m. The first camera center is (0, 0, 0), and the centroid of the valid points lies one normalized unit away.

Each pixel ray starts at camera center o and points along the unit direction d = normalize(RK⁻¹p). Its six Plücker channels are [d, o × d]. The moment o × d locates the line relative to the origin and stays the same if o moves along that line. On four training clips the maximum point and ray reprojection errors were both below 0.00013 pixels.

Full data checker output · four training clips
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00010844774078577757,
      "max_ray_reprojection_error_px": 9.41436446737498e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.0001220703125,
      "max_ray_reprojection_error_px": 0.0001078959321603179,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.281559323426336e-05,
      "max_ray_reprojection_error_px": 8.446346328128129e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.00010504279634915292,
      "max_ray_reprojection_error_px": 9.423537994734943e-05,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [
      2,
      8,
      3,
      224,
      224
    ],
    "depth": [
      2,
      8,
      1,
      224,
      224
    ],
    "depth_valid": [
      2,
      8,
      1,
      224,
      224
    ],
    "K": [
      2,
      8,
      3,
      3
    ],
    "frame_numbers": [
      2,
      8
    ],
    "T_world_from_camera": [
      2,
      8,
      4,
      4
    ],
    "original_world_from_reference": [
      2,
      4,
      4
    ],
    "scene_scale": [
      2
    ],
    "points_world": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_origins": [
      2,
      8,
      3,
      224,
      224
    ],
    "ray_directions": [
      2,
      8,
      3,
      224,
      224
    ],
    "plucker": [
      2,
      8,
      6,
      224,
      224
    ]
  }
}

Part 2Novel-view synthesis with Mini-LVSM

2.0 · Predicting an RGB image from target rays

For each source view I combined RGB with its six ray channels and embedded 8 × 8 patches. Each target contributes a separate set of ray-only query tokens. After attention over the sources and one target, the RGB head decodes the target tokens and rearranges the predicted patches into an image.

2.1 · Fitting one training example

After 2,000 updates the fixed overfit example reached 30.99 dB PSNR. The predicted table and chairs follow the target viewpoint; their edges are softer than in the target.

Overfit result · two source views, ground truth, prediction
Overfit result · two source views, ground truth, prediction

2.2 · Training and ray-conditioning ablations

I trained the 14.6M-parameter transformer from scratch for 100 epochs (5,600 updates), using width 384, eight layers, two source views, batch size four and MSE. AdamW used a peak learning rate of 0.0002 with 200 warmup steps followed by cosine decay. Validation PSNR peaked at epoch 99.

Training and validation MSE and PSNR, and validation SSIM over 100 epochs
100-epoch baseline training history
Validation predictions during training · epoch 0
Validation predictions during training · epoch 0
Validation predictions during training · epoch 95
Validation predictions during training · epoch 95
Source viewsTarget raysTarget imagesPSNR (dB) ↑SSIM ↑
1correct19615.200.5581
1shuffled19611.800.4874
1none19612.490.5264
2correct16816.120.5599
2shuffled16811.490.4721
2none16812.460.5256
4correct11216.870.5749
4shuffled11211.590.4726
4none11212.460.5355

With correct rays, PSNR rose from 15.20 dB with one source to 16.87 dB with four. At two sources, giving the model rays from the wrong target cost 4.63 dB; zeroing the rays cost 3.66 dB. The mismatched rays appear to steer the prediction toward the wrong view. The number of held-out targets also changes with source count, so the three view-count settings cover different target sets.

Evaluation images · all nine settings
S1 · correct · sources, ground truth, prediction
S1 · correct · sources, ground truth, prediction
S1 · none · sources, ground truth, prediction
S1 · none · sources, ground truth, prediction
S1 · shuffled · sources, ground truth, prediction
S1 · shuffled · sources, ground truth, prediction
S2 · correct · sources, ground truth, prediction
S2 · correct · sources, ground truth, prediction
S2 · none · sources, ground truth, prediction
S2 · none · sources, ground truth, prediction
S2 · shuffled · sources, ground truth, prediction
S2 · shuffled · sources, ground truth, prediction
S4 · correct · sources, ground truth, prediction
S4 · correct · sources, ground truth, prediction
S4 · none · sources, ground truth, prediction
S4 · none · sources, ground truth, prediction
S4 · shuffled · sources, ground truth, prediction
S4 · shuffled · sources, ground truth, prediction

2.3 · Does an L1 term improve the prediction?

I trained a second model from the same seed with the same 100-epoch schedule, changing the objective to MSE + 0.1 × mean absolute RGB error. I compared the best-validation-PSNR checkpoint of each run on the same 168 target images with two sources and correct rays.

Training objectiveValidation MSE ↓PSNR (dB) ↑SSIM ↑
MSE0.0294616.120.5599
MSE + 0.1 × L10.0289616.230.5616
MSE · two sources, target, prediction
MSE · two sources, target, prediction
MSE + 0.1 × L1 · same validation example
MSE + 0.1 × L1 · same validation example

The added L1 term improved PSNR by about 0.10 dB and SSIM by 0.0017 on the unrounded scores. The overall room layout is recognizable, but the prediction still smooths over fine furniture edges. This single-seed comparison supports a small numerical gain; it gives little evidence of a large change in sharpness.

Part 3Camera and depth estimation with Mini-VGGT

3.0 · Reconstructing a scene from eight images

A frozen DINOv2 ViT-B/14 encoder supplies patch features. Eight transformer pairs alternate attention within each image and across the eight views. The camera head predicts translation, a quaternion and field of view. The depth head predicts positive depth and confidence with RGB-guided refinement. I used a camera loss weight of 5 and a depth loss weight of 1, including confidence-weighted regression and a multiscale depth-gradient term.

3.1 · Single-clip overfit

I trained on one fixed clip for 40 epochs of 25 steps at an initial learning rate of 0.001. The prediction recovers the back wall, central opening and foreground furniture, while some point boundaries remain thicker than in the ground truth.

Single-clip training · total, camera and depth losses
Single-clip training · total, camera and depth losses
Ground truth · training clip
Ground truth · training clip
Prediction · same clip
Prediction · same clip

3.2 · Reconstruction on held-out scenes

Validation PSNR in Part 2 was still improving at epoch 99, so I kept the longer default schedule of 1,000 epochs for both Part 3 experiments. I used eight 224 × 224 views from a 32-frame window, batch size 16 and AdamW with a peak learning rate of 0.0006, ten warmup epochs and cosine decay. I selected the joint checkpoint by minimum validation loss and evaluated it on the full validation and test splits with seed 825 and eight views drawn from the first 32 prepared frames.

Joint training · total, camera and depth losses
Joint training · total, camera and depth losses
Metric · best checkpointValidationTest
Depth AbsRel ↓0.14340.1661
Depth δ1 ↑0.81850.7890
Rotation AUC @ 5° ↑0.18440.1834
Rotation AUC @ 15° ↑0.52550.5024
Rotation AUC @ 30° ↑0.70700.7080
Translation AUC @ 5° ↑0.01280.0094
Translation AUC @ 15° ↑0.08570.0452
Translation AUC @ 30° ↑0.19980.1402
Joint pose AUC @ 5° ↑0.00310.0013
Joint pose AUC @ 15° ↑0.06480.0281
Joint pose AUC @ 30° ↑0.17880.1239

The metric gap between rotation and translation is visible at 15°: validation rotation AUC is 0.5255, while translation-direction AUC is 0.0857. A scene can retain a recognizable floor and walls even when individual camera positions and object boundaries do not align closely.

Scene 1 · 09c1414f1b

Ground truth
Ground truth
Predicted camera and depth
Predicted camera and depth

The table and window occupy the correct region, but the chair legs merge into the table in the predicted depth map.

Scene 2 · 21d970d8de

Ground truth
Ground truth
Predicted camera and depth
Predicted camera and depth

The floor remains continuous. The prediction softens the boundary along the narrow table and the red chair.

Scene 3 · 280b83fcf3

Ground truth
Ground truth
Predicted camera and depth
Predicted camera and depth

The hallway and doorway are visible in both reconstructions. Depth transitions along the door frames are smoother in the prediction.

Scene 4 · 2b1dc6d6a5

Ground truth
Ground truth
Predicted camera and depth
Predicted camera and depth

The office chair is visible, but the seat and armrest have less distinct boundaries than in the ground truth.

3.3 · Does camera supervision help depth?

I compared joint training against an otherwise identical run with the camera loss weight set to zero. Both started with the same initialization, used seed 825 and ran for 1,000 epochs. I evaluated the final checkpoint of each run on the same validation split so that checkpoint selection would not favor either objective.

Supervision · final checkpointDepth AbsRel ↓Depth δ1 ↑
Depth only0.160379.45%
Camera + depth0.144283.06%
Depth accuracy of joint versus depth-only training

Camera supervision reduced relative depth error by about 10% and increased the fraction of valid pixels within the δ1 threshold by 3.61 percentage points. Predicting pose may encourage the shared features to encode cross-view geometry. The comparison used one seed, and camera gradients also affect the shared gradient-clipping step, so this experiment cannot separate those effects.