16-825 · Learning for 3D Vision

Assignment 4: Multi-View Transformers

1. Multi-View Data Preprocessing

1.2 Depth Unprojection

Unprojected points and eight cameras for scene 036bce3393 in world coordinates, with the RGB and depth of view 0
Scene 036bce3393, original world frame (meters). Red frustum: first camera.
Unprojected points and eight cameras for scene 07f5b601ee in world coordinates, with the RGB and depth of view 0
Scene 07f5b601ee, original world frame (meters).

1.3 Canonicalization

Canonicalized points and cameras for scene 036bce3393; first camera center (0, 0, 0), scene scale 1.867 m, centroid distance 1.000
Scene 036bce3393, canonical frame
Canonicalized points and cameras for scene 07f5b601ee; first camera center (0, 0, 0), scene scale 0.924 m, centroid distance 1.000
Scene 07f5b601ee, canonical frame
SceneFirst camera centerCentroid distance
036bce3393(0, 0, 0)1.000
07f5b601ee(0, 0, 0)1.000

1.7 Final Check

SampleMax point reproj. (px)Max ray reproj. (px)Centroid distance
e050c15a8d_frame_0070801.13×10−41.09×10−41.000
5f99900f09_frame_0028201.23×10−41.11×10−41.000
47b37eb6f9_frame_0027200.99×10−40.89×10−41.000
4c5c60fa76_frame_0066901.09×10−41.03×10−41.000
Full check_data.py output
{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001089862416847609,
      "max_ray_reprojection_error_px": 0.00010257320536766201,
      "centroid_distance": 0.9999999403953552
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

Canonicalization, Ray Directions and the Plücker Moment

Canonicalization. Poses are expressed in the first camera's frame, \(T'_i = T_0^{-1} T_i\), and translations, depth and points are divided by the distance of the valid-point centroid. Camera 0 becomes the identity and the centroid lies at distance 1, so every sample shares one frame and scale.

World-space ray directions. \(\mathbf{d} = R\,K^{-1}[u, v, 1]^\top\), normalized to unit length, where \(R\) is the camera-to-world rotation. The ray origin is the camera center \(\mathbf{o} = \mathbf{t}\).

Plücker moment. \(\mathbf{m} = \mathbf{o} \times \mathbf{d}\) is the same for every point on the ray, so \([\mathbf{d}, \mathbf{m}]\) describes the ray itself. \(\lVert\mathbf{m}\rVert\) is the ray's distance from the origin, which is zero for camera 0.

2. Novel-View Synthesis with Mini-LVSM

2.1 Overfitting

Final overfit PSNR: 30.88 dB

Four target views from the overfit batch: two source views, ground truth and prediction per row; the predictions match the ground truth
val_epoch000.png. Columns: source 1, source 2, ground truth, prediction.

2.2 Training and Evaluation

SettingValue
ConfigDefault configs/lvsm.json, unchanged
Views8 per sample: 2 sources, 6 targets
Training100 epochs, batch size 4 (5,600 steps)
OptimizerAdamW, lr 2×10−4, weight decay 0.05; 200-step warmup, cosine decay
ModelPatch 8, width 384, 8 layers; 14.6M parameters
HardwareOne RTX 4090, 30.5 min
Training curves over 100 epochs: MSE falls, PSNR rises from about 12 to 16 dB, validation SSIM rises to 0.56
Training curves

Validation PSNR / SSIM

Source viewsCorrect raysShuffled raysNo rays
115.20 / 0.55811.81 / 0.48812.47 / 0.528
216.14 / 0.56011.49 / 0.47212.44 / 0.527
416.88 / 0.57511.59 / 0.47312.46 / 0.538

Training visualization

Validation predictions at epoch 95: four rows of two source views, ground truth and a blurry prediction
val_epoch095.png. Columns: source 1, source 2, ground truth, prediction.

Evaluation visualizations

Evaluation with one source view and correct rays: source, ground truth, prediction per row
1 source, correct rays. Columns: source, ground truth, prediction.
Evaluation with two source views and correct rays: two sources, ground truth, prediction per row
2 sources, correct rays
Evaluation with four source views and correct rays: four sources, ground truth, prediction per row
4 sources, correct rays
Two sources with shuffled target rays: predictions show the layout of a different target view
2 sources, shuffled rays
Two sources with zero target rays: predictions are a blurry average image
2 sources, no rays

Number of source views. With correct rays PSNR rises from 15.20 to 16.14 to 16.88 dB for 1, 2 and 4 sources. More sources cover more of the target view, so less has to be guessed. The model was trained with 2 sources but also works with 1 and 4, because attention accepts any number of source tokens.

Ray-conditioning ablation. Correct rays beat shuffled rays by 3.4–5.3 dB and zero rays by 2.7–4.4 dB, and only correct rays improve with more sources, so the model uses the target camera. Shuffled rays are worst: the model renders another view's layout in the wrong place, while with zero rays it falls back to a blurry scene average.

2.3 Bonus: Analysis and Exploration

Question. The 2.2 predictions are blurry and show 8×8 blocks. Do a decoder that sees across patches and a perceptual loss (LPIPS) fix this, and how do PSNR, SSIM and sharpness change?

Settings

  • LVSM+ (bonus/lvsm_plus.py): the linear head outputs 3 + 16 channels per pixel, followed by four 3×3 conv layers that cross patch borders. Width 768, 12 layers (86.8M parameters), batch size 8, lr 3×10−4, 400 epochs. The MSE+LPIPS run was then resumed to 1000 epochs (learning-rate schedule restarted at epoch 400).
  • Controlled pair: LVSM+ MSE vs. LVSM+ MSE+LPIPS(VGG), where the LPIPS weight is 0 until epoch 60 and ramps to 0.1 by epoch 120. Everything else is identical.
  • Reference: the 2.2 model trained for 1000 epochs (baseline long), which takes the same wall-clock time as 400 epochs of LVSM+ (about 2.7 h).
  • Extra metrics: LPIPS(AlexNet); sharpness = mean image gradient of prediction / GT; blockiness = gradient across 8-px patch borders / gradient inside patches (≈1 means no seams).

Results

Model (val, 2 sources)EpochsPSNR ↑SSIM ↑LPIPS ↓SharpnessBlockiness
Baseline (2.2)10016.140.5600.6720.337.97
Baseline long100017.620.6120.5660.423.43
LVSM+ MSE40017.000.6090.6930.281.06
LVSM+ MSE+LPIPS40017.190.5890.5080.511.37
LVSM+ MSE+LPIPS100017.670.6120.4500.541.15

At epoch 399 baseline long has 16.78 dB / 0.585, below both LVSM+ runs. With 4 sources the 1000-epoch models reach 19.03 dB (LVSM+) and 18.84 dB (baseline long).

Four validation targets; columns ground truth, baseline, baseline long, LVSM+ MSE, LVSM+ MSE+LPIPS at 400 and 1000 epochs
Columns: ground truth, baseline (2.2), baseline long, LVSM+ MSE (400), LVSM+ MSE+LPIPS (400), LVSM+ MSE+LPIPS (1000).
Validation PSNR over 1000 epochs for baseline long and LVSM+ MSE+LPIPS and 400 epochs for LVSM+ MSE, and training LPIPS for the two LVSM+ runs
Left: validation PSNR. Right: training LPIPS of the two LVSM+ runs. Dashed line: LVSM+ MSE+LPIPS resumed.

Conclusions

  • The blocks come from the per-patch linear decoder. The conv head lowers blockiness from 3.43 to 1.06.
  • Removing the blocks alone does not sharpen the image. LVSM+ MSE has the lowest sharpness (0.28): MSE still averages detail the model is unsure about.
  • LPIPS brings back edges. Sharpness rises to 0.51 and LPIPS(AlexNet) falls to 0.508 without losing PSNR (+0.19 dB). The cost is 0.02 SSIM and a faint grid pattern with the 8-px patch period.
  • With equal training, LVSM+ is the clearest model. At 1000 epochs it matches baseline long in PSNR and SSIM (17.67 / 0.612 vs. 17.62 / 0.612), has much lower LPIPS (0.450 vs. 0.566) and higher sharpness (0.54 vs. 0.42), and has no block seams. Shelves, the chair and the poster become recognisable.
  • The LPIPS weight must start late and small. With weight 0.5 from epoch 0, the model output a dot texture with no scene layout (below).
Failed run: with LPIPS weight 0.5 from the start the prediction is a dot texture; MSE-only shows the blurry layout
Epoch 50. Columns: ground truth, LPIPS 0.5 from epoch 0, MSE only.

Limitations

Baseline vs. LVSM+ changes the decoder and capacity together, and each setting was run once. The MSE-only LVSM+ run was not extended, so the loss comparison is at 400 epochs, and the 1000-epoch LVSM+ run used a restarted learning-rate schedule. Detailed regions still show a faint 8-px grid pattern, and content the sources do not show stays blurry.

3. Multi-View Geometry with Mini-VGGT

3.1 Overfitting

Overfit training loss over 40 epochs: total, camera and depth loss all drop steeply in the first five epochs and keep decreasing
Overfit loss: total, camera and depth

GT vs. predicted geometry

Ground-truth cameras and point cloud of the overfit clip in canonical coordinates
Ground truth
Predicted cameras and point cloud of the overfit clip from the same viewpoint; cameras and geometry match the ground truth
Prediction

3.2 Training and Evaluation

SettingValue
ConfigDefault configs/vggt.json, unchanged
Views / sampling range8 / 32
Training1,000 epochs, batch size 16
OptimizerAdamW, lr 6×10−4, weight decay 0.01; 10-epoch warmup
Loss weights5 × camera + 1 × depth
HardwareOne RTX PRO 6000 (96 GB), 2 h 11 min
Training and validation loss over 1,000 epochs: total, camera and depth loss all decrease
Training loss: total, camera and depth

Metrics

MetricValTest
Depth AbsRel ↓0.1510.164
Depth δ1 ↑0.7990.775
Rotation AUC @5° / 15° / 30° ↑0.198 / 0.529 / 0.7110.178 / 0.495 / 0.691
Translation AUC @5° / 15° / 30° ↑0.014 / 0.071 / 0.1750.006 / 0.052 / 0.150
Pose AUC @5° / 15° / 30° ↑0.006 / 0.055 / 0.1540.000 / 0.031 / 0.129

best.pt; full splits, 8 views from the first 32 frames, 224px, seed 825.

GT vs. predicted geometry, four validation scenes

Each row: view 0's RGB and depth, ground truth, prediction.

Scene 09c1414f1b, a dining room: the prediction recovers the table, chairs and wall but bunches the cameras together
Scene 09c1414f1b
Scene 21d970d8de: ground truth and prediction from the same viewpoint
Scene 21d970d8de
Scene 280b83fcf3, a hallway: the prediction gets the corridor's layout but the walls are thicker and the far end is shortened
Scene 280b83fcf3
Scene 2b1dc6d6a5, an office: the camera path is roughly right but compressed; the desk and chair are smeared
Scene 2b1dc6d6a5

Errors on held-out scenes

Depth recovers the room layout in every scene (δ1 0.80). Errors are at object boundaries and thin structures such as chair legs, which are smeared because depth is decoded from 14×14 patches.

Cameras: rotation is reasonable (AUC@30° 0.71), but translation direction is poor. The predicted cameras are bunched together, and because cameras in a 32-frame window are close to each other, a small position error gives a large direction error. This limits the joint pose AUC.

Val and test results are close, and training loss stays above validation loss, so the model underfits rather than overfits.