Assignment 4

Jianing Zhang (jianing7)

1. Multi-View Data Preprocessing

1.2 Depth unprojection

Points and cameras in the original world frame. The red frustum is the first camera.

Unprojected points and cameras, scene 036bce3393
036bce3393_frame_004470
Unprojected points and cameras, scene 076c822ecc
076c822ecc_frame_002890

1.3 Camera and depth canonicalization

The same two scenes after canonicalization. The first camera is at the origin and the centroid distance is 1 (see the text at the bottom right).

Canonicalized scene 036bce3393
036bce3393_frame_004470
Canonicalized scene 076c822ecc
076c822ecc_frame_002890

1.7 Final check

{
  "split": "train",
  "dataset_samples": 224,
  "checked_samples": 4,
  "samples": [
    {
      "sequence_name": "e050c15a8d_frame_007080",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9779127240180969,
      "max_reprojection_error_px": 0.00011279189493507147,
      "max_ray_reprojection_error_px": 0.00010904046939685941,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "5f99900f09_frame_002820",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9903116226196289,
      "max_reprojection_error_px": 0.00012302029062993824,
      "max_ray_reprojection_error_px": 0.00011108566104667261,
      "centroid_distance": 0.9999999403953552
    },
    {
      "sequence_name": "47b37eb6f9_frame_002720",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.9609947800636292,
      "max_reprojection_error_px": 9.881925507215783e-05,
      "max_ray_reprojection_error_px": 8.897326188161969e-05,
      "centroid_distance": 1.0
    },
    {
      "sequence_name": "4c5c60fa76_frame_006690",
      "views": 8,
      "resolution": 224,
      "valid_depth_fraction": 0.7516716122627258,
      "max_reprojection_error_px": 0.0001155954014393501,
      "max_ray_reprojection_error_px": 0.000106735089502763,
      "centroid_distance": 1.0
    }
  ],
  "batch_shapes": {
    "rgb": [2, 8, 3, 224, 224],
    "depth": [2, 8, 1, 224, 224],
    "depth_valid": [2, 8, 1, 224, 224],
    "K": [2, 8, 3, 3],
    "frame_numbers": [2, 8],
    "T_world_from_camera": [2, 8, 4, 4],
    "original_world_from_reference": [2, 4, 4],
    "scene_scale": [2],
    "points_world": [2, 8, 3, 224, 224],
    "ray_origins": [2, 8, 3, 224, 224],
    "ray_directions": [2, 8, 3, 224, 224],
    "plucker": [2, 8, 6, 224, 224]
  }
}

All checks pass: the reprojection errors are tiny and the centroid distance is 1.

Explanations

Canonicalization. I move everything into the first camera's frame, so the first camera sits at the origin with no rotation. Then I scale the scene so the center of the valid points is 1 away from the first camera. This way every scene has the same reference frame and the same size, no matter where it was or how big the room is.

World-space ray directions. For each pixel, the inverse intrinsics give the ray direction in the camera frame. I rotate it into the world with the camera rotation (no translation, since it is a direction) and normalize it. All rays of a camera start at the camera center.

Plücker moment. The moment is the cross product of the ray origin and direction. Any point on the same line gives the same moment, so direction plus moment describes the ray itself. This gives every pixel a 6-D camera encoding, which is how LVSM gets the cameras.

2. Mini-LVSM

2.1 Overfitting one batch

Final overfit PSNR: 33.82 dB.

Overfit visualization
val_epoch000.png. Columns: source views, ground truth, prediction.

2.2 Training and evaluation

Training settings. I used the default configs/lvsm.json (2 source views, 100 epochs) with one change: batch size 2 instead of 4. I train on my own GPU, which has limited memory, and batch size 4 runs out of memory, so I can only use batch size 2.

Mini-LVSM training curves
Training curves.
Training visualization
Training visualization (val_epoch095.png). Columns: source views, ground truth, prediction.

Validation results (PSNR / SSIM):

Source viewscorrectshufflednone
115.13 / 0.55811.82 / 0.49012.46 / 0.532
216.04 / 0.55911.50 / 0.47512.42 / 0.530
416.72 / 0.57511.67 / 0.48012.48 / 0.541
1 source view, correct rays
1 source view, correct
2 source views, correct rays
2 source views, correct
4 source views, correct rays
4 source views, correct
2 source views, shuffled rays
2 source views, shuffled
2 source views, no rays
2 source views, none

Evaluation visualizations. Columns: source views, ground truth, prediction.

Number of source views. More source views give better results, even though I trained with 2. More views show more of what the target camera sees, so the model has to guess less. The predictions get the layout and colors right but are still blurry and blocky, since the model is small and trained briefly.

Ray conditioning. The target rays matter a lot. With shuffled rays, the model draws blurry shapes in the wrong place. With no rays, it outputs an almost flat image in the scene's average color. Without the correct rays, adding source views does not help anymore, which shows the model uses the target rays to know where to look.

2.3 Bonus: adding an SSIM loss

Question. Does adding an SSIM term to the MSE loss make the novel views sharper?

Setup. I train with MSE + 0.1 · (1 − SSIM) and keep everything else the same as 2.2, so the loss is the only difference.

Validation PSNR and SSIM for both losses
Validation PSNR and SSIM during training.
Source viewsMSE (2.2)MSE + SSIM
115.13 / 0.55814.45 / 0.567
216.04 / 0.55914.94 / 0.569
416.72 / 0.57515.20 / 0.581

Validation results with correct rays (PSNR / SSIM).

MSE baseline on top, MSE + SSIM below
Same scenes. Top 4 rows: MSE. Bottom 4 rows: MSE + SSIM.

Results. Each model wins on what it was trained for: the SSIM loss gives slightly higher SSIM but lower PSNR. Visually it is not sharper. It is less blocky, but smoother and blurrier, with duller colors.

Why. SSIM cares about local structure and contrast, so it punishes the blocky patch edges. But it cares less about exact brightness and color, which is what MSE and PSNR measure. With a small model, the SSIM term trades color accuracy for smoothness.

Limitations. I only tried one weight, and both models are still far from converged. A smaller weight or a perceptual loss might reduce the blockiness without losing as much PSNR.

3. Mini-VGGT

3.1 Overfitting one clip

Overfit loss curves: total, camera and depth
Overfit loss curves (total, camera, depth).
Overfit clip, ground truth geometry
Ground truth
Overfit clip, predicted geometry
Prediction

All losses go down quickly, and the total loss goes slightly below zero because of the confidence term. The predicted cameras and point cloud almost match the ground truth, so the model can fit a single clip.

3.2 Training and evaluation

Training settings. I used the default configs/vggt.json (8 views, 1000 epochs, frozen DINO encoder) with one change: batch size 6 instead of 16. I train on my own GPU, which has limited memory, and batch size 16 runs out of memory, so I used the largest batch that fits. I also used 4 data loader workers instead of 8, since that was faster on my machine.

Training loss curves: total, camera and depth
Training loss curves (total, camera, depth).

Results of best.pt:

SplitAbsRel ↓δ1 ↑Rot. error (°) ↓Trans. error (°) ↓Rot. AUC@30 ↑Trans. AUC@30 ↑Pose AUC@30 ↑
Val0.1430.83514.844.60.6650.1510.131
Test0.1640.79810.343.20.7060.1420.127

Ground truth (left) and prediction (right) for four held-out scenes:

Scene 09c1414f1b, ground truth
09c1414f1b, ground truth
Scene 09c1414f1b, prediction
09c1414f1b, prediction
Scene 21d970d8de, ground truth
21d970d8de, ground truth
Scene 21d970d8de, prediction
21d970d8de, prediction
Scene 280b83fcf3, ground truth
280b83fcf3, ground truth
Scene 280b83fcf3, prediction
280b83fcf3, prediction
Scene 2b1dc6d6a5, ground truth
2b1dc6d6a5, ground truth
Scene 2b1dc6d6a5, prediction
2b1dc6d6a5, prediction

Errors on held-out scenes. Validation and test results are close, so the model generalizes to new scenes. The rough room shape and the camera layout are right, but the details are not. The predicted depth is smooth and blurry, so walls bend, corners are rounded and thin things like chair legs get smeared out. Rotation is learned fairly well, but the translation direction is the weakest part. The cameras in a clip are close together, so the direction they move in is hard to tell from the images, and a small mistake there gives a large angle error. The model is also small and only trained on small 224×224 images with a frozen encoder, so it cannot recover fine geometry.

3.3 Bonus: number of input views

Question. How does the number of input views at test time affect accuracy, runtime and memory?

Setup. I take the trained model from 3.2 (trained with 8 views) and evaluate it on the same val and test scenes with 2, 4, 8, 16 and 32 views, one sample at a time. I time the forward pass and record the peak GPU memory. No retraining.

Accuracy, runtime and memory vs number of input views
Depth, pose, runtime and memory against the number of views.
ViewsVal AbsRel ↓Val δ1 ↑Val rot. error (°) ↓Test rot. error (°) ↓Val pose AUC@30 ↑Time (ms)Memory (GB)
20.1130.87217.113.90.110420.69
40.1360.83816.210.90.098400.71
80.1430.83514.810.30.131460.75
160.1710.79213.210.20.119650.83
320.1780.77413.89.80.1191380.98

Results. More views make depth worse but rotation a bit better, and translation stays about the same. Runtime is flat up to 8 views and then grows quickly, while memory grows only a little.

Why. Views are picked from a fixed window, so with more views, more of them are far from the first camera, and their depth has to be placed in the first camera's frame, which is harder. Going past 8 views also means the model sees more views than it was trained on. Rotation gets better because more views give more overlap to match. Runtime grows fast because global attention looks at all tokens of all views together, so its cost grows with the square of the number of views.

Limitations. I only changed the number of views at test time. Training with more (or a random number of) views might change the picture. Each view count also uses a different random set of frames, which adds some noise.