1. Multi-View Data Preprocessing
I implemented the five geometry functions in a4data/geometry.py, keeping the supplied dataloader and visualization code. This part uses the provided RGB-D measurements and cameras; it does not need training.
1.1. Resizing camera intrinsics
The loader handles the center crop first. I scale the focal lengths for the new image size and apply the half-pixel correction to the principal point. This matches the supplied align_corners=False resize.
sx = W_out / W_in, sy = H_out / H_in
fx_new = sx * fx, fy_new = sy * fy
cx_new = sx * (cx + 0.5) - 0.5
cy_new = sy * (cy + 0.5) - 0.5
1.2. Depth unprojection
For an integer pixel center p = (u, v, 1), I compute X_camera = depth * K^-1 * p, then X_world = R * X_camera + t. The poses are camera-to-world. Depth is optical-axis z-depth, so I do not normalize the backprojected direction before multiplying by depth.
These screenshots use five training scenes, eight views per scene, 224px images and seed 825. The provided viewer displays up to 10,000 valid points per view. Red marks the first sampled camera; blue marks the others. Coordinates here are in the original world frame, in meters.

Four more training scenes




1.3. Camera and depth canonicalization
I express the cameras in camera 0's frame using T0^-1 * Ti. I compute this transform in float64, then cast back to float32. After unprojection in that frame, the provided compute_scene_scale gives s = ||mean(valid points)||. I divide camera translations, depth and points by the same scale.
I do not subtract the centroid. Camera 0 stays at the origin with an identity pose, while the valid-point centroid is one unit away. The original first pose and scale are kept so the original world coordinates can be recovered.
The five scenes and sampled frames match the unprojection results above. Each screenshot shows the first camera center and centroid distance in the right-hand panel. The point clouds still have gaps from missing depth and occlusion; normalization does not fill these in.

Four more training scenes




1.4. Camera rays
Each ray starts at the camera center o = t. Its unit direction in the canonical world frame is d = normalize(R * K^-1 * p). I rotate the direction but do not translate it. Rays are computed for every pixel, including pixels with missing depth.
1.5. Plücker coordinates
I concatenate the direction and moment in the required order: [d_x, d_y, d_z, m_x, m_y, m_z], where m = o × d. The moment describes where the ray's line lies relative to the origin. Moving the origin along the same line leaves this moment unchanged. It is perpendicular to the direction, and rays from camera 0 have zero moments.
1.6. Provided dataloader
The training loader randomly orders eight frames from a sampling window of up to 32 frames. The first selected frame defines camera 0, so it is not necessarily the earliest frame. The shared defaults use 224px and seed 825.
Validation and test use the prepared loader with deterministic view selection and frame 0 first. That loader bypasses my Q1 functions and applies the shared scale after sampling, so it cannot replace the training-data check below. Its shared default prefix is 128 prepared frames; the model sections specify their own evaluation settings.
1.7. Final check
I ran the required check on four training samples using the default batch size 2 and four workers:
python scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4 --report outputs/q1/check_data.json
All four samples pass. The largest point reprojection error is 0.00012303 pixels and the largest ray reprojection error is 0.00011109 pixels, both below 0.01 pixels. Centroid distances are approximately 1, with a maximum difference of 6 × 10−8. Invalid depth and point entries stay zero.
| Training sample | Valid depth | Max point error (px) | Max ray error (px) | Centroid distance |
|---|---|---|---|---|
e050c15a8d_frame_007080 | 97.79% | 0.00011279 | 0.00010904 | 1.00000000 |
5f99900f09_frame_002820 | 99.03% | 0.00012302 | 0.00011109 | 0.99999994 |
47b37eb6f9_frame_002720 | 96.10% | 0.00009882 | 0.00008897 | 1.00000000 |
4c5c60fa76_frame_006690 | 75.17% | 0.00010925 | 0.00010257 | 1.00000000 |
The RGB batch shape is (2, 8, 3, 224, 224), depth is (2, 8, 1, 224, 224), and Plücker coordinates are (2, 8, 6, 224, 224). Seven small analytic tests also pass, covering resize alignment, rotated-camera unprojection, canonicalization, masks, unit rays and the Plücker sign/order.
Full check_data.py output
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.00010924567322945222,
"max_ray_reprojection_error_px": 0.00010257320536766201,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
2. Novel-View Synthesis with Mini-LVSM
2.0. Implementation
I completed patchify/unpatchify, token assembly, the forward pass, MSE loss and training step in the supplied LVSM files. Patches are flattened in row, column, channel order. Each target gets its own sequence: all source tokens followed by that target's ray-query tokens. Sources and queries share the supplied 2D position embedding.
Source RGB is shifted to [−1, 1] and concatenated with its six Plücker channels. Targets provide rays only, not RGB. After the transformer, I decode only the target-query tokens with the RGB head and sigmoid. Training uses BF16 autocast; the RGB MSE is computed in float32 over every pixel, without a depth mask.
2.1. Fixed-batch overfitting
I ran the prescribed 2,000 updates on one fixed batch: four samples, two source views and six targets per sample. Fixed-batch MSE fell from 0.071160 at initialization to 0.000859. Final overfit PSNR, measured on that same batch, is 30.95 dB. This is a debugging result, not held-out performance. Full training started from a fresh model after this check.

2.2. Full training and evaluation
I used the supplied baseline config unchanged: patch size 8, width 384, eight transformer blocks, six attention heads, and MLP ratio 4. Training uses all 224 training scenes for 100 epochs, with batch size 4: 56 updates per epoch and 5,600 updates total. Each clip has eight 224 × 224 views drawn from a 32-frame window; seed is 825. Two views are sources and the other six are targets.
The optimizer is AdamW with learning rate 0.0002, weight decay 0.05, betas (0.9, 0.95), gradient clipping at 1, 200 warmup steps and the supplied cosine decay to 10% of the peak rate. The baseline ran on GPU 0. Best validation PSNR selected the checkpoint from epoch 100; all 100 epochs were completed.

Here is a fixed training-split example rendered with the best full-training checkpoint:

Validation previews saved during training


I evaluated the best checkpoint on all 28 validation scenes, using eight views from the first 32 prepared frames, 224px and seed 825. All nine source-count/conditioning combinations use the provided evaluator. No validation batches were omitted.
| Sources | Target rays | PSNR (dB) | SSIM | Target images |
|---|---|---|---|---|
| 1 | correct | 15.19 | 0.5582 | 196 |
| 1 | shuffled | 11.80 | 0.4876 | 196 |
| 1 | none | 12.46 | 0.5258 | 196 |
| 2 | correct | 16.13 | 0.5603 | 168 |
| 2 | shuffled | 11.49 | 0.4724 | 168 |
| 2 | none | 12.43 | 0.5260 | 168 |
| 4 | correct | 16.89 | 0.5759 | 112 |
| 4 | shuffled | 11.59 | 0.4732 | 112 |
| 4 | none | 12.44 | 0.5369 | 112 |
With two sources, shuffled rays change PSNR by -4.65 dB relative to correct rays; removing rays changes it by -3.70 dB. With correct rays, the one-, two- and four-source scores are 15.19, 16.13 and 16.89 dB. More sources supply additional appearance and coverage, but the model was trained with two sources. The supplied source partitions also change which views remain targets, so these averages are not an identical-target comparison.
Shuffling gives the model a different requested viewpoint while keeping the RGB target unchanged. Zero rays remove the query's camera information. Neither changes the source images. Both ray ablations reduce PSNR at two sources, which supports the model using target-camera information. The images below show the resulting errors rather than only the average scores.
The held-out images are still heavily blurred, with visible patch boundaries. The baseline gets some room colors and large bright regions, but furniture layout and thin structures are often inaccurate. Passing the fixed-batch check does not mean strong generalization. These are the results of the full supplied 100-epoch schedule; whether substantially longer training would fix them needs a separate experiment.
All nine evaluation visualizations
1 source view(s)



2 source view(s)



4 source view(s)



Three held-out scene examples with correct rays



2.3. Bonus: does a shallower model help?
I compared the eight-layer baseline with four transformer layers. For normal training, only layer count and the output directory change: data, seed, source count, image/patch size, optimizer and the 100-epoch schedule are the same. The four-layer model ran independently on GPU 1. Its initial 2,000-step diagnostic reached only 26.28 dB, so I repeated that fixed-batch check with 6,000 updates and reached 35.00 dB before full training. Neither normal-training run was initialized from overfit weights.
| Model | Parameters | Epochs / updates | Logged train + val time | PSNR (dB) | SSIM |
|---|---|---|---|---|---|
| 8 layers | 14.63 M | 100 / 5,600 | 22.8 min | 16.13 | 0.5603 |
| 4 layers | 7.54 M | 100 / 5,600 | 17.2 min | 15.97 | 0.5573 |

At the normal two-source/correct-ray evaluation, four layers change PSNR by -0.16 dB and SSIM by -0.0030. Each model uses its own best validation-PSNR checkpoint; the four-layer checkpoint is from epoch 100. The smaller model has fewer parameters and less transformer computation per update. This is an equal-epoch/equal-data comparison, not equal compute. Runtime includes validation, uses different GPUs and was measured during concurrent runs; GPU 1 also had an unrelated memory allocation. It is not a clean hardware-speed benchmark.
FP32 weights plus Adam's two moment tensors take roughly 175.6 MB for eight layers and 90.5 MB for four layers, excluding gradients and activations. This is tensor-storage accounting, not a measured peak GPU-memory benchmark. Fewer layers also store fewer block activations, but token counts and per-block attention shapes stay the same, so total memory does not simply halve.
At this training budget, eight layers do better on both held-out metrics. The extra capacity helps here, although the smaller model is cheaper per update. This is one seed on the supplied training subset. Capacity and optimization interact, and validation averages do not guarantee sharp details in every room. The matched examples below compare image quality on the same targets without substituting overfit results for generalization.
Matched eight-layer / four-layer predictions on three validation scenes






3. Multi-View Geometry with Mini-VGGT
3.0. Implementation
I completed attention aggregation, camera/depth prediction and the losses in the three supplied VGGT files. Frame blocks attend within each view; global blocks attend over all views. I repeat the 2D positions for global attention, then concatenate the final frame/global features. The DINOv2 ViT-B/14 encoder stays frozen.
The camera head predicts world-to-camera translation, an XYZW quaternion and vertical/horizontal FoV. Only FoV gets a ReLU; camera loss compares the raw encoding directly to GT. The depth head unpatchifies its features, adds RGB refinement, and uses exp for depth and 1 + exp for confidence.
Camera loss combines translation L1 (clamped at 100), quaternion L1 and half the FoV L1. Depth loss uses the provided 98% trimmed regression/confidence terms plus masked residual gradients at strides 1, 2, 4 and 8. Total loss is 5 × camera + depth. Evaluation still uses all GT-valid pixels, without trimming or a predicted-confidence mask.
3.1. Fixed-clip overfitting
I used one fixed eight-view training clip for the prescribed 40 epochs × 25 updates = 1,000 updates, with learning rate 0.001. Total loss fell from 5.4835 to -0.1612. On the fitted clip, depth AbsRel is 0.0372, δ1 is 0.9548, and joint pose AUC@15° is 0.9905. These are training-clip results, not held-out metrics.
The depth objective includes −0.2 log(confidence), so total/depth loss can be negative. The depth and pose metrics are useful checks alongside that objective.



3.2. Full training and evaluation
I trained from a fresh initialization for all 1,000 epochs over the 224 training scenes: 14 updates per epoch, 14,000 updates total. DINO uses the provided pretrained weights but remains frozen. The model has eight frame/global pairs (16 blocks), width 384 and six heads; the camera head has four cross-view blocks.
The supplied config is unchanged: eight 224 × 224 views, sampling range 32, seed 825, AdamW learning rate 0.0006, weight decay 0.01, gradient clipping 1, ten warmup epochs and cosine decay to 5% of the peak rate. Training uses BF16 autocast. Two RTX 4090s run DDP with the provided loader's global batch size 16: eight samples per GPU, not 16 per GPU.
Each epoch validates all 28 validation scenes. Minimum validation loss selects the best checkpoint, from epoch 945. No overfit weights initialize normal training. The required evaluations use that checkpoint on all 28 validation and all 28 test scenes, with eight views from the first 32 prepared frames, 224px and seed 825. There is no max-batch limit.

| Metric | Validation | Test |
|---|---|---|
| Depth AbsRel ↓ | 0.1406 | 0.1716 |
| Depth δ1 ↑ | 0.8413 | 0.7776 |
| Rotation AUC@5° ↑ | 0.1745 | 0.1875 |
| Rotation AUC@15° ↑ | 0.4754 | 0.4561 |
| Rotation AUC@30° ↑ | 0.6680 | 0.6616 |
| Translation AUC@5° ↑ | 0.0036 | 0.0038 |
| Translation AUC@15° ↑ | 0.0625 | 0.0386 |
| Translation AUC@30° ↑ | 0.1858 | 0.1282 |
| Joint pose AUC@5° ↑ | 0.0010 | 0.0008 |
| Joint pose AUC@15° ↑ | 0.0455 | 0.0204 |
| Joint pose AUC@30° ↑ | 0.1574 | 0.1046 |
AUC values are in [0, 1]. Rotation AUC includes all camera pairs. Translation/joint AUC excludes undefined GT translation directions and allows the supplied translation-sign ambiguity. The reports contain 10,347,720 validation and 10,517,881 test depth pixels.
The broad room shape is recognizable, but the held-out geometry is much less precise than the overfit example. In the dining room, chair backs and legs merge into a smooth depth blob. The corridor has rounded wall boundaries and overlapping surfaces instead of a clean doorway; the office chair and desk also lose thin details. Camera positions are often too clustered or follow the wrong direction, so combining the views produces thick, misregistered surfaces even when individual depth maps look reasonable. Mean translation-direction error is about 43 degrees on both splits, and joint pose AUC@15 degrees is only 0.0455 on validation and 0.0204 on test. These are not accurate camera reconstructions. The test depth scores are also worse than validation, so fitting one clip well does not establish generalization. Black regions in the displayed depth maps use the GT missing-depth mask; they do not indicate that the model learned to detect missing measurements.
Below are four different validation scenes. Both GT and prediction use the same display ranges and camera viewpoint. I do not post-align the predictions or overwrite the predicted first-camera pose.
09c1414f1b_frame_000000


Scene 2: 21d970d8de_frame_000000
21d970d8de_frame_000000


Scene 3: 280b83fcf3_frame_000000
280b83fcf3_frame_000000


Scene 4: 2b1dc6d6a5_frame_000000
2b1dc6d6a5_frame_000000


3.3. Bonus: input-view count at evaluation
I kept the trained checkpoint fixed and supplied the first 2, 4 or 8 RGB views from each prepared eight-view validation clip. To make the accuracy comparison controlled, all settings score the same first two views, using the original eight-view canonicalization and scale. This is a matched-view context study, not the standard eight-view evaluation above.
All 28 validation scenes participate. The table reports common-view depth/pose metrics and batch-one inference cost on NVIDIA GeForce RTX 4090. Timing is GPU-synchronized forward time only, after five warmup forwards per setting; it excludes loading and metric computation. Memory is peak PyTorch-allocated forward memory, including resident tensors, not whole-device memory or a training-memory benchmark.
| Input views | AbsRel ↓ | δ1 ↑ | Rot. AUC@15° | Trans. AUC@15° | Joint AUC@15° | Forward ms | Peak MiB |
|---|---|---|---|---|---|---|---|
| 2 | 0.1791 | 0.6823 | 0.5024 | 0.0714 | 0.0690 | 29.35 | 715.6 |
| 4 | 0.1473 | 0.8122 | 0.5286 | 0.0643 | 0.0619 | 29.35 | 732.7 |
| 8 | 0.1371 | 0.8487 | 0.5452 | 0.0976 | 0.0690 | 29.22 | 769.0 |

Common-view depth AbsRel changes from 0.1791 to 0.1473 to 0.1371 as context increases. Joint pose AUC@15° is 0.0690, 0.0619 and 0.0690; the camera result does not follow the same steady trend. In the depth grid, extra views help the dining-room depth scale, but chair legs and wall edges remain blurred.
Mean forward times are 29.35, 29.35 and 29.22 ms. They are essentially flat in this batch-one measurement; the small difference is not evidence that eight views are faster. GPU underutilization and host/kernel-launch overhead are possible explanations, but I did not profile them. Peak allocated memory grows from 715.6 to 769.0 MiB (7.5%). Resident model weights are shared across settings, so total memory need not grow in proportion to the view count.
More views increase encoder work and global-attention sequence length, even though this benchmark does not show a latency increase. The model was trained with eight views, so two/four views are a distribution change. This is one checkpoint and one seed; the matched-view scores are not directly comparable to the full eight-view metrics above, and the first two views provide only 28 camera pairs across the split.