November 2026
All experiments were run on a single NVIDIA L4 (24 GB) with bf16 autocast. Image paths are relative to this file.
python scripts/visualize_data.py --stage unprojection --num-scenes 5 --max-points 10000 --output outputs/unprojection
Depth is z-depth, so each pixel is unprojected as \(X_c = z\,K^{-1}[u, v, 1]^\top\) and mapped
to world space with the camera-to-world pose. Points from all 8 views
overlap into one consistent room, and each camera frustum sits where it
should relative to the geometry it observes. The first camera is at an
arbitrary world location: for scene 036bce3393, the camera
center is (3.35, 4.68, 1.47) m and the centroid is 6.67 m from the
origin.
python scripts/visualize_data.py --stage canonicalization --num-scenes 5 --max-points 10000 --output outputs/canonicalization
These are the same scenes and frames as above. In every scene the first camera center is (0, 0, 0) and the valid-point centroid distance is 1.000. The info panel on the right of each screenshot shows both values.
| Scene | First camera center | Scene scale (m) | Centroid distance |
|---|---|---|---|
| 036bce3393_frame_004470 | (0.0000, −0.0000, 0.0000) | 1.867 | 1.000 |
| 076c822ecc_frame_002890 | (0.0000, −0.0000, 0.0000) | 2.830 | 1.000 |
check_data.py outputpython scripts/check_data.py --config configs/scannetpp/iphone/data.json --split train --max-samples 4
| Sequence | Views | Res | Valid depth | Max point reproj. err (px) | Max ray reproj. err (px) | Centroid dist. |
|---|---|---|---|---|---|---|
| e050c15a8d_frame_007080 | 8 | 224 | 0.978 | 1.13e−4 | 1.09e−4 | 1.0000 |
| 5f99900f09_frame_002820 | 8 | 224 | 0.990 | 1.23e−4 | 1.11e−4 | 1.0000 |
| 47b37eb6f9_frame_002720 | 8 | 224 | 0.961 | 9.88e−5 | 8.90e−5 | 1.0000 |
| 4c5c60fa76_frame_006690 | 8 | 224 | 0.752 | 1.09e−4 | 1.03e−4 | 1.0000 |
Batch shapes (batch size 2): rgb [2,8,3,224,224],
depth/depth_valid [2,8,1,224,224],
K [2,8,3,3], T_world_from_camera [2,8,4,4],
original_world_from_reference [2,4,4],
scene_scale [2], points_world /
ray_origins / ray_directions [2,8,3,224,224],
plucker [2,8,6,224,224]. All reprojection errors are about
1e−4 px, far below the 0.01 px threshold. The full raw output is in output1-7.txt.
Canonicalization. Every pose is left-multiplied by \(T_{\text{ref}}^{-1}\), the inverse of the first sampled camera’s camera-to-world pose. Camera 0 becomes the identity, and all geometry is expressed in its coordinate frame. Then a single scene scale \(s\) is computed: the distance in meters from the origin to the centroid of the valid points. Camera translations, depths and points are all divided by \(s\). Rotations are unchanged, and depth/point consistency is preserved because everything shares the same \(s\). After this step, every training sample has the same gauge: no arbitrary world origin or orientation, and a metric scale near 1. Without it, the network would have to predict poses in an arbitrary world frame, which cannot be determined from the images.
World-space ray directions. For pixel \((u, v)\), the camera-space direction is \(K^{-1}[u, v, 1]^\top\), using the intrinsics after resizing. It is rotated into the canonical world frame with \(R_{wc}\) and then normalized to unit length. The ray origin is the camera center \(t_{wc}\), which is the same for every pixel of a view. Reprojecting origin + depth-scaled direction lands back on the source pixel to within about 1e−4 px.
Plücker moment. A ray is encoded as 6 channels \([\mathbf d, \mathbf m]\) with \(\mathbf m = \mathbf o \times \mathbf d\). The moment is the same for every point on the line: for any point \(\mathbf p = \mathbf o + t\mathbf d\), \(\mathbf p \times \mathbf d = \mathbf o \times \mathbf d\). Its magnitude is the line’s distance from the origin, and its direction is the normal of the plane containing the line and the origin. So \([\mathbf d, \mathbf m]\) identifies the ray as a line, regardless of which point on it is called the origin. It is a smooth per-pixel camera embedding that a transformer can use directly.
python -m models.lvsm.train --overfit 2000 --output-dir outputs/lvsm_overfit
After 2000 steps on one fixed batch, the train loss fell to 6.9e−3. Final overfit PSNR: 30.90 dB (SSIM 0.870, 24 target images), which clears the 30 dB target.
Training settings (configs/lvsm.json):
num_source_views = 2 sources and 6
targets. Batch size 4, 224×224 images, patch size 8 (784 tokens per
view).best.pt
is the final epoch (99); validation PSNR was still slowly rising.
Validation ablations
(python -m models.lvsm.eval --checkpoint outputs/lvsm/best.pt --num-source-views 1 2 4 --conditioning correct shuffled none):
| Source views | Conditioning | PSNR (dB) ↑ | SSIM ↑ | # images |
|---|---|---|---|---|
| 1 | correct | 15.20 | 0.558 | 196 |
| 1 | shuffled | 11.81 | 0.488 | 196 |
| 1 | none | 12.48 | 0.528 | 196 |
| 2 | correct | 16.13 | 0.561 | 168 |
| 2 | shuffled | 11.50 | 0.473 | 168 |
| 2 | none | 12.45 | 0.526 | 168 |
| 4 | correct | 16.85 | 0.576 | 112 |
| 4 | shuffled | 11.59 | 0.474 | 112 |
| 4 | none | 12.46 | 0.536 | 112 |
Training visualizations. Columns: source views, ground truth, prediction.
Evaluation visualizations (columns: source views, ground truth, prediction):
Discussion.
none) drops PSNR
by about 3.7–4.4 dB. Shuffling them is even worse (about 4.5–5.3 dB
drop, lowest SSIM), because the model confidently renders the wrong
viewpoint. In contrast, none falls back to a blurry average
of the scene colors, which costs less under MSE. Both conditions are
flat across source-view counts (about 12.5 dB for none,
about 11.5 dB for shuffled): without correct target
geometry, extra sources do not help. Together, these results show the
model really uses the Plücker target rays and does not just copy colors
from the sources.Question. Does adding a structural loss term to the photometric MSE make predictions sharper and improve SSIM, and what does it cost in PSNR?
Setup. The loss is \(\mathcal L = \text{MSE} + 0.2\,(1 -
\text{SSIM})\) (configs/lvsm_ssim.json,
ssim_weight: 0.2, implemented in
view_synthesis_loss). Everything else matches the baseline:
architecture, data, seed, optimizer, schedule, 100 epochs, and 2
training source views. Both models were evaluated on the same validation
views with correct rays.
| Source views | Baseline PSNR | +SSIM PSNR | Baseline SSIM | +SSIM SSIM |
|---|---|---|---|---|
| 1 | 15.20 | 14.78 | 0.558 | 0.576 |
| 2 | 16.13 | 15.41 | 0.561 | 0.579 |
| 4 | 16.85 | 15.78 | 0.576 | 0.593 |
Results and conclusions.
python -m models.vggt.train --config configs/vggt.json --overfit --output outputs/vggt_overfit
Settings: 1 scene, 40 epochs × 25 steps, learning rate 1e−3. All three losses fall smoothly. The camera loss goes 0.34 → 0.0026. The depth loss becomes negative, which is expected: it includes the \(-\log(\text{confidence})\) regularizer, so it goes below zero once the model predicts accurate depth with high confidence. The predicted cameras and point cloud closely match the ground truth.
Training settings (configs/vggt.json, run
outputs/vggt_train2):
num_views=8, sampling_range=32, 224 px,
batch size 2 (112 steps per epoch).
Metrics (8 views from the first 32 prepared frames, 224 px, seed 825, full splits of 28 scenes / 784 pose pairs each):
| Split | AbsRel ↓ | δ1 ↑ | Rot AUC@5/15/30 ↑ | Trans AUC@5/15/30 ↑ | Pose AUC@5/15/30 ↑ |
|---|---|---|---|---|---|
| Val | 0.145 | 0.832 | 0.214 / 0.539 / 0.717 | 0.010 / 0.078 / 0.191 | 0.004 / 0.060 / 0.173 |
| Test | 0.159 | 0.795 | 0.188 / 0.501 / 0.698 | 0.013 / 0.076 / 0.194 | 0.001 / 0.047 / 0.167 |
Mean errors: rotation 11.8° (val) / 10.5° (test); translation direction 41.2° (val) / 39.1° (test).
GT vs. prediction on four validation scenes (left: ground truth, right: predicted camera + depth):
| Ground truth | Prediction |
|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Scenes: 09c1414f1b, 21d970d8de,
280b83fcf3, 2b1dc6d6a5.
Discussion of held-out errors.
280b83fcf3 the predicted valid-point centroid
is 0.94 instead of 1.0. Thin structures and reflective or dark regions,
such as windows and screens, are where the frozen-DINO features and the
light depth decoder struggle most.Question. The baseline gets 3D points indirectly, by unprojecting predicted depth with predicted cameras. (1) Is a direct world-space point-map head a better way to get 3D points? (2) Does adding point-map supervision as an extra task change the camera and depth predictions?
Method.
PointHead (models/vggt/heads.py)
follows the same pattern as the depth head. A linear layer maps each
patch token to a 14×14×3 block of points. A small RGB-conditioned
convolutional residual refines the result. There is no exponential
activation, so coordinates can be negative.points_world (B,V,3,H,W) directly in the
canonical first-camera frame.Settings (configs/vggt_points.json,
run outputs/vggt_points_b4w2_2):
num_views=8 and sampling_range=32 as
the Q3.2 baseline.
(1) Point head vs. camera + depth unprojection. All
rows are scored against the GT canonical points over every GT-valid
pixel, in normalized units (scene centroid at distance 1). The script is
compare_pointmaps.py;
raw results are in pointmap_val.json and pointmap_test.json.
| Split | Model / point source | Mean L2 ↓ | Median L2 ↓ | Rel. L2 ↓ | Acc@0.05 ↑ | Acc@0.10 ↑ |
|---|---|---|---|---|---|---|
| Val | Baseline / camera + depth | 0.257 | 0.159 | 0.224 | 0.112 | 0.296 |
| Val | Point model / camera + depth | 0.218 | 0.134 | 0.196 | 0.130 | 0.357 |
| Val | Point model / point head | 0.235 | 0.144 | 0.214 | 0.102 | 0.327 |
| Test | Baseline / camera + depth | 0.240 | 0.163 | 0.207 | 0.103 | 0.280 |
| Test | Point model / camera + depth | 0.210 | 0.136 | 0.182 | 0.117 | 0.351 |
| Test | Point model / point head | 0.219 | 0.143 | 0.190 | 0.088 | 0.323 |
(2) Effect on camera and depth metrics (standard evaluation; baseline numbers are from Q3.2):
| Split | Model | AbsRel ↓ | δ1 ↑ | Rot AUC@5/15/30 ↑ | Trans AUC@5/15/30 ↑ | Pose AUC@5/15/30 ↑ | Rot / trans err (°) ↓ |
|---|---|---|---|---|---|---|---|
| Val | Baseline | 0.145 | 0.832 | 0.214 / 0.539 / 0.717 | 0.010 / 0.078 / 0.191 | 0.004 / 0.060 / 0.173 | 11.8 / 41.2 |
| Val | Point model | 0.133 | 0.843 | 0.274 / 0.597 / 0.763 | 0.026 / 0.101 / 0.223 | 0.011 / 0.081 / 0.206 | 8.9 / 38.5 |
| Test | Baseline | 0.159 | 0.795 | 0.188 / 0.501 / 0.698 | 0.013 / 0.076 / 0.194 | 0.001 / 0.047 / 0.167 | 10.5 / 39.1 |
| Test | Point model | 0.154 | 0.816 | 0.250 / 0.583 / 0.755 | 0.010 / 0.067 / 0.190 | 0.003 / 0.052 / 0.176 | 9.1 / 38.4 |
Qualitative comparison on two validation scenes (left to right: ground truth, point model camera + depth, point model point head):
| Ground truth | Camera + depth | Point head |
|---|---|---|
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
Conclusions.