Name: Hiroki Mii
Andrew ID: hmii
Scene 2
Scene 3
Scene 4
Scene 5
{
"split": "train",
"dataset_samples": 224,
"checked_samples": 4,
"samples": [
{
"sequence_name": "e050c15a8d_frame_007080",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9779127240180969,
"max_reprojection_error_px": 0.00011279189493507147,
"max_ray_reprojection_error_px": 0.00010904046939685941,
"centroid_distance": 1.0
},
{
"sequence_name": "5f99900f09_frame_002820",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9903116226196289,
"max_reprojection_error_px": 0.00012302029062993824,
"max_ray_reprojection_error_px": 0.00011108566104667261,
"centroid_distance": 0.9999999403953552
},
{
"sequence_name": "47b37eb6f9_frame_002720",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.9609947800636292,
"max_reprojection_error_px": 9.881925507215783e-05,
"max_ray_reprojection_error_px": 8.897326188161969e-05,
"centroid_distance": 1.0
},
{
"sequence_name": "4c5c60fa76_frame_006690",
"views": 8,
"resolution": 224,
"valid_depth_fraction": 0.7516716122627258,
"max_reprojection_error_px": 0.0001155954014393501,
"max_ray_reprojection_error_px": 0.000106735089502763,
"centroid_distance": 1.0
}
],
"batch_shapes": {
"rgb": [
2,
8,
3,
224,
224
],
"depth": [
2,
8,
1,
224,
224
],
"depth_valid": [
2,
8,
1,
224,
224
],
"K": [
2,
8,
3,
3
],
"frame_numbers": [
2,
8
],
"T_world_from_camera": [
2,
8,
4,
4
],
"original_world_from_reference": [
2,
4,
4
],
"scene_scale": [
2
],
"points_world": [
2,
8,
3,
224,
224
],
"ray_origins": [
2,
8,
3,
224,
224
],
"ray_directions": [
2,
8,
3,
224,
224
],
"plucker": [
2,
8,
6,
224,
224
]
}
}
Canonicalization is expressing each sample in its first camera's coordinate frame and normalizing its scale. centroid_distance = 1.0 in every sample confirming the shared scale.
World-space ray directions is the unit direction of each pixel's ray in world-coordinates. It is computed through K-1[u, v, 1] (the direction in the camera's own axes), rotated into the world frame by the camera's rotation matrix, and normalized to unit length.
Plücker moment is the cross product of a ray's origin and unit direction. A ray is represented in Plücker coordinates as [d, m], direction followed by moment.
val_epoch000
val_epoch045
val_epoch095
PSNR = 15.193, SSIM = 0.558
S=1, conditioning=shuffled
PSNR = 11.797, SSIM = 0.487
S=1, conditioning=none
PSNR = 12.468, SSIM = 0.528
S=2, conditioning=correct
PSNR = 16.124, SSIM = 0.560
S=2, conditioning=shuffled
PSNR = 11.485, SSIM = 0.472
S=2, conditioning=none
PSNR = 12.448, SSIM = 0.527
S=4, conditioning=correct
PSNR = 16.876, SSIM = 0.575
S=4, conditioning=shuffled
PSNR = 11.577, SSIM = 0.472
S=4, conditioning=none
PSNR = 12.460, SSIM = 0.538
The training configurations were left as is in configs/lvsm.json with epochs=100, batch_size=4, learning_rate=0.0002, patch_size=8, d_model=384, n_layers=8.
The results of the ablations were consistent to what was expected.
As the number of source views increased, both PSNR and SSIM scores increased (with correct rays).
More source views cover more of the target view, so the model has more of the scene to draw from.
For ray conditioning, correct uses true target rays so it got the highest PSNR and SSIM scores.
Shuffled and None both got lower scores because the model no longer knows which view to render.
Shuffled gives it rays of a different view, and None gives it no camera information.
We investigate how adding an L1 or SSIM term to the MSE loss changes PSNR, SSIM, and visual sharpness. All other training configurations were left as is from 2.2.
S=1, conditioning=correct
PSNR = 15.328, SSIM = 0.559
S=2, conditioning=correct
PSNR = 16.287, SSIM = 0.563
S=4, conditioning=correct
PSNR = 17.096, SSIM = 0.579
S=1, conditioning=correct
PSNR = 14.876, SSIM = 0.575
S=2, conditioning=correct
PSNR = 15.551, SSIM = 0.578
S=4, conditioning=correct
PSNR = 15.975, SSIM = 0.591
Neither the addition of an L1 or SSIM term contributed to significant improvements.
L1 term raised PSNR by about 0.1-0.2 dB with no change in SSIM, while SSIM term raised SSIM by about 0.02, but lowered PSNR by about 0.3-0.9.
Like 2.2, the predicted outputs are still blocky, and the continued decrease in training and validation loss indicates the model is underfitted and needs further training.
The loss functions can be re-evaluated after training for more epochs.
"depth_abs_rel": 0.13722808236251358,
"depth_delta1": 0.8515819910086473,
"rotation_auc_3": 0.11224489795918367,
"rotation_auc_5": 0.2114795918367347,
"rotation_auc_15": 0.5318877551020408,
"rotation_auc_30": 0.7153911564625851,
"translation_auc_3": 0.004251700680272109,
"translation_auc_5": 0.012244897959183673,
"translation_auc_15": 0.08095238095238094,
"translation_auc_30": 0.2037840136054422,
"pose_auc_3": 0.0,
"pose_auc_5": 0.004081632653061224,
"pose_auc_15": 0.06301020408163266,
"pose_auc_30": 0.1876700680272109
"depth_abs_rel": 0.15694556708130455,
"depth_delta1": 0.7960929582679249,
"rotation_auc_3": 0.13477891156462585,
"rotation_auc_5": 0.2298469387755102,
"rotation_auc_15": 0.5437074829931973,
"rotation_auc_30": 0.734141156462585,
"translation_auc_3": 0.005527210884353741,
"translation_auc_5": 0.013520408163265307,
"translation_auc_15": 0.08112244897959184,
"translation_auc_30": 0.1993197278911565,
"pose_auc_3": 0.0012755102040816326,
"pose_auc_5": 0.004591836734693878,
"pose_auc_15": 0.05748299319727891,
"pose_auc_30": 0.18142006802721092
Ground truths on left, predictions on right.
Scene 1
Besides changing to num_workers=4 to avoid OOM errors, training configurations were left as is in configs/vggt.json with epochs=1000, batch_size=16, learning_rate=0.0006, num_views=8, sampling_range=32.
On held-out scenes, depth and rotation are reasonably accurate.
depth_abs_rel was 0.137 on validation, and 0.157 on test. rotation_auc_30 was 0.715 on validation, and 0.734 on test.
Translation was the weakest with translation_auc_30 0.204 on validation, and 0.199 on test.
Training and validation loss were still decreasing at epoch 1000 so longer training could help.
In the visualizations, the predicted point clouds capture the room layout and furniture, but surfaces are still a little noisy with stray points and smearing.