16-825 Assignment 4: Multi-View Transformers

Name: Ruinian (Ray) Xue

AndrewID: ruinianx

1. Multi-View Data Preprocessing (20 points)

1.2. Depth Unprojection

Implement unproject_depth to convert depth pixels into world-space 3D points using the supplied camera poses. Depth is optical-axis z-depth, not distance along a unit ray.

After Q1.1–1.2, visualize points and cameras in the original world frame (meters):

Screenshot

1.3. Camera and Depth Canonicalization

After Q1.3, visualize canonicalized geometry. Camera 0 should be identity, with the valid-point centroid approximately 1 unit from the origin:

Screenshot

Note: I tried to put them together, but if I scroll down, the visualization will disappear so I made the screenshot separately. But all of them are correspondent to each other and of the same scene.

SceneImgFirst camera center & centroid distance
1
2

1.7. Final Check

check_data.py output

Discussions

Canonicalization

Canonicalization is important, it changes different scenes into the same scale. We can look at the picture in 1.2, and compare with the in 1.3, the scale in 1.2 was from 0 to 8, but in 1.3 is within 2. This allows network in later implementation to learn without knowing about different scales, and makes it simpler than putting scale S into loss like the way in π3.

World-space ray directions

This allows pixels to have information of directions. In the picture above, we can notice that there are triangles representing camera of different positions and their directions of ray. With this, we can recognise whether two rays has eye on the same point.

Plücker moment

The |M|equals the distance from origin to the ray.

2. Novel-View Synthesis with Mini-LVSM (40 points)

2.1. Overfitting (10 points)

Once your implementation is complete, check that it can memorize one fixed batch before full training:

Check that the loss decreases and PSNR reaches about 30 dB or more, and inspect outputs/lvsm_overfit/val_epoch000.png.

Submission (Q2.1): Include the final overfit PSNR and the val_epoch000.png visualization on your assignment webpage.

Overfit PSNR

PSNR: 34.09 dB

SSIM: 0.9156

Overfit visualization

2.2. Training and Evaluation (30 points)

This prints train and validation metrics every epoch and saves best.pt, last.pt, log.jsonl and val_epoch*.png (visualizations) under outputs/lvsm/.

Evaluate best.pt with ablations over the number of source views and the target-ray conditioning.

Settings:

correct uses the true target rays, shuffled uses the rays of a different target view, and none uses all-zero rays.

Metrics:

MetricDefinitionBetter
PSNRHow close the predicted image is to the real one, in dB, based on the pixel error (MSE)Higher
SSIMHow similar the structure (edges, contrast, local patterns) of the two images is, from 0 to 1Higher

Training curves

Results for ablation setting

Source views (S)ConditioningPSNR (dB)SSIM
1correct15.140.5578
1shuffled11.830.4894
1none12.490.5270
2correct16.070.5598
2shuffled11.520.4751
2none12.450.5280
4correct16.760.5745
4shuffled11.650.4782
4none12.490.5410

Training visualizations

Epoch 0

Epoch 50

Epoch 95

Evaluation visualizations

SettingVisualization
S=1, correct
S=1, shuffled
S=1, none
S=2, correct
S=2, shuffled
S=2, none
S=4, correct
S=4, shuffled
S=4, none

Training settings

Using almost default settings except changed batch size from 4 to 2.

Training took around 2h on a 8GB 3070Ti Laptop GPU. Setting batch size low to suit on 8GB RAM for GPU.

Discussions

The overall picture is vague, and I noticed in Piazza discussion that it is normal within this number of epochs and settings.

We can notice that the model act the best in the setting of S equals four and correct for the shuffled one the performance is worse. But it's interesting that shuffled has picutre that's funny and clearer than none, but it has worse PSNR (11.65) than none (12.49), it seems that giving a wrong perspective will ever make things worse(?) not really. It might also due to the bias of PSNR and SSIM score.

For the S=4, it's the clearest picture out of prediction, S=4 > S=2 > S=1. It is easy to understand because if we have more perspectives, then we have better understanding of a certain scene.

2.3. Bonus: Analysis and Exploration (10 points)

Question & Experimental setting

Question: Train with a different number of source views, change numer of source views into 4, figure out what affect will be made on metrics and visualization.

Experimental setting: only change num_source_views from 2 to 4 in the config file, keep all others the same.

Result & Conclusion

Eval SConditioningTrain S=2 PSNRTrain S=2 SSIMTrain S=4 PSNRTrain S=4 SSIM
1correct15.140.557815.150.5574
1shuffled11.830.489411.860.4865
1none12.490.527012.350.5322
2correct16.070.559816.240.5595
2shuffled11.520.475111.500.4673
2none12.450.528012.320.5292
4correct16.760.574517.250.5766
4shuffled11.650.478211.470.4625
4none12.490.541012.370.5401

Visualization

Epoch 0

Epoch 50

Epoch 95

Explanation & Limitation

We can notice both in metric results and visual results that when number of source views turn into 4, the performance is better with PNSR and SSIM: 17.25 and 0.5766, compared to 16.76 and 0.5745 in the last case. Because if we have more perspectives during training, the model is more used to 4 pictures and seems can get more information out of the same setting. But it is interesting that for S=1 during testing didn't change much, even drop some in SSIM.

Limitation: During training, with more numbers of source view, the training speed get slower.

3. Multi-View Geometry with Mini-VGGT (40 points)

3.1. Overfitting (10 points)

Default model: A frozen DINOv2 ViT-B/14 encoder feeds 8 frame/global transformer pairs (16 blocks; width 384; 6 attention heads). The camera head adds 4 cross-view transformer blocks, and the depth head decodes patch features with RGB refinement.

Inspect outputs/vggt_overfit/loss_curve.png and outputs/vggt_overfit/visualization/index.html. Check that loss decreases and predicted geometry approaches GT.

Submission (Q3.1): Include the overfit loss curve image (total, camera and depth loss) and screenshots comparing GT and predicted geometry on your assignment webpage.

Loss curve image

Screenshot

GT geometryPredicted geometry

3.2. Training and Evaluation (30 points)

Once single-scene overfitting works, train on the full training split. We recommend starting with the defaults in configs/vggt.json. You may adjust other hyperparameters while keeping num_views=8 and sampling_range=32.

For four GPUs, replace python with torchrun --standalone --nproc_per_node=4. The script saves loss_curve.png, last.pt, best.pt and visualization/index.html under the output directory. The HTML compares GT and predicted geometry on validation scenes; best.pt is selected by validation loss.

To continue an interrupted run with its saved settings, optimizer and schedule:

Evaluate best.pt on validation and test using 8 views from the first 32 prepared frames, 224px and seed 825. These flags override the checkpoint's data settings. Run the full splits without --max-batches:

Metrics:

MetricDefinitionBetter
Depth AbsRelMean abs(pred − GT) / GT over valid pixelsLower
Depth δ1Fraction with max(pred/GT, GT/pred) < 1.25Higher
Rotation AUCPairwise relative rotation accuracyHigher
Translation AUCPairwise translation-direction accuracy, allowing sign ambiguityHigher
Joint Pose AUCUses the larger rotation/translation error per pairHigher

Report each AUC at 5°, 15° and 30°; values range from 0 to 1. Depth metrics use all GT-valid pixels; translation/joint AUC excludes pairs with undefined GT translation direction.

Submission (Q3.2): Include the normal-training loss curve image (total, camera and depth loss), validation/test metrics, and screenshots comparing GT and predicted geometry for four different scenes. State your training settings and briefly discuss errors on held-out scenes.

Loss curve image

Validation/test metrics

MetricValTest
Depth Abs Rel0.16990.1718
Depth delta10.75010.7634
Rotation AUC 5°0.20540.2293
Rotation AUC 15°0.52080.5688
Rotation AUC 30°0.70190.7511
Translation AUC 5°0.01280.0084
Translation AUC 15°0.08340.0648
Translation AUC 30°0.20500.1864
Joint Pose AUC 5°0.00360.0043
Joint Pose AUC 15°0.05830.0497
Joint Pose AUC 30°0.18070.1712

Screenshot

I tried my best to adjust to the best position but still somewhat different:(

SceneGT geometryPredicted geometry
1
2
3
4

Training setting

Using almost default settings except changing batch size from 16 to 1, and changed number of workers from 8 to 2.

Training took around 12h on a 8GB 3070Ti Laptop GPU. Setting numbers above low to suit on the 8GB 3070Ti GPU, and limited RAM on WSL system.

Discussion

We can notice that in overfitting, on the Screenshot, the prediction is almost the same as the ground truth. This is because when training, the network optimize the network to just fit the data. It is like opening test answer to a student and the student will get full points, So we need held out, split the testing and training datasets.

On the test datasets, the network seems to predict vague on the edge of pictures. It is because on the edge of pictures, the information within the frame is less, covered by less viewpoints. And in the first case, the edges of desks and chairs aren't clear. But the depths and the relative position of objects is correct.