16-825 Learning for 3D Vision | Fall 2026 | Jinyao Zhou (jinyaozh)
Left: xy_grid. Right: rays.
Left: color. Right: depth, divided by its maximum value.
Box center: (0.25, 0.25, 0.00)
Box side lengths: (2.00, 1.50, 1.50)
MLP: 6 layers of 128 with ReLU, position input added again at layer 3, 10 harmonic functions for position. No view dependence. 250 epochs, lr 5e-4, 128 x 128 images.
Mean validation PSNR: 23.82 dB (100 views).
Materials scene, 400 x 400. Both runs use the same settings: 6 layers of 256, 10 harmonic functions for position, 400 epochs, lr 1e-4. In the view-dependent run, density uses only the position. The color head takes the position features and the view direction (2 harmonic functions), with one hidden layer of 128.
| Model | Validation PSNR (dB, 100 views) |
|---|---|
| No view dependence | 23.94 |
| View dependence | 25.18 |
Left: no view dependence. Right: view dependence.
Train and validation curves. Top: no view dependence. Bottom: view dependence.

View dependence gives higher PSNR on both train and validation views. But I think the two look very similar, and the effect of view dependence is not obvious. So I tried the parameters from the paper, with a bigger and deeper network, and then you can see the difference (below).
In both runs, the gap between train and validation PSNR stays below 0.5 dB, so I don't see clear overfitting at 400 epochs. I think this comes from the network design in the NeRF paper: the position goes through a shared backbone, and density comes only from it, so the shape is the same from all views. The view direction is only added to a small color head on top of the backbone features. So the backbone has to learn one shape for all views, and the direction only changes the color on it.
According to NeRF++ (Zhang et al. 2020), the risk of view dependence is the shape-radiance ambiguity: a wrong shape can still match all training images if its color can change freely with the view, but new views will be wrong. In NeRF, the direction only has a few frequencies (2 in my runs, 4 in the paper) and a small head, so the color can only change a little with the view. This is not enough to hide a wrong shape, so the network has to learn the correct shape.
Network from the NeRF paper: 8 ReLU layers of 256, position input added again at layer 5, a linear feature layer, and a color head of 128. 10 harmonic functions for position and 4 for direction. 128 samples per ray (no coarse/fine sampling), 1024 rays per step, lr 5e-4, 400 epochs.
With PyTorch default initialization, this deep network collapsed to an empty scene: density was 0 everywhere and got no gradient. These changes fixed it: Kaiming initialization, density bias 0.1, rays only from the image center (half size) for the first 5 epochs, and a fixed seed.
| Model | Validation PSNR (dB, 100 views) |
|---|---|
| Paper architecture, view dependence | 27.41 |
Two train views and two validation views. Left to right: ground truth, prediction, absolute error x4. Brighter means larger error.
I find the mirror material shows larger error, which I think can be explained by that our model tends to blur things and may fit diffuse surface appearance better.
The metal highlights are clear, and the mirror sphere shows the environment. The gap between train and validation PSNR is about 1 dB (28.1 and 27.2 on random rays at epoch 400), which is larger than in the smaller runs. A larger network and more direction frequencies fit the train views better, but validation still improves with train, so it does not overfit badly at 400 epochs.
Isotropic Gaussians, L1 loss, Adam, 1000 iterations.
| Parameter | Learning rate |
|---|---|
| Opacities | 0.05 |
| Scales | 0.005 |
| Colors | 0.02 |
| Means | 0.0005 |
Mean PSNR: 29.54. Mean SSIM: 0.936.
Training progress. Top: prediction. Bottom: reference.

Final renders of training views.
Without spherical harmonics (only the constant component):
With spherical harmonics (degree 3):
Same views side by side. Left: without spherical harmonics. Right: with spherical harmonics.
Without spherical harmonics, each Gaussian has one color for all views. With spherical harmonics, I can very clearly see that when the chair rotates, the silk velvet cushion gets brighter or darker at different angles, and the strength of the reflection also changes.
View 1: in the right image, the center of the cushion has a very clear golden reflection, and there is a shading from left to right. There is no sharp boundary between light and dark like in the left image; it is softer and silkier.
View 10: in the right image, the position of the reflection is completely different from view 1, and the contrast between light and dark is much better. In the left image, the colors of all parts are very thin and uniform. It feels like the left image has a soft light from all directions, and the right image has a more real light source.