Guo Yu · 16-825 Learning for 3D Vision
Parts 1–6 and 7.1. NeRF extensions: view dependence, coarse/fine sampling, and VolSDF.
Pixel coordinates are generated in normalized device coordinates and unprojected into world space. Each ray starts at the camera center and follows the normalized direction toward its unprojected pixel.
Points are sampled between the near and far bounds along each ray: x(t) = o + td.
Colors and depths are accumulated using emission–absorption weights. Transmittance accounts for absorption by preceding samples.
αᵢ = 1 − exp(−σᵢΔtᵢ), Tᵢ = exp(−∑ⱼ<ᵢ σⱼΔtⱼ), wᵢ = Tᵢαᵢ
A subset of image pixels is sampled at each iteration to reduce the memory required for differentiable rendering.
The box center and side lengths are optimized using mean squared error between rendered RGB values and ground-truth colors.
| Parameter | Optimized value (rounded to 0.01) |
|---|---|
| Center (x, y, z) | (0.25, 0.25, -0.00) |
| Side lengths (x, y, z) | (2.01, 1.50, 1.50) |
An MLP maps encoded 3D positions to density and RGB color. Density is constrained to be nonnegative using ReLU; color is mapped to [0, 1] using sigmoid. The field is optimized with image reconstruction supervision.
[TODO: report the actual network architecture, positional encoding, training epochs, and image resolution.]
Color prediction additionally depends on viewing direction, while density depends only on position. This separates scene geometry from directional appearance.
Trade-off. Directional conditioning can represent specular highlights and reflections. Excessive directional capacity can also fit training views without learning appearance that interpolates reliably to unseen cameras. Restricting direction conditioning to the color branch and controlling its encoding and network capacity can improve generalization.
View dependence allows NeRF to represent appearance changes across viewing directions, such as specular highlights and reflections in the materials scene. However, increasing directional capacity can encourage the model to memorize training views rather than learn appearance that generalizes to unseen viewpoints. Conditioning only the color branch on viewing direction, while keeping density position-dependent, helps separate directional appearance from geometry. A moderate directional encoding provides a practical balance between modeling these effects and maintaining novel-view consistency.
A coarse pass estimates where samples contribute along each ray. Additional samples are drawn from the resulting weight distribution; the fine pass renders the combined, depth-sorted samples.
Speed / quality trade-off. Importance sampling concentrates evaluations near contributing regions and can recover detail more efficiently than uniform sampling alone. Two network passes and resampling add computation and memory; total runtime depends on both sample counts and network sizes. A poor coarse estimate can miss regions that should receive fine samples.
Coarse-to-fine sampling uses the coarse network’s rendering weights to allocate additional samples to regions likely to contribute to the final image. This can improve sampling efficiency and recover finer details, but requires an additional network pass and resampling. In my experiments, iteration time increased from approximately 9 seconds with the baseline to 11 seconds with coarse-to-fine sampling, an overhead of about 22%. The rendered results looked similar, with no substantial visual improvement. Under the current settings, coarse-to-fine sampling therefore introduced a moderate computational cost for limited visible benefit.
A neural field predicts signed distance and color. Signed distance is converted to volume density using a scaled Laplace CDF, and the resulting field is rendered with the volume renderer.
α and β. With negative signed distance inside the surface, σ(x) = α · Fβ(−s(x)). The parameter α controls the density scale; β controls the width of the transition around the zero level set.
High β. A broad density transition distributes contributions across a thicker region around the surface. This can provide gradients over a wider range and make early optimization easier, but it can also introduce geometric bias and soften the recovered surface.
Low β. A narrow transition concentrates rendering near the surface and can support more accurate localization once geometry is reasonably estimated. It may make optimization harder because useful gradients are more localized. A lower β therefore does not by itself guarantee a more accurate learned surface.
Eikonal loss. The regularizer encourages the spatial gradient of the predicted distance to have unit norm, helping the field behave like a signed distance function.
LEikonal = Ex [(‖∇s(x)‖₂ − 1)²]
alpha: 10.0 beta: 0.05 eikonal_weight: 0.02 num_epochs: 100 batch_size: 1024 lr: 0.0005
Each Gaussian’s scale and rotation define its 3D covariance. The center is projected into image space, and the covariance is transformed using the camera rotation and perspective-projection Jacobian.
The projected covariance defines the quadratic Gaussian exponent at each pixel.
Gaussian centers are transformed into camera space. Gaussians behind the camera are removed, and the remaining Gaussians are sorted from near to far.
Pixel alpha is the Gaussian opacity multiplied by its unnormalized Gaussian response. Exclusive cumulative products of (1 − α) produce front-to-back transmittance.
RGB, depth, and mask are accumulated with weights αᵢTᵢ. Remaining transmittance contributes the background color.
Isotropic Gaussians are initialized from the provided truck point cloud. Their means, scales, opacities, and colors are optimized with Adam. Rotation is not optimized for isotropic Gaussians.
| Parameter | Learning rate |
|---|---|
| Means | 0.0001 |
| Scales | 0.005 |
| Opacities | 0.01 |
| Colors | 0.0025 |
Training minimizes L1 loss between rendered images and the corresponding ground-truth images. Gradient checkpointing is used for the Gaussian batches to reduce the memory required for backward propagation.
| Setting / metric | Value |
|---|---|
| Training iterations | 1000 |
| Gaussians per splat | 128 |
| Held-out PSNR (dB) | 25.881 |
| Held-out SSIM | 0.882 |
The pretrained spherical harmonic coefficients are evaluated using normalized camera-to-Gaussian directions. Degree 0–3 basis functions produce view-dependent RGB colors; geometry and opacity are unchanged.
The rendering with spherical harmonics shows slightly more natural shading, particularly in the darker regions of the chair. Compared with the DC-only rendering, it captures more subtle variations in appearance across viewing directions, although the overall difference is modest. This improvement comes from view-dependent color modeling rather than changes to geometry or explicit shadow computation.