16-825 Learning for 3D Vision
get_pixels_from_image places one sample at each pixel
centre and maps it to NDC in
.
get_rays_from_pixels lifts each NDC point to depth 1 and
unprojects it to world space with
camera.unproject_points(..., from_ndc=True). Each ray
starts at the camera centre and points through its unprojected pixel.
Directions are normalised, so sample depths are distances along the
ray.
| xy grid | ray directions |
|---|---|
![]() |
![]() |
StratifiedRaysampler places n_pts_per_ray
evenly spaced depths
along every ray and returns the points
.
The figure shows every tenth sample from the first camera.
Weights follow the emission–absorption model, with :
The transmittance is an exclusive cumulative product of the
per-segment survival probabilities. The same weights aggregate colour
(_aggregate) and depth, so the depth map comes from the
same pass. The depth map below is normalised by its maximum value.
Spiral render
(part_1.gif) |
Depth (normalised) |
|---|---|
![]() |
![]() |
get_random_pixels_from_image takes a random permutation
of the full pixel grid and keeps the first n_pixels
entries. The loss is the mean squared error between rendered and
ground-truth colours. After 1000 epochs (batch 4096, lr 5e-4):
| x | y | z | |
|---|---|---|---|
| Box center | 0.25 | 0.25 | 0.00 |
| Box side lengths | 2.01 | 1.50 | 1.50 |
| View 0 | View 1 | View 2 | View 3 | |
|---|---|---|---|---|
| Before | ![]() |
![]() |
![]() |
![]() |
| After | ![]() |
![]() |
![]() |
![]() |
Architecture. The input point is positionally
encoded with 6 frequencies (39 dims) and passed through a 6-layer,
128-unit ReLU MLP (MLPWithInputSkips). The encoding is
concatenated back in at layer 3. Two heads read the trunk features:
Linear(128 → 1) followed by
ReLU, so that
.Linear(128 → 128), then
Linear(128 → 64) with ReLU, then
Linear(64 → 3) with a sigmoid.View dependence is implemented behind a use_view_dir
config flag, but it is off here because this section asks for a
position-only model.
Training. The nerf_lego config: 250
epochs at 128×128, 1024 rays per step, 128 samples per ray in
,
Adam at lr 5e-4 decaying ×0.8 every 50 epochs, and an MSE loss. The
final training MSE is about 0.0025.
The render recovers the bulldozer’s overall shape, the arm and bucket, the tracks, and the texture of the base plate. Since the model has no view dependence, the colours look matte from every angle.
Implementation.
NeuralSurface: a shared trunk
(positional encoding with 6 frequencies, then 6 × 128 ReLU layers) feeds
two heads. The distance head is a linear layer with no activation,
because a signed distance can be negative. The colour head is a 2-layer
MLP with a sigmoid output. get_distance_color runs the
trunk once for both outputs.eikonal_loss:
,
computed on random points in
with weight 0.02.sdf_to_density: follows VolSDF §3.1,
,
where
is the CDF of a zero-mean Laplace distribution with scale
:The code evaluates this as 0.5 * exp(-|d| / β) so that
the exponential never overflows.
Geometry
(part_4_3_geometry.gif) |
Colour (part_4_3.gif) |
|---|---|
![]() |
![]() |
Settings. I used the provided
volsdf_surface settings: α = 10, β = 0.05, eikonal weight
0.02, 250 epochs, and lr 5e-4. With β = 0.05 the surface transition is
about 1–2% of the scene size, sharp enough to resolve the bucket, cab
and tracks. With α = 10 the interior becomes opaque within a few of the
128 samples per ray. The remaining artifacts are small floaters near the
base plate, where the training views give little constraint.
What α and β do. α is the density deep inside the object: as grows, , so interior points get density α. It also caps how opaque a thin shell can become. β is the scale of the Laplace CDF, so it sets the width of the transition from empty (density ≈ 0) to solid (density ≈ α) around the zero level set. Small β gives a sharp, almost binary surface. Large β gives a soft, foggy shell several β thick.
How does high β bias the learned SDF? What about low β? With high β, density leaks well outside the surface and stays low just inside it. The rendered surface therefore sits in front of the true zero level set, and the network can explain the images with blurry, over-smoothed geometry: thin parts and sharp edges get rounded away. With low β, density approaches an indicator function of the interior. The renderer then sees only the zero crossing, which pushes the zero level set onto the true surface, but only points very close to the surface receive gradient.
Would an SDF be easier to train with low β or high β? Why? High β is easier. Every sample within a few β of the surface gets non-zero weight and gradient, so even a poorly initialised SDF gets a learning signal from distant samples, and the loss surface is smooth. With low β the density is close to a step function. Samples that miss the thin transition band receive no gradient, so optimisation stalls easily and becomes sensitive to sample spacing. This is why VolSDF and NeuS anneal β from large to small, or learn it.
Would you be more likely to learn an accurate surface with high β or low β? Why? Low β, as long as training still converges. With a sharp transition, colour is attributed to points right at , so the zero level set has to lie on the real surface to explain the images. With high β, many surface placements explain the images equally well, and the recovered surface is blurrier.
5.1 Projection. The 3D covariance is
(quaternion_to_matrix for
).
For isotropic Gaussians it is simply
,
and the quaternion is never touched. The 2D covariance is
,
using the provided Jacobian and the world-to-view rotation
.
Means are projected to pixels with
camera.transform_points_screen.
5.2 Evaluation.
,
evaluated for all Gaussians and pixels at once with a single
einsum.
5.3 Filter and sort. Depth is the view-space
of each mean. The Gaussians are sorted by argsort, and
those with
are dropped.
5.4 Alphas and transmittance.
and
.
The transmittance is a cumprod over the batch, with the
previous mini-batch’s final transmittance prepended.
5.5 Splatting. With : image , depth , and silhouette .
10,000 isotropic Gaussians are initialised from the provided point cloud. Means, log-scales, colours and opacity logits are trainable. Quaternions are not, since rotation has no effect on an isotropic Gaussian. Each step renders one training view on a black background and applies an L1 loss. The optimiser is Adam.
| Parameter | Learning rate |
|---|---|
| Opacities (pre-sigmoid) | 0.05 |
| Colours | 0.02 |
| Scales (log) | 0.005 |
| Means | 0.0005 |
The means get the smallest learning rate: the point cloud already places them well, and large steps scatter them. Opacity logits start at 8, where the sigmoid is saturated, so they need a large learning rate to move at all.
colours_from_spherical_harmonics reshapes the 48
coefficients per Gaussian to
(degree 0–3, RGB). It evaluates the 16 real SH basis functions at the
unit direction from the camera centre to each Gaussian
(calculate_gaussian_directions), and returns
.
All Gaussians are processed in one vectorised pass.
render.py --use_sh enables it and writes
part_7_1_render.gif.
5.5: DC only
(part_5_render.gif) |
7.1: full SH
(part_7_1_render.gif) |
|---|---|
![]() |
![]() |
Side-by-side comparisons (left: DC only, right: full SH, same frame).
Without SH, the seat and back are a flat mid-green. With SH, the fabric under the backrest and on the inner seat is darker. This is the shading and self-occlusion present in the training images. The damask pattern also has more contrast.
At this grazing angle into the seat, the SH render is noticeably darker and more saturated, and the golden embroidery stands out. The DC colour is an average over all viewing directions, which washes out this view-dependent sheen of the silk.
From below, the two renders differ little because the cream base is matte in both. SH adds slightly warmer highlights on the gilded carving and the armrest scrolls. Diffuse surfaces gain little from higher-order terms, while specular trim gains the most.