All experiments were run on a CPU (Apple M4 Max) in PyTorch, so every hard-coded
.cuda() call was replaced by a device variable. No other change was made to the
provided training or rendering loops unless stated below.
get_pixels_from_image maps pixel indices to NDC with
x = 2i / (W − 1) − 1 (same for y), so the corner pixels land
exactly on ±1, which matches grid_sample(align_corners=True) used later to
read ground-truth colours. get_rays_from_pixels places the NDC points on the plane
z = 1, lifts them to world space with camera.unproject_points, uses the
camera centre as the ray origin, and normalizes the direction so that distances along a ray are
metric.
Both images match the reference outputs. As a sanity check, the centre ray direction is (0, 0, 1) and every corner ray is (±0.577, ±0.577, 0.577), i.e. 1/√3 per component for a 90° field of view.
StratifiedRaysampler places n_pts_per_ray depths uniformly in
[min_depth, max_depth] and computes
p = o + t·d for every ray by broadcasting. The figure shows all
65,536 × 64 samples of the first camera. The far boundary is an arc, not a line, because
the samples are placed at equal distance from the camera along unit directions.
For each ray with samples ti:
T is computed with an exclusive cumulative product (a column of ones is prepended to
1 − α before torch.cumprod), and _aggregate is a
weighted sum over the sample dimension, which is reused for both colour and depth.
get_random_pixels_from_image draws a random permutation of the pixel indices and keeps
the first n_pixels (sampling without replacement). The loss is the mean squared error
between rendered and ground-truth colours. Training uses 1000 epochs over the 4 views with 4096
rays per step.
| x | y | z | |
|---|---|---|---|
| Box center | 0.25 | 0.25 | 0.00 |
| Box side lengths | 2.01 | 1.50 | 1.50 |
Comparison with the reference GIF. The two GIFs are nearly identical. Across the 20 frames, the foreground area of our box (pixels whose R + G + B exceeds 30 on the 0 to 255 scale) is 98.5% to 100.2% of the reference (mean 99.2%), the bounding boxes differ by at most one pixel, and the mean absolute pixel difference is 0.3%, which is at the level of GIF colour quantization.
Architecture. The 3D position is encoded with HarmonicEmbedding
(6 frequencies, 39 dimensions with the raw input). A 6-layer MLP with 128 hidden units and a skip
connection that re-injects the embedding at layer 3 (MLPWithInputSkips) produces a
128-d feature, followed by a ReLU. Density is ReLU(Linear(feature)) and colour is
Sigmoid(Linear(feature)). The loss is MSE; training follows
nerf_lego.yaml (250 epochs, 128×128, 1024 rays and 128 samples per ray,
lr 5e-4 with step decay).
Initialization detail: with a ReLU on the density output, roughly one third of random initializations produce zero density for every sample, so the gradient is exactly zero and the model never trains. I later initialized the density head bias to 0.1, which removed this failure in 200 out of 200 seeds. The lego result above was trained before this change and did not hit the degenerate case.
Method. Density depends only on position, as in Q3. For colour, the 128-d
position feature is concatenated with the harmonic embedding of the viewing direction
(2 frequencies, 15 dimensions) and passed through a small head
Linear(143, 64) → ReLU → Linear(64, 3) → Sigmoid. The model is
trained on the materials scene with the settings of nerf_materials.yaml
(128×128, 250 epochs, lr 1e-4).
Trade-offs. View dependence lets the model reproduce effects such as the highlights on the metal spheres, which a position-only colour has to average out. The cost is that the network gains an easy way to fit the training images without learning consistent geometry: it can paint each training view with a different colour instead of placing surfaces correctly, which looks good on training views and breaks on novel ones. Three choices in the design limit this. Density never sees the direction, so geometry is shared by all views (I checked that flipping the direction leaves density unchanged and only changes colour). The direction uses a low-frequency embedding (2 frequencies), so colour can only vary smoothly with the viewpoint. The direction branch is small (one 64-unit layer), so most of the capacity stays in the view-independent trunk. More view dependence (more frequencies or a larger branch) would capture sharper reflections but increases the risk of overfitting, especially with few training views.
Method. Two NeRF networks with the Q3 architecture are trained jointly. The
coarse pass uses 64 uniformly spaced samples per ray and renders with the coarse network. Its
weights (excluding the first and last sample, plus 1e-5) are normalized into a piecewise-constant
PDF over the bins between sample midpoints, and 128 additional depths are drawn by inverse
transform sampling (torch.searchsorted on the CDF with linear interpolation inside the
bin). The fine network then renders all 192 sorted samples. The loss is the sum of the coarse and
fine MSE, and the fine samples are detached so the coarse network is only supervised by its own
rendering.
| Network evaluations per ray | Time / epoch | Total time | PSNR vs reference GIF | |
|---|---|---|---|---|
| Q3 (uniform, 128 samples) | 128 | ~15 s | 75 min | 31.80 dB |
| 4.2 (64 coarse + 192 fine) | 256 | ~23 s | 110 min | 31.38 dB |
Trade-offs. Hierarchical sampling spends samples where the coarse pass found mass, so in principle it resolves surfaces more finely than uniform sampling with the same number of fine samples. It roughly doubles the work per ray, since two networks are evaluated and the fine network sees 192 points, and it made training about 1.5× slower here. At this low resolution (128×128) the visual difference to Q3 is small. Two caveats on the comparison: the reference GIF is itself a NeRF rendering rather than a ground-truth photo, so the PSNR column measures agreement with the reference implementation, and the two models do not use the same total number of samples. The benefit of coarse/fine sampling should grow at higher resolution or when the sample budget per ray is small, which is when uniform samples miss thin structures.
mean((‖∇d‖ − 1)²) on
random points in the bounding box, weight 0.02.σ(x) = α Ψβ(−d(x)) where
Ψβ is the CDF of a zero-mean Laplace distribution with scale β. Each
branch of the piecewise formula is clamped so that the unused branch of
torch.where cannot overflow and produce NaN gradients.All three versions use α = 10, β = 0.05, 128 samples per ray and 250 epochs. They differ in how the network is initialized and how colour is predicted.
| Version | Colour PSNR vs reference GIF | Geometry silhouette IoU vs reference |
|---|---|---|
| (1) Independent colour MLP, default init | 30.24 dB | 0.24 |
| (2) + geometric (sphere) initialization | 23.58 dB | 0.85 |
| (3) + colour from geometry features | 33.24 dB | 0.77 |
(1) Default initialization. In this run the initial network output was a distance of about −0.01 everywhere (the sign depends on the random seed), so the whole volume started "inside" with a density of 10·(1 − ½e−0.01/0.05) ≈ 5.9, more than half of α. Because the background of every training image is black, a black opaque wall explains the background just as well as empty space, and training never carves out the regions behind and below the object. The colour looks fine, but the extracted geometry is a large slab with the bulldozer embedded in it.
(2) Geometric initialization. The distance MLP is initialized so that it outputs
approximately ‖x‖ − 1 (the IGR/SAL scheme: hidden weights
N(0, 2/out), output weights with mean √(π/in) and bias −r, and the sine/cosine
inputs zeroed in the first layer). Space starts empty outside a unit sphere and training only has
to sculpt the sphere into the object. The geometry becomes clean, but the colour MLP, which is
shallow (one hidden layer) and reads the high-frequency position encoding directly, paints
coloured stripes in the thin shell outside the surface where density is small but non-zero.
(3) Geometry features. The distance MLP outputs 1 + 128 values: the first is the signed distance and the remaining 128 form a feature vector that is fed, together with the position encoding, to the colour MLP (the IDR/VolSDF design). Geometric initialization is applied only to the distance row. Colour now follows the surface, the stripes disappear, and the colour PSNR rises by almost 10 dB. Because the colour loss now also updates the shared trunk, the geometry is slightly worse than in (2) around the base plate. Version 3 is my best model overall because of its much better colour, while version 2 has the cleanest geometry.
| Setting | Value | Reason |
|---|---|---|
| α | 10 | With 128 samples in [2, 6], samples are 0.031 apart. Inside the object each sample then has opacity 1 − exp(−10 × 0.031) ≈ 0.27, so a ray becomes nearly opaque within about ten samples (0.3 units) after crossing the surface. This is solid enough for the surface to occlude what is behind it, while a single sample does not saturate. |
| β | 0.05 | The density goes from 0.9α to 0.1α over about 3.2β = 0.16 units, which spans about five samples. A much smaller β would make the transition thinner than the sample spacing, so samples could step over the surface and receive little gradient. |
| Eikonal weight | 0.02 | Keeps ‖∇d‖ close to 1 as a regularizer without dominating the colour loss. |
| Geometric init radius | 1.0 | Comparable to the size of the object. It only has to start with empty space outside; training reshapes the sphere into the bulldozer. |
| Geometry feature size | 128 | Equal to the hidden width of the distance MLP. |
α, β and the eikonal weight are the values provided in volsdf_surface.yaml; I did not sweep them.
My experiments instead varied the initialization and the colour branch, which turned out to be the factors limiting the
results above.
Ψβ(−d) goes from 0 far outside the surface to 1 deep inside it and equals 1/2 on the surface. α scales this, so it is the density of the interior: deep inside the object σ ≈ α, on the surface σ = α/2, and a larger α makes the solid more opaque per unit length. β is the width of the transition: the density falls from α to 0 over a band of a few β around the zero level set. As β → 0 the density becomes a step function at the surface, and a large β turns the surface into a thick fog.
camera.transform_points_screen.All four unit tests pass, and frame 29 of the GIF matches the provided reference frame (PSNR 74.9 dB on the RGB panel).
The submitted results use W = camera.R, as instructed. While implementing 5.1 I found
that the geometrically correct choice is W = camera.RT.
Derivation. PyTorch3D uses row vectors: x_cam = x_world R + T. The
covariance formula Σ' = WΣWT uses column vectors,
x_cam = W x_world. Transposing the PyTorch3D form gives
x_camT = RT x_worldT, so W = RT.
Monte Carlo check. For the five anisotropic Gaussians of unit test 2 (scaled down
so the linearization holds), I sampled 200,000 points from each 3D Gaussian, projected them with
camera.transform_points_screen, and measured the covariance of the projected pixels.
With W = RT the predicted 2D covariance matches the empirical one within 0.8%; with
W = R the error reaches 131% and the sign of the off-diagonal term is flipped for several
Gaussians, i.e. the ellipses are tilted the wrong way.
Effect on renders. The pre-trained chair was optimized with the correct projection in the original codebase, so W = RT reproduces it more faithfully: with W = R, flat Gaussians are rotated off the surface and appear as thin streaks.
With W = RT, unit tests 2 and 4 fail because their expected values were generated with W = R. Isotropic Gaussians (Q6 and Q7.2) are unaffected: Σ = s²I and W(s²I)WT = s²I for any rotation, which I also confirmed numerically (maximum difference 1.2e-7).
Means, scales, colours and opacities are made trainable; quaternions are not, since the Gaussians are isotropic and a rotation has no effect. The loss is L1 between the rendered and ground-truth images (black background).
| Parameter | Learning rate |
|---|---|
| Opacities (pre-activation) | 0.01 |
| Scales (log) | 0.001 |
| Colours | 0.01 |
| Means | 0.0001 |
The means get the smallest learning rate because they are initialized from the provided point cloud and are already close to the right positions.
| Iterations | Mean PSNR (test) | Mean SSIM (test) |
|---|---|---|
| 1000 | 28.20 | 0.930 |
colours_from_spherical_harmonics evaluates all 16 real SH basis functions up to degree
3 for every Gaussian in a vectorized way and returns clamp(SH·Y(d) + 0.5, 0, 1),
where d is the normalized direction from the camera centre to the Gaussian mean. It matches the
eval_sh function of the original 3DGS code to 2e-7 on random inputs, and keeping only
the degree-0 coefficients reproduces the DC colours used in Q5 exactly. A --use_sh
flag in render.py switches between the two colour models.
The fabric on the back and seat is darker and the golden pattern stands out much more with SH. The DC colour is the average over all viewing directions, which washes out the sheen of the fabric and makes it look flat and bright; the higher-order terms restore how the fabric reflects light towards this particular viewpoint.
Shading on the back and seat varies more with SH, and the carved golden trim on the frame looks glossier. These are view-dependent effects, so the DC-only render cannot express them.
Almost no difference. This view mostly shows the matte wooden underside, which is close to diffuse, so its colour hardly depends on the viewing direction and the higher-order coefficients are small. (The streaks around the legs come from W = R, see Section 5.x.)
The baseline uses the Q6 setup unchanged: 10,000 isotropic Gaussians with
init_type="random", the Q6 learning rates, L1 loss and 1000 iterations. All runs in
this section share the same random seed, the same order of training views, and are evaluated on
the same 100 validation views.
| Method (1000 iterations) | PSNR | SSIM |
|---|---|---|
| Baseline (Q6 setup) | 16.27 | 0.616 |
| A+L (best) | 21.23 | 0.672 |
In the baseline the Gaussians never leave the small cluster around the origin where they are initialized, so every sphere is missing. With A+L every sphere appears at the correct position and both metrics improve; the remaining errors are blur and missing material detail, which is expected from 10,000 isotropic Gaussians and 1000 iterations.
knn_points with K = 4,
excluding the point itself), as in the original 3DGS, so the Gaussians tile the volume
without large gaps.I initialized the Gaussians with each radius, rendered 20 training views, and measured the fraction of ground-truth object pixels covered by the initial silhouette (mask > 0.3).
| Radius | Object pixels covered (mean) | Worst view | Covered area that is background |
|---|---|---|---|
| 1.0 | 89.7% | 84.4% | 61% |
| 1.3 | 100.0% | 99.9% | 72% |
| 1.5 | 100.0% | 100.0% | 74% |
| 2.0 | 100.0% | 100.0% | 75% |
A radius of 1.0 misses the outer spheres. Larger radii spread the same 10,000 Gaussians over more empty space, so I used the smallest radius that covers the scene plus a small margin (1.4).
| Run | Init | Means lr | PSNR | SSIM |
|---|---|---|---|---|
| B | baseline | 1e-4 | 16.12 | 0.607 |
| A | new | 1e-4 | 18.59 | 0.462 |
| L | baseline | 1e-3 | 16.25 | 0.614 |
| A+L | new | 1e-3 | 19.08 | 0.504 |
L alone changes nothing visible: a 10× larger step is still too small for points that start in a tight cluster. Initialization is what matters, and the larger learning rate helps only once the Gaussians start in the right region (A+L > A). The SSIM of B and L is misleading here: about 75% of each image is black background, and a render that is almost entirely black scores well on those regions even though every object is missing. At 300 iterations A and A+L were still clearly improving while B had plateaued, so A+L was continued to 1000 iterations.
| Run | PSNR | SSIM |
|---|---|---|
| Baseline | 16.27 | 0.616 |
| A+L | 21.23 | 0.672 |
| A+L + exponential decay of the means lr (1e-3 to 1e-4) | 21.13 | 0.666 |
| A+L + decay + SSIM loss (0.8·L1 + 0.2·(1 − SSIM)) | 21.05 | 0.621 |
Neither learning-rate decay nor the SSIM loss improved the result within 1000 iterations. With the decay, the PSNR difference was approximately 0.1 dB; more seeds are needed to assess its significance. The SSIM loss lowered validation SSIM by 0.045. I did not run experiments to isolate the causes, so the following are possible explanations rather than conclusions:
A repeat of the baseline without a fixed seed gave 16.26 / 0.617, differing by 0.01 dB from the seeded run. This single repeat suggests that seed variation is small compared with the 5 dB gap between the baseline and A+L, although more seeds would be needed to quantify it.