0. Transmittance Calculation (10 points)

1. Differentiable Volume Rendering

1.3. Ray sampling (5 points)

Grid:

Rays:

Point sampling (5 points)

1.5. Volume rendering (20 points)

images/part_1.gif

depth

2. Optimizing a basic implicit volume

2.2. Loss and training (5 points)

Box center: (0.25, 0.25, 0.00) Box side lengths: (2.00, 1.50, 1.50)

2.3. Visualization

3. Optimizing a Neural Radiance Field (NeRF) (20 points)

4. NeRF Extras

4.1 View Dependence (10 points)

Outputs for: No view dependence (left) Vs. View dependence (right)

We can see the shine on the balls change as the view changes for the right gif compared to the one on the left.

Increased view dependence can lead to the model quickly overfitting to the training views, while not generalizing on novel views. The network will put more emphasis on the view rather than the structure during optimization, leading to overfitting.

5. Sphere Tracing (10 points)

The sphere tracing algorithm iteratively moves along the ray by the SDF value at any point. Once the SDF is less than a certain threshold (epsilon), it no longer continues to move further away. We run this algorithm on all rays upto a certain maximum number of iterations. In the end, certain rays would have hit the surface and stopped, while others would have crossed the far point.

6. Optimizing a Neural SDF (15 points)

Network Used:

self.harmonic_embedding_xyz = HarmonicEmbedding(3, cfg.n_harmonic_functions_xyz)
embedding_dim_xyz = self.harmonic_embedding_xyz.output_dim

self.sdf_mlp = MLPWithInputSkips(
    n_layers=cfg.n_layers_distance,
    input_dim=embedding_dim_xyz,
    output_dim=1, # sdf
    skip_dim=embedding_dim_xyz,
    hidden_dim=cfg.n_hidden_neurons_distance,
    input_skips=cfg.append_distance,
)

Here the distance can be negative so we don't apply the ReLU.

Hyperparameters (default):

implicit_function:
  type: neural_surface

  n_harmonic_functions_xyz: 4

  n_layers_distance: 6
  n_hidden_neurons_distance: 128
  append_distance: []

  n_layers_color: 2
  n_hidden_neurons_color: 128
  append_color: []

The eikonal loss is basically guiding the neural network to behave like an SDF with the property that for any given point, the magnitude of the gradient is 1.

7. VolSDF (15 points)

$$ \Psi_\beta(s) = \begin{cases} 1 - \frac{1}{2}e^{\frac{-s}{\beta}} & s \ge 0 \\ \frac{1}{2} e^{\frac{s}{\beta}} & s < 0 \end{cases} $$

• This function is the Cumulative Distribution Function (CDF) of a Laplace distribution, which peaks at $s=0$ (i.e., at the surface $d=0$) and falls off symmetrically.

• alpha is a simple scalar multiplier. It directly controls the maximum possible density in the volume. A high alpha means the surface is very dense and opaque, and light will be blocked (or colored) very quickly. A low alpha results in a more transparent, "ghostly" surface, where light can pass through.

• beta controls the spread of the density falloff as you move away from the $d=0$ isosurface. High beta (means slow exponent change) gives a thick, soft-looking surface where density is high over a wide range of distances. Low beta gives a hard thin surface.

• An SDF would be easier to train with volume rendering with a high beta initially, since it can assign some density to several points. With a low beta, the model needs to accurately find the surfaces and gradients for other points would be very low.

• Low beta would give more accurate surfaces.

hyperparameters that work the best:

alpha: 10.0
beta: 0.05
implicit_function:
  type: neural_surface

  n_harmonic_functions_xyz: 6

  n_layers_distance: 6
  n_hidden_neurons_distance: 128
  append_distance: []

  n_layers_color: 2
  n_hidden_neurons_color: 128
  append_color: []

Other Hyperparameters:

alpha=10, beta=0.15

As we can see, this doesn't give sharp surfaces

Trying alpha=2, beta=0.01 gives nans in the loss.

8. Neural Surface Extras

8.1. Render a Large Scene with Sphere Tracing (10 points)

8.2 Fewer Training Views (10 points)

Num views = 20 The NeRF doesn't give a clean output with 20 views.

VolSDF works with fewer views:

8.3 Alternate SDF to Density Conversions (10 points)

Naive solution from neus: $\phi_s(x) = \frac{se^{-sx}}{(1 + e^{-sx})^2}$