Assignment 0: Getting Started

Robert Ren robertre 16-825 Learning for 3D Vision

1. ChatGPT

I asked about the fundamental limits of multi-view 3D reconstruction. The part of the answer I found most interesting is that it refuses the premise of the question: the ceiling has almost nothing to do with the model being feedforward, and everything to do with which parts of the scene the input rays actually constrain.

Prompt

Assume N cameras with perfectly known intrinsics and extrinsics, noiseless sensors, and unlimited compute. What is the fundamental upper bound on the geometry and appearance fidelity I can recover from a scene using feedforward3D reconstruction models?

ChatGPT

The fundamental ceiling is not set by whether the reconstruction model is feedforward. It is set by the information contained in the \(N\) images.

Let the true scene be

$$ S=(G,A) $$

with geometry \(G\) and appearance \(A\), and let the noiseless images be

$$ I_i=\mathcal R(S,C_i),\qquad i=1,\ldots,N, $$

where \(C_i\) is perfectly known. The observations identify only the equivalence class

$$ \mathcal E(I)= \left\{ S' : \mathcal R(S',C_i)=I_i \quad \forall i \right\}. $$

No reconstruction method—feedforward network, bundle adjustment, NeRF optimization, unlimited-compute search—can distinguish two scenes in \(\mathcal E(I)\).

So for an arbitrary scene and a finite number of cameras, perfect full-scene reconstruction is fundamentally impossible.

Geometry

For geometry, there are three qualitatively different regions.

1. Surfaces observed from multiple sufficiently different viewpoints

This is where the upper bound can be extremely high. With perfect calibration, noiseless measurements, good correspondences, and a sufficiently nice opaque surface, geometry can in principle approach the measurement-resolution limit.

For a simple stereo pair,

$$ Z=\frac{fB}{d}, $$

so approximately

$$ \delta Z \approx \frac{Z^2}{fB}\delta d. $$

Thus larger baselines, higher resolution, and more favorably positioned cameras improve the achievable depth precision. With ideal continuous images and exact correspondences, the mathematical triangulation error could even go to zero.

But finite-resolution cameras still impose a spatial-frequency/sampling limit.

2. Surfaces visible from only one camera

Their depth is generally not identifiable.

For example, a textured patch could lie at multiple depths with appropriately warped texture and produce the exact same image. Priors can guess its geometry, but the observation itself does not determine it.

3. Surfaces visible from no camera

There is essentially no geometric information at all.

If this is the observed object:

$$ \boxed{\text{front}} $$

the backside could be

$$ \boxed{\text{flat}},\quad \boxed{\text{rounded}},\quad \boxed{\text{deep cavity}},\quad \boxed{\text{arbitrary structure}} $$

without changing any of your \(N\) images.

This remains true even if the images are infinite-resolution and noiseless.

There are also ambiguities even in visible regions. A perfectly textureless wall, for instance, may give you silhouettes and occlusion boundaries but not enough correspondence information to uniquely determine its exact shape without additional assumptions.


Appearance

There's an important distinction between observed radiance and intrinsic appearance.

For every captured camera ray,

$$ r_{ij} \rightarrow I_i[j], $$

you know the outgoing radiance along that ray essentially perfectly. Therefore, in the ideal setting a model can reproduce the training views exactly.

But that does not mean you know how the surface should look from arbitrary new viewpoints.

Imagine a material whose outgoing radiance is

$$ L_o(x,\omega). $$

Your \(N\) cameras only observe

$$ L_o(x,\omega_1), \ldots, L_o(x,\omega_k) $$

for the directions from which that point happened to be visible.

Infinitely many functions \(L_o(x,\omega)\) can agree at those \(k\) directions and differ everywhere else.

This becomes particularly severe for specular, glossy, translucent, refractive, anisotropic, or otherwise strongly view-dependent materials.

And if by "appearance" you mean physical properties such as

$$ \text{albedo},\quad \text{BRDF},\quad \text{roughness},\quad \text{metallicity},\quad \text{lighting}, $$

then there are additional inverse-rendering ambiguities. Under one fixed illumination,

$$ \text{image} = \text{geometry} + \text{material} + \text{lighting}, $$

and different combinations of those variables can produce exactly the same images.

So even observing a point from many cameras does not necessarily let you uniquely recover its physical BRDF.


The most useful way to state the bound

For finite \(N\):

QuantityFundamental best case
Captured RGB pixelsExact
Geometry jointly visible in several good viewsPotentially arbitrarily accurate, down to imaging/sampling limits
Geometry seen in only one viewGenerally ambiguous
Occluded/unseen geometryUnrecoverable from observations
Radiance along observed viewing directionsExact
Radiance at unobserved viewing directionsUnderdetermined
Albedo / BRDF / lighting decompositionGenerally not uniquely identifiable
Entire true 3D sceneNo exact guarantee for finite \(N\)

And there's an even stronger theoretical result lurking here: if you put essentially no restrictions on what counts as a scene, you can construct completely different 3D geometry that reproduces all \(N\) camera images exactly. So in the worst case, even huge portions of seemingly "reconstructed" geometry need not be determined by the images.

What a feedforward model changes

A feedforward model effectively learns

$$ \hat S=f_\theta(I_1,\ldots,I_N). $$

When

$$ |\mathcal E(I)|>1, $$

it has to choose one member of the equivalence class using its learned prior.

That's why modern feedforward systems can produce convincing backsides, fill holes, make textureless surfaces smooth, and complete objects. They are effectively estimating something like

$$ p(S\mid I_1,\ldots,I_N). $$

A strong prior can make the expected reconstruction extraordinarily good on the natural-scene distribution. But that extra fidelity is prediction, not information recovered from the cameras.

A useful decomposition is therefore

$$ \boxed{ \text{recoverable 3D} = \text{image-constrained information} + \text{prior-based inference} } $$

The first has a hard physical/information-theoretic limit. The second can improve indefinitely with better priors, but cannot have a worst-case guarantee.

This distinction is especially important when evaluating feedforward 3D methods: with enough views, the interesting question increasingly becomes not "how powerful is the network?" but "which parts of its output are actually determined by the input rays?"

2. A 3D Shape

A dolphin represented as a triangle mesh.

A dolphin rendered as a triangle mesh, with visible wireframe edges
Source: Dolphin triangle mesh, Wikimedia Commons. Created by Wikipedia user Chrschn; released into the public domain.

3. A Funny GIF

An animated GIF of a Minion with a flashing siren on its head
Source: giphy.com/gifs/1wDy0NewJUS4w — clip from Despicable Me (Illumination). Re-encoded to 320 px / 15 fps to fit the 1 MB page limit.