Robert Ren robertre 16-825 Learning for 3D Vision
I asked about the fundamental limits of multi-view 3D reconstruction. The part of the answer I found most interesting is that it refuses the premise of the question: the ceiling has almost nothing to do with the model being feedforward, and everything to do with which parts of the scene the input rays actually constrain.
Prompt
Assume N cameras with perfectly known intrinsics and extrinsics, noiseless sensors, and unlimited compute. What is the fundamental upper bound on the geometry and appearance fidelity I can recover from a scene using feedforward3D reconstruction models?
ChatGPT
The fundamental ceiling is not set by whether the reconstruction model is feedforward. It is set by the information contained in the \(N\) images.
Let the true scene be
$$ S=(G,A) $$with geometry \(G\) and appearance \(A\), and let the noiseless images be
$$ I_i=\mathcal R(S,C_i),\qquad i=1,\ldots,N, $$where \(C_i\) is perfectly known. The observations identify only the equivalence class
$$ \mathcal E(I)= \left\{ S' : \mathcal R(S',C_i)=I_i \quad \forall i \right\}. $$No reconstruction method—feedforward network, bundle adjustment, NeRF optimization, unlimited-compute search—can distinguish two scenes in \(\mathcal E(I)\).
So for an arbitrary scene and a finite number of cameras, perfect full-scene reconstruction is fundamentally impossible.
For geometry, there are three qualitatively different regions.
This is where the upper bound can be extremely high. With perfect calibration, noiseless measurements, good correspondences, and a sufficiently nice opaque surface, geometry can in principle approach the measurement-resolution limit.
For a simple stereo pair,
$$ Z=\frac{fB}{d}, $$so approximately
$$ \delta Z \approx \frac{Z^2}{fB}\delta d. $$Thus larger baselines, higher resolution, and more favorably positioned cameras improve the achievable depth precision. With ideal continuous images and exact correspondences, the mathematical triangulation error could even go to zero.
But finite-resolution cameras still impose a spatial-frequency/sampling limit.
Their depth is generally not identifiable.
For example, a textured patch could lie at multiple depths with appropriately warped texture and produce the exact same image. Priors can guess its geometry, but the observation itself does not determine it.
There is essentially no geometric information at all.
If this is the observed object:
$$ \boxed{\text{front}} $$the backside could be
$$ \boxed{\text{flat}},\quad \boxed{\text{rounded}},\quad \boxed{\text{deep cavity}},\quad \boxed{\text{arbitrary structure}} $$without changing any of your \(N\) images.
This remains true even if the images are infinite-resolution and noiseless.
There are also ambiguities even in visible regions. A perfectly textureless wall, for instance, may give you silhouettes and occlusion boundaries but not enough correspondence information to uniquely determine its exact shape without additional assumptions.
There's an important distinction between observed radiance and intrinsic appearance.
For every captured camera ray,
$$ r_{ij} \rightarrow I_i[j], $$you know the outgoing radiance along that ray essentially perfectly. Therefore, in the ideal setting a model can reproduce the training views exactly.
But that does not mean you know how the surface should look from arbitrary new viewpoints.
Imagine a material whose outgoing radiance is
$$ L_o(x,\omega). $$Your \(N\) cameras only observe
$$ L_o(x,\omega_1), \ldots, L_o(x,\omega_k) $$for the directions from which that point happened to be visible.
Infinitely many functions \(L_o(x,\omega)\) can agree at those \(k\) directions and differ everywhere else.
This becomes particularly severe for specular, glossy, translucent, refractive, anisotropic, or otherwise strongly view-dependent materials.
And if by "appearance" you mean physical properties such as
$$ \text{albedo},\quad \text{BRDF},\quad \text{roughness},\quad \text{metallicity},\quad \text{lighting}, $$then there are additional inverse-rendering ambiguities. Under one fixed illumination,
$$ \text{image} = \text{geometry} + \text{material} + \text{lighting}, $$and different combinations of those variables can produce exactly the same images.
So even observing a point from many cameras does not necessarily let you uniquely recover its physical BRDF.
For finite \(N\):
| Quantity | Fundamental best case |
|---|---|
| Captured RGB pixels | Exact |
| Geometry jointly visible in several good views | Potentially arbitrarily accurate, down to imaging/sampling limits |
| Geometry seen in only one view | Generally ambiguous |
| Occluded/unseen geometry | Unrecoverable from observations |
| Radiance along observed viewing directions | Exact |
| Radiance at unobserved viewing directions | Underdetermined |
| Albedo / BRDF / lighting decomposition | Generally not uniquely identifiable |
| Entire true 3D scene | No exact guarantee for finite \(N\) |
And there's an even stronger theoretical result lurking here: if you put essentially no restrictions on what counts as a scene, you can construct completely different 3D geometry that reproduces all \(N\) camera images exactly. So in the worst case, even huge portions of seemingly "reconstructed" geometry need not be determined by the images.
A feedforward model effectively learns
$$ \hat S=f_\theta(I_1,\ldots,I_N). $$When
$$ |\mathcal E(I)|>1, $$it has to choose one member of the equivalence class using its learned prior.
That's why modern feedforward systems can produce convincing backsides, fill holes, make textureless surfaces smooth, and complete objects. They are effectively estimating something like
$$ p(S\mid I_1,\ldots,I_N). $$A strong prior can make the expected reconstruction extraordinarily good on the natural-scene distribution. But that extra fidelity is prediction, not information recovered from the cameras.
A useful decomposition is therefore
$$ \boxed{ \text{recoverable 3D} = \text{image-constrained information} + \text{prior-based inference} } $$The first has a hard physical/information-theoretic limit. The second can improve indefinitely with better priors, but cannot have a worst-case guarantee.
This distinction is especially important when evaluating feedforward 3D methods: with enough views, the interesting question increasingly becomes not "how powerful is the network?" but "which parts of its output are actually determined by the input rays?"
A dolphin represented as a triangle mesh.