Method¶
3D Asset Generation¶
Our model automates 3D asset generation by mapping 2D images of objects to textured 3D meshes with plausible physics parameters. The images can be:
- Real images taken in the robot's environment,
- Real images provided by Google search under relevant category names, e.g., "avocado", or
- Images generated by pre-trained text-conditioned diffusion models, such as stable diffusion, prompted appropriately to generate uncluttered images of the relevant objects, e.g., "an image of an individual avocado".
Given a real or generated 2D image of an object, we lift it to a 3D model by minimizing re-reprojection error and maximizing likelihood of its image renderings using a diffusion model. We provide background on image diffusion models below before we describe our 3D model fitting approach.
Image diffusion models¶
A diffusion model learns to model a probability distribution $p(x)$ by inverting a process that gradually adds noise to the image $x$. The diffusion process is associated with a variance schedule $\{\beta_t \in (0,1)\}_{t=1}^{T}$, which defines how much noise is added at each time step. The noisy version of sample $x$ at time $t$ can then be written as $x_t = \sqrt{\bar{\alpha}_t} x + \sqrt{1-\bar{\alpha}_t} \epsilon$, where $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{1})$ is a sample from a Gaussian distribution (with the same dimensionality as $x$), $\alpha_t = 1- \beta_t$, and $\bar{\alpha}_t = \prod_{i=1}^t \alpha_i$. One then learns a denoising neural network $\hat \epsilon = \epsilon_{\phi}(x_t;t)$ that takes as input the noisy image $x_t$ and the noise level $t$ and tries to predict the noise component $\epsilon$. Diffusion models can be easily extended to draw samples from a distribution $p(x|\text{cond})$ conditioned on a prompt $\text{cond}$, where $\text{cond}$ can be a text description, a camera pose, and image semantic map, etc. Conditioning on the prompt can be done by adding $text{cond}$ as an additional input of the network $\epsilon_\phi$. For 3D lifting, we build on Zero-1-to-3, a diffusion model for novel object view synthesis that conditions on an image view of an object and a relative camera rotation around the object to generate plausible images for the target object viewpoint, $text{cond} = [I_1,\pi]$. It is trained on a large collection $\mathcal{D'} = \{(x^i, text{cond}^i)\}_{i=1}^N$ of images paired with views and relative camera orientations as conditioning prompt by minimizing the loss:
\begin{equation*} \mathcal{L}_\text{diff}(\phi;\mathcal{D}') = \tfrac{1}{|\mathcal{D}'|} \sum_{x^i, text{cond}^i\in\mathcal{D}'} || \epsilon_{\phi}(\sqrt{\bar{\alpha}_t} x^i + \sqrt{1-\bar{\alpha}_t} \epsilon, text{cond}^i , t) - \epsilon ||^2. \end{equation*}Image-to-3D Mesh using Score Distillation Sampling¶
Given an image and relative camera pose 2D diffusion model $p(I|[I_0,\pi])$, we extract from it a 3D rendition of the input image $I_0$, represented by a differential 3D representation using Score Distillation Sampling (SDS). We do so by randomly sampling a camera pose $\pi$, rendering a corresponding view $I_\pi$, assessing the likelihood of the view based on a diffusion model $p(I_\pi|[I_0,\pi])$, and updating the differentiable 3D representation to increase the likelihood of the generated view based on the model. Specifically, the diffusion model is frozen and the 3D model is updated as:
\begin{equation*} \nabla(\theta)\mathcal{L}_{SDS}(\theta; \pi, text{cond}, t) = \mathbb{E}_{t,\epsilon}[w(t)(\epsilon_\phi(a_t I + \sigma_t\epsilon;t, text{cond}) - \epsilon) \cdot \nabla_{\theta} I ], \end{equation*}where $I = R(\theta, \pi)$ is the image rendered from a given viewpoint $\pi$. The loss we use to backpropagate to the 3D model parameters $\theta$ includes an image re-projection loss for the camera viewpoint of the input image, and score distillation for the other views, using a pre-trained view and pose conditioned image diffusion model of to measure 2D image likelihood. We use a two-stage fitting, wherein the first stage an instantNGP NeRF representation is used, similar to RealFusion, and in the second stage a mesh-based representation is initialized from the NeRF and finetuned differentiably, similar to Fantasia3D. More information of our score distillation sampling can be found in our website.
Texture generation¶
We augment the textures of our generated assets using the method of TEXTure which iteratively edits a mesh's texture by rendering the mesh from different viewpoints and updating the rendered 2D images. While domain randomization randomly re-textures simulated assets, TEXTure produces diverse yet plausible texture augmentations.
Generating plausible physical properties¶
The visual and collision parameters of an asset are generated from the Image-to-Mesh pipeline discussed above. To define 3D sizes and physics parameters for the generated 3D meshes, we query GPT-4 regarding the range of plausible width, height, and depth for each object, and the range of its mass given its category. We then scale the generated 3D mesh based on the generated size range. We feed the mass and 3D mesh information to MeshLab to get the inertia matrix for the asset. Our prompts for querying GPT for mass and 3D object size can be found on our website. We wrap the generated mesh information, its semantic name, as well as the physical parameters into URDF files to be loaded into our simulator.
Our architecture is as follows:
Physical Parameters using LLMs¶
Once we have the texture, geometry and physical parameters, we populate the URDF for use in simulations.
