Project 1: Rendering Basics with PyTorch3D

By Teja Kanthamneni


1. Practicing with Cameras

1.1 360-Degree Render

Rendered a 360-degree view of a cow mesh using look_at_view_transform.

360 Cow
Cow orbiting 360 degrees.

1.2 Dolly Zoom

Recreated the Vertigo effect by modifying camera distance and FOV simultaneously.

Dolly Zoom
Dolly zoom effect on cow mesh.

2. Practicing with Meshes

Constructed basic 3D shapes from scratch by manually defining vertices and faces, then applied a 360-degree rotation.

Tetrahedron
2.1 Tetrahedron: 4 vertices, 4 faces.
Cube
2.2 Cube: 8 vertices, 12 faces.

3. Re-texturing a Mesh

Retextured the cow mesh based on the relative Z-coordinate of each vertex to create a smooth color gradient across the cow mesh. Colors: Green [0, 0.7,0.5] to Purple [0.5, 0.2, 1]

Retextured Cow
Cow textured with a Z-axis color gradient.

4. Camera Transformations

Modified the relative rotation ($R_{rel}$) and translation ($T_{rel}$) to achieve four distinct camera viewpoints. ($R_{rel}$) rotates the camera from its initial configuration (inverse of how the mesh rotates). ($T_{rel}$) defines where the world origin is located relative to the camera.

Transform 1
Transform 1: 90-degree rotation around the Z-axis.
$$R_{rel} = \begin{bmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}, T_{rel} = \begin{bmatrix} 0 & 0 & 0 \end{bmatrix}$$
Transform 2
Transform 2: Translated back along the Z-axis to zoom out.
$$R_{rel} = I, T_{rel} = \begin{bmatrix} 0 & 0 & 2 \end{bmatrix}$$
Transform 3
Transform 3: Shifted slightly right and down.
$$T_{rel} = \begin{bmatrix} 0.5 & -0.5 & 0 \end{bmatrix}$$
Transform 4
Transform 4: 90-degree rotation around Y to view from the side, adjusted translation to keep subject in frame.
$$R_{rel} = \begin{bmatrix} 0 & 0 & 1 \\ 0 & 1 & 0 \\ -1 & 0 & 0 \end{bmatrix}, T_{rel} = \begin{bmatrix} -3 & 0 & 3 \end{bmatrix}$$

5. Rendering Generic 3D Representations

5.1 Rendering Point Clouds from RGB-D Images

Unprojected RGB-D images into 3D point clouds, applying a coordinate flip to map them properly to PyTorch3D space.

Plant View 1
View 1 Point Cloud
Plant View 2
View 2 Point Cloud
Combined Plant
Combined Point Cloud (Union)

5.2 Parametric Functions

Sampled continuous parametric equations to generate 3D point clouds.

Torus Point Cloud
Torus Point Cloud
$$x = (R + r \cos \phi) \cos \theta$$ $$y = r \sin \phi$$ $$z = (R + r \cos \phi) \sin \theta$$
Mobius Point Cloud
Möbius Strip Point Cloud
$$x = (R + \phi \cos(\frac{\theta}{2})) \cos \theta$$ $$y = \phi \sin(\frac{\theta}{2})$$ $$z = (R + \phi \cos(\frac{\theta}{2})) \sin \theta$$

5.3 Implicit Surfaces

Utilized marching cubes to extract a mesh from an implicitly defined 3D voxel grid $F(x,y,z) = 0$.

Implicit Torus
Implicit Torus
$$F(x,y,z) = (x^2 + z^2 + y^2 + R^2 - r^2)^2 - 4R^2(x^2 + z^2) = 0$$
Custom Implicit Surface
Taubin's Heart
$$F(x,y,z) = (x^2 + 2.25z^2 + y^2 - 1)^3 - x^2y^3 - 0.1125z^2y^3 = 0$$

Tradeoffs: Meshes vs. Point Clouds

When selecting a 3D representation for rendering or processing, there are several key tradeoffs to consider:

Criteria Comparison Summary
Rendering Speed Meshes are generally slower to render because they require rasterization, normal calculations, z-buffering, and shading algorithms. Point clouds are significantly faster since they only require projecting 3D coordinates onto the 2D screen. However, depending on the desired quality/fidelty of the rendering, a greater number of points may be required for a point cloud, which can make it slower than a mesh rendering.
Rendering Quality Meshes provide higher rendering quality due to their continuous surfaces, support for lighting and smooth texturing. In contrast, point clouds have lower visual fidelity and can appear to have holes unless the point density is high.
Ease of Use Point clouds are simpler to process and capture directly from hardware because they are unorganized, raw coordinates. Meshes are more complex to generate and manipulate due to their connectivity.
Memory Usage Point clouds are generally more memory efficient as they only store a 3D coordinate and potentially a color, while meshes require the face connectivity matrix. However, to achieve similar quality renderings, meshes could end up using less memory as it can leverage its connectivity to fill in gaps.

6. Extra Credit: Point Cloud Morphing

Implemented a linear interpolation algorithm to morph a parametric Torus point cloud into a Sphere over time.

Torus to Trefoil Morph
Point cloud morphing from a Torus to a Sphere.

7. Extra Credit: Sampling Points on Meshes

Converted a 3D mesh into a point cloud by sampling faces proportionally to their area, then uniformly sampling barycentric coordinates using the square root trick to prevent vertex clustering.

10 Sampled Points
10 Points
100 Sampled Points
100 Points
1000 Sampled Points
1,000 Points
10000 Sampled Points
10,000 Points
Original Cow Mesh
Original Textured Mesh (For Comparison)