3D Rendering & Mesh Processing with PyTorch3D

Introduction

This project explores different 3D representations and rendering techniques using PyTorch3D. We generate point clouds, parametric surfaces, and implicit meshes, then render them into 360-degree GIFs.

The project consists of:

• 1: Practicing Camera Movements (360° Views & Dolly Zoom)
• 2: Constructing Simple Meshes (Tetrahedron & Cube)
• 3: Re-Texturing Meshes
• 4: Camera Transformations
• 5: Rendering Generic 3D Representations
• 6: Creative Rendering Challenge!

---

1. Practicing with Cameras

1.1 360-degree Renders (5 points)

We generate a 360-degree animation of the cow mesh by rotating the camera around it.

Cow Mesh (360° GIF)

How It Works:

• Used pytorch3d.renderer.look_at_view_transform() to control camera angles.
• Rendered frames from multiple azimuth angles and compiled them into a GIF.

---

1.2 Re-Creating the Dolly Zoom Effect (10 points)

The Dolly Zoom (Hitchcock Effect) keeps the subject the same size while changing focal length and distance.

Dolly Zoom GIF

How It Works:

• Moved the camera closer while adjusting focal length.
• Rendered a sequence of frames with FoVPerspectiveCameras.

---

2. Constructing Simple Meshes

2.1 Constructing a Tetrahedron (5 points)

We manually defined a tetrahedron mesh with 4 vertices and 4 triangular faces.

Tetrahedron 360° GIF

Mesh Details:

• Vertices: 4
• Faces: 4 triangular faces
• Used TexturesVertex to color the tetrahedron.

---

2.2 Constructing a Cube (5 points)

We built a cube as a triangular mesh (since each face is made of two triangles).

Cube 360° GIF

Mesh Details:

• Vertices: 8
• Faces: 12 triangles (2 per cube face)

---

3. Re-Texturing the Cow Mesh (10 points)

We smoothly changed the cow’s color from front to back by interpolating between two colors based on z-coordinates.

🖼 Retextured Cow

Chosen Colors:

• Front: color1 = [0, 0, 1] (Blue)
• Back: color2 = [1, 0, 0] (Red)

Formula Used: [ \text{color} = \alpha \cdot \text{color}2 + (1 - \alpha) \cdot \text{color}_1 ] where (\alpha = \frac{z - z{min}}{z_{max} - z_{min}})

---

4. Camera Transformations (10 points)

We adjusted camera extrinsics (R, T) to transform the cow mesh's viewpoint.

🖼 Camera Transformations

1. First Image - Side View (Rotated -90° Around Z-Axis)

Transformations Applied:

• Rotation (R_relative):
  • Rotates -90° around the Z-axis to turn the camera to the side.
  • The X-axis and Y-axis are swapped: [ R_{rel} = \begin{bmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} ]
• Translation (T_relative):
  • Moves the object forward (z = 4) so it appears in front of the camera.

Result:

• The cow is rotated 90° sideways and centered in front of the camera.

Rendered Image:

---

2. Second Image - Smaller Cow (Moves Closer to Camera)

Transformations Applied:

• Rotation (R_relative):
  • No rotation applied (Identity matrix): [ R_{rel} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} ]
• Translation (T_relative):
  • Moves the object closer to the camera (z = 2), making it appear smaller.

Result:

• The cow is not rotated but moves closer to the camera, reducing its size.

Rendered Image:

---

3. Third Image - Side View (Rotated 90° Counterclockwise Around Y-Axis)

Transformations Applied:

• Rotation (R_relative):
  • Rotates 90° counterclockwise about the Y-axis using: [ R_{rel} = \begin{bmatrix} \cos(90^\circ) & 0 & \sin(90^\circ) \\ 0 & 1 & 0 \\ -\sin(90^\circ) & 0 & \cos(90^\circ) \end{bmatrix} ]
  • This turns the cow sideways.
• Translation (T_relative):
  • Moves the cow to the left (x = -3) and in front (z = 4).

Result:

• The cow is turned 90° and moved forward.

Rendered Image:

---

4. Fourth Image - Larger Cow (Slight Translation)

Transformations Applied:

• Rotation (R_relative):
  • No rotation applied (Identity matrix): [ R_{rel} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} ]
• Translation (T_relative):
  • Moves the cow slightly to the right (x = 1) and closer (z = 1), making it appear larger.

Result:

• The cow appears larger due to its closer distance to the camera.

Rendered Image:

---

Summary of Transformations

Image	Rotation (R_{rel})	Translation (T_{rel})	Effect
1. Side View (-90° Z-axis)	[[0,1,0], [-1,0,0], [0,0,1]]	[0,0,4]	Rotates cow 90° to the side
2. Smaller Cow (Moves Closer)	Identity matrix	[0,0,2]	Cow moves closer (appears smaller)
3. Side View (90° Y-axis)	[[cos(90),0,sin(90)], [0,1,0], [-sin(90),0,cos(90)]]	[-3,0,4]	Turns cow left
4. Larger Cow (Moves Forward)	Identity matrix	[1,0,1]	Cow moves forward (appears larger)

---

🌐 5. Rendering Generic 3D Representations

5.1 Rendering Point Clouds from RGB-D Images (10 points)

We converted RGB-D images into 3D point clouds and visualized them.

🎥 Point Cloud Renders

---

5.2 Parametric Functions (10 + 5 points)

We generated a torus point cloud using a parametric function.

🎥 Torus & Custom Parametric Shape

Torus	Möbius Strip

Torus Equation: [ x = (R + r \cos\phi) \cos\theta, \quad y = (R + r \cos\phi) \sin\theta, \quad z = r \sin\phi ]

---

5.3 Implicit Surfaces (15 + 5 points)

Instead of sampling points, we defined 3D surfaces using implicit equations and extracted them using Marching Cubes.

🎥 Torus Mesh

🎥 Gyroid

Mesh vs. Point Cloud Comparison:

Feature	Mesh	Point Cloud
Rendering Speed	Fast (Rasterization)	Slow (Many points)
Quality	Smooth surfaces	Dotted look
File Size	Compact	Large
Good For	Rendering & Simulation	Neural Networks & Sampling

---

6. Creative Rendering Challenge (10 points)

I created a fun 3D scene combining multiple objects into a single animated composition.

🎥 Final Fun Scene

How It Works:

• Objects: A sphere, Möbius strip, and torus.
• Scene Composition: Objects arranged at different positions.
• Rotation: 360-degree orbiting camera.

---

Conclusions

This project demonstrated various 3D representations:

1. Point Clouds are great for raw depth data but have lower rendering quality.
2. Parametric Surfaces give precise mathematical control over shapes.
3. Implicit Meshes enable smooth surfaces but require marching cubes.
4. Creative 3D composition brings everything together.

Future Work

• Load real-world 3D models (OBJ, STL).
• Implement Neural Implicit Representations (NeRF/SDFs).
• Improve texturing and materials.

---

References

• PyTorch3D Documentation
• Marching Cubes Algorithm
• Common 3D Datasets (CO3D)

---

Contact

• 📧 Email: vhsingh@andrew.cmu.edu
• 🔗 GitHub
• 🌐 Personal Website: yourwebsite.com

---

Thanks for Reading!

Hope you enjoyed this 3D visualization project!