Rendering Basics with PyTorch3D

16-825 Learning for 3D Vision · Assignment 1

Student
Joshua Jacob
Andrew ID
JJACOB2

1. Practicing with cameras

1.1. Cow turntable

Cow mesh rotating through a full turn.
The camera moves through 360° at a fixed distance of 4 and elevation of 20°.

1.2. Dolly zoom

Textured cow on a checkerboard during a dolly zoom.
The field of view changes from 20° to 100° and back while the camera moves toward and away from the cow.

y′i = yi / [(zi + d) tan(fov / 2)]
max(y′) − min(y′) = constant

For each field of view, binary search finds the camera distance that keeps the cow's projected height at 112.86 pixels, matching the 60° view. This accounts for the different depths of its vertices. The height varies by less than 0.001 pixels across the camera sweep; the cow and ground still change perspective.

2. Practicing with meshes

Lavender tetrahedron turntable.
2.1. Tetrahedron: 4 vertices and 4 triangular faces.
Green cube turntable.
2.2. Cube: 8 vertices and 12 triangular faces, with two triangles per square side.

Both meshes use manually listed vertices and faces, with triangle vertices ordered so the normals point outward.

3. Re-texturing a mesh

Cow with a blue-to-orange gradient from front to back.
Blue at the front, orange at the back.

color1 = (0.1, 0.3, 0.9) and color2 = (1.0, 0.45, 0.1).

α = (z − zmin) / (zmax − zmin)
color = (1 − α) color1 + α color2

Vertex colors depend on z; the renderer interpolates them across each triangle.

4. Camera transformations

The base camera has R0 = I and T0 = (0, 0, 3). The transformed camera uses R = Rrelative R0 and T = Rrelative T0 + Trelative.

Rotate

Cow rotated clockwise by 90 degrees in the image.
R_relative = [[ 0, 1, 0],
              [-1, 0, 0],
              [ 0, 0, 1]]
T_relative = [0, 0, 0]

This rolls the view so the cow appears rotated 90° clockwise. Its head and the green y-axis point right.

Zoom out

Cow smaller in the frame after moving the camera farther away.
R_relative = I
T_relative = [0, 0, 2]

The translation becomes (0, 0, 5). Increasing the distance makes the cow smaller without changing the field of view.

Shift

Cow shifted toward the lower-left corner.
R_relative = I
T_relative = [0.5, -0.5, 0]

Positive view-space x points left in PyTorch3D, and negative y points down. The cow moves left and down.

Side view

Cow viewed from the side, facing left.
R_relative = [[ 0, 0, 1],
              [ 0, 1, 0],
              [-1, 0, 0]]
T_relative = [-3, 0, 3]

The rotation turns the view by 90° around y. RrelativeT0 is (3, 0, 0), so the added translation restores T = (0, 0, 3) and keeps the cow centered.

5. Rendering generic 3D representations

5.1. Point clouds from RGB-D images

Synchronized turntables of the first plant point cloud, second point cloud, and their union.
Left: first RGB-D image. Middle: second image. Right: union in the shared world coordinate system.

Unprojecting each masked depth image with its supplied camera gives 125,035 and 123,749 points. These CPU renders use every eighth point: 15,630 and 15,469 points, with 31,099 in the union. Use --point_stride 1 to render all points.

All three clouds use the same camera orbit, with −y pointing up to keep the plant upright. The union fills in parts of the leaves missing from either view alone. Gaps around the leaves and pot, along with stray points outside the plant, remain from imperfect depth estimates and masks.

Individual GIFs: view 1, view 2, union.

5.2. Parametric functions

Colored torus point cloud with its central hole visible.
Torus with major radius 1 and tube radius 0.35.
Colored ellipsoid point cloud.
Ellipsoid with axis radii 0.8, 1.2, and 0.65.

Torus: (x, y, z) = ((1 + 0.35 cos v) cos u, 0.35 sin v, (1 + 0.35 cos v) sin u)
Ellipsoid: (x, y, z) = (0.8 sin v cos u, 1.2 cos v, 0.65 sin v sin u)

Both surfaces use 100 × 100 parameter samples, colored by normalized xyz coordinates. For the torus, both angles cover a full turn. For the ellipsoid, u covers a full turn and v runs from 0 to π. The torus's points form visible rows, with wider gaps around the outside of the ring. Uniform parameter spacing does not give uniform surface-area density.

5.3. Implicit surfaces

Torus mesh extracted from an implicit function.
Torus zero level set.
Ellipsoid mesh extracted from an implicit function.
Ellipsoid zero level set.

Torus: F(x, y, z) = (√(x² + z²) − 1)² + y² − 0.35²
Ellipsoid: F(x, y, z) = (x / 0.8)² + (y / 1.2)² + (z / 0.65)² − 1

Marching cubes extracts F = 0 from a 48³ grid over [−1.6, 1.6]³. The returned voxel indices are converted to world coordinates using the grid spacing, 3.2 / 47.

Mesh versus point cloud

The torus mesh has a continuous, shaded surface. In the point cloud, gaps between rows depend on the point spacing and radius. Point clouds are easy to build from depth images because no faces need to be connected. Meshes need triangle indices and, for an implicit surface, an extraction step.

Point clouds store positions and colors. Meshes also store triangle indices, and marching cubes temporarily needs the voxel grid. Total memory depends on the number of points, vertices, faces, and voxels; neither representation is always smaller.

Measured CPU rendering time for 60 frames at 256 × 256, excluding GIF encoding and geometry construction.
ObjectPoint cloudMesh
Torus44.7 s29.8 s
Ellipsoid44.4 s21.4 s

Point counts and mesh resolutions differ, so these timings apply to the examples shown. Exact counts and times are in timings.json.

6. Do something fun — snowman

A snowman with three snowballs, a black top hat, buttons, and an orange carrot nose.
Snowman turntable.

I made this by using three scaled spheres for the body. Smaller spheres make the eyes and buttons, two cylinders make the hat, and a cone makes the nose. The parts have separate vertex colors and are joined into one mesh for rendering.

7. Sampling points on meshes

Cow surface sampled at 10, 100, 1000, and 10000 points, followed by the original cow mesh.
10, 100, 1,000, and 10,000 samples, followed by the original mesh. All use the same camera. Open full-size comparison.

Triangles are chosen with probability proportional to their area. Given independent u, v drawn uniformly from [0, 1), the barycentric weights are:

(w₀, w₁, w₂) = (1 − √u, √u (1 − v), √u v)
p = w₀ a + w₁ b + w₂ c

The square root avoids concentrating samples near one vertex. Vertex colors are interpolated with the same weights. Ten points barely suggest the shape. At 1,000 points, the cow's outline is recognizable; at 10,000, most of the visible surface is filled in.

Reproduce the results

python -m starter.assignment --section all
python test_geometry.py

All renders use PyTorch3D on the CPU, with four PyTorch threads and gradient tracking disabled. Animations contain 60 frames and loop at an average of 15 fps. The sampling seed is 7.

Setup and commands · Environment · Runner · Geometry code