16-825 · Learning for 3D Vision

Assignment 1: Rendering Basics with PyTorch3D

1. Practicing with Cameras 15 points

1.1 360-degree Renders 5 pointsdone

360-degree turntable render of the cow mesh
The cow is rendered from 36 azimuth angles at 10° intervals, with a camera distance of 3, an elevation of 0°, and a 60° field of view. Code: part1_1.py

1.2 Re-creating the Dolly Zoom 10 pointsdone

Dolly zoom on the cow-on-plane mesh
The field of view increases from 5° to 120°, while the camera distance follows 3 / tan(fov/2) to keep the cow at a constant size in the frame. Code: starter/dolly_zoom.py

2. Practicing with Meshes 10 points

2.1 Constructing a Tetrahedron 5 pointsdone

360-degree render of a hand-built tetrahedron
A tetrahedron rendered from 36 azimuth angles.

Vertices and faces

The tetrahedron contains 4 vertices and 4 triangular faces. Each face corresponds to one of the C(4,3) = 4 possible combinations of 3 vertices.

Code: part2.py

2.2 Constructing a Cube 5 pointsdone

360-degree render of a hand-built cube
A unit cube spanning [−1, 1]³, rendered from 36 azimuth angles at an elevation of 20°.

Vertices and faces

The cube contains 8 vertices and 12 triangular faces. The vertices are located at the 8 corners (±1, ±1, ±1). Each of the 6 square sides is divided into 2 triangles.

Code: part2.py

3. Re-texturing a Mesh 10 pointsdone

Cow re-textured with a front-to-back color gradient
A per-vertex blue-to-red gradient along the z-axis, rendered from 36 azimuth angles.

Color choice

I selected color1 = [0, 0, 1] (blue at z_min) and color2 = [1, 0, 0] (red at z_max).

Since blue and red don't share an RGB channel, interpolation gives each alpha value a unique color, smoothly transitioning through magenta. This gradient really helps us see depth. Colors that share a channel, like green and yellow, would blend together less clearly.

Code: part3.py

4. Camera Transformations 10 pointsdone

The baseline is R_0 = I and T_0 = (0, 0, 3). Since these are world-to-camera extrinsics, R_relative rotates the camera frame, and a positive T_relative shifts the scene in camera coordinates.

Cow rotated 90 degrees in the image plane
1. In-plane roll. R_relative = [[0,1,0],[−1,0,0],[0,0,1]], T_relative = 0. This rotates the image 90° about the camera's optical axis without changing the viewpoint, so the cow appears to lie on its side.
Cow seen from farther away
2. Move back. R_relative = I, T_relative = [0, 0, 2]. This moves the cow 2 units along +z, equivalent to moving the camera back. The cow therefore appears smaller.
Cow shifted within the frame
3. In-plane translation. R_relative = I, T_relative = [0.5, −0.5, 0]. This shifts the cow parallel to the image plane without changing its depth, size, or orientation.
Cow seen from its left side
4. Orbit 90°. R_relative = [[0,0,1],[0,1,0],[−1,0,0]], T_relative = [−3, 0, 3]. A 90° rotation about the up axis directs the camera along the world −x axis. Because this rotation moves T_0 to (3,0,0), the translation re-centers the cow and produces a side view.

Code: part4.py

5. Rendering Generic 3D Representations 45 points

5.1 Rendering Point Clouds from RGB-D Images 10 pointsdone

Point cloud unprojected from the first plant image
Image 1
Point cloud unprojected from the second plant image
Image 2
Union of both plant point clouds
Union of both

Notes

Both clouds are unprojected directly into world coordinates, so they can be combined with torch.cat without registration.

Two implementation details require special handling. First, the stored cameras follow a y-down convention, as indicated by R[1,1] ≈ −1 in the saved extrinsics. The turntable therefore uses up=((0, -1, 0),) to keep the plant upright. Second, pixels without valid depth values unproject to NaN, which prevents the renderer from computing a valid bounding box. These non-finite points are removed before rendering.

Code: part5.py, --render plant_rgbd

5.2 Parametric Functions 10 + 5 pointsdone

Torus point cloud sampled from its parametric function
Torus with R = 1.0, r = 0.3, 400×400 samples
Mobius strip point cloud
Möbius strip (new object), 200×200 samples

Parameterizations

The torus is parameterized as x = (R + r·cosθ)cosφ, y = (R + r·cosθ)sinφ, and z = r·sinθ, sampled over (φ, θ) ∈ [0, 2π]².

The second object is a Möbius strip, parameterized as x = (1 + v·cos(u/2))cos u, y = (1 + v·cos(u/2))sin u, and z = v·sin(u/2). The u/2 term introduces a half-twist, creating a single-sided surface with a topology distinct from that of the torus.

Code: part5.py

5.3 Implicit Surfaces 15 + 5 pointsdone

Torus mesh extracted from an implicit function
Torus mesh from a 64³ grid plus marching cubes
Gyroid minimal surface mesh
Gyroid (new object), 96³ grid

Implicit functions

The torus is defined as the zero level set of F = (sqrt(x² + y²) − R)² + z² − r². The function is evaluated on a 64³ grid, and the surface is extracted using mcubes.marching_cubes. Because the resulting vertices are expressed in voxel-index space, they must be rescaled to world coordinates.

The second object is a gyroid defined by sin x·cos y + sin y·cos z + sin z·cos x = 0. It is a triply periodic minimal surface without a simple parametric form, making an implicit representation well suited to this geometry.

Code: part5.py

Mesh vs. point cloud tradeoffs

Rendering quality. Mesh faces create a seamless surface, which means you get smooth shading and accurate occlusion. Point clouds, on the other hand, reveal gaps when you get close, and points from the background can peek through at slight angles. The quality of the mesh still hinges on how many voxels are used: at 64³, the torus looks a bit faceted.

Rendering speed. At comparable quality, the mesh is faster here: roughly 10k triangles versus 160,000 points for a solid-looking torus. Point clouds are easy to subsample, while meshes require decimation.

Memory. Point clouds use less memory per primitive because they store only position and color, without connectivity. However, the implicit-surface pipeline requires a dense intermediate grid. The 96³ gyroid grid contains approximately 885,000 values before marching cubes begins, and this cost grows cubically with resolution.

Ease of use. Point Cloud is easiest to acquire and process initially, as raw sensor data can be used directly without managing complex surfaces or topology. Mesh takes more effort to build up front, but is much easier to use in downstream applications like rendering, 3D editing, and physics simulations thanks to its defined surfaces and structural normals.

6. Do Something Fun extra credit · +10done

Two metaballs merging and separating as an implicit surface
A 48-frame metaball morph. Each frame is generated by applying marching cubes to an 80³ grid while the camera simultaneously orbits the surface.

What this is

Two metaballs orbit toward each other and then apart again. The surface is the zero level-set of a time-varying implicit function,

F(x, t) = Σᵢ r² / ||x − cᵢ(t)||² − 1

which I re-evaluate on the voxel grid and re-mesh at every frame. What makes this interesting is that the surface changes topology partway through. When the balls are far apart it has two connected components, and once they get close enough those merge into a single one. A mesh with fixed connectivity cannot represent this, and neither can a parametric function, since both have their topology fixed from the start. An implicit function gives it to you for free, which is exactly the advantage I described in 5.3.

The merge threshold is also predictable. At the midpoint between the two centers the field reduces to 2r²/s² − 1, so the lobes first touch when s = r√2. With r = 0.78 that works out to 1.1031. To confirm this, I counted the connected components of the extracted mesh across the animation: every sampled frame below that separation produced one component and every frame above it produced two, with no mismatches.

The balls also counter-rotate in the xy-plane, and the separation follows a cosine, so the loop joins up seamlessly and the merge stays visible from more than one viewing angle.

Code: part6.py

7. Sampling Points on Meshes extra credit · +10done

10 sampled points beside the cow mesh
10 points (left) next to the original mesh (right)
100 sampled points beside the cow mesh
100 points
1000 sampled points beside the cow mesh
1000 points
10000 sampled points beside the cow mesh
10000 points