Assignment 1: Rendering Basics with PyTorch3D

16-825: Learning for 3D Vision · Kashu Yamazaki

1. Practicing with Cameras

1.1 — 360-degree renders

A turntable sweeps the azimuth through a full turn with look_at_view_transform, building one camera per frame and rasterizing them a batch at a time. Every other turntable on this page reuses the same helper.

The cow mesh rotating through a full 360-degree turn.
36 frames, one per 10° of azimuth.

1.2 — Dolly zoom

The frustum is 2 · distance · tan(fov / 2) wide at the subject, so keeping the subject the same size while the field of view opens up means holding that width fixed: distance = width / (2 tan(fov / 2)). The sweep runs from 5° to 120°.

The scene itself is only 3×3 with its near edge at z = −1.5, so the constant cannot simply be the width of the plane — anything under about 5 walks the camera inside the geometry at the wide end of the sweep. I used 5.

Dolly zoom: the cow stays the same size while the background stretches and the field of view widens.
Field of view 5° → 120°, with the distance tracking it.

2. Practicing with Meshes

2.1 — Tetrahedron

An equilateral base of radius 1 with the apex over its centre. Putting the base plane at −h/4 and the apex at 3h/4 with h = √2 makes the tetrahedron regular and centres it on the origin. Each face is listed counter-clockwise as seen from outside, so the normals point outward — the signed volume comes out positive, which is the cheap way to check the winding.

A tetrahedron rotating through a full turn.
Regular tetrahedron, viewed from 20° elevation.

2.2 — Cube

I listed the six square faces as quads wound counter-clockwise from outside, then split each one as (a, b, c) and (a, c, d). Doing it that way means the two triangles inherit the quad's winding, so the outward normals come for free rather than having to be checked face by face.

A cube rotating through a full turn.
Unit cube spanning ±1 on each axis.

3. Re-texturing a Mesh

color1 = [0, 0, 1] (blue) at the nose, the vertex with the smallest z, blending linearly to color2 = [1, 0, 0] (red) at the tail, the largest z. Blue to red reads clearly against the light background and the direction of the gradient stays obvious from any viewpoint, which a pair of neighbouring hues would not manage.

The cow mesh with a blue-to-red gradient running from nose to tail, rotating through a full turn.
Per-vertex colour interpolated along z.

4. Camera Transformations

PyTorch3D maps the world into the camera as x_cam = x_world @ R + T, and the camera looks down +z with +x to the left and +y up. Two consequences drive every answer below: R_relative is the transpose of the rotation the picture appears to undergo, and T_relative is added directly in camera coordinates, so a positive x offset pushes the cow left rather than right.

The cow lying on its side, rotated a quarter turn clockwise in the image plane.
transform1 — roll 90° about the view axis.
R = [[0,1,0], [−1,0,0], [0,0,1]]
T = [0, 0, 0]
The cow seen from its side, with the blue z-axis pointing right.
transform2 — orbit a quarter turn about world y.
R = [[0,0,1], [0,1,0], [−1,0,0]]
T = [−3, 0, 3]
The cow in the same pose as the reference but smaller, seen from further away.
transform3 — slide straight back to 5 units out.
R = I
T = [0, 0, 2]
The cow in the same pose and size as the reference but shifted left and down.
transform4 — slide sideways, same size and pose.
R = I
T = [0.5, −0.5, 0]

The two rotations are the interesting ones. transform1 only rolls the camera about the direction it is already looking, so nothing moves in depth and T_relative stays zero; the red world-x axis goes from pointing left to pointing up, which is the whole change. transform2 swings the camera round to the cow's side, but R_relative alone would rotate the old translation with it and leave the camera at [−3, 0, 0] looking past the origin — so T_relative = [−3, 0, 3] is what re-seats it 3 units from the cow.

5. Rendering Generic 3D Representations

5.1 — Point clouds from RGB-D images

Each view is unprojected with unproject_depth_image and the union is just the two clouds concatenated — they already share a world frame, so no registration is needed. Two things had to be handled. A large fraction of the masked pixels carry a depth of exactly 0, and unprojecting those divides by zero and sprays inf and nan through the cloud, so the mask has to be intersected with depth > 0 (that alone drops view 1 from 125k points to 63k valid ones). And the capture's world frame is upside down relative to look_at_view_transform, so x and y get flipped to stand the plant back up.

Three point clouds of a potted plant side by side, rotating together: the first view, the second view, and their union.
Left: view 1. Middle: view 2. Right: the union. Cameras 6 units out.

5.2 — Parametric surfaces

The torus is a tube of radius r swept around a circle of radius R in the xz plane, so the hole faces up the y axis and an elevated camera looks straight through it. For a second object I kept the same sweep but turned the cross-section half a turn over the loop, which gives a Möbius strip — the surface has one side, and you can follow the twist round in the animation.

A torus point cloud rotating, with the hole clearly visible.
Torus, 200×200 samples of (θ, φ).
A Moebius strip point cloud rotating, showing its half twist.
Möbius strip, same sampling density.

5.3 — Implicit surfaces

The same torus as the zero level set of (√(x² + z²) − R)² + y² − r², sampled onto a voxel grid and run through marching cubes. For the second object I used a gyroid: g = sin x cos y + sin y cos z + sin z cos x is a minimal surface dividing space into two interlocking labyrinths. |g| < t gives that sheet a wall thickness, and taking the max with a cube field trims it to a block, since a max of two implicit functions is the intersection of the two solids. It is a one-liner implicitly and a nightmare to model by hand, which is rather the point.

A torus mesh rotating, with the hole clearly visible.
Torus mesh, 64³ grid. Euler characteristic 0, so genus 1.
A block of gyroid lattice rotating, showing its interlocking channels.
Gyroid block, 128³ grid.

Mesh vs. point cloud

The mesh wins on quality. It is a real surface, so it occludes properly, shades with normals, and holds up under a close camera. A point cloud at the same budget shows gaps at the silhouettes and has to be tuned by eye through the splat radius — too small and the surface leaks, too large and fine detail turns to blobs.

The mesh is also far more compact. 9.8k vertices for the torus mesh against 40k points for a comparable point cloud, and the gap grows with area: points have to cover a surface densely, whereas triangles only have to span it. A flat region costs one quad and an unbounded number of points.

The point cloud wins on everything upstream of rendering. It comes straight out of a sensor or a parametric formula, it has no topology to get wrong, it rasterizes as independent splats so it stays fast under heavy subsampling, and it represents open, noisy, partial geometry — the plant in 5.1 — that a mesh could not express at all. The implicit route also costs a voxel grid on top: 128³ samples and a marching-cubes pass to recover a torus that the parametric form produces directly.

6. Something Fun

A carousel of eight cows, one hue each, riding around a ring and bobbing on their own phase. The camera never moves — instead every frame of the animation is its own mesh in a single batch, which is both the point of the effect and the cheapest way to render the whole loop.

Eight rainbow-coloured cows riding around a ring, bobbing up and down.
Eight cows, joined into one mesh per frame.

7. Sampling Points on Meshes

Face areas come from half the norm of the cross product, torch.multinomial picks faces in proportion to those areas, and each sample gets a uniform barycentric point inside its face: draw (u, v) from the unit square and fold the u + v > 1 half back onto the other one, which is a uniform draw over the triangle.

At 10 points the cow is unrecognisable, at 100 you can just about place the legs, at 1000 the silhouette is there, and at 10000 it is hard to tell from the mesh at this resolution.

The cow mesh beside point clouds sampled at 10, 100, 1000 and 10000 points, all rotating together.
Left to right: the original mesh, then 10, 100, 1000 and 10000 sampled points.