16-825 · Learning for 3D Vision

Assignment 1: Rendering Basics with PyTorch3D

Question 1.1

360 degree renders

The cow mesh is rendered from 36 viewpoints spaced 10 degrees apart in azimuth. For each frame, look_at_view_transform returns the rotation and translation for a camera placed 3 units from the origin and pointed at it. The point light is placed at the camera centre for every frame, so the surface facing the viewer stays lit through the full rotation.

gif of the cow mesh
Cow mesh 360 degree gif

Question 1.2

Re-creating the dolly zoom

The field of view increases from 5 to 120 degrees across the frames. To hold the cow at a constant size while the field of view changes, the camera distance for each frame is computed from the field of view itself. Half the subject, half the field of view and the distance form a right triangle, which gives d = (w / 2) / tan(fov / 2) where w is the width of the subject. The result is that the cow occupies the same part of the frame throughout while the plane behind it stretches.

Dolly zoom effect on the cow, field of view from 5 to 120 degrees
Dolly zoom

Question 2.1

Constructing a tetrahedron

The tetrahedron is built from 4 vertices and 4 triangular faces. Each face lists its three vertex indices in counter clockwise order as seen from outside the solid, so every face normal points away from the centroid. A turntable at a fixed elevation cannot show all four faces, because any face whose normal is parallel to the orbit axis will stay on the edge for the whole rotation. The elevation therefore follows a sine curve of the azimuth with an amplitude of 45 degrees, which brings every face into a direct view at some point and returns to its starting value so that the loop closes.

gif of a tetrahedron
Tetrahedron

Question 2.2

Constructing a cube

The cube is built from 8 vertices and 12 triangular faces, two triangles for each of the six square sides. The corner coordinates are at plus or minus 0.5 on each axis, which puts the centre of the cube at the origin and lets the camera aim at the origin directly.

gif of a cube
Cube

Question 3

Re-texturing a mesh

The colour of each vertex is set by linear interpolation along the z axis of the cow, which runs from the head to the tail. With z_min and z_max taken from the vertex coordinates, the blend factor is alpha = (z - z_min) / (z_max - z_min) and each vertex's colour is alpha * color2 + (1 - alpha) * color1.

I chose color1 = [0, 0, 1], a pure blue, at z_min, and color2 = [1, 0, 0], a pure red, at z_max. The two are at opposite ends of the hue range, so the gradient stays easy to read across the body and passes through purple in the middle of the cow.

Cow mesh coloured with a blue to red gradient along its z axis
Retextured cow

Question 4

Camera transformations

Each view is produced by a relative transform applied to the original extrinsics, with R = R_relative @ R_0 and T = R_relative @ T_0 + T_relative. One point is worth noting about the second view. Because the translation is composed as R_relative @ T_0, a rotation on its own also moves the camera, and for a 90 degree rotation about the y axis it sends the cow to a position beside the camera at zero depth, where it falls outside the field of view. The translation term is what places the camera back in front of the subject.

Cow rotated by 90 degrees in the image plane
R_relative is a rotation of 90 degrees about the z axis of the camera, and T_relative is zero. The camera does not move, it only rolls about its own viewing direction, so the image content is rotated by 90 degrees.
Cow seen in profile from its left side
R_relative is a rotation of 90 degrees about the y axis and T_relative is [-3, 0, 3]. The rotation turns the camera to face the side of the cow and the translation returns it to a position 3 units in front of the subject.
Cow rendered smaller, with the camera moved back
R_relative is the identity and T_relative is [0, 0, 2]. The camera moves 2 units further along its viewing direction, so the cow keeps its orientation and only becomes smaller in the frame.
Cow shifted within the image plane
R_relative is the identity and T_relative is [0.5, -0.5, 0]. The camera is displaced within its own image plane without any rotation. The positive x axis of the PyTorch3D screen convention points to the left, so the cow moves left and down.

Question 5.1

Rendering point clouds from RGB-D images

Each RGB-D image is converted to a point cloud with unproject_depth_image, which casts a ray through every masked pixel and places a point at the distance given by the depth map. Pixels that lie inside the mask but carry a depth of zero unproject to non finite coordinates, so these are removed before the cloud is built. This accounts for 62,386 of the 125,035 masked pixels in the first image and 21,562 of the 123,749 in the second, which leaves 62,649 and 102,187 points respectively and 164,836 in the union. The difference in how many points survive is visible in the result. The first cloud is noticeably sparser and breaks up along the leaves, the second is denser, and the union covers most of the gaps left by either view on its own.

Three point clouds of a plant side by side, from the first image, the second image and their union
Left: first image. Centre: second image. Right: union of the two.

Question 5.2

Parametric functions

A torus is sampled from its parametric form. For an angle phi around the main ring and an angle theta around the tube, the surface is ((R + r cos(theta)) cos(phi), (R + r cos(theta)) sin(phi), r sin(theta)) with a major radius R = 1 and a minor radius r = 0.4. Sampling both angles on a 200 by 200 grid gives 40,000 points, and each point is coloured by its normalised position. The camera holds an elevation of 30 degrees so that the hole stays visible through the rotation.

gif of a torus point cloud
Torus point cloud sampled from its parametric function.

Second object

The second surface is a Mobius strip, sampled as ((1 + v cos(u / 2)) cos(u), (1 + v cos(u / 2)) sin(u), v sin(u / 2)) for u over a full turn and v across the width of the band. The half angle u / 2 is what gives the strip its single twist, so the band returns to its starting point with the opposite face outward.

gif of a Mobius strip point cloud
Mobius strip point cloud.

Question 5.3

Implicit surfaces

The same torus is represented as the zero level set of (sqrt(x^2 + y^2) - R)^2 + z^2 - r^2 with R = 1 and r = 0.35. The function is evaluated on a 64 by 64 by 64 voxel grid spanning -1.5 to 1.5 on each axis, and the surface is extracted with marching cubes. The vertex coordinates come out indexed by grid position, so they are rescaled back to the original range before the mesh is built.

gif of a torus mesh from an implicit function
Torus mesh recovered from an implicit function by marching cubes.

Second object

The second surface is the heart given by (x^2 + (9/4) y^2 + z^2 - 1)^3 - x^2 z^3 - (9/80) y^2 z^3, evaluated on the same grid. The equation is written with its axis of symmetry along z, while the renderer treats y as the up direction, so the roles of the two axes are exchanged when the voxel grid is filled. Without that change the surface is viewed end on and reads as a flat blob rather than a heart.

gif of a heart shaped mesh from an implicit function
Heart mesh recovered from an implicit function by marching cubes.

Meshes compared with point clouds

The two representations differ most clearly in rendering cost/time at the same quality. A single 256 pixel view of the 164,836 point plant cloud took about 11 seconds on the CPU, and the 36 frame gif of the three clouds required 108 such renders. The torus mesh, which covers its surface with 17,088 triangles, rendered its full gif in a small fraction of that time. A point cloud has no connectivity, so the cost/time scales with the number of points, and a surface only looks solid once enough points are present to close the gaps between them.

The quality is also better with a mesh. Its triangles join up, so the renderer knows which way each part of the surface is facing and can light it correctly, and the outline of the shape comes out clean. A point cloud is only a set of dots with nothing joining them, so gaps show through when there are too few points or when the surface is seen at a shallow angle, and the size of each dot has to be set by hand to suit how closely the points are spaced. The plant shows this clearly. The first view has fewer points and breaks up along the leaves, while the union of the two views looks much more solid.

Memory goes the other way. A mesh has to store a list of triangles on top of its points, but it needs far fewer points to describe the same smooth surface. The torus mesh here has 8,544 points and 17,088 triangles, while the torus point cloud needs 40,000 points and still looks grainier. Point clouds are easier to obtain. The plant clouds came straight from the depth images with no extra work, whereas each implicit surface needed a voxel grid, a choice of grid size and a marching cubes step. That grid size also limits how much fine detail the mesh can hold, so a shape with sharp features would need a finer grid and the memory saving would shrink.