16-825 · Learning for 3D Vision
The cow mesh is rendered from 36 viewpoints spaced 10 degrees apart in azimuth. For each frame,
look_at_view_transform returns the rotation and translation for a camera placed 3 units from the
origin and pointed at it. The point light is placed at the camera centre for every frame, so the surface facing
the viewer stays lit through the full rotation.
The field of view increases from 5 to 120 degrees across the frames. To hold the cow at a constant size while
the field of view changes, the camera distance for each frame is computed from the field of view itself. Half the
subject, half the field of view and the distance form a right triangle, which gives
d = (w / 2) / tan(fov / 2) where w is the width of the subject. The result is that the cow occupies the same part of the frame throughout while the plane behind it stretches.
The tetrahedron is built from 4 vertices and 4 triangular faces. Each face lists its three vertex indices in counter clockwise order as seen from outside the solid, so every face normal points away from the centroid. A turntable at a fixed elevation cannot show all four faces, because any face whose normal is parallel to the orbit axis will stay on the edge for the whole rotation. The elevation therefore follows a sine curve of the azimuth with an amplitude of 45 degrees, which brings every face into a direct view at some point and returns to its starting value so that the loop closes.
The cube is built from 8 vertices and 12 triangular faces, two triangles for each of the six square sides. The corner coordinates are at plus or minus 0.5 on each axis, which puts the centre of the cube at the origin and lets the camera aim at the origin directly.
The colour of each vertex is set by linear interpolation along the z axis of the cow, which runs from the head
to the tail. With z_min and z_max taken from the vertex coordinates, the blend factor is
alpha = (z - z_min) / (z_max - z_min) and each vertex's colour is
alpha * color2 + (1 - alpha) * color1.
I chose color1 = [0, 0, 1], a pure blue, at z_min, and
color2 = [1, 0, 0], a pure red, at z_max. The two are at opposite ends of the hue range,
so the gradient stays easy to read across the body and passes through purple in the middle of the cow.
Each view is produced by a relative transform applied to the original extrinsics, with
R = R_relative @ R_0 and T = R_relative @ T_0 + T_relative. One point is worth noting
about the second view. Because the translation is composed as R_relative @ T_0, a rotation on its own
also moves the camera, and for a 90 degree rotation about the y axis it sends the cow to a position beside the
camera at zero depth, where it falls outside the field of view. The translation term is what
places the camera back in front of the subject.
[-3, 0, 3]. The rotation turns the camera to face the side of the
cow and the translation returns it to a position 3 units in front of the subject.
[0, 0, 2]. The camera moves 2 units further along its viewing direction, so the cow keeps its
orientation and only becomes smaller in the frame.
[0.5, -0.5, 0]. The camera is displaced within its own image plane without any rotation. The
positive x axis of the PyTorch3D screen convention points to the left, so the cow moves left and down.Each RGB-D image is converted to a point cloud with unproject_depth_image, which casts a ray
through every masked pixel and places a point at the distance given by the depth map. Pixels that lie inside the
mask but carry a depth of zero unproject to non finite coordinates, so these are removed before the cloud is
built. This accounts for 62,386 of the 125,035 masked pixels in the first image and 21,562 of the 123,749 in the
second, which leaves 62,649 and 102,187 points respectively and 164,836 in the union.
The difference in how many points survive is visible in the result. The first cloud is noticeably sparser and
breaks up along the leaves, the second is denser, and the union covers most of the gaps left by either view on its
own.
A torus is sampled from its parametric form. For an angle phi around the main ring and an angle
theta around the tube, the surface is
((R + r cos(theta)) cos(phi), (R + r cos(theta)) sin(phi), r sin(theta)) with a major radius
R = 1 and a minor radius r = 0.4. Sampling both angles on a 200 by 200 grid gives 40,000
points, and each point is coloured by its normalised position. The camera holds an elevation of 30 degrees so that
the hole stays visible through the rotation.
The second surface is a Mobius strip, sampled as
((1 + v cos(u / 2)) cos(u), (1 + v cos(u / 2)) sin(u), v sin(u / 2)) for u over a full
turn and v across the width of the band. The half angle u / 2 is what gives the strip its
single twist, so the band returns to its starting point with the opposite face outward.
The same torus is represented as the zero level set of
(sqrt(x^2 + y^2) - R)^2 + z^2 - r^2 with R = 1 and r = 0.35. The function is
evaluated on a 64 by 64 by 64 voxel grid spanning -1.5 to 1.5 on each axis, and the surface is extracted with
marching cubes. The vertex coordinates come out indexed by grid position, so they are rescaled back to the
original range before the mesh is built.
The second surface is the heart given by
(x^2 + (9/4) y^2 + z^2 - 1)^3 - x^2 z^3 - (9/80) y^2 z^3, evaluated on the same grid. The equation is
written with its axis of symmetry along z, while the renderer treats y as the up direction, so the roles of the two
axes are exchanged when the voxel grid is filled. Without that change the surface is viewed end on and reads as a
flat blob rather than a heart.
The two representations differ most clearly in rendering cost/time at the same quality. A single 256 pixel view of the 164,836 point plant cloud took about 11 seconds on the CPU, and the 36 frame gif of the three clouds required 108 such renders. The torus mesh, which covers its surface with 17,088 triangles, rendered its full gif in a small fraction of that time. A point cloud has no connectivity, so the cost/time scales with the number of points, and a surface only looks solid once enough points are present to close the gaps between them.
The quality is also better with a mesh. Its triangles join up, so the renderer knows which way each part of the surface is facing and can light it correctly, and the outline of the shape comes out clean. A point cloud is only a set of dots with nothing joining them, so gaps show through when there are too few points or when the surface is seen at a shallow angle, and the size of each dot has to be set by hand to suit how closely the points are spaced. The plant shows this clearly. The first view has fewer points and breaks up along the leaves, while the union of the two views looks much more solid.
Memory goes the other way. A mesh has to store a list of triangles on top of its points, but it needs far fewer points to describe the same smooth surface. The torus mesh here has 8,544 points and 17,088 triangles, while the torus point cloud needs 40,000 points and still looks grainier. Point clouds are easier to obtain. The plant clouds came straight from the depth images with no extra work, whereas each implicit surface needed a voxel grid, a choice of grid size and a marching cubes step. That grid size also limits how much fine detail the mesh can hold, so a shape with sharp features would need a finer grid and the memory saving would shrink.