16-825 Assignment 1: Rendering Basics with PyTorch3D

Name: Ruinian (Ray) Xue

AndrewID: ruinianx

Goals: In this assignment, you will learn the basics of rendering with PyTorch3D, explore 3D representations, and practice constructing simple geometry.

You may also find it helpful to follow the Pytorch3D tutorials.

Table of Contents

  1. Practicing with Cameras (15 Points)

  2. Practicing with Meshes (10 Points)

  3. Re-texturing a mesh (10 Points)

  4. Camera Transformations (10 Points)

  5. Rendering Generic 3D Representations (45 Points)

  6. Do Something Fun (+10 Extra Credit)

1. Practicing with Cameras (15 Points)

1.1. 360-degree Renders (5 points)

Your first task is to create a 360-degree gif video that shows many continuous views of the provided cow mesh. For many of your results this semester, you will be expected to show full turntable views of your outputs. You may find the following helpful:

dolly_360

1.2 Re-creating the Dolly Zoom (10 points)

The Dolly Zoom is a famous camera effect, first used in the Alfred Hitchcock film Vertigo. The core idea is to change the focal length of the camera while moving the camera in a way such that the subject is the same size in the frame, producing a rather unsettling effect.

 

dolly_zoom

2. Practicing with Meshes (10 Points)

2.1 Constructing a Tetrahedron (5 points)

In this part, you will practice working with the geometry of 3D meshes. Construct a tetrahedron mesh and then render it from multiple viewpoints. Your tetrahedron does not need to be a regular tetrahedron (i.e. not all faces need to be equilateral triangles) as long as it is obvious from the renderings that the shape is a tetrahedron.

Number of Vertices: 4

Number of Faces: 4

tetrahedron_360

2.2 Constructing a Cube (5 points)

Construct a cube mesh and then render it from multiple viewpoints. Remember that we are still working with triangle meshes, so you will need to use two sets of triangle faces to represent one face of the cube.

Number of Vertices: 8

Number of Faces: 12

 

cube_360

3. Re-texturing a mesh (10 points)

Now let's practice re-texturing a mesh. For this task, we will be retexturing the cow mesh such that the color smoothly changes from the front of the cow to the back of the cow.

Choice of color1 and color2: use normal color picker for hex code of colors, I notice in Microsoft Excel we can easily choose color (is quicker than opening Adobe Photoshop) and get both hex code and RGB number. Then we can divide it into 255.

bicolor_cow_360

4. Camera Transformations (10 points)

When working with 3D, finding a reasonable camera pose is often the first step to producing a useful visualization, and an important first step toward debugging.

Running python -m starter.camera_transforms produces the following image using the camera extrinsics rotation R_0 and translation T_0:

Cow render

Overall explanation (base idea)

First, evaluate the difference between the formula in class and the formula in pytorch3d.

We have Xc~=R(Xw−C~) and t=−RC~ in the slides, represting the transformation from world to camera, and the C~ means the camera position in world coordinates. But points in the PyTorch3D pytorch3d.renderer.FoVPerspectiveCameras are right-multiplied by R, so we need to reconstruct Rrelative using the formula R−1=Rrelative, RT=R−1.

In the code, we have R=Rrelative@R0 and T=Rrelative@T0+Trelative, and in the following 4 cases, R0=I.

Given the fact that Rrelative=RT, following cases will use this conclusion directly.

What's more, the T0 in the code, is the result of tinitial=−RinitialC~initial, coming from the initial position of the camera in the world coordination. The T in the code, is the result of tafter=−RafterC~after, so thr Trelative shows the movement from C~initial to C~after.

Howerver, we cannot simply equal Trelative to any tinitial or tafter above, due to the multiply and add action in the code, Trelative isn't simply t's. (I also noticed this as a question in Piazza.)

Transform 1: Rotate Camera Counterclockwise

Cow render

Based on the overall explanation above, in this case, we only rotate the camera counterclockwise, not affecting the camera position in the world coordinate system.

So the Trelative keeps default, which is an all zero vector.

For the rotation, we changed from (x,y,z) to (−y,x,z), so the rotation matrix R should be

(1)[0−10100001]

But the R is not Rrelative, we need transpose it.

Thus, Rrelative is:

(2)[010−100001]

In this case, Trelative is [0,0,0].

Transform 2 : Move Camera Further Back

Cow render

Based on the overall explanation above, in this case, we only move the camera further away, not rotating the camera coordinate system.

So the Rrelative keeps default, which is an identity matrix.

Because the Rrelative is an identity matrix, we can simply calculate Trelative in this case, as same as the representing the movement of the world coordinate system origin.

As we move it further back along the -z axis, with the tinitial as [0,0,3], tafter as anything further from the z-axis, I chose it to be [0,0,5], so Trelative=tafter−tinitial=[0,0,2]

So the parameter to fill in this case is:

Rrelative=I, Trelative[0,0,2]

Transform 3: Shift Camera to the right

Cow render

Similar to transform 3, we don't need to rotate the camera and the Rrelative keeps identity matrix, and we move the camera slightly to the right, the Trelative=t=[0.5,0,0], 0.5 is a number I approximate according to the example picture.

So the parameter to fill in this case is:

Rrelative=I, Trelative[0.5,0,0]

Transform 4:Move Camera to the Right Side

Cow render

This is a somewhat more complicated case, and due to the complexity of the formula in the code, based on the right-multiplation in PyTorch3D, I try to prove and make conclusion for the calculation of Trelative, although we can simply change the code and make it easier, or just calculate Trelative using the known T=[0,0,3] after transformation to calculate back Trelative, I was really curious about the geometric meaning of Trelative in this case so made this informal proof.

Conclusions on the camera and world based on lecture:

  1. C=−R−1T=−R⊤T=−RrelativeT, C is the position of camera origin in the world system.

  2. T=−RC, T is the position of world origin in the camera system.

  3. T=RT0, based on the definition of T0 in the code.

  4. Based on T=RrelativeT0+Trelative,

  5. based on 1, we know Cafter=−RrelativeT.

  6. After the rotation, without moving the camera, we cannot capture any photo of the cow, because the z-axis of the camera is not facing the cow, so we need to move the camera. Define the movement of camera after rotation, based on camera coordinate system, Mcamera, and the movement of camera based on world coordinate system, Mworld, please notice both of them are representing movement of camera, we assume world not move. We can simply know Mcamera=RMworld.

  7. Assuming world not move so we can get Mworld=Cafter−Cinitial, based on geometry, we can get the movement as Mworld=Cafter−Cinitial=[3,0,0]−[0,0,−3]=[3,0,3]. By the way, Tafter−Tinitial=[0,0,3]−[0,0,3]=0, so T keeps unchanged.

  8. Digging it further, we can find it out that geometrically, Trelative not equal this movement.

  9. Mcamera=RMworld=[−3,0,3], it's actually the Trelative in this case but only in this case.

  10. Trelative=T−RrelativeT0=−RrelativeCafter−RrelativeT0

  11. T0=−R0C0=−C0 (because R0=I, based on the definition in the code)

  12. based on 10&11, Trelative=−RrelativeMworld

  13. Trelative=−RTMworld

And this is the geometrical meaning of Trelative? Maybe, actually initially I tried many answers because the formula in the code looks different from the lecture, after filling in the Mcamera=[−3,0,3] in the parameter for T_relative, I got it right. But I really wonder what this Trelative means in geography, if there's no geographical meaning, I just have to use T - R_relative @T_0, that's correct, but I really want to figure out why it equals Mcamera and it seems to be a coincidence:(.

In summary,

(3)Rrelative=[001010−100]

And, Trelative=[−3,0,3]

5. Rendering Generic 3D Representations (45 Points)

The simplest possible 3D representation is simply a collection of 3D points, each possibly associated with a color feature. PyTorch3D provides functionality for rendering point clouds.

5.1 Rendering Point Clouds from RGB-D Images (10 points)

In this part, we will practice rendering point clouds constructed from 2 RGB-D images from the Common Objects in 3D Dataset.

In render_generic.py, the load_rgbd_data function will load the data for 2 images of the same plant. The dictionary should contain the RGB image, a depth map, a mask, and a Pytorch3D camera corresponding to the pose that the image was taken from.

You should use the unproject_depth_image function in utils.py to convert a depth image into a point cloud (parameterized as a set of 3D coordinates and corresponding color values). The unproject_depth_image function uses the camera intrinsics and extrinisics to cast a ray from every pixel in the image into world coordinates space. The ray's final distance is the depth value at that pixel, and the color of each point can be determined from the corresponding image pixel.

Construct 3 different point clouds side by side:

plant_1plant_2plant_3

5.2 Parametric Functions (10 + 5 points)

A parametric function generates a 3D point for each point in the source domain. For example, given an elevation theta and azimuth phi, we can parameterize the surface of a unit sphere as (sin(theta) * cos(phi), cos(theta), sin(theta) * sin(phi)).

By sampling values of theta and phi, we can generate a sphere point cloud.

Render a torus point cloud

Your task is to render a torus point cloud by sampling its parametric function.

According to Wikipedia: θ,φ∈[0,2π)]

(4)x(θ,φ)=(R+rsin⁡θ)cos⁡φy(θ,φ)=(R+rsin⁡θ)sin⁡φz(θ,φ)=rcos⁡θ

torus

A new object of my chioce

I chose to draw a heart using the parameter equation online.

Equation credit from: Mathmetics-Parametric equation for Taubin heart surface

heart

After using the parameter equation, I found out the heart rotates facing strangely, so I swapped the points in z-axis and y-axis:

5.3 Implicit Surfaces (15 + 5 points)

In this part, we will explore representing geometry as a function in the form of an implicit function. In general, given a function F(x, y, z), we can define the surface to be the zero level-set of F i.e. (x,y,z) such that F(x, y, z) = 0. The function F can be a mathematical equation or even a neural network. To visualize such a representation, we can discretize the 3D space and evaluate the implicit function, storing the values in a voxel grid. Finally, to recover the mesh, we can run the marching cubes algorithm to extract the 0-level set.

Torus mesh

 

Torus mesh

Tradeoff between mesh and point cloud

  1. Rendering speed: In our case, the plants has really less points, so the rendering speed is really fast. But if the points get more, in the parameter funtion vs implicit surface case, for the same object, the point clouds take more time to render.

  2. Rendering quality: Mesh has better overall quality, especially when using implicit surfaces for sampling. Comparing parameter function using point cloud and implicit surfaces using mesh, meshes are smoother and have less noise. Point clouds has really explicit points especially when there are less points, and they are not really affected by lights.

  3. Ease of use: Point clouds are easier to use, in fewerm lines of codes can we render them. For meshes, we need to indicate vertices and faces.

  4. Memory usage: Representing point cloud needs more memory, but rendering complicated meshes needs a big amount of memory. Especially in my following case for extra point, the Loomis Model I downloaded from internet has much more vertices and faces and it takes time and memory to load and render. So we cannot simply say which one exactly uses more, it depends on the quality.

  5. Summary: Meshes has better quality but sometimes are limited in shape and implement (more difficult to implement), point clouds has more flexibility but not really interactive to lights and look not smooth, and if points get more, it costs to compute and render.

Render the heart again

According to the reference link above, the heart implicit function is:

(5)(x2+9y24+z2−1)3−x2z3−9y2z380=0

Thus we can put it in the code, and the result is listed below:

heart-mesh


(Extra Credit) 6. Do Something Fun (+10 points)

Now that you have learned to work with various 3D represenations and render them, it is time to try something fun. Create your own 3D structures, or render something in an interesting way, or creatively texture, or anything else that appeals to you - the (3D) world is your oyster! If you wish to download additional meshes, Free3D is a good place to start.

During the past summer, I learned Loomis Model in manually drawing human head, when drawing a human head in mangas, we need to know about the strcuture of the head, and Loomis Model is a really interesting and simple sample to do this.

loomis

So I downloaded a .glb document from sketchfab and converted it into meshes using trimesh, after that, I tried to make it shining, rotating with different light, which is funny, looks like meme. So I changed the light and angle within every frame, and make the color from pink to blue (the same in the bi-color cow).