Assignment 0: Getting started with webpage submission

Learning for 3D Vision · Graham Felton (gtf)

Task 1. ChatGPT

For this task, play around with ChatGPT. Be creative! Now, mention the prompt and the answer given by ChatGPT in your webpage. You can even mention an interesting conversation.

Some of the prompt examples include:

  1. Write a song/poem/story/joke.
  2. Some puzzle or riddle.
Answer

Prompt

Frame an open problem in the 3D computer vision literature as a LeetCode style problem.

Response

Absolutely. A good way to do this is to take a genuinely unresolved 3D vision problem and strip away the research-specific machinery until it looks like a coding-interview problem—while keeping the underlying difficulty intact.

Open-problem-inspired LeetCode problem: Dynamic Point Cloud Consistency

Problem

You are given a sequence of 3D point clouds captured by a moving camera:

P[0], P[1], ..., P[T-1]

Each point cloud contains points in the camera's coordinate system: (x, y, z). The scene contains both static objects (walls, buildings, roads) and moving objects (cars, pedestrians, cyclists).

Between consecutive frames, the camera undergoes an unknown rigid transformation consisting of a 3D rotation R and a 3D translation t, so that a static point p should satisfy approximately:

p_next ≈ R p + t

However, you are not told which points are static. Your task is to recover the camera motion between every pair of frames.

Input

points[t][i] = [x, y, z]

where points[t] is the point cloud at time t. You may assume:

Return, for every consecutive pair of frames, the rigid transformation (R[t], t[t]) that best explains the motion of the static portion of the scene.

Example

Suppose frame 0 contains [(0,0,5), (1,0,5), (0,1,5), (10,0,8), ...] and frame 1 contains the same static wall after the camera has translated by (1, 0, 0), while a car has independently moved by (5, 0, 0). Your algorithm should recover the camera translation (-1, 0, 0) rather than being fooled by the moving car.

Constraints

1 <= T <= 100
1000 <= number of points per frame <= 10^6
-1000 <= x, y, z <= 1000

Target complexity: O(N log N) per frame pair, with N the number of points.

Follow-up

Now remove the assumption that 60% of the scene is static. Can you still guarantee recovery of the camera motion?

Why this is actually an open research problem

This superficially resembles ICP / point-cloud registration, but the important difficulty is that the correspondence and the static-vs-dynamic segmentation are both unknown. Real autonomous-driving and robotics systems face exactly this problem: estimate ego-motion from a point cloud when an unknown subset of the scene is moving. Existing approaches use combinations of geometric registration, learned features, scene-flow estimation, object detection, temporal consistency, and robust optimization—but there isn't a universally reliable solution under realistic conditions.

The particularly nasty cases are:

So the LeetCode-looking core is: given two unordered noisy sets of 3D points, recover the rigid transformation explaining the largest geometrically consistent subset, without knowing which points belong to that subset. That is a much more interesting research problem than ordinary ICP.

Even harder version

A particularly clean research-grade formulation is: given two unlabeled point sets P, Q ⊂ ℝ³, find the rigid transformation (R, t) maximizing

| { p ∈ P : ∃ q ∈ Q, ||R p + t - q|| < ε } |

when an unknown fraction of points are outliers.

Question: Can you design an algorithm that is both computationally efficient and provably robust when the inlier fraction is unknown?

That formulation connects directly to robust 3D registration, correspondence-free pose estimation, dynamic-scene understanding, and SLAM—and unlike a standard coding problem, there is no known algorithm that solves all realistic instances optimally.

Task 2. 3D shape images

To get you used to uploading images on the webpage, search for a image with an interesting 3D shape. Please mention the source as well.

Answer
Four 3D renderings of a multilayer perceptron, with each layer drawn as a
    stacked plane and activations drawn as red and blue points connected across layers, shown at
    increasing training iterations.
This is an illustration that I really love for how a source data distribution is transformed as it passes through layers in an MLP.
Source

Task 3. Funny GIFs

For this task, search around the web for the funniest GIF. Please mention the source as well.

Answer
A white cat that is invisible against a white background until the
    background fades to dark, revealing the cat.

Source