Darkroom Implementation of the CMU Smart Headlight

Vinay Palakkode

My final project for the course 15-869: Visual Computing Systems

The CMU Smart Headlight is a reactive visual system developed at Carnegie Mellon university under the leadership of prof. Srinivasa Narasimhan of Robotics Institute. I intend to implement a similar high speed image processing/ analysis pipeline using darkroom which is a language for describing hardware image processing pipelines embedded in Terra.

Summary

I will implement an image processing pipeline consisting of the High Speed Image Capture, Background Subtraction, thresholding, dilation and display using darkroom and analyse the effectiveness of the langauage. A quick demo of this would be illuminating a ping pong ball with a dark background using this system. (Which is popularly tagged as one of the ping pong ball experiments in the context of the smart headlight demos.)

Background

The CMU programmable headlight has a number of features which includes anti-glare, rain and snow disillumination. The concept of selective illumination and dis-illumination of particles of significant dimensions, is a challenging problem in computational photography. And it is often tedious to try out different types of workload scheduling by manually trying out all the permutations and combinations of the same. And it is even more difficult to port the system from one platform to another, or when there is a newer hardware revision available. This project is intended to analyze the capabilities of the darkroom to handle all these challenges.

Challenges

The most challenging aspect of the project is the use of Darkroom language itself. As this is a research language and also since it is very young, the probability of hitting road blocks due to untested corner cases in the compiler code are high. Also at this point of time it is not clear about the fact that how well the stencil operations are gonna map to the hardware and how efficient is it going to be when compare against handcrafted workload scheduling. Another challenge is the open endedness of the algorithm for selective illumination/ dis-illumination.

Resources

There is only one open implementation. The concept is explained in Programmable Automotive Headlights . The key resource for this project is the support offered by James Hagerty, the developer of darkroom.

Goals

Through this project ideally, I must be able to deliver a single codebase for the starter version of the smart headlight written using darkroom which can generate binaries that directly map on to x86, ARM and FPGAs.

Schedule

  • Nov 01 - Nov 07: Ramp up time to understand the Darkroom language and trying out toy algorithms on static images to get a hang of the language.
  • Nov 08 - Nov 14: Implement background subtraction and thresholding using 2 static images. Once I find success, I will extend the pipeline by adding image dilation
  • Nov 15 - Nov 21: Run the pipeline using a video file as the input. Once I succeed in that, I will replace the file based input with High Speed NIR Camera input stream.
  • Nov 22 - Nov 28: Assuming ARM support will be available by then, I will port the existing system to Jetson-Tk1 platform.
  • Nov 29 - Dec 12: Bug fixing, performance analysis and final report preparation.