Darkroom Implementation of the CMU Smart Headlight

Vinay Palakkode

My final project for the course 15-869: Visual Computing Systems

The CMU Smart Headlight is a reactive visual system developed at Carnegie Mellon university under the leadership of prof. Srinivasa Narasimhan of Robotics Institute. I intend to implement a similar high speed image processing/ analysis pipeline using darkroom which is a language for describing hardware image processing pipelines embedded in Terra.

Checkpoint 1

As per the proposed schedule, I have completed the tasks plan for the first two weeks with the exception of adding, dilation on the binarized image. Meanwhile I got the Darkroom up and running on the Tegra K1 development board which wasn’t a task listed in the checklist for week1 and week2.

  • Week 1: Utilized this week for the ramp up. I utilized this time to understand the operations supported and not supported by darkroom. Also digged into a big of metaprogramming in Lua to understand how to parameterize convolutions. I will need this mainly for the dilation kernel.
  • Week 2: Implemented (file based i/o) image binarization, background subtraction and thresholding with 2 input images. Need to add image dilation.

Test Dataset

I have generated a range of input images with a number of illuminated circles with different intensity and sizes to simulate varying illumination as well as varied number of particles. And I will make a couple of uncompressed video files by drawing particles of random sizes (offcourse bounded with in a range).

Deliverables

As planned, I'll deliver the darkroom implementation of the CMU smartheadlight pipeline for tracking and dislluminating/illuminating particles. The demo test case will be pingball ball tracking and illumination. I will try my level best to demo using live inputs from the GS3 Near Infrared Camera as long as there are no logistical issues that cannot be resolved before December 1st. Also I intend to report the end to end latency figures on Tegra K1, because this pipeline has never been ported onto an ARM based platform.

Checkpoint 2

The current status of the project is that, I'm facing issues in integrating the darkroom generated code to the application written in C/C++. The darkroom kernels, and pipeline as a fused kernel works well as a standalone entity with the aid of lua jit compiler. I am aware of the fact that terra lets me compile stuff into objects which can be linked to my application written in C/C++. I've sought help from James. So I'm setting a checkpoint 2.5 to achieve this by Friday. I will be able to profile my code only if I surpass this stage. And this is equally important for me to easily generate the disassembly to analyze how well darkroom leverages NEON and SSE4/AVX capabilities of the host CPU.

  • Week 3: I spent a couple of days trying to implement the bit packing stage (used as a hack in the current implementation to save bandwidth ) using darkroom. And I figured out that since dark room is more of a functional programming language, it does not let me pack binary bytes as bits. I will have to detach this stage from the darkroom pipeline.
  • Week 4: I wrote a baseline for ARM. Found that arm-linux-gcc was n't doing a good job in auto-vectorization. I spent a good amount of time in understanding the auto vectorization scheme followed by gcc on ARM and X86. I also ran through the schemes used by llvm (mainly because the darkroom ARM/ X86 SIMD optimizations are as good/bad as the IR code generated by llvm). I implemented dilation in two different ways. one based on the sum of the pixels in a window and the other one using the map reduce operator provided by darkroom.

Schedule

  • Nov 01 - Nov 07: Ramp up time to understand the Darkroom language and trying out toy algorithms on static images to get a hang of the language.
  • Nov 08 - Nov 14: Implement background subtraction and thresholding using 2 static images. Once I find success, I will extend the pipeline by adding image dilation
  • Nov 15 - Nov 21: Run the pipeline using a video file as the input. Once I succeed in that, I will replace the file based input with High Speed NIR Camera input stream.
  • Nov 22 - Nov 28: Assuming ARM support will be available by then, I will port the existing system to Jetson-Tk1 platform.
  • Nov 29 - Dec 12: Bug fixing, performance analysis and final report preparation.