Overview

The purpose of this assignment is just to get you set up with the tools of the trade. You are asked to install them on your computer and use them for a couple of simple tasks.

No Partners

As the only goal of this assignment is to "Turn a very small wheel once", you are asked to do it by yourself. You'll have trouble contributing as an equal partner in the subsequent parts of this assignment, unless you can do something small-ish by yourself, to be sure.

Required Software

Please install the following software on your own computer, or confirm that it works for you in the WeH 5419CD cluster:

• Java, if you need it (You know how to do this from Intro courses)
• Hadoop-0.17.* (See the tutorial for a link)
• Eclipse

Links

• Quickstart Guide
• Hadoop Map/Reduce Tutorial
• Eclipse

Setup Instructions -- Commandline From the links above, follow the quickstart guide where it is talking about Standalone mode. Standalone mode runs everything in a single Java process. There is another mode called psuedo-distributed which you can ignore. Psuedo-distributed mode is where it runs all the different parts of a full cluster all on your single computer. Read the Hadoop Map/Reduce Tutorial and run few of the example bundled with the hadoop distribution to get a feel for how to run jobs and to make sure it works.

Setup Instructions -- Eclipse Now, you should try to get eclipse working with the hadoop plugin. First download and install eclipse. Then you have to copy the hadoop plugin from contrib/eclipse-plugin/ in the hadoop archive to the eclipse plugins directory. When you start eclipse now, the plugin should be detected and you will have to go to the settings and give the plugin the path to where you extracted hadoop. Then try loading and running some more example programs so that you can see how the plugin works.

Setup Instructions -- Cluster If working in Weh5419CD, Eclipse and the PlugIn are already installed, but you'll need to unip and configure hadoop. You'll probably want to put this into your "MyAFS" space, rather than on the Desktop, as the Desktops are local, per-machine and you might not get the same seat every time you go to work. You will still have to do the step where you tell the plugin where hadoop has been extracted to.

While you are downloading and unzipping things, also get your one choice of the following:

• Web traces from NASA
• Web Traces from the EPA

Your Tasks

• Carefully follow the Hadoop tutorial
• Get yourself set up to use Hadoop in Standalone mode via the Eclipse Plug-In
• Get the canonical "Word Count" example working. Turn in not only the source code, but the Output when run upon the Declaration of Independence of the United States.
• Write your own Hadoop/Map-Reduce Program that processes your choice of Web traces and determines the most frequently accessed object (Much like Word Count, eh?)

We're Here To Help!

As always -- remember, we're here to help!