{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><h1>Introduction to the Cloud and Apache Spark</h1></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Overview</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul>\n",
    "    <li>Introduction to the Cloud</li>\n",
    "    <li>Context</li>\n",
    "<li>Basics of Apache Spark</li>\n",
    "    <li>Run Apache Spark on the Cloud</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Why would you go with the Cloud?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l2/cloud_characteristics.png\" height=\"250px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>What is Cloud in the REAL World?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul>\n",
    "<li><b>“Cloud”</b> refers to large Internet services running on 10,000s of machines (Amazon, Google, Microsoft, etc) </li>\n",
    "\n",
    "<li><b>“Cloud computing”</b> refers to services by these companies that let external customers rent cycles and storage\n",
    "\n",
    "<ul>\n",
    "    <li>Amazon EC2: virtual machines at 8.5¢/hour</li>\n",
    "    <li>Amazon S3: storage at 21¢/GB/month </li>\n",
    "    <li>Google Cloud AppEngine</li>\n",
    "    <li>Windows Azure</li>\n",
    "    </ul>\n",
    "</li>\n",
    "    </ul>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Redeem your Google Cloud Coupon</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul><li>Redeem your coupon by copying and pasting the coupon URL that can be found on Canvas into your browser and type your @andrew.cmu.edu email.<br/></li>\n",
    "<li>After typing your @andrew email, you will be emailed a URL to use your Google Cloud coupon. Click on the link and switch the email to your @gmail.com email. In other words, relogin to your GMAIL account and make sure that the account shown on your profile page when redeeming is your @gmail.com (Check the screenshot next page)</li> \n",
    "<li>Remember that you get only one coupon. Make sure your google account on the <b>top right</b> corner of the screen shows your GMAIL account.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<ul><li>As a student, you get a 50 dollars cloud credit coupon. This coupon can be refilled by the instructor.</li>\n",
    "<li>You <b>MUST</b> disable your billing account when you finish using your cloud account. Failure to do so will lead to your account running out of balance.</li>\n",
    "<li>Do not redeem the free trial on your Google Cloud account. If you redeemed Google cloud credit earlier, use different GMAIL email. If you are not sure, use different email. Your coupon will be lost (and unredeemable) if it was redeemed in an account that used free trial credits before.</li></ul>\n",
    "\n",
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l4/Coupon_Redemption_using_Gmail.png\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<ul><li>If it is your first time on this cloud account, you will be asked to create a project and associate the billing account with your project. The billing account used by this grant is named \"Billing Account for Education\"</li>\n",
    "<li>Remember, each time you finish the use of your cloud, you need to disable the billing on your account. In order to do so, navigate to the <b>Billing</b> section from left navigation menu and view your projects. You will see three vertical dots next to your project, from which you will be able to disable the billing on your account (Check the image below). You may enable the billing back using the same approach.<br/>\n",
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l4/disable_billing.png\"/>\n",
    "</li></ul>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Context</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Last lecture, we talked about the data science process. Now, let's switch gears and discuss our technical infrastructure that can be deployed later to the cloud. You will need Spark installed on your machine in order to run the code snippets in this lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this class, we will use <b>Apache Spark</b> for data ingestion, wrangling and EDA. But why Spark?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Why Spark?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul>\n",
    "<li>Developed in 2009 at UC Berkeley AMPLab, then open sourced in 2010, </li>\n",
    "<li>Spark is the next revolution of the popular Hadoop MapReduce framework.</li>\n",
    "\n",
    "<li>Gartner, Advanced Analytics and Data Science (2014) \"Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model. It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.\"</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Pandas vs PySpark (Spark developed in Python)</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In a research done by Databricks, an industrial leader in the domain of big data storage and processing, PySpark shows superior performance compared to traditional implementations.\n",
    "<center><figure><img src=\"http://stat.cmu.edu/~mfarag/14810/l6/pandas_vs_pyspark.png\"/></figure></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>What is Spark?</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul>\n",
    "<li>Apache Spark is a fast and general-purpose cluster computing system for large scale data processing.</li>\n",
    "<li>Spark was originally written in Scala, which allows concise function syntax and interactive use.</li>\n",
    "    <li>Apache Spark provides High-level APIs in Java, Scala, Python (PySpark)and R.</li>    \n",
    "    <li>Apache Spark combines two different modes of processing:<ul>\n",
    "        <li><b>Batch-based Processing</b> which can be provided via Apache Hadoop MapReduce</li>\n",
    "        <li><b>Real-time Processing</b> which can be provided via Apache Storm.</li>\n",
    "        </ul>\n",
    "        </li>\n",
    "    </ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><figure><img src=\"http://stat.cmu.edu/~mfarag/14810/l3/batch_vs_realtime.png\"/></figure></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Spark Ecosytem</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><figure><img src=\"http://stat.cmu.edu/~mfarag/14810/l3/spark_ecosystem.png\"/></figure></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Spark Componenets</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><figure><img src=\"http://stat.cmu.edu/~mfarag/14810/l3/spark_components.png\"/></figure></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3>Spark Core</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Spark Core is the general execution engine for the Spark platform that other functionalities are built on top of it. \n",
    "Spark has several advantages: \n",
    "<ul>\n",
    "<li>Speed: runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk</li>\n",
    "    <li>Ease of Use: Write applications quickly in Java, Scala, Python, R</li>\n",
    "<li>Generality: Combine SQL, streaming, and complex analytics</li>\n",
    "<li>Runs Everywhere: Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: findspark in c:\\users\\mfarag\\appdata\\roaming\\python\\python39\\site-packages (2.0.1)\n"
     ]
    }
   ],
   "source": [
    "# if you installed Spark on windows, \n",
    "# you may need findspark and need to initialize it prior to being able to use pyspark\n",
    "\n",
    "!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Uncomment the following lines if you are using Windows!\n",
    "#import findspark\n",
    "#findspark.init()\n",
    "#findspark.find()\n",
    "\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "\n",
    "\n",
    "spark = SparkSession.builder.master(\"local[*]\").appName('SparkTest').getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Download Data Files Remotely</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: wget in c:\\users\\mfarag\\appdata\\roaming\\python\\python39\\site-packages (3.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Saved under KDDTrain+ (3).txt\n"
     ]
    }
   ],
   "source": [
    "# Downloading and preprocessing KDD Train Data \n",
    "!python -m wget https://www.andrew.cmu.edu/user/mfarag/14763/KDDTrain+.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Cloud Consideration</h2>\n",
    "Your data need to be moved to a special storage server if you are running Spark on the Cloud. This special storage is called HDFS. The following command is used to move your data from your local storage to the special storage server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Uncomment and Execute this line if you are running your notebook on the Cloud\n",
    "#!hadoop fs -put KDDTrain+.txt /"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Spark Dataframes</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inspired by pandas DataFrames in structure, format, and a few specific operations, Spark DataFrames are like distributed in-memory tables with named columns and schemas, where each column has a specific data type: integer, string, array, map, real, date, timestamp, etc. To a human’s eye, a Spark DataFrame is like a table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When data are visualized as a structured table, it’s not only easy to digest but also easy to work with when it comes to common operations you might want to execute on rows and columns. <br/>Also, DataFrames are immutable and Spark keeps a lineage of all transformations. You can add or change the names and data types of the columns, creating new DataFrames while the previous versions are preserved. A named column in a DataFrame and its associated Spark data type can be declared in the schema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's examine the generic and structured data types available in Spark before we use them to define a schema. Then we'll illustrate how to create a DataFrame with a schema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2><a href=\"https://spark.apache.org/docs/latest/sql-ref-datatypes.html\">Spark’s Basic Data Types</a></h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Spark supports basic internal data types. These data types can be declared in your Spark application or defined in your schema"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><figure><img src=\"http://stat.cmu.edu/~mfarag/14810/l5/spark_data_types.png\"/></figure></center>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Load data from csv to a dataframe on a local machine. \n",
    "# header=False means the first row is not a header \n",
    "# sep=',' means the column are seperated using ','\n",
    "df = spark.read.csv('KDDTrain+.txt', header=False, sep=\",\")\n",
    "# on the Cloud, the files will have to be at the root level. So, the cloud version is:\n",
    "#df = spark.read.csv('/KDDTrain+.txt', header=False, sep=\",\")\n",
    "\n",
    "df.show(5, vertical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Avoiding Auto-assigned Column Names: Read CSV and Specify the Column Names</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "col_names = [\"duration\",\"protocol_type\",\"service\",\"flag\",\"src_bytes\",\n",
    "    \"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\n",
    "    \"logged_in\",\"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\n",
    "    \"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
    "    \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\n",
    "    \"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
    "    \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\n",
    "    \"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\"dst_host_same_src_port_rate\",\n",
    "    \"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\n",
    "    \"dst_host_rerror_rate\",\"dst_host_srv_rerror_rate\",\"classes\",\"difficulty_level\"]\n",
    "\n",
    "df = spark.read.csv(\"KDDTrain+.txt\",header=False, inferSchema= True).toDF(*col_names)\n",
    "\n",
    "# on the Cloud, the files will have to be at the root level. So, the cloud version is:\n",
    "# df = spark.read.csv(\"/KDDTrain+.txt\",header=False, inferSchema= True).toDF(*col_names)\n",
    "\n",
    "df.show(5, vertical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>More ways to Display Dataframes</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "1.   `df.take(5)` will return a list of five Row objects. \n",
    "2.   `df.collect()` will get all of the data from the entire DataFrame. Be really careful when using it, because if you have a large data set, you can easily crash the driver node. \n",
    "3.   `df.show()` is the most commonly used method to view a dataframe. There are a few parameters we can pass to this method, like the number of rows and truncaiton. For example, `df.show(5, False)` or ` df.show(5, truncate=False)` will show the entire data wihtout any truncation.\n",
    "4.   `df.limit(5)` will **return a new DataFrame** by taking the first n rows. As spark is distributed in nature, there is no guarantee that `df.limit()` will give you the same results each time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Schemas and Creating DataFrames</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "You can think about the dataframes as a table. A schema in Spark defines the column names and associated data types for a DataFrame. Most often, schemas come into play when you are reading structured data from an external data source. Defining a schema up front as opposed to taking a schema-on-read approach offers three benefits:\n",
    "\n",
    "<ul>\n",
    "<li>You relieve Spark from the onus of inferring data types.</li>\n",
    "<li>You prevent Spark from creating a separate job just to read a large portion of your file to ascertain the schema, which for a large data file can be expensive and time-consuming.</li>\n",
    "\n",
    "<li>You can detect errors early if data doesn’t match the schema.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You may also create your own schema using data field name followed by data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "schema = \"id INT, firstName STRING, WEBSITE STRING\"\n",
    "data = [[1, \"John\", \"https://tinyurl.1\"],\n",
    "       [2, \"Brooke\", \"https://tinyurl.2\"]]\n",
    "\n",
    "test_df = spark.createDataFrame(data, schema)\n",
    "test_df.show()\n",
    "test_df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Display Schema Information for Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.printSchema() # or df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Print Column Names in Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Print Total Number of Your Records in Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Print Sample Record from Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.show(1, vertical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>DataFrame Operations on Columns</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Selecting Columns & Creating Subset Dataframes\n",
    "2. Adding New Columns\n",
    "3. Renaming Columns\n",
    "4. Removing Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Create a Subset Dataframe from Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "small_df = df.select(\"duration\",\"protocol_type\",\"service\",\"classes\",\"difficulty_level\")\n",
    "small_df.show(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Display Summary Statistics in Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.describe().show(vertical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Display Unique Values from a Column in Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.select(\"classes\").distinct().show(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Add a Column to Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# We will add a new column called 'first_column' at the end\n",
    "from pyspark.sql.functions import lit\n",
    "df = df.withColumn('first_column',lit(1)) \n",
    "# lit means literal. It populates the row with the literal value given.\n",
    "# When adding static data / constant values, it is a good practice to use it.\n",
    "df.show(1,vertical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Renaming a Column in Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df = df.withColumnRenamed('first_column', 'new_column_one')\n",
    "\n",
    "df.printSchema()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Delete a Column from Your Dataframe</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df = df.drop('new_column_one')\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Lab: Run Spark on the Cloud</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Create Clusters (e.g. Hadoop Clusters)</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<li>A cluster is group of machines, servers, or nodes. It helps providing the sum of the computational power offered by all incorporated machines. It's difficult to build a local machine with 64GB RAM and 20TB of Storage but that is not difficult when you are running on the cloud.</li> \n",
    "<li>You may start by navigating to Dataproc and click on the Clusters section \n",
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l4/create_your_cluster.png\"/>\n",
    "</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>Cluster - Setup</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<li>Next, you need to create your cluster and choose the cluster to use \"Google Compute Engine\"</li>\n",
    "\n",
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l4/create_cluster.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3>Cluster Configuration</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Make sure to conduct the following:\n",
    "1. Enable component gateway\n",
    "2. Select to add `Docker` and `Jupyter Notebooks` from the list of available components\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3>Running Cluster</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Once you click on the create button, Google Cloud will work on creating your own cluster and if it's successful, you will see your cluster running.\n",
    "<img src=\"http://stat.cmu.edu/~mfarag/652/lectures/l4/running_cluster.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3>Connect to Your Cluster</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "From the Web Interfaces, open the Jupyter Notebook. Upload your Notebook to `Local Disk/home` folder and run the cells."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}