---
title: 'Lecture 4: <br> Basic cleaning, loops, and alternatives'
author: "Prof. Alexandra Chouldechova"
date: "Fall 2020"
output:
  ioslides_presentation:
    highlight: tango
    widescreen: true
    smaller: true
---

## Agenda

- Lists
- A common data cleaning task
- Factor variables, and when they're useful
- Functions
- If-else statements
- For/while loops to iterate over data
- R coding style

- Rather than picking up where Lecture 3 left off I've woven the Lecture 3 content we haven't yet covered into the Lecture 4 notes

## Package loading
```{r, message=FALSE, warning=FALSE}
library(tidyverse)
Cars93 <- MASS::Cars93  # For Cars93 data again
```

## Basics of lists

> A list is a **data structure** that can be used to store **different kinds** of data

- Recall: a vector is a data structure for storing *similar kinds of data*

- To better understand the difference, consider the following example.

```{r}
my.vector.1 <- c("Michael", 165, TRUE) # (name, weight, is.male)
my.vector.1 
typeof(my.vector.1)  # All the elements are now character strings!
```

## Lists vs. vectors

```{r}
my.vector.2 <- c(FALSE, TRUE, 27) # (is.male, is.citizen, age)
my.vector.2
typeof(my.vector.2)
```

- Vectors expect elements to be all of the same type (e.g., `Boolean`, `numeric`, `character`)

- When data of different types are put into a vector, the R converts everything to a common type

## Lists

- To store data of different types in the same object, we use lists

- Simple way to construct lists: use **`list()`** function

- (We'll learn about functions like `map` and `map_chr` soon.)

```{r}
my.list <- list("Michael", 165, TRUE)
my.list
map_chr(my.list, typeof)
```

## Named elements
```{r}
patient.1 <- list(name="Michael", weight=165, is.male=TRUE)
patient.1
```

## Referencing elements of a list (similar to data frames)
```{r}
patient.1$name # Get "name" element (returns a string)
patient.1[["name"]] # Get "name" element (returns a string)
patient.1["name"] # Get "name" slice (returns a list)
c(typeof(patient.1$name), typeof(patient.1["name"]))
```

## A common problem

- One of the most common problems you'll encounter when importing manually-entered data is inconsistent data types within columns

- For a simple example, let's look at `TVhours` column in a messy version of the survey data from Lecture 2

```{r}
survey.messy <- read.csv("http://www.andrew.cmu.edu/user/achoulde/94842/data/survey_data2020_messy.csv", 
                         header=TRUE, stringsAsFactors = FALSE)
# Print out first 20 elements
head(survey.messy$TVhours, 20) 
```

- **NOTE**: If you've installed R within the past few months, your version will automatically default to `stringsAsFactors = FALSE`.  My version of R is older and still has the old `stringsAsFactors = TRUE` default, a convention that dates back to 1998.
  - For a thrilling read, take a look at this [this blog post](https://developer.r-project.org/Blog/public/2020/02/16/stringsasfactors/) by the R development team


## What's happening?
```{r}
str(survey.messy)
```
- Several of the entries have non-numeric values in them (they contain strings)

- As a result, `TVhours` is being imported as `character` vector

## A look at the TVhours column

```{r}
survey.messy$TVhours
```

## Partial fix
- In Lecture 1 we saw that there exists a family of `as.`*type* functions that will try to objects from one data type to the specified *type*

- We want TVhours to be numeric, so let's try `as.numeric`

```{r}
as.numeric(survey.messy$TVhours)
```


## We can do a bit better

```{r}
as.numeric(survey.messy$TVhours)
```

- All the corrupted cells now appear as `NA`, which is R's missing indicator

- We can do a little better by looking at the corrupted entries and seeing if we can recover more information from the cells that contained non-numeric values

## Deleting non-numeric (or .) characters 

- Here we'll use the **`gsub()`** function (global substitution) to clean up more of the corruption

```{r}
head(survey.messy$TVhours, 40)
# Use gsub() to replace everything except digits and '.' with a blank ""
gsub("[^0-9.]", "", survey.messy$TVhours) 
```

- As a last step, we should go through and figure out if any of the `NA` values should really be `0`.  
  - This step is not shown here.

## One-line cleanup 

- Let's clean up the `TVhours` column and cast it to numeric all in one command

```{r}
survey <- mutate(survey.messy, 
                 TVhours = as.numeric(gsub("[^0-9.]", "", TVhours)))
str(survey)
```

## Another common problem

- On Homework 2 you'll learn how to wrangle with another common problem

- When data is entered manually, misspellings and case changes are very common

- E.g., a column showing Program information may look like,

```{r}
program <- c("ppm", "PPM", "MISM", "HCA", "hca", "mism", "PPM-DA", "PPM-DA", "MSHCA", "MSPMM-DA", "PPM")

table(program)
```

## 
```{r}
table(program)
```

- This vector has a lot of redundant unique values that we won't want to carry through our entire analysis

- E.g., hca and HCA, mism and MISM, ppm and PPM should certainly be combined.  We might even want to combine PPM and PPM-DA together.

- On HW 2 you'll see a quick way to fix capitalization issues.  For other forms of redundancy, you'll likely want to use a function like `recode()` introduced in Lecture 3.  

## When are factor variables useful?

- Factor variables are handy when it's important to have control over the ordering of the variable values.

- E.g., What happens when we plot everyone's prior programming experience?

```{r, fig.height = 3, fig.width = 5, fig.align='center'}
qplot(survey$PriorExp)
```

- The x-axis values appear in alphabetical order.  Not always desirable.
- What if we wanted the values to appear in ascending order of experience?

## Factor variables

- We can mutate `PriorExp` into a factor with levels in a specified order using the `factor()` command, specifying the `levels` of the variable in the order we want them to appear

```{r}
survey <- survey %>%
  mutate(PriorExp = factor(PriorExp,
                           levels = c("Never programmed before",
                                      "Some experience",
                                      "Extensive experience")))
head(survey$PriorExp)
```

- Now `PriorExp` is a factor variable, with values ordered from "Never programmed before" to "Extensive experience"

## Reconstructing the plot

- Here's what we get if we run the exact same plotting command again

```{r, fig.height = 3, fig.width = 5, fig.align='center'}
qplot(survey$PriorExp)
```

- Better!  This more clearly communicates the distribution of prior programming experience among survey respondents.  


## Functions
- We have used a lot of built-in functions: `mean()`, `subset()`, `plot()`, `read.table()`...

- An important part of programming and data analysis is to write custom functions

- Functions help make code **modular**

- Functions make debugging easier

- Remember: this entire class is about applying *functions* to *data*

## What is a function?

> A function is a machine that turns **input objects** (arguments) into an **output object** (return value) according to a definite rule.

- Let's look at a really simple function

```{r}
addOne <- function(x) {
  x + 1
}
```

- `x` is the **argument** or **input**

- The function **output** is the input `x` incremented by 1

```{r}
addOne(12)
```

## More interesting example

- Here's a function that returns a % given a numerator, denominator, and desired number of decimal values

```{r}
# Ended here
calculatePercentage <- function(x, y, d) {
  decimal <- x / y  # Calculate decimal value
  round(100 * decimal, d)  # Convert to % and round to d digits
}

calculatePercentage(27, 80, 1)
```

- If you're calculating several %'s for your report, you should use this kind of function instead of repeatedly copying and pasting code

## Function returning a list
- Here's a function that takes a person's full name (FirstName LastName), weight in lb and height in inches and converts it into a list with the person's first name, person's last name, weight in kg, height in m, and BMI.

```{r}
createPatientRecord <- function(full.name, weight, height) {
  name.vec <- strsplit(full.name, split=" ")[[1]]
  first.name <- name.vec[1]
  last.name <- name.vec[2]
  weight.in.kg <- weight / 2.2
  height.in.m <- height * 0.0254
  bmi <- weight.in.kg / (height.in.m ^ 2)
  list(first.name=first.name, last.name=last.name, weight=weight.in.kg, height=height.in.m,
       bmi=bmi)
}
```
 
## Trying out the function
```{r}
createPatientRecord("Michael Smith", 185, 12 * 6 + 1)
```

## Another example: 3 number summary
- Calculate mean, median and standard deviation
```{r}
threeNumberSummary <- function(x) {
  c(mean=mean(x), median=median(x), sd=sd(x))
}
x <- rnorm(100, mean=5, sd=2) # Vector of 100 normals with mean 5 and sd 2
threeNumberSummary(x)
```


## If-else statements

- Oftentimes we want our code to have different effects depending on the features of the input

- Example: Calculating a student's letter grade
  - If grade >= 90, assign A
  - Otherwise, if grade >= 80, assign B
  - Otherwise, if grade >= 70, assign C
  - In all other cases, assign F

- To code this up, we use if-else statements

## If-else Example: Letter grades
```{r}
calculateLetterGrade <- function(x) {
  if(x >= 90) {
    grade <- "A"
  } else if(x >= 80) {
    grade <- "B"
  } else if(x >= 70) {
    grade <- "C"
  } else {
    grade <- "F"
  }
  grade
}

course.grades <- c(92, 78, 87, 91, 62)
map_chr(course.grades, calculateLetterGrade)
```

## `return()`

- In the previous examples we specified the output simply by writing the output variable as the last line of the function

- More explicitly, we can use the **`return()`** function

```{r}
addOne <- function(x) {
  return(x + 1)
}

addOne(12)
```

- We will generally avoid the `return()` function, but you can use it if necessary or if it makes writing a particular function easier.
- Google's style guide suggests explicit returns.  Most do not. 


## More programming basics: loops

- We'll now learn about loops and some more efficient/syntactically simple loop alternatives

- **loops** are ways of iterating over data


## For loops: a pair of examples
```{r}
for(i in 1:4) {
  print(i)
}

phrase <- "Good Night,"
for(word in c("and", "Good", "Luck")) {
  phrase <- paste(phrase, word)
  print(phrase)
}
```

## For loops: syntax

> A **for loop** executes a chunk of code for every value of an **index variable** in an **index set**

- The basic syntax takes the form

```{r, eval=FALSE}
for(index.variable in index.set) {
  code to be repeated at every value of index.variable
}
```

- The index set is often a vector of integers, but can be more general

## Example
```{r}
index.set <- list(name="Michael", weight=185, is.male=TRUE) # a list
for(i in index.set) {
  print(c(i, typeof(i)))
}
```

## Example: Calculate sum of each column
```{r}
fake.data <- matrix(rnorm(500), ncol=5) # create fake 100 x 5 data set
head(fake.data,2) # print first two rows

col.sums <- numeric(ncol(fake.data)) # variable to store running column sums
for(i in 1:nrow(fake.data)) {
  col.sums <- col.sums + fake.data[i,] # add ith observation to the sum
}
col.sums

colSums(fake.data) # A better approach (see also colMeans())
```

## while loops

- **while loops** repeat a chunk of code while the specified condition remains true

```{r, eval=FALSE}
day <- 1
num.days <- 365
while(day <= num.days) {
  day <- day + 1
}
```

- We won't really be using while loops in this class

- Just be aware that they exist, and that they may become useful to you at some point in your analytics career

## Loop alternatives

Command | Description
--------|------------
`apply(X, MARGIN, FUN)` | Obtain a vector/array/list by applying `FUN` along the specified `MARGIN` of an array or matrix `X`
`map(.x, .f, ...)` | Obtain a *list* by applying `.f` to every element of a list or atomic vector `.x`
`map_<type>(.x, .f, ...)` | For `<type>` given by `lgl` (logical), `int` (integer), `dbl` (double) or `chr` (character), return a *vector* of this type obtained by applying `.f` to each element of `.x` 
`map_at(.x, .at, .f)` | Obtain a *list* by applying  `.f` to the elements of `.x` specified by name or index given in `.at`
`map_if(.x, .p, .f)` | Obtain a *list* `.f` to the elements of `.x` specified by `.p` (a predicate function, or a logical vector)
`mutate_all/_at/_if` | Mutate all variables, specified (at) variables, or those selected by a predicate (if)
`summarize_all/_at/_if` | Summarize all variables, specified variables, or those selected by a predicate (if)

- These take practice to get used to, but make analysis easier to debug and less prone to error when used effectively

- The best way to learn them is by looking at a bunch of examples.  The end of each help file contains some examples.  

## Example: apply()
```{r}
colMeans(fake.data)
apply(fake.data, MARGIN=2, FUN=mean) # MARGIN = 1 for rows, 2 for columns
# Function that calculates proportion of vector indexes that are > 0
propPositive <- function(x) mean(x > 0)
apply(fake.data, MARGIN=2, FUN=propPositive) 
```

## Example: map, map_<type>()
```{r}
map(survey, is.numeric) # Returns a list
map_lgl(survey, is.numeric) # Returns a logical vector with named elements
```

## Example: apply(), map(), map_<type>()
```{r}
apply(cars, 2, FUN=mean) # Data frames are arrays
map(cars, mean) # Data frames are also lists
map_dbl(cars, mean) # map output as a double vector
```

## Example: mutate_if

Let's convert all factor variables in Cars93 to lowercase

```{r}
head(Cars93$Type)
Cars93.lower <- mutate_if(Cars93, is.factor, tolower)
head(Cars93.lower$Type)
```

- Note: this has the effect of producing a copy of the `Cars93` data where all of the factor variables have been replaced with versions containing lowercase values

## Example: mutate_if, adding instead of replacing columns

If you pass the functions in as a list with named elements, those names get appended to create modified versions of variables instead of replacing existing variables

```{r}
Cars93.lower <- mutate_if(Cars93, is.factor, list(lower = tolower))
head(Cars93.lower$Type)
head(Cars93.lower$Type_lower)
```

## Example: mutate_at

Let's convert from MPG to KPML but this time using `mutate_at`
```{r}
Cars93.metric <- Cars93 %>% 
  mutate_at(c("MPG.city", "MPG.highway"), list(KMPL = ~ 0.425 * .x))
tail(colnames(Cars93.metric))
```
Here, `~ 0.425 * .x` is an example of specifying a "lambda" (anonymous) function.  It is permitted short-hand for 

```{r, eval = FALSE}
function(.x){0.425 * .x}
```


## Example: summarize_if

Let's get the mean of every numeric column in Cars93

```{r}
Cars93 %>% summarize_if(is.numeric, mean)

Cars93 %>% summarize_if(is.numeric, list(mean = mean), na.rm=TRUE)
```

## Example: summarize_at

Let's get the average fuel economy of all vehicles, grouped by their Type

```{r}
Cars93 %>%
  group_by(Type) %>%
  summarize_at(c("MPG.city", "MPG.highway"), mean)
```

## Another approach

We'll learn about a bunch of select helper functions like `contains()` and `starts_with()`.  

Here's one way of performing the previous operation with the help of these functions, and appending `_mean` to the resulting output. 

```{r}
Cars93 %>%
  group_by(Type) %>%
  summarize_at(vars(contains("MPG")), list(mean = mean))
```

## More than one grouping variable

```{r}
Cars93 %>%
  group_by(Origin, AirBags) %>%
  summarize_at(vars(contains("MPG")), list(mean = mean))
```

## R coding style

Let's return back to the [last few slides of lecture 2](http://www.andrew.cmu.edu/user/achoulde/94842/lectures/lecture02/lecture02-94842.html#33)

## Assignments

- **Homework 2** will be posted today
  - **Due: Wednesday, November 11, 1:30pm ET**
  - Submit your .Rmd and .html files on Canvas

- **Lab 4** is available on Canvas and the course website
  - You have until Friday evening to complete it
  - Friday's lab session will go over this week's material and help you complete the labs