Generators

We've spent some time playing with lists -- they are ordered collections that are iterable, indexable -- and finite. But, what if we'd like to have an iterable collection -- that isn't necessarily finite. Imagine an ordered collection of the even numbers or the digits of pi. Although it isn't indexable, Python does give us a way of representing such an ordered sequence in an iterable way -- generators.

To do this, we write what looks a lot like a function that computes each item in the sequence, one at a time, stopping after each one. Then, each time we "iterate" upon the generator, it computes and yields the next item. The syntax of the generator looks a lot like the syntax of a function, but we "yield" a value instead of "return"ing the value. A "yield" is like a reusmable return -- the next time we poke the generator, it will pick up there.

In class we built up the example below. Notice that we used a loop to genrate the sequence; that if the loop isn't infinite, the sequence isn't either; that we can iterate using next(); and we can also iterate using a for-loop, as we can for any other iterable

---

def generateEvens(start):

  if ((start % 2) != 0): start += 1

  even = start
  while True:
    yield even
    even +=2

evens = generateEvens(3)

print next(evens)
print next(evens)
print next(evens)
print next(evens)


# Reset the generator: 
# Technically, create a new one and assign it to the same variable
evens = generateEvens(3)

# This would be infinite!
# for even in evens: print even

def generateEvensRangeInclusive(start, end):

  if ((start % 2) != 0): start += 1

  even = start
  while (even <= end):
    yield even
    even +=2

evens = generateEvensRangeInclusive(3, 10)
for even in evens: print even

# Reset 
evens = generateEvensRangeInclusive(3, 10)
print next(evens)
print next(evens)
print next(evens)
print next(evens)
print next(evens) # Notice the StopIterator exception
  

---

Generators "Comprehensions" Expressions

Just like we could use a closed form to initialize a list, we can do the same for generators. Instead of using []-brackets to define the comprehension, we use ()-parenthesis to define the generator "expression".

The example below shows a couple of genertor expressions and their use:

  

sentence = "The quick brown fox jumps over the lazy old dog."

# List comprehension
wordList = [word for word in sentence.split()]
print wordList # Prints the list as a list
for word in wordList: print word # Works as expected
print wordList[2] # Legal

# Similar generator expression
wordGenerator = (word for word in sentence.split())
print wordGenerator # Prints nothing useful
for word in wordList: print word # Works as expected
# print wordGenerator[2] # Syntax Error
  

  

Pipelining Generators

One of my favorite features of generators is the ability to use them in a pipeline for filtering. The example below does several steps in a pipeline: parses words, removes periods from them, upper cases them, numbers them. Notice how one generator is able to "pull" data from its predecessor, moving it from the first to the last:

---

sentence = "The quick brown fox jumps over the lazy old dog."

words = (sentence.split())
#for word in words: print word

wordsNoPeriods = (word.replace(".","") for word in words)
#for word in wordsNoPeriods: print word

wordsUpper = (word.upper() for word in wordsNoPeriods)
#for word in wordsUpper: print word

def numbersGenerator():
  number = 0
  while True:
    yield number
    number = number+1
numbers = numbersGenerator()

numberedWords = (str(numbers.next()) + ":" + word for word in wordsUpper)
for word in numberedWords: print word
  

---

Please also consider the example below, which is more complex, but may actually be clearer. In it, we don't rely upon comprehensions. Instead we actually define generators old school and go from there:

---

#!/usr/bin/python

import random # Random number generators

def randomNumber(minIncl, maxExcl):
  while True:
    yield random.randint(minIncl, maxExcl)

def doubleNumber(generatorForInputs):
  while True:
    yield 2*generatorForInputs.next()

def squareNumber(generatorForInputs):
  while True:
    yield generatorForInputs.next() ** 2

# Here is a very clear pipeline, assembled from parts
rng = randomNumber(1, 1001)
dng = doubleNumber(rng)
sng = squareNumber(dng)

# Prints the square of a doubled random number
print sng.next()
print sng.next()
print sng.next()
print sng.next()
  

---

Coroutines

Coroutines are, like generators, a very unique and interesting feature of Python. They enable us to implement an algorithm, using a nice, clean, function like defintion -- but to pause the execution of the algorithm for input.

Generators paused after generating output, waiting for the caller to request more. Coroutines pause execution waiting for the caller to send input. In Coroutines, the "yield" represents a value and will most often appear on the right side of an equal sign, or as the argument to a function -- by contrast, in generators, "yield" is a command, much like "return".

Coroutines are probably best explained by example. Consider the following simple example, in which the coroutine waits for a vlaue, and then prints it. We define the coroutine, create one and assign it to a variable, advance it to the "yield", and then send it values, one at a time:

---

def printValue():
  while True:
    value = (yield)
    print value

pv = printValue() # Create an assign
pv.next() # Advance until it blocks at the first "(yield)"
pv.send("This string is sent and becomes the value of the (yield)")
pv.send("And again...")
pv.send("And again...")

# We can also do it in a loop
sentence = "The quick brown fox jumps over the lazy old dog."
for word in sentence.split(): pv.send(word)
  

---

Much as we could with generators, we can also organize coroutines into a pipeline. But, there is an important, be it somewhat subtle, difference. When pipelining generators, the data is pulled through the pipeline of generators. When we pipeline coroutines, the data is pushed through from last-to-first. Compare the example below to the generator example in the prior lecture.

---

def printWord():
  while True:
    word = (yield)
    print word

def numberWord(targetCR):
  number = 0
  while True:
    word = (yield)
    targetCR.send(str(number) + ": " + word)
    number += 1

def upperWord(targetCR):
  while True:
    word = (yield)
    targetCR.send(word.upper())

def wordNoPeriods(targetCR):
  while True:
    word = (yield)
    targetCR.send(word.replace(".",""))

# I like to use the ;-semicolon to intialize and advance on the same line
pw = printWord(); pw.next()
nw = numberWord(pw); nw.next()
uw = upperWord(nw); uw.next()
wnp = wordNoPeriods(uw); wnp.next()
  
sentence = "The quick brown fox jumps over the lazy old dog."
for word in sentence.split(): wnp.send(word)
  
  

---