An Introduction to Sorts
You probably did at least a small amount of sorting in your introductory course. Sorting a collection of items is an important computing application. There are many commonly-used sorting algorithms. Some of them are good for sorting small sets of data, while others are good for sorting large sets of data. Some are good under certain conditions and horrid in other conditions. Some take about the same length of time to run no matter what the conditions. Some are simple, others are a bit more complicated.In order to keep things simple and focus our attention on sorting, we'll sort an array of integers, from lowest to highest, in our examples. In reality, you can sort any collection of Objects (remember Comparable).
Bubble Sort
A bubble sort traverses the array, or a selected portion of the array, length-1 times (a less efficient version of the bubble sort could pass through the array length times and still produce a sorted array). With each pass through the array, the bubble sort compares adjacent values, swapping them if necessary. This actually results in "bubbling" the highest value in the array up to the highest index (length-1) by the end of the first pass through the array, the second-highest value up to the second-highest index (length-2) after the second pass through the array, and so on. By the time the bubble sort has made length-1 passes through the array (or a selected portion of the array), every item, including the lowest item, is guaranteed to be in its proper place in the array.Let's see how the bubble sort works on an array of integers. We start at the lowest pair of indexes, and work our way up to the highest.
The following shows one entire pass through the array using a bubble sort. Turn your head sideways to the left to read this array. The array indexes are the left-hand, red column. The two adjacent items currently being compared are bold. Note that there are eight total items in the array. Notice that we will make seven comparisons (there are eight columns after the red column) during this first pass.
7 7 7 7 7 7 7 7 9 6 4 4 4 4 4 4 9 7 5 5 5 5 5 5 9 4 4 4 6 6 6 6 9 5 5 5 3 3 3 3 9 6 6 6 6 2 1 1 9 3 3 3 3 3 1 2 9 1 1 1 1 1 1 0 9 2 2 2 2 2 2 2 What has happened to the 9 that started at index 0? It happened to be the highest value in the array, so every time we compared adjacent values we had to swap until the 9 was finally in the last position in the array. After one pass through the array, we have successfully moved the highest value into the highest index.
On our next pass through the array, we'll move the second-highest value into the second-highest index. This time we will stop at the second-to-last pair of values, since we already know that the highest value is at the highest index. The 9 is in blue, since it is already at its proper index.
7 9 9 9 9 9 9 9 6 7 7 7 7 7 7 7 5 4 4 4 4 4 6 6 4 5 5 5 5 6 4 4 3 6 6 6 6 5 5 5 2 3 3 3 3 3 3 3 1 1 2 2 2 2 2 2 0 2 1 1 1 1 1 1 On the second pass through, the second-highest value was already in the second-highest position in the array, but we did move some of the other values one step closer to being in the correct position. Notice that this time, we made only six comparisons (see that there are only seven columns after the red column). There's no need to make the seventh comparison, since the 9 is already in its proper place.
We continue this process until we have moved every value to the correct index.
Bubble Sort Code
public void bubbleSort(int[] numbers) { /*traverse the array (or a subset of the array) length-1 times*/ for (int highest_unsorted=numbers.length-1; highest_unsorted != 0; highest_unsorted--) { /*makes the necessary comparisons for one pass through the array*/ for (int best_so_far=0; best_so_far < highest_unsorted; best_so_far++) { /*compare adjacent items and swap them if necessary*/ if (numbers[best_so_far] > numbers[best_so_far+1]) swapNumbers (best_so_far, best_so_far+1); } } } public void swapNumbers(int i, int j) { int temp = numbers[i]; /*put numbers[i] somewhere to keep it safe*/ numbers[i] = numbers[j]; /*put numbers[j] at index i*/ numbers[j] = temp; /*put numbers[i] at index j*/ }Now we're going to pause for a minute to introduce something called "algorithmic complexity". It basically measures how expensive an algorithm is. There are many different ways to measure cost, whether by how much money something is, how much time is spent, power, etc. Computers are often measured on what is called "runtime", essentially how much time on a stopwatch it takes to run an algorithm. This can be made much faster with more ram, a faster processor, etc. Algorithmic complexity, on the other hand, just looks at the algorithm independent of the environment it is run on, and simply answer the question, 'how complex is it?' To measure, we have to specify what case we're measuring, whether the best case, the worst case, or the average case. Usually, computer scientists concentrate on the worst case, often called an outer bound. The notation for this is O, spoken as "big O". To do this, we have to ignore many things, such as constant time factors.
Analyzing Running Time
The bubble sort is a sorting algorithm which is very easy to implement, but how good is this algorithm? In order to answer this question, we need to look at how much work the algorithm has to do. When we analyze the running time of an algorithm, we want to look at the bigger picture, so we will make some approximations and ignore constant factors in order to simplify the analysis.
Let's say we have 10 numbers. The outer for loop of the bubble sort has to run 9 times (once when last_one = 9, once when it = 8, etc., up until it = 1, and when it hits 0 it stops). If we had 100 numbers, the outer loop would run 99 times. If we had 1000 numbers, the outer loop would run 999 times. In general, if there are N numbers, the outer loop will run (N-1) times, but to simplify our math we will say that it runs N times.
If the outer loop runs N times, then the total running time is N times the amount of work done in one iteration of the loop. So how much work is done in one iteration? Well, one iteration of the outer for loop includes one complete running of the inner for loop. The first time through, the inner loop goes 9 times, the second time through it goes 8 times, then 7, and so on. On average, the inner loop goes N/2 times, so the total time is N for the outer loop times N/2 for the inner loop times the amount of work done in one iteration of the inner loop.
For one iteration of the inner loop, we either do nothing if the number in the less than the one after it, or we set three values if we need to swap. In the worst case, we will need to swap at every step, so we will say that the one iteration of the inner loop requires 3 operations. That makes the total time for the bubble sort algorithm N*(N/2)*3 operations, or (3/2)*N2
Selection Sort
Now let's look at another easy sort, called the selection sort. With Bubble sort, we remembered the "smallest value so far" by swapping it into current position of the list during each step of each iteration. Although this works, it take a lot of effort to swap numbers around -- especially if all we really need to know is the index of the "smallest value so far". This is the critical difference in thw two sorts -- Selection Sort just remembers the position fo the "best so far", instead of swapping it into the current position. This reduces the amount of work required during each step of the sort. Then, at the end of each pass, it swaps just once.So, let's take a look at this step-by-step. In the selection sort, we find the smallest value in the array and move it to the first index, then we find the next-smallest value and move it to the second index, and so on. We start at the first index and walk through the entire list, keeping track of where we saw the lowest value. If the lowest value is not currently at the first index, we swap it with the lowest value.
Now, let's take a look at an example. We will start with the same set of numbers we used before.
0 1 2 3 4 5 6 7 9 2 1 3 6 5 4 7 So the first step in the selection sort is to get the lowest value into the lowest index (index 0 in this case). We will loop through all of the values, keeping track of which index contains the lowest value. Since we are trying to fill index 0, we will assume that the lowest value is already there unless we find one that is lower - we initialize our best index to be index 0.
As we scan through, we find that the lowest value is the 1 which is at index 2, so we swap the value at index 0 with the value at index 2.
0 1 2 3 4 5 6 7 1 2 9 3 6 5 4 7 Now the lowest value is at index 0, so the next step is to get the next-lowest value into index 1. When we scan through the numbers, we find that the next-lowest value, the 2, is already at index 1, so we do not need to swap.
If we continue through the array one index at a time, we will eventually move every value into the appropriate index, resulting in a sorted array.
Selection Sort Code
public void selectionSort(int[] numbers)
{
/*For every index in the array, with the exception of the last one,*/
for(int searcherIndex=0; searcherIndex < numbers.length-1; searcherIndex++)
{
/*Assume that the number is where it's supposed to be*/
int correctIndex = searcherIndex;
/*Try out other candidate indexes*/
for (int candidateIndex=searcherIndex+1; candidateIndex < numbers.length; candidateIndex++)
{
/*If you find a smaller number, make it's index the correct index*/
if(numbers[candidateIndex] < numbers[correctIndex])
correctIndex = candidateIndex;
}
/*At this point correctIndex really will be the correct index*/
swapNumbers (searcherIndex, correctIndex);
}
}
Running Time of Selection Sort
Selection sort slightly better than the bubble sort, because we swap at most once for every index, instead of potentially once for each item. We find the correct index for that particular number. Then we swap it into its correct place in the array.
Insertion Sort
Insertion sort works, as before, by viewing the list as two sub-lists, one of unsorted items, initially full, and one of unsorted items, initially empty. As before, each iteration, the next item from the unsorted list enters the sorted list.Bascially, we remove the next item from the unsorted list, and that gives us an extra slot in the sorted list. We then move each item, one at a time, into that slot, until we make a "hole" in the right place to insert the new item. When we do, we place it there.
Much like Bubble sort and Selection sort, Insertion sort is still requires a multiple of N2 operations. We move one item to the sorted list each time, so we need to make n passes. And then, each pass, we need to compare it to n/2 items (amortized over all runs).
It is however, quite a bit more expensive in real time. Displacing an item in an array is quite expensive -- everything after it needs to be copied.
But, what is special about insertion sort is that it can be started before all of the data arrives. New data can still be sorted, even if it is lower (or higher) than those already fixed in place.
Also, the cost of displacing an item isn't present, if a Linked List is being sorted.
Here's an example:
4 7 9 2 5 8 1 3 6 4 7 9 2 5 8 1 3 6 4 7 9 2 5 8 1 3 6 4 7 9 2 5 8 1 3 6 4 7 9 2 5 8 1 3 6 2 4 7 9 5 8 1 3 6 2 4 7 9 5 8 1 3 6 2 4 5 7 9 8 1 3 6 2 4 5 7 9 8 1 3 6 2 4 5 7 8 9 1 3 6 2 4 5 7 8 9 1 3 6 1 2 4 5 7 8 9 3 6 1 2 4 5 7 8 9 3 6 1 2 3 4 5 7 8 9 6 1 2 3 4 5 7 8 9 6 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
Insertion Sort Code
public void insertionSort(int[] numbers)
{
/*going through all of the items in the array*/
for (int insertMe=1; insertMe < numbers.length; insertMe++)
{
/*find the correct index for the item*/
for (int newPosn=0; newPosn < insertMe; newPosn++)
{
/*stop when you come to an item greater than the item in question*/
if (numbers[insertMe] < numbers[newPosn])
{
/*put the item in question somewhere for safe keeping*/
int temp = numbers[insertMe];
/*move everything after the correct index down to make room*/
for (int shift=insertMe; shift > newPosn; )
numbers[shift] = numbers[--shift];
/*put the item in its correct index*/
numbers[newPosn] = temp;
/*You've found the right index for the item and it's time to stop*/
break;
}
}
}
}
Quick Sort: The Strategy
Although the sorts we've discussed so far have differed in the details, they have all had the same basic strategy: Fill the sorted list in order, one item at a time, by selecting the right element from the unsorted list. Select this item by moving from one side of the unsorted list to the other, making comparisons along the way.
We are now going to take a completely different approach. For the moment, we need a small amount of magic. So, let me grant you a magic wand (Poof!). If you point this magical wand at an unsorted list of numbers, it can, much like a divining rod, find the "middle" number in the list. For example, if we point it at the numbers
5, 7, 2, 1, 6, 8, 9it will point at "6", the 3rd (starting with 0) number in the list -- the middle one. This is a very powerful magic wand -- because we've now found the right value for the 3rd position, and can just swap it into place. We'll call this place the "pivot point", because it is right in the middle of the list.
So, this magic wand makes sorting perfectly simple, right? We can just zap some more numbers, right? Well, not exactly. We've got (at least) one problem -- we've now got two lists, not one. The list to the left of the pivot, and the list to the right of the pivot. Which one do we zap?
Well, we need to zap each of them, each has its own pivot point. But, there is a problem with that, too. Each of the lists is unordered. Take a look at the list again -- this time with the one pivot we know swapped into place:
5, 7, 2, 6, 1, 8, 9The list on the left has 7, which is greater than the pivot, and the list on the right has 1, which is less than the pivot. 7 might be in the right place, but these two lists don't have the right values.
So, let's start at the beginning of the left list and move right until we find a value that should be in the right list -- 7. Then, let's start at the right side of the right list, until we find a value that belongs in the left list -- 1. Then, let's swap these two. We should repeat this process, until everything in the left list is less than the pivot and everything in the right list is greater than the pivot:
5, 1, 2, 6, 7, 8, 9Now, we can just use that magic wand twice more -- once on the left list and once on the right list. In fact, we are just going to do this recursively until the list size is 2, at which time we'll just compare the two numbers, and swap if necessary.
So, in the left list, we pick 2 as the pivot and swap it into place
5, 2, 1, 6, 7, 8, 9Then, we swap the 1 and 5 with each other, since each is on the wrong side of the pivot value, 2.
1, 2, 5, 6, 7, 8, 9Now we repeat the same for the right list, selecting 8 as the pivot, discoverign that it is in the right place, and then checking the values in the left and right lists -- each of whcih contains one number that is in the right place:
1, 2, 5, 6, 7, 8, 9Now, we repeat this, again recursively on each of the sublists. But, we discover that each has a length of 1, so we know that each is in the right place (how to get a list of one to be out of order?)
Think about what we're doing here, we're dividing thi list in half, and then half again, and again, until the list has only one item. For this reason, Quicksort is often known as a divide adn conquer algorithm. By using recursion, we cave the problem into smaller and smaller pieces, each of which is much easier to solve.
1, 2, 5, 6, 7, 8, 9
Quick Sort: Making the Magic Disappear (Poof!)
So, given this magic wand, we can sort a list. Pretty cool, huh? But, what about without this magic wand? Easy answer -- we just pretend. We pick a value, and pretend it is the pivot. If we are right, the sort proceeds as we've discussed. If we aren't right, no big deal -- the left and right lists will just be different sizes. This is suboptimal, but still functional. The only problem is that we have learned less about the positions of the numbers than we otherwise could. Imagine if we'd pick the lowest (or highest) number in the list as the pivot. We'd still put it into its place -- but all of the vlaue would be on one side of it. We wouldn't be able to divide the problem at all.
At the implementation level, to facilitate things, weh we guess at the pivot, we are going to swap it out of the list, into the last element, to get it out of the way. Unlike our magically perfect pivot value, we don't know exactly where it goes, until we divide the list into those things that are less than it and those things that are greater than it. At that point, we know where the two lists meet, and can swap the pivot value into the proper place.
Which number do we guess is the pivot? Well, since we don't know, if the unsorted list was truely in a random order, we could pick any -- the first, the last, or anything in between -- we'd have an equal change of being right -- and no greater a chance of being more wrong. But, if the list doesn't happen to be in a random order, there are better and worse choices. For example, if the list is sorted, and we always pick the first value, it will always be the worst choice -- everything will be greater than it. The same is true if we always pick the alst value and the list is sorted the opposite way. We could pick a random value each time -- just pull a number out of the hat, per se. And this will prevent a consistently bad choice -- but it'll waste some time picking a random number and could still be bad, sometimes.
So, instead, we'll take the number in the middle of the list. If the list is in a random order, it is as good as any. But, if the list is in sorted order, whether forward or backward, it is optimal. The middle value will be in the middle. Of course, there is still a worst case ordering for this approach -- but it is somewhat less likely given human behavior.
Quick Sort: A Real Example
Let's take a look at quicksort in a more structured example. This time, we're going to pick the middle value as the pivot. And, we're going to swap it out of the list, until we divide the list into two sublists, each of which has either vlaue less than the pivot, or greater than the pivot. Once we do that, where the two lists meet is the pivot point, so we can swap it into its proper place.
This is the initial state of our quicksort example (sorted elements are italicized).
5 8 1 4 3 7 6 9 11 10 12 2 After sorting the rest of the list based on if it's in the right place relative to the pivot, we find that the pivot should be where the 8 now is.
5 6 1 4 3 2 8 9 11 10 12 7 So you swap them and QuickSort is then recursively called on the first half of the sequence up to the old pivot, with 4 as the new pivot, up unto the pivot.
5 6 1 4 3 2 7 9 11 10 12 8 The 4 gets swapped with the 2, which is the last element of the partition, and all the elements are tested to see if they belong in the right place, and it's determined that in the lower array, where the 6 is, is where the 4 should be.
3 2 1 6 5 4 7 9 11 10 12 8 Now the 4 is in place, so the 2 is the new pivot of that lower partition.
3 2 1 4 5 6 7 9 11 10 12 8 The 1 and 3 get swapped because they're in the right place, and the two gets put back. Since the partitions are each one element big, we know they're done.
1 2 3 4 5 6 7 9 11 10 12 8 The six becomes the next pivot for the next sub-array, and since it's in the right place, that's done too. So now we look at the upper partition from placing the first pivot in its spot. First we choose the the 10 as the pivot.
1 2 3 4 5 6 7 9 11 10 12 8 We switch it with the 8, which is at the end, and check the elements to see if they're on the correct sides. We end up having to swap the 11 and the 8. So now we know where the 10 goes, and we swap with the 11.
1 2 3 4 5 6 7 9 8 11 12 10 We switch it with the 8, which is at the end, and check the elements to see if they're on the correct sides. We end up having to swap the 11 and the 8. So now we know where the 10 goes, and we swap with the 11.
1 2 3 4 5 6 7 9 8 10 12 11 After doing the same recursive calls on the two-element partitions, we have our ordered list.
1 2 3 4 5 6 7 8 9 10 11 12
Quick Sort: A Look at the Performance
The best, average, and worst cases of quicksort actually turn out to be very different, because of the variabilitity introduced by the choice of pivots.
In the best case, the pivot exactly divides each the list. So, we have to pick Log2N pivots, before each sublist is of size 1. So, we have Log2 N lists to sort using recursion. For each of these lists, we walk through the elements linearly, one at a time, looking for items that are out of place and should be swapped. So, we have a linear process. So, in the best case, quick sort requires N*Log N operations. The "N" is from the traverse-and-swap within the sublist, and the "Log N" is the number of sublists.
In the worst case, where we pick exactly the wrong pivot, we don't divide things in half each time. Instead, we end up with a list of size 0 and a list of size N-1 to sort each time. So, the divide step requires N operations, and the traverse-and-swap still requires N operations, for a total of a multiple of N2) operations -- we have to traverse-and-swap for each of the N lists.
In this respect, a worst-case quick sort is much like a bubble or selection sort. But, as it turns out, either is more efficient than a worst-case quicksort. The reason is that quicksort uses recursion, which is quite expensive -- even more expensive than bubble sort's constant swapping. So, if the pivot is bad, there is very little algorithmic gain -- but a lot of additional overhead.
The same is actually true for small lists, even in the optimal case -- the overhead of recursion outweights the algorithmic gain for small lists. So, in truth, it doesn't make sense to quicksort down to a list size of 1 or two. A more typical strategy is to quick sort down to a list size of 50-100, and then use a selection sort.
In the average case, empirical analysis has shown performance to be much closer to optimal than worst case -- still N*Log N.
Quicksort is known as an unstable sort, because its best and worst cases are different. By contrast, bubble sort and selection sort are stable sorts -- their performance is constant, regardless of the "luck of the draw".
Quick Sort Code
/*
*quickSort() calls sortPartition()
*/
public void quickSort(int[] numbers)
{
sortPartition (0, numbers.length-1);
}
/*
* This is a helper method which is used by findPartition() to
* find the "pivot point" - the place to divide the partition.
*/
private int findPivot (int left, int right)
{
return ((right + left)/2);
}
/*
* This is a helper method called by sortNumbers(). It sorts
* an individual partition about the given pivot point.
*/
private int partition (int left, int right, int pivot)
{
do
{
while (numbers[++left] < numbers[pivot]);
while ((right !=0) && (numbers[--right] > numbers[pivot]));
swapNumbers (left, right);
} while (left < right);
swapNumbers (left, right);
return left;
}
/*
* This is a helper method called by sortNumbers(). It recursively
* calls itself on subpartitions to sort the numbers. The actual
* sorting within the partition is done by sortPartition(), which
* is iterative.
*/
private void sortPartition (int left, int right)
{
int pivot = findPivot (left, right);
swapNumbers (pivot, right);
int newpivot= partition (left-1, right, right);
swapNumbers (newpivot, right);
if ((newpivot-left) > 1) sortPartition (left, newpivot-1));
if ((right - newpivot) > 1) sortPartition (newpivot+1, right));
}