Return to the Lecture Notes Index

Lecture 3 (January 21, 2000)


Reading

Project Description
It Is An Exciting Day! Project #1
 
Brief Overview

You'll be implementing a UNIX shell with job control, pipes, I/O redirection, and some accouting. In many ways it will function like your favorite UNIX shells.

The project description is also availabel via the web:

 Project Description (HTML)
Deadline

This project is due at 11:59pm on Friday, January 28th, 2000.
 

Groups

Please register your groups by Monday night or Tuesday at the absolute latest using the form available via the Web:
 Project #1 Group Registration
Groups should be two people. Permission is required to work by yourself -- this practice is not encouraged.

Deadlines

The general rule is that late work is not accepted. Please turn in whatever you have on or before the deadline. If something unusual and insurmountable comes up, please speak to us as early as possible. We will always do our best ot work with you -- and to be fair to the rest of the class. The later you are in bringing these matters to our attention, the less helpful we can be, while remaining fair.

Environment

Whereas you can do this assignment on any UNIX, it must run on the Solaris machines for your demo. We encourage you to work in that environment, because the libraries ofr future projects are only available in the andrew enviroment.

Although you can solve this assingment in your choice of languages, it would probably be more difficult in anything other than C (or perhaps C++).  We strongly encourage you to use C.

For future projects you'll almost certainly have to use C. C++ may work, but we offer no guarantees. There is a difference in the linking convention between the two languages that causes incompatibilities. We've tried to correct this, but we make no promises about C++.

The Demo

After projects are over, we'll do demos. This will be your moment in the spotlight to show us what you've done. Hopefully you'll take great pride in demonstrating your project passing our various tests.

We'll also give you an oral exam of sorts to make sure that both people involved learned from the exercise and seem familar with the code. Please try to understand the whole project, not just the parts that you implemented.

And of course, we'll offer you some feedback about what you did well and what you could have done better.

The Parser

Much like a real shell, you'll have to parse the command line to determine the various components of the user input. We've provided a grammar that describes the langauge of the shell. By enforcing this grammar, you'll be preventing many types of non-sensical and troublesome to implement input:
 

Example: ls > outfile | more
But let us caution you. the parser is very uninteresting to us in a systems course. Spend as little time on this part of the project as you can. This is one case where sufficient is good enough.

We are really interested in the system programming part of this project. But many sutdents in the past have spent massive amounts of time on the parser and had difficulty finishing the rest. Paresers are interesting -- but please be cautious.

Getting Help

We're here to help! Please give us the opportunity. My office hours and the office hours of the TAs have recently been posted to the Web and bboards.

The staff-412@cs mailing list is often the most prefered resourse. It is constantly monitored and answered for questions. Since we all service the list, the response time is much better than any one of us could individually obtain. We're happy to answer questions that are directed individually to us, but the list will probably be faster.

We'll also be posting anonymized version of the questions and answers to the web frequently, so check for that new feature soon.

Although we generally don't monitor Zephyr, we will place a link to a Zephyr class 412 archive on the Web. Many students like this forum for interacting with each other and 412 survivors from years past.

Hopefully the archive will be useful to you.

We're here to help.
 

Some Useful Information
Process Creation

To create a new process we use the fork() system call. The fork system call actually clones the calling process, with very few differences. The clone has a different process id  (PID) and parent process id (PPID). There are some other minor differences, see the man page for details.

The return value of the fork() is the only way that the process can tell if it is the parent or the child (the child is the new one).

The fork returns the PID of the child to the parent and 0 to the child.

This subtle difference allows the two separate processes to take two different paths, if necessary.

The wait_() family of functions allows a parent process to wait for a child process to complete. You may want to do this when you create a foreground process form your shell.

It is important to note that the wait_() family of functions returns any time the child changes status -- not just when it rolls over or exits. Many status changes you may want to ignore. You may also want to take a look at some of the flags in the man page for waitpid(), you may find WNOHANG, and others helpful. (WNOHANG makes the wait non-blocking, if there's no news -- it just let's you collect information, if available)

The following example shows a waitpid(). It waits for a specific child. wait() will wait for any child. There are several other flavors. We'll talk more about what the execve() within the child does shortly.

It is important for your shells to wait for the children that they create. This can either be done in a blocking fashion for foreground processes, or in a non-blocking fashion (WNOHANG) when the child signals. Although many of the resources composing a process are freed when it dies, the process control block(PCB) is not. The PCB contains status information that the parent can collect via wait_(). A process that is in this state is called defunct. After the wait_(), the PCB is freed. If the parent dies before the child, the child is reparented to the init() process which will perform a wait_() for any such process, allowing the PCB to be freed. Orphan process that are waiting for init to clean them up are called zombies.

 Fork-Execvp Example


Process State

This is a good time to introduce the state diagram that describes the life cycle of a process:

After the operating system is done assembling and intializing the necessary data structures and the process is runable, it enter the ready queue. At this point, we say that the process has been admitted. The ready queue is a collection of processes that are runnable. Processes from the ready queue are allocated a CPU based on a scheduling policy also known as a scheduling discipline.

When a process is given a CPU, we say that is has been dispatched. A running process may be preempted by the operating system to allow another process to run. In the case, the process is returned to the ready queue.

The process can also move itself to any of many wait queues. When process do this, we say that they are waiting or blocked. There are wait queues for almost every possible reason. Processed typically block themselves when they can't be productive so that other processes can run.

Once the blocking I/O operation or event occurs, the operating systems handler will return the blocked process to the ready queue. This makes it eligible to run.

Once a process terminates it enters one of two states: zombie or defunct We say that a process is defunct if it is terminated, but its parent hasn't waited for it. We say that a process is a zombie, if its parent has died and the init process will clean it up. A defuct process can become a zombie process, if the parent dies.


What If I Don't Want A Clone?

The exec_() family of calls allows a process to substitute another program for itself. Typically a program will call fork() to generate a duplicate copy of itself and the child will call an exec_() function to start another process.

There are several different flavors of exec_(). They all boil down to the same call within the kernel. One parameterization may be more or less convenient from time-to-time.

An exec'd process isn't completely different from the calling process. It does inherit some things, PPID, GID, and signal mask, but not signal handlers. Please see the man page for the details.

The exec_() fucntions do not return (a new process is now in charge). At least it is fair to say tat if they do return, something bad has happened.

We walked through a simple fork()-exec_() example in class.

 Fork-Execvp Example


I/O Redirection

To implement I/O redirection, you'll need to use the dup2() function:

int dup2(int fildes, int fildes2);
 
Each process contains a table with one entry for each open file. This table contains some information about the state of the open file, such as the current offset into the file (the location where the next operation will occur). It also contains a pointer to the system-wide open file table.

This table contains exactly one entry for each open file in the system. If multiple processes have the same file open, the corresponding entry in each process's file descriptor table will point to the same entry in the system-wide open file table. This table contains some information about the file, including a count of how many processes currently have it open. It also contains a pointer to the file's inode, the data structure that associates a file with its physical storage on disk. We'll talk more about this when we get to fiel systems.

It is also important to realize that many non-files use the same interface, although they operate differently under the hood. Foe example, in many ways, terminals can be manipulated as if they were files.

By default the first three entries in each process's open file table are open and reference the terminal: stdin (0), stdout(1), and stderr(2).

To perform I/O redirection, we open a file and then copy this file's file descriptor entry over either standard in or standard out (or standard error). If we need to restore the orignal entry later, we need to save it in another entry in the table.

An example of I/O redirection is given on the Web site (we walked through it in class):

 An example of I/O Redirection

Signals

Signals are the simplest primitive for interprocess communication (IPC). We'll talk more about these tools later in the semester.

Signals allow one process to communicate the occurance of an event to another process. The number of the signal indicates which event occured. No other information can be communicated via signals.

But signals will be very important in this project. They will indicate changes in the state of a child background process -- such as its termination, and other important events....that its time for a process to sleep, for example.

When a prcoess recieves a signal, it can take an action. many dignals have a default action.  For example, certain signals, by default, cause core dumps, or process's to suspend themselves.

We can also specify how we want our process to handle a particular signal (Except for KILL, which isn't really a signal, although it looks like one to the programmer). We do this by sepcifying a signal handler. We walked through the following example in class:

 Simple Signal Handler
 
Editor's Note: Some people asked about a more powerful type of signal hander that can determine which process sent the signal, &c. This is useful for handling the SIGCHLD signal -- which child needs attention?

I added another example to the course website and psoted to the bboard:

 Who Sent The Signal? Handler
 
Pipes

Pipes are a more sophisticated IPC tool. They allow for a one-way flow of data from one process to another. (Okay -- SYVR4 pipes can be bidirectional, but we'll stick to Posix pipes for this discussion).

We'll talk more about pipes later in the semester, but here's the basic idea. A pipe is basically a circular buffer that hides in the file system. We use it in a producer-consumer fashion. One process writes to the pipe, and blocks if the buffer becomes full. Another process reads from the pipe and blcoks if it becomes empty.

A read will fail if the producer closes  the pipe or dies. And a write will fail if the consumer closes the pipe or dies.
 

  1. Here's how it works. We create a pipe in the parent process using the pipe() system call, by passing it an array of two file descriptors: pfd[0] and pfd[1].
  2. Much like file descriptor 0, we will use pfd[0] for input. And we will use pfd[1] for output.
  3. We fork and create a child.
  4. Now the child and the parent both share the pipe file descriptors. Each will close one side of the pipe (whcih side depends on whether they will be reading or writing.
  5. Next each process will use dup2 to copy the open pipe file descriptor over stdin or stdout, as appropriate. We then close the pipe file descriptors, since they are no longer needed. (If we will later need to restore stdin, or stdout, they should be saved, as we discussed with redirection).
  6. Now the two processes can communicate using the pipe via stdin and stdout.


If we do this inbetwene the time we fork and we exec_(), we can tie processes together using pipes -- even though they are ignorantly communicating using stdin and stdout.

We walked through this example in class:

 Pipe Example

Processes, Process Groups, Sessions, and Job Control

When we log into a system, the operating system allocates a terminal for our session. A session is an environment for processes that is (or at least can be) associated with one controlling terminal.

Our shell is placed into the foreground process group within this session. A process group is a collection of one process or of related processes -- they are usually related by one or more pipes. At most, one terminal can be associated with a process group. The foreground process group is the group within a session that currently has access to the controlling terminal.

There can also be background process groups. These are process groups that do not currently have access to the sessions controlling terminal. Since there is only one controlling terminal per session, there can only be one foreground process groups. The other process groups can't perform terminal I/O.

Processes are placed into process groups using the setpgid() function. Process groups are named by the PID of the group leader. The group leader is the first process to create a group -- it's PID becomes the GID. The group leader can die and the group can remain.

A group becomes the forground group using the tcsetpgrp() call. This call makes the specified group the foreground group. It can affect itself or any of its children.

If a process forms a new session by calling setsid(), it becomes both a session leader and a group leader. For a new session to interact with a terminal, it must allocate a new one -- you won't need to create a new session or alocate a terminal.

But you will have to create process groups, and manipulate the foreground process group to match the current foreground process (which could be your shell).

By making the right group the foreground process group, you are not only ensuring that it has a connection to the temrinal for stdin, stdout, and stderror, but you are also ensuring that every process in the foreground group will receive terminal control signals like SIGTSTP.

Please note that your colleagues form prior semesters didn't follow this real world model. Instead they left all processes i the same group as their shell and masked SIGTSTP when they created children, so that only thier shell recieved it. Their then propogated an equivalent SIGSTOP to the appropriate processes. We will accept this solution this semester -- but I won't promise that it will get full credit. We'd really like you to do it using the real-world tools.


 

We talked though the following example in class. Since then it has been fixed to prevent underlying job-control shells from interfering with your shell (important for t/csh users) by masking SITTOU and SIGTTIN and also to fix a tcsetpgrp() that I left out(sorry!):
 

 Terminal Group Example #1
 
We also talked through this example (ditto for revisions):
 Terminal Group Example #2