Research Report

Computing Workshop/Human Computer Interaction Institute

Carnegie Mellon University

Mary I. Hart,* Albert Corbett** and Jill Lehman**

Dec. 28, 2001

*Human-Computer Interaction Institute and Center for Innovation in Learning, Carnegie Mellon University. Email: mh5r@andrew.cmu.edu

**Human-Computer Interaction Institute, Carnegie Mellon University. Email: corbett@andrew.cmu.edu

***Computer Science Department, Carnegie Mellon University (adjunct), and KidAccess, Inc. Email: jill@kidaccess.com

Abstract: Five high school students with ASD participating in the Excel/Autism study at CMU were able to demonstrate mastery of a set of Excel topics. The Excel curriculum covered approximately the same topics as are covered in the Excel portion of Computer Business Applications, a class for regular education students at Fox Chapel Area High School, a high school in suburban Pittsburgh. The students with ASD were provided with one-on-one tutoring support. Two of the five ASD participants self-initiated activities and engaged in generative thinking to a substantial degree over the course of the eight instructional sessions for which data was recorded. Two others demonstrated lesser amounts of this behavior, and one participant did not demonstrate any. The ASD experimental participants, as compared to a treatment group of three students with ASD who did not receive instruction in Excel, demonstrated improvement in a multi-step planning task which was significant.

INTRODUCTION

Autism was first identified by Kanner (1943). It has since been described by Wing and Gould (1979) in terms of a primary “triad” of differences commonly observed, which include decreased ability to interact in a “normal" way with others in social settings, delay in language acquisition and ongoing difficulties in communication, and impairment in imaginative abilities. Individuals with autism are also likely to display a low tolerance for change, interest in objects rather than in people, and repetitive behaviors. “Islets” of higher functioning may be observed in the individuals. The great majority of individuals with ASD (Autism Spectrum Disorder) suffer from mental retardation (Frith, 1992), based upon standardized tests of IQ. Educational expectations are thus likely to be lower for many individuals with autism than they are for the neurotypical student. Many secondary school students with autism are identified as “Life Skills”students in school, where the emphasis in their schooling will not be on higher-level academics or preparation for further schooling.

Researchers since Kanner have concentrated on particular areas of functional difference in attempting to provide an account of autism. Baron-Cohen, et al (1994) look at the Theory of Mind (ToM), or the ability to make inferences about other’s mental states. There are various levels of ToM, which is thought to be developmental in nature, from the ability to point something out to another individual, through envisioning another’s state of mind, and even the understanding that someone else may hold a mistaken idea. The range of ToM abilities displayed by individuals with autism is thought to be lower than that displayed by neurotypical individuals. ToM provides a possible account for the social difficulties experienced by many individuals with autism.

An account which concentrates on a different aspect of autistic difference is one based on executive function (Griffin, Pennington, Wehner, and Rogers

, 1999)]. Some of the observed phenomena which might be accounted for in terms of executive function are planning ability and ability to inhibit actions, both of which are problematic for individuals with autism. The “weak central coherence” account (Frith, 1992) addresses the autistic individual’s tendency to see component parts rather than wholes, possibly accounting for superior skills on some visually-presented tasks. Recent work by Minshew and Goldstein (1998) provides a more inclusive account that explains a variety of observed differences in terms of capacity for complex information processing. A variety of activities, from planning to envisioning another’s state of mind, require a heavy load of cognitive processing, and the individual with autism is seen as having a lesser capacity in this area, leading to more difficulty in performing these “computationally heavy” activities.

These accounts are compatible with several observations which are commonly made of individuals with autism. The first of these observations is that individuals with autism are likely to be primarily visual thinkers. The verbal or written skills of autistic individuals may be less well developed, although there is tremendous variability in this area. This combination of visual thinking with the possibility of less success in verbal thinking and verbal or written communication has some implications for educators of students with autism. One implication is that instruction should take into account a visual learning style. Another is that traditional assessments, which may be oriented more toward verbal than visual thinking, may fall short of accurately assessing the capabilities of individuals with autism. Using the wrong modality for asking questions may result in underestimating the functional capability of students with autism. To the extent that this happens, students with autism may have their educational opportunities inappropriately limited based upon standardized test scores.

A second observation is that individuals with autism have greater difficulty with tasks requiring planning. Different accounts might focus on the cognitive load involved in planning, the factor of inhibition as a necessary component, or the imaginative capacity required to look ahead or envision a different state of affairs, but the various accounts are likely to be in agreement that tasks involving multi-step planning are more difficult for the individual with autism to carry out optimally.

Related to the second observation, a third point, which is specifically noted in much of research on autism, is a lack of imaginative capacity or ability to do imaginative thinking. This is seen to be at work in ToM, manifesting itself as the lack of ability to envision, depending upon the level, another mind, the thoughts of the other mind, or the mistaken thoughts of the other mind. Pretend play is posited to be less likely in the child with autism, as is the generative thinking in areas such as mathematics, where neurotypical students are observed seeking alternative solutions to problems already solved, as part of the normal developmental process (Seigler, 1996).

A belief that generative thinking is unlikely to be demonstrated by individuals with autism may become a self-fulfilling prophecy in the classroom if the prevailing pedagogy suggests that rote learning is what is within the capabilities of these students. Appropriately adapted instruction, taking into account characteristics such as visual learning style and the notion of cognitive load, may enable individuals with autism to demonstrate skills not demonstrated in other contexts.

The current study is motivated in part by observations from instructional sessions in which individuals with autism received instruction in Excel. Over the course of the instructional sessions, individuals with autism acted in ways which were not consistent with what the literature on autism had led us to expect. One student in particular, JB, regularly exhibited these behaviors, which included attempting to find a novel and alternative solution for a task for which a solution was already known; envisioning and attempting a “shortcut”; verbalizing, “what if I do this?”; and trying something known to be incorrect in order to see “what kind of error message I will get”. Envisioning a possible state of affairs and generating a novel solution strategy appeared to contradict the categorization of the autistic individual which posits less likelihood of imaginative thinking or counterfactual thinking. Observations with other students included instances of a variety of mathematics-related skills being demonstrated for the first time in the context of using Excel.

We speculate that Excel (and spreadsheet applications in general) may provide optimal scaffolding for individuals with autism, thus enabling the demonstration of skills that would be found more difficult in other environments. A specific property of Excel that may provide such scaffolding is the graphic/visual nature of the application. The presentation of information in a spreadsheet format not only appears to be appropriate for visual learners, but may also lessen the cognitive load of the individual, providing, in effect, external working memory, and thus freeing up cognitive resources for other processing. An additional factor is that Excel may help with organizing the knowledge at hand, and in providing an ongoing organizational scheme for the user.

We report the results of an experimental study examining individuals with autism learning Excel. High school age participants with autism who would not normally have the opportunity to take a class in this area because of their classification within the education system were assigned to either treatment (Excel instruction) or control conditions. We evaluate the learning of the subject matter by the treatment group, answering first the question of whether these students are capable of learning such material. Additionally, we look for examples of generative thinking over the course of instruction for the treatment group. Participants take a pretest and a posttest in which the three types of problems (rule induction, planning, and mathematics) are intended to elicit the behaviors of interest (generative thinking) and scores are compared to assess transfer to generative thinking tasks.

METHODS

Participants:

The study included nine male students with ASD (Autistic Spectrum Disorder, as diagnosed by a physician). Eight of the nine were classified by their school districts as Life Skills students. These participants attended special schools for part or all of the school day, or the Life Skills classroom in a regular school. The remaining participant was a Learning Support student at a regular school. Participants ranged in age from thirteen to nineteen, and had no prior experience with Excel. Referrers were told that participants should be able to understand and follow instructions and be able to engage in a non-preferred activity for a period of approximately twenty minutes. Participants were required to have used a computer before, but no particular level of experience or competence was required. Referrals to the study were based on these criteria. Personnel who referred participants to the study included therapeutic and counseling personnel at schools, a counselor at the State (PA) Office of Vocational Rehabilitation, and a doctor at the University of Pittsburgh Medical Center.

Design:

Assignment to groups:

All students completed a problem solving pretest and posttest. Pretest scores provided the basis for assignment to treatment and control conditions. Based on numeric scores on the pretest, participants were ranked from highest to lowest score and divided into three groups. From each group, one individual was randomly assigned to the control condition, and the other two were assigned to the treatment condition. This procedure matched controls to treatment participants so that controls were drawn from a distribution of pretest scores.

Structure of study:

Participants in the treatment condition completed a sequence of eight Excel lessons with individual tutoring support, while controls did not receive any instruction. The same curriculum was used for all students, and covered inputting data, editing data, formatting data, doing arithmetic, graphing data, and using controls. In a separate session following the eighth lesson, students completed a posttest.

Procedure:

Administration of pretest and posttest:

The assessments were administered online. They took place in a location familiar to the participant, at home or at school. An individual known to the student (parent, teacher, or counselor) administered the test jointly with the experimenter. Participants were permitted as much time as needed to complete the tests.

Instruction:

Sessions consisted of instruction while participants worked through exercises in Excel with feedback from the instructor, and self-directed activities. Participants received encouragement at each session to make use of items of interest to them as data (for instance, one’s collection of CDs or one’s golf scores), but were provided with data if they did not choose to do so.

Participants who expressed interest in modifying the task at hand to fit their interests or to engage in another task within the Excel context were encouraged to do so. Participants pursued self-directed tasks in Excel until the participants judged the tasks complete or until they had been engaged in the particular task for 1.5 hours, at which time they were asked to save their work and shut down the computer.

Curricular items were covered in the same order by all students. Differences in the speed at which participants worked and in the time spent on self-directed activities resulted in some differences in which session particular topics were covered in. Length of sessions ranged from twenty minutes to slightly over 1.5 hours.

Instructional sessions were videotaped.

Materials

Pretests and posttests:

The tests were presented in an online format using testing software that was developed for the study. The pretest and posttest were identical in structure. Questions were different on the pretest and the posttest, but corresponding questions on the pretest and the posttest were designed to be at a similar level of difficulty.

Rule Induction Test

The first test section consisted of nine questions. An example of this can be seen in Figure 1a. Each question presented the participant with a 3 by 3 grid in which each of the locations, except for the last, contained a graphic. Participants were instructed to choose, from six possibilities, the best graphic for completing the grid. The testing software allowed the participant to try out possible answers by moving them to the blank location. Graphics made use of elements such as shape and number of items presented. Participants had to induce two or three rules, such as “the number of items is increasing going down the column” and “the shape is the same going across the row” in order to select the optimal answer. In the scoring of this section, partial credit was given for answers that were based on some, but not all, of the relevant rules. The trying out of answers, as well as the final answer, were recorded in a log file with time stamps.

Figure 1a. A question from the Rule Induction Test.

Planning Test

In the second section, participants were presented with a graphic of a set of cars and trucks in a grid which was described as a parking lot (Figure 2b). The parking lot had a single exit. Participants were instructed to move their car, which was initially positioned in the center of the lot, to the exit, which was blocked by other vehicles, all of which were positioned horizontally or vertically. Each vehicle could be moved forward or backward by dragging with the mouse, and solution of the puzzle required moving vehicles which were in the way. All moves were logged, providing the number of moves, the number of optimal moves, number of attempts at making illegal moves, and amount of time spent on the puzzle, as well as whether or not it was correctly solved.

Figure 1b. The posttest version of the Planning Test.

Arithmetic Test

The third section of the pretest and posttest presented the participant with four arithmetic questions, two each at two different levels of difficulty. For each difficulty level, there was a question requiring exact arithmetic, and an approximation question. The first question (easy, exact) was a problem that could be solved by counting. The second question (easy, approximation) involved a comparison of two quantities. The third (medium, approximation) and fourth (medium, exact) questions involved the addition of dollar and cent amounts. The third question required that the participant choose the closest answer from a list of possible answers. The fourth question asked the participant to write in the exact answer in a blank space. Results were logged, with timestamps. Posttest questions are presented below as Figures 1c, 1d, 1e, and 1f.

Figure 1c. Question 1 of the Arithmetic Test is at the easy level of difficulty, and is an “exact” rather than approximation question.

Figure 1d. Question 2 of the Arithmetic Test is at the easy level of difficulty, and is an approximation question.

Figure 1e. Question 3 of the Arithmetic Test is at the medium level of difficulty, and is an approximation question.

Figure 1f. Question 4 of the Arithmetic Test is at the medium level of difficulty, and is an exact arithmetic question.

Instructional sessions:

The instructor utilized a syllabus (Table 1) along with individual session-specific lists of topics and exercises. Participants were prompted by the instructor to engage in activities in Excel. Excel worksheets prepared by the instructor were utilized by the participants (Figure 2a and 2b) in conjunction with some topics, and when the student did not provide his own data.

Syllabus:

Session 1: Opening up a new spreadsheet, navigation with arrow keys, tab, and enter key; inputting items into Excel; deleting; saving files.

Session 2: Doing arithmetic; arithmetic operators; placement of the “=” sign; comparison to calculator; switching back and forth between calculator and spreadsheet for typing formulas, all using constants; counting.

Session 3:”Names” of cells; Using cell names rather than constants. Summing columns, rows. The difference between using “+” and using “=SUM”. Labels. Shortcuts. Control sequences.

Session 4: More complicated arithmetic expressions using formulas; copying, cutting, and pasting; column heading; inserting/deleting rows/column; resizing columns.

Session 5: Functions: =SUM, =AVERAGE; Freezing panes. Copying formulas. Copying to multiple locations.

Session 6: =COUNT; @IF; =COUNTIF; formatting. Sorting data

Session 7: Charts: creating a chart; creating a second type of chart; saving a chart; moving/resizing charts/chart components; assigning a title to a chart.

Session 8: Controls: placing command buttons, labels, and textboxes; changing size/location of controls; switching between design and run modes; using VBA code with controls; modifying properties of controls

Table 1. Excel syllabus by session.

Experimenter-provided worksheets and data

Figure 2a shows a worksheet that was provided to all participants during the third session. After having been introduced to the topic of cell names, the student was verbally instructed to fill in the cell names that are missing from the first ten rows and six columns. Figure 2b shows experimenter-provided data that was provided to participants if they did not come up with their own data. The candy data, or other numeric data provided by the student, was initially used in session 3. The participant was verbally prompted to use arithmetic operators to sum up the numeric amounts. The participant was told to use cell names rather than constants in the formula. Later in the session, the participant was instructed in the use of the “=SUM” function, and instructed to add up the numeric amounts using the function. In a subsequent session (Session 7 for most participants), the same data was used for creating a chart.

Figure 2a. Instructor-provided worksheet: participant was prompted to fill in the missing cell names.

Figure 2b. Experimenter-provided data. The participant was asked in Session 3 to add up the numbers, and was prompted to make use of cell names in doing arithmetic with the column of numbers.

RESULTS

One student, who was assigned to the treatment group, took the pretest dropped out of the study without participating in any Excel sessions or the posttest. The five participants remaining in the treatment condition each completed eight sessions and subsequently took a posttest. Control participants took posttests during the approximate same calendar period as the treatment participants’ pretests. Following are results of pretests and posttests, Excel outcomes, and comparisons of pretest and posttest differences for treatment and control groups.

Pretest Assignment:

Pretest and posttest scores are the sum of the points for all questions on the three sections of the test. The original pretest scores were listed in ranked order, as follows, and divided into the three highest scores, the next three scores, and the last three scores. From each group of three, a random assignment was made of two to treatment group, and one to control group. One participant took the pretest and was assigned to treatment, but dropped out of the study before going any further.

10	treatment
11	control
11.2	treatment
11.2	treatment
11.3	control
11.8	treatment
12	control
12.1	dropped out
13.2	treatment

Table 2. Pretest scores and status.

Mastery of Excel topics:

During the course of the Excel sessions, all of the participants in the treatment condition were able to demonstrate some level of mastery of most of the Excel topics as indicated in Table 3 below. Mastery was judged by the experimenter over the course of instruction. The experimenter used a checklist of skills and noted on the checklist whether the participant could demonstrate the use of these skills independently or with one or more prompts.

Excel topics:	Participants:
	T1	T2	T3	T4	T5
Start Excel from Start menu	M	M	M	M	M
Start Excel by double-clicking on an Excel file	M	M	M	M	M
Navigate with arrows	M	M	M	M	M
Navigate with tab key	M	M	M	M	M
Navigate with enter key	M	M	M	M	M
Save file	M	M	M	M	M
Save file in a specific location	M	M	M	M	MP
Find file saved to a specific location	MP	M	MP	MP	MP
Input numeric data	M	M	M	M	M
Input character data	M	M	M	M	M
Tell a difference between the two kinds (aligned)	MP	M	MP	MP	MP
Delete contents of entire cell	M	M	M	M	M
Delete single character or digit	M	M	M	M	M
Copy text using edit/copy	M	M	M	M	M
Copy numeric data using edit/copy	M	M	M	M	M
Copy data using shortcuts	MP	M	M	M	MP
Correctly type arithmetic equation using constants	M	M	M	M	M
Add constants	M	M	M	M	M
Subtract constants	M	M	M	M	M
Multiply constants	M	M	M	M	M
Divide constants	M	M	M	M	M
Identify cells names	M	M	M	M	M
Fill in cell names for specified locations	M	M	M	M	M
Find a cell by name	MP	M	M	M	MP
Sum columns and rows using cell names and "+"	M	M	M	M	M
Average column or row using cell names and "+", "/"	MP	MP	MP	M	MP
Use =SUM function	M	M	MP	M	MP
Use =AVERAGE function	MP	M	MP	MP	MP
Copy formulas w. cell names to appropriate locations for use	M	M	M	M	MP
Inserting rows/columns	N	M	N	N	N
Resizing columns	M	M	M	M	M
Column headings	M	M	MP	MP	MP
Freezing panes	MP	M	MP	MP	MP
Changing fonts for text data	M	M	MP	M	MP
Using =COUNT	M	M	M	M	MP
Using =COUNTIF	M	M	M	M	MP
Sorting	MP	M	MP	M	MP
Creating a column chart with a single data range	M	M	M	M	M
Creating a second kind of chart with a single data range	M	M	M	M	M
Saving charts	MP	M	MP	MP	MP
Moving/resizing charts	MP	M	MP	MP	MP
Moving/resizing components of charts	MP	M	MP	MP	MP
Adding a chart title	MP	M	MP	MP	MP
Modifying the chart legend	MP	M	MP	MP	MP
Modifying data labels	MP	M