ADFL Bulletin
22, no. 3 (Spring 1991): 39-43
To the Editor Search

Table of Contents
Previous Article Next Article
Works Cited

Assessing the Gain in Oral Proficiency from Summer Foreign Study


Margo Milleret


THE effect of study abroad on foreign language learners is of increasing interest now that the profession is attending to the development of communicative skills and the evaluation of proficiency. Several recent studies that analyze the oral proficiency of students in foreign language programs in the United States have uncovered, as John Carroll did in his landmark 1967 study, the positive relation between foreign study and foreign language learning. Sally Sieloff Magnan noted that students of French with foreign-study experience scored higher on oral-proficiency testing than their counterparts did (433), while Tomás L. Graman found that students of Spanish with language experience outside the classroom, either abroad or in the home, could move easily into upper-division courses, while their counterparts without that experience could not (930­31).

Although students have been studying abroad in large numbers since the 1960s, the assessment of student performance has not kept pace with that activity. The major limitation to such evaluation has been the lack of tools for testing readiness for, and gains from, the study-abroad experience. The traditional tools used for such measurements have been achievement tests, such as the cooperative tests of the College Entrance Examination Board (CEEB) and the Modern Language Association. These tests measure language usage, not language use, that is, they tell what the student knows about the operation of the language but not how the student performs when producing the language. Although these tools have the advantage of standardization, they have been criticized as outdated in their use of the target language (the MLA test is from 1961) and in their pedagogical approach. Dan Davidson complains that the CEEB test “tells us nothing about what students can actually do with the language” (222).

The best way to assess the language skills of students participating in foreign study may be the ACTFL oral-proficiency interview. The ACTFL OPI measures global language skills, as opposed to language achievement, and conforms more closely to communicative pedagogy. Several researchers have already used the OPI in foreign-study programs. Roma Hoff and Judith Liskin-Gasparro used it to measure improvement in students participating in one-year study-abroad programs in Spain. Barbara Freed used achievement tests and the ACTFL OPI with summer-study students in France to try to determine whether out-of-class contact had an effect on gains.

One argument for the use of the OPI in this area is that the measurement of oral skills can reflect gain in all skill areas. Foreign-study directors have often noted that students make the greatest gain in oral skills as a result of foreign study (Gordon 31). Foreign-study programs have traditionally used pre- and posttesting of oral skills to help in assigning credit or a grade (Eustis 103). Davidson discovered that Russian oral interviews had good correlation with other skill exams, such as the American Council of Teachers of Russian qualifying examination for study abroad (228).

Yet even as the OPI becomes a more widely used testing tool, several concerns have been raised regarding its use in foreign-study programs. Roberto Veguez discovered that the increased fluency of study-abroad returnees from Spain was causing raters to give false superior ratings (5­8). He found that students were especially creative at avoiding the Spanish subjunctive, which is an important marker for determining superior-level placement. Veguez's study outlines how th Middlebury program used his research to change the curriculum of students studying abroad. His findings, however, suggest that OPI raters may need to be sensitized to the language use of foreign-study returnees.

More serious are the questions raised by Freed about the sensitivity of the OPI to short-term gains. Her study of students in a summer program in France revealed that only the lowest-ranking students made progress. Since the OPI is weighted so that gains at the bottom are easier to make than those at the top, Freed wondered whether more advanced students were not making real gains or whether the OPI did not measure those gains.

The subject of measuring gains from foreign study with the ACTFL OPI is a branch of the ongoing debate about proficiency testing that has been communicated from the pages and the podiums of the profession since the ACTFL guidelines were introduced. ( ADFL Bulletins 18.1, 20.2, and 21.2 provide a history and bibliography of the debate.) There is general agreement that the ACTFL OPI can determine a level of professional ability and measure gain after significant exposure to the target language. Some have suggested that its descriptions of levels facilitate its use as a diagnostic test. But can and should such a test measure gains from short-term contexts like summer foreign study? Liskin-Gasparro is currently involved in a project that uses the OPI to compare the proficiency of students in summer immersion programs in Spanish and Russian at Middlebury College with that of students attending regular language classes. Her results, along with those from our study described below, should help provide some answers to that question.

The Portuguese Speaking Test and Summer Study

Summer foreign study has become a popular option for both students and faculty members at universities because it requires a smaller commitment of time and funds while offering a potential for results similar to that of longer foreign-study programs. Summer-study programs in which students speak only the target language and receive communicative classroom experiences and cultural immersion have reported impressive results. Isidore Fish (37) and Gregory K. Armstrong (367) found that students at the high school level of Hebrew and Spanish made one year's worth of progress during one summer of study abroad.

As director of a summer-study program in northeastern Brazil, I am interested in tailoring our language and culture program to the students' needs, using our on-site facilities in the best possible way, and documenting the students' progress. In the summer of 1989 I worked in conjunction with the Center for Applied Linguistics (CAL) in Washington, DC, to administer the Portuguese Speaking Test (PST) to the participants of our summer-study program. Our purpose was to validate the Portuguese Speaking Test as a placement test for the study-abroad program and as a measurement of gain. 1

The PST is a prerecorded oral-proficiency interview that follows the format of the ACTFL oral-proficiency interview intended for students at proficiency levels from low intermediate to superior on the ACTFL scale or 1 to 3+ on the Interagency Language Roundtable (ILR) scale. Examinees with abilities that exceed those of the ACTFL superior rating can earn high superior (which corresponds to a 3+ to 5 on the ILR) on the PST. Charles Stansfield and a team of researchers at CAL developed and tested the PST for reliability and concurrent validity with the face-to-face ACTFL interview in 1988. CAL distributes the test and arranges for ACTFL-certified raters to score it. The PST can be administered at any campus in a language laboratory by untrained individuals and is available in several formats so that it can be given as a pre- and posttest.

The PST consists of six sections. The first section corresponds to the warm-up of a direct interview and includes questions in Portuguese about personal subjects such as the family, hobbies, and school. The second, third, and fourth sections require students to respond to information shown in pictures, in the manual. For instance, examinees may be asked to give directions from one point on the map to another, to describe the objects in a picture, and to sequence events. Sections 5 and 6 require examinees to use their Portuguese in more demanding ways, such as hypothesizing or speaking in culturally defined situations. Students may be asked to talk about the advantages and disadvantages of using public transportation and to convince a dealer in an open-air market that the price of an item is too high. Instructions for sections 2 through 6 are in English, and students are given 15 to 30 seconds to prepare their answers and from 45 seconds to 1 minute and 45 seconds to respond (Stansfield et al. 642­43).

Eleven of the fifteen students from four universities that participated in the summer 1989 study program in Brazil took the PST as a pre- and posttest. Those in our date sample had completed from two to six semesters of Portuguese language before leaving for Brazil. One member of the group was a native speaker of Spanish, one was bilingual in Spanish and English, and all the rest were native speakers of English. All but one of the participants had studied another foreign language in high school. Three of the students had lived abroad, seven had traveled in either Europe or Latin America, and five had never been outside the United States. One of the students had lived in Brazil for a short time as a child, but none of the students had studied in Brazil before.

Data for the project were collected from measurements of oral proficiency (the PST and an oral face-to-face structured interview conducted by the instructors when the students arrived in Brazil) and from questionnaires completed by the instructors and the students. The scores on the PST were compared with the interview scores given by the instructors, and both scores were compared with instructor and student opinions about progress made during the program.

The results, showed that the PST functioned well as a placement tool. The PST scores gave insights into the proficiency levels of students that the applications and “seat time” could not provide. The PST scores agreed with the divisions into class levels made by the instructors in Brazil except in two cases. In these two cases student questionnaires suggested that the two individuals were not happy with the progress they had made. The PST scores confirmed that those students had been misplaced according to their skill levels.

Because the PST-level descriptions give more information about students' language skills than is usually available to program directors, the results can be used to tailor the curriculum, collect materials, and arrange for teaching staff for each class before the students leave for Brazil. The information gathered can answer questions such as how broad a range of skill levels we have and how we can divide students into classes, what emphasis each class should be given, and what kinds of situations are most difficult for students and how we can address them. Since the program consists of about five weeks of classes, planning with the PST can help ensure that class time in Brazil is used to the fullest.

The PST also measured statistically significant gains made by the students as a group. The average gain score for the students who took the PST as both pre- and posttests was approximately the difference between mid intermediate and high intermediate on the ACTFL scale. One way to appreciate that gain is to compare it with Magnan's findings in her preliminary study of proficiency levels in the French program at the University of Wisconsin. Magnan discovered that after one year of study, students on the average rated low intermediate, after two years mid intermediate, and for minors and majors high intermediate. The mean pretest score for the Brazil summer program was low intermediate, and the mean posttest score was almost high intermediate. On the basis of Magnan's figures, we could say that students in the summer program averaged almost one year's worth of language-proficiency progress. Although Magnan's findings do not translate into other contexts, it is tempting to view the gains of the Brazil summer program as confirming earlier work by Armstrong and Fish in which students made a year's worth of progress during a summer of study abroad.

When we divided our data of gains into class levels, we discovered an unequal distribution of gain. The PST registered a greater gain for those students with a lower PST score than those who were ranked higher. The students in the lower level began the summer with a mean rating of mid novice on the ACTFL scale. By the end of the summer, that group had a mean rating of low intermediate. In contrast, the advanced class began the summer with a mean rating just below mid intermediate and ended with a mean rating just above mid intermediate, a gain that was not statistically significant. 2

The reason the advanced students made so little progress may be that the ACTFL OPI measures improvement over a long period of time and that its rating scale demands greater levels of improvement as one moves up the scale. That is, students who know less can show gains more easily than can students who know more and must make more complex improvements in their skills. It may be that the summer program is long enough to measure gains at the novice level but too short to measure more sophisticated improvements. The problem with this answer is that it flies in the face of a commonly accepted tenet of foreign study: that better-prepared students—that is, those who have more time invested in learning the language—make more and better progress during a foreign-study experience. Our research found that the PST could not really confirm this belief, even though information gathered from student and instructor questionnaires revealed that both the instructors in the program and the advanced students themselves felt they had made more progress than those at the lower level had.

Another explanation may stem from the large number of Spanish speakers in the advanced level. Several students in the advanced class either spoke Spanish as a first language, were bilingual, or had studied Spanish in high school. John B. Jensen's work with Spanish-speaking students of Portuguese (SPs) has led him to observe that Spanish speakers can demonstrate impressive degrees of proficiency in Portuguese without any training at all (119). Jensen's observation may indicate that the enthusiasm of the advanced-level students and their instructors was due more to proficiency in Spanish than to actual command of Portuguese.

The presence of Spanish in the students' backgrounds may have skewed the results of the PST. The meager gains of the advanced students could be due, in part, to artificially high pretest scores of Spanish speakers. Jensen observes that, “considering the very high proportion of common vocabulary between Spanish and Portuguese, it is easy to see why SP's may never qualify as ‘novices’” (122). He hypothesizes that under the current ACTFL guidelines, his Spanish-speaking students would rate low intermediate after only a few weeks in class and would finish the semester at high intermediate. While it appears that Spanish does prepare students to be more proficient in Portuguese, the real problem is, as Jensen points out, that under the existing proficiency guidelines candidates may be able to demonstrate Portuguese proficiency without having learned the language (127). If our data sample had included more students, especially advanced students with no previous exposure to Spanish, it might have been possible to determine what degree of influence Spanish had on the PST scores.

Our validation study confirmed that using the PST scores adds precision and speed to the placement of students and the development of curriculum in the Brazil summer program. The prerecorded format, which offers mobility and reliability, is especially helpful for a program that accepts students from many different foreign language environments throughout the United States.

However, our results regarding the ability of the PST to measure gain were not as conclusive. Our study would have benefited from greater numbers or from a longitudinal approach. Nonetheless, when the students were measured as a group, the gains were significant and in agreement with previous findings of foreign-study programs in Mexico and Israel. But when the gains were analyzed by class levels, the more advanced students showed no gains.

Is the PST and its ACTFL-OPI model the wrong test, or do its measurements just need refinement to function better in a foreign-study context? Liskin-Gasparro suggests that raters may need to develop greater sensitivity to individual improvements when they rate short-term foreign-study interviews. The study by Veguez, mentioned earlier, confirms this view. But Jensen proposes that speakers of Spanish who study Portuguese may require a different kind of proficiency test, one that would measure how much communicative ability comes from mastery of Spanish and how much is based on command of Portuguese (127­28). This distinction does not, of course, matter to the proficiency tester qua tester, who does not care how interviewees have come by their proficiency. It does, however, matter greatly to the teacher.

One possible measure to aid the teacher has been suggested by Stephen L. Graham, who worked with the Foreign Service Institute at the Language Training Mission in Utah. He proposes developing a diagnostic sheet on which to record the student's performance in detail by strengths and weaknesses (31­37). The sheet would pinpoint common grammar, vocabulary, pronunciation, and fluency areas. After taking the test, the students would have a language profile of their skills that they and the instructors could use to inform the curriculum. At the end of the summer the students would take the test again, and the results would show what progress was made. A detailed diagnostic might be specific enough to detect short-term improvements or, for Spanish-speaking students, to determine a real mastery of Portuguese.

The evaluation of sociolinguistic aspects of communication, such as proxemics, gestures, and other nonverbal expressions, has not yet been addressed by the ACTFL OPI but is even more difficult to determine in the prerecorded format of the PST. It may be advisable to ascertain whether videotaping students as they respond to the questions and situations would provide meaningful nonverbal information that could be evaluated along with the existing or revised descriptions.

Probably the need for other refinements will surface as we continue to use the PST in our Brazil summer-study program. Certainly the need to know about the effect of foreign study on the foreign language performance of our students justifies an ongoing investment in research and development. Our validation study of the PST contributes in a small way to that effort. But the need to find good measures of gains from study abroad must continue not only because it is an issue for foreign language teachers but also because it is of concern to parents and administrators. Foreign study has an increasing role to play in postsecondary education an universities attempt to internationalize their curricula. As we evaluate our programs and the performance of our students, we gain useful information about the importance of foreign study for language learners. But we also demonstrate our credibility and our willingness to be accountable to the academy. 3


The author is Assistant Professor of Portuguese at the University of Tennessee, Knoxville.


Notes


1 The complete project report, “The Validity of the Portuguese Speaking Test for Use in a Summer Study Abroad Program,” by Margo Milleret, Charles Stansfield, and Dorry Mann Kenyon, was submitted in July 1990 to the Center for Assessment Research and Development at the University of Tennessee, the financial sponsor for the project.

2 For data analysis the PST scores were coded in the following manner: 0.2, low novice, 0.5, mid novice; 0.8, high novice; 1.0, low intermediate; 1.5, mid intermediate; 1.8, high intermediate; 2.0, advanced. References in the text to just below or just above levels refer to mathematical averages of level scores.

3 This article is based on a paper presented at the American Association of Teachers of Spanish and Portuguese, 10­14 August, 1990, in Miami, Florida.


Works Cited


Armstrong, Gregory K. “Language Study Abroad for High School Students: Indiana's Program for Proficiency and Recruitment.” Foreign Language Annals 15.5 (1992): 365­70.

Caroll, John. “Foreign Language Proficiency Levels Attained by Language Majors near Graduation from College.” Foreign Language Annals 1.1 (1967): 131­51.

Davidson, Dan. “Assessing Language Proficiency Levels of American Participants in Russian Language Programs in the Soviet Union.” Russian Language Journal 36.125 (1982): 221­32.

Eustis, Christopher. “Is Summer Study Abroad Worth It? For the Students? For the Director?” Study Abroad in the Eighties. Ed. Deborah J. Hill. Columbus: Renaissance, 1985. 92­104.

Fish, Isidore. “Foreign Language Learning and Short Term Study Abroad.” American Foreign Language Teacher 4.4 (1974): 36­38.

Freed, Barbara. “Language Learning in a Study Abroad Context: The Effects of Interactive and Non-interactive Out-of-Class Contact on Grammatical Achievement and Oral Proficiency.” Proceedings of the Georgetown University Roundtable on Language and Linguistics [Spring 1990]. Forthcoming.

Graham, Stephen L. “Using the FSI Interview as a Diagnostic Evaluation Instrument.” Direct Testing of Speaking Proficiency: Theory and Application. Ed. John L.D. Clark, Princeton: ETS, 1978. 31­37.

Graman, Tomás L. “The Gap between Lower- and Upper-Division Spanish Courses: A Barrier to Coming Up through the Ranks.” Hispania 70.4 (1987): 929­35.

Gordon, Kenneth A. “Study Abroad and Language Proficiency: Methods of Developing and Evaluating Student Skills.” ADFL Bulletin 19.1 (1987): 30­32. [Show Article]

Hoff, Roma. “The Miracle Happens ‘Over There’: Tracing the Development of Productive Skills of Two Study Abroad Students.” Central States Conference on the Teaching of Foreign Languages. Milwaukee, 1986.

Jensen, John B. “Evaluating Portuguese Performance of Spanish-Speaking Students.” Negotiating for Meaning: Papers on Foreign Language Teaching and Testing. Ed. Dale A. Koike and Antônio R.M. Simões. Austin: U of Texas, 1989. 119­30.

Liskin-Gasparro, Judith. Phone conversation. 2 July 1990.

Magnan, Sally Sieloff. “Assessing Speaking Proficiency in the Undergraduate Curriculum: Data from French.” Foreign Language Annals 19.5 (1986): 429­38.

Stansfield, Charles W., et. al. “The Development and Validation of the Portuguese Speaking Test.” Hispania 73.3 (1990): 641­49.

Veguez, Roberto. “The Oral Proficiency Interview and the Junior Year Abroad: Some Unexpected Results.” Northeast Conference on the Teaching of Foreign Languages. New York, 1984.


© 1991 by the Association of Departments of Foreign Languages. All Rights Reserved.

ADFL Bulletin 22, no. 3 (Spring 1991): 39-43


Table of Contents
Previous Article Next Article
Works Cited