Corpus-based approaches to problem solving

Firechief

Duress

ANSIM

pipes

(c) Jose Quesada, 2003

Return to my homepage

The objective of this page is to create a community of modellers using the same datasets and explaining the same phenomena. In computational linguistics, it is common practice to share datasets, and we believe this would be the case also for problem solving. It is easy to find datasets for things like chess, go, etc., but not so easy to find logs of complex, dynamic tasks such as the ones offered in the site.

Having a common set of effects and data would facilitate model comparison. We expect this site to be open to the problem solving community, with new corpora being contributed by researchers around the world. Please contact me If you have any corpora that you would like to upload, or any questions about the ones offered here.

Fierchief-1 Dataset

Task description

Firechief (Omodei & Wearing, 1995) simulates a forest where a fire is spreading. Participants’ task is to extinguish the fire as soon as possible. In order to do so, they can use helicopters and trucks (each one with particular characteristics) that can be controlled by mouse movements and key presses. There are three commands that are used to control the movement and functions of the appliances: (1) Drop water on the current landscape segment; (2) Start a control fire (trucks only) on the current landscape segment; (3) Move an appliance to a specified landscape segment.

Every time a participant performs an action, it is saved in a log file as a row containing action number, command (e.g. drop water or move) or event (e.g., a wind change or a new fire), current performance score, appliance number, appliance type, position, and landscape type. Most of these variables are not continuous, but on a nominal scale, such as type of movement. For more information on the structure of the log files, see Omodei and Wearing (1995).

Where to get the task: contact Mary Omodei (La Trobe University, Australia) at m.omodei @ latrobe.edu.au

A copy of the manual can be obtained here.

There is a newer version of Firechef, called Networked Firechef; it can be obtained from the Complex Decision Research Group at La Trobe. the map can be a lot larger than in the original verion, and up to 4 people can control the system simultaneously; There are no available corpus as yet, however.

Dataset description

To create an LPSA space, data from experiments 1 and 2 described in Quesada et al. (2000) were used as the corpus, plus data from the experiment described in Cañas et al. (2003). The experiments were designed to test hypothesis about cognitive flexibility when facing new, changing conditions after training under constant conditions in the dynamic task Firechief. Thus, the experiments consisted of a long period of environmental constant conditions followed by a short period of variable conditions. The conditions manipulated were wind direction and fire extinguishing efficiency of the appliances. Since here we are interested in a sample of representative behavior when interacting with the system, the actual hypothesis and design of the former experiments are not relevant.

contact person: Jose Quesada (quesadaj @ psych.colorado.edu)

Related publications

Omodei, M. M., & Wearing, A. J. (1995). The Fire Chief microworld generating program: An illustration of computer-simulated microworlds as an experimental paradigm for studying complex decision-making behavior. Behavior Research Methods, Instruments & Computers, 27, 303-316.

Cañas, J. J., Quesada, J. F., Antolí, A., & Fajardo, I. (2003). Cognitive flexibility and adaptability to environmental changes in dynamic complex problem solving tasks. Ergonomics, 46(5), 482-501.

Quesada, J. F., Cañas, J. J., & Antoli, A. (2000). An explanation of human errors based on environmental changes and problem solving strategies. In C. P. Warren (Ed.), ECCE-10: Confronting Reality. Sweden: EACE.

Quesada, J. F., Kintsch, W., & Gomez, E. (2002). A theory of Complex Problem Solving using Latent Semantic Analysis. In W. D. Gray & C. D. Schunn (Eds.), 24th Annual Conference of the Cognitive Science Society (pp. 750-755). Fairfax, VA.: Lawrence Erlbaum Associates, Mahwah, NJ.

Universities using the dataset

University of Granada, Emilio Gomez, Jose J. Canas
University of Colorado, Boulder, Jose Quesada

 

Duress-1 Dataset

Task description

. An example of this kind of tasks is DURESS (DUal REServoir System). DURESS is a thermal-hydraulic process control simulation that was designed to be representative of Industrial process control systems. It consists of two redundant feedwater streams that can be configured to supply either, both or neither of the two reservoirs. The goals of the game is to keep each of the reservoir temperatures (T1 and T2) at a prescribed level (e.g., 40 C and 20 C, respectively), and to satisfy the current mass (water) output demand (5 liters per second and 7 liters per second, respectively). Thanks to the seminar work of Vicente and collaborators (Christoffersen, Hunter, & Vicente, 1996, 1997, 1998), the equivalent of three years of experience with the system DURESS II is available and we used it in our LPSA simulations.

where to get the task: CEL laboratory (Toronto, Canada) m directed by Kim Vicente (vicente @ mie.utoronto.ca)

Dataset description

The dataset contains the performaces of 6 people controlling Duress in different situations for 6 months. The concrete experiment is described in the articles cited. So overall, a sampleof 3 years of performance in this system is available from Vicente.

contact person: vicente@mie.utoronto.ca

Related publications

Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1996). A longitudinal study of the effects of ecological interface design on skill acquisition. Human Factors, 38, 523-541.

Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1997). A longitudinal study of the effects of ecological interface design on fault management performance. International Journal of Cognitive Ergonomics, 1, 1-24.

Christoffersen, K., Hunter, C. N., & Vicente, K. J. (1998). A longitudinal study of the impact of ecological interface design on deep knowledge. International Journal of human-Computer Studies, 48(6), 729-762.

Vicente, K. J. (1999). Cognitive Work Analysis. Mahwah, NY: LEA.

Quesada, J. F., Kintsch, W., & Gomez, E. (2003). Latent Problem Solving Analysis as an explanation of expertise effects in a complex, dynamic task, Proceedings of the 25th Annual Conference of the Cognitive Science Society. Boston, MA.

Universities using the dataset

University of Colorado, Boulder, Jose Quesada
Univerity of Toronto, Kim Vicente

ANSIM dataset

Task description

The Research Flight Simulator form the National aerospace laboratory (NLR) consists of a modern transport aircraft cockpit on a 4 degrees of freedom motion system. The pilots view a computer generated outside world through two displays wide angle collimated displays mounted in front of the cockpit. We used a B747 simulation landining in the San Francisco Airport.

 

Dataset description

The dataset consists of 400 landings performed by professional pilots, with grades given by two human experts (instructors). The data were collected at the NLR (national simulation facility) in Amsterdam (Neederlands).

The dataset was generated using two experts that graded landings in different wind conditions. The experts used the following criteria: (1) Flare initiation height: The flare has to be initiated at a particular height; this height is not rigid as lower flares can be compensated by a higher pitch rate for example. Three levels (too high, correct, and too slow) were used. (2) Thrust Reduction: The reduction should be progressive, and it has to be started in a particular moment in time. It was judged using three levels: (too fast, correct, and too slow). (3) Pitch rate. The pitch was evaluated using five discrete levels, from too high to too low. (4) Overall landing score. This is a general rating that expressed how good the landing was, from one to five.

The variables logged in the simulator were the following:

 

To visualize the 400 landings, and see the human gradings, there is an online landing viewer. You can get two landings displayed simultaneously.

Related publications

Quesada, J. F., Kintsch, W., & Gomez, E. (2003). Automatic Landing Technique Assessment using Latent Problem Solving Analysis, 25th Annual Conference of the Cognitive Science Society.

Universities using the dataset

University of New Mexico, Peder Johnson
University of Granada, Emilio Gomez
University of Colorado, Boulder, Jose Quesada

Pipes-1 Dataset

Task description

A pipes participant plays the role of a water purification plant operator. In this plant, water enters different purification tanks, which the operator then activates while attempting to meet a set of deadlines.
the figure shows a screenshot of pipes, which contains 22 tanks with 2 pumps per tank; a set of deadlines is visible on the right of the screen. The tanks are connected by pipes that indicate the path the water traverses to reach the deadline. The set of connected tanks is called a chain, and the length of the chain dictates the amount of time required to pump the water out of the system. The operator must remove the water that enters the various tanks at different times as the simulation advances. The operator’s goal is to distribute all the water within the allotted amount of time by activating and deactivating the pumps assigned to each tank. Only 5 pumps can be activated at any one time, and after a pump is used there is a delay of 10 simulation minutes (cleaning time) before this pump (or a different one) can be re-activated. The simulation time begins at 2 o’clock and finishes at 10 o’clock when the final deadline expires.

Where to get the program: contact Cleotilde Gonzalez (conzalez @ cyrus.andrew.cmu.edu)

Dataset Description

No dataset is available as yet (coming soon).

Relevant publications

Gonzalez, C., Lerch, F. J., & Lebiere, C. (2003). Instance-based learning in dynamic decision making. Cognitive Science, 27, 591-635.

Gonzalez, C., & Quesada, J. F. (in press). Learning in a Dynamic Decision Making Task: The Recognition Process. Computational and Mathematical Organization Theory.

Universities using the dataset

Carnegie Mellon University, Cleotilde Gonzalez
University of Colorado, Boulder, Jose Quesada

Monday, December 15, 2003 0:29 AM