Ralph Morelli
Department of Engineering and Computer Science
Trinity College
Hartford, CT 06106
203-297-2220
ram@troy.trincoll.edu
In Proceedings of the Sixth Florida AI Research Symposium, pp. 315-319, April, 1993.
Abstract
This paper describes Soar/ITS, a prototype intelligent tutoring architecture based on Soar. Soar is a computational paradigm capable of modeling human cognition at a variety of levels. Three novel features of Soar/ITS are described: (1) its ability to represent task knowledge in a flexible and open-ended manner; (2) its ability to represent perceptual and motor knowledge in a cognitively plausible way; and (3), its ability to learn new representations of both the task and the student during the tutoring process. It is shown how these features address some of the outstanding problems in current ITS research.
Introduction
Despite their promise, Intelligent Tutoring Systems (ITSs) have largely failed to make a major impact in the classroom. One of the main reasons for their failure is that most ITS architectures are not based on a suitable cognitive theoretical foundation. Instead they are based on ad-hoc mechanisms and heuristics derived from AI research. The exception is Anderson's work, which is based on the ACT* theory of cognition (Anderson, 1983). It is probably not coincidental that Anderson's group is one of the few within the ITS community to report successful deployment of ITSs, most recently in the Pittsburgh public schools (Anderson, 1992).
Three limitations in current ITS design are: (1) the incompleteness of hand-crafted student models, (2) the ad-hoc representation and processing of theory-laden diagrams and other visual data, and (3) the lack of flexible approaches for representing different granularities of task knowledge. The remaining sections describe each of these limitations and then show briefly how the Soar/ITS architecture purports to overcome them.
Overview of Soar/ITS
Soar is a unified theory of cognition based on the Problem Space Computational Model (PSCM). Its most important hypotheses about human cognition are that all cognitive activities occur through problem-solving and that learning occurs automatically during all activities (Newell, 1990). According to the PSCM, tasks are formulated in terms of problem spaces in which operators are repeatedly selected and applied to the current state until a goal state is achieved. The knowledge that governs problem-solving is represented in production rules stored in long-term memory. A production rule is a condition-action pair of the form "IF C THEN A". When the conditions match the current state of working memory, the production "fires" thereby adding the contents of its actions to working memory. Processing continues as long as the knowledge stored in long term memory can be applied to the contents of working memory. When Soar lacks sufficient knowledge for processing to continue, an impasse occurs, in response to which Soar automatically sets up a subgoal and a problem space to resolve the impasse, thus allowing processing to continue in the original space. This is known as universal subgoaling. When an impasse is resolved, Soar's learning mechanism automatically creates a chunk that represents the knowledge of how to avoid similar impasses during future problem solving and adds this new knowledge to long-term memory.
The Soar/ITS architecture applies Soar's universal subgoaling and chunking mechanisms to problems of student modeling and knowledge representation. As we show below, universal subgoaling provides an arbitrarily rich collection of representational idioms through which knowledge can be communicated. Tasks are represented as a hierarchy of subgoals that can be decomposed to arbitrary depth, depending on the types of skills one is attempting to model (and tutor). This is especially important for designing ITSs that are capable of interacting intelligently with students who have a wide range conceptual models of the task domain. The chunking mechanism is used to acquire new knowledge about the task domain, including perceptual and motor knowledge, and to model the student's knowledge of the domain.
A Problem-Space Description of Soar/ITS
Figure ## provides an overview of Soar/ITS's problem space hierarchy. Soar/ITS can be viewed as a combination of three major architectural components: a perceptual & cognitive architecture, a tutoring and a task domain. In this case the task domain is a very small portion of discrete electrostatics.
Extensible Student Models
Most tutoring systems employ some form of hand-crafted student model augmented with rules that model incorrect behavior (Anderson, 1984). The correct rules represent the knowledge of an "ideal" student whereas the "mal-rules" represent some of the more common observable errors. By comparing the student's problem solving performance against what the ideal student would do and against the set of mal-rules, the tutor decides when to intervene. While this approach to student modeling has proved to be effective in formal domains such as programming and geometry (Corbett and Anderson, 1991), construction of hand-crafted ideal models and mal-rules is labor intensive and tends to be somewhat ad-hoc. Exclusive reliance on them tends to limit the applicability of ITSs to domains that are easily formalizable.
Rather than rely exclusively on a hand-crafted model, Soar/ITS is designed to extend and refine its built-in models to account for unexpected, but potentially correct, student behavior. It is also designed to learn new mal-rules that prevent the student from reaching a correct solution. Together these capabilities provide a more flexible approach to diagnosis and student modeling, an approach that is potentially much better suited to highly visual, experimental natural science domains such as physics.
For example, in Soar/ITS's basic physics domain, students play electric field hockey (EFH), a game designed to improve their understanding of basic electrostatics and mechanics. In the EFH microworld, the student places electric charges on the screen in a such a way that their combined forces will cause a unit-charge particle (the puck) to move around a given set of obstacles and into the goal (Figure ## -- Electric Field Hockey). The trajetory of the puck, which is determined by the microworld's internal physical model, is shown when then the GO button is clicked on, with the spacing of the dots in the trajetory providing a static represention of the puck's velocity.
For such a problem, it is impossible to delineate all correct moves, and there are innumerable configurations that could lead to a correct solution. Moreover, any attempt to construct an ideal model for such a problem would be necessarily ad-hoc. Solving this problem is largely a matter of trial-and-error: place some charges in the field, observe their effects and keep trying until you converge on a successful array of charges. Therefore, if the tutor fails to detect a match between the student's action and what it would do, this does not necessarily imply that the student has made an error. It may indicate, instead, the system's ignorance of the student's strategy.
In such cases, it might be counterproductive to constrain the student's learning to just those solutions represented in the built-in model. In might be better to allow the student to proceed down the unknown path which, if correct, can be learned by the tutor as a new way to solve the problem, and, if incorrect, can be learned by the tutor as a new unsuccessful strategy. The current Soar/ITS prototype is able to track the student's actions and recognize when they don't match its expectations, and our present work is focused developing a more sophisticated diagnostic and remedial response.
Representation of Visual Displays
Theory-based representations (e.g., motion and force in physics) have been found to be an important component of expert problem-solving (Larkin and Simon, 1987). As a result of this insight, such representations have become a prominent feature of computer-based microworlds. The hypothesis behind theory-based representations is that experts envision an internal mental model of the problem situation, and the point of using such representations in microworlds and ITSs is to move the student towards acquiring such models. A number of these studies have reported successful improvements in learning as a result of relying on theory-based representations.
One of the main design features of Soar/ITS is to provide a general cognitive framework for representing highly interactive visual environments. (Figure ## -- Total Cognitive Architecture). Such a framework should support the perception and comprehension of visual displays of arbitrary complexity and the use of this knowledge in the tutoring process. In Soar/ITS, perception of the interactive instructional environment is based on the Model Human Processor, according to which independent perceptual, motor and cognitive processors communicate via a shared memory (Card, Moran and Newell, 1983). This organization provides the system with a general architecture for focusing attention, comprehending visual input, and performing intentional motor actions.
For example, the current Soar/ITS prototype has successfully learned to attend to objects in a physics microworld, comprehend their significance with respect to solving a given problem, and decide upon and carry out appropriate motor actions such as drawing vectors and presenting text. It is equipped with sufficient electrostatics knowledge to solve simple qualitative physics problems. For example, when given the task of showing the forces acting on two charged particles on the display, it can correctly draw arrows that represent these forces. The primitive motor actions it uses to accomplish this task are at the level of mouse manipulations -- i.e., move-the-mouse-to-X-Y, click-the-mouse-button. Moreover, it has successfully learned how to build composite actions (e.g., drawing a force arrow) out of these primitive motor actions.
Significantly, once Soar/ITS has learned a composite action, it can use it as a primitive operator in future problem solving. This ability is easily extended and generalized in Soar. Thus, Soar/ITS's grounding in a cognitively-based perceptual and motor model could have signficant implications for ITS design and development. In addition to facilitating the arduous task of building the interface between an ITS and the instructional environment, these perceptual and motor capabilities give Soar/ITS the ability to learn task knowledge from the bottom up -- i.e., by watching an expert solve a problem and then remembering how it was done.
Representation of Task Knowledge
The ability of a tutoring system to adapt to the needs of a student has been widely recognized as a desirable feature. A well-designed tutor should be able to communicate in a idiom that closely matches the student's current knowledge state (Wenger, 1987). Yet even the best systems display a limited capacity for such adaptation. For example, the CMU Lisp tutor represents programming tasks in terms of two problem spaces, a planning space and a Lisp-coding space, which means that it can communicate with the student about both planning and coding decisions (Reiser et al, 1985). In a sense this feature represents the tutor's ability to view its domain at different levels of abstraction. Obviously, the more levels of abstraction the better.
Soar/ITS is able to extend the levels of abstraction to an arbitrary degree, being limited only by the computational costs of the additional levels. For example, the current prototype represents the task of drawing force arrows on an interactive display at the level of discrete mouse operations (e.g., move-the-mouse-to-X-Y, click-the-mouse). Such a representation avoids any attempt to model hand-eye coordination. However, if it were important (pedagogically) to model student hand-eye coordination, the existing problem space hierarchy could easily be so extended, thus yielding an additional level of abstraction with which to communicate knowledge. This ability to switch flexibly from one level of abstraction to another is a direct effect of Soar's universal subgoaling mechanism. It enables Soar/ITS to detect and address student errors at various levels of problem decomposition. Most significantly, it provides these capabilities in a principled way within the context of unified theory of cognition.
Discussion
The cumulative result of the features outlined above is a perception-driven ITS architecture based on the same kinds of cognitive capabilities found in human tutors. While this is an ambitious design, the present work has the advantage of being able to draw upon much previous and on-going work on machine learning and Soar-based cognitive modeling. Soar models have been developed for a wide range of tasks of appreciable complexity (e.g., natural language comprehension, playing Nintendo, individual differences in solving syllogisms).
Current and future work on Soar/ITS is focused on the following tasks: 1) developing effective and general student modeling mechanisms capable of learning both correct and incorrect solution strategies by observing student behavior; 2) developing effective and general decision making mechanisms that allow tutoring to take place at arbitrarily fine-grained levels of abstraction; 3) refining the system's existing perceptual/motor capabilities to support an open-ended interface with a visual instructional environment; and 4) extending the system's physics knowledge to a level that is rich enough to allow adequate testing of the prototype. Classroom testing of the prototype is scheduled for Fall 1993.
References
Anderson, J.R. (1983). The Architecture of Cognition. Harvard University Press, Cambridge, MA.
Anderson, J.R. (1984). Cognitive psychology and intelligent tutoring. Proceedings of the Cognitive Science Society Conference, Boulder, Colorado, pp. 37-43.
Anderson, J.R. (1992). Intelligent tutoring and high school mathematics. In Frasson, C.; Gauthier, G.; and McCalla, G.I. (Eds) Intelligent Tutoring Systems: Second International Conference, ITS '92, Montreal Canada, June 1992 Proceedings. New York: Springer-Verlag, pp. 1-10.
Card, S.K, Moran, T.P and Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ.
Corbett, A.T. and Anderson, J.R. (1990). The effect of feedback control on learning to program with the Lisp Tutor. Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, Cambridge, MA.
Larkin, J.H and Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science 11, 65-69.
Reiser, B.J., Anderson, J.R., and Farrell, R.G. (1985). Dynamic student modeling in an intelligent tutor for LISP programming. Proceedings of the Ninth International Joint Conference on Artificial Intelligence, Los Angeles, 8-14.
Wenger, E. (1987). Artificial Intelligence and Tutoring Systems. Morgan Kaufmann, Los Altos, CA.