Multimedia pedagogues: interactive systems for teaching and learning

  • Published on

  • View

  • Download



    Beverly Park Woolf University of Massachusetts

    Wendy Hall University of Southampton

    m Multimedia technologies

    inspire active learning by

    letting students participate in

    the instructional process.

    Advancements in these

    technologies may significantly

    improve education.



    udiovisual material can provide valuable aids for teaching sys- tems. However, a system is only useful if the learner remains active and motivated. It is well known that page turning or

    browsing does not ensure effective learning. Students must want to learn and should be involved. They need to be challenged to reason about the material presented. Flashy graphics and simulations are not enough; the experience has to be authentic and relevant to the learners life.

    have advocated active learning multimedia environ- ments to enable students to make authentic choices that impact the envi- ronment. We continue this theme and assert that multimedia, when coupled with other improvements in educational software design, can support effective, quality instruction. We propose an instruction model that moves beyond the Tyranny of the B u t t ~ n ~ and uses intelligent sim- ulation, dynamic links (on-line generation of links based on student behavior), and multimedia composition and creation. Key elements of an active learning environment include


    parameters of a database or simulation that the user can change; system knowledge of the components in the environment (for exam- ple, a model of the situation through mathematical formulas, scripts, or semantic representation); and system reasoning about user actions, and immediate response.

    Although technological barriers exist that prevent the widespread devel- opment of effective systems, we discuss ways to break them down. Finally, we suggest that truly interactive systems can evolve into multimedia learn- ing environments, orpedagogues, and facilitate a shift in teaching and learn- ing. We present several case studies to illustrate these advantages.

    CASE STUDIES A variety of multimedia technologies are being used to achieve rich

    interactivity with students (see Background sidebar). Examples include simulations, the Cardiac Tutor, systems for composition and collabora- tion, and explanatory systems.

    Multimedia simulations Simulation places multimedia in a project-based context and engages

    learners in situations where they can solve relevant problems. Simulations can take many forms within two main categories: those based on scenar- ios and those based on knowledge.

    Scenario-based multimedia simulations can use video, graphics, sound, or voice to engage the user in typical situations. Examples include systems that teach cardiopulmonary resuscitation, anesthesiology, or strategies for combating trauma.4 However, many of these simulations provide only a few paths through a problem, no knowledge of the problem beyond those set points, and no ability to adapt the presentation to perceived user needs or individual knowledge. In other words, these simulations have no

    0018-9162/95/$4.00 1995 IEEE

  • domain knowledge about the topics they present and can- not respond to student questions or explain the informa- tion beyond the presentations.

    On the other hand, knowledge-based simulations con- tain a model of the situation and may use a planner, plan recognizer, or user model to make assumptions about the situation and the users state of knowledge and learning needs. These simulations require complex representations and sophisticated control structures to respond flexibly to the user.

    Scenario-based simulations often classify student actions as correct or incorrect, and provide little advice. However, knowledge-based systems can respond to idiosyncratic stu- dent activity-for example, suggesting that a students action is out of order or too late, compared to an experts action. Dynamic assessment, or on-line calculations and rea- soning about user actions, can provide real-time compar- isons between student and expert actions, in the context of the simulation. Because the systems recommendations reflect the current context, they are relevant and robust. Knowledge-based systems can reason about tutorial goals and identify the ones toward which the simulation should be directed.

    The Cardiac Tutor Chris Eliot, at the University of Massachusetts Computer

    Science Department, has built a knowledge-based simula-

    tion for teaching about cardiac resuscitation, based on extensive domain knowledge about cardiac resuscitation procedure^.^ Knowledge was encoded into the system through an iterative process involving extensive expert inputs. A formative system evaluation (measured by physician-administered final exams) with two classes of fourth-year medical students suggested that working with the Cardiac Tutor was equivalent to working one on one with an emergency room physician.

    The tutor presents a graphical view of an emergency room patient. The goal is clear: Save the patient by select- ing the proper procedures on the screen. The tutor pro- vides clues, including

    spoken advice; emergency room sounds; and graphic indications of ECG (electrocardiogram) trace, blood gases, and vital signs.

    The simulation proceeds in physiologically and clinically plausible sequences, accounting for all actions thus far applied. Simulation events drive the multimedia and pass information to the tutoring module. Student input is treated as simulation events (see Figure 1).

    In addition to recording, restoring, critiquing, and grad- ing student performance, the tutor offers automated tuto- rial help. It customizes the problems presented to suit

    Background In recent years, the term multimedia has evolved to

    describe the storage and display of audiovisual material as well as text and graphics in computer systems. To enhance their students learning experience, teachers have used mul- tiple media presentations such as slide shows, television broadcasts, and videos, for as long as the technology has been available. As early as the 1960s, experts foresaw the integration of audiovisual material with computer- generated text and graphics, and its potential impact on education.

    One of the earliest documented, interactive, educational videodisc projects was The Puzzle of the Tacoma Narrows Bridge Collapse, which was recently cited as prior art in over- turning Comptons patent on interactive CD-ROM based multimedia. Pioneering work at MlT led to such well-known and -documented projects as Electric Charles (an interactive video trip along the Charles River in Boston), Galatea (a video server) and Project Athena. In the U K , one of the biggest educational interactive videodisc projects was the BBC Domesday Project, undertaken in 1985.

    Digital video technology has been under development for almost 20 years. High-quality video playback is just becom- ing available on PCs and workstations. The large storage requirements for high-quality video are now easily attained from inexpensive CD-ROM drives. Early examples of digital video technologys educational use include the Palenque pro- ject, codeveloped by Bank Street College of Education in New York, and the video compression algorithms that became known as digital video interactive (DVl).* Palenque was essen-

    tially a n educational simulation based on video that let users explore an ancient Mayan site, visit the ra in forest or the Mayans, and examine area maps or glyph writing.

    QuickTime, digital video software for the Apple Macin- tosh, was one of the major driving forces behind the multi- media revolution. QuickTime videos can be cut and pasted into any Macintosh application. This development forced other vendors to deliver video on desktop machines that didnt require special-purpose hardware. The availability of sound and moving video on desktop computers has started a revolution in the way information is viewed. Specifically, people now perceive sound and video as information that can be stored and manipulated, rather than as material delivered from a cinema or television (over which they have no control).

    The technology has moved on. We now have interna- tional standards for S t i l l and moving picture compression (JPEG and MPEG, respectively) and a firm basis on which to develop educational software, but we have a long way to go before we can integrate multimedia systems to the teaching and learning culture.

    References 1. M.E. Hodges and R.M. Sasnett, Multimedia Computing Case

    Studies from MIT Project Athena, Addison-Wesley, Reading, Mass.,1993.

    2. G.D. Ripley, DVI: A Digital Multimedia Technology, Comm. of the ACM, Vol. 32, No. 7, 1988, pp. 81 1-823.


  • Figure 1. The Cardiac Tutor: simulated patient, ECG (electrocardio- gram) traces, and blood labs.

    different previous levels of achievement, and assists the learning process dynamically. It gives positive feedback for good or improved performance, and categorizes and comments on incorrect behavior.

    Early feedback from users has been very positive. In many cases, the system stimulated active student discus- sions and directed review of the textbook material, pro- viding highly productive learning experiences. In addition, students reacted strongly to the tutors very simple negative feedback by actively exploring the problem on their own. During the tutor evaluation, both doctors and students sought to understand the medical text based on the simu- lation. They argued over the exact procedure for cardiac life support and found that the simulation teased apart the required procedure more clearly than the text did.

    Multimedia composition and collaboration Producing a multimedia document requires that a stu-

    dent do more real scholarship and significant research than preparing a text-based presentation. Students may be involved in very different multimedia aspects, including

    creation, producing text, drawings, or digitized pictures; organization, enabling the reader to connect and move items, mark and classify them, and view the same item differently; access, indexing or filtering certain types of items and searching for patterns; and communication, sharing work and ideas.

    Literacy skills for creating and using multimedia com- positions may one day be essential. Several environments allow inexperienced students to create multimedia com- munications-working alone or together through distrib- uted networks. Well-connected multimedia interfaces let students access several media forms for each piece of


    information and move among media presentations (text simulations, process descriptions, or graphics) that refer to a sin- gle topic. In the Live Information interface,6 the student can share and reuse informa- tion, and choose the media to display it. In addition, all needed displays and user pro- cedures are available from a common place (see The Explanation Planner sidebar).

    The Microcosm system7 uses another approach, providing a hypermedia service that supports both explicit and implicit linking. Explicit, or authored, links are maintained in databases, separate from the documents to which these links apply. Because any number of link databases can be operative at a given time, users can main- tain private links and still access other links (such as those authored by teachers or other students). This lets the student view the data in different ways.

    Explicit links define relationships between different multimedia objects. Implicit links are created at runtime by a dynamic process such as a database query, a text-retrieval search, a similarity-match-

    ing algorithm, or an intelligent rule base. The sharing of resources by students and tutors creates economies of scale and permits collaborative work. Microcosm links, both explicit and implicit, are applied to objects (such as a piece of text or part of an image) in any appropriate application rather than fixed to a particular location. Hence, students can access links-for example to gain tutorial help or look something up in a dictionary-when working in a standard desktop environment such as a word processor or spread- sheet. This creates a fully integrated interactive multime- dia environment.

    Multimedia and explanatory systems Several instructional systems incorporate multimedia

    and explanation. They incorporate a model of student questions or activities and generate real-time context- based responses (see The Explanation Planner sidebar). The Microcosm-Hides project at the University of Southampton8 gives students feedback about the rele- vance of documents that may support or counter an argu- ment they are presenting or an analytic task for which they are collecting multimedia (see The Scholar Project at the University of Southampton sidebar on page 78). The Microcosm system suggests additional encoded material and supports search-through information from many dif- ferent media types, including text, graphics, video, and sound archives. The student accesses multimedia docu- ments through Microcosm in the same way, using hyper- media links and similarity-matching techniques such as text retrieval and database queries. (History students have used a Microcosm application as an integral part of their studies for the last two years.)

    Students present their results with standard word pro- cessing packages. Then they use Microcosm to link refer- ences in their essays to documents held in the multimedia archive. They can thus create multimedia documents and

  • The Explanatkm Planner


    The Explanation Planner, developed at the University of Massachusetts, responds to student questions in real time, like a private tutor.' It parses typed student queries, using dialogue history to disambiguate (establish a single seman- t i c for) referents such as "the switch" in Figure A. And it plans the response with internal representations before translating into natural language. The domain knowledge for the Explanation Planner is basic electricity and electrical networks. Although not shown in the figure, the Explanation Planner reasons about and selects the appropriate text and graphics based on the learner's question. The dialogue occurs through a direct-manipulation interface (see the sec- ond question, Q2, in Figure A, for an example). In addition to typing the nonbold text, the student selects and drags (via mouse) two phrases from the Explanation Planner's first response (boldface in the figure). Displayed information remains attached to i t s internal representations, and the machine directly accesses the referents of boldface phrases. Students can also use mouse selections t o update an inter- nal focus-of-attention data structure by bringing the selected context into the foreground. The system uses the input mouse selections to determine on which concept the stu- dent is focused. It infers the student's preference for media type by tracking media choices.

    Media is not just an adjunct t o the program or a separa- ble piece of data that terminates upon display.2 Each media

    I_______ txplsiner hDlebOOt - +-







    What happen; When )'GU

  • use different media sources to support their arguments without copying the actual multimedia data into their work. This has two important advantages:

    the data can be copyright protected, and very large files can be linked to small documents.

    Although HTML (Hypertext Markup Language) or Microsoft's OLE (object linking and embedding) could have been used to implement the system, Microcosm's search and retrieval tools provide the advantage of an inte-

    grated environment, which allows access to the database of the links used and therefore can offer advice on appro- priate documents that may have been overlooked. Because the links are maintained in separate link databases, semi- automatic procedures can be encoded to evaluate which documents the students have considered and which they have not. The data in the link databases effectively describes the sources that the student has considered in writing an essay. This makes marking the essay much eas- ier, since a record is made of the source files referenced. This l i s t can be matched against a "model" l i s t provided

    The Scholar Project at the University of Southampton Figure C shows sample documents from an application

    developed in the Department of Human Morphology for the Locomotor course. Students can use this material for a resource base to supplement a practical dissection labora- tory; for revision and review after a dissection; or for browsing to supplement an explanation or definition. Self- assessment tests (specific questions integrated into the envi- ronment) check students' knowledge of the topic. Students can find answers to the tests or explore a definition by cross- referencing the text or definition with other media, such as video sequences, line diagrams, or photographic images.

    A second example of the Scholar Project is the Properties of Materials course given by the Engineering and Materials Department to every first-year engineering student at 'outhampton. This course is laboratory based and therefore !xpensive t o run in terms of staff time. The Scholar Project is eworking some difficult laboratory sessions as multimedia ourseware. Figure D shows a screen shot from the course- vare, developed to replace a laboratory on phase diagrams- I subject notoriously hard to teach. This application completely eplaced the laboratory sessions in the 1994-95 school year. Self-assessment material is built into the application.)

    The Department of Oceanography has developed a third ipplication, which it uses to teach sampling and observa- ion techniques before student oceanographic field study see Figure E).

    Figure C. The Locomotor course module.

    The Scholar Project provides flexible, customizable, and reusable learning environments within Microcosm. It also offers a variety of teaching methods, including simulation, guided learning, and problem solving.

    Figure D. The Phase Diagram module.

    Time/ Phasi

    Figure E. Sampling and observation techniques in the Oceanography module.

    Computer -!I!!!

  • by the teacher. The teacher can use the results of this matching process either for grading purposes or for giv- ing students feedback about important sources that they might have missed.


    more effective and widely available include Technological challenges for making multimedia tools

    transitioning to knowledge-based multimedia systems that reason about the curriculum, student, and situa- tion before responding; transitioning to network-based multimedia systems rather than individually equipped workstations; and developing authoring tools to support rapid develop- ment of multimedia materials.

    Knowledge-based multimedia systems Current click-and-show software does not respond

    well to the individual students needs, since it is frequently designed around combinations of media and situations available to the student only at specific instances. Click- and-show software is also difficult to build and is inflexi- ble to idiosyncratic student behavior.

    On the other hand, knowledge-based tutors dynami- cally adapt their response and presentation to instruc- tional needs by reasoning about domain, student, and tutoring knowledge and human-computer interaction. Evaluation of knowledge-based tutors shows that stu- dents progress to the same mastery level with these tutors in one-third of the time required by conventional instructional methodology. Furthermore, students using these systems show a 40 percent improvement over their performance from classroom instruction.9 Research shows success with these tutors in military, high school, and college courses. Unprecedented learning advances are possible, since knowledge-based instructional sys- tems can reason about the problem being solved and make assumptions about the students learning at a fine level of detail. This fosters instruction optimized to indi- vidual students.

    However, some problems continue to plague develop- ment of knowledge-based multimedia systems. Until recently, these systems were notoriously difficult to build, requiring several years for a small, focused system. Object- oriented languages, easy-to-use graphical interfaces, and emerging authoring tools now provide the resources to more easily build these systems. In one case, a knowledge- based tutor was built more quickly than a traditional instructional system. Researchers are addressing prob- lems of knowledge representation, error diagnosis and remediation, and student modeling to expand the scope of knowledge-based systems and facilitate their rapid development.

    Network-based multimedia systems We expect the proliferation of high-speed networks with

    quality of service guarantees to support instructional mul- timedia in virtually any desktop environment (see Resource Management in Networked Multimedia Systems by K. Nahrsted and R. Steinmetz, in this issue). Further research in the field of digital video and distance learning is neces-

    sary to allow delivery of video images and lecture and instructional material in real time, permitting remote stu- dents and teachers to dynamically interact. This technology could dramatically reduce hardware costs for distributed instructional systems. The World Wide Web, a matrix of interconnected Internet hosts, is part of the solution, but only one part. The connections merely let a user easily access information from another machine; they provide no domain knowledge of the information being presented.

    Instructional programs on the network should contain a few multimedia resources along with routines and dri- vers for accessing a predefined inventory of other avail- able multimedia resources. Thus, a single program could use many resources. Systems stored on the network could hold supplementary resources in a legally and techno- logically accessible format. Media holdings might be remote archives (like the Library of Congress) or hetero- geneous networks of file servers with terabytes of cata- loged sounds and images available to any program that needs them. The main instructional program could pro- vide still images and text, and the network could supply full-motion video.

    Authoring tools Effective instruction or training requires more than sim-

    ple navigation among media. Authoring tools or templates for developing training systems should contain conditional branching with programming logic to evaluate a students response or review the current question. Authoring tools must track student activities (for example, Authorware, Icon Author, and CourseBuilder) and let authors identify screens to be repeated or mastery levels to be achieved. However, few commercial authoring tools encode

    domain knowledge, inferences about student knowledge, or tutoring strategies for responding to a student idiosyn- cratically.

    Research laboratories are developing more powerful authoring tools. The Department of Defense, through ARPAs Technology Reinvestment Project (TRP), has funded a major consortium to commercialize such tools, thereby reducing the time and resources required to build new instructional systems. (Consortium members include Apple Computer, Houghton Mifflin Company, PWS Publishing, the University of Massachusetts, Carnegie Mellon University, Stanford University Medical School, and the University of Colorado.) In addition, commercial cross-platform script-based tools will provide the foun- dation for authoring tools that let teachers input domain knowledge, pedagogical strategy, and multimedia through graphical interfaces. Automating the building process will support transfer of expert knowledge to train- ing systems and feedback to computer scientists from cog- nitive scientists about the effectiveness and specificity of tutorial systems. One cross-platform authoring language, ScriptX, developed by Kaleida Labs (a joint partnership ofApple and IBM), enables creation of multimedia objects that can run on Mac, PC, and Unix workstations and TV set-top boxes.

  • Efficiency of development effort Developing multimedia instructional systems involves

    extensive costs for multiple expertise (such as content experts, programmers, and instructional and graphics designers) and resources for planning, programming, observing student behavior, debugging, and making the sys- tem bulletproof. For now, its more art than science. Commercial companies and universities are beginning to develop multiple instructional systems, but many systems still come from first-time, unguided design-and-build teams. Moreover, the financial investment for deployment is high, considering the need for fast computers, CD-ROM drives, audio capability, and color displays for every student seat. Fortunately, the cost of these items keeps decreasing.

    MULTIMEDIA LEARNING ENVIRONMENTS, O R PEDAGOGUES, must be educationally effective and provide students the same quality of experience as traditional teaching meth- ods do, if not better. At the very least, such systems will enhance and improve the quality of learning. When they work well, they can also educate in cutting-edge ways, let- ting faculty members spend less time lecturing and more time working with individual students.

    The global network will enable easy access to informa- tion. Multimedia and artificial intelligence technology should play a central role in making this information more realistic and practical, providing access to more knowl- edge and a variety of media forms.

    Desktop multimedia can make computer education as effective as one-on-one human tutoring yet as compelling, affordable, and widespread as television. The instructional systems weve described merely provide the foundation for what should become the global multimedia infra- structure for future education and training. Besides con- tributing to a shift in education, multimedia teaching systems can inspire active and motivated learning, leading to more effective instruction. I

    Acknowledgments We acknowledge support for systems development from

    National Science Foundation Grant MDR 8751362, and the original work of Chris Eliot, Dan Suthers, and Matt Cornel1 in developing these systems. In addition, Robert Rose made helpful editorial comments, and Gwyn Mitchell prepared the manuscript. We also acknowledge the fol- lowing for their contributions: the Microcosm team in the Multimedia Research Laboratory at Southampton; Frank Colson, director of the Hides project; the staff at Southampton; and all staff and students who contributed to the Scholar Project, particularly the authors of the appli- cations cited, namely Philippa Reed, Julian Bailey, Paul Riddy, and Sandra Peel. And we acknowledge Arturo Rodriquez for bringing the authoring team together and assisting in the editing.

    References 1. S.M. Stevens, Intelligent Interactive Video Simulation of a

    Code Inspection, Comm. ofthe ACM, Vol. 32, No. 7, July 1989, pp. 832-843.

    2. R.C. Schank, Active Learning Through Multimedia, IEEE MultiMedia, Vol. 1, No. 1, Spring 1994, pp. 69-78.

    3. W. Hall, Ending the m a n n y of the Button, IEEEMultiMe- dia, Vol. 1, No. 1, Spring 1994, pp. 60-68.

    4. J. Henderson, Interactive Videodisc to Teach Combat Trauma Care,J. ofMedical Systems, Vol. 10,1986.

    5. C. Eliot and B. Woolf, Reasoning About the User Within a Simulation-Based Real-Time Training System, Proc. Fourth Intl Conf. User Modeling, Mitre Corp., Bedford, Mass., 1994.

    6. M. Cornell, B. Woolf, and D. Suthers, Using Live Informa- tion in a Multimedia Framework, in Intelligent Multimedia Interfaces, M. Maybury, ed., AAAI/MIT Press, Menlo Park, Calif., 1993, pp. 307-327.

    7. W. Hall and H.C. Davis, Hypermedia Link Services and Their Application to Multimedia Information Management, Infor- mation and Software Tech., Vol. 36, No. 4,1994, pp. 197-202.

    8. W. Hall and F.R. Colson, Multimedia Teaching with Micro- cosm-Hides: Viceroy Mountbatten and the Partition of India, History and Computing, Vol. 3, No. 2,1991, pp. 89-98.

    9. S. Lajoie and A. Lesgold, Apprenticeship Training in the Workplace: Computer-Coached Practice Environment as a New Form of Apprenticeship, in Intelligent Instruction by Computer: TheoryandPractice, J. Farr and M. J. Psotka, eds., Taylor and Francis, Washington, D.C., 1992.

    Beverly Park Woolf is a senior research scientist and direc- tor of the Center for Knowledge Communication at the Uni- versity of Massachusetts at Amherst. She has more than 15 years experience in educational computer science, intelligent tutoringsystems, and multimediasystems development. She has a PhD in computerscience and an E& in education, both from the Universityof Massachusetts atAmherst. She haspub- lished over 50 articles and has delivered keynote addresses, panels, and tutorials in several countries. She is a councilor on theExecutive Board of the American Association ofArtifi- cia1 Intelligence (M), AVMultimedia editorfor Computer, and assistant editor for Interactive Learning Environments. She also cochaired the AAAl Spring Symposium on Knowl- edge-Based Systems for Learning and Teaching.

    Wendy Hall is a professor in the Department of Electronics and Computer Science at the University of Southampton, UK. Her research interests include development of multime- dia databases and applications of multimedia information systems in education, industry, and commerce. Major pro- jects include the development of the open hypermedia sys- tem, Microcosm, and the application of multimedia technology to electronic archives. She received a BSc in math- ematics and a PhD inpure mathematics, bothfrom the Uni- versity of Southampton, and an MSc in computer science from City University. She is codirector of the Multimedia Research Group at Southampton and of the Universitys Teaching and Learning Technology Project and the Digital Libraries Research Centre. She is a member of the British Computer Society and is a chartered engineer.

    Readers can contact BeverlyPark Woolfat the Department of Computer Science, University of Massachusetts, Amherst, MA 01003; e-mail; and Wendy Hall at the Department of Electronics and Computer Science, Uni- versity of Southampton, Southampton SO17 1 BJ, UK; e-mail uk.



View more >