Virtual Environments and the Acceleration of Experiential Learning

International Statistical Review (2007), 75, 3, 322–335 doi:10.1111/j.1751-5823.2007.00033.x

Virtual Environments and theAcceleration of Experiential Learning

Chris Wild

University of Auckland, New Zealand. E-mail: [email protected]

Summary

Darius et al. (2007) and Nolan & Temple Lang (2007) give examples of virtual environments thatcan, for specific purposes, substitute for the real world. We are in the early stages of developments thatcould revolutionize statistics education by making it possible to capture efficiently important aspectsof the thinking and practice of professional statisticians previously learned only from long yearsof experience. The ability of virtual environments to automate processes provides a potent weaponfor tackling the tyranny that Time exercises over such modes of learning. We discuss the many newpossibilities that are opened up by virtual environments together with cognitive and pedagogicalimperatives to be addressed to ensure that environments actually do teach the lessons they weredesigned to teach. We echo Nolan and Temple Lang’s call for the development of environments to bemodular and open source. Taking the R-project as a model, this can lead to a growing repository ofbuilding blocks that make the construction of future environments less costly, thus facilitating therealization of more and more ambitious conceptions.

Key words: Experimental design; interactive learning environments; java applets; open source develop-ment; statistical thinking.

1 Introduction

1.1 Revolution

“You say you want a revolution / Well, you know / We all want to change the world”

– Beatles, White Album, 1968.

With the papers by Darius et al. (2007) and Nolan & Temple Lang (2007) we are experiencingearly stirrings of a new revolution in statistical education. Its potential may even be commensuratewith that of its predecessor that was ushered in by the advent of user-friendly software fordata analysis. Others have already experienced similar revolutions, with uncannily realisticflight simulators, battleground simulators and virtual medical environments. Motivated byconsiderations of cost and safety the builders of these systems have striven to create believableenvironments in which trainees can begin to hone their professional skills both inexpensively andwithout endangering either themselves or others. Even more important for our revolution willbe the ability to make journeys through landscapes of rich learning experiences on dramaticallycollapsed time scales.

C© 2007 The Authors. Journal compilation C© 2007 International Statistical Institute. Published by Blackwell Publishing Ltd, 9600 Garsington Road,

Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.

Virtual Environments and the Acceleration 323

The precursors of the coming revolution are the now ubiquitous simulation-based java appletsdemonstrating facets of random behaviour. Their goals have been very limited, however, typicallytargeting an understanding of a few generic concepts such as sampling variation and the behaviourof confidence intervals, or the patterns that can be seen in data generated by a particular model.The simulation-based applets have relatives that are specialized calculators and analysis engines,performing computations, drawing graphs, or showing how the behaviour of a model is governedby its parameters. Additionally, we have had internet-delivered instructional materials of varyingdegrees of sophistication. All of these things have been minor improvements on what has alreadybeen done in many ways for many years. Where we are heading is different in kind. The newvirtual environments will no longer simply “replace” books, analysis tools, etc., but will besurrogates for “the real world”; virtual worlds in which students can design and conduct statisticalinvestigations. This is most obvious in the two environments Darius et al. describe. While whatwe are currently seeing are baby steps on the journey towards sophisticated virtual environments,they are hugely significant in showing us what can already be done and, furthermore, openingour minds to entirely new possibilities.

Darius et al. with their experimental design environments (their website also has sampling en-vironments), and Nolan & Temple Lang with dynamic documents are not the only revolutionaries,however. There is a cell in Simon Fraser University, Canada, which includes Carl Schwarz with hissurvey environment StatVillage (Schwarz, 1997) and “Online experimental design” (Schwarz,2003) (both systems available at www.stat.sfu.ca/∼cschwarz) and, more recently, Derek Binghamand Sigal Blay with a series of small environments delivered from a webpage called “StatisticalExperiment Simulations” (http://cgi.sfu.ca/∼sblay/cgi-bin/stat/). Michael Bulmer (2003, 2004)describes data-generating environments growing virtual plants, rats and humans.

In an exciting new development, Jock MacKay and Stefan Steiner of the University of Waterloohave developed a virtual environment simulating the high-volume manufacturing of an engine-component (see www.stats.uwaterloo.ca/Faculty/Steiner.shtml). Plant output is the result of asequence of processes, each of which has varying external inputs and environmental factors,control factors and outputs feeding into the next process. Initially, there is excess variation ina critical product characteristic. Student teams, each with their own version of the process, areassigned the task of reducing the variation to an acceptable level. The teams are given partialinformation about how the process operates, how the inputs vary and so on. Following a process-improvement framework, the students have to devise and conduct a sequence of investigations,each one built on what has come before, to identify a major cause of the variation and thenfind and implement a solution that will meet the goal. There is a wide variety of study plansavailable (retrospective and prospective observational studies, experimental designs of varyingcomplexity etc.). Carrying out any study, making measurements, changing control factors andfollowing parts through the process have associated costs. Each team has a budgetary cap.

While the environments above are becoming increasingly ambitious in terms of their statisticalobjectives, they are still naıve in some other respects when compared with some of what hasbeen done in other areas such as science education, as we will see in the next subsection.

1.2 The Revolution in Scientific Inquiry

Nelson & Ketelhut (2007) review research into the problems of implementing “authenticscientific inquiry” curricula in schools and into the emerging use of virtual environmentsas a means for providing students with experience in interactive scientific inquiry practices.“Authentic scientific inquiry” was intended to translate into classroom practice in the US asfollows, “Inquiry is a multifaceted activity that involves making observations; posing questions;

International Statistical Review (2007), 75, 3, 322–335C© 2007 The Authors. Journal compilation C© 2007 International Statistical Institute

324 C. WILD

examining books and other sources of information to see what is already known; planninginvestigations; reviewing what is already known in light of experimental evidence; using toolsto gather, analyze and interpret data; proposing answers, explanations and predictions; andcommunicating the results” (National Research Council 1996, p. 23). The overlap betweenthis and reform agendas for statistics education (see Garfield & Gal, 1999) is almost total. Notsurprisingly, these laudable but ambitious goals have proved much easier to write than to translateinto the actual experience of real students.

Nelson and Ketelhut discuss several virtual environments that seem to have made substantialprogress towards delivering affordable and interesting experiences of scientific inquiry. Wewill highlight just one, River City (see http://muve.gse.harvard.edu/rivercityproject/). QuotingNelson and Ketelhut, “The River City virtual world is set in the late 1800s . . . . River Cityincludes a main street with shops, a library, and elementary school, along with institutions suchas a hospital, university, and city hall. Upon entering the city, the students’ avatars can interactwith computer-based agents (residents of the city), digital objects (pictures and video clips),and the avatars of other students. In exploring, students also encounter visual stimuli such asmuddy dirt streets, and auditory stimuli such as the sounds of coughing town residents. Contentin the right-hand interface-window shifts based on what the student encounters or activates in thevirtual environment, such as a dialogue with an agent or historic photos and accompanying textthat provide additional information about the town and its residents. . . Students work in teams ofthree or four to develop and test hypotheses about why residents are ill. Three different illnesses(water-borne, air-borne, and insect-borne) are integrated with historical, social and geographicalcontent, allowing students to develop and practice the inquiry skills involved in disentanglingmulti-causal problems embedded within a complex environment . . . . Over the course of a 4 weeklong curriculum, students experience a year of virtual time in River City. First visiting RiverCity in October 1878, student teams return several times to find that 2–3 months have passed inRiver City on each subsequent visit. A final sharing day at the end of the project allows studentsto compare their research with other teams of students in their class and to piece together someof the many potential hypotheses and causal relationships embedded in the virtual environment.While exploring the River City world, students can also make use of several interactive toolsdesigned to scaffold their inquiry, manage complexity, and mimic real-world scientific inquiryprocesses . . . These “tools for inquiry” include a water sampling tool, mosquito catcher, stooltester, lice test, an environmental health meter, and a unique tool for running experiments bychanging elements in a world to see the results.” (Nelson and Ketelhut then go on to discussresearch on student use and effectiveness.)

Sounds fascinating, doesn’t it? And intensely statistical! I am not suggesting that statisticseducation goes anywhere near anything this sophisticated in the short term, particularly at thetertiary level. Too much hard thinking needs to be done about goals and the means of achievingthem before the required levels of funding could be spent effectively even if we had accessto them. But seeing ambitious systems like this in the backyards of others is a wake-up callnonetheless.

1.3 Reform

Yet even while the new revolution is brewing, we are still struggling to learn to surf thetsunamis triggered by the last one. We have not yet taken on board fully its implications foreducational content. The bottom-line, what-is-most-worth-knowing question is, “Of all thatstatistics education might potentially deliver for a given group of students, what will benefit themmost in the real world?” Answers change with time. The biggest change-agent affecting rational



responses to the what-is-most-worth-knowing question is advancing technology, which begsthe precursor questions, “What can machines do and what thinking remains critically human?”Statistics education should clearly pay particular attention to the latter. So what can machinesdo? More and more they can do almost anything that is entirely algorithmic or procedural. Whatthey cannot do is creative thinking, real-world modelling, and arriving at synthesis and insight.

Statistics students have severe difficulties with even highly sanitized versions of real-worldproblems because there is no obvious path to “the right answer”. They are much more likelyto freeze than to experiment and just “try things out”. This tendency is undesirable becauseof its limiting effects on the ability to make any progress at all when on unfamiliar ground. Acontributing factor is that most of the educational activities that statistics students experienceare very structured, very procedural, laid out step-by-step. The balance between creative idea-generation and the operation of learned mechanical procedures is set far too close to theprocedural. Shifting the balance is difficult. Many of us are ourselves intensely uncomfortablein unstructured teaching-situations without obvious “right answers”, where some students mayhave a far better grasp of the application area than we do and possibly even better ideas abouthow to proceed.

We live in a world where machines do calculations and data processing. The essential humanrole is to ask questions, and to obtain, interpret and synthesize information from a variety ofsources. We want to help our students to prepare themselves for this new world. The biggestholes in our educational fabric, limiting the ability of graduates to apply statistics, occur wheremethodology meets context (i.e. the real world). Engaging with context is fundamental to theup-front phases of investigation (problem abstraction and question generation, study design,measurement and variable choices), but is also important in aspects of analysis (recognizingproblem structure, and interpreting data features, i.e., answering, “What does this mean?”) andin communicating what has been learned to others (see Wild & Pfannkuch, 1999). Virtualenvironments that could help us to make real progress in any of these areas would be invaluable.

The remainder of this paper is organized as follows. Section 2 tries to abstract some principlesto inform the design of virtual environments. It does this by taking aspects of the way we learnusing ideas from statistics education research and cognitive psychology, and considering themin the light of the potential strengths of virtual environments and gaps in our current attemptsto develop statistical thinking. Section 3 focuses on two of the larger gap areas, the building ofrecognition and interpretational skills. Section 4 discusses collaborative development. The paperconcludes with a discussion in Section 5.

2 Some Aspects of Learning

2.1 Learning by Experience

To many statistical professionals, the phrase “statistical thinking” evokes characteristics ofthe way that they think that are much more the product of war stories, intuition and on-the-job experience than of any formal instruction they had been through (Wild & Pfannkuch,1999). Formal education seems to have been at its best in the transmission of received wisdomcoded in theoretical structures. Lessons and skills that are less well defined, but equally if notmore important, and all the fertilizers for insight, intuition and creativity are accumulated “byexperience”. “Experience is the best teacher”, the old adage tells us, “but the tuition is high”the Norwegian proverb continues. Where the goal is to try to capture the benefits of learningby experience within the confines of formal education, then high among the tuition costs is thesheer length of time it all takes.


326 C. WILD

Figure 1. Making the connections that matter.

Learning by experience occurs in environments that are much less structured than theclassroom. In life we are continually coming up with ideas about how to proceed, making choicesand learning by abstracting lessons from the experienced consequences of those choices. Theessential elements of learning by experience are choices and consequences. Virtual environmentswill enable us to encourage students to experiment, to “just try it out” in constructed worlds wherethey are not ground down by the burdensome time costs that accompany exploring choices inthe real world. We can see this in the main example of Nolan & Temple Lang (2007) wheredifferent types of analysis can be explored, tweaked and compared very efficiently, and inDarius et al. (2007) where many different experimental designs can be tried out and their resultscompared in a tiny fraction of the time taken by any real experiment. We will further arguein Section 2.3 that where environments can be structured to minimize the time and cognitivedistances between choices and consequences the lessons may be learned much more effectively aswell.

The imagery I have been using has its limits. It is unlikely that time figured highly in the genesisof the Norwegian proverb. Tuition costs refer, in all probability, to the painful consequences ofbad choices. The reason why the lessons of life can be burned so indelibly into our being maywell be the pain that accompanies their learning. Those pain levels will also be substantiallyreduced in a virtual world. But, with time as the severely limited resource it is we have a choicebetween a few hard-won experiences and more experiences gained time-efficiently. The formermay lead to a narrower but deeper education and the latter to a broader, but less deep, education.We cannot really know, however, to what extent such a dichotomy is real as there has been solittle experimentation with fostering efficiently gained “experience”.

Additionally, “Experience by itself teaches nothing.” “Without theory, experience has nomeaning. Without theory, one has no questions to ask. Hence, without theory, there is no learning.”(W. Edwards Deming as quoted by Wikipedia, 15 September 2007). Unfortunately, to many ofus, “theory” is synonymous with “mathematical theory”. For discussion and references about theconstruction of theories for the many aspects of statistics that cannot be addressed mathematically,see Wild & Pfannkuch (1999) and Snee (1999).

2.2 Making the Connections that Matter

The storing of, and forming mental connections between, ideas and pieces of information iscentral to all learning. Figure 1 illustrates a very simple and obvious idea. Let A and B representideas or pieces of information. To make a connection between A and B we need first to knowabout A and B. Having both in working (rather than long-term) memory facilitates seeing theconnection between the two. Effective connection between A and B will be blocked if there aretoo many ideas being entertained at once or there is too much distracting busy work separatingour forming of an appreciation of A and an appreciation of B.



The interview of field-leader Richard Mayer by Veronikas & Shaughnessy (2005) andthe special issue of Educational Psychology Review on interactive learning environments(Volume 19, Issue 3, 2007) give a very accessible introduction to the concerns of cognitive-psychology researchers working in the area of multimedia learning. The cognitive psychologiststalk of “holding a mental representation in working memory during the meaning makingprocess”, of “reducing extraneous cognitive processing” and make statements like, “only a fewpieces of information can be actively processed at any one time in working memory within eachchannel” (Moreno & Mayer 2007). Mayer & Moreno (2003) warn of the potential for cognitiveoverload in which the cognitive processing required of learners exceeds their available cognitivecapacity. Kalyuga (2007) says, “In contrast to the essential load, an extraneous cognitive load isassociated with a diversion of cognitive resources on activities that are irrelevant to learning andcaused by poor instructional design.” Such considerations are surely fundamental for any typeof learning activity we design.

We are very good at producing ever-lengthening lists of things our educational offeringsshould accomplish. Our biggest enemy is the idea that everything is important and we needto pay attention to everything at once. Trying to do everything is a recipe for accomplishingnothing. A clear corollary of the above is the following. We should explicitly form goals forlearning activities—goals that are consistent with well-formulated overall educational goals. Weshould scrutinize goals and analyze learning activities paying particular attention to the minimalsets of mental connections that are critical for the goals to be reached. These are questions weshould be able to answer:

• What is the purpose of this educational element?• What are the mental connections that have to be made to accomplish this purpose?• What are the essential elements that have to be in place for these connections to be made?• What is distracting detail that might get in the way?• How does this element fit into the way the larger goals are being addressed?

The following is an example of “identifying the connections that matter” in the context of aset of experiences focussed on teaching students about a statistical model so that they can use itsuccessfully in practical data analysis. The connections that matter include the connections thatfoster:

• an understanding of the data structures for which the model might prove useful;• an understanding of the model itself including its assumptions, properties of the data it would

generate and how its parameters can be mapped onto practical reality;• an ability to fit the model to data using software, to do assumption checking, and an appreciation

for how far assumptions can be bent without the “technology” breaking;• an ability to map features discovered back onto the original context and interpret inferential

information, and an appreciation of the limitations of any conclusions.

If the model we are dealing with is one-way analysis of variance, I would include an intuitivefeel for how the F-test obtains evidence as a connection that matters, but a derivation of theF-test statistic and its sampling distribution as contributing to extraneous cognitive load giventhe expressed purpose of this particular set of experiences to address practical use. There are goodreasons for establishing the mathematics for advancing students but the set of mental connections-that-matter for understanding the mathematical development has very limited overlap with theset of connections-that-matter for practical use. (An obvious overlap is in what motivates thedevelopment of the mathematics.)


328 C. WILD

In courses (“units” in Australia) and programmes of courses we need a variety of experienceseach of which concentrates on building a specific set of primary connections with the collectionas a whole gradually building a web of connections. This does not preclude experiences that aretrying to give familiarity with “the whole statistical-enquiry process”. It does suggest, however,that unless these experiences are of high level with almost all of the detail hidden, the desiredconnections are unlikely to be made. It is much more likely that the lessons that stick will relateto isolated details (some clever teacher-intervention may help). With whole programmes wehave much more scope than with single courses. I am not aware, however, of anything beingdone anywhere at the programme level which is much more meaningful than looking at lists oftopics covered in courses and throwing a few thoughts like, “they should do a project somewhere”,“they should give a talk some time” and so on.

Where, if anywhere, do we see in-depth analysis and debate about goals and the minimalmental connections that are critical for achieving those goals? I would dearly love to see thembecome prominent, both at conferences and in the statistics-education literature.

2.3 Implications for Virtual Environments

Virtual environments give us the opportunity to minimize both the time and cognitive distancesbetween making a choice and experiencing the consequence of that choice, because much of whatneeds to happen in between can be programmed to happen automatically. Dynamic documents(Nolan & Temple Lang, 2007) appear to be particularly powerful tools for such purposes.

There are subtleties here which only experience with virtual environments can help us resolve.Take, for example, the greenhouse environment of Darius et al. (2007). The environment playsthe role of the real data-generating world. Students design and run their experiments and theenvironment stops at the point of giving them their data. The downstream analysis work is donein the way a statistician normally would using the tools she would normally use. This is anexcellent conception for many purposes, and in particular for building the feeling of doing realresearch and building competence with data-analytic tools. The illusion of reality is importantfor students’ engagement with the experience and belief in the relevance of its lessons to theirsubsequent real lives. Once these things have been established, however, where the focus is theconsequences of choosing a particular design over some other design, this laboriously performedanalysis becomes an onerous extraneous-cognitive-load obscuring the connection between choiceof design and the consequence of that choice. As such, the analysis becomes an intermediatestep that is ripe for automation. As a general principle, any parts of a process which employalready-mastered skills should be automated to the maximum possible extent. The time studentscan and will devote to their studies is a scarce resource, far too scarce for us to squander.

Virtual environments can provide journeys through rich learning experiences on dramaticallycollapsed time scales. By varying the extent of automation, some parts of the journey can betravelled slowly allowing students to pay full attention to the detail, other parts traversed at warpspeed but still offering fleeting glimpses of passing galaxies, giving students a gentle reminderwhat is going on in the background, and still others can be hidden entirely (teleporting?). Virtualworlds can allow users to go backwards in time (time travel!) and revise earlier decisions.Furthermore, the level of complexity can be controlled to prevent students from becomingoverwhelmed. For example, developers can control whether or not experimental runs willsometimes fail for reasons beyond the experimenter’s control, resulting in outliers or missingdata. Additional real-world complexities can be added as the student’s experience levels andsophistication increase, paralleling the usage of “levels” in the gaming world. And a virtualenvironment would be made useful for a wider range of students and courses if teachers could



be given some control over the complexity levels. There is enormous flexibility here, far beyondanything we have ever had at our disposal before. All we have to do is learn how to use thatflexibility effectively!

Optimizing the use of automation and time-acceleration is an argument against working oncomprehensive virtual worlds like River City. Virtual environments will not be able to do therequisite acceleration if they try to do everything. If they do not have a specific purpose howwill you know what processes to accelerate progress through and what to slow down and paydetailed attention to? What parts of reality do we want to capture and why? What parts do wewant to hide and why? As a data-generating world, the greenhouse environment of Darius et al.hides many of the processes involved in real experimentation. It hides problem abstraction andformulation including recognition of the desirability of performing an experiment. It hides theprocesses involved in deciding what to measure and how to measure it. It hides all the real-worldproblems of dealing with people, processes, budgets and other practical constraints on what onecan and cannot feasibly do. These are all aspects of experimentation to which someone comingout of an experimental design class should have been exposed. But the greenhouse environmentignores all of this and focuses in on experimental layout. Subject to the understanding thatexperience with the greenhouse environment is to be combined with other experiences thataddress these gaps, this may well have been a good decision (to target a defined, limitedset of objectives and address them well). There are already other tools in the teacher’s toolbox foraddressing other aspects including war stories, in-class brainstorming around scenarios and realbut toy experiments like the paper helicopters. (There are lessons of real value in toy-but-realexperiments but they may be lost on students. After more than 15 years of teaching innovativecourses built to a large extent around such experiments with many colleagues, Jock MacKaysaid in a personal communication, “we had terrible experiences with our labs—we could not getthe students to accept the connections between the labs and any real context in spite of lots ofeffort!”)

2.4 Seeing that Which is Meant to be Seen

In the interview in Veronikas & Shaughnessy (2005) about multimedia learning, Mayerlists five principles aimed at reducing extraneous cognitive-processing (coherence, signalling,redundancy, spatial contiguity and temporal contiguity), three principles aimed at managingintrinsic cognitive processing (segmenting, pre-training and modality) and two principles aimedat promoting generative processing. Generative processing is the deeper and more reflectiveprocessing related to the instructional goals. Many of these should be transferable to, or at leastsuggestive for, virtual environments in statistics.

Our experience with simple applets also gives transferable lessons for virtual environments.One lesson is that which seems to make something blindingly obvious to the initiated is oftencompletely opaque to the novice. The evidence includes the research of Lipson et al. (2003), anobservation and interview-based investigation of what students thought they were seeing whenusing a simple applet. The applet was built around a news story in which a journalist tested anAustralian Post claim that 96% of letters arrive on time by sending out 60 letters of which only52 (88%) arrived on time. The applet allowed students to continually repeat an idealized versionof the process with an underlying on-time rate of 96%. Onscreen they could see simple, familiarplots giving the results of the current run and the frequency distribution of the number of on-time letters generated by all runs to date. Teaching goals included an improved appreciation ofrandom behaviour, the distribution and use for an informal hypothesis test. The variety of ways inwhich the onscreen information was interpreted, misinterpreted or passed right over the heads of


330 C. WILD

Figure 2. Applying any technique or framework.

students is amazing, and therefore very illuminating. And yet this was a very simple applet usinga very simple concrete scenario. Lipson et al. (2006) discuss cognitive-psychology principles ofmultimedia design and then apply them to the improvement of this applet—strategies for helpingstudents actually connect with what is happening onscreen.

So students often do not see what we think they will see and do “see” things we did notintend or do not believe are actually there. If we get to the stage where most students read thescreens reliably (first base), there is then the issue of ensuring that students take on board fromthe experience and retain the desired messages (home base). Garfield & Ben-Zvi (2007) warn,“Research on simulation training indicates that even a well-designed simulation is unlikely tobe an effective teaching tool unless students’ interaction with it is carefully structured.” In themore general arena, Moreno & Mayer (2007) say, “Despite the fact that interactive multimodallearning environments allow students to manipulate the instructional materials, deep learningfrom these environments depends on opportunities for students to reflect on their actions . . . acomponent that is often absent.”

A compounding difficulty is the value clash between students wanting to do and the teacher’sdesire for them to think. “Just tell me what I have to do” is a common refrain. The resultantchallenge for the environment developer (or teacher) is to try to somehow use the student’s desireto do (or to accomplish something) to make them actually want to think. Complex computergames seem to be able to manage this. Can we? Designing environments so that students interactwith them in such a way that they abstract the desired lessons (and are able to communicate thoselessons) is a critical and complex area, fraught with difficulty. Clearly, environments cannot bedesigned in a vacuum. The design of an environment should go hand in hand with the design ofthe journeys that are to be taken through it, the tasks to be undertaken and the overall goals.

3 Recognition and Interpretation

Wild (1994) discussed various aspects of recognition. In Wild & Pfannkuch (1999, p. 231) wetalked of the use of problem archetypes as a problem-solving strategy whereby one finds a wayof mapping a new problem onto a problem that has already been solved so that the previouslydevised solution can be applied or adapted. This strategy is at the heart of the way statisticsoperates (see also Cobb, 2007, on abstraction and transfer).

The practical application of any technique, algorithm or theoretical framework involves thethree steps shown in Figure 2. Our instruction tends to focus on step 2, mechanical application.



However, steps 1 (recognition) and 3 (interpretation in context) are: first, vital to step 2 havingany utility, and second, inordinately more difficult and require inordinately more practice.

Traditionally we teach about a model with a couple of examples and exercises using the model,including some with a few wrinkles that illustrate some slightly perverse data features. The skill“apply the model/procedure” is reasonably well, even if sub-optimally, catered for. Nowadaysmost of us motivate the need for the model with a real-world example before introducing it andeven endeavour to close the gap by applying the model to the motivating example and drawinginferences by the time the educational experience has ended. We are slowly improving. But whatis missing almost entirely is building the enabling skills that permit the recognition of usefulstructure in a real-world problem that will be helpful in solving the problem. Over time studentslearn about a large number of theoretical structures. These structures become useful only whenthey also learn to recognize the elements of an applicable few through the dense fogs generatedby the diverse and complex contexts encountered in practice. Until then, the procedural skillsthat statistics educators spend so much time teaching are of very limited use indeed.

There are ways that we can do parts of this in a time-efficient manner by using large numbersof scenarios coupled with tasks that focus solely on the recognition of the appropriate tool ortheoretical structure. This can be done with or without software. We have done this at Aucklandin our first and second applied courses for quite a few years in a limited way. The vehicle is setsof exercises and examination questions that go under the catch-phrase title “Spot the analysis”.It is intended to lead to “which of all the tools accumulating in your tool bag are likely to be themost useful for getting started on this problem” but we are really just taking the first steps withsimple short scenarios and a limited range of possibilities. The structure to be recognized is notso much buried in deep contextual cover (the real world) as thinly shrouded. Our little scenariosshould, but do not, lead into a progression of scenarios with ever richer and more detailed contextinformation that reach beyond these courses and span the whole programme. Statistics educationprogrammes make far too little use of unifying, or bridging, experiences. It is hard to overstatehow hard-won recognition skills are for most students, making it imperative to start with simplerecognition tasks as a precursor to ever more subtle and realistic ones.

What can virtual environments add? Automated multiple choice with feedback has been astandard technology for some time now that can be, and is, used for these purposes. This is just aslightly improved delivery mechanism for existing instructional methods, however. Other placeswhere a great deal of experience is needed because of the interaction with diverse context isin the interpretation of features seen in data and conclusion writing (or communication moregenerally).

The greatest beauty of the virtual environments is the ability to accelerate parts of a journeyusing automation and the ability to incorporate random elements. Recognition and tool-choiceskills stemming from varieties of data behaviour may be an area where random data generation canbeneficially be applied, leading to more highly evolved descendents of the “guess the correlationcoefficient” applets. Environments that present contexts and data, give choices for analysis, theworking through of which is automated but with the ability to explore and tweak (a la Nolan andTemple Lang) and are then coupled with “communicate” tasks are an obvious use. If the problemsused are real and interesting this brings closure to the story. Using an interesting problem to getstudents intrigued to use for our own purposes but without providing closure feels like a teaseand cheat.

There are other reasons why experience with many real data sets is important. Each new realdata set brings its own wrinkles and helps add to our intuition and “data sense”. Even within oneanalysis framework, e.g. multiple linear regression, there are model-specification subtleties themastery of which spells the difference between false conclusions and real insight. In particular,there are many ways in which explanatory variables can be recoded or combined in order to better


332 C. WILD

address substantive questions of interest. Fitting models probably requires a lot less experiencethan devising models that address a scientific context well. The problem is the time it takes towork through all of this. We need ways to focus attention on what is new and accelerate throughwhat has already been mastered. Dynamic documents (Nolan & Temple Lang, 2007) would seemto give us a mechanism for doing that.

4 Collaborative Development

The amount of work required to develop virtual environments is huge and ours is not an areathat has historically attracted large amounts of development money. Nolan and Temple Langadvocate collaborative development citing the R model (R Development Core Team, 2007). Thisis music to the ears of an Aucklander. R was born right here in my home department circa 1992.The parents who gave it life, nurtured it and then sent it out into the world as a ward of the widercommunity were Ross Ihaka and Robert Gentleman. It has become a real phenomenon, bringingever larger numbers of statisticians together in the collaborative development of a huge, cutting-edge software system for the benefit of researchers and data analysts everywhere. It shows justwhat a voluntary, community, building project can do.

But the fact that R is a phenomenon also means that it is highly unusual and, as such, perhaps anaberration. Can lightning strike twice? In one sense it already has with the Robert Gentleman-initiated Bioconductor project for bioinformatics, which has also taken hold and, in just sixyears, attracted a substantial number of contributors. Bioconductor is perhaps a too-closely-related development to R to count as a separate lightning strike, however. So what are some ofthe factors in the way R took hold in the research and data-analysis community and to what extentcan the R experience be replicated on the educational-technology side? Part of it is being an opensource project, being modular, extensible and having a stellar team looking after the essentialcore of the system. Part of it is starting with a substantial collection of well-designed and veryuseful building blocks that, from the onset, provide substantial savings in time to developerswanting to build something new. Another component seems to be the existence of a substantialuser base for the existing tools and an unsatisfied demand for even more tools. But the user basemust also contain within it a subset, with sufficient critical mass, of people who have the skillsand willingness to create new tools.

We are in the infancy of virtual environments for statistical education, a time when a lot ofteacher-led experimentation is desirable: first to generate a rich set of possibilities, and thenbegin to understand what works. There are considerations affecting developers, with respect totime and making the connections that matter, that are very similar to those affecting studentlearners. Where realizing a new creation takes a great deal of time and effort it is hard to give fullattention to what really matters. What really matters is a compelling scenario and the featuresthat most effectively promote the desired student learning. These include a convincing look andfeel for the finished product to help make something, which is actually artificial, believableenough to students that its lessons are transferred. Where the costs of change are prohibitive, it isnot possible to keep tinkering and making incremental improvements. For substantial numbersof people to participate in experimentation, we will need the tools that heavily reduce the timeand effort separating teacher conception from a working prototype. (Outsiders like instructionaldesigners cannot help us much until we know quite what it is that we want.) Perhaps then wewill be able to divert the duplicated effort that produces hundreds of applet clones into buildingsomething really substantial. Tool builders like Nolan and Temple Lang deserve a great dealof praise and gratitude. A large number of lower-level tools have already been assembled inOmegahat giving us the rough equivalents of computations with multidimensional arrays and



solve(), but many of the less technologically inclined of us will be waiting for the higher-leveltools, at least for optim() if not for glm(). And while these will help with certain types ofenvironments, I wonder what the building blocks for realistic-looking worlds will look like, whowill be the champion for those and where they might accumulate?

5 Discussion

“Asking whether computers are a good instructional technology is like asking whetherbooks are a good instructional technology. It all depends on how they are used, that is,on the instructional method.” – Richard Mayer, 2005

“Technology matters only to the extent that it may allow new instructional methods.” –Richard Mayer, 2005

Mayer is a little harsh in the second statement, using “matter” in a deeply fundamental sense.At a more superficial, but not unimportant, level, technology can improve matters by reducingcosts, increasing flexibility, overcoming problems of scheduling and distance, reducing cheatingby giving students individualized assignments and helping students enjoy their education more.But, where the instructional methods are largely the same, it is not surprising that we do notget large shifts in the effectiveness of instruction. Affectiveness (promoting a positive emotionalresponse; to be contrasted with effectiveness) should not be underestimated either. It plays anon-negligible role in attracting and retaining students, and gaining their interest. We are onmuch more compelling ground, however, when technology enables us to do things that werenot previously possible. Where we are addressing new areas, however, historically used testsand other assessments are no longer relevant. This creates difficulty for assessing progress ineducational delivery using student performance measures such as grades over time because weare “moving the goalposts”. Sometimes goalposts are moved haphazardly, but at other times thegoalposts are moved for a very good reason indeed. Someone has finally realized that they werein the wrong place.

Darius et al. (2007) with their experimental design and sampling environments enable studentsto learn to design statistical studies, as opposed to analyzing data from designed studies,by actually designing statistical studies. Nolan & Temple Lang’s (2007) dynamic documentscombine a rational linear presentation of the analysis of data that reveals “the answers” with anaccount of the journey, its turnings and wrong turnings and allows students to explore modifiedpathways. Both sets of authors are facilitating new learnings about aspects of statistical practicethat are simply not possible in realistic time frames without clever empowerment by computertechnology. Although Nolan & Temple Lang (2007) has been written entirely in terms of statisticseducation, the system they are describing provides a potent infrastructure for communicationin many other contexts as well. One area of application is “reproducible research” (Gentleman& Temple Lang, 2007). Another is empowering statistical consultants to produce sophisticateddecision-support documents for their clients based upon the consultant’s data analyses. Suchdocuments would enable clients to explore, dynamically, very complex scenarios—varyingenvironmental assumptions, projections, inputs and business strategies. The documents couldalso allow for variations on the assumptions used and decisions made in the underpinning dataanalyses.

The authors of Darius et al. 2007 (Paul Darius, Kenneth Portier and Eddie Schrevens) andNolan & Temple Lang 2007 (Deborah Nolan and Duncan Temple Lang), together with others(e.g. Carl Schwarz, Jock MacKay and Stefan Steiner, Derek Bingham and Sigal Blay, and


334 C. WILD

Michael Bulmer) have done us a great service by demonstrating new capabilities using virtualenvironments and opening up new possibilities, giving the stimulus for others of us to startimagining new environments that push the boundaries still further.

Virtual environments give us the potential to provide worlds in which:

• students can design and conduct statistical investigations;• investigative time scales are collapsed;• rich context information can be included in a wide variety of ways (cf. River City);• complexity levels are controlled, set at levels students can cope with and increasing levels of

complexity incorporated (cf. “levels” in games);• parts of processes corresponding to already-mastered skills are automated;• different students/groups have unique experiences, thus eliminating copying problems and

providing a basis for valuable lessons about variation.

The most exciting prospect of virtual environments is in substituting, for specific purposes,for the real world allowing students to learn lessons about aspects of statistical investigation thatprevious generations have only learned later, from long real-world experience. They are a potentweapon for tackling the tyranny of Time. I look forward to virtual environments that

• address clearly thought out educational goals;• mirror easily understood concrete realities;• look and feel real to maximize student engagement and belief in the relevance of the

environment’s lessons to their future lives;• allow us to capture vital elements of “learning by experience”;• are designed to target the mental connections that matter and minimize extraneous cognitive-

processing;• minimize the time and cognitive distances between making choices and experiencing conse-

quences;• are designed in concert with the journeys to be travelled within them and the tasks to be

undertaken using them, thus ensuring that students actually see that what was intended thatthey see and abstract the lessons they were intended to abstract;

• embed assessment;• patch the holes in our educational fabric where methodology meets context;• display a mastery of the art of what to reveal and what to hide;• allow teacher control of complexity and other environmental settings;• automate all parts of processes that correspond to already-mastered skills;• are rich in terms of pathways to explore;• allow students to efficiently “unlock the stories in data”;• encourage students to “just try things out” in worlds that minimize the time costs of taking a

wrong turning.

But this is early days. In the meantime, I am hoping that we will see a great deal of teacher-ledexperimentation, just trying things out and, along the way, spinning off useful building blocksas Nolan and Temple Lang are doing into open-source ventures like R and Omegahat—to makefuture building less costly for everyone and facilitate the realization of ever more ambitiousconceptions. Adapting the words of Mao Zedong, “Let a thousand flowers bloom, let a thousandschools of thought contend.”



Acknowledgements

I would like to thank Jock MacKay, Chris Triggs and Maxine Pfannkuch for some very helpfulsuggestions.

References

Bulmer, M. (2003). Growing virtual plants for teaching and learning statistics. In Proceedings of the AppleUniversity Consortium Conference, Ed. N. Smythe, pp. 3-1–3-7. Sydney, Australia: Apple Computer Australia,auc.uow.edu.au/conf/conf03.

Bulmer, M. (2004). Virtual worlds for teaching statistics. CAL-laborate, 11, June, 1–4.Cobb, G.W. (2007). One possible frame for thinking about experiential learning. Int. Statist. Rev., 75, 336–347.Darius, P.L., Portier, K.M. & Schrevens, E. (2007). Virtual experiments and their use in teaching experimental design.

Int. Statist. Rev., 75, 281–294.Garfield, J. & Ben-Zvi, D. (2007). How students learn statistics revisited: A current review of research on teaching and

learning statistics. Int. Statist. Rev., 75, 372–396.Garfield, J.B. & Gal, I. (1999). Assessment and statistics education: Current challenges and directions. Int. Statist.

Rev., 67, 1–12.Gentleman, R. & Temple Lang, D. (2007). Statistical analyses and reproducible research. Journal of Computational

and Graphical Statistics, 16(1), 1–23.Kalyuga, S. (2007). Enhancing instructional efficiency of interactive e-learning environments: A cognitive load

perspective. Educational Psychology Review, 19, 387–399.Lipson, K., Francis, G. & Kokonis, S. (2006). Developing a computer interaction to enhance student under-

standing in statistical inference. Proceedings of the Seventh International Conference on Teaching Statistics:Working cooperatively in statistics education, Salvador, Brazil, July, 2006. [CD-ROM.] Voorburg, The Nether-lands: International Statistical Institute. [Retrieved 1 September 2007, from http://www.stat.auckland.ac.nz/∼iase/publications/17/6E2 LIPS.pdf].

Lipson, K., Kokonis, S. & Francis, G. (2003). Investigation of Students’ Experiences with a Web-Based ComputerSimulation. Proceedings of the IASE Satellite Conference on Statistics and the Internet, Berlin, August 2003.[CD-ROM.] Voorburg, The Netherlands: International Statistical Institute. [Retrieved 1 September 2007 fromhttp://www.stat.auckland.ac.nz/∼iase/publications/6/Lipson.pdf].

Mayer, R.E. & Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. EducationalPsychologist, 38, 43–52.

Moreno, R. & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19,309–326.

National Research Council (1996). National Science Education Standards: Observe, interact, change, learn. Washing-ton, DC: National Academy Press.

Nelson, B.C. & Ketelhut, D.J. (2007). Scientific inquiry in educational multi-user virtual environments. EducationalPsychology Review, 19, 265–283.

Nolan, D. & Temple Lang, D. (2007). Dynamic, interactive documents for teaching statistical practice. Int. Statist. Rev.,75, 295–321.

R Development Core Team. (2007). R: A language and environment for statistical computing. R Foundation forStatistical Computing, Vienna, Austria, 2007. URL http://www.R-project.org. ISBN 3–900051-07–0.

Schwarz, C.J. (1997). StatVillage: An On-Line, WWW-Accessible, Hypothetical City Based on Real Data forUse in an Introductory Class in Survey Sampling. J. Statist. Educat., [Online: www.amstat.org/publications/jse/v5n2/schwarz.html] 5(2).

Schwarz, C.J. (2003). An Online System for teaching the design (and analysis) of experiments. 2003 Proceedings of theAmerican Statistical Association, Statistical Education Section [CD-ROM], Alexandria, VA: American StatisticalAssociation: 3732–3738.

Snee, R.D. (1999). Development and use of statistical thinking: A new era. Int. Statist. Rev., 67, 255–258.Veronikas, S. & Shaughnessy, M.F. (2005). An interview with Richard Mayer. Educational Psychology Review, 17,

179–189.Wild, C.J. (1994). On embracing the ‘wider view’ of statistics. The American Statistician, 48, 163–171.Wild, C.J. & Pfannkuch, M. (1999). Statistical thinking in empirical enquiry (with discussion). Int. Statist. Rev., 67,

223–265.

[Received August 2007, accepted September 2007]


Documents

Virtual Environments and the Acceleration of Experiential Learning