Computers & Education 56 (2011) 418–428

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier .com/locate/compedu

A case study on using prediction markets as a rich environment for active learning

Patrick Buckley a,*, John Garvey b, Fergal McGrath c

aAIB Center for Information and Knowledge Management, S120, Schuman Building, University of Limerick, Limerick, IrelandbDepartment of Accounting and Finance, University of Limerick, Limerick, IrelandcDepartment of Management and Marketing, University of Limerick, Limerick, Ireland

a r t i c l e i n f o

Article history:Received 16 December 2009Received in revised form3 September 2010Accepted 3 September 2010

Keywords:Teaching/learning strategiesInteractive learning environmentsDistributed learning environmentsMultimedia/hypermedia systemsEvaluation of CAL systems

* Corresponding author. Tel.: þ353 61 202751; fax:E-mail address: patrick.buckley@ul.ie (P. Buckley).

0360-1315/$ – see front matter � 2010 Elsevier Ltd. Adoi:10.1016/j.compedu.2010.09.001

a b s t r a c t

In this paper, prediction markets are presented as an innovative pedagogical tool which can be used tocreate a Rich Environment for Active Learning (REAL). Prediction markets are designed to make forecastsabout specific future events by using a market mechanism to aggregate the information held by a largegroup of traders about that event into a single value. Prediction markets can be used to create decisionscenarios which are linked to real-world events. The advantages of this approach in the cognitive andaffective domains of learning are examined. The unique ability of prediction markets to enable activelearning in large group teaching environments is explored. Building on this theoretical work, a detailedcase study is presented describing how a prediction market can be deployed as a pedagogical tool inpractice. Empirical evidence is presented exploring the effect prediction market participation has onlearners in the cognitive domain.

� 2010 Elsevier Ltd. All rights reserved.

1. Introduction

Mass higher education presents serious problems to implementing active learning. Large classes at undergraduate level are regularlymeasured in hundreds of students (Hogan & Kwiatkowski, 1998). Classes of this size lead to increasing student/staff ratios (McAvinia &Oliver, 2002), student bodies which are diverse in terms of ability, aptitude and motivation (Gibbs, 1992), and increased resourceconstraints. Hogan and Kwiatkowski (1998) note that in large groups, learning can be become a solitary experience, leading to feelings ofalienation. This isolation is linked to poormotivation that inevitably affects learning (Benbunan-Fich, 2002). Individual learnersmay find thepacing of content delivery too fast or too slow, leading to distraction on the part of the learner and a diminished learning experience (Hadley,2002). Academics face pressure to pursue their research agendas and “publish or perish” (Roettger, Roettger, & Walugembe, 2007). Theseconflicting demands mean it is often difficult for academic staff to implement active learning pedagogical strategies given the largeadministrative overheads that can be involved.

Many authors have pointed out that Information Technology (IT) can have a major role in promoting active learning in the educationalenvironment (Rouet & Puustinen, 2009; Tsai, 2008). The appropriate use of IT allied with careful pedagogical design can deal with thechallenges posed by large group teaching (Wang, 2009). Information technology can deliver richmultimedia applications instantaneously tolarge diverse groups of learners and has the ability to handle large volumes of data rapidly and efficiently. Some pedagogical activities canleverage IT to scale to large groups. Once created, a multiple choice assessment can be delivered to 300 students as efficiently as to 30. Otheractivities do not scale as well. While IT can be used to create a web based message board that enables peer-to-peer discussion and debate,without moderation and leadership from academic staff, the educational benefit of such tools is open to debate. In this case, the amount oftime that staff will invest in managing the educational process in an online forum is similar to the amount of time theywill spendmanagingan offline process and is proportional to the class size (Lemak, Reed, Montgomery, & Shin, 2005). In extreme cases, IT may just becomea burdenwhere teachers are required tomake the same content and learning activities available in a variety of different formats (McAvinia &Oliver, 2002).

In this paper prediction markets are presented as a pedagogical tool to implement a Rich Environment for Active Learning (REAL) asdescribed by Grabinger and Dunlap, 1995. By using IT to leverage the unique attributes of prediction markets, REALs can be created which

þ353 86 8568162.

ll rights reserved.

P. Buckley et al. / Computers & Education 56 (2011) 418–428 419

can scale efficiently to very large classes. A prediction market is defined as a market “designed and run for the primary purpose of mining andaggregating information scattered among traders and subsequently using this information in the form of market values.” (Tziralis & Tatsiopoulos,2007, p. 75). Our core insight is that prediction markets can be used to create decision scenarios that allow learners to apply knowledgedelivered in lectures to real-world problems.

In the following paper, Section 2 introduces the concept of prediction markets and provides theoretical justification for our contentionthat prediction markets can be viewed as REALs. This section also describes the pedagogical benefits of prediction market participation forboth learners and academics. Section 3 presents a case study describing how a prediction market can be deployed as a pedagogical tool.Section 4 presents a number of research questions this paper poses in order to empirically evaluate the effectiveness of prediction marketsas a pedagogical tool in the cognitive domain. The methodology used to analyse these questions is also introduced. Section 5 reports theresults of the research, while Section 6 concludes the paper with a brief summary and suggests other possible research avenues.

2. Prediction markets a pedagogical tool

2.1. Introduction to prediction markets

A predictionmarket is a tool used to make forecasts about the future outcome of large, complex systems (Wolfers & Zitzewitz, 2004). Thetheoretical roots of prediction markets are found in the discipline of economics, specifically the efficient market hypothesis (Hayek, 1945).According to the strong version of the efficient market hypothesis, the current value of an asset traded in a liquid market reflects all publiclyavailable information about that asset (Kolb, 1997). This concept has led researchers to view markets as information aggregation mecha-nisms, which in turn motivated the development of prediction markets.

In the simplest form of a prediction market, an asset is created whose value is dependent upon a future uncertain event. As an example,consider asset A, whose value is dependent upon the identity of the next British Prime Minister. This asset will return 100 virtual currencyunits if the next British Prime Minister is a Conservative and 0 virtual currency units in every other case. This asset is offered for sale ina market where participants can buy or sell the asset. If many participants believe that a Conservative is likely to be the next PrimeMinister,they will buy the asset, causing the price of the asset to rise. Conversely, if many participants believe it is unlikely the next British PrimeMinister will be a Conservative, then they will sell the asset, causing its price to fall. Therefore the price of the asset can be used as anindicator of the group’s collective estimation as to the probability of a Conservative being the next PrimeMinister. This simple scheme can beextended trivially to allow for group estimation of the probability distribution of a set of multiple disjoint events.

In an extensive literature review Tziralis and Tatsiopoulos (2007) point out the increasing interest in prediction markets both fromacademics and practitioners. Much research has focussed on the performance of prediction markets vis-à-vis polls and other surveymechanisms (Berg, Nelson, & Rietz, 2008; O’Connor & Zhou, 2008). “Information Markets: A new way of making decisions” is an excellentguide to current research in the area (Hahn & Tetlock, 2006).

There are awide variety of permutations in the design of predictionmarkets. The nature of themarket structuremeans that some form ofcurrency must be used to enable transactions. This may sometimes be real currency, but more often the market sponsor specifies thatparticipants use virtual currency to participate in a prediction market (Servan-Schreiber, Wolfers, Pennock, & Galebach, 2004). If virtualcurrency is used then other rewards are often offered to participants, linked to performance as measured in virtual currency.

Many familiar financial markets such as the New York Stock Exchange uses a continuous double auction to enable trading. In this case,a buyer and seller must agree on a price before a trade can occur. Most prediction markets use an automated market maker. An automatedmarket maker automatically fills buy or sell orders from participants and adjusts the price of the asset accordingly. It is not necessary tomatch buyers and sellers. By allowing transactions to occur immediately it reduces the complexity of the market interface, which has theeffect of lowering knowledge barriers and promoting participation (Christiansen, 2007). Examples of automatedmarket markers include theMarket Scoring Rule (Hanson, 2007) and the Dynamic Parimutuel Market Maker (Pennock, 2004).

Our core insight is that prediction markets in an educational context can be used to create decision scenarios that are linked to learningoutcomes. The challenge in our methodology is defining suitable decision scenarios. This approach is not universally applicable. Deter-ministic subjects where the solution to a problem is calculable, such as mathematics, physics or engineering, are obviously not suitable forthis approach. However, a wide range of disciplines, particularly in business and the social sciences, are aimed at developing learner’sdecision making skills in complex problem domains. In these subjects this methodology is readily applicable. As examples, decisionscenarios on stock movements can be envisaged as part of a finance module, with macro-economic indicators and political events playingcorresponding roles in economics and politics modules. In the case study described more fully in Section 3, decision scenarios are createdwhere learners are asked to predict what insurance losses will occur for a specific time period in three particular states in the United Statesof America. These decision scenarios link directly to the content and theory of the corresponding module on risk management.

Creating prediction markets are facilitated by IT. They can be deployed over the Internet via a website. Administrative tasks such asmanaging user accounts, recording trades and updating prices are handled automatically by the prediction market. The performance ofparticipants within the market can be easily and frequently assessed through an automated data download from the market. Theseattributes of prediction markets mean they scale easily and efficiently to large groups.

2.2. Prediction markets as a REAL

The foundations of REALs are found in constructivist learning theories. Constructivism is not a new concept. Its roots can be traced backto the Socratic school of philosophic inquiry, and its conceptual development can be traced through authors and educational philosopherssuch as Piaget, Vgotsky, Dewey and Simon. Fundamentally, constructivism asserts that individuals learn by modifying the internal repre-sentations they hold of reality through repeated interactions with the environment (Jonassen, 1994).

Constructivist learning theories share three major characteristics. First, knowledge is not a product to be accumulated, but rather anactive and evolving process by the learner (Gurney, 1989). A second characteristic of constructivist learning theories is that people inter-nalise their knowledge in personal ways. Rather than viewing learning as the transmission of facts from one person to another,

P. Buckley et al. / Computers & Education 56 (2011) 418–428420

constructivism emphasises the need for learning to occur within a context. Learning proceeds from “..the social, collective activity of the childto his more individualised activity.” (Vygotsky, 1986, p. 226). Finally, constructivist learning theories stress that learning is a product of socialinteraction. Learning occurs through expressing ideas, testing and comparing those ideas to others’ ideas and concepts, and modifyinginternal representations in response to these interactions and negotiations (Bednar, Cunningham, Duffy, & Perry, 1995).

The concept of a REAL was introduced by Grabinger and Dunlop in their paper “Rich Environments for Active Learning: A definition”(1995), and Grabinger et al., (1997), and Bostock (1998). REALs are presented by these authors as a comprehensive framework which canhelp educators to develop pedagogical interventions which are based upon constructivist learning principles. These authors provide fiveattributes which a pedagogical innovation must include in order to be considered an active learning environment that embodiesconstructivist learning principles. These principles are:

� Authentic Learning Contexts� Promoting Learner Responsibility and Initiative� Authentic Assessment Strategies� Generative Learning Activities� Collaborative Learning.

Other authors, such as Lebow (1993) and Simons (1993), provide similar lists.We now demonstrate how prediction markets meet these criteria and therefore can be considered a REAL. Firstly, REALs require that the

learning experience should be realistic and should accurately reflect both the situationwhere the learning is expected to be applied and thedecisions that will be required in that situation. In general, the learning experience should be built around real-world problems, events andissues (Bostock, 1998). In this proposal, prediction markets are used to present decision scenarios to the learner. For example, in the casestudy discussed below, students are asked to estimate what actual insurance losses will occur in a given region in the United States fora fixed period of time. More generally, the decision scenarios that are offered in prediction markets can be linked to real-world events. Withcareful design, these decision scenarios can closely mimic the decisions that learners will have to make in their future careers. The outcomeof these event is unknown in advance and is subject to a large number of known and unknown variables. In order to engage with thesedecision scenarios, learners will have to deliberate on the scenario. It is not possible to meaningfully engage with these decision scenariosthough the application of rote formulae and the recall of simple facts. This more closely represents how learning will be applied in the realworld and reflects the complexity that learners can expect to encounter in the problem domains under consideration in this paper.

A second attribute of a REAL is that it should promote initiative and responsibility in the learning process (Bostock, 1998). Predictionmarkets participation encourages this in two ways. First, by changing the learning event from the passive receipt of material and recall offacts to active decisionmaking, learners are challenged to engage in the learning process. Second, in our model the decision scenarios underconsideration are constantly affected by real-world events. Using the case study described below as an example, it is obvious that anenvironmental occurrence such as a hurricane will impact upon insurance losses suffered in nearby regions. The dynamic nature of thedecision scenarios presented, encourages learners to actively seek out relevant information and integrate it into their decision makingprocesses. Learners who perform these activities successfully will be rewarded through superior performance in the prediction market.

REALs should utilise assessment mechanisms that require learners to use skills rather than describe them, have a reasonable level ofcomplexity and be exemplars of how these skills would be applied in a realistic setting. Authentic assessment mechanisms are described ascontextualised, complex intellectual challenges (Wiggins, 1989). In our methodology, a prediction market is used to create decisionscenarios based on real-world events. The educational process thus changes from being a passive experience, where learners are presentedwith facts and expected tomemorise them, to onewhere learners are asked tomake decisions. Assessment of learners’ performance is basedupon the outcome of the events under consideration and is directly linked to the decisions that learners make regarding these events. Thesedecision scenarios can be constructed in such a way as to make them similar to the types of decisions learners will be making in theirprofessional careers, heightening the authenticity of the assessment mechanism.

Active learning ideally involves the creation of a knowledge artefact. This may be a physical artefact, such an essay, or an intellectualartefact. The fourth criteria associated with REALs, -generative learning, implies that learners should actively engage in understandingmodule content by creating artefacts related to the module (Bostock, 1998; Zantow, Knowlton, & Sharp, 2005).

In our framework, the knowledge artefact that is created by the learners interacting with the prediction market is the price of the assetbeing traded. This market price reflects the groups’ estimation as to the likelihood of the event occurring. It is shared amongst all thelearners participating in the market. Once the initial price of the asset is set by the market creator, it is in constant flux, and is determined bythe actions of all the participants in the market. By buying or selling assets to reflect their individual estimation as to the likelihood of anevent occurring or not, learners are constantly articulating their individual opinion to the wider group.

Additionally, learners also create internal knowledge artefacts. From a constructivist viewpoint, knowledge is created by participating ina social activity (Vygotsky, 1986). Individuals create mental models of the world to guide their decision making. They then interact with anexternal entity. On the basis of these interactions, learners’ mental models are updated through feedback and reinforcement, and theprocess begins again. Learners create personal, internalised knowledge artefacts through interaction with an external entity. Our meth-odology offers the same basic approach. The presentation of decision scenarios prompts learners to interact with a dynamic, external entity,namely themarket. Through engagement with themarket, learners create and evolve the cognitive rules that allow them tomake successfuldecisions. These cognitive rules, representing the amalgamation of knowledge gathered from lectures and the experience of the decisionmaking scenarios, are knowledge artefacts, stored by and unique to an individual learner.

Social collaboration is seen as being a crucial enabler of learning according to constructivist learning philosophies. Enabling this socialcollaboration is the final attribute that is associated with REALs. Social collaboration in a learning context is traditionally seen asa communicative process, such as that between a teacher and a student, or between peers. Collaboration in a learning environment can takemany forms including “..teaching each other, viewing from different perspectives, dividing tasks, pooling results, brainstorming, critiquing,negotiating, compromising, agreeing.”(Stahl, 2004, p. 54). The crucial point is that communication occurs between independent entities.Prediction markets can enable social collaboration in two ways. First, we believe that interaction with the market is a form of

P. Buckley et al. / Computers & Education 56 (2011) 418–428 421

communication between the learner and an external entity, namely, the market. Second, the interactive, group nature of prediction marketparticipation itself encourages social interaction and discussion.

In order to support our first point, we turn to the efficient market hypothesis first expounded by Hayek. This explicitly providesa theoretical platform that views markets as communication channels (Hayek, 1945). While there is still considerable debate as to theefficiency of markets, the underlying insight that markets transmit information is widely accepted (Kolb, 1997). Using this perspective,a market is an external entity to the individual interacting with it. Due to the operations of other students, the market prices can movewithout the direct input of an individual learner. The price of the asset is a signal to the participant, indicating the combined estimation of allthe participants in the market as to the correct price of the asset for sale in that market. Similarly, an individual learner can communicatewith the market. By buying or selling assets, the learner has a measurable effect on the price of assets in the market.

In this way, student interaction with a prediction market can be seen as a dialog. The market communicates with a student throughthe mechanism of asset prices. The learner can communicate with the market through buying and selling assets. While the bandwidthof this communication channel is lower than that associated with more traditional communication channels, such as face to faceconversation, two way information exchange does occur. While the signals being used are not what we would traditionally considerlanguage, as Vgotsky points out “the medium is besides the point; what matters are the functional use of signs, any signs that could playa role corresponding to that of speech in humans” (Vygotsky, 1986, p. 76). In the act of participating in a prediction market, learners enterinto a collaborative, knowledge building experience with an external third party that fulfils the basic criteria required of socialcollaboration.

While enabling communication and social collaboration at the problem solving level, we also believe it is reasonable to suggest thatprediction markets will prompt conversations with other peers in social contexts outside of the educational sphere. Prediction markets canbe used to present dynamic, real-world problems, which will be affected by dramatic events, depending on the nature of the decisionscenarios under consideration. In and of itself, we believe it is reasonable to suggest that this will be more likely to engage students’attention than more traditionally formulated problems.

The nature of prediction markets is that they are a group activity. The operation of a prediction market clearly indicates that a largenumber of people are participating in it. Prediction markets provide a shared, dynamic problem space which we believe it is reasonable tosuggest will act as a catalyst to prompt discussion and debate amongst learners. Finally, the predictionmarket creates a competitive elementin problem solving, which we believe will further serve to increase students’ interest in the problem under consideration. While thecompetitive element may affect students’ willingness to discuss certain issues, we do not believe it will deter it outright. For example, inmany situations, learners are ultimately competing against each other for grades which will affect their future careers, but this competitiondoes not prevent students from revising for exams together, sharing lecture notes or any other of the myriad of activities students engage incollaboratively to prepare for exams.

The above discussion demonstrates that the use of prediction markets as a collaborative tool to present learners with decisionscenarios linked to real-world events meets the conditions required for prediction markets to be considered a Rich Environment for ActiveLearning.

2.3. Pedagogical benefits of prediction markets

In the previous section prediction markets were identified as a REAL. In this section, specific benefits for learners andacademics are identified. The benefits for learners are analysed in terms of Blooms taxonomy of learning domains. Specificbenefits are identified in both the cognitive and affective domains of learning. Benefits for teaching and academic staff are alsoillustrated.

2.3.1. Developing students’ cognitive skillsPrediction market participation contributes to the development of cognitive skills in three main ways. By making decisions, learners are

encouraged to internalise and operationalise knowledge and information from a wide range of sources. Models and theories that aredescribed in lectures should be applied by learners when making decisions. Historical data should also be used by learners to evaluatedecisions. However, learners not only memorise these theories and formulae by rote, but actually apply them in a decision-making context.The link between decision scenarios and real-world events familiarises learners with decision-making scenarios that exists in theirprofessional lives, providing the opportunity to develop an appreciation of how the theoretical underpinnings of their discipline can be usedto solve real-world problems. Prediction markets provide rapid feedback to learners, both through the medium of market price movement,which allows them to see the opinion of other learners, and also through the resolution of the market. Providing regular feedback is seen asbeing important to ensuring successful e-Learning systems (Wall & Ahmed, 2008). This feedback allows learners to identify and correcterrors in their decision making processes. As a consequence of these educational processes, learners’ decision making abilities shouldimprove in the specific problem domain of interest.

In traditional formulation of problems all the relevant information is revealed to learners in advance. This is far-removed from the realityof the complex issues that learners will face in the problem domains under consideration in this paper. Complex real-world situations arecharacterised by constant change. Prediction markets allow the creation of decision scenarios that closely mirror these conditions. Theyallow learners to re-evaluate and, where appropriate, modify their existing decisions in response to new information. Thus, by usingprediction markets to implement decision scenarios, this approach encourages learners to develop skills in analysing new information as itbecomes available and factoring it into their decision making.

Another factor that distinguishes complex problems is that relevant information is widely distributed. In these situations, all other thingsbeing equal, searching out additional information should improve decision making. Under these conditions, information literacy skills suchas information search, retrieval and evaluation take on a heightened importance. Participation in a prediction market enables the devel-opment of these information skills in two ways. Attentive learners will become more sensitised to information relevant to the decisionscenarios under consideration when they encounter it. It also prompts investigation and encourages learners to actively search out infor-mation relating to the decision scenarios.

P. Buckley et al. / Computers & Education 56 (2011) 418–428422

2.3.2. The affective domainThis approach is also beneficial to the affective domain of learning. The real-world problems that are the focus of the decision scenarios

will help ground the module for learners, improving student motivation. Understanding how academic theories are actually applied to real-world problems is a major challenge for many learners. Participation in a prediction market allows learners to take academic theory “out ofthe classroom” and use it to solve problems. This will help learners to see how theory informs practice in the real-world, thus increasingengagement in the educational process (Burke & Moore, 2003). The decision making processes that learners will engage in while partici-pating in a prediction market are similar in many ways to the decision making situations they will face in their professional careers. Studiesalso show that actively engaging in decision making in a problem domain improves student’s self-efficacy (Tompson & Dass, 2000).

Motivating students to engage in the learning process is another topic highlighted in the literature (Shih, Feng, & Tsai, 2008). Predictionmarkets activate the competitive spirit of students. Those learners who make accurate predictions will ultimately end up with more virtualcash then those who don’t. While this competitive element will have to be controlled and monitored carefully to ensure it doesn’t impactnegatively on the educational process, when properly harnessed the competitive element of prediction markets provides a powerful tool tomotivate students.

2.3.3. Benefits for academic staffThis methodology provides academic staff with a new tool to enable active learning. In particular, by leveraging IT and the unique

attribute’s of prediction markets, lecturers can initiate active learning when teaching large groups where other active learning strategieswould have prohibitive administrative burdens.

It provides lecturers with real-time, real-life examples that are familiar to learners and can be drawn upon in lectures and tutorials. Forexample, if a real-world event impacts upon the decision scenario presented in the prediction market, the lecturer can discuss this event.The impact of the event can be examined and discussed with reference to a decision scenario that learners will already be familiar with.

Another important benefit of prediction markets is that academic staff can use them as a real-time feedback mechanism. For example, ifa lecturer reveals information in a lecture that should affect learners’ decisions, they can then monitor the prediction market to observewhether learners assimilated the revealed information in the correct manner. This provides academic staff with a simple way of monitoringthe effectiveness of their teaching.

3. Prediction markets as a pedagogical tool: a case study

The Insurance Loss Market (ILM) is a prediction market designed specifically for an undergraduate module in risk management.This module introduces learners to the qualitative and quantitative skills required in risk assessment, risk control and risk financing.The ILM presents a set of decision scenarios drawn from the risk management industry. These scenarios are real-world problems thatallow learners to activate their skills in mathematical competency and qualitative risk assessment of current events. The ILM requireslearners to predict Weekly Insured Property Losses Estimates for California, New York and Florida. The Weekly Insured PropertyLosses are the total monetary amounts that all insurance companies will have to pay due to claims on the policies offered in thoseregions for that week. For the rest of this paper, we refer to the Weekly Insured Property Losses as insurance losses for the sake ofconvenience.

These decision scenarios were selected to fulfil a number of criteria. Most importantly, estimation of insurance losses is a skill directlylinked to the learning outcomes for the module. Insured property losses for a period are easily quantifiable after that time period haselapsed, which makes them suitable for use in a prediction market. Variations in regional insurance losses are influenced by phenomenasuch as hurricane activity, wildfires or sub-zero temperatures. Learners were thus required to undertake decisions under conditions ofuncertainty. However, this uncertainty occurs within a reasonably well-understood system, where only environmental events are likely toimpact on regional insurance losses. Data on these events is readily available and widely dispersed, which should have encouraged learnersto engage in information search.

The ILM was live for a ten-week period from Monday, September 14th to Friday, November 20th, 2009. This was most of the 2009 fallsemester. For each week in the period, the market opened on Monday at 9am and closed on Friday at 5pm. At the market close each Friday,students’ forecasts were evaluated against the insurance losses as provided by Insurance Data Provider, Xactware. The simplicity of the ILMinterface and data provided by Xactware concealed a sophisticated process that allowed for the provision of highly accurate data at the endof each week.

At 9am on the Monday of every week, each participant was provided withV5000 in notional ‘risk’ capital that they needed to allocate bypredicting insured property losses in each of the three US States. This was made operational by providing a series of loss bands whereparticipants could allocate their capital in the ratio they considered to be optimal. Fig. 1 provides an example of the ILM interface. It showsthe range of loss bands available to participants for the New York market for the week ending 9th of October 2009.

As trading activity commenced, the market dynamic produced a distribution of outcomes as participants evaluated historical infor-mation, such as recent weather patterns and recent insurance claims, as well as forward-looking information such as hurricane develop-ment, weather forecasts and potential hazards such as wildfires. There is wide availability of new information on weather-related hazardssuch as fires, windstorms and hail, as well as other relevant information. Market participants had to evaluate the importance of the availablehistorical information as well as the relevance of new information when making a decision.

The structure of a prediction market is such that the value attributed to a specific loss band will increase or decrease as the number ofparticipants investing in it fluctuates. As more participants select a specific loss band its value increases and simultaneously the value of allother loss bands will decrease proportionately. This structure also means that at any time throughout the execution of the market, learnershad the opportunity to re-evaluate the decisions they hadmade in light of newly revealed information and change their decisionwhere theydeemed it appropriate.

The activity in this market is similar to that carried out in the real insurance markets every day as insurance and reinsurance under-writers transfer insurance risks. The ILM website makes a number of information sources available to participants, including lecture notes,research articles on insurance and risk management and theoretical models that are applied in risk management and forecasting.

Fig. 1. The ILM loss bands.

P. Buckley et al. / Computers & Education 56 (2011) 418–428 423

Participants were also encouraged to search for other relevant information. Ultimately, success in the ILM is directly linked to the partic-ipant’s understanding of the available information, their ability to evaluate its relevance and their utilisation of it in the decision makingprocess.

As part of the overall implementation of the ILM, the problem difficulty was adjusted over time. The individual markets in the ILM startedwith very large losses bandswhichwere progressively tightened over a number of weeks. Eachmarket has ten distinct loss bands. In the firstweek of operation, the lowest loss bandwas “Losses less than 5million” and the highest loss bandwas “Losses greater than 45million”, withthe eight loss bands in between these two extremes changing in steps of 5 million. The loss bands were tightened in all the markets untilfifth week, when the loss bands ranged from “Losses less than 17 million” to Losses greater than 25 million”, with individual loss bandschanging in steps of 1 million. It became more difficult to select the correct loss band as the bands become tighter.

The ILM contributed 17% of the total marks for the module. Eight percent of the marks were assigned for participation. Students received0.8% of the total marks available for the module for each week they participated in the ILM. In order to gain participation marks for a week,ILM participants had to undertake at least one trade in each State. There was no upper limit on the number of trades they could make andthey could trade whenever they like throughout the week.

The other 9% were allocated on the basis of performance in the ILM, meaning that 0.9% of the marks for the module were available tostudents each week. At the end of each week, the ILM was resolved, converting student’s portfolios back to virtual cash. Students received100 units of virtual currency for every share they had in the correct loss band and 0 units for every other share. After this, students’performance is ranked in order from highest to lowest in terms of their virtual cash balance. It is important to note here that students’performance was measured relative to the rest of the class rather than in terms of absolute performance. Students whose virtual cashbalance placed them in the top 20% of all students for that week received the full 0.9% of marks available for that week, with fewer marksbeing allocated to lower percentiles.

This assessment procedure had a number of benefits. By allocating marks just for participation, learners were given an incentive toparticipate in the market. At the same time, allocation of marks for performance encouraged careful decision making and maintained thecompetitive element of the ILM, which is important in providing motivation. Resetting students’ virtual cash balance every week ensuredthat learners had an incentive to participate to the best of their ability every week and avoided a situation where students who made poordecisions at the start of the term would be subsequently disadvantaged for the entire term.

4. Research questions and methodology

This paper presents exploratory research that investigates the effect of prediction market participation on the cognitive domain oflearning. Investigating the benefits that participation in predictionmarkets will bring to learners in the affective domain is beyond the scope

Fig. 2. Average portfolio value after each week.

P. Buckley et al. / Computers & Education 56 (2011) 418–428424

of this paper. From the discussion in Section 2.3.1 three main benefits to learners in the cognitive domain of learning are proposed. First, it issuggested that learners’ decision making skills will improve due to participation in a prediction market. Second, learners should integratenewly revealed information into their decisionmaking processes. Finally, it is proposed that participation in a predictionmarket encourageslearners to engage in information search for unrevealed information pertinent to the decision scenarios.

To address these questions, this paper draws on the data that was recorded by the prediction market. Four hundred and five studentswere registered for the module. On average 347 users participated in the ILM per week. As part of the prediction markets operation, everytrade by every individual was recorded. For each individual trade, the date, time and volume of the trade was stored, as well as informationon how that trade changed the price of the associated loss band. By the end of term 115, 258 individual trades had been recorded by theprediction market.

At the end of every week, the market was resolved when the actual outcome of the decision scenarios was revealed. This operationoccurred simultaneously in all three markets in the ILM. After resolution of the market, all of a participants share holdings are converted tovirtual cash, in accordancewith the actual outcome of the decision scenarios for theweek in question. Learners are then ranked according tothe procedure described in Section 3. Performance data on every individual who participated in every week was recorded, providing a totalof 3469 performance records across 10 separate weeks.

Trading data and performance data can be cross referenced using a unique identifier assigned to every participant in the predictionmarket. For each individual trader, we can identify the trades they made, when they traded, what states and loss bands they traded in andtheir performance both absolutely and relative to their peers. For the purposes of analysis, only trading and performance data from weeksthree through ten was included. It was felt that data from the first two weeks was unreliable, as students were learning how to use thepredictionmarket itself during this period, rather than focussing on the decision scenarios. This data set was used to investigate the researchquestions posed.

5. Results

In section 3, we suggest that prediction market participationwill have three effects on the cognitive domain of learning. We suggest thatstudents’ decision making skills will improve. We suggest that students will incorporate newly revealed information into their decisionmaking. We suggest that students will actively search out relevant information.

Table 1Average portfolio value by percentile.

Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10

0 – 20 Percentile 10596.85 12141.03 12761.55 10242.66 9695.25 9772.70 9720.48 9704.3720 – 40 Percentile 5525.09 5486.15 4955.16 4743.37 5112.62 5350.49 4768.11 5436.3140 – 60 Percentile 4766.49 3958.00 3569.89 3638.01 4066.01 4559.50 3905.94 4625.5060 – 80 Percentile 4022.40 2162.72 3569.89 2174.02 2801.84 3543.90 2790.30 3723.9180–100 Percentile 2240.40 250.30 434.08 338.08 744.03 1576.31 1185.50 2112.13

Table 2Average number of positions held.

Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10

Average no. of positions 3.84 3.96 4.5 4.8 5.1 5.75 5.44 5.22

P. Buckley et al. / Computers & Education 56 (2011) 418–428 425

Two points are pertinent to this analysis. First, the group of students participating in the prediction market remained constantthroughout the execution of the market. The variances in student decision making observed over the course of the experiment cannot bedue to a student’s background or ability, since the population is essentially invariant throughout the case study. Second, prediction marketsenable continuous assessment on a very large scale, without imposing prohibitive administrative burdens. It is easy to imagine alternativeformulations of similar problems that would prompt similar behaviours in students. However, the unique scalability of prediction marketsmeans they are applicable in situations where other pedagogical tools are not.

5.1. Decision making

The first research objective of this paper is to explore the effect prediction market participation had on learner’s decision making skills ina risk management context. To answer this question, a distinction is drawn between how learners were assigned marks and how theirperformance is measured. The mechanism used to assign grades to students is discussed in Section 3. Students were assigned marks basedon their performance relative to their peers. While this is a suitable way of assigning marks to students, it cannot be used as a metric tomeasure decision making performance.

Absolute performance is bettermeasured by the value of a learners virtual cash portfolio aftermarket resolution. As an individual learnerhas more shares invested in the correct loss band, the value of their portfolio will rise. It naturally follows that as more individual learnershave more shares in the correct loss band the average portfolio value will also increase.

Fig. 2 plots the average portfolio value of all learners week byweek. There are two observable trends in this graph. The first major trend isthat the average portfolio value drops from week two to week seven. This drop in average portfolio is explained by our policy of graduallyincreasing the difficulty of the problem for the first fiveweeks through themechanism of narrowing the loss bands described in Section 3. Asthe problem difficulty increases the average portfolio value drops as would be expected.

In week six the problem bands were stabilised and did not change subsequently throughout the term. From week seven onwards, thegeneral trend in the average portfolio value is upwards. This upwards trend suggests improved decision making performance.

While this demonstrates that decisionmaking improved over themodule, the nature of a predictionmarket means that improvements inperformance could be driven by a subset of themarket participants who excelled at the ILM driving up the average portfolio value. In Table 1,learners are first ranked by performance marks over the entire term. This ranked list is then divided into five separate percentiles. The row“0–20 Percentile” represents the top 20 percent of students ranked by performance, and so on. For each percentile created, the averageportfolio value for all the learners in that percentile is calculated for each week.

Two trends emerge here. In all the percentiles except the highest, the pattern observed in Fig. 2 is replicated. The average portfolio valuedeclines until week six and then begins to rise. This is an indication that once the problem difficulty stabilised, learners’ decision makingbegan to improve over time, regardless of their decision making skill in the ILM vis-à-vis their peers.

Fig. 3. Trading volumes by day.

Table 3Number of Trades in California.

Week Start Date Monday Tuesday Wednesday Thursday Friday

3 28/09/2009 1863 1095 608 456 3084 05/10/2009 1187 696 484 725 6405 12/10/2009 1191 286 533 1121 3496 19/10/2009 1361 600 344 423 1807 26/10/2009 340 560 439 600 2718 02/11/2009 1135 663 959 535 3559 09/11/2009 1078 412 433 278 25010 16/11/2009 974 652 379 524 325

P. Buckley et al. / Computers & Education 56 (2011) 418–428426

The average portfolio value of the top 20% of performers has a different trend. Performance peaks during week six and then graduallydeclines from there. One property of a prediction market using an automated market marker is that the possible loss to the market sponsoris limited. This means that a prediction market is a zero sum game across all the participants. The rise in average portfolio value for the toppercentile of students early in the ILM is an indication that the there was an appreciable gap in the decision making of the stronger studentsearlier on in the module. However, as the ILM continued, the weaker students’ decision making began to improve. This reduced theadvantage the stronger students had and therefore reduced their average portfolio value.

A participant holds a position in the ILM if they hold at least one stock in a particular loss band when the market closes. In Table 2, theaverage number of positions held by all students’ across the three states in the ILM is shown. As the loss bands available to select fromgrow tighter, a rational decision maker will engage in risk diversification and hold more positions, weighting their investmentaccording to their assessment of the probability of the event occurring. Risk diversification is a risk management strategy, which aims toreduce the overall risk associated with a portfolio through investing in several different assets. In the case of the ILM, a learner engagesin risk diversification by investing in a number of different loss bands, rather than just investing in the one outcome that the learnerthinks is most likely to occur.

In the above data, an overall increase in risk diversification from the start of the ILM to the end can be observed. This is an indication ofimproved decisionmaking for two reasons. Firstly, it indicates increasingly rational behaviour amongst traders. Secondly, risk diversificationis a key theoretical concept that is explored in the module under consideration. The application of risk diversification is a clear indicationthat learners are taking theoretical concepts from the lectures and utilising them in their decision making process.

The data collected suggests that students make better decisions as the module progresses. We can observe two trends in tradingbehaviour and performance that support this statement. First, as measured by virtual capital at the end of a week, individual student’sperformance tends to improve once the problem difficulty stabilised. Second, we also observe that students engage in more diversificationover the duration of the module. Both of these observations support the suggestion that students’ decision making had improved as themodule progressed.

One obvious explanation for this improved performance is that students are being simultaneously exposed to the theory of riskmanagement as part of the module. It is reasonable to suppose that improved knowledge leads to improved performance. However, theconstructivist framework we are using to examine this pedagogical tool emphasises that the simple delivery of information is not enough toprompt learning and create knowledge. Learners must be given the opportunity to apply knowledge in a realistic context to fully enablelearning. Without the use of a prediction market as a pedagogical tool, in a large group such as the one under consideration, it would beadministratively prohibitive to provide an environment where students can apply that theoretical knowledge. Additionally, repeatedlyprompting students to engage with a dynamic decision making scenario also improves their decision making skills.

5.2. Information use

The second objective of this paper is to search for evidence in learners’ trading behaviours that indicates they are reacting to newlyrevealed information. To investigate this, a suitable information event needs to be identified. Every Thursday, data regarding losses in thethree states in the previous weekwas released to students. Recent historical data such as this should impact upon student’s decisionmaking.

The chart in Fig. 3 shows the average number of trades per day, broken up by state. In all three states, the trading behaviour is broadlyconsistent. The most active trading day is Monday, with the average number of trades per day declining as the week progresses.

However, a noticeable increase in the number of trades can be observed on Thursday, particularly in California and New York. Thisindicates increased trading activity on a Thursday, which is consistent with the behaviour that would be expected if learners were re-evaluating their decisions based on new information, in this case the disclosure of the previous week’s losses.

Table 4Number of Trades in Florida.

Week Start Date Monday Tuesday Wednesday Thursday Friday

3 28/09/2009 1804 406 670 416 3324 05/10/2009 1149 520 342 455 2575 12/10/2009 919 316 488 1005 3396 19/10/2009 1044 588 432 476 2157 26/10/2009 333 459 438 620 3358 02/11/2009 1155 607 861 524 2279 09/11/2009 1143 800 377 454 36310 16/11/2009 799 591 289 453 230

Table 5Number of trades in New York.

Week Start Date Monday Tuesday Wednesday Thursday Friday

3 28/09/2009 1623 597 552 414 2714 05/10/2009 1230 602 378 451 5105 12/10/2009 1345 310 459 1122 4286 19/10/2009 1571 717 358 1008 2627 26/10/2009 421 578 547 724 4658 02/11/2009 1600 681 641 826 3929 09/11/2009 1247 417 417 318 18810 16/11/2009 932 721 828 680 325

P. Buckley et al. / Computers & Education 56 (2011) 418–428 427

The evidence presented in the case study suggests that learners are dynamically incorporating new information into their decisionmaking as it becomes available. When information on the previous week’s losses is presented to learners, they use this information to adjusttheir decisions, as is evidenced by the upsurge in trading that occurs on the market.

This behaviour is a clear indication of active engagement by students. It is fair to say that learners will not heed or utilise informationpresented to them, unless it has a utility to them.Without the prompt of continuous assessment provided by the predictionmarket, studentswould have no incentive to notice, evaluate or apply newly revealed information. Furthermore, the nature of prediction markets allowsstudents to alter their decisions, in contrast to other possible solutions such as a poll. Prediction markets enable learners to evaluate andre-evaluate their decisions at any stage while the market is running.

It is easy to envisage other pedagogical initiatives that could prompt similar behaviours. For example, students could be asked to engagein an online discussion with peers and teachers. In such discussions, it is logical to suggest that students would similarly react to newlyrevealed information. However, while is it is possible to envisage other approaches that would prompt similar behaviours, the crucial pointis that many of these pedagogical mechanisms would not scale efficiently. In contrast, prediction markets, when deployed in the mannersuggested by this paper, do scale to very large groups.

5.3. Information search

The final objective of this research is to search for evidence to suggest that students are actively searching for relevant information toinform their decision making. To probe this question, events that have a number of particular properties must be identified. First, the eventshould be reasonably expected to impact upon the ILM. Second, the event should not have been revealed to students through traditionalacademic channels. It should be an event that learners would have had to actively search out and discover through non-traditional channels.

Events which fit these criteria areweather events which could be reasonably expected to affect insurance losses in the three states underconsideration. For the purposes of identifying such events, we selected events severe enough to be reported nationally by two of major newsnetworks in the US, in order to ensure that learners could be expected to have found out about these events given a reasonable level ofsearch. The two news networks chosen were CNN and ABC.

Two specific events occurredwhichmet our criteria over the time period in question. The first eventwas amajor stormwhich occurred inCalifornia on the 13th and 14th of October. The storm threatened to cause major flooding and land-slides in California, as reported by bothCNN and ABC.

The second major event was the formation of Hurricane IDA. The formal classification of Hurricane IDA was announced by the NationalHurricane Center on the 4th of November and was reported by both CNN and ABC.

Tables 3, 4 and 5 respectively show the trading data for California, Florida and New York. Each cell shows the number of trades thatoccurred in the market on the day in question for the given week. The highlighted cells in the table above are days in which the number oftrades conducted in a particular state are two standard deviations above the average number of trades in that state for that day.

The impact of both of these events can be clearly identified from the data above. An increase in trading occurred in both the Florida andCalifornia markets on the 15th of October. A similar increase in trading can be observed on the 15th of October in the New York market,although this does not reach the level of two standard deviations. An increase in trading in all three states is to be expected, as participantsre-evaluate their portfolios in response to the external shock.

The effect of Hurricane IDA can also be clearly identified in the market. A significant surge in trading can be observed on the 4th ofNovember in both the Florida and California markets. As with the storms in California, the increase in trading can also be observed in theNew York market, although again this does not reach significance.

There are three other significant spikes observed in the trading data. The increase in trading observed on Tuesday the 29th of Septemberand Friday the 9th of October is most likely attributable to learners responding to wildfires reported in California at that time. However,a number ofwildfireswere occurring in California at the time andwe cannot conclusively tie these spikes to anyone event. The last significantincrease in tradingwas in the NewYorkmarket on the 18th of November 2009. No significant event can be correlatedwith this trading event.

The data presented above suggests that students are prompted to engage in information search and actively seek out new information toimprove their decisionmaking. The large surges in trading that can be observed can be correlatedwith events in the real-world. This is againa function of the incentive of continuous assessment which is uniquely enabled in large groups by prediction markets. It is also encouragedby linking the decision scenarios to real-world events. The decisions learners are making are affected by real-world events. Improvedknowledge of real-world events will improve decision making, thus creating the incentive to actively search for information.

6. Conclusions

This paper makes a number of unique contributions. We position prediction markets as being a pedagogical tool which can be used tocreate a Rich Environment for Active Learning. We identify how they are particularly useful in promoting active learning in large groups.Weoffer a detailed case study describing how a prediction market can be deployed as a pedagogical tool in practice.

P. Buckley et al. / Computers & Education 56 (2011) 418–428428

In this paper, we have primarily focussed on the benefits of prediction markets as a tool for improving the cognitive domain of learning.Our exploratory research has demonstrated that learners’ decision making in a specific problem domain has improved over the module. Wehave identified trading behaviours that demonstrate learners will integrate new information into their cognitive framework and alter theirdecisions based upon this new information. This is a clear indication of active engagement by learners. Finally, we have demonstrated thatlearners are actively searching for relevant information that is not supplied to them in lectures and are integrating this information into theirdecision making processes.

Our analysis of our approach allows us to propose a number of benefits to learners in the affective domain. We believe that thepresentation of real-world problems will improve learners’motivation to study. We also believe that the competitive element of predictionmarkets provides a further incentive to improve motivation in learners. We believe further research to explore these issues is called for.

The scalability of prediction markets raises other interesting possibilities. Once operational, a prediction market can be used to deliverproblem scenarios to very large numbers of participants. This raises the intriguing possibility of creating learning communities that gobeyond traditional module boundaries. We believe it is possible to leverage prediction markets to create learning communities that includeparticipants from many different classes and universities. It may be possible to include professionals working in industry. The creation andmanagement of such a diverse community of learners raises many challenges, but also offers many potential benefits to all concerned. Webelieve prediction markets are a tool that can be used to unlock the potential of such diverse communities, and welcome research aimed atexamining the practical and pedagogical benefits and challenges of creating such communities.

References

Bednar, A., Cunningham, D., Duffy, T., & Perry, D. (1995). Theory into practice: how do we link? In G. J. Anglin (Ed.), Instructional technology: Past, present, and future (2nd ed.).(pp. 88–101) Libraries Unlimited.

Benbunan-Fich, R. (2002). Improving education and training with IT. Communications of the ACM, 45(6), 94–99. doi:10.1145/508448.508454.Berg, J. E., Nelson, F. D., & Rietz, T. A. (2008). Prediction market accuracy in the long run. International Journal of Forecasting, 24(2), 285–300. doi:10.1016/

j.ijforecast.2008.03.007.Bostock, S. J. (1998). Constructivism in mass higher education: a case study. British Journal of Educational Technology, 29(3), 225–240. doi:10.1111/1467-8535.00066.Burke, L. A., & Moore, J. E. (2003). A perennial dilemma in OB education: engaging the traditional student. Academy of Management Learning & Education, 2(1), 37–52.Christiansen, J. D. (2007). Prediction markets: practical experiments in small markets and behaviours observed. The Journal of Prediction Markets, 1(1), 17–41.Gibbs, G. (1992). Teaching large classes in higher education: How to maintain quality with reduced resources. London: Kogan Page.Grabinger, R. S., & Dunlap, J. C. (1995). Rich environments for active learning: a definition. Alt-J, 3(2), 5. doi:10.1080/0968776950030202.Grabinger, S., Dunlap, J. C., & Duffield, J. A. (1997). Rich environments for active learning in action: problem-based learning. Alt-J, 5(2), 5. doi:10.1080/0968776970050202.Gurney, B. (1989). Constructivism and professional development: a stereoscopic view. Retrieved from. http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/

ERICServlet?accno¼ED305259.Hadley, N. (2002). An empowering pedagogy sequence for teaching technology. Journal of Computing in Teacher Education, 19(1), 18–22.Hahn, R., & Tetlock, P. (Eds.). (2006). Information markets: A new way of making decisions. Washington D.C./Blue Ridge Summit PA: AEI-Brookings Joint Center for Regulatory

Studies. Distributed to the trade by National Book Network.Hanson, R. (2007). Logarithmic market scoring rules for modular combinatorial information aggregation. The Journal of Prediction Markets, 1(1), 3–15.Hayek, F. A. (1945). The use of knowledge in society. American Economic Review, 35(4), 519.Hogan, D., & Kwiatkowski, R. (1998). Emotional aspects of large group teaching. Human Relations, 51(11), 1403–1417. doi:10.1177/001872679805101104.Jonassen, D. H. (1994). Thinking technology: toward a constructivist design model. Educational Technology, 34(4), 34–37.Kolb, R. (1997). Understanding futures markets (5th ed.). Malden MA: Blackwell Publishers.Lebow, D. (1993). Constructivist values for instructional systems design: five principles toward a new mindset. Educational Technology Research and Development, 41(3), 4–16.

doi:10.1007/BF02297354.Lemak, D. J., Reed, R., Montgomery, J. C., & Shin, S. J. (2005). Technology, transactional distance, and instructor effectiveness: an empirical investigation. Academy of

Management Learning & Education, 4(2), 150–159.McAvinia, C., & Oliver, M. (2002). “But my subject’s different”: a web-based approach to supporting disciplinary lifelong learning skills. Computers & Education, 38(1–3),

209–220. doi:10.1016/S0360-1315(01)00080-X.O’Connor, P., & Zhou, F. (2008). The tradesports NFL prediction market: an analysis of market efficiency, transaction costs, and bettor preferences. The Journal of Prediction

Markets, 2(1), 45–71.Pennock, D. M. (2004). A dynamic pari-mutuel market for hedging, wagering, and information aggregation. In Proceedings of the 5th ACM conference on electronic commerce

(pp. 170–179). New York, NY, USA: ACM. doi:10.1145/988772.988799.Roettger, C., Roettger, L., & Walugembe, F. (2007). Teaching: more than just lecturing. Journal of University Teaching and Learning Practice, 4(2), 119–133.Rouet, J., & Puustinen, M. (2009). Introduction to “learning with ICT: new perspectives on help seeking and information searching”. Computers & Education, 53(4), 1011–1013.

doi:10.1016/j.compedu.2009.07.001.Servan-Schreiber, E., Wolfers, J., Pennock, D. M., & Galebach, B. (2004). Prediction markets: does money matter? Electronic Markets, 14(3), 243. doi:10.1080/

1019678042000245254.Shih, M., Feng, J., & Tsai, C. (2008). Research and trends in the field of e-learning from 2001 to 2005: a content analysis of cognitive studies in selected journals. Computers &

Education, 51(2), 955–967. doi:10.1016/j.compedu.2007.10.004.Simons, P. R. (1993). Constructive learning: the role of the learner. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), Designing environments for constructive learning (pp. 291–313).

Springer-Verlag.Stahl, G. (2004). Building collaborative knowing: elements of a social theory of CSCL. In J. Strijbos, P. Kirschner, & R. Martens (Eds.), What we know about CSCL and imple-

menting it in higher education (pp. 53–85). The Netherlands: Kluwer Academic Publishers. http://portal.acm.org/citation.cfm?id¼1011204 Retrieved from.Tompson, G. H., & Dass, P. (2000). Improving students’ self-efficacy in strategic management: the relative impact of cases and simulations. Simulation Gaming, 31(1), 22–41.

doi:10.1177/104687810003100102.Tsai, C. (2008). The preferences toward constructivist Internet-based learning environments among university students in Taiwan. Computers in Human Behavior, 24(1), 16–31.

doi:10.1016/j.chb.2006.12.002.Tziralis, G., & Tatsiopoulos, I. (2007). Prediction markets: an extended literature review. The Journal of Prediction Markets, 1(1), 75–91.Vygotsky, L. (1986). Thought and language (Translation newly rev. and edited/.). Cambridge Mass.: MIT Press.Wall, J., & Ahmed, V. (2008). Use of a simulation game in delivering blended lifelong learning in the construction industry – opportunities and challenges. Computers &

Education, 50(4), 1383–1393. doi:10.1016/j.compedu.2006.12.012.Wang, Q. (2009). Design and evaluation of a collaborative learning environment. Computers & Education, 53(4), 1138–1146. doi:10.1016/j.compedu.2009.05.023.Wiggins, G. (1989). A true test: toward more authentic and equitable assessment. The Phi Delta Kappan, 70(9), 703–713.Wolfers, J., & Zitzewitz, E. (2004). Prediction markets. The Journal of Economic Perspectives, 18(2), 107–126.Zantow, K., Knowlton, D. S., & Sharp, D. C. (2005). More than fun and games: reconsidering the virtues of strategic management simulations. Academy of Management Learning

& Education, 4(4), 451–458.