Information Sciences 264 (2014) 91–103

Contents lists available at ScienceDirect

Information Sciences

journal homepage: www.elsevier .com/locate / ins

An extensible micro-world for learning in the data networkingprofessions

0020-0255/$ - see front matter � 2013 Elsevier Inc. All rights reserved.http://dx.doi.org/10.1016/j.ins.2013.10.024

⇑ Corresponding author. Address: Pearson Austin Operations Center, 400 Center Ridge Dr., Austin, TX 78753, United States. Tel.: +1 623 238 3E-mail address: kristen.dicerbo@pearson.com (K.E. DiCerbo).

Dennis C. Frezzo a, Kristen E. DiCerbo b,⇑, John T. Behrens b, Mark Chen c

a Cisco, United Statesb Pearson, United Statesc Independent Developer, United States

a r t i c l e i n f o a b s t r a c t

Article history:Available online 30 October 2013

Keywords:Serious gameComputer networkingSimulation

This paper describes the rationale, implementation logic, and user data related to asimulation-based serious game in the domain of computer networking. The resultingmicro-world was created to provide rich and open user experiences that mimic importantaspects of the real world relevant to technical knowledge, social understanding, and theapplication of skills in the networking domain. Simulation software called Packet Tracerprovides a comprehensive micro-world authoring tool that allows the construction anduse of network micro-worlds while a game layer called Aspire overlays Packet Tracer tosimulate complex social interactions, requirements of problem formulation and solution,recovery from failure and other important skills needed for success in computer network-ing professions. These tools were designed from an ECD framework over a number of yearsand initial studies regarding their use are discussed.

� 2013 Elsevier Inc. All rights reserved.

1. Introduction

In this paper we discuss efforts to create digital environments to support individuals learning about computer networksand the practice of being computer network technicians, administrators, and engineers. Students and instructors are lessconcerned with what the learners know and can do in situations that are uniquely created for learning or assessment itself,but rather are concerned with the physical, affective, and cognitive reality of the world of work for computer networkingprofessionals. We attempt to provide the opportunity for students to experience this world and collect data for feedbackand decision support in the context of simulated micro-worlds that are part of a comprehensive instruction and educationprogram.

The work described here takes place in the context of the Cisco Networking Academy (CNA). CNA is a global program inwhich beginning computer network engineering and Information and Communications Technology (ICT) literacy is taughtthrough a blended program of face-to-face classroom instruction, hands-on lab experiences, an online curriculum, and onlineassessments [16,23]. Courses are delivered at high schools, 2- and 3-year community college and technical schools, 4-yearcolleges and universities, and non-profits. Since its inception in 1997, the Networking Academy has grown to reach a diversepopulation of about a million students each year in more than 165 countries.

A core motivation of this work is a focus on directly improving student learning by giving students and instructorsincreasingly detailed feedback regarding student knowledge, skills, and attributes in contexts that reflect those in which theywill need to apply the information outside the classroom. As such, the systems we describe here are neither ‘‘assessment’’

511.

92 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

systems nor ‘‘instructional’’ systems, but learning systems that break down the traditional barrier between instruction andassessment. As natural tasks become increasingly digital, the divide between the instructional experience and the assess-ment experience is unobtrusively closed [14]. As learners engage in what, for them, is a natural activity of game play, theywork through challenges at a given instructional level, acquiring new knowledge and skills, while information about theirperformance is collected and analyzed to provide feedback within the game and to teachers outside. As such, the distinctionbetween instruction and assessment fades away in the manner that occurs in the classroom of highly skilled teacher whosecycle of practice and observation are closely tied.

2. The state of the art

There have recently been two large, comprehensive meta-analyses recently completed examining the question of gamesand learning [42,10]. Both begin by looking at whether games were more effective than traditional classroom methods interms of learning, and both found that they were, with similar effect size estimates of 0.29 [42] and 0.32 [10]. These resultssuggest that the recent interest in creating digital games for learning is justified.

However, the term ‘‘digital games’’ covers a wide array of software and implementations. The research question of mostinterest may now be, ‘‘what designs work for whom in under what conditions?’’ The meta-analyses above begin to point toquestions and answers. Wouters et al. report that students learned more when: playing in groups than playing alone, gameswere supplemented with other instructional methods, and multiple training sessions were involved. In addition, the advan-tage of games over conventional instruction was largely seen when non-random assignment was used. This suggests thatthere may be factors that make particular students a good match for particular games. Although these factors remain tobe specified, the studies with non-random assignment to games appeared to capitalize on them through student, teacher,or researcher selection.

Clark et al. also summarized studies comparing ‘‘enhanced’’ games with learner supports, enhanced interfaces, and otheradvanced designs to those using basic designs and found an overall effect size of .29 favoring the advanced designs. Unfor-tunately, there were too few studies of any given enhancement to determine which enhancements seem to work best underwhat conditions. Much of the current research on games for learning focuses on further developing complex enhancementsand affordances. The following four topics are a representative (but not exhaustive) view of the work being done at the fore-front of complex learning environments in an effort to further improve learning outcomes.

2.1. Adaptive/intelligent environments

Adaptive environments are those in which events that happen in the microworld depend upon the learners’ actions andestimates of their knowledge, skills, and attributes. Environments can be adaptive in at least three ways: presentation (lookand feel), problem sequencing, and problem-solving support [25]. Problem solving support typically involves providing hintsand feedback based on player results. Work in the field of intelligent tutors that provide step-level feedback and requireimmediate error correction has shown that this method improves the outcomes of learners [17]. However, research withgames suggests that players rarely ask for hints, and whether players attend to hints appears to be a complex interactionbetween when in game play it occurs, what type of hint it is, whether it is in response to a correct or incorrect move, anda players’ general attitude toward hints [34].

To create a system in which the game adapts to the learner in ways that do not interrupt game play, the system must beable to assess levels of prior knowledge, learning progress, motivational states, and preferences. In the ELEKTRA game [25], a3-D adventure game to teach eighth grade optics, the game engine uses players’ actions to update the probability distribu-tions of competence states. Based on the estimated probabilities, hints and supports are selected from a set of possible inter-ventions. Further, if a player does something that is inconsistent with the estimated competencies (as often happens whenplayers are exploring an open game environment), an assessment clarification, or activity designed to help provide assess-ment information, can be introduced. The game also contains competence activation, competence acquisition, and motiva-tional adaptations to offer based on learner states. This use of probabilistic estimates of a variety of constructs and change togame states ‘‘on the fly’’ is representative of current work in this area.

2.2. Intelligent virtual agents/mentors

Related to the research on adaptive environments generally is research on intelligent virtual agents in games. Intelligentagents, or computer-generated characters that adaptively interact with the learner, often have the added challenge of beingable to respond appropriately to natural language input. In order to build these agents, researchers have used complex sta-tistical models to investigate dialogue structure and tutoring effectiveness in human pairs [8], while others have sought toconstruct computational models of socially normative conversational behavior [35].

Art Graesser and colleagues implemented a tutoring system, AutoTutor, in which agents hold conversations with learnersin natural language [21]. After success as a tutor, the technology was expanded in the game Operation Aries, in which thehuman learner has ‘‘trialog’’ conversations with a student agent and a tutor agent in the process of detecting flaws in scien-tific studies (placed there by aliens attempting to take over the world). The agents respond to the natural language inputs of

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 93

the learners to guide them first in training modules where they learn about common flaws, then through competitive playagainst the student agent, and finally into interrogating aliens about the flaws in their research [29].

A further expansion of AutoTutor has been through the creation of AutoMentor, that acts as a mentor responding viaemail and chat using speech act classifiers, newness, relevance, epistemic network analysis, and state transition networks[41]. AutoMentor is currently embedded in the game Land Science, in which players are interns at an urban planning firmwhere they must develop land use plans for ecologically-sensitive areas balancing stakeholder interests. As interns, playershave mentors, who were originally human, to guide them through the tasks. AutoMentor has been found to adequately servethe mentoring function, allowing for scaling of the game beyond what was possible when human mentors were required.

A different use of intelligent agents is to position them as learners to be taught by the players. For example, in the militarycontext, the game NERO requires players to train a team of 50 agents skills for military combat. Players devise training exer-cises to prepare their agents. The training exercises inform neural networks that determine the agents’ current states ofknowledge. When the player decides performance is satisfactory, the team can be sent into battle [28]. This approach tothe use of intelligent agents, particularly when players can interact with them as they do with humans, presents a rangeof opportunities for new roles and interactions.

2.3. Role of affect in learning

In addition to cognitive factors, affective states impact learning outcomes. Baker and his colleagues [3] examined the im-pact of boredom, frustration, confusion, engaged concentration, delight, and surprise on learning outcomes. They reportedthat boredom was common across environments and associated with poorer learning. Confusion was also common, butwas linked to learning gains, although learners may require help regulating their confusion. These findings led the authorsto recommend methods of detecting affective states within learning environments so they can be addressed in a way thatpromotes positive affect.

Research teams have shown success analyzing facial expressions with Hidden Markov Models to uncover patterns relatedto emotion [22]. Emerging research also focuses on using text, body language, and multimodal systems to accurately classifyaffective states [9]. However, while there has been success in identifying emotion, there has been much less documentedintervention attempting to change affect. Work remains to be done regarding how to overcome boredom, manage frustrationand confusion, and perhaps to increase the frequency of surprise and delight.

2.4. Stealth assessment

Shute coined the term stealth assessment to describe the process of using information from learners’ actions with digitallearning environments to make inferences about their knowledge, skills, and attributes [39]. Rather than intrusive assess-ments that are not connected to the learning environment, stealth assessment unobtrusively gathers data from students’every day interactions with the instructional environment. Proponents of stealth assessment argue that they allow for theassessment of skills that are difficult to assess with traditional assessment tools. They also provide streams of data that allowfor the examination of the problem-solving process rather than simply the end-product.

A number of projects have sought to assess science concepts in game-like environments [1,10]. Shute and colleagues havedesigned a game called Newton’s Playground to facilitate learning and assessment of players’ qualitative physics under-standing, persistence, and creativity. Validity studies have shown that the estimates of these constructs from game activitycorrelate as expected to outside measures [40]. In addition, playing the game for 4 h across 1.5 weeks led to improved under-standing of physics, demonstrating how both learning and assessment took place in the same activity by the learner.

3. Principles for design

The previous section outlined promising lines of research into microworlds. In this section, we describe the principlesundergirding the design of these learning systems. These follow primarily from recognition of the social nature of educa-tional and learning activity [4] as well as the need to make principled arguments from what we observe to the inferenceswe make about students. The work described is derived from a number of principles of instructional and assessment designwhich we discuss below.

3.1. A focus on skills ‘‘in the wild’’

Technological limitations on interacting with students, providing experience and capturing relevant data, especially inthe classroom, often lead to dramatic truncation in the goals and aspirations expressed in learning outcomes. Sometimesthe truncation makes its way back to the original conceptual frame of the problem so that designers do not even considerthe target activity to which we wish to infer, but stop at distal and common formulations that may have severe inferentialweaknesses for claims of generalization or transfer. To counter this, we encourage specification of the claims we want tomake about activity ‘‘in the wild’’ [24]. That is, we try to understand the claims as contextualized in practice outside ofthe learning environment. Most practitioners would argue that all good learning conceptualizations should do this, but

94 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

likewise many experienced practitioners will confide that one’s ability to think beyond the constraints of their authoringenvironment is often quite difficult.

As an example of skills in the wild, the Foldit game requires players to learn to fold real proteins using direct manipula-tion tools and user-friendly versions of algorithms. Players work collaboratively and competitively to build on each other’ssolutions and strategies such that a collection of players, most with limited science background, uncovered previously un-known structures that have been subsequently published in first tier science journals [12]. Players are engaging with realscience problems with real results, but in the context of a multiplayer game.

Creating an artificial environment to allow action consistent with living ‘‘in the wild’’ is an important aspect of educa-tional technology [7]. An open, simulated environment allows for a full range of knowledge, skills, and attributes to evolveover time including: recognition of cues regarding problem situations, formulation of problems, recovery from mistakes,understanding and responding to environmental feedback, and other complex emotional and information processing skills.Shaffer’s work on epistemic games that place students in urban planning and engineering offices with authentic tasks drawstudents into real scenarios [38]. The ability to digitally create authentic experiences gets to heart of the affective decision tobecome and remain a professional.

The focus on skills in the wild extends to the types of responses learners make in the system. By using a flexible scoringsystem behind an open ended activity presentation system, the user flow and evidence identification goals of an activity canboth be met. For example, the authentic tasks of emails, instant messages, and reports in the epistemic games are analyzedwith epistemic network analysis to make inferences about players’ skills, knowledge, and professional identities [37]. Tech-nology shifts the burden of inference from presentation to evidence identification. That is, in the world of multiple choiceexams, the work is in creating useful items in the constrained space. Identifying evidence from these items is simple. Withtechnology, the space for presentation of tasks is much larger and the difficulty is shifted to how to identify appropriate evi-dence from the bounty of responses in that environment.

3.2. An evidentiary perspective

We have found the principles of Evidence Centered Design (ECD) [32] and its logical bases [30] extremely useful in ourwork in automated classroom assessment [6]. First, ECD emphasizes the logical form of the inferential argument and sug-gests careful consideration of the train of reasoning from activities learners complete to skills, knowledge, and attributesof interest. Second, while many discussions of ECD emphasize this important evidentiary aspect of assessment we equallyfound benefit from ECD’s detailing the elements of assessment delivery in a way that is sufficiently abstract as to includehuman language [33], a broad range of classroom activities [31], games [3], and simulation in general [19]. The four-processmodel of the cycle of activity completion, scoring, accumulation into a profile, and selection of the next activity provides aflexible conceptualization of the activity of an instructional and assessment system.

3.3. An ecosystem perspective

If we take the ECD perspective seriously, we are led to deeply consider the purpose of our activity and how different goalsfor feedback may lead to different and multiple forms of interaction with the learners. The ecosystem approach attempts tounderstand the broad range of stakeholder (e.g., learner, instructor, school system, etc.) needs and tailor individual activitiesto specific needs, but also create a design across activities and events to ensure all needs are met appropriately. Consider-ations such as physical location in formal versus informal learning environments and the technology itself (e.g., handhelddevices versus pc’s) may influence interactions. In addition, factors such as mandatory assignment of play by teachersmay have unforeseen effects on motivation [23].

For example, in Networking Academy classes, instructors have varying goals for assessment activity, from a quick under-standing of whether students are on track to knowing whether they are prepared to take a certification exam. Behrens et al.[5] described six different types of assessment activities that were undertaken in the Networking Academies at that time(Quiz, Module Exam, Practice Final, Final Exam, Voucher Exam, Practice Certification Exam) in terms of six different featuretypes (organizational purpose, instructional purpose, grain size of feedback, grain size of claims and tasks, level of taskcomplexity, level of task security). Variation in assessment activity goal led to different patterns of design features affordedthose activities. For example, quizzes were designed to provide small grain size feedback with low security while final examsare designed to assess higher grain claims with larger grain size feedback and higher security. Practice certification examswere created in alignment with the professional certification exam given to examinees under third party certificationconditions.

While in some ways such an enumeration seems commonsensical, it is a departure from many assessment formula-tions that use ‘‘the test’’ as the unit of analysis. In such an approach, purposes aligned with specific assessment goalscan be missed and assessment activities (items or tests) may be developed with one purpose in mind, which are theninappropriately applied in other contexts. The same is true in understanding the ecosystem in which learning tools areused, which includes everything from use in whole class demonstrations, in-class guided and independent practice, home-work, and assessment.

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 95

4. Packet Tracer: an integrative micro-world authoring tool for computer networking

To reach the assessment and instructional goals for promoting knowledge and skills related to designing, configuring andtroubleshooting networks, the Networking Academy team created and implemented an extensible micro-world authoringtool called Packet Tracer [11,13], which then also became the foundation of a simulation-based game. While Packet Tracermay at first blush appear as a network simulator, there are a number of features of the software that go beyond simple sim-ulation of network environments. Frezzo et al. [18] discussed the affordances at 5 different levels; since 2009 these affor-dances have evolved and are summarized in Table 1.

Level 1 is the level of machinery for the simulation. Here network traffic is simulated to move across devices based on thestate of machines simulated on the network. The simulator can handle networks consisting of hundreds of devices using thevast majority of entry-level program commands and underlying logical protocols (Fig. 1).

The second level of the system concerns the user experience which is built primarily out of variations of a representationof the network using a network diagram as illustrated by the diagrams of a router connecting three computers and a hubconnecting another three computers on the left side of Fig. 2. Additional representations allow for visualization of movement

Table 1Affordances of Packet Tracer software.

Level Experiential goal Affordances

1. Simulation ofcomputernetworks

Allow experience and feedback from world withverisimilitude to real world in key areas

Comprehensive Network Simulator recreates behavior ofnetworks and devices

2. Interactioninterface

Support understanding, exploration and manipulation of themicroworld

Intuitive user interface based on primary representations in field(network topology diagram, images of devices and cables,command line interfaces, GUIs)

3. Authoringinterface

Support re-use, portability, contextualization and use forexplanation

Authoring interface. Save networks and device states. Createstories and notes in local languages

4. Assessmentinterface

Support scenarios extended to assessment scoring andfeedback.

Create and save ‘‘correct answer’’ network final states andcorresponding activities that support automated scoring; createinitial network conditions and choices of interface elements tolock. Add instructions, timing and network connectivity testingcriterion

5. Variablemanager andscriptmodules

Support Platform based re-use and flexibility, extendingscoring to a wider range of realistic design, configuration,and troubleshooting tasks

Create isomorphic and variant experiential and assessmentpatterns automatically accounted for in assessment authoring.Examples include variations hypothesized to be isomorphicincluding topology orientation, device names, and deviceaddresses, or ability to dynamically inject problems into thenetwork

Fig. 1. Palette of devices and protocols for network modeling, visualization, inquiry.

Fig. 2. A network diagram with visualization of multiple data packets.

96 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

of data across the network that would not be readily visible from real equipment (represented by the ‘‘envelopes’’ of data inFig. 2) and access and configuration of devices.

The third level is a core aspect of the system which sets it apart from many simulators in that the end user (student,instructor, assessment designer, or anyone else) has access to the authoring tools to create their own network micro-worldsin order to learn, experiment, practice, observe or participate in assessment activities. This is part of a so-called ‘‘ActivityWizard’’ that lets users author activities and scenarios. Multiple files associated with each micro-world can be saved andshared so that Packet Tracer is not simply a simulator, but rather a simulation based micro-world authoring tool. This pro-vides learners (whether they are instructors, students or assessment designers) to explore, experiment, and experience acomplex array of possible simulated real networking worlds in order to ‘‘strives for the emergence of consciousness and crit-ical intervention in reality.’’ ([11], p. 68).

To support the construction of specific inferences, most aspects of the interface as well as access to individual devices ordevice features on the network can be locked by the author before subsequent distribution of the network micro-world file.Accordingly an instructor (or student or assessment designer) can author a micro-world with assessment scoring features(the next level) and distribute it in a manner that restricts the recipient from certain activities that might compromisethe instructional or assessment goals of the activity.

The fourth level of the system is also part of Activity Wizard and concerned with assessment authoring. The Activity Wiz-ard supports the specification of a feature network (the ‘‘answer network’’) that will act as a key against which a learner cre-ated network can be compared. The activity wizard scans the components of the authored network (modeled features caneasily reach into the thousands) and makes a list of all devices and their features and presents them to all the author tochoose which features and feature values will be compared to the learner network after the learner submits the work.Fig. 3 illustrates this interface as it existed in the early version of 2009. In this case, the assessment author was interestedin assessing students’ port configuration, so the port elements for various devices are checked for inclusion in scoring whileother elements are not. Additional facilities exist for the authoring of specific end-state tests of connectivity, for examplewhether a test message called ‘‘ping’’ travels back and forth across the network, that are common requirements in network-ing activities. This allows testing proper connectivity between two devices on the network (‘‘is the network working’’) orspecific failures of communication that may be desired (‘‘can we keep all X out of Y’’) for security or other network manage-ment reasons. This connectivity test mimicks the ‘‘instructor standing over the students shoulder’’ practice of giving feed-back in real classroom situations.

The fifth level of the hierarchy of affordances consists of special support for the automatic generation and tracking ofisomorphs and variants by randomly or using other algorithmic means to seed features into the network (also part of theActivity Wizard, called Variable Manager and Script Modules). This allows the real-time generation of networks that are sim-ilar in surface structure but vary in the topological orientation, device names, IP addresses, and other features that may or

Fig. 3. Interface allowing authors to select components of a network to score.

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 97

may not affect difficulty. These features are important both for security and for generalization. By re-invoking the micro-world repeatedly, learners have access to repeated examples of a problem structure without identical work to be accom-plished. Explicit support for the ECD design-oriented language (work product features, observables, compoundobservables, proficiency estimates, reporting variables) was added in which a macrolanguage could be used to extract, scoreand combine a chain of mathematical inference. Authors start from student work product features, identify evidence to ex-tract, and program elementary evidence accumulation algorithms. An elaborated scripting interface has been made availablethat allows more direct access to simulator data by allowing the interaction of java script based programming to interact ascode fragments with the data layers of the simulation. This allows a very wide array of possible scoring and observation rou-tines in the simulation system by the assessment authors.

Packet Tracer is embedded throughout the curriculum in the Networking Academies. To assist in attaining basic network-ing skills, students undertake Packet Tracer activities to learn, practice and explore networking concepts by clicking on linksin the curriculum that launch the Packet Tracer software with the micro-world instance associated with that curriculum sec-tion. Many chapters end with a Packet Tracer Skills Integration Challenge, allowing practicing of all skills accumulated to thatpoint in the course. As their name suggests, these challenges allow learners to complete more complex tasks that requireboth review of learned skills and their combination in new ways. Not surprisingly, these are also more authentic tasks thanthose involving a one-to-one pairing of claims to tasks. In addition, at the end of each semester, a Packet Tracer Skills BasedAssessment is taken as part of the evidentiary portfolio for the instructor and student. The output of that activity is placed inthe grade book including device logs, final configurations, lists of observables, scores and proficiency estimates, available asfeedback to instructors and students. Because students have been using Packet Tracer throughout the courses, the end-of-course exams avoid the complication of construct irrelevant variance based on novel interface experience.

5. Aspire: extending Packet Tracer to professionalization

While Packet Tracer’s affordances allowed for exploration and interaction with a simulation of the physical and logicalaspects of the computer devices and networks themselves, there was a recognition that the skill set required for successfulactivity in the profession included a broad array of skills associated with social interaction, interpretation, and decision mak-ing as we would experience in the workplace. To more completely support learners’ ‘‘critical intervention into reality,’’ thesystems would need to be extended to provide an electronic reality of the social and business issues typically encountered inaddition to the technical concerns already modeled in Packet Tracer. As Jim Gee states,

A good instructional game, like many good commercial games, should be built around what I call ‘‘authentic profession-alism.’’ In such games, skills, knowledge, and values are distributed between the virtual characters and the real-worldplayer in a way that allows the player to experience first-hand how members of that profession think, behave, and solveproblems ([20], para. 1).

This argument is similar to that made by Shaffer in his work on epistemic games, which he argues help players think likethe professionals, giving them tools they need to participate in these communities [38]. In order to create this experience fornetworking students, a broad range of business and entrepreneurial adventures were authored and a game called Aspire was

Fig. 4. Internet café venue in Aspire game.

Fig. 5. Game board for Aspire.

98 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

created in which players are entrepreneurs starting a small networking business. It is built on Packet Tracer, allowing playersto bid on networking contracts for different businesses throughout the city, and then complete them using the simulationtools in Packet Tracer. For example, one contract asks players to configure four computers for Maria, who is setting up aninternet café (see Fig. 4).

Players in the game face real-world challenges such as juggling multiple projects at the same time, deciding whether tovolunteer computer services for needy organizations, and when it makes sense to take out a business loan. Actions such asnot buying maintenance contracts for refurbished computers have consequences later in the game when those computers

Fig. 6. State diagram of finite state machine from Aspire that depicts the logical dependencies in computer agent activities.

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 99

fail and customers need them fixed. Players’ actions are tied to three technical scores (configuration, troubleshooting, andphysical labor) and three business scores (business sense, money management, and reputation).

Aspire’s game board (see Fig. 5) includes multiple locations, including a bank to take loans, a store at which to buy net-working supplies, a learning center, and the player’s home. Most of the locations, including a combination Laundromat/Inter-net café, a medical office, a school, and a food bank serve as venues in which players complete contracts to configure andtroubleshoot networks. Throughout the game, players interact with clients and other characters via their (simulated, in-game) smart phone. These characters help drive the action in the game, from determining whether a hint is needed to con-gratulating players on a job well done.

Behind each client in a venue is a finite state machine (FSM) that acts as the ‘‘brain’’ of the client, and an overall FSM thatrepresents the player’s progress in the game (see Fig. 4). A FSM is a model used by computer programs to describe the statusof a ‘‘machine’’ (in this case the clients and the overall game all in parallel). Any state machine is in one state at a time, but atany one time, there are more than 10 state machines running in parallel. The parallelism lets the game introduce distracters(phone calls, opportunities, dilemmas) at controlled ‘‘random’’ (not always at the same point in the game) times and allowsthe user choice in contract paths. These parallel state machines can change states (‘‘transition’’) given a triggering event. Inthe game, the state of a contract, for example, could be ‘‘no hints,’’ but when the timer hits 2 min (triggering event) and theerror on the device remains, the contract transitions to ‘‘give hint.’’ Because the characters are acting like agents waiting forspecific events to occur, the experience of the learner is that the environment is dynamic with random elements (randomonly because they do not know the rule structure).

The overall FSM monitors aspects of the scoring model in the game and based on the values, determines what happensnext. It essentially asks things like, ‘‘Did the player configure the network correctly? If so, what should I say next? If not, do Igive a hint yet?’’ This represents a level of adaptivity in the system that is agent based rather than procedural.

The scoring engine in Aspire based on the scoring engine from the Packet Tracer simulation tool (as described above). Itallows subject matter experts/game authors to define the aspects of the devices and network configuration that are impor-tant to observe. A simple example might be to observe whether the work product feature named ip_address1 is equal to192.168.1.1. However, the scoring model also allows specification of ranges of correct events (e.g., does ip_address1 equalsome number between 192.168.1.1 and 192.168.1.126?), allowing for more open-ended problems and solutions. In the con-tract laid out in Fig. 6, there is a circle that says fire. There are in fact three paths, including one in which the player ignoresspecific instructions from his/her boss, which result in the player being fired. The entire contract is mapped out in the statemachine, laying out the multiple paths a player could take from the time they are presented with a contract to the time it iscomplete. Finite state machines are, by definition, finite; they only contain a limited, defined number of states, meaning they

100 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

will not support completely open-ended game applications. However, in Aspire, combined with the Packet Tracer scoringmodel, they allow for a degree of adaptivity in response to player actions and considerable replayability.

6. Research summary

6.1. Use and satisfaction

Although evidence of validity and learning is important for a game, it is also important that teachers and students like it. Ifthey do not, even the most carefully designed game would not get played. Instructor satisfaction was gathered during aninitial testing period. On average, instructors (n = 50) gave a 4.1 (out of 5) rating to the game. An open-ended commentbox elicited various comments from instructors exemplified by the following, ‘‘Good game to verify what the students havelearned, to help them apply the theory to real life problems and situations.’’

In the first 6 months of release, more than 12,000 students played Aspire. Depending on how they progress, players haveup to three chances to report on their satisfaction with the game. At the end of their first game session they get a survey(level 1 unless they progressed to level 2). They also get a survey after game sessions in which they get to level 2 for the firsttime (there are two levels in the game) and after they win the game. Examination of the satisfaction ratings and their com-ments revealed an interesting pattern. Satisfaction on the level 1 survey was relatively lower, it peaked on the level 2 survey,and then dropped again on the final survey. Reading the comments, it was clear that respondents on the level 1 survey werefrustrated by some user interface elements and the lack of help available. Respondents at level 2 liked the game and theircomments reflected this, writing things like, ‘‘The Aspire game brings what you learned in the class, into a realistic expec-tation of what may be in ‘real life’.’’ Finally, those who won the game were again somewhat less satisfied. Their commentsindicated that they were disappointed that there was not more content and levels for them to play. By looking at satisfactionlevels over time in this way, we were able to identify multiple issues that were affecting satisfaction. The user interface wasmodified and more hints and scaffolding were provided in the next version of the game and other editions with expandedcontent were released.

6.2. Validity evidence: response process

While the capability of games to motivate is fairly well documented [13], their ability to encourage players to utilize par-ticular cognitive processes, access particular prior knowledge concepts, and/or apply given procedural skills to solve prob-lems is less clear. In the world of assessment, this question of whether a task is completed using the processes the authorsintended is the essence of substantive validity [27] or evidence based on response process [2]. For example, a student whocompletes a math game by using algebraic processes to find the answers to the problems is using the intended processeswhile the student who uses brute force trial and error to find the solution is not. If the students using brute force are lucky,or if there are game clues that hint at the right answers, they may end up with game outcomes similar to those who haveused math processes to solve the problems. When we make inferences about students’ abilities from the outcomes of thegame, we need to ensure we understand the processes that can lead to those outcomes.

DiCerbo et al. [15] sought to determine whether Aspire game players used a cognitive troubleshooting process in com-bination with domain-specific knowledge and skills to solve problems in the game. They recorded pairs of students playingthe game (6 novice pairs and 5 advanced pairs) and coded both the recorded discussions and game play for problem solving/troubleshooting and specific content knowledge. Troubleshooting is a type of problem solving and it was expected that inorder to solve the problem, students would need to identify the fault symptoms, diagnose the fault, and generate and verifysolutions. These steps were indeed observed across student pairs. In addition, use of domain-specific content knowledge wasrequired to complete the task. All of this confirms that students did use the intended cognitive processes during game play.No instances of successful completion without these processes were observed. In addition, performance of more advancedstudents was differentiated from performance of novice students. The advanced students solved problems more quickly be-cause they were more likely to diagnose the correct fault initially, resulting in fewer cycles of diagnosis and solution gener-ation and verification. This type of research is particularly important in context rich environments in order to determine theextent to which use of construct-irrelevant cues can result in successful outcomes.

6.3. Validity evidence: internal structure

A second type of validity evidence discussed in the assessment literature is evidence regarding the internal structure ofthe assessment [2]. Internal structure refers to the relationship among the things being measured in an assessment. So, if wesay a particular assessment measures three subskills and particular activities are aligned to those subskill, does the data con-firm those relationships? We do not often think about validating internal structure of scoring in games and analysis of thescoring structure in Aspire has not been examined. However, if we want to give feedback about skills from how players con-figure networks, we need to understand this structure better, and it has been examined closely in Packet Tracer Skills-BasedAssessments (PTSBAs). These assessments are delivered on the simulation engine that also underlies the Aspire game andrequire students to configure and troubleshoot networks as the Aspire game does.

Fig. 7. Sociograms for two players with the same score. Each node represents a device and command issued.

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 101

Rupp et al. [36] recently examined the product and process results from one particular PTSBA. The assessment askedlearners to perform basic configuration and troubleshooting tasks to establish communication between two computers, arouter, and a switch. The exam consisted of 36 primary observables (elements of the completed network configuration thatare evaluated and scored) that the assessment authors combined into four performance components that then combinedinto a total score. These four components are: device connection, basic device configuration, IP addressing design and con-figuration, and verification and troubleshooting. The researchers examined the dimensionality across the 36 primary observ-ables using exploratory and confirmatory factor analysis methods, Diagnostic Classification Models (DCMs), and BayesianNetworks (BNs).

In sum, descriptive, exploratory, and confirmatory statistics painted a mixed picture of the relationship between scorevariables at different levels of the postulated scoring hierarchy. On the one hand, descriptive analyses and confirmatory anal-yses using DCMs with a linear hierarchy and Bayes nets suggested that there is some evidence that the four scoring dimen-sions are defensible, particularly for a sample collected under summative conditions. On the other hand, the comprehensiveBayes net model did not support this conclusion for a formative subsample. Moreover, results from the exploratory modelssuggested that certain pairs, triplets, or higher-order tuples of primary observables hung together in a relatively stable man-ner but not necessarily in the way that the operational scoring structure would suggest. The researchers then worked withsubject matter experts to investigate alternative scoring models with differing combinations of observables but that stillmade conceptual sense. Based on these efforts, the researchers and assessment authors were able to arrive at a scoring struc-ture might be changed to better reflect the relationships among the observables.

6.4. Use of process data

Digital simulations and games produce two kinds of data: product and process. Product data reflects the end result of thedigital interaction, for example: final game scores, completed problems, or configured networks, as investigated above. Pro-cess data is the recording of how the player got to that final product. This product scoring information is important for sum-marizing the final results that students produce. However, it is also often important in formative assessment situations tounderstand how students came to arrive at a particular answer. The initial investigation of student process data is oftenhighly exploratory and may or may not have theoretical grounding. DiCerbo et al. [16] discussed how visualization tech-niques can help identify patterns in these data, for example, with sociograms (Fig. 7) can be used to examine the paths indi-viduals took through a task.

Rupp et al. [36] demonstrated that 11 students who all earned scores of 94 on the PTSBA described above had markedlydifferent processes. For example, the number of commands used by these 11 students to arrive at the final solution rangedfrom 51 to 106. The authors examined the log files using descriptive analysis of number and type of commands, sociogramsto represent command sequences, and state transition networks. They also employed statistical natural language processingtools such as stemming and tagging. In the end, a number of metrics consistent with the concept of efficiency emerged, andthe researchers demonstrated that even students with the same final results could arrive there with markedly different effi-ciency. Ultimately, it is hoped that patterns discovered can be translated into information to help digital systems, designers,teachers, and students make decisions about student learning.

7. Conclusions

In this paper we discuss the rationale, foundational logic and use of the Packet Tracer simulation and Aspire game aslearning micro-worlds. We believe the highly integrated nature of the experience with the curriculum and its rich social

102 D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103

and technical simulation offer learners valuable opportunities for understanding and improvement. Having built the systemswith an eye toward flexible and automated scoring from the ground up, the systems offer extremely high re-use for students,instructors and development teams. Weaknesses in individual assessment activities in Packet Tracer (exams, etc.) can be cor-rected through authorable changes by assessment personnel without the involvement of technologists. Because of the com-plexity of Aspire based simulation, modifications in this system typically require changes represented in program code.Nevertheless, the flexibility of the system provides a sandbox for data collection and refinement centered around specifictasks and conceptualizations without having to recreate an artificial environment each time.

While the richness of the new data available provides challenges for authoring and analysis, the scoring system can workat a number of levels of complexity that can allow for production in simpler strategies (e.g., activity wizard based networkcomparison) while research is undertaken regarding newer strategies for scoring. The simulation basis allows us to impactequity: all students in the Networking Academy program have some form of access to computers and hence to feedback fromthese tools; the game basis allows us to address the common, powerful learning modality of games ‘‘where many of our stu-dents are.’’ From Freire’s conception to The Power of Their Ideas [26], we envision creating systems to help instructors andstudents navigate a digital ocean of feedback supporting the emergence of their networking professionalism.

References

[1] J. Asbell-Clarke, E. Rowe, E. Sylvan, Working through impulse: assessment of emergent learning in a physics game, in: Paper Presented at theGames + Learning + Society 9.0 (GLS 9.0) Conference, Madison, WI, 2013.

[2] American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Standards forEducational and Psychological Testing, American Psychological Association, Washington, DC, 1999.

[3] R.S.J.d. Baker, S.K. D’Mello, M.M.T. Rodrigo, A.C. Graesser, Better to be frustrated than bored: the incidence, persistence, and impact of learners’cognitive-affective states during interactions with three different computer-based learning environments, International Journal of Human-ComputerStudies 68 (4) (2010) 223–241.

[4] A. Bandura, Social Learning Theory, General Learning Press, 1977.[5] J.T. Behrens, T.A. Collison, S.F. DeMark, The seven Cs of comprehensive assessment: lessons learned from 40 million classroom exams in the Cisco

Networking Academy Program, in: S. Howell, M. Hricko (Eds.), Online Assessment and Measurement: Case Studies in Higher Education, K-12 andCorporate, Information Science Publishing, Hershey, PA, 2005, pp. 229–245.

[6] J.T. Behrens, D.C. Frezzo, R.J. Mislevy, M. Kroopnick, D. Wise, Structural, Functional, and Semiotic Symmetries in Simulation-based Games andAssessments, in: e. Baker, J. Dickieson, W. Wulfeck, H.F. O’Neil (Eds.), Assessment of Problem Solving using Simulations, Erlbaum, New York, 2008, pp.59–80.

[7] J.T. Behrens, R.J. Mislevy, M. Bauer, D.W. Williamson, R. Levy, Introduction to evidence centered design and lessons learned from its application in aglobal e-learning program, International Journal of Testing 4 (2004) 295–301.

[8] K.E. Boyer, R. Phillips, A. Ingram, E.Y. Ha, M. Wallis, M. Vouk, J. Lester, Investigating the relationship between dialogue structure and tutoringeffectiveness: a hidden Markov modeling approach, International Journal of Artificial Intelligence in Education 21 (1–2) (2011) 65–81.

[9] R.A. Calvo, S.K. D’Mello, Affect detection: an interdisciplinary review of models, methods, and their applications, IEEE Transactions on AffectiveComputing 1 (1) (2010) 18–37.

[10] D.B. Clark, E. Tanner-Smith, S. Killingsworth, A Systematic Review and Meta-Analysis of Digital Games for Learning: Moving Beyond Media-Comparison Studies, in preparation.

[11] J. Clarke-Midura, M. Mayrath, C. Dede, Thinking outside the bubble: virtual performance assessments for measuring inquiry learning, in: M. Mayrath, J.Clarke-Midura, D. Robinson (Eds.), Technology Based Assessment for 21st Century Skills: Theoretical and Practical Implications from Modern Research,Springer-Verlag, New York, NY, 2012.

[12] S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, F. Players, Predicting protein structures with a multiplayer online game, Nature 466 (7307)(2010) 756–760, http://dx.doi.org/10.1038/nature09304.

[13] D.I. Cordova, M.R. Lepper, Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice, Journalof Educational Psychology 88 (1996) 715–730.

[14] K.E. DiCerbo, J.T. Behrens, Implications of the digital ocean on current and future assessment, in: R. Lissitz, H. Jiao (Eds.), Computers and their Impacton State Assessment: Recent History and Predictions for the Future, Information Age Publishing, Charlotte, North Carolina, 2012, pp. 273–306.

[15] K.E. DiCerbo, D.C. Frezzo, T. Deng, Substantive validity of a simulation-based game, Research and Practice in Technology-Enabled Learning 6(3) (2011)161–185. <http://apsce.net/RPTEL/RPTEL2011NovIssue-Article2_pp161-185.pdf>.

[16] K.E. DiCerbo, J. Liu, D.W. Rutstein, Y. Choi, J.T. Behrens, Visual analysis of sequential log data from complex performance assessments, in: PaperPresented at the Annual Meeting of the American Educational Research Association, New Orleans, LA, 2011.

[17] M.W. Easterday, V. Aleven, R. Scheines, S.M. Carver, Using tutors to improve educational games, Artificial Intelligence in Education 6738 (2011) 63–71.[18] D.C. Frezzo, J.T. Behrens, R.J. Mislevy, Activity theory and assessment theory in the design and understanding of the Packet Tracer ecosystem. The

International Journal of Learning and Media 2 (2009). <http://ijlm.net/knowinganddoing/10.1162/ijlm.2009.0015>.[19] D.C. Frezzo, J.T. Behrens, R.J. Mislevy, P. West, K.E. DiCerbo, Psychometric and evidentiary approaches to simulation assessment in Packet Tracer

Software, in: Paper Presented at the International Conference on Networking and Services, Valencia, Spain, 2009.[20] J. Gee, What would a state of the art instructional video game look like? Innovate 1(6) (2005). <http://www.innovateonline.info/

index.php?view=article&id=80>.[21] A.C. Graesser, V. Rus, S.K. D’Mello, G.T. Jackson, AutoTutor: learning through natural language dialogue that adapts to the cognitive and affective states

of the learner, in: D.H. Robinson, G. Schraw (Eds.), Recent Innovations in Educational Technology that Facilitate Student Learning, Information AgePublishing, 2008, pp. 95–125.

[22] J. Grafsgaard, K.E. Boyer, J. Lester, Toward a machine learning framework for understanding affective tutorial interaction, in: Proceedings of theEleventh International Conference on Intelligent Tutoring Systems, Crete, Greece, 2012, pp. 52–58.

[23] M.A. Honey, M. Hilton, Learning Science through Computer Games and Simulations, National Academies Press, 2010.[24] E. Hutchins, Cognition in the Wild, MIT Press, Cambridge, Mass, 1995.[25] M.D. Kickmeier-Rust, D. Albert, Micro adaptivity: protecting immersion in didactically adaptive digital educational games, Journal of Computer

Assisted Learning 26 (2010) 95–105.[26] D. Meier, The Power of their Ideas: Lessons for America from a Small School in Harlem, Beacon Press, Boston, MA, 1995.[27] S. Messick, Validity of psychological assessment: validation of inferences from a person’s responses and performances as scientific inquiry into score

meaning, American Psychologist 50 (1995) 741–749.[28] R. Miikkulainen, Creating intelligent agents in games, The Bridge 36 (4) (2006) 5–13.[29] K. Millis, C. Forsyth, H.A. Butler, P. Wallace, A.C. Graesser, D.F. Halpern, Operation ARIES! A serious game for teaching scientific inquiry, in: M. Ma, A.

Oikonomou, J. Lakhmi (Eds.), Serious Games and Edutainment Applications, Springer-Verlag, London, UK, 2011.

D.C. Frezzo et al. / Information Sciences 264 (2014) 91–103 103

[30] R.J. Mislevy, Evidence and inference in educational assessment, Psychometrika 59 (1994) 439–483.[31] R.J. Mislevy, J.T. Behrens, K.E. DiCerbo, R. Levy, Design and discovery in educational assessment: Evidence-centered design, psychometrics and

educational data mining, Journal of Educational Data Mining 4 (2012) 11–48.[32] R.J. Mislevy, L.S. Steinberg, R.G. Almond, On the structure of educational assessments, Measurement: Interdisciplinary Research and Perspectives 1

(2003) 3–62.[33] R.J. Mislevy, L.S. Steinberg, R.G. Almond, Design and Analysis in Task-based Language Assessment. CRESST Technical Report 579. <http://

www.cse.ucla.edu/products/Reports/TR579.pdf>, 2002.[34] M. Muir, C. Conati, An analysis of attention to student-adaptive hints in an educational game, Proceedings of ITS 2012, 11th International Conference of

Intelligent Tutoring Systems, vol. 7315, Springer, LNCS, 2012, pp. 112–122.[35] C. Pedica, H. Vilhjalmsson, Social perception and steering for online avatars, in: H. Prendinger, J. Lester, M. Ishizuka (Eds.), IVA 2008. LNCS (LNAI), vol.

5208, Springer, Heidelberg, 2008, pp. 104–116.[36] A.A. Rupp, K.E. DiCerbo, R. Levy, M. Benson, S. Sweet, A.V. Crawford, D. Fay, K.L. Kunze, T. Caliço, J.T. Behrens, The interplay of theory and data: evidence

identification and aggregation for product and process data within a digital learning environment, Journal of Educational Data Mining 4 (2012) 49–110.

[37] A.A. Rupp, M. Gushta, R.J. Mislevy, D.W. Shaffer, Evidence-centered design of epistemic games: measurement principles for complex learningenvironments, The Journal of Technology, Learning and Assessment 8(4) (2010). <http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1623>.

[38] D.W. Shaffer, How Computer Games Help Children Learn, Palgrave Macmillan, New York, 2006.[39] V.J. Shute, M. Ventura, M.I. Bauer, D. Zapata-Rivera, Melding the power of serious games and embedded assessment to monitor and foster learning:

flow and grow, in: U. Ritterfeld, M. Cody, P. Vorderer (Eds.), Serious Games: Mechanisms and Effects, Routledge, Taylor and Francis, Mahwah, NJ, 2009,pp. 295–321.

[40] V.J. Shute, M. Ventura, Y.J. Kim, Assessment and learning of qualitative physics in Newton’s playground, Journal of Educational Research 106 (2013)423–430.

[41] J. Wang, H. Li, Z. Cai, F. Keshtkar, A.C. Graesser, D.W. Shaffer, AutoMentor: artificial intelligent mentor in educational game, AIED 2013 (2013) 940–941.[42] P. Wouters, C. van Nimwegen, H. van Oostendorp, E.D. van der Spek, A meta-analysis of the cognitive and motivational effects of serious games, Journal

of Educational Psychology (2013). doi: 10.1037/a0031311, February 4.