Upload
bernice-thornton
View
213
Download
0
Embed Size (px)
Citation preview
Researching the Effectiveness Researching the Effectiveness of Serious Games: Asking the of Serious Games: Asking the
Right QuestionsRight Questions
Dr. Jennifer SolbergDr. Jennifer Solberg
US Army Research Institute for theUS Army Research Institute for theBehavioral and Social SciencesBehavioral and Social Sciences
Serious games are being increasingly incorporated into learning environments.
Examples:• Classroom Education• Games for Health• Military Training
What sorts of questions should we be asking?Really, there’s only one.
The Rise of Serious GamesThe Rise of Serious Games
The One Big QuestionThe One Big Question
Do Serious Games Work? Are serious games an effective means of training?
Are they effective above and beyond current methods?
It’s that simple!
• If not, nothing else matters
It’s not that simple!
• Many factors are undoubtedly involved in the effectiveness of serious games.
OverviewOverview
Current thought on the One Big Question (OBQ)
• Answers so far
• Are we asking the right questions?
What questions we should be asking
• The Player
• The Game
• The Process
Where we can find the answers
• Performance within and outside the game
• Individual difference measures
• Preference measures
Where do we go from here?
Current Thought on the OBQCurrent Thought on the OBQ
Is there evidence that serious games are effective for training?
• Sometimes.
Is there evidence that serious games in general are effective for training above and beyond traditional methods of instruction ?
• Not really. (Hays, 2005)
What Does this Mean?
Current Thought on the OBQCurrent Thought on the OBQ
“The empirical research of the instructional effectiveness of games is fragmented, filled with ill defined terms, and plagued with methodological flaws.” (Hays, 2005)
“There appears to be a consensus among a large number of researchers with regard to the negative, mixed, or null findings of game research, suggesting that the cause might be a lack of sound instructional design embedded in the games” (O’Neil et al, 2005)
“Successful learning in serious games involves a complex interaction between the player and the game. In order to develop serious games that work, we must ask the right questions about this interaction.” (Solberg, this week)
Asking the Right QuestionsAsking the Right Questions
Any type of learning involves three components: the player, the instructional material, and the learning process itself.
We need to ask questions about all three.
The Player: What characteristics of the player make them more likely to learn through serious games?
The Game: What aspects of the game itself can we manipulate to maximize effectiveness?
The Process: In what context should the game be delivered to the player to maximize the likelihood of learning?
Is a game-based environment conducive to our normal learning and memory processes?
• Kirschener et al (2006). The trend of relying on unstructured learning is contrary to cognitive architecture.
• Clark’s talk yesterday
Are serious games conducive to normal interactions in a multi-player setting?
• Method of communication
• Restriction of actions
The PlayerThe Player
The PlayerThe Player
Do some people have characteristics that lend themselves to success in game-based learning? You may be surprised!
Examples:Sex
•Carr (2005). Female players avoid female avatars in game environments.
Age/Generation Cognitive ability Personality
•Learning style preferencePropensity to play games
•Orvis et al. (2005, 2006). Cadets’ gaming experience
Prior experience with training material
The GameThe Game
How can we design serious games to maximize the likelihood of learning?
Current thought:– Serious games should be tied to structured learning objectives.– Serious games should be motivational.
What aspects of serious games can we manipulate to assess effectiveness?
•Modality (of instructions, feedback) •Type of instructional material
•Changes in difficulty level •Complexity (part v. whole task)
•Transparency of instructional material •Duration (effects on motivation)
•In game feedback/score •Help (scaffolding, availability)
The ProcessThe Process
Does the way in which serious games are administered affect the quality of learning?
Possible factors:
• Pre-game instruction
(game play and training domain)
• Presence of instructor
• Feedback during/after game
(modality and timing)
• Effects of retesting
We have lots of questions. Where can we find the answers?
Finding the AnswersFinding the Answers
Where can we find the answers?
• Performance measures within and outside the game
• Individual difference measures
• Preference measures
These measures make up the types of data used in serious games research.
– It’s important to consider including measures in the game design so that research is possible later on.
• Validation
• Improvement
• Trouble-shooting
Performance MeasuresPerformance Measures
Within-game: How well do players play the game?• Score • Time to completion• Qualitative decisions/actions• Relative position to others?• Communication between players
Outside the game: How well do players transfer skills to other tasks?
• “Real world” taskNolan & Jones (2005) squad exercise performance after
training with COTS game• Tests
Retention, recognition– Training domain knowledge– KSAs
Individual Difference TestsIndividual Difference Tests
In order to determine the extent to which player characteristics can affect learning in serious games, measures of these characteristics should be considered.
• Perceptual/attentional tests
• Cognitive factors (e.g. decision-making ability, memory span)
• Personality factors (e.g. extraversion, conscientiousness, learning style preference)
Preference MeasuresPreference Measures
An important aspect of serious games is their ability to motivate. Gaming should be fun.
Preference measures provide data on the enjoyability of the game.
Also,
• Preference for game-based instruction
• Preference for modality of instruction (e.g. visual v. audio)
Too Much Information?Too Much Information?
How much performance information is enough? What kind is most informative?
This is a topic of debate (even in my lab), and largely depends on constraints of project
• Subjectivity v. objectivity
• Need for rapid deployment (especially in military)
• Need for troubleshooting
• Need to answer theoretical questions
Should there be a “standard” methodology?
Where Do We Go Now?Where Do We Go Now?
Assessing the answer to the OBQ will require asking questions about the interaction between
• The Player
• The Game
• The Process
Some answers can be found in:
• Performance measures within and outside the game
• Individual difference measures
• Preference measures
Where Do We Go Now?Where Do We Go Now?
These are all empirical questions that need to be answered!
Serious games are being implemented whether or not we answer the OBQ!
Non-trivial training domains
• Military training
• Medical/health instruction
An AfterthoughtAn Afterthought
The Author
• How easily does the author of the game interact with the authoring interface?
– May affect ability to incorporate training objectives into game environment.
• How much control does the author need?
– Sophistication of scenarios v. accessibility
– Decision based on finances, access to facilities, need for rapid deployment