ASSESSING GAME-BASED PROGRAMMINGIssues, challenges, and models to follow

WHO I AM

Kyle Felker

Grand Valley State University Libraries

Digital Initiatives Librarian

Investigates, prototypes, integrates

felkerk@gvsu.edu, @gwydion9

STRUCTURE OF THIS TALK

What games can do for us

Basic assessment principles

Unique Challenges and approaches

LibraryQuest assessment program

WHY GAMES?

Greater, lengthier, or more focused engagement with facilities, staff, services, etc.

Reaching one or more learning objectives

GENERAL PRINCIPLE : SET CLEAR AND MEASURABLE GOALS

Engagement based-goal: Increase participation in our summer reading program by at least 50%

Learning goal: To teach incoming undergraduates how to locate books using the online catalog and how to find them on the shelf.

Skills-based vs Conceptual

GENERAL PRINCIPLE: CONTEXTUALIZE AND HARMONIZE GOALS

Should be in accord with institutional goals and objectives

Should not duplicate other programs

Should take into account national and institutional standards, frameworks, etc.

EVALUATION VS. ASSESSMENT

Evaluation: How can we improve this?

Assessment: Did this do what it was supposed to do?

Evaluation often easier, mistaken for assessment

Assessment often requires more effort and thought

GENERAL PRINCIPLE: BUILD ASSESSMENT INTO PROJECTS AT INCEPTION

Consider assessment plan as part of overall plan

Design, schedule, and plan opportunities for both evaluation and assessment

Secure resources such as staff time and money

Initial pilot: evaluation, subsequent pilot: assessment

GENERAL PRINCIPLE: INVOLVE OTHERS

What do you needs to know/do to fulfill your assessment plan?

Do you have all the necessary skills?

Who does, and can you get them to help you?

GAME-SPECIFIC CHALLENGES

The conundrum of “fun”

Asynchronous games

Complexity

INTEGRATING ASSESSMENT INTO GAMEPLAY

Can interactions include assessment components?

Can players complete assessment as part of gameplay?

What information can be collected unobtrusively? By staff? By the system?

ELECTRONIC GAMES AND METRICS

Number of players or interactions

Length of interaction

Path through the game or app

Privacy concerns

CASE STUDY: LIBRARYQUEST

LIBRARYQUEST

Asynchronous mobile game

Offers players “quests” or tasks

Complete for points, which can be turned in for prizes

Learn by doing: accomplishing quest = successfully using skill

Second pilot: Assessment focused

OBJECTIVES:

To teach incoming freshmen ten distinct skills and knowledge-based goals relating to library services and spaces (example: how to use the online catalog to locate a book).

To teach freshman that the library is a support service that exists to help them in their work.

To increase freshman sense of self-sufficiency and confidence in using the library.

INSTITUTIONAL AND NATIONAL CONTEXT

ACRL Information Literacy Standards: PK Morgan Critique “Pausing at the Threshold.”

Institutional IR assessment

Research on Student Retention

Emphasis on incoming freshmen

OUTSIDE COLLABORATORS

Head of Instructional Services

First-Year Experience Librarian

Statistical Consulting Center

ASSESSMENT COMPONENT: LEARNING GOALS/STUDENT CONFIDENCE

Student selects quest

Quest app assesses student ability through skills-based quiz

Exposes results to student

Quest asks student to rate confidence on a ten-point scale

Student performs quest

Quest app asks student to rate confidence once again using ten point scale

App rewards student

EXAMPLE QUESTIONS: CALL NUMBER QUESTWhich of the following is shelved first?

EA 301.E799 1990

E 302.E799 1989

E 302.A799 1999

Which part of this call number designates a general subject area?

HK

302.1

.E99

1990

On a scale of 0-9, rate how confident you are that you could find a book on the shelf using a call number:

Not Confident 0 1 2 3 4 5 6 7 8 9 Very Confident

ASSESSMENT: IDENTIFYING LIBRARY AS SUPPORT SERVICE

Pre and post-play questions:

Which of the following offer services that support you in completing your assignments?

Registrar’s office

Writing Center

Student life office

University Libraries

EVALUATIVE DATA

Statistics:

Number of players

Number of quests completed

Which quests completed and abandoned

What steps completed before abandoned

When completed

What rewards were selected (and how many)

EVALUATION COMPONENTS

Periodic surveys throughout the game on elements of gameplay:

“What is your favorite quest so far? Why?”

Post-game Interviews

“If you could tell the designer of this game any one thing, what would it be?”

Questions?