HuskyHunt Technology Integration project

University of Connecticut

HuskyHunt

A Study of Engagement and Transfer

Pufahl, Jason12-7-2014

HuskyHunt 1

ContentsTitle..................................................................................................................................................2

Instructional Overview....................................................................................................................3

Purpose/Objectives..........................................................................................................................4

Formative Assessment.....................................................................................................................4

Data Section.....................................................................................................................................6

Player Overview and Demographic Information.........................................................................6

Engagement Data.........................................................................................................................8

Knowledge Transfer Data..........................................................................................................13

Pre-Survey Data.........................................................................................................................15

Post-Survey Data........................................................................................................................17

Analysis and Results Section.........................................................................................................18

Reflections Section........................................................................................................................24

Future Recommendations Section.................................................................................................26

Appendices....................................................................................................................................28

Appendix A - Social Network Reach.........................................................................................28

Appendix B - Game Schedule and Module Format...................................................................29

Appendix C - Sample Player Badge Profile Page......................................................................30

Appendix D - List of All Available Badges...............................................................................31

Appendix E - Introductory Video Script....................................................................................35

Appendix F - Pre-Game and Post-Game Survey Questions......................................................36

Appendix G - Awards Ceremony Agenda and Interview Questions.........................................38

Appendix H - Awards Dinner Interviews Transcriptions..........................................................39

HuskyHunt 2

Title

As the Chief Information Security Officer for the University of Connecticut one of my responsibilities is providing security awareness training to faculty, staff and students. Because the University has chosen not mandate this training I need to seek out opportunities to educate and am required to find ways to encourage people to take training. This challenge led me to create HuskyHunt.

HuskyHunt is an information security awareness training game and is intended to meet the following specific goals:

Teach students practical lessons about Internet and information security awareness topics that they can relate to personally

Utilize gaming mechanics to increase engagement and participation Leverage Social media, specifically Facebook and Twitter to reinforce learning that

occurred in the game modules. Leverage social media to increase the likelihood that UConn students not

participating in HuskyHunt are exposed to the security training.

The technology integration project was designed to use the HuskyHunt framework to measure the impact that digital badges (“Digital badges - Wikipedia, the free encyclopedia,” n.d.) have on player engagement, learning transfer and players attitudes regarding technology and security.

The game was originally designed without digital badges and was redesigned for this project. 17 digital badges were designed for the game and the game was configured to randomly distribute badges to players at registration time. Badges and descriptions can be seen in Appendix D – List of All Available Badges.

The game was developed to be both desktop and mobile friendly which proved important in supporting both the learning modules and scavenger hunt.

Social networks, Facebook and Twitter, were used for three reasons:

Communicate of game events to players in a medium they prefer Provide us with a longer term capability of communicating other information to students Expand the reach of security information to friends of players not playing the game

(Appendix A – Social Network Reach)

Briefly HuskyHunt is a game consisting of 2 primary components, online learning modules and real world scavenger hunt modules. The game has incentives structured throughout the game, with grand prizes of $500, $100, and $50 to the UConn CO-OP and periodic incentives provided by local business sponsors. The game was originally designed solely as a security training tool and is now partnered with the OneCard Office and includes content related to their functions.

HuskyHunt 3

HuskyHunt started on October, 6th 2014 and ran through October, 26th 2014. The game was broken down into 3 weekly challenges. Each week included 6 daily online learning modules and culminated with a physical scavenger hunt. Players were alerted to new content via postings to social networks, specifically Facebook and Twitter.

HuskyHunt, as a learning tool leveraging gaming mechanics and digital badging, is directly in line with the 2014 K-12 Horizon Report that suggests that the next three years will show a significant increase in the amount of game based learning tools.

Instructional Overview

I don’t have a classroom, but have an assigned and ethical responsibility to try and teach greater than 20,000 students about basic technology security concepts. HuskyHunt is designed to be delivered at a large scale and has no technical limitations to the number of individuals that can participate in an available and active course. The framework aligns most closely to an xMOOC, which is geared towards an individual seeking an achievement, is likely offered in addition to other complementary education and is designed to be repeated (“elearnspace › MOOCs are really a platform,” n.d.).

HuskyHunt was developed to appeal, primarily, to University students. There are a variety of challenges connecting with this demographic.

Students are geographically dispersed across seven campuses Semester timeframes and varying course loads make getting sustained attention

difficult Marketing/Advertising challenges

HuskyHunt was developed and implemented to deliver “the entities, relations and attributes that the learner must ‘know’” (Duffy & Jonassen, 1992 location 192). It also shares similarities with a current standards based curriculum concepts (Carr-Chellman, 2011) leveraging directed instruction and assessments. HuskyHunt establishes clearly communicated learning objectives and leverages online multiple choice assessments designed to test specifically what was taught in the lesson.

The game was designed to collect assessment and behavioral data throughout the game, for the purpose of aligning with the ISTE Standards for teachers to “provide students with multiple and varied formative and summative assessments aligned with content and technology standards, and use resulting data to inform learning and teaching.” (“ISTE Standards for Teachers,” n.d.) The whole intention of the game is to educate and improve reach and quality for each semester.

The curriculum was intentionally re-designed to meet the following criteria outlined in the ISTE Standards for Teachers (“ISTE Standards for Teachers,” n.d.):

1. “Design and develop digital age learning experiences and assessments. Communicate relevant information and ideas effectively to students....using a

HuskyHunt 4

variety of digital age media and formats.” HuskyHunt utilized videos, mobile devices, online content, social media, online surveys, digital badges and QR codes to enable players to interact and engage with the game.

2. Promote and model digital citizenship and responsibility “Teachers understand local and global societal issues and responsibilities in an evolving digital culture…”

a. Advocate, model, and teach safe, legal, and ethical use of digital information and technology, including respect for copyright, intellectual property, and the appropriate documentation of sources

b. promote and model digital etiquette and responsible social interactions related the use of technology and information

The game learning modules, listed in Appendix B – Game Schedule and Module Format, were designed to address both items a. and b. above.

Purpose/ObjectivesThe purpose of this study was to evaluate the affect that awarding digital badges to

players would have on game completion, task completion and engagement. Training materials, such as information security awareness often has low participation and completion rates but are considered useful tools for ensuring that students or employees are taught relevant information. I assert that these tools can actually be made to be effective if design elements, such as digital badges, are designed and incorporated to increase a participants motivation to complete the training. I believe that learning transfer can be increased through more active participation with the content. P.R. Subramanium Says that ‘Task design certainly plays a major role in impacting situational interest” and that “By increasing the interestingness of the learning task, even students with different individual dispositions are more likely to exhibit interest and be actively engaged.” (Subramaniam, 2009). It is this specific idea that I sought to support with this implementation of HuskyHunt.

As additional support for the idea that digital badges can increase engagement I considered this New York City program where schools implemented a digital badge award program that is described as having positive results (“Digital Badges for Microcredentials, Student Engagement & Persistence | Online Learning Consortium, Inc,” 2012). I designed this version of HuskyHunt such that approximately half of the players would receive digital badges for certain achievements and other players would not. It is assumed that greater engagement would lead to higher completion rates and correspondingly higher learning transfer.

Formative AssessmentThe following qualitative data were collected:

Pre-Game Survey

There was a short, six question, optional survey provided as the first online activity of the game. All players had the option of taking the survey and it was made available at the start of the game, on October, 6th immediately following the introductory video. The survey was available to any new player starting on or after October 6th.

HuskyHunt 5

Post-Game Survey

There was an eight question optional survey provided as the final activity of the game. The survey was only available to players that completed all of the modules and was presented only on October 26th, the last day of the game.

The complete list of all Pre-Game and Post-Game survey questions are listed in Appendix F – Pre-Game and Post-Game Survey Questions. An analysis of survey results are described in the analysis and results section below.

Awards Dinner Discussions

There was an awards dinner held on October 29th. The agenda and format of this event is described in Appendix G – Awards Ceremony Agenda and Interview Questions. I took advantage of the time players were arriving to engage in informal discussions to get a general sense of their experiences. I also arranged for a group of players to remain after the dinner and engaged in a more structured discussion. A summarized transcription is listed in Appendix H – Awards Dinner Interviews Transcriptions . This conversation was captured in both audio and video format and the transcription was made from the recorded conversation.

The following quantitative data elements were collected:

From game interaction and authentication sources:

NetId, Name, University Affiliation, University Status, Badges Enabled, Academic Year, Gender, At least one module completed, Furthest Module Completed, Pre-Game Survey Completed, Post-Game Survey Completed, Scavenger Hunt Participation, Created Player Alias, Posted to Social Media, Completed Every Module, Completed Final Assessment, Entered Credentials in Final Assessment.

The source data listed above was utilized to generate the following statistics. These statistics were designed to be collected and compared to determine engagement, skills development and attitude changes:

% of players completing each module % of players remaining after each module % of players completing all modules % of players abandoning game % of players abandoning by module Total players receiving all badges Total players receiving at least 1 badge Total players that checked the badge description page How far average player with badges got How far average player without badges got Total unique scavenger hunt participants % of players completing post-game survey % of players completing pre-game survey % players changing profile name % of players sharing in social media

HuskyHunt 6

Average times needed to answer an assessment question correctly Average Final Assessment Score % of people providing username and password (final assessment criteria) Player’s thoughts and knowledge about Internet and information security before and after

the game. Player’s thoughts on how badges influenced their desire to play the game. Players attitudes towards the technologies and concepts used in the game

Data Section

Player Overview and Demographic InformationThe data in this section is meant to describe player statistics and demographics. The

intention with gathering this data was ensure a complete understanding of overall player breakdown so that any inferences from the data could be compared with the general player information.

25

75

125

175

225

275

325

375

Total Game Registrations

Figure 1

Figure 1. Illustrates the total game registrants. This data was generated from all players that registered, but may not have played, the game.

HuskyHunt 7

25

75

125

175

225

Total Active Players

Figure 2

Figure 2. Illustrates only the active players. This data was generated from all players who participated in at least one game module. This is the population from which all future inferences will be derived.

25

75

125

175

225

Total Active Players Badge Status

Figure 3

Figure 3. Illustrates the breakdown of active players with and without badges. It is important to recognize that nearly 70% of players had badges and 30% did not.

HuskyHunt 8

Total0

50

100

150

200

250

192

2010.42%

71

36.98%

28

14.58%

64

33.33% 7 3.65% 2 1.04%

Active Players By University Affiliation

Active Players Freshman Freshman % Sophomore Sophomore % Junior Junior % Senior Senior % Graduate Student Graduate Student % Other Other %

Figure 4

Figure 4. Illustrates the breakdown of active players by University affiliation. The intention of gathering this data was primarily to gain a more thorough understanding of players and help guide future player recruiting efforts.

Players interacted with the game using both mobile devices such as cell phones (Android and iOS primarily), iPods and full computers (Mac, PC). There was no effort made to distinguish between laptops or desktops. Game modules produced a Mobile/Computer ratio of: 33.02%:66.98% while the scavenger hunt produced a Mobile/ Computer ratio of 65.09%:34.9. The ratio for the scavenger hunts is expected, but the ratio for the online modules is skewed somewhat unexpectedly towards computers as we had assumed that most people would interact with the game using their mobile devices.

Engagement DataThe following data is designed to describe how players interacted with the game, and

provide clarity regarding the breakdown of players with badges and players without badges. All of the following data is based on active players (players playing at least one module) not registered players. Recalling the data above, 69% of players had badges enabled and 31% of players did not.

HuskyHunt 9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Total Active Players

205

152

134

124

120

106

104

96 90 86 85 77 77 74 70 68 65 24 35

Active Players with Badges

130

104

93 84 82 70 69 63 61 58 57 51 51 48 45 44 41 18 23

Active Players without Badges

75 48 41 40 38 36 35 33 29 28 28 26 26 26 25 24 24 6 12

25

75

125

175

225

Active Players by Module

Total Active Players Active Players with Badges Active Players without BadgesGame Level

Acti

ve P

laye

rs

Figure 5

Figure 5. Illustrates the active players by module. This clearly shows a steady decrease of player activity over the course of the game. The decrease, however, is statistically consistent between players with and without badges throughout the game.

10

30

50

70

90

110

130

150

170

Pre-Game Survey Participants

Figure 6

Figure 6. Illustrates the players that participated in the optional pre-game survey. These numbers are excellent with an 82.2% completion rate and likely influenced by the fact that the survey immediately followed the introductory video in which I specifically request they fill out the survey.

HuskyHunt 10

2.5

7.5

12.5

17.5

22.5

27.5

Post-Game Survey Participants

Figure 7

Figure 7. Illustrates the players that participated in the optional post-game survey. The response rate of 93% is excellent, and is even higher than the response rate for the pre-survey questions. This is one of the more positive indicators that players had a good game experience and were engaged.

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

3.5

Post-Survey Question 8

Figure 8

Figure 8. describes players attitudes regarding the digital badges, after game completion.

Do you feel that having badges in the game made you more likely to play?1 - Yes2 - No3 - There were no badges

HuskyHunt 11

2.5

7.5

12.5

17.5

22.5

27.5

Scavenger Hunt Statistics

Figure 9

Figure 9. Illustrates the players that participated in the scavenger hunt.

5

15

25

35

45

55

65

Alias Creation Statistics

Figure 10

Figure 10. Illustrates the players that created in-game nicknames (aliases). This data represents another positive engagement metric as 35% of players explored game settings and identified that they could create aliases.

HuskyHunt 12

5

15

25

35

45

55

65

75

Social Media Sharing Statistics

Figure 11

Figure 11. Illustrates the players that shared content to their social media networks. This data represents another positive engagement metric as 38% of players were willing to share preconfigured posts to their community.

1

3

5

7

9

11

13

Players Completing Entire Game

Total Players With Badges With Badges % W/O Badges W/O Badges %

Figure 12

Figure 12. Illustrates the players that completed all modules. This total, 13 people (6%) is much lower than expected and clearly lower than the expectations (as indicated by Pre-Survey Question 6) of players prior to the start of the game.

HuskyHunt 13

2.5

7.5

12.5

17.5

22.5

27.5

32.5

Players Completing Final Assessment

Figure 13

Figure 13. Illustrates the players that completed the final assessment.

0 50 100 150 200 2500

0.5

1

1.5

2

2.5

3

3.5

Pre-Survey Question 6

Do you expect to complete the whole game?1 - Yes2 - No3 - Maybe

Figure 14

Figure 14. Describes player’s thoughts, entering the game, regarding their expectation of completing the game.

Finally, an example of the scoreboard, showing the current badge status of all players is available in Appendix C - Sample Player Badge Profile Page. This was designed to provide a current status of earned and unearned badges and it was hoped that it would increase players motivation.

Knowledge Transfer Data

HuskyHunt 14

1

3

5

7

9

11

13

15

17

19

Average Level Completed

Figure 15

Figure 15. Illustrates the average levels completed. The game was unable to keep players engaged for at least 50% of the game and players with badges actually abandoned 1.5 modules earlier than those without.

Players answered 80.28% of the module assessment question correctly on the first attempt. There were 5188 total question attempts and 4165 correct attempts. Players were allowed as many chances as necessary to answer questions correctly, as the primary goal of HuskyHunt is to train rather than assess.

The average final assessment score was 74.8, which represents an average of 82.33 points out of a possible 110. Players were not allowed to answer questions more than one time for the final assessment, as the intention here was to provide score differentiation and produce a clear winner.

The average time to answer a question was eight seconds indicating that the question difficulty was too easy.

1

3

5

7

9

11

Players Providing Credentials

Figure 16

HuskyHunt 15

Figure 16. Illustrates the players that completed the final assessment, but provided their username and password when asked.

Pre-Survey Data

0 50 100 150 200 2500

0.5

1

1.5

2

2.5

3

3.5


Figure 17

Figure 17. Shows players that completed pre-game survey: 244/338. There were 192 active players, but 336 registrants that could have participated in the surveys.

0 50 100 150 200 2500

0.5

1

1.5

2

2.5

3

3.5


Figure 18

Why are you playing the game?1 - To learn about information security and privacy2 - To win prizes3 - Saw it advertised and decided to try it

How would you describe your attitude towards security?1 - I am very security conscious2 - I know there are risks but am not concerned3 - I don't think about security at all

HuskyHunt 16

0 50 100 150 200 2500

0.5

1

1.5

2

2.5


Figure 19

0 50 100 150 200 2500

0.5

1

1.5

2

2.5

3

3.5


Figure 20

Figure 20. Provides useful contextual information regarding players technology attitudes and can be utilized to inform future implementation decisions.

How would you describe your attitude towards personal privacy?1 - I am very conscious of my privacy2 - I know there are risks but am not concerned3 - I don't think about privacy at all

How would you describe your attitude towards technology?1 - I am a technology enthusiast2 - I use technology3 - I don't like technology but am required to use it

HuskyHunt 17

Post-Survey Data

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5


Figure 21

Figure 21. Shows Players that completed post-game survey: 27/29. There were 29 players that completed the final assessment, the post-game survey was presented after the final assessment.

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

3.5


Figure 22

Did you learn something about security, privacy or technology from the game?1 - Yes, I learned a lot of useful information2 - Yes, but I knew most of the information already3 - No

Did you attitude regarding security change?1 - Yes2 - Somewhat3 - No

HuskyHunt 18

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

3.5


Figure 23

0 5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

3.5


Figure 24

Analysis and Results SectionThe data collected was a mix of quantitative data and qualitative data. The majority of

the quantitative data I collected and evaluated was stored in an Excel document called ‘HuskyHunt Demographic and Game Data’ which contained the following data elements:

NetId, Name, University Affiliation, University Status, Badges Enabled, Academic Year, Gender, At least one module completed, Furthest Module Completed, Pre-Game Survey Completed, Post-Game Survey Completed, Scavenger Hunt Participation, Created Player Alias, Posted to Social Media, Completed Every Module, Completed Final Assessment, Entered Credentials in Final Assessment.

Did your attitude regarding personal privacy change?1 - Yes2 - Somewhat3 - No

Did your attitude towards technology change?1 - Yes2 - Somewhat3 - No

HuskyHunt 19

The data was structured in this format so that the filtering and graphing functions of Excel could be utilized to produce most of the information provided in the Data Section and was intentionally collected to ensure that the questions identified in the Formative Assessment section could be answered.

The qualitative data, primarily the responses to the Pre and Post Game Surveys and the interviews that occurred at the awards dinner were all collected electronically and stored in the HuskyHunt Demographic and Game Data Excel file as the following elements:

Pre-Survey Question 1, Pre-Survey Question 2, Pre-Survey Question 3, Pre-Survey Question 4, Pre-Survey Question 5, Pre-Survey Question 6, Post-Survey Question 1, Post-Survey Question 2, Post-Survey Question 3, Post-Survey Question 4, Post-Survey Question 5, Post-Survey Question 6, Post-Survey Question 7, Post-Survey Question 8

The survey results were collected in Excel spreadsheets and graphed. The interviews were all recorded and I created transcriptions in Word for each conversation. I utilized these interviews as an anecdotal account in an effort to get game feedback and see if those results align with the quantitative data collected.

The online survey data, figures 16 through 23, provided some interesting information for game improvements and some optimism that the game can provide real learning value. Figure 16 clearly indicates that players are joining the game because they are interested in winning prizes and that the topic itself is not the motivating factor. The same respondents overwhelmingly indicate that they are concerned about security and privacy, but that concern is not enough to sustain long term interest in the game. The post-game surveys do suggest that the game did provide value for those that played it to completion. The responses to questions 2-5 (figures 20-23) indicate that the majority of players are thinking differently about security and privacy as a result of the game. These results are supported by the in person conversations I had with players at the awards dinner.

The quantitative data shows very little difference in engagement or knowledge gain between players that received badges and those that didn’t. A close examination of the data shows that approximately two of every three players were configured to receive badges. That ratio remains close, with any differences being insignificant, for every category measured as illustrated in the following graphs:

HuskyHunt 20

69.79%

30.21%

Badge DistributionRatio - 2.31:1

Players With Badges % Players W/O Badges %

Figure 25

68.99%

31.01%

Pre-Game Survey Statistics CompletionRatio - 2.25:1


Figure 26

70.37%

29.63%

Post-Game Survey Statistics CompletionRatio - 2.37:1


Figure 27

57.69%

42.31%

Scavenger Hunt ParticipationRatio - 1.36:1


Figure 28

HuskyHunt 21

72.60%

27.40%

Players Sharing on Social MediaRatio - 2.64:1


Figure 29

69.35%

30.65%

Players Creating an AliasRatio - 2.26:1


Figure 30

61.64%

38.46%

Players Completing All ModulesRatio - 1.6:1


Figure 31

68.97%

31.03%

Players Completing Final AssessmentRatio - 2.22:1


Figure 32

Players Failing Final AssessmentRatio - 1.2:1


HuskyHunt 22

Figure 33

The data shows that badges did not have any positive impact in engagement. In fact most data show players without badges progressed further and participated more. In all categories players with badges participated at approximately a 2:1 ratio to those without badges, which is consistent with the overall badge distribution. Furthermore, the categories of scavenger hunt statistics, all modules completed actually show a ratio approaching 1:1 indicating that players without badges had a slightly higher engagement level than those with badges. Finally, the data shows that of the 18 possible game levels players with badges abandoned the game by level 7.32 while players without badges abandoned the game by level 8.8 indicating that players without badges were more engaged in completing the game.

The only category where there is some evidence that players with badges performed better than those without is ‘players providing credentials.’ Fewer players with badges provided their username/password combination than those without, however there were only 11 total players that provided their credentials so the comparison of this data is likely statistically insignificant.

The quantitative data describes similar results. The players interviewed at the awards dinner had mixed opinions regarding the influence badges had on gameplay with most players indicating that the badges had no influence on their desire to play the game. Two players, however, indicated strongly that they liked receiving the badges and felt some amount of reward when they received a badge. These same players suggested that badges could have been displayed more prominently and could have provided additional extrinsic motivation if other awards were associated with them.

I collected demographic data related to student status (Freshman-Senior) and Sex. Evaluating a small portion of data to compare game interactions between Male and Female provided the following results:

44.27%

55.73%

Active Player Distribution by Sex

Male Players % Female Players %

Figure 34

Figure 34. Illustrates the total overall player distribution by sex.

HuskyHunt 23

52.29%47.71%

Pre-Game Survey Completion By Sex

Male Players With Badges % Female Players With Badges

Figure 35

Figure 35. Illustrates the player distribution that participated in the optional pre-game survey.

52.63%47.37%

Post-Game Survey Completion By Sex


Figure 36

Figure 36. Illustrates the player distribution that participated in the Post-Game Survey.

51.16%48.84%

Players Creating Aliases By Sex


Figure 37

Figure 37. Illustrates the players that created a game alias.

HuskyHunt 24

Because these sample results did not show any statistically significant differences I chose not to pursue additional results using student status as I expect the results to be similar. I had expected to see a difference between males and females creating aliases and was surprised to see no difference.

The final assessment metric is the single most concerning because 3 of the 18 modules were related to password management in some way and the 11 people that submitted their credentials represents 37.9% of the players that took the final assessment

Reflections SectionIt was my expectation prior to starting the game that digital badge awards would provide

significantly improved incentive to play the game, which would result in greater learning transfer due to engagement. While the data indicates no significant difference between the players populations I continue to believe that awarding digital badges would have a positive impact on game engagement. More accurate data to support this theory could be gathered if the following game changes were made:

1. The digital badges need to be displayed on the users score and profile more prominently. 2. There was a digital badge awarded at the end that was transportable outside the game.

This would increase badge value by enabling players to leverage this training experience in places like LinkedIn

3. The game was run at multiple institutions so that a greater sample set of data could be collected.

4. The game was run with entire player populations receiving, or not receiving badges. I believe having a mixed population in the same game instance confuses the purpose and significance of badges.

One of the challenges of the data collection was related to the fact that there was a coding error that allowed players to register for the game, and become active, prior to the actual start of the game. While this didn’t affect gameplay or scoring it caused the ratio between players with badges and those without to be approximately 70:30 while it was intended that the ratio would be 50:50. We did not have the luxury of fixing this without unregistering players and we felt it was more important to have players register than it was to restart and risk potentially losing players.

I also think that doing individualize interviews would have been more effective and accurate than doing a roundtable interview. Individualized interviews would have provided more data points and a greater ability to attempt to identify trends that I was able to do with the data collected from a group.

This version of the game included an introductory video of me and Jonathan Moore describing the game. I wanted players to be able to understand why the game was being run and get a sense of who was running. However, I do think that this video may have caused a few of the players to provide their credentials at the final assessment due to a certain established level of trust created by the video. I debated, for years, whether a practical final assessment of asking

HuskyHunt 25

people for their username and password a good idea and continue to be concerned about it. It did provide an excellent (and alarming) data point that I do believe would remain statistically consistent across a larger player population, but I’m not actually sure that our implementation of that component taught players anything about the importance of password secrecy and may actually have been discouraging.

I underestimated the time required to create a high quality video, and game launch timing forced me to use a video that I wasn’t particularly happy with. The video script content was good (Appendix G – Awards Ceremony Agenda and Interview Questions), but the video setting was stale and my demeanor was too formal. This s can be addressed in future versions now that I better understand how to approach producing a video. In general I found collecting video for this project to be challenging due to lack of good equipment and the ad-hoc nature of many of the discussions regarding game planning and implementation.

I feel like the data collected through electronic processes was adequate and sufficient to address the questions I had when initially considering the scope of the project and I think it could have been especially interesting if the overall player population was larger. However as I review the data I have I continue to identify questions that would be useful to aid in future game development that cannot be answered by the data I’ve currently collected. I believe I’ve approached some of the questions too narrowly and addressing that limitation for future implementations of the game would produce more interesting data. I am also confident that I would be able to provide much better data and draw more complete correlations, with the data I have, if I had a stronger background in statistics.

I should have considered additional ways to collect more qualitative data. I placed a greater emphasis on application generated data (and continue to believe this provides the best data) however as I listened and transcribed the interviews I did I realized that actual personal accounts are very helpful in making improvements. I missed opportunities, during scavenger hunts, to be able to talk to game players and get additional anecdotal game information. I also found running the formal focus group to be a different experience that I had expected. I tried to keep the conversation structured, but informal, and am not sure that is the most effective approach to obtaining high quality data. It would likely be more effective to interview each person individually and it would enable more complete data collection across all questions and prevent participant’s responses from being influenced by other people.

Overall, though, I feel that this project went extremely well and the data I collected does provide me valuable insight as I look to do design and gameplay updates. While I did not obtain the results I expected or hoped for I feel like I’ve identified a tool, in HuskyHunt, that can be leveraged to continue pursuing my theory that digital badges can and do provide improved engagement and learning transfer.

Future Recommendations Section

The data indicates HuskyHunt can be an effective training tool. In the current model of MOOC’s it is rare to get a completion rate above 1% (although participant numbers are much larger) and HuskyHunt, even with its modest active player numbers, obtained 15% completion. I realize that the incentives contribute to that total, but it still significantly higher than an average

HuskyHunt 26

MOOC completion percentage. I’m confident that the participation could be increased if we improve our scoring such that players can catch up if they fall behind. Making the scavenger hunt occur online would also help, as the current requirement to be a Storrs student is limiting participation.

However, the player numbers do have to be increased to justify the effort of running the game. The advertising, coordination, content generation and scavenger hunts require significant staff time and cannot continue if the player counts don’t increase. Partnering with the OneCard office has shown that the game can easily be extended beyond information security content and other departments have shown interest in using the tool as well. If we can tie the start of the game into the start of the semester, especially the fall semester, and leverage the framework as a training tool for other departmental activities that occur at that time I believe we could significantly increase participation in the game and provide greater institutional value. My goal is simply to expose as many students as possible to critical security concepts, but recognize that this is not high on their list of concerns. I’m willing to explore significantly expanding content beyond information security if it enables me to ultimately present fewer, but critical, information security concepts to a greater number of students.

We have also had conversations with a variety of other Colleges and Universities regarding an inter-university ‘HuskyHunt’ challenge, run during security awareness month in October. This could provide the potential to leverage school pride to increase engagement, as well as helping other institutions train their students. We are hoping that we can run this across Connecticut’s four state universities next year.

I have also considered, and am exploring ways to implement, using HuskyHunt as a tool to provide data access. Many data protection regulations, HIPAA for example, require that people with access to certain types of data are trained. I think it would be possible to leverage HuskyHunt as that training vehicle and utilize the badge awards as access tokens. This would significantly strengthen the importance of badge awards, as they could then be tied directly to system privileges and would reinforce to the users that the effort they extended to learn information provided value.

One of the major challenges of digital badges remains their value for badges outside of the ecosystem that awarded the badges and a long term desire of mine is to identify ways to establish that value. Currently there is little incentive to display the ‘HuskyHero’ badge, which is the final badge, on a LinkedIn page, as it has no meaning to people that did not participate in HuskyHunt. I think that the ideas I’ve described above, which are all intended to increase game participation and purpose, would significantly help increase its purpose and value and continue to strengthen the idea that digital badges have value and can provide participation incentives.

HuskyHunt 27

Appendices

Appendix A - Social Network Reach

Average # of Followers (age 18-21) = 200

HuskyHunt 28

Appendix B - Game Schedule and Module Format

HuskyHunt 29

Appendix C - Sample Player Badge Profile Page

HuskyHunt 30

Appendix D - List of All Available Badges

HuskyHunt 31

HuskyHunt 32

HuskyHunt 33

HuskyHunt 34

Appendix E - Introductory Video ScriptJason:

Hi, I’m Jason Pufahl the CISO for the Univ. of Ct. If you have any questions regarding data security or huskyhunt please email me at [email protected]

Jon: Hi, I’m Joh Moore, I’m the huskytech manager, student technology services and you can contact me at xxxx.

Jason:

HuskyHunt is a security awareness game that we designed to give you practical tips you can use at school or home regarding computer security, technology or online privacy. Our goal is for you to understand some of the most common risks and to provide you tools to better protect yourself or your device.

Jon:

Like Jason said, huskyhunt is an online game that will start October 6 and run for 3 weeks. Each day a new level will appear at a random time with content related to tech security. Be the first to beat the level and you can win a number of prizes. Earn points as you progress through levels and you will compete for the grand incentive, $500 from the UConn coop. The game is student focused but faculty and staff are welcome to play, just not eligible for the incentives.

Jason:

This year we will be running the game in 2 versions concurrently. One will award badges for activities and one won’t so don’t be surprised if you have a different version than one of your friends. The information we collect will be used to improve future versions of the game.

Finally, there is a short survey immediately following this video. If you would take a few minutes to complete that I would appreciate it. Thanks and enjoy the game.

Jon:

Join the hunt at huskyhunt.uconn.edu.

mailto:[email protected]

HuskyHunt 35

Appendix F - Pre-Game and Post-Game Survey QuestionsPre-Game Survey

Why are you playing the game?o To learn about security and privacyo To win prizeso Saw it advertised and just decided to try it

Have you played the game before?o Yeso No

How would you describe your attitude towards securityo I am very security consciouso I know there are risks but am not concernedo I don’t think about security at all

How would you describe your attitude towards personal privacyo I am very privacy consciouso I know there are risks but am not concernedo I don’t think about privacy at all

How would you describe your attitude towards technology?o I am a technology enthusiasto I use technologyo I don’t like technology but am required to use it

Do you expect to complete the whole game?o Yeso Noo Maybe

Is there anything specific you hope to learn in this game?o Open ended

Post-Game Survey

Did you enjoy the gameo Yeso Somewhato No

Did the badge awards provide additional motivation to complete modules?o Yeso Somewhato No

Did you learn something about Security, Privacy or Technology from the game?o Yes, I learned a lot of useful informationo Yes, but I knew most of the information alreadyo No

HuskyHunt 36

Did your attitude regarding security change?o Yeso Somewhato No

Did your attitude regarding personal privacy change?o Yeso Somewhato No

Did your attitude regarding technology change?o Yeso Somewhato No

How likely would you be to recommend HuskyHunt to a friendo Very Likelyo Maybeo Not Likelyo No way

Describe the game contento It was interestingo It was uninterestingo It was OK

Do you have any suggestions for game improvement?o Open Ended Question

HuskyHunt 37

Appendix G - Awards Ceremony Agenda and Interview QuestionsAwards Ceremony Notes and Agenda

Awards gathering planning

Pizza/drinks/cookies provided Invite a limited set of players (to be determined based on player pool. We will limit the

invites so that we assume approximately 75 attendees. We may select top 100 players, or perhaps some percentage of users that have completed and will make this determination at the end of the game.

Serve pizza immediately

Awards

Thank players for coming Describe some major outcomes of the game (player numbers, unique awards) Introduce desire for a conversation (below) Award top 3 prizes

Q&A

This will be loosely structured and as conversational as possible. I am assuming that pre-game and post-game surveys have gathered some consistent data. I will attempt to get the information related to my key goals, but will also allow the conversation to evolve organically. This data will likely not be associated to an individual but the outcome artifact will be an aggregate compilation of the group.

Key goals for this discussion:o Did they learn anything?o Did they feel content was broad enough?o Was there anything particularly useful?o Why did they complete the whole game?o What are the thoughts around badges?o What would they like to see change?o How did they find out about it?o Why do they think the participation was low?o

I expect the whole event will last between 30 and 45 minutes at most.

HuskyHunt 38

Appendix H - Awards Dinner Interviews Transcriptions

Awards Dinner Discussion

Informal Interview 1 – From Evernote Recording 20141029 17:06:05

Did you do it before: A few people did

Did you finish it: A few people did

Didn’t manage to finish it either time, because there was no opportunity to catch up when the player fell behind.

Player didn’t understand the point decay. Ray, the developer, described the point decay and solicited some feedback related to how to implement it better so that players could catch up.

Did anyone learn anything?

One person did – about HTTPS specifically

Another learned about UConn Software licensing options


Did anyone watch the video? – Yes, people did.

Did anyone find the video helpful – No particular response.

Did you finish?

Rebecca Day - she finished it because she started and wanted to win.

Michael Lau - played because they were in a position to win.

2 people played the game, were off campus, and couldn’t really do the scavenger hunt.

Michael Lau - A person described a timing issue where the module announcements may come after the module was actually released. Described the activity as a ‘mindless speed game.’


Rebecca Day

Won because she timed the module releases

She felt it would have been more interesting if the game was more difficult and had a bit more of a quest feel.

HuskyHunt 39

Girl says that it was annoying when it came out during class and she couldn’t respond quickly enough

Formal Interview 4 – From Evernote Recording 20141029 17:41:56 – ~ 21 Minutes long. There is an accompanying video as well, which is this exact discussion captured in video.

Question 1 – Did you learn anything?

Learned ‘stuff’- some of it stuck with him Importance of using UConn Security vs UConn Guest. How stuff gets stolen

Nobody felt the content wasn’t valuable, but perhaps not quite challenging enough

Question 2 – When would you like to pick up prizes?

Generally prefer prizes at end of week 1 person expressed desire not to have to go so far for scavenger hunt and to make

them more flexible for prize awards

Question 3 – How many of you completed the whole game

6

Question 4 – Why didn’t you finish the whole game?

Discouraged because of inability to catch up Just didn’t have time to do the game with their coursework

Question 5 – Did the possibility of earning badges matter?

1 person clear that they didn’t care 1 person did think it was ‘cool to see them’ like a reward system. Thought it was

fun Rebecca Day – liked earning them as well. Would have liked to also tie a point

incentive to the badges in some way. She liked collecting them all. 1 person said they just liked seeing points increase, didn’t care about the badges

Question 6 – How many of you play online games?

About Half

Question 7 – Did you have any questions from other players because you had badges and they didn’t?

No, because people generally don’t know about the game

Question 8 – How did you hear about the event? (Asked again later by Jon Moore)

Flyer

HuskyHunt 40

People felt most people didn’t know about it. People felt there was no reason to start after day 1 Word of mouth (3) Daily Digest (3) There were still dining table advertising from the last time the game was run Posters in the Library

Question 9 – If we ran it again how many of you would play again?

All of them. Michael Lau changed his mind….I think perhaps he liked the personal touch of

this discussion. It is something that would likely be worthwhile to do more of in the future.

Question 10 – What would you like to see in the game?

Shorter time frame Ability to catch up Closer Scavenger hunts Ability to get the regional campuses involved (scavenger hunt limits this)

Significant ancillary discussion about how to get people to know that the game is running.

HuskyHunt 41

Carr-Chellman, A. A. (2011). Instructional design for teachers: improving classroom practice. New

York: Routledge.

Digital Badges for Microcredentials, Student Engaement & Persistence | Online Learning Consortium,

Inc. (2012, June 20). Retrieved December 6, 2014, from

http://olc.onlinelearningconsortium.org/effective_practices/digital-badges-microcredentials-

student-engagement-persistence

Digital badges - Wikipedia, the free encyclopedia. (n.d.). Retrieved August 9, 2014, from

http://en.wikipedia.org/wiki/Digital_badges

Duffy, T. M., & Jonassen, D. H. (Eds.). (1992). Constructivism and the technology of instruction: a

conversation. Hillsdale, N.J: Lawrence Erlbaum Associates Publishers.

elearnspace › MOOCs are really a platform. (n.d.). Retrieved from

http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform/

ISTE Standards for Teachers. (n.d.). Retrieved December 6, 2014, from

http://www.iste.org/standards/standards-for-teachers

Subramaniam, P. R. (2009). Motivational effects of interest on student engagement and learning in

physical education: a review. International Journal of Physical Education, 46(2), 11–19.

Documents

HuskyHunt Technology Integration project