Butterfly Effect:An Augmented Reality Puzzle Game

Marleigh Norton

Blair MacIntyre

Steven DowMaribeth Gandy

Game Overview

• Spatial puzzle where physical world is not modeled

• Virtual butterflies in physical environment• Catch them!

– Butterflies are stationary– Challenge comes from reaching the butterflies

• Bring butterflies into reach by rotating the virtual world– Virtual world rotates in 90 degree chunks about a

player defined axis.

Equipment

Head-Mounted Display

Tornado Stick

Rotations

• Tornado Stick controls Virtual Axis– First Button – Place / Remove Axis– Second Button – Pause / Unpause Axis– Third Button – Rotate Butterflies

Tornado Stick

• People had trouble understanding rotations– Not everyone is used to

3D geometry– Though 3D puzzles have worked before

• Change the metaphor, not the interaction– “Tornado Stick”

Butterfly Effect

Butterfly Effect Video

QuickTime™ and aH.263 decompressor

are needed to see this picture.

Design Context

• Start with medium, then design game– Mapping physical and virtual worlds is hard– AR works best when user is moving slowly

• Tracking more accurate• Safer for player – Cognitive Tunneling

Design Context

• Home environment– Run on next generation game consoles– Must be easy to set up – Minimize use of special equipment

1st Prototype: 2D Paper

2nd Prototype: 2D Interactive

3rd Prototype: Video

Tuning

• Two Major Challenges– 90 degree rotation constraint– Understanding virtual space and physical space

• Possible Changes– Loosening rotation constraint– Adding more manipulations (translations)– Adding better depth perception cues

Open Issues: Depth Perception

• Static Depth Cues

Open Issues: Depth Perception

• Remove Automatic Depth Cues

Open Issues: Depth Perception

• Remove Unavailable Depth Cues

Open Issues: Depth Perception

• Texture probably the best

Without Grid

With Grid

Is Occlusion Really Impossible?

• Ad hoc modeling– Automatic modeling during calibration– Fiducials

• Real time depth– Depth from stereo

• Something to be explored

Virtual Obstacles - Bees

• No consequences for using poor strategy

• Bees behave like butterflies• “Sting” if caught• Penalty

– Lose life– Butterflies escape

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Questions?

Old Slides from Past Talks

(In case people as a question I have slides for.)

Butterfly Effect:An Augmented Reality Puzzle Game

Marleigh Norton

Technical Advisor: Blair MacIntyre

Person Walks Through Game Area

Person Walks Through Game Area

Person Walks Through Game Area

Derive Walls from Path

Derive Walls from Path

Why this is Fun

• Fun in the same ways Rubik’s Cubes are fun

• Caillois’s Four Categories of Games [1]– agôn, alea, mimicry, ilinx

• MacTavish’s “astonishment at visual and auditory technology” [2]

Open Issues

• Scoring

• Capture Interaction

• Depth Perception

Open Issues: Scoring

• Score based on efficiency, not speed

• Penalized for moving too much– Accrue score as you walk– High score bad

• Problem: High Score Traditionally Good

Open Issues: Capture Interaction

• Location-Based Capture– Capture butterfly by getting close enough– Pros: Technically simple. No additional cost.– Cons: Precise head tracking more important.

Less thematically appropriate. Less fun.

• Butterfly Net Capture– Capture butterfly in a net– Pros: More active, thematically appropriate

interaction.– Cons: Requires an additional tracking system,

likely expensive.

Open Issues: Depth Perception

• Judging depth in virtual worlds is difficult– Overcoming this difficulty is part of the game, yet it

should not be impossible

• Problem: What can we do to help the player understand the relative locations of the butterflies?

• Research on Depth Perception yields several techniques [4] [5] [6] [7]

Questions & Feedback

• Scoring – Efficiency

• Capture Interaction– Location-based capture– Butterfly Net Capture

• Depth Perception– Which cues to use? More ideas?

• Other Feedback

References: Game Theory

• [1] Caillois, Roger, “Man, Play, and Games,” trans. Meyer Barash, University of Illiniois Press, 1961 (orig.1958) ch 2.

• [2] Mactavish, Andrew, “Technological Pleasure: The Performance and Narrative of Technology in Half-Life and other High-Tech Computer Games”, in: Geoff King and Tanja Krzywinska (eds.) Screenplay: Cinema/ Videogames/ Interfaces (London: Wallflower Press, 2002) 33-50

References: Depth Perception 1

• [4] Cutting, J. E., and P.M. Vishton, 1995. “Perceiving layout and knowing distance: The integration, relative potency and contextual use of different information about depth”, in Perception of Space and Motion, W. Epstein and S. Rogers, Eds. Academic Press, New York, 69--117.

• [5] Furmanski, Chris, Ronald Azuma, and Mike Daily. “Augmented-reality visualizations guided by cognition: Perceptual heuristics for combining visible and obscured information”, in Proc. IEEE and ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2002) (Darmstadt, Germany, 30 Sept. - 1 Oct. 2002), pp. 215-224.

References: Depth Perception 2

• [6] Sinai, M.J., W. K. Krebs, R. P. Darken, J. H. Rowland, and J. S. McCarley, 1999. “Egocentric distance perception in a virtual environment using a perceptual matching task”, in Proceedings of the 43 rd Annual Meeting Human Factors and Ergonomics Society, 43, 1256-1260.

• [7] Surdick, R. T., E. T. Davis, R. A. King, and L. F. Hodges, 1997. “The perception of distance in simulated visual displays: A comparison of the effectiveness and accuracy of multiple depth cues across viewing distances”, in Presence: Teleoperators and Virtual Environments 6, 5 (October), 513--531.

References: DART

• [8] MacIntyre, Blair, Maribeth Gandy, Steven Dow, and Jay David Bolter. "DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences." To appear at conference on User Interface Software and Technology (UIST'04), October 24-27, 2004, Sante Fe, New Mexico.

Implementation Status

• It’s playable!*– Populate the world with a pre-defined set

of butterflies– Butterflies can be captured– Tallies of caught and remaining butterflies

are displayed– Butterflies can be rotated about the axis– Physical axis interface

* When all of the equipment is working.

The Screen

Feedback

• Rotation Interface

• Depth Perception

• Score

• Aesthetics

Rotation Interface

• Problem: Rotating about an axis? Huh?– Suggested alternatives:

• Randomization• Jar, Mini-version of gamespace

– (Rotation about a point)

• “Tornado Stick”

• Changes: Tornado Stick– Axis/Tornado Stick only interface in which all

manipulations are valid moves– Arrows pointing to butterflies’ final location

Depth Perception

• Problem: It’s hard to tell where the butterflies are, especially when they’re “behind walls”– Complication: can’t change appearance of

butterfly based on presence of wall(or can we…?)

• Changes: Add grid with drop shadows– Also, it’s less of a problem when you’re

actually playing

Score

• Problem: for Efficiency-based score, lower was better (confusing players)– Suggestions

• Timer• Subtract from High initial score• No Score

• Changes: No Score– Score isn’t really that important– Was causing more problems than it solved

Aesthetics

• Problem: What will the game look like?– Collecting the butterflies should be

compelling

• Changes: Aesthetics more fully fleshed out– Childlike graphic design with scanned

crayon textures– Player is frog, eating butterflies

Frog has Butterfly in Sights

Frog Catches Butterfly

Frog Eats Butterfly

Future Work: Finish Implementation

• Sounds

• Graphics

• Animations

• Refine Stick– Are those the right buttons?– Buttons or Switches or ???– InertiaCube Issues with Ferris

Environments

Questions?

Thanks to the Augmented Environments Lab, especially:

• Blair MacIntyre

• Steven Dow

• Enylton Machado Coelho