Final Year Project 2004/2005Final Year Project 2004/2005LYU0402LYU0402

Augmented Reality Table for Augmented Reality Table for Interactive Card GamesInteractive Card Games

Supervisor: Prof Michael LyuSupervisor: Prof Michael Lyu

Presented by: Kevin Chow, Albert LamPresented by: Kevin Chow, Albert Lam

OverviewOverview

IntroductionIntroduction ObjectiveObjective Trading Card GameTrading Card Game Architecture of ARTArchitecture of ART Implementation in Second TermImplementation in Second Term ExperimentExperiment ConclusionConclusion Future WorkFuture Work

IntroductionIntroduction

What is What is Augmented RealityAugmented Reality??

• Mixed Reality combines the content from Mixed Reality combines the content from the real world with virtual imaginarythe real world with virtual imaginary

• Augmented Reality is a subset of this Augmented Reality is a subset of this where virtual content is overlaid into real where virtual content is overlaid into real objects of the worldobjects of the world

Introduction (Con’t)Introduction (Con’t)

• Paul Milgram characterized Mixed Paul Milgram characterized Mixed Reality interfaces on his “Reality-Reality interfaces on his “Reality-Virtuality Continuum” in 1994Virtuality Continuum” in 1994

Mixed Reality (MR)

VirtualEnvironment

Augmented Virtuality (AV)

AugmentedReality (AR)

RealEnvironment

Reality-Virtuality (RV) Continuum

Introduction (Con’t)Introduction (Con’t)

An AR system has the following three An AR system has the following three main characteristics:main characteristics:• Combines real and virtual objects in a Combines real and virtual objects in a

real environmentreal environment• Runs interactively, and in real timeRuns interactively, and in real time• Registers virtual objects onto the real Registers virtual objects onto the real

worldworld

Introduction (Con’t)Introduction (Con’t)

ART stands for ART stands for Augmented Reality TableAugmented Reality Table• User can play Card Games on the ART.User can play Card Games on the ART.

• ART can enhanceART can enhance the visual effect the visual effect during the playingduring the playing of the card game. of the card game.

ObjectiveObjective

Develop a generic Interactive ART Develop a generic Interactive ART Trading Card Game application. Trading Card Game application.

Enhance the traditional card games:Enhance the traditional card games:• Visual effectVisual effect• Sound effectSound effect• Complex calculationComplex calculation

Trading Card GameTrading Card Game

Trading Card GamesTrading Card Games• Two players play a match.Two players play a match.• Players play cards to fight with each Players play cards to fight with each

other. It includes summoning monster, other. It includes summoning monster, casting magic, or setting traps, etc.casting magic, or setting traps, etc.

• We choose We choose “YU-GI-OH”“YU-GI-OH” as our as our implementationimplementation

Trading Card Game (Con’t)Trading Card Game (Con’t)

Trading card game includes five kinds of Trading card game includes five kinds of actionsactions• RestoreRestore• Draw cardsDraw cards• Play cardsPlay cards• Attack/ChallengeAttack/Challenge• Discard cardsDiscard cards

An Example - YU-GI-OHAn Example - YU-GI-OH In YU-GI-OH, Different information are In YU-GI-OH, Different information are

stored on the card.stored on the card.

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

Game Mat of YU-GI-OHGame Mat of YU-GI-OH Different kinds of card must be put inside Different kinds of card must be put inside

particular card zonesparticular card zones

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

LP : 2000 LP : 2000

ATK : 500 ATK : 1200

1200 – 500= 700-700

LP : 1300

An Example - YU-GI-OH (Con’t)An Example - YU-GI-OH (Con’t)

LP : 1300 LP : 2000

DEF : 1300 ATK : 1200

1300 – 1200= 100-100

LP : 1900

ArchitectureArchitecture

Hardware SetupHardware Setup• Plasma monitorPlasma monitor• Overhead cameraOverhead camera

System architectureSystem architecture• Augmented Reality Perception Augmented Reality Perception • Game CoreGame Core• DatabaseDatabase• Game Enhancement Game Enhancement

Hardware SetupHardware SetupOverhead mounted camera

Analyze the video captured from the camera, and generate output to the screen

The plasma monitor for playing trading card games

Plasma monitor act as a table

Hardware Setup (Con’t)Hardware Setup (Con’t)

ComputerComputer• processes perceiving imageprocesses perceiving image• generates visual and audio enhancementgenerates visual and audio enhancement• controls game flowcontrols game flow

Plasma MonitorPlasma Monitor..• acts as the Game Tableacts as the Game Table• displays computer-generated scenedisplays computer-generated scene

Overhead mounted cameraOverhead mounted camera• captures the cards and the screen of the captures the cards and the screen of the

plasma.plasma.• only input of the systemonly input of the system

System ArchitectureSystem Architecture

Game Core ModuleGame Core Module Perception ModulePerception Module Database ModuleDatabase Module Game Enhancement ModuleGame Enhancement Module

System Architecture (Con’t)System Architecture (Con’t)

ART Card Game

ART Card Game CoreGeneric Card Game Database

Augmented Reality Perception

CommandDetector

Calibration

Card Detector Game Enhancement

Video Decoder

Card Locator

Calibration

Card Recognizer

3D Animation

Image Database

Input Analyzer

Sound Effect

Game CoreRule Database

DisplayCamera

Card Information

Input Information

Environment Information Output Generator

Game Manager

Rule ManagerCard Database

ImplementationImplementation

Database ModuleDatabase Module• Access card informationAccess card information• Access game rules and card rules Access game rules and card rules • Identify cardsIdentify cards

GameCore ModuleGameCore Module• Control game flowControl game flow• Store game stateStore game state• Rulebase game engineRulebase game engine

Database ModuleDatabase Module

Contains all game informationContains all game information

Card DatabaseCard Database• Contain all cardContain all card

information information• Simple editorSimple editor

Rule DatabaseRule Database• Contain all game ruleContain all game rule

and card rule and card rule• Simple editorSimple editor

Image DatabaseImage Database

Image DatabaseImage Database• Query with a card imageQuery with a card image• Retrieve card idRetrieve card id• Use improved block matching algorithmUse improved block matching algorithm

?

Select the best matched Select the best matched card from the candidates card from the candidates

• Split the captured image Split the captured image into 4 channelsinto 4 channels

• Compare only inner imageCompare only inner image• Compare the different Compare the different

channels to the channels to the database separatelydatabase separately

Block Matching AlgorithmBlock Matching Algorithm

Block Matching AlgorithmBlock Matching Algorithm

• Find the pixel difference byFind the pixel difference by

• Reject the image if the pixel difference Reject the image if the pixel difference is larger than the thresholdis larger than the threshold

• Accept the image with minimum pixel Accept the image with minimum pixel differencedifference

w

i

h

jDQDQP jiIjiIIID

1 1

2)),(),((),(

Improved Block Matching AlgorithmImproved Block Matching Algorithm

• However, the Block However, the Block Matching Algorithm cause Matching Algorithm cause some error in real time some error in real time gamegame

• Solution, we use an Solution, we use an improved Block Matching improved Block Matching AlgorithmAlgorithm

• Divide the image into 9 Divide the image into 9 squaressquares

• Apply Block Matching Apply Block Matching Algorithm to each squaresAlgorithm to each squares

GameCoreGameCore

Core Part of ArtTableCore Part of ArtTable Contains:Contains:

• Input Manager – handles inputInput Manager – handles input• Output Manager – handles outputOutput Manager – handles output• Rule Manager – stores rules and inference rulesRule Manager – stores rules and inference rules• Game Manager – stores game statesGame Manager – stores game states

Control how these managers interact and Control how these managers interact and flow of the game.flow of the game.

GameCoreGameCore

GameCore

Output ManagerInput Manager

Rule ManagerGame Manager

1. Receive input

3. Update game states

2. Pass to game manager

4. Inference rules

7. Display output to player

5. Perform actions if any

6. Generate output if any

RulebaseRulebase

Why rulebase?Why rulebase?• Flexible – Card games are very flexible Flexible – Card games are very flexible

and some card effect are unpredictable.and some card effect are unpredictable.• Extensible – New cards can be extended Extensible – New cards can be extended

by adding new rules.by adding new rules.• Generic – New card games can be Generic – New card games can be

implemented by new rule models.implemented by new rule models.

RulebaseRulebase

Load game rulesLoad game rules Forward checking for predicateForward checking for predicate While conclusion reachedWhile conclusion reached

• Perform the action of the rulePerform the action of the rule• further forward checkingfurther forward checking

Predicate1 Predicate2 Conclusion ActionList

P.S. Action may be load another rule, remove a rule, reset a rule,update game states, or show some outputs, etc.

RulebaseRulebase

MainPhase PutCard1 NormalSummon

MainPhase PutCard2 PlaySpell

HandFromHell PlaySpell DestroyMonsterSetTarget

MysticTomato SendGraveYard SpecialSummonPutCard1

ExperimentExperiment

We have used 90 cards to do the We have used 90 cards to do the recognition experiment. The result is as recognition experiment. The result is as follow:follow:

Number of hitsNumber of hits number of cardsnumber of cards percentagepercentage

AlwaysAlways(No miss)(No miss)

7070 78%78%

SometimesSometimes(miss 1-5 times)(miss 1-5 times)

1616 18%18%

NeverNever(miss >5 times)(miss >5 times)

44 4%4%

Experimental Result DiscussionExperimental Result Discussion

Reasons decrease the accuracyReasons decrease the accuracy• Image are captured in low resolution. Image are captured in low resolution.

The small error in cards position The small error in cards position detection will lead to inaccuracy in detection will lead to inaccuracy in recognition process.recognition process.

• Reference images are recorded from Reference images are recorded from camera. Errors in recording will lead to camera. Errors in recording will lead to errors in recognition process.errors in recognition process.

ConclusionConclusion

We have designed and implemented We have designed and implemented the prototype of ART systemthe prototype of ART system

We have improved the card We have improved the card recognition algorithm with increased recognition algorithm with increased accuracy and efficiencyaccuracy and efficiency

We have developed a rulebase game We have developed a rulebase game engine which is generic and engine which is generic and extensible for card gamesextensible for card games

Future WorkFuture Work

Color CalibrationColor Calibration Improve in card recognition Improve in card recognition

algorithmalgorithm Game EnhancementGame Enhancement

End of PresentationEnd of Presentation

Thank you very muchThank you very much

Q & A SessionQ & A Session