Embed Size (px)
DESCRIPTION
Developing intuitive interactive applications that are easy to maintain by developers is quite challenging, due to the complexity and the many technical aspects involved in such applications. In this article, we tackle the problem in two complementary ways. First, we propose a gestural interface to improve the user experience when interacting with applications that require the manipulation of 3D graphical scenes. Second, we reduce the complexity of developing such applications by modeling their executable behaviour using statecharts. We validate our approach by creating a modular and extensible Java framework for the development of in- teractive gesture-based applications. We developed a proof- of-concept application using this framework, that allows the user to construct and manipulate 3D scenes in OpenGL by using hand gestures only. These hand gestures are captured by the Kinect sensor, and translated into events and actions that are interpreted and executed by communicating statecharts that model the main behaviour of the interactive application.
Citation preview
Statechart modeling of interactive gesture-based applications
Statechart modeling of interactivegesture-based applications
Romuald Deshayes Tom Mens
Software Engineering Lab
2010-2011
Deshayes Romuald – UMONS 1 / 20
Statechart modeling of interactive gesture-based applications
Table des matières
1 Problem Statement
2 Proof-of-concept application
3 Modeling interactive behaviour
4 Statechart models
5 The application framework
6 Conclusion and Future Work
Deshayes Romuald – UMONS 2 / 20
Statechart modeling of interactive gesture-based applications
Problem Statement
Outline : Problem Statement
Problem Statement
Deshayes Romuald – UMONS 3 / 20
Statechart modeling of interactive gesture-based applications
Problem Statement
Problem Statement
ChallengeDevelop intuitive interactive applications that are easy to maintain
Why is it a challenge ?Complex behaviourNondeterministic user actionsSame events lead to different actions
ContributionsGestural user interface for 3D objects manipulationModeling executable behaviour using a visual formalismValidation with an application framework
Deshayes Romuald – UMONS 4 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Outline : Proof-of-concept application
Proof-of-concept application
Deshayes Romuald – UMONS 5 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Proof-of-concept application
3D visual drawing toolUses gestures to create and manipulate 3D objectsOpenGL graphical libraryMicrosoft Kinect + NITE
Deshayes Romuald – UMONS 6 / 20
Statechart modeling of interactive gesture-based applications
Proof-of-concept application
Small video
Deshayes Romuald – UMONS 7 / 20
Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Outline : Modeling interactive behaviour
Modeling interactive behaviour
Deshayes Romuald – UMONS 8 / 20
Statechart modeling of interactive gesture-based applications
Modeling interactive behaviour
Modeling interactive behaviour
ContextHighly reactive event-driven systemsGesture-based interaction
Proposed solutionsVisual modeling language
Statecharts
Petri nets
Amenable to formal analysisEasier to evolveReduced complexity
Deshayes Romuald – UMONS 9 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Outline : Statechart models
Statechart models
Deshayes Romuald – UMONS 10 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Hand
Deshayes Romuald – UMONS 11 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Component
Deshayes Romuald – UMONS 12 / 20
Statechart modeling of interactive gesture-based applications
Statechart models
Statechart models - Scene
Deshayes Romuald – UMONS 13 / 20
Statechart modeling of interactive gesture-based applications
The application framework
Outline : The application framework
The application framework
Deshayes Romuald – UMONS 14 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework
Framework for developing interactive applicationsExecutable behaviour specified using statechartsClient-server architecture
Deshayes Romuald – UMONS 15 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework II
FeaturesGeneric
Abstract classes
Observer design pattern
Communication between statechartsUses Java SwingStates library
Deshayes Romuald – UMONS 16 / 20
Statechart modeling of interactive gesture-based applications
The application framework
The application framework III
OpenGLWidgetsBodyAdapters
Framework
-body : BodyBoyEvent extends VirtualEvent
-open : boolean-noCompTouched : boolean
ComponentEvent
-position : double[]-speed : double[]-pixelpos : int[]-lefthand : boolean
Body
Main Application
<<library>>SwingStates
+addAsListenerOf(s : Server)BodyAdapter
-serverSocket : DatagramSocketServer extends Thread
+handMovedX(e : BodyEvent)+handMovedY(e : BodyEvent)+handMovedZ(e : BodyEvent)+handClosed(e : BodyEvent)+handOpened(e : BodyEvent)
<<Interface>>BodyListener extends
EventListener
-machine : StateMachineHand
-machine : StateMachine+addComponent(c : Component)+addPointer(p : Pointer)+addScene(s : Scene)
Controller
-open : boolean-machine : StateMachine
Pointer
-boundingBox : Rectangle-machine : StateMachine
Component
-machine : StateMachineScene
+extend() : void+retract() : void+drawMenu(gl : GL) : void
ExtendableMenu+changeActiveTexture() : void+drawButton(gl : GL) : void+startCounter() : void
GLButton
+addButton(button : GLButton) : void+addText(text : String) : void+drawMenu(gl : GL) : void+spawn() : void+reduce() : void
SpawningMenu
+addButton(button : GLBUtton) : void+drawMenu(gl : GL) : void
ButtonMenu +drawMenu(gl : GL) : void+onModel() : void
ImageMenu
-selected : booleanModel
StateMachine
MainScene
3DModel
1..*
models
0..1
r ight
1..*
1..*
0..1
left
0..1selectedModel
1..**
1
Visual Paradigm for UML Standard Edition(University of Mons)
Deshayes Romuald – UMONS 17 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Outline : Conclusion and Future Work
Conclusion and Future Work
Deshayes Romuald – UMONS 18 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Conclusion and Future Work
StatechartsAppropriate and scalable formalism for modeling theinteractive gesture-based behaviour ?
ImprovementsExternalise/decouple statechart representation from frameworkCompare statecharts with Petri nets
Deshayes Romuald – UMONS 19 / 20
Statechart modeling of interactive gesture-based applications
Conclusion and Future Work
Future Work II
Apply our approach to other HCI applicationsOther types of natural interfacesCompare/combine with other UIMLControlled experiments, carry out user studies, empiricalstudies
Deshayes Romuald – UMONS 20 / 20
