Challenges in using MDE for HMI

Multi-Paradigm Modelingof HMI Applications

Tom Mens Romuald Deshayes

Software Engineering Lab

MoVES event, Universiteit Antwerpen, 20 December 2011Tom Mens – UMONS 1 / 23

Challenges in using MDE for HMI

HMI modeling

Outline : HMI modeling

HMI modeling

Tom Mens – UMONS 2 / 23

Challenges in using MDE for HMI

HMI modeling

Example of a HMI application

3D visual drawing toolHaptic (gestural) user interface(using hands) based on KinectCreation and manipulation of 3Dobjects, using OpenGLComplex UI behaviour representedusing statecharts www.youtube.com/

watch?v=lVcqzWTnpMY

Tom Mens – UMONS 3 / 23

Challenges in using MDE for HMI

HMI modeling

Multi-Paradigm Modeling for HMI

GoalUse MDE techniques to advance the state-of-the-art in HMI (HCI)

Multi-paradigm modeling. Existing approaches focus on :Task modeling : understand user tasks to improve UIs (cf.Jean Vanderdonckt - UCL)Presentation modeling : focus on visual appearance of theapplication (e.g., OpenGL)User modeling : focus on the user characteristicsBehavioural modeling : model how the application reacts tousers actions (e.g. event-based)Physical modeling : physical behaviour of real or virtual objects. . .

Tom Mens – UMONS 4 / 23

Challenges in using MDE for HMI

HMI modeling

Task modeling

Example : ConcurTaskTrees task model for a digital home

Tom Mens – UMONS 5 / 23

Challenges in using MDE for HMI

HMI modeling

Behavioural modeling

Example : Statecharts model of hand behaviour [DM11]

Tom Mens – UMONS 6 / 23

Challenges in using MDE for HMI

HMI modeling

Behavioural modeling

Example : Statecharts mode of UI component behaviour [DM11]

Tom Mens – UMONS 7 / 23

Challenges in using MDE for HMI

HMI modeling

Behavioural modeling

Example : Statecharts model of graphical scene behaviour [DM11]

Tom Mens – UMONS 8 / 23

Challenges in using MDE for HMI

HMI modeling

Behavioural modeling

Example : Petri net model of double click behaviour(Palanque - Toulouse) [NPLB09]

Tom Mens – UMONS 9 / 23

Challenges in using MDE for HMI

HMI modeling

Physical modeling

GoalModeling physical behaviour or interaction of virtual objects

E.g., falling, rolling, colliding, bouncing, breaking, . . .Requires continuous models (e.g., differential equations)

ChallengeCombining discrete and continuous models

Possible solution : hybrid automataDiscrete automata containing differential equations in the states

hybridAutomaton.png

Tom Mens – UMONS 10 / 23

Challenges in using MDE for HMI

Research Challenges

Outline : Research Challenges

Research Challenges

Tom Mens – UMONS 11 / 23

Challenges in using MDE for HMI

Research Challenges

Research Challenges

Top-down

Come up with domain-specific modeling languages (DSML) thatcombine multiple views on HMI modeling.

Bottom-upEnable formal reasoning over HMI models

Detect unintended behavioural problems (nondeterminism,non-termination, ...)Detect HMI vulnerabilities, inconsistencies, incompletenessFacilitate code generation. . .

Tom Mens – UMONS 12 / 23

Challenges in using MDE for HMI

Research Challenges

Formal reasoning - Example

Work by Combéfis and Pecheur to analyse HMI vulnerabilities[CP09]Using labeled-transition systems (LTS) semantics to generate andanalyse users’ mental modelsExample : LTS of user’s mental model of a lamp, compared to thereal system model.

Tom Mens – UMONS 13 / 23

Challenges in using MDE for HMI

Research Challenges

Formal reasoning - Example

Work by Combéfis and Pecheur to analyse HMI vulnerabilitiesExample : accident with Therac-25 medical system due to aincorrect operator-machine interaction

Fig. 7. Statechart model of the Therac-25 medical system [16]. The enter command is represented with

�

and the up command is represented with ↑.

X-ray beam (selectX), validating it (enter) and administeringthe treatment (fire), the X-ray command may or may notbe available depending on the execution followed in thesystem. This means that the system may end up either inthe X-ray or in the E-beam mode, non-deterministically andwith no observable difference. That behaviour is due to aninternal transition which occurs when the treatment has beenadministered, which represents the fact that the system is resetto its initial state. The revealed controllability issue indicatesthat there should be an observation informing the user when thesystem is reset. Adding a reset observation makes the systemfull-controllable and a minimal full-control mental model forit can be generated, with 24 states.

The other issue with that system, the well-known one,can also be detected with our framework, after fixing theother issues in the model. If the operator is not aware ofthe 8-seconds timer (or does not track the countdown), theissue described in [2], [16] can also be found. It suffices toturn the observation 8seconds into an internal transition τ andto make the system being reset when the operator pressesenter after the treatment has been administered. The lastactions of the returned counterexample (the most relevantpart) is: �. . . , selectE, up, E-beam, selectX, E-beam�. Thatcorresponds to the user selecting the electron beam mode, thenchanging his mind by pushing on the up button and selectingthe X-ray mode. After that, the system may either be in E-beamor X-ray mode.

B. The Video Cassette Recorder example

The second example is a model of a video-cassette recorder(VCR). That model has been obtained from an ADEPT modelwhich has been translated into a JPF statechart Java program,manually. The system has six main operating modes: play,stopped, fast forward, rewind, pause and record. There is onecommand to activate each of those modes and one additional

command to turn the system on or off. Moreover, there areseveral speeds for the fast forward and rewind modes. Thosespeeds are automatically chosen by the system according to theremaining tape length. Finally, the system enters automaticallyinto the rewind mode when reaching the end of the tape. Thesystem model has 1088 states and 3740 transitions. There areseven commands (one to activate each mode and a power button)and two observations (tape moving forward and backward).

Analyzing the system against the full-control relation high-lights some bad interaction that may happen and lead toconfusion by the operator. The issue comes from the [ON,STOP, 0.01] state which corresponds to the VCR being turnedon, the tape being stopped and the remaining length of tapebeing 0.01. In this state, when the user presses the playcommand, the system can go to one of the two followingstates: [ON, PLAY, 0.01] or [ON, REWIND FULL, 0.00]. Inthe first case, the system just switches in the play mode. In thesecond case, after the user presses the play button, the tapehas just moved forward until it remains zero, in which case thesystem automatically goes to rewind mode. The controllabilityissue pointed by the framework is that the pause commandis not available in the two states that are reachable from [ON,STOP, 0.01] after executing the action play.

Such kinds of error can be harmful. For the VCR example,suppose that the operator wants to press the play button andthen, someone calls him and he presses the pause button as,for him, it is an authorized operation. When he comes backto his VCR, he observes that the tape was rewinded up tothe beginning. That situation occurs because after pressing theplay button, the system can either transition into a state wherethe pause button is enabled or in another state where it isnot. For critical systems, such an error can have more seriousconsequences.

Tom Mens – UMONS 14 / 23

Challenges in using MDE for HMI

Research Challenges

Domain-Specific Modeling for HMI

DSMLHow to define a DSML that combines the multiple modelinglanguages/paradigms used for HMI ?

Tom Mens – UMONS 15 / 23

Challenges in using MDE for HMI

Research Challenges

Domain-Specific Modeling for HMI

Domain-specific model : specific to application domainE.g., robotics, gaming, home automation, virtualreality

Platform model : describes platform specificitiesmodality (vocal, tactile, WIMP)input devices (hardware sensors, keyboard,mouse, Kinect)output (screen, VR goggles, voice). . .

Transformation model : weaves both models into platform-specificmodel, using model transformation technology

Tom Mens – UMONS 16 / 23

Challenges in using MDE for HMI

Research Challenges

Visual formalismsAre used for formal reasoning and analysis over the models

Questions :Which formalisms to use for which purpose ?How to combine results from different formalisms ?How to feed back results into DSML ?

Remark : model transformations are models too, amenable toformal analysis

Tom Mens – UMONS 17 / 23

Challenges in using MDE for HMI

Research Challenges

Visual Formalisms

Use most appropriate formalism, at most suitable abstraction level,for a given purpose

Possible candidatesStatecharts : for event-driven, deterministic, reactive systemsPetri nets : for analysing concurrency-related issuesLabelled Transition SystemsHybrid automata (combining discrete and continuous models)Logic-based or constraint-based formalismsGraph transformation (e.g., for model transformation)

Tom Mens – UMONS 18 / 23

Challenges in using MDE for HMI

Research Challenges

Visual Formalisms

Which semantics to associate to each formalism ?Which tools to use for formal reasoning ?

Graph transformation tools (e.g., AGG, Fujaba, GrGen)Model manipulation tools (ATL, EMF, Epsilon, AtoMPM,TCore, . . . )HyTech tool (for hybrid automata)Model checkersSAT solvers. . .

Tom Mens – UMONS 19 / 23

Challenges in using MDE for HMI

Research Challenges

More challenges

How to facilitate evolution ?How to generate code from models ?How to develop generic framework (independent of particularDSM or platform model) ?Where and how to use graph transformation ?Killer app ? Augmented reality, gaming, home automation,robotics ?

Tom Mens – UMONS 20 / 23

Challenges in using MDE for HMI

Research Challenges

Thank you

Thank you for your attention !

Questions, suggestions ?

Tom Mens – UMONS 21 / 23

Challenges in using MDE for HMI

Research Challenges

Bibliography I

Sébastien Combéfis and Charles Pecheur.A bisimulation-based approach to the analysis ofhuman-computer interaction.In EICS, pages 101–110, 2009.

Romuald Deshayes and Tom Mens.Statechart modelling of interactive gesture-based applications.In CoDeisMoto Workshop, co-located with Interact 2011, 2011.

Tom Mens – UMONS 22 / 23

Challenges in using MDE for HMI

Research Challenges

Bibliography II

David Navarre, Philippe A. Palanque, Jean-François Ladry, andEric Barboni.Icos : A model-based user interface description techniquededicated to interactive systems addressing usability, reliabilityand scalability.ACM Trans. Comput.-Hum. Interact., 16(4) :56, 2009.

Tom Mens – UMONS 23 / 23