59
ReaX -- a reactive ReaX -- a reactive programming programming language for language for multimodal multimodal applications applications Nils Klarlund, AT&T Labs Nils Klarlund, AT&T Labs With contributions from With contributions from Anders Møller, Jennifer Anders Møller, Jennifer Beckham, Giuseppe di Beckham, Giuseppe di Fabbrizio Fabbrizio

ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Embed Size (px)

Citation preview

Page 1: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

ReaX -- a reactive ReaX -- a reactive programming programming language for language for multimodal multimodal applicationsapplicationsNils Klarlund, AT&T LabsNils Klarlund, AT&T Labs

With contributions from Anders With contributions from Anders Møller, Jennifer Beckham, Møller, Jennifer Beckham,

Giuseppe di FabbrizioGiuseppe di Fabbrizio

Page 2: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

GoalsGoals Introduce Introduce reaction treesreaction trees and and ReaXReaX, ,

a reactive scripting languagea reactive scripting language Argue reaction trees are what has Argue reaction trees are what has

been missing in event driven been missing in event driven programming notationsprogramming notations

Explain causality in simple Explain causality in simple operational and explicit terms operational and explicit terms without nondeterminism, fixed without nondeterminism, fixed points, intuitionistic logic, …points, intuitionistic logic, …

Page 3: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Motivation: Motivation: unfinished business in UIunfinished business in UI Time seek and undo missing or Time seek and undo missing or

inconsistentinconsistent Why a reactive core language is Why a reactive core language is

essential to a user agent or browseressential to a user agent or browser Especially if speech recognition is involvedEspecially if speech recognition is involved

How SMIL, a W3C technology for How SMIL, a W3C technology for multimedia presentation, represents multimedia presentation, represents important progress in controlling the important progress in controlling the user interfaceuser interface

Page 4: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

OpportunitiesOpportunities

ReaX work is conceptualReaX work is conceptual Exploiting trend: XML is turning Exploiting trend: XML is turning

trees into an important paradigm trees into an important paradigm of abstractionof abstraction

Exciting thing: Exciting thing: trees are actually now used by trees are actually now used by

software practitionerssoftware practitioners they open new venues for they open new venues for

programming language designprogramming language design

Page 5: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Talk overviewTalk overview

I. MotivationI. Motivation II. SMIL demoII. SMIL demo III. Event woesIII. Event woes IV. Solution: reaction treesIV. Solution: reaction trees V. Tie things togetherV. Tie things together Appendix. ReaX overviewAppendix. ReaX overview

Page 6: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Part 1. Motivation Part 1. Motivation spiralspiralUI issues UI issues

Central problem about eventsCentral problem about events

Page 7: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

My backgroundMy background

For a decade, I’ve been looking for a For a decade, I’ve been looking for a solution to work on the computer withsolution to work on the computer with Much less use of arms and handsMuch less use of arms and hands Reason: typing injuryReason: typing injury

What I have learned:What I have learned: Efficiency is the keyword to solve this problemEfficiency is the keyword to solve this problem Efficiency is Efficiency is thethe motivationmotivation for learning for learning

anything newanything new Don’t look for alternatives, look for more Don’t look for alternatives, look for more

efficient waysefficient ways

Page 8: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

User interface work User interface work driven by entropy ratedriven by entropy rate

Foot keyboardFoot keyboard Must have sufficiently many keys for repetitive Must have sufficiently many keys for repetitive

workwork Dictation system spoken command interfaceDictation system spoken command interface

Design ShortTalk, a “perfect language” for Design ShortTalk, a “perfect language” for commands, maximizing bits/secondcommands, maximizing bits/second

Errors that put system in strange states are Errors that put system in strange states are offensiveoffensive

Any error (by human or by machine) must be Any error (by human or by machine) must be immediately correctableimmediately correctable

For a hundred years, typewriter necessitated an For a hundred years, typewriter necessitated an education until delete button was inventededucation until delete button was invented

It has all to do with efficiencyIt has all to do with efficiency

Page 9: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

User interfaces take User interfaces take effortsefforts

I spent 8 years onI spent 8 years on

Foot keyboard design Foot keyboard design

““Perfect language”, butPerfect language”, but ““perfect implementation”perfect implementation” 10,000 lines of Emacs Lisp10,000 lines of Emacs Lisp

This work is currently being This work is currently being monetizedmonetized

Page 10: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Common UI problemsCommon UI problems Flash Web site: BBC “Life of Mammals”Flash Web site: BBC “Life of Mammals”

Embedded back button, no forward buttonEmbedded back button, no forward button ““Explore another species link” sometimes does not workExplore another species link” sometimes does not work Movie sound and animal sound not mutually exclusiveMovie sound and animal sound not mutually exclusive Accessibility issues: some common keyboard shortcuts Accessibility issues: some common keyboard shortcuts

don’t workdon’t work ATM: enter a wrong amount---and go through the ATM: enter a wrong amount---and go through the

whole cycle again after waiting for a slip of paper!whole cycle again after waiting for a slip of paper! IVR systems: say or press something wrong---and IVR systems: say or press something wrong---and

end up cornered!end up cornered! Voicemail: to hear that phone number again, Voicemail: to hear that phone number again,

rewind the whole message!rewind the whole message! Dictation systems: say something that is Dictation systems: say something that is

misinterpreted and end up in a strange state!misinterpreted and end up in a strange state!

Page 11: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Undo claimUndo claim

The lack of undo is a most The lack of undo is a most fundamental and most easily solved fundamental and most easily solved problem in speech and gesture problem in speech and gesture recognition interfacerecognition interface

The undo button would be the most The undo button would be the most frequently used of them all and frequently used of them all and should be sized accordinglyshould be sized accordingly

One success: the “back button” on One success: the “back button” on browsersbrowsers

Page 12: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Undo and seek are horribly Undo and seek are horribly implemented in generalimplemented in general

Even as check-pointing and recovery Even as check-pointing and recovery techniques known for yearstechniques known for years

The problem:The problem: In practice, it’s hard to doIn practice, it’s hard to do

But a programming language that But a programming language that incorporates classic knowledge about incorporates classic knowledge about recovery can alleviate almost all the recovery can alleviate almost all the complexitycomplexity

We will show that the principal obstacle We will show that the principal obstacle is a classic issue: understanding events is a classic issue: understanding events and synchronizationand synchronization

Page 13: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Part II. SMIL Part II. SMIL demodemoEleganceElegance

Semantic gremlinsSemantic gremlins

Page 14: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

SMIL demoSMIL demo

SMIL (Synchronous Multimedia SMIL (Synchronous Multimedia Language)Language) Declarative and very convincing Declarative and very convincing

approach to managing the sequential approach to managing the sequential and parallel activities of UIand parallel activities of UI

Does for the temporal dimension what Does for the temporal dimension what HMTL does for the two dimensions of HMTL does for the two dimensions of layoutlayout

Link to AVILink to AVI

Page 15: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

The SMIL declarationThe SMIL declaration<SMIL:seq repeatCount="indefinite"> <SMIL:seq repeatCount="indefinite"> <button class="time" end="c.click" id="c"><button class="time" end="c.click" id="c"> Click to launch!Click to launch! </button> </button> <SMIL:par end="clip.click" id="example"> <SMIL:par end="clip.click" id="example"> <span class="time">Launching!</span> <span class="time">Launching!</span> <SMIL:video src="spaceshuttle.avi" id="clip"> <SMIL:video src="spaceshuttle.avi" id="clip"> <SMIL:animate begin="10s" <SMIL:animate begin="10s" dur="5" to="true"dur="5" to="true" attributeName="mute"/> attributeName="mute"/> </SMIL:video> </SMIL:video> <SMIL:seq begin="rewind.click" dur="0"<SMIL:seq begin="rewind.click" dur="0" onend="rewind(example);"/>onend="rewind(example);"/> </SMIL:par> </SMIL:par> </SMIL:seq> </SMIL:seq>

Page 16: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Time seek and undo are Time seek and undo are hardhard

Because howBecause how eventsevents, which take place over time, and, which take place over time, and synchronizationsynchronization, which is instantaneous,, which is instantaneous,are related is not well-understoodare related is not well-understood

Once we have a model that tie these Once we have a model that tie these concepts togetherconcepts together solving undo and seek semantics solving undo and seek semantics

become rather simple become rather simple Main goal for rest of talk: understand Main goal for rest of talk: understand

events and synchronizationevents and synchronization

Page 17: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Part III. Event Part III. Event woeswoes

EmacsEmacs

StatechartsStatecharts

Sync arcsSync arcs

Page 18: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Event woes I: EmacsEvent woes I: Emacs

An event (mouse click, key press) is an An event (mouse click, key press) is an abstraction unitabstraction unit

Namely, in keyboard macrosNamely, in keyboard macros Events are queued in unread-command-Events are queued in unread-command-

eventsevents A macro is a sequence of eventsA macro is a sequence of events A macro can be assigned to a keyA macro can be assigned to a key Thus, macros are definable in terms of Thus, macros are definable in terms of

macrosmacros

Page 19: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Does execute-kbd-macro add events to Does execute-kbd-macro add events to front or to back of queue?front or to back of queue? Neither works in case of macro of macros!Neither works in case of macro of macros!

Analogy: consider a programming Analogy: consider a programming language where yet-to-be-executed language where yet-to-be-executed procedures are kept in a queue. One procedures are kept in a queue. One procedure at a time, from the front of the procedure at a time, from the front of the queue, is picked for executionqueue, is picked for execution Is this a good idea?Is this a good idea?

For ShortTalk prototype written in Emacs For ShortTalk prototype written in Emacs LispLisp Event model creates serious headaches Event model creates serious headaches

Renegade thinkingRenegade thinking

Page 20: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Event woes II: Event woes II: StatechartsStatecharts

Events are propositionsEvents are propositions Not consumed, but generated in a Not consumed, but generated in a reactionreaction On {e} as stimulus, least fixed point is On {e} as stimulus, least fixed point is

{e,f}{e,f}

*f gf fg

Now on {e}, Now on {e}, we get we get {e,f,g}{e,f,g}

Page 21: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

*f gf fg

MP1MP1

{I: * {I: * f |f |

J: g J: g f |f |

K: f K: f g }g }

I J K

gf gf fg

I J K

MP2MP2

{I: {I: g f |f |

J: g J: g f |f |

K: f K: f g }g }

Mystery programs 1 and 2Do this first

Then, this

Try others, then check

Try this first

Page 22: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Event Woes III: SMILEvent Woes III: SMIL

In SMIL, you can say “time containers I In SMIL, you can say “time containers I and J are to start at the same time”and J are to start at the same time”

““time container” = “interval” time container” = “interval” We use “thread”We use “thread” Example: two videos to be started togetherExample: two videos to be started together<video src=’head’ id=’I’ begin=’J.begin’/><video src=’head’ id=’I’ begin=’J.begin’/>

… …......

<video src=’side’ id=’I’ begin=’I.begin’/><video src=’side’ id=’I’ begin=’I.begin’/> Time arc from I to J and from J to I Time arc from I to J and from J to I

synchronizing start of videosynchronizing start of video

Page 23: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Binding world of Binding world of synchronization to synchronization to

programmatic world of programmatic world of eventsevents Declarative notion inadequate in Declarative notion inadequate in

generalgeneral Programmatic way to begin an Programmatic way to begin an

element; that is, way to go from element; that is, way to go from push events into declarative worldpush events into declarative world I.beginElement()I.beginElement()forces I to beginforces I to begin And by synchrony J beginsAnd by synchrony J begins

Page 24: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

More bindings between More bindings between worldsworlds

Also, SMIL (Microsoft Also, SMIL (Microsoft implementation) offersimplementation) offers

Programmatic inspection of Programmatic inspection of declarative worlddeclarative world I.isActive I.isActive property (is interval property (is interval II

active or does thread active or does thread I I run?)run?) Cast synchronization instants as Cast synchronization instants as

eventsevents onbegin onbegin event fired when thread startsevent fired when thread starts

Page 25: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Part IV. Solution: Part IV. Solution: reaction treesreaction trees

CausalityCausality

Principle of no self-causationPrinciple of no self-causation

Page 26: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Concepts towards an Concepts towards an understandingunderstanding

An An eventevent is identified by a is identified by a namename A A targettarget is a block (that’s where event is a block (that’s where event

appears)appears) bE = beginElement b = begin event ? = accept/receive/read/handle (pattern) event @ target ! = emit/fire/raise/signal

(statement) A handler is of form pattern statement ;}* Co-routine model: one handler executed at a

time

Page 27: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Modeling bE to b and Modeling bE to b and sync arcssync arcs

bE? b@I!I:

J:

Accept beginElement, then signal begin at I

b@J? bE@I!

b@I? bE@J!

When begin is signaled at J, signal beginElement at I

bE? b@J!

co-event

Page 28: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Cyclic dependencies lead to cyclic Cyclic dependencies lead to cyclic behaviorsbehaviors

bE@I!bE@I! Somebody kicks Somebody kicks off Ioff I

b@I!b@I! I, kick-off: bE? b@I!

bE@J!bE@J! I, sync arc: I, sync arc: b? bE@J!

b@J!b@J! J, kick-off: bE? b@J!

bE@I!bE@I! J, sync arc: J, sync arc: b? bE@I!

Why not allow this? Make it Principle of no self-causality

Causality chain ends before a second occurrence

Page 29: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Co-events happen when?Co-events happen when? Augment I and J with Augment I and J with isActiveisActive variable variable

We expect “true” and We expect “true” and “true” to be printed“true” to be printed

bb signaled at signaled at II causes causes event event bb at at I I (execution)(execution) co-event co-event bE@IbE@I at at J J

(observation)(observation) So, co-event must So, co-event must

happen before happen before execution (here)execution (here)

bE? b@I!; I.isActive := true;b@J? bE@I!b? print(J.isActive)

I:

bE? b@J!; J.isActive := true;b@I? bE@J!b? print(I.isActive)

J:

Page 30: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Temporal ordering of co-Temporal ordering of co-eventsevents

Replace “?” with “-” or “+” - means before, + means after

to specify temporal order of co-events relative to their source event

bE? b@I!; I.isActive := true;

b@J-? bE@I!E? print(J.isActive)

Page 31: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

The reaction treeThe reaction tree The root is an empty node (The root is an empty node (document document

nodenode)) It holds the external event or injected It holds the external event or injected

eventevent After an event is handled,After an event is handled,

the signaled events are appended as children the signaled events are appended as children in left-to-right order with timein left-to-right order with time

each child is accompanied by its pre-events, each child is accompanied by its pre-events, which are siblings inserted in front of itwhich are siblings inserted in front of it

When an event has executedWhen an event has executed its post-events are inserted after it as siblings, its post-events are inserted after it as siblings,

andand the next node in pre-order traversal is the next node in pre-order traversal is

selected for handlingselected for handling

Page 32: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Example of tree growthExample of tree growth

bE@I

bE@I isActive.I := true

bE@J isActive.J := true

When event node p is handled, the already handled nodes are the ones before p in document order (preorder)

b@Jb@J

b@Ib@I@J

-

b@J@I

-

Goal: when b@I and b@J are

handled, both isActive

variables are true

Page 33: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

When to be able to When to be able to handle?handle?

A guard of A guard of truetrue always enabled? always enabled? Goes against idea of event consumptionGoes against idea of event consumption

So, tie this predicate to an event!So, tie this predicate to an event! Which one?Which one?

The external eventThe external event NotNot every event! every event!

Page 34: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Reify reaction tree Reify reaction tree

Expose reaction tree to program and programmers to account for all causality

Reify reaction tree

Page 35: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Node matching: Node matching: XSLT/XPathXSLT/XPath

Convention: a pattern is now that of Convention: a pattern is now that of XSLTXSLT

Examples:Examples: / the root, but never / the root, but never

signaled/handledsignaled/handled ee the event named the event named ee /* any external event/* any external event /click/click the external event click the external event click /*///*//clickclick any mouse click that was any mouse click that was

synthesizedsynthesized

Page 36: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Use filtersUse filters

*[not(//g)] an external event, but only if g has not

been signaled so far /*//e[not(ancestor::f) and

not(ancestor::g] any e, when not external and when not

indirectly caused by f or g

Page 37: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

*/f g/f f/g

MP1:MP1:

{I: /*@MP1 - {I: /*@MP1 - f !f !MP1|MP1|

J: g J: g f !MP1 |f !MP1 |

K: f K: f g !MP1 }g !MP1 }

I J K

g/f g/f f/g

I J K

MP2:MP2:

{I: {I: *[not(//g)]@MP2 + f f ||

J: g J: g f!MP2 |f!MP2 |

K: f K: f g!MP2 }g!MP2 }

Mystery Program Solutions

Page 38: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Mystery Program 1Mystery Program 1

Let us inject e at location MP1 This event is never handled

There is no handler! Still f is generated as expected

e@MP1

e@MP1@I

-

MP1:MP1:

{I: /*@MP1-? {I: /*@MP1-? f !f !MP1|MP1|

J: g J: g f !MP1 |f !MP1 |

K: f K: f g !MP1 }g !MP1 }

f@MP1

g@MP1

f@MP1

Page 39: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

f@MP1

Mystery Program 2Mystery Program 2

Let us inject e at location MP2 Post-event generates f

e@MP1

e@MP1@I

+

f@MP1

g@MP1

MP2:MP2:

{I: {I: *[not(//g)]@MP2+? f f ||

J: g J: g f!MP2 |f!MP2 |

K: f K: f g!MP2 }g!MP2 }

Page 40: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Namespace for event Namespace for event valuesvalues

Events, of course, are not atomic, Events, of course, are not atomic, but are treesbut are trees

So reaction tree is tree of trees!So reaction tree is tree of trees! Use namespace notation withUse namespace notation with

ReaxEvtReaxEvt for top element of an eventfor top element of an event ReaxEvtPreReaxEvtPre top element of pre-event top element of pre-event ReaxEvtPostReaxEvtPost for top element of post- for top element of post-

eventevent

Page 41: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Example of namespace Example of namespace useuse

<ReaxEvt:e loc=’id1’><ReaxEvt:e loc=’id1’>

<ReaxEvtPre:f at=’id3’ loc=’id4’/><ReaxEvtPre:f at=’id3’ loc=’id4’/>

<name>Anna</name><name>Anna</name>

<ReaxEvtPre:f><ReaxEvtPre:f>

<ReaxEvt:f loc=’id3’><ReaxEvt:f loc=’id3’>

<name>Anna</name><name>Anna</name>

<ReaxEvt:f><ReaxEvt:f>

<person gender=’male’><person gender=’male’>

<name>Bob</name><name>Bob</name>

</person></person>

<ReaxEvt:e/><ReaxEvt:e/>

Page 42: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

XML for locations as wellXML for locations as well

Broadcast for event signalsBroadcast for event signals Observe several places for co-eventsObserve several places for co-events So, program syntax itself should be So, program syntax itself should be

XMLXML Then, we can writeThen, we can write

//* //* any locationany location id(helperProc)//* id(helperProc)//* any location within any location within

helperProchelperProc .. .. for the block above for the block above

Page 43: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Modeling exceptionsModeling exceptions

try SS{… throw {… throw e; … throw ; … throw f; …} …}

catch {catch {e Se Se ; ; f Sf Sf }}becomesbecomes init@I!;!;

I:{init:{init SS{… e!; break; break; … … f!; break; break; …}

e SeSe; exit;

f SfSf; exit; }}

Page 44: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Program analysisProgram analysis

Good opportunities becauseGood opportunities because ReaX is simple (relatively speaking) and ReaX is simple (relatively speaking) and

deterministicdeterministic Trees are at the foundationTrees are at the foundation Analogy to XSLTAnalogy to XSLT

For exampleFor example While loop should be annotated with While loop should be annotated with

variants to insure terminationvariants to insure termination Or, simple analysis will reveal Or, simple analysis will reveal

termination of must while loopstermination of must while loops

Page 45: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Related workRelated work

EsterelEsterel UML Statecharts UML Statecharts

As in our approach, one event is presented at As in our approach, one event is presented at a time. If several transitions are enabled, then a time. If several transitions are enabled, then either they are in different threads or the either they are in different threads or the deepest one takes precedencedeepest one takes precedence

By default all events are broadcastBy default all events are broadcast [Varro 2002] A Formal Semantics of UML [Varro 2002] A Formal Semantics of UML

Statecharts by Model TransitionsStatecharts by Model Transitions Uses a hierarchy of queuesUses a hierarchy of queues

Page 46: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Part VI. Tying it Part VI. Tying it togethertogether

Page 47: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

UI issues and undo & seekUI issues and undo & seek

In principle, we can now with confidence In principle, we can now with confidence explain a SMIL-like languageexplain a SMIL-like language With declarative synchronization and With declarative synchronization and

temporal featurestemporal features Based on one event model comprisesBased on one event model comprises

Reified causalityReified causality Principle of no self-causationPrinciple of no self-causation

And, we can augment this notation with And, we can augment this notation with undo and seek as built-in mechanismsundo and seek as built-in mechanisms

Page 48: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Time seek and undoTime seek and undo

Add to ReaXAdd to ReaX Notion of moduleNotion of module Checkpoints to identify unit tasks that Checkpoints to identify unit tasks that

are to be undoneare to be undone Ways of categorizing events asWays of categorizing events as

UndoableUndoable With or without specific undo eventWith or without specific undo event Various roll-back characteristics of modulesVarious roll-back characteristics of modules

Not undoableNot undoable Notion of real time and module that Notion of real time and module that

allow time seekallow time seek

Page 49: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

In conclusionIn conclusion Causality explained by Causality explained by reaction treesreaction trees, not queues, , not queues,

that can be exposed in program, debugger, etcthat can be exposed in program, debugger, etc All (?) the engineering advantages of Esterel and All (?) the engineering advantages of Esterel and

Statecharts in an operational framework that Statecharts in an operational framework that explains synchronization, broadcast, and time explains synchronization, broadcast, and time arcsarcs

XML to the rescue twiceXML to the rescue twice for pattern matching in reaction treesfor pattern matching in reaction trees for explaining broadcast, event propagationfor explaining broadcast, event propagation

Consequently, ReaX can be extended with time Consequently, ReaX can be extended with time seek, undo, and redoseek, undo, and redo

Result: better user interfaces and better modeling Result: better user interfaces and better modeling languageslanguages

Page 50: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Appendix. ReaX Appendix. ReaX overviewoverview

SyntaxSyntax

SemanticsSemantics

Page 51: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

ReaX expressionsReaX expressions

LocELocE ::=::= XPath expression with XPath expression with context node = node of innermost block context node = node of innermost block

containing expressioncontaining expression EvtEEvtE ::=::= XPath expression with XPath expression with

context node = current event being context node = current event being handledhandled

EE ::=::= EvtEEvtE or XSLT-like tree fragment or XSLT-like tree fragment In XPath: In XPath: event()event()refers to current refers to current

eventevent

Page 52: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

ReaX syntax: IReaX syntax: I StmtStmt::=::= while while EvtEEvtE do { do {StmtsStmts}};; ||

if if EvtEEvtE then { then {StmtsStmts} else } else {{StmtsStmts}};;|| varvar :=:= EE; ; ||

EvtNnm@LocEEvtNnm@LocE ! ! (( < < || > > )); ; ||

break; break; ||

exit; exit; ||

HandlrsHandlrs ||

DeclsDecls StmtsStmts::= ::= StmtStmt**

Exit handler

Exit handlers block

Page 53: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

ReaX syntax: IIReaX syntax: II

DeclsDecls ::=::=

{{((var var VarNvmVarNvm ((:= := EE)?)?;;)*)* StmtsStmts}};;

HandlrsHandlrs ::=::=

{{((EvtNmeEvtNme{{[[EvtEEvtE]]}?}? ||

EvtNme@locE EvtNme@locE {{[[EvtEEvtE]]}?{}?{+ + || - -}?}?))

StmtsStmts)*)*}};;

Page 54: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Run-time modelRun-time model

The run status of a thread is The run status of a thread is nonenone, , waitingwaiting, or , or executingexecuting

When status is waiting or executing, When status is waiting or executing, the thread has a program counterthe thread has a program counter

Between reactions all thread statuses Between reactions all thread statuses are none or waitingare none or waiting

Co-routine model: at most one thread Co-routine model: at most one thread is executing at a timeis executing at a time

If a thread is waiting, then its parent If a thread is waiting, then its parent is, toois, too

Page 55: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

The reactionThe reaction Add(Add(e@le@l at at pp):):

Append pre-events of Append pre-events of e@le@l as children of as children of pp Append Append e@le@l

Process(Process(e@le@l at at pp):): If there is a handler If there is a handler ee[[EvtEEvtE]] StmtsStmts at at ll with thread with thread

of of l l waiting at waiting at l l and with and with EvtEEvtE evaluating to evaluating to truetrue, then , then executeexecute Stmts Stmts as a subthread as a subthread

For each For each e@Le@L that is signaledthat is signaled, , Add(Add(e@Le@L at at pp)) Append post-events as siblings of Append post-events as siblings of pp Process(event at next node) [in pre-order traversal]Process(event at next node) [in pre-order traversal]

Pre-events and post-events are processed Pre-events and post-events are processed similarlysimilarly No additional pre- and post-events addedNo additional pre- and post-events added

Page 56: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Statement executionStatement execution Simple do the obvious, advancing the PC of the Simple do the obvious, advancing the PC of the

thread, until a handlers block in encountered; thread, until a handlers block in encountered; then stopthen stop This is where the thread waitsThis is where the thread waits A subthread’s run status is (still) noneA subthread’s run status is (still) none

If If breakbreak or end of thread is encountered, then or end of thread is encountered, then stopstop The thread’s run status becomes none (parent thread The thread’s run status becomes none (parent thread

still waiting)still waiting) If If exitexit is encountered is encountered

Make run status none for this thread and all siblingsMake run status none for this thread and all siblings Continue execution of parent thread after parent Continue execution of parent thread after parent

blockblock

Page 57: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Add co-events for Add co-events for e@le@l

For eachFor eache e @ @ locE locE [[EvtEEvtE]-]-StmtsStmts

at location at location lltt such thatsuch that locE locE evaluated at evaluated at lltt contains contains ll EvtE EvtE evaluates to trueevaluates to true

add add e e @ @ l @l @ lltt to set of co-eventsto set of co-events Order co-events according to Order co-events according to

document order on document order on lltt (or use (or use static priority declarations)static priority declarations)

Page 58: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Termination propertyTermination property

If every while loop terminates, then If every while loop terminates, then the ReaX semantics guarantee the ReaX semantics guarantee termination of a reactiontermination of a reaction ProofProof

König’s Lemma states that a finitely König’s Lemma states that a finitely branching tree is infinite only if it has an branching tree is infinite only if it has an infinite pathinfinite path

Termination of while loops imply that the Termination of while loops imply that the reaction tree is finitely branchingreaction tree is finitely branching

Principle of no self-causation implies no Principle of no self-causation implies no infinite pathsinfinite paths

Page 59: ReaX -- a reactive programming language for multimodal applications Nils Klarlund, AT&T Labs With contributions from Anders Møller, Jennifer Beckham, Giuseppe

Exceptions through Exceptions through transformations,…transformations,…

try SS{… throw {… throw e; … throw ; … throw f; …} …}

catch {catch {e Se Se ; ; f Sf Sf }}becomesbecomes init@I!;!;

I:{init:{init SS{… e!; break; break; … … f!; break; break; …}

e SeSe; exit;

f SfSf; exit; }}