A Demonstration of Continuous Interaction with Elckerlyc

A Demonstration of Continuous Interaction with Elckerlyc

Herwin van Welbergen, Dennis ReidsmaJob Zwiers

Temporal coordination: from turn based interaction to continuous interaction

Current interaction with virtual humans• Using speech, facial

expression and gestures• Turn-based, ‘push-to-talk’

interaction paradigm

Our goals

• Allow continuous interaction

• All partners express themselves continuously

• In parallel• Needs prediction

and anticipation

Continuous interaction

Temporal coordination in human behavior- Timing of backchannel- Allowing room for (intrusive) backchannel from

partner- Overlap during non-intrusive backchannel- …

In this talk- Focus on specifying continuous interaction- Using minor timing and parameter value adjustments

in ongoing behavior

Elckerlyc

Temporal coordination in interaction with virtual humans• Elckerlyc: open source platform for behavior generation

designed for such temporally coordinated behavior (… and other things)

• Executes behavior specified in the behavior markup language (BML)

• Implements SAIBA Framework

Why BML/SAIBA?

> 10 years of research on ECAs• Increasingly sophisticated models

– Directed to different aspects of behavior

• Building a state-of-the-art ECA entails re-implementing all these models

– Allow reuse of models

SAIBA framework

Intent Planning

BehaviorPlanning

Behavior Realization(Smartbody, Greta,

ACE, Elckerlyc)

FML BML

FeedbackFeedback

• Three-level setup of many existing ECAs• Separation between stages• Each stage is a black box• Clear-cut definition of interfaces between stages

BML Example

Bml Request

<bml id=”bml1"><gaze type=”AT” id="gaze1" target="AUDIENCE"/>

<speech id="speech1"> <text>Welcome ladies and gentlemen!

</text></speech>

</bml>

Behaviors

BML Example

BML Behaviors

Gesture Head Gaze Speech Locomotion Posture Facial expression

eyeseyes

torsotorso

legslegs

Coordin

ation

Phases and sync-points

Synchronizing behaviors

<bml><gaze id="gaze1" target="AUDIENCE"/> <speech start=”gaze1:ready” id="speech1"> <text>Welcome ladies and gentlemen!

</text></speech>

</bml>

Example: gaze shift to moving target

<bml id="bml1"><speech id="bluespeech" start="gaze1:ready">

<text>I'm staring at the blue cube.</text></speech>

<speech id="greenspeech" start="gaze2:start"><text>Look! A green sphere.</text>

</speech>

<speech id="bluespeech2" start="gaze2:end"><text>Looking at the blue cube again.</text>

</speech>

<gaze id="gaze1" start="1" ready="2" type="AT" modality="NECK" target="bluebox"/>

<gaze id="gaze2" start="4" ready="5" relax="6" end="7" type="AT" modality="NECK" dynamic="true" target="greensphere"/>

</bml>

Continuous interaction using SAIBA

Intent Planner

BehaviorPlanner

FML BML Anticipator

SensorUser

VirtualHuman

movement & speech

Sensor

Sensor+interpretation

Realizer

Continuous Interaction Scenarios

Intent Planner

BehaviorPlanner

[increase tempo]

Sensors +Interpretation

exercise istoo easy

increasedifficulty

BehaviorRealizer

exercisedescription

TempoAnticipator

predictedtempo

Sensor

User

movement & speech

VirtualTrainer

Virtual Trainer

Continuous Interaction Scenarios

Virtual Trainer<bml id="bml1">

<bmlt:procanimation id="exercise1" name="squat"/><constraint id="c1">

<synchronize ref="exercise1:beat1"><sync ref="excerciseAnticipator:beat1-0.5"/></synchronize><synchronize ref="exercise1:beat2"><sync ref="excerciseAnticipator:beat2-0.5"/></synchronize>...

</constraint></bml>

Demo

Continuous Interaction Scenarios

Intent Planner

BehaviorPlanner

[wait for turn then speak]

take turnand perform

communicativeact

(politeness,

arousal, friendliness)

BehaviorRealizer

speech withstart sync linked to

anticipator

SpeechStopAnticipator

predictedstop time

SensorUserVirtualHuman

movement & speech

<bml id="bml1"><speech id="speech1" start="speechStopAnticipator:stop+x">

<text>Bla bla</text></speech>

</bml>

Taking the turn (as soon as possible)

Demo

• Uses pre-planning• Easily allows incremental updates of speech end

prediction

Continuous Interaction Scenarios

Intent Planner

BehaviorPlanner[keep turn]

keep turn(politeness,

arousal,

friendliness)

BehaviorRealizer

increasevolume of speech1

UserVirtualHuman

movement & speech

Sensor +Interpretation

user wants turn

Keeping the turn

Demo

Conclusion

• Elckerlyc provides mechanisms for the specification and execution of behaviors synchronized to predicted human behaviour

• Flexible pre-planning• Incremental update of user behavior prediction

Open issues

Specifying parameter changes <bmlt:setparameter id=“pchange1" start="10"

end="speech1:end” target="speech1" parameter="volume" curve="linear" startvalue="25" endvalue="100"/>

• Send a parameter specification message– Allow easy experimentation with parameter change– Synchronization to behaviors– Parameter value curve (e.g. linear, ease-in ease-out, …)– Using BML?

• Conceptually doesn’t match very well with other behaviors• Should not modify synchronization constraints• Requires specialized mechanisms to refer to behaviors in

previous blocks

Open Issues

Creating anticipators• Currently we use placeholders• Real-time music tempo tracking and prediction (Pieter

Bos et al)• Predicting listener backchannel relevant moments

(Louis-Philippe Morency, Iwan de Kok, ...)• Fitness excercise tempo tracking and prediction (Eike

Dehling)• Real-time turn-giving prediction (Jonsdottir et al 2008)

Enterface 2010

Backchannel aware guide• Predicts and anticipates completions, acknowledgement and

continuers of a user• Response handling strategies

– Modulated by politeness– Ignore (and speak louder)– Acknowledge response– Wait for acknowledgement before continuing– Change route-giving plan– Etc

• Requires – Adaption of timing of ongoing behavior– Adaptation of parameterization of ongoing behavior– Graceful interruption

Thank you for your attention

Acknowledgments Stefan Kopp’s

group Bart van Straalen Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers SmartBody

developers

More information/demo• http://hmi.ewi.utwente.nl/showcase/Elckerlyc• Contact Dennis, Herwin:

{dennisr,welberge}@ewi.utwente.nl

END

The End.

Enterface 2010

Route recap: dealing with acknowledgements A1: So, we went to the square with the obelisk… A2: We crossed the river using the suspension bridge… A3: Crossed the square with the arch monument… A4: And took the first street to the right.• Do not deal with it: simply elicit the feedback, pause a bit, then continue

speaking, whatever the subject does• Deal with it: elicit feedback, pause a bit, if feedback is there, wait till it is

finished before acknowledging (smile, nod?) and continuing• Deal with it, the “continuous interaction way”: elicit feedback, wait, but

if feedback comes and it is non-intrusive, immediately acknowledge the feedback (smile, nod?) and continue speaking even while the user is still finishing the feedback

• Evaluate responsiveness, attentiveness, politeness, etc

Enterface 2010

Eliciting of completion A: Do you know the way to the square with the… euhm… [makes

thinking gestures]… euhm… that thing [makes iconic gesture for the arch monument]… euhm… [looks at the listener inviting a response]… euhm… how do you call it?... etc.

[correct completion:] U: arch monument? A: Yes, the arch monument. [Incorrect completion:] U: Obelisk? A: No, I mean… euhm... the square with the arch monument.

Some boring technical facts

• Written in Java• ≈80,000 lines of code (Greta: 45,000; SmartBody: 35,000)

– including ≈ 30,000 lines code for Collada loading and custom graphics

– ≈ 200 (J)Unit test cases, ≈ 1100 tests

• Takes advantage of multiple processors/cores– Physical simulation (OdeJava)– Rendering (Jogl)– Text To Speech (MS Speech API, Mary TTS, Text-based)

Licensing

• Pre-release under the GPL v3 license– No source repository yet, contact Herwin for code– ‘Official’ versioned release soon

• Inverse dynamics code is available under MIT license– http://www.herwinvanwelbergen.nl/index.php?selected=phd

Elckerlyc design concerns

Continuous interaction• Allow fast and flexible planning and re-planning• Allow specification of synchronization to predictions

Flexibility• Easily exchange internal modules, add new modalities, …

Motion generation• Not so much focus on designing (procedural) animation• But on combining motion from all kinds of paradigms

– In sequence and parallel– Physical simulation, mocap/keyframe animation, procedural

motion, biomechanical models, …– New: mixed dynamics

Elckerlyc Architecture

Highlights• Scheduling and playing

stages• Scheduler can easily be

exchanged• Scheduler resolves

timing of a sync point per constraint

• Sync points can slightly adjusted

Scheduler

SpeechPlanner ... Planner

BehaviorBinding

BML stream

Parser

behaviors,(time) constraints

animation plan

peg board

AnimationPlanner

speech plan ... plan

joint rotations

Animation Engine Speech Engine

behavior, syncs to resolve

resolved syncs

behavior, syncs to resolve

resolved syncs

Anticipator Sensor

anticipation change

AnimationPlayer SpeechPlayer

speech

time pegupdate

... Engine

SAIBABehaviorPlanner

feedback

Scheduling

Playing

.. player

add

be

havi

or

add

be

havi

or

set t

ime

peg

SAIBAIntent

Planner

Animation plan

Animation plan• Result of planned

BML• Specification of

motion to be executed by the player

• Contains timed motion units

• Timing linked to time pegs

Organization of motion

Motion Units• (≈LMPs, gestures, controllers, …)• Parameters, linked to BML parameters• Can be executed, typically rotate joints

– At canonical time (0..1)

• Contain phases, key times can be aligned to time pegs– Key times are assigned canonical time values

• Physical, procedural, mocap/keyframe, specialized

Procedural motion units

Procedural motion units• Function of time (0..1) and parameter vector• of end effector position and/or joint rotation

– Custom functions for perlin noise, splines, …– Example:

<EndEffector local="false" target="r_wrist” translation="0;(1-t)*starty+t*endy;0.3"/>

• Key frames• Mocap• Can be imported from Greta gestures

Physical motion units

Physical controllers• Input: desired state • Minimize discrepancy between

current and desired state• Output: joint torques that

guides the VH closer to this state

• Can cope with external perturbation

Video from Abe et al.

2007

Physical simulation for gesture

Advantages• Models force transference between body parts• Models momentum effects

– Overshoot– Models pendulum like motion easily

• Realistic balance• Desired state specification fits some motions well

– Loosely hanging arm (desired state: slightly damp arm motion)– Balance (desired state: projected CoM in support polygon)– Spine (desired state: spine pose)

Physical simulation for gesture

Disadvantages• Precise timing and limb placement is an open issue

– It is unknown if and when a controller achieves a desired limb position

– These aspects are crucial for (speech accompanying) gesture

Mixed dynamics

Mixed dynamics• Use kinematic motion on

body parts that require precise timing/placement

• Use physical simulation the remaining part of the body

• Calculated the forces the kinematic parts apply on the physical part

– Using inverse dynamics

• Allow switching between different kinematic/physical representations

– Depending on the needs of the moment

Comparison with mocap

Executing a Greta gesture

Executing a SmartBody gesture

Physical spine (experimental)

Switching

Transition motion units

• Define a transition animation from physics to kinematics• Or kinematics to kinematics• Flexible start state• Predicted end state• Only define type, joints and start and end time <bmlt:transition id="trans1" class="HermiteSplinePhysicalTransition" start="armhang:end"

end="7"> <bmlt:parameter name="joints" value="l_shoulder,l_elbow,l_wrist"/> </bmlt:transition>

Prediction

Animation predictor• Can predict a joint configuration at a certain time• Uses a copy of the animation plan with predictable timed

motion units• Predictable timed motion units

– Deterministically define pose at each time• Procedural motion unit

– Determinism during certain motion phases• Stroke of gesture• Gaze at static target

Custom kinematic motion units

Adhere to the motion unit interface• Move joints on the basis of time (0..1) and parameter

values (can be changing during motion)• Provide canonical key positions for synchronization• Motion should be smooth (C0 continuous)• Can make use of predicted motion• Can use a flexible start position• Currently:

– Gaze, pointing

• Planned: MURML motion units

Animation player

Animation player• Executes an animation

plan• Timing is linked to time

pegs• Time warping: converts

‘real’ time to canonical time

• Combines physical simulation and kinematic motion automatically

Scheduling

Scheduling• Determine

synchronization constraints between behaviors

• Distribute behavior over planners

• BML => plans• Assign preliminary

times to pegs

Scheduling

• Conflicting constraints• Stretching/skewing/skipping if necessary

Scheduling in Elckerlyc

Using the SmartBody scheduling algorithm• Assumes that the behaviors in the XML are ordered

by time importance• The timing of the 1st behavior is never adapted• The timing of the 2nd behavior is adapted to fit the

time constraints imposed by the 1st behavior• The timing of the 3rd behavior is adapted to fit the

time constraints imposed by the 1st and 2nd behavior• Etc• So, ‘later’ behaviors never change the timing of

earlier behaviors

Scheduling

Scheduling• Modularity: separate parser and uni-modal planners• Each planner

– Can provide timing information on BML behaviors given time constraints• Desired duration, requested timing

– Can construct (motion) units from BML behaviors– Can validate the current plan

• Allows us to easily exchange the scheduler– Use ACE’s chunk based scheduler?

a b c d

a db c

?time constraints,

requestedmotion timing

motion unitpreferred timing

timedmotion unit

Animation Planner

Animation planner• Construct the

animation plan• BML => timed motion

units• Resolves execution

times for unknown Time Pegs for a timed motion unit , given constraints

• Can validate the current animation plan

From BML to timed motion units

Gesture binding• Maps behaviors to motion

units• Handles parameter

assignment in motion units– Maps BML parameter– Or provide defaults

• Allows easy modification of BML realization

• Allows easy addition of new BML behaviors

• Mapping is currently 1 to 1

Gesture binding<gesturebinding><MotionUnitSpec type="head"> <constraints> <constraint name="action" value="ROTATION"/> <constraint name="rotation" value="NOD"/> </constraints> <parametermap> <parameter src="amount" dst="a"/> <parameter src="repeats" dst="r"/> </parametermap> <parameterdefaults> <parameterdefault name="a" value="0.5"/> <parameterdefault name="r" value="1"/> </parameterdefaults> <MotionUnit type="ProcAnimation" file="nod.xml"/></MotionUnitSpec>

<MotionUnitSpec type="head"> <constraints> <constraint name="action" value="ROTATION"/> <constraint name="rotation" value="SHAKE"/> ...</gesturebinding>

BML specification of a head nod<bml id="bml1"> <head id="nod1" repeats=”3" action="ROTATION" rotation="NOD"/></bml>

Procedural motion unit defined in nod.xml<ProcAnimation prefDuration="1.0"> <Rotation target="vc4" rotation="a*0.5*sin(alpha*2*pi*r);0;0"/> <Rotation target="skullbase" rotation="a*0.5*sin(alpha*2*pi*r);0;0"/> <Parameter id="a"/> <Parameter id="r"/> </ProcAnimation>

Replanning

• Moving Time Pegs– Keys of timed motion units are linked to time pegs => retiming is

handled automatically– Planner can be queried for the validity of the motion plan after

moving time pegs

• Access to motion units, speech units, ability to remove or adapt them

Anticipation

Specify synchronization to predicted times<bml id="bml1" xmlns:bmlt="http://hmi.ewi.utwente.nl/bmlt">

<bmlt:procanimation id="conduct1" name="3-beat"/>...<bmlt:procanimation id="conduct8" name="3-beat"/><bmlt:controller id="balance1" class="BalanceController"/><constraint id="c1">

<synchronize ref="conduct1:start"> <sync ref="metronome1:tick1"/> </synchronize>

<synchronize ref="conduct1:beat2"> <sync ref="metronome1:tick2"/> </synchronize> ...

<synchronize ref="conduct8:end"> <sync ref="metronome1:tick25"/>

</synchronize></constraint>

</bml>

Demo with anticipation

Other uses for Time Pegs

• Chunk based/last minute scheduling– MORE???

• Communication between Engines– Communicate time drift of TTS in Speech Engine to Face Engine

Further work

Implementing Anticipators for turn-taking• End of turn, interruption point, user wants turn, ..

Combining Timed Motion Units How to best implement interruption?

• Directly hook into Elckerlyc?• Define some BML strategies?

– Core BML can only replace, append and insert behaviors– No synchronization between different BML requests– Create strategy for replace, tighter merge, change?

Further work

Specifying parameter value changes• Using BML??

<bml id="bml2" scheduling="tight-merge"/><bmlt:setparameter id="reparam1" start="10" end="speech1:end"target="speech1" parameter="volume" curve="linear"

startvalue="25" endvalue="100"/></bml>

Collaboration

Exchanging Motion Units, other modules? Anticipation/prediction

• Predicting listener backchannels with USC/ICT– (Louis-Philippe Morency, Iwan de Kok, ...)

Rendering/output generation• Bonebus interface for Elckerlyc?

Some boring technical facts

• Written in Java• ≈80,000 lines of code (Greta: 45,000; SmartBody: 35,000)

– including ≈ 30,000 lines code for Collada loading and custom graphics

– ≈ 200 (J)Unit test cases, ≈ 1100 tests

• Takes advantage of multiple processors/cores– Physical simulation (OdeJava)– Rendering (Jogl)– Text To Speech (MS Speech API)

Licensing

• Pre-release under the GPL v3 license– No source repository yet, contact Herwin for code– ‘Official’ versioned release soon

• Inverse dynamics code is available under MIT license– http://www.herwinvanwelbergen.nl/index.php?selected=phd

Thank you for your attention

Acknowledgments Job Zwiers Ronald Paul Mark ter Maat Zsofia Ruttkay Greta developers SmartBody

developers

More information/demo• http://hmi.ewi.utwente.nl/showcase/Elckerlyc• Contact Dennis, Herwin:

{dennisr,welberge}@ewi.utwente.nl

END

The End.

RESTJES

De volgende dingen horen typisch in de afdeling Easter Eggs:

• Scheduling• Hoe maak je je eigen controllers• Hoe maak je je eigen engine• Hoe neem je een nieuwe avatar op in het systeem• Hoe bouw je procedurele animaties• Hoe maak je mocap animaties

(daar is ergens een oude herwin tutorial van, no?• …

FUTURE WORK

• OPEN: Multiple BML blocks, BML streams, …? Hoe gaan we dat oplossen? Wat waren de mogelijkheden ook weer?

• OPEN: meer, zie einde deze presentatie

BML Behaviors

More behaviors…

Open Issues

Handling multiple BML Requests

• How to handle both long sequences and graceful interruption of behavior?

Granularity of BML Requests is left open• Full monologue• Or short spurts of behavior

– Single gaze shift– Speech clause + related gestures

Handling multiple BML Requests

Handling successive blocks• In development• Required for BML core compliance:

– Replace– Append (but what about persistent behaviors?)– Merge (default) merges the new BML block into the current BML block(s). Behaviors in this request start immediately, provided they do not

conflict with on-going behaviors (and behaviors scheduled in the future) of prior BML requests (for the given character?). If such a potential conflict is detected with required behaviors, the realizer must not perform any behaviors of the new request and notify via an error / exception. If only non-required behaviors potentially conflict with prior behaviors, the realizer may choose not to perform these behaviors, as long as warning notifying of the omission are sent. If this results in the omission of all behaviors, an error/exception must be sent instead of a warning. Alternatively, the schedule of these non-required

behaviors may be adjusted to avoid conflicts, as long as required synchronization constraints are not violated. – tight-merge: (Elckerlyc specific) Like merge, but the new BML block may refer to

behaviors in the current block(s). – mergeandreplace (Elckerlyc specific): like tight-merge, but replaces all behaviors from

the current block(s) with those in the new block if their ids match. Optionally allows the (fluent) replacement of currently running behaviors.

– removeandreplace (Elckerlyc specific): removes all current behaviors except those with ids matching behaviors in the new block. Then replaces those behaviors with the new behaviors and adds the new behaviors. This is different from replace in that it allows replacing behaviors that are currently running and retains persistent behavior.

– TODO: wat is nodig voor enterface?

Stuff from old presentation

Animation Player

Kinematic Motion

InverseDynamics

PhysicalControllers

activecontrollers,

desired states

joint rotationsjoint velocities

joint accelerations

connector torques

connectorvelocity,

connector acceleration

joint torques

joint rotations

joint rotationsVirtual Human

current state

Animation Player

AnimationPlanPlayer

Physical/kinematicbody selection

active controllers

Physical body

ForwardDynamics

Physical Simulation

animation plan

Open issues

Conflict resolution• What if multiple behaviors want to use the right hand?• Resource allocation in the planner

– Use left hand if right hand is in use

• Combination of animations– Store in motion units how they combine with other motion units

• Nod motion should be additively blended• Manipulative hand gesture should constrain hand position

– In planner => hierarchical motion plans (as in SmartBody)– In realizer => final combination phase (as in EMBR)– Use existing computer animation techniques for the combination

(see Welbergen et al 2009 for an overview)

Open issues

Easily obtaining more motion units• Because authoring motion units is a lot of work• With other research groups?

– Cooperation?– Automatic conversion of Greta gestures to our proc animation

End of stuff from old presentation

Feedback from the realizer

To the planner• BML performance start• BML performance stop• Sync-Point progress• Exception• Warning

Intent Planning

BehaviorPlanning

BehaviorRealization

FML BML

FeedbackFeedback

Advanced synchronization

<constraint> Allows synchronization of sync points

• Before another sync-point• After another sync-point• At another sync-point

Advanced synchronization

<constraint id="before_example"> <before ref="speech_1:start"> <sync

ref="gaze_1:stroke"/> <sync

ref=“nod_1:stroke"/> </before>

</constraint> • Before/after not implemented in any realizer• Elckerlyc supports synchronize

<constraint id=“sync_example"> <synchronize ref="speech_1:start">

<sync ref="gaze_1:stroke"/>

<sync ref=“nod_1:stroke"/>

</synchronize>

</constraint>

Gesture behavior

• Coordinated movement with arms and hands

Gesture types: Point

Gesture types: beats

Gesture types: conduit

Gesture types: lexicalized

<speech>

<bml><speech id="s1"> <text>

This is a complete core level BML <sync id="tm1"/> speech description.</text></speech><gesture id="g1" stroke="s1:tm1" type="BEAT">

</bml>

Persistent behaviors

No end time• Posture• Gaze• Point (?)• ‘New’ behaviors overwrite old behaviors• TODO, moet dit hier? In Elckerlyc: implemented with

replacement groups

BML Design

• Describes occurrence of behaviors• Relative timing and synchronization of behaviors• Form of behaviors• Realizer-independent• But allows extensions for realizer-dependent behavior

BML Design: Realizer independence

• Cannot rely on skeleton joints, speech synthesis systems, available animations, ...

• Refers to body-parts, lexicalized locations, common verbs– speech, face, gesture, center, left

• Allows specification of nuanced/detailed behavior through extensions

BROADER VIEW

• wat zijn VH toepassingen en waar zijn die goed voor? • hoe zit de methodische cyclus in elkaar van

observeren, modelleren en nabootsen van menselijk gedrag?

• Wat is continuous interaction, waar is dat goed voor, waarom moeten VHs dat ook kunnen?

ELCKERLYC

• SAIBA Global architecture SAIBA (met name: de feedback loops met warnings enzo)

• BMLWat zijn de belangrijkste issues bij behavior generation die door BML worden aangepakt? Wat is BML en hoe werkt het? Synchronisatie van behaviors, parallel specificeren van allerlei behaviors, abstractie (!), WAT ZIJN BEHAVIORS (voorbeelden)?

• Design concerns voor Elckerlyc• Globale architectuur Elckerlyc

met peg board, planners, players, etc. Feedback loops…• Feature: organisation of motion• Bridge: The Animation Player (& Animation Plan)• Feature: Physics and mixed dynamics

switching & mixed dynamics & physics (waarom uberhaupt?)• Feature: persistent behaviors & replacement groups• Feature: prediction of motion (voor transitiecontrollers enzo)• Extension: Hoe maak je je eigen motion units

(voor bielefeld relevant). Uitleg procmotionunits, uitleg mocap (kort, = pmu), voorbeeld greta en smartbody, vraag “en Max?”

• Feature: gesture binding en planning

ELCKERLYC IN USE , CONCLUDING REMARKS

• hoe wordt Elckerlyc gebruikt voor een VH toepassing? Wat is de relatie tov intent&behavior planning? Die vallen buiten Elckerlyc. Als je behavior gepland hebt: (1) maak elckerlyc, stuur BML (2) maak Elckerlyc, maak MUs en pas die aan en voeg die toe aan player

• Wat kan Elckerlyc dat andere realizers niet kunnen? (of: wat zijn de tekortkomingen van andere realizers die je aan begin presentatie noemt als ‘het doel van Elckerlyc’?) Refereer terug naar de design concerns

• Hoe blijkt die continuous interaction nu daadwerkelijk in Elckerlyc te zitten? Hebben we al goede voorbeelden? Wanneer wel?

• Boring technical facts

Switching

<bml id="bml1">

<!-– Some conducting gestures --> <bmlt:procanimation id="conduct1" name="3-beat"/> <bmlt:procanimation id="conduct2" name="3-beat" start="conduct1:end"/> <bmlt:procanimation id="conduct3" name="3-beat" start="conduct2:end"/> <bmlt:procanimation id="conduct4" name="3-beat" start="conduct3:end"/> <bmlt:procanimation id="conduct5" name="3-beat" start="conduct4:end"/>

<!–- Physical controllers --> <bmlt:controller id="balance1" class="BalanceController"/> <bmlt:controller id="armhang" class="CompoundController" name="leftarmhang" start="3.5"

end="6.5"/>

<!–- Transtion --> <bmlt:transition id="trans1" class="HermiteSplinePhysicalTransition" start="armhang:end"

end="7"> <bmlt:parameter name="joints" value="l_shoulder,l_elbow,l_wrist"/> </bmlt:transition>

</bml>

Continuous interaction

Continuous interaction• Observation and interpretation• Cognition and deliberation• Multi-modal Expression

Occurring in parallel

Expression timed tightly, from moment to moment, to perceived movements and behavior of other

E.g.: Conduct an orchestra; Count along with exercising user; Talk about co-occurring physical task performed by user (device maintenance, tutoring) or about fixed media (presenting info about multi media content in IR or QA)

Feedback loop.