XISL language

XISL= eXtensible Interaction Sheet Language

or

XISL=eXtensible Interaction Scenario Language

2

Characteristics

• XML-based markup language for web-based MMI• Grammar is defined by DTD• Interaction between a XML/HTML element and a user

User operation (click, speech input,…) Action (screen update, speech output,…)

• Separates XML/HTML content from XML interaction• Allows user initiative interaction, system initiative

interaction and mixed initiative interaction• Enables the use of various type of terminals: mobile

phone, PC, digital TV, PDA,…• Uses I/O cooperativley: parallel I/O, sequential I/O,

alternative I• Flexibility to expand I/O modalities

3

Goals of XISL

• Provide a common language for web-based multimodal interaction

• Satisfy 3 conditions:1. Control dialog flow/transition: employed from

VoiceXML

2. Synchronize input/output modalities:employed from SMIL

3. Modality-extensibility:offered by XISL

4

Exemple

5

Dialog levels

1) Exchange

2) Dialog

3) Document

4) Application

5) Session

6

Dialog levels

1) Exchange: primary unit of interaction between a user and a system

• non-prompting description (user operation, system action)• prompting description (a prompt, user operation,system action)

2) Dialog:• composed of exchanges• user is always execution a dialog

3) Document: XISL file composed of some dialogs

4) Application: • set of XISL documents• contains 0..N leaf document, 1 application root document

5) Session• begins when user connects to the XISL execution system• ends when requested by the user or the XISL execution system

7

Architecture of XISL execution system

8

XISL execution system

1. Front-end module: • user interface terminal (microphone, speaker, screen,…)• Depends of application and terminal

2. Dialog manager module: • interprets XISL documents• manages dialog flow• controls I and O• Independent of application and terminal

3. Document server module (general web server):• holds XISL, XML/HTML documents • other documents (XISL stylesheet, speech grammar files,…)

9

A subset of XISL elements

10

Tree structure of XISL elements

11

Exemple

12

Dialog manager sub-modules

1. XISL interpreter

2. Input integrator

3. Action module

4. Document manager

13

Outline of XISL execution system

• XISL interpreter recives the XISL document from Document Manager devides it into

<operation>s : sends to Input Integrator <prompt>s: sends to Action Module <action>s: sends to Action Module

• Input Integrator extracts <input>s from <operation>s and sends them to front-end module parses the <operation>s to prepare for input integration matches the parsed <operation>s with the multimodal inputs sent from

front-end module informs the result to the action module

• Action module executes action corresponding to the <operation>s from Input Integrator extracts <output>s from <prompt>s and <action>s 2 types of actions:

terminal-dependent: outputs to a user are sent to the front-end Terminal-independent: executed inside the action module (<switch>,<goto>)

14

Separation of content from XISL document (1)

• XISL document contains only interaction scenario

• Content is held separately into XML/HTML files

• Advantagessystem developers can reuse XML/HTML

documents and XISL documentsimproves readabability

15

Separation of content from XISL document (2)

• <output> : output to the userAttributes:

type: output modality (browser, speech, audio,…) event: output event (navigate, close, play,...)

• <input> : accepts the events raised on XML/HTML documentsAttributes:

type: input modality (speech, pointing, key,…) event: input event (recognize, click, press,...) match : link to XML/HTML element return: return values

• for flexibility strict attribute values and contents of above tags are not specified by XISL (it should be done by terminal developers)

16

Modalities to control interactions in XISL documents:

1. comb attribute in <dialog> tag• Comb=par -> all exchange are executed in parallel• Comb=alt -> one of the exchange is executed alternatively• Comb=seq -> all exchange are executed in document

order

2. <par_exchange>

<seq_exchange>

<alt_exchange>

the exchange elements bound by these tags have the same semnification as above

17

Exemple

18

Types of front-end

1. PC terminal• On Line Shopping application• User input: touch-screen display, keyboard,

microphone• Outputs: touch-screen(anthropomorphic

agent), speeaker

19

Input specification for PC terminal

20

Output specification for PC terminal

21

Exemple

22

2. Mobile phone terminal

• On Line Shopping application

• Sequential scenario due to terminal’s limits

• Inputs: user’s speech, DTMF inputs

• Output:speech

23

Input specification for phone terminal

24

Output specification for phone terminal

25

Exemple

26

3. PDA terminal

• On Line Shopping application

• Inputs: cursor key, tap-pen (software keyboard), speech

• Output: screen, sound device

27

Input specification for PDA terminal

28

Output specification for PDA terminal

29

Exemple

30

Comparation with other MMI approaches (1)

1. SALT: offers tags for describing speech recognition and TTS included in HTML documents

Advantages of XISL: • better suited to describe complex MMI using sequential,

parallel or alternative combination• interaction is explicitly separated from XML/HTML

contents

31

Comparation with other MMI approaches (2)

2. XHTML+Voice: • add VoiceXML to XHTML pages • provides voice interaction as additional modalities

to web pages• VoiceXML comes with interaction flow: control

more complex MMI then SALT

Advantage of XISL:• more flexibility to add modalities

32

Galatea Interaction Builder – prototyping tool

• Runs on PC• Handels input modalities of speech, mouse,

keyboard• Handels output modalities of speech(TTS), facial

expresion, window display• Implementation of input and output can be made

without the knowleadge of XISL• GUI design for domain-specific prototyping

(airline ticket reservation, secretary service,…)

33

Interaction Builder

34

Conclusions

• XML-based markup language for web-based MMI scenarios

• XISL is separately described from XML/HTML contents (reusability)

• enables the use of various type of terminals

• flexibility to describe user’s input and system’s output