User Interface User Interface AgentsAgents

Roope Raisamo (rr@cs.uta.fi)Department of Computer and Information Sciences

University of Tampere

http://www.cs.uta.fi/~rr/

User Interface AgentsUser Interface Agents

A user interface agent guides and helps the user– Many user interface agents observe the

activities of the user and suggest better ways for carrying out the same operations

– They can also automate a series of operations based on observing the users

Many user interface agents are based on the principles of programming by example (PBE)

Two examples of Two examples of user interface agents:user interface agents:

EagerEagerLetiziaLetizia

Eager – automated macro Eager – automated macro generatorgenerator

Allen Cypher, 1991http://www.acypher.com/Eager/ Observes the activities of the user and tries

to detect repeating sequences of actions. When such a sequence is detected, offers a possibility to automate that task.

like an automated macro generator this kind of functionality is still not a part of

common applications, even if it could be.

EagerEager

Eager observes repeating sequences of actions

When Eager finds one, it jumps on the screen and suggests the next phase

EagerEager

When all the phases suggested by Eager have been shown and accepted, the user can give Eager the permission to carry out the automated task.

Letizia – a browser companion Letizia – a browser companion agentagent

Letizia observes the user and tries to preload interesting web pages at the same time as the user browses through the web

LetiziaLetizia

LetiziaLetizia

Traditional browsing leads the user into doing a depth first search of the Web

Letizia conducts a concurrent breadth-first search rooted from the user's current position

The appearence of agentsThe appearence of agents

The appearance of an agentThe appearance of an agent

The appearance of an agent is a very important feature when a user tries to find out what some agent can do.

It is a bad mistake to use such an appearance that the user thinks an agent to be more intelligent than it really is.

The appearance must not be disturbing.

Computer-generated talking Computer-generated talking head head

one of the most demanding forms of agent presentation

a human head suggests the agent to be rather intelligent

a talking head probably is the most natural way to present an agent in a conversational user interface.

Drawn or animated charactersDrawn or animated characters

the apperance has a great effect on the expectations of the user– a paper clip vs. a dog vs. Merlin the Sorceror

Continuously animated, slowly changing or static presentation

Textual presentationTextual presentation

Textual feedback of the actions of an agent– Concerning textual user interfaces we usually

should avoid textual input if it is not a part of the main task that the agent is observing.

Chatterbots– e.g., Julia that is a user in a MUD world. It can

also answer to questions concerning this world.http://lcs.www.media.mit.edu/people/foner/Yenta/julia.html

– so called NPCs (non-person characters) in multiplayer role-playing computer games.

Auditory presentationAuditory presentation

An agent can also be presented only by voice or sound, the auditory channel– ambient sound– beeps, signals– melodies, music– recorded speech– synthetic speech

Haptic presentationHaptic presentation

In addition to auditory channel, or to replace it an agent can present information by haptic feedback

Haptic simulation modalities– force and position– tactile– vibration– thermal– electrical

Haptic output devicesHaptic output devices

Inexpensive devices: – The most common haptic devices

are still the different force-feedback controllers used in computer games, for example force-feedback joysticks and wheels.

– In 1999 Immersion Corporation’s force feedback mouse was introduced as Logitech Wingman Force Feedback Gaming Mouse

– In 2000 Immersion Corporation’s tactile feedback mouse was introduced as Logitech iFeel Tactile Feedback Mouse

Haptic output devicesHaptic output devices

More sophisticated devices:– SensAble Technologies: PHANTOM– Immersion Corporation: Impulse Engine– Often very expensive, and non-ergonomic.

VTi CyberForce Impulse Engine 2000VTi CyberTouch

PHANTOM

No direct presentation at allNo direct presentation at all

An agent helps the user by carrying out different supporting actions– e.g., prefetching needed information,

automatic hard disk management, … An indirectly controlled background

agent– question: How to implement this indirect

control?– multisensory input: the agent is observing a

system, an environment, or the user

Related user interface Related user interface metaphors:metaphors:

Conversational User Conversational User InterfaceInterface

Multimodal User InterfaceMultimodal User Interface

Conversational User Conversational User InterfacesInterfaces

Why conversation?– a natural way of communication– learnt at quite a young age– tries to fix the problems of a direct

manipulation user interface Conversation augments, not necessarily

replaces a traditional user interface– the failure of Microsoft Bob– Microsoft Office Assistant

Microsoft Office AssistantMicrosoft Office Assistant

Office assistant tries to help in the use of Microsoft Office programs with a variable rate of success.

The user can choose the appearance of the agent– unfortunately, this has no effect

on the capabilities of the agent A paper clip most likely is a

better presentation for the current assistant than a Merlin character.

Multimodal User InterfacesMultimodal User Interfaces

”Multimodal interfaces combine many simultaneous input modalities and may present the information using synergistic representation of many different output modalities” [Raisamo, 1999]

Multimodal User InterfacesMultimodal User Interfaces

An agent makes use of multimodality when observing the user:– speech recognition

reacts on speech commands, or observes the user without requiring actual commands

– machine vision, pattern recognition: recognizing facial gestures recognizing gaze direction recognizing gestures

Multimodal User InterfacesMultimodal User Interfaces

a specific problem in multimodal interaction is to combine the simultaneous inputs.– this requires a certain amount of task

knowledge and ”intelligence”– this way every multimodal user interface is

at least in some respect a user interface agent that tries to find out what the user wants based on the available information

A high-level architecture for A high-level architecture for multimodal user interfacesmultimodal user interfaces

Inputprocessing- motor- speech- vision- …

Outputgeneration- graphics- animation- speech- sound- …

Mediaanalysis- language- recognition- gesture- …

Mediadesign- language- modality- gesture- …

Interactionmanagement

- media fusion

- discoursemodeling

- planrecognitionandgeneration

- usermodeling

- presentationdesign

Ap

plic

atio

n in

terf

ace

Adapted from [Maybury and Wahlster, 1998]

ModelingModeling

[Nigay and Coutaz, 1993]

PutPut– That – That – There– There[Bolt, 1980]

Example: Digital Smart KioskExample: Digital Smart Kiosk

Smart Kiosk was a research project at Compaq-Digital Cambridge Research Laboratory in which an easy-to-use information kiosk has been built to be used by all people

Combines new technology:– machine vision, pattern recognition– speech synthesis (DECtalk)– speech recognition– animated talking head (DECface)

[Christian and Avery, 1998]

Example: Digital Smart KioskExample: Digital Smart Kiosk

Vision

DECface

Netscape Navigator

Active vision zone

Touchscreen

Example: Digital Smart KioskExample: Digital Smart Kiosk

Example: Digital Smart KioskExample: Digital Smart Kiosk

Example: Digital Smart KioskExample: Digital Smart Kiosk

ReferencesReferences

[Bolt, 1980]  Richard A. Bolt, Put-that-there. SIGGRAPH ‘80 Conference Proceedings, ACM Press, 1980, 262-270.

[Christian and Avery, 1998] Andrew D. Christian and Brian L. Avery, Digital Smart Kiosk project. Human Factors in Computing Systems, CHI ’98 Conference Proceedings, ACM Press, 1998, 155-162.

[Nigay and Coutaz, 1993]  Laurence Nigay and Joëlle Coutaz, A design space for multimodal systems: concurrent processing and data fusion. Human Factors in Computing Systems, INTERCHI ’93 Conference Proceedings, ACM Press, 1993, 172-178.

[Raisamo, 1999]   Roope Raisamo, Multimodal Human-Computer Interaction: a constructive and empirical study. Ph.D. dissertation. Report A-1999-13, Department of Computer Science, University of Tampere. http://granum.uta.fi/pdf/951-44-4702-6.pdf

[Maybury and Wahlster, 1998]  Mark T. Maybury and Wolfgang Wahlster (Eds.), Readings in Intelligent User Interfaces. Morgan Kaufmann Publishers, 1998.