Mobile Hci L Nigay

Laurence Nigay - [email protected]

CNRS Summer School on Mobile Learning 4-9 July 2009, Brest, France

Mobile Human-Computer Interaction

Introductory Course

Laurence NigayUniversity of Grenoble

Grenoble Informatics Laboratory - LIGEngineering Human-Computer Interaction Group - EHCI


Outline

• Focus on Human-Computer Interaction

• EHCI Group• Mobility: a broad concept• Interaction with handheld devices

• EHCI Group

EHCI group & Mobile HCI

• “EHCI” for“Engineering for Human-ComputerInteraction”– Created in 1990– 7 professors and assistant-professors– 15 PhD students– 4 research engineers and 1 post-doc

EHCI group

• EHCI group belongs to LIG(Computer Science Laboratory of Grenoble)

• 165 academic researchers• 240 doctoral and post-doctoral students• 65 technical and administrative staff

– 24 autonomous research teams

– 4 principal themes• Infrastructure (networks and data)• Software (foundations and design models)• Interaction (perception, action and dialog)• Knowledge (learning, agent models and web-ontologies)

• Software Engineering for HCI

• Development of conceptual and technical toolsbased on HCI principles : Utility, usability, context

• Establishing links between psychology-ergonomics and software engineering

EHCI group: Research thematic

Human Supported byHuman sciences

Computer sciencecontribution

InteractionSystem



EHCI group: Research axes

• 4 complementary research axes– Models and processes for designing HCI,

formal testing– Interaction plasticity

• Adaptation of HCI to interaction context– Post-WIMP interaction

• Vision based finger tracking and two-handedinteraction …

• Mixed reality– Modality and multimodality


• Models and processes for designingHCI, formal testing

• Task Hierarchy: a model at runtime


• Interaction plasticity• Adaptation of HCI to interaction context• Context: interaction resources / user preferences /

social context / location etc.


• Interaction plasticity• Adaptation of HCI to interaction context

1. PC only

Slides

Control panel


• Interaction plasticity

1. PC

2. PDA: Partialmigration (only thecontrol panel, not thespeaker video)


• Interaction plasticity



Control panel


• Interaction plasticity» camnote_shortRepetWeaving.mov





• Vision based finger tracking and two-handedinteraction


• WIMP– Windows: scroll, resize, move– Icons: representations, drag/drop– Menus: pop-up, pull-down– Pointers: mouse, trackball

• Post-WIMP

EHCI group: Research axes EHCI group: Research axes

• Vision-based tracking (finger, token)


• Mixed Reality» Video Geonote_26.avi

get


• New technique of menu: multi-strokemarking menu

» Video MenuWave.wmv» Video WaveMenuIphone.mov

1) Menu appears

2) Menu is enlarged

3) Submenu appears

4) Final item is selected

•Submenus Previsualization– Rapid scan of possibilities

•Parent menus:– Always visible– Can be directly selected

•Visual stability of the focus of attention







• Vision based finger tracking and two-handedinteraction


EHCI group: Research axes• Modality and multimodality

• RAZZLE: Mobile game– The goal of the player is to collect the pieces of a digital

puzzle– The digital puzzle pieces are scattered all over a

modeled playground

• Modality and multimodality• Multimodality for the task of collecting a selected

puzzle piece– Speech modality

– Gestural modality

– Tactile modality

Get


Video RAZZLERAZZLEExpe1.dv

Video RAZZLERAZZLEExpe2.dv

• Modality and multimodality• RAZZLE experimental tests (10 subjects / 3 sessions)

– Use of the modalities– All sessions / All subjects


31%

45%24% Tactile

GesturalSpeech


– Usage of the modalities per subject


0%10%20%30%40%50%60%70%80%90%

100%

1 2 3 4 5 6 7 8 9 10

TactileGesturalSpeech


– Subject 4: Use of the modalities per action(for the three sessions)



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3




– Subject 9: Use of the modalities per action(for the three sessions)



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Rotate 1 Collect 1 Rotate 2 Collect 2 Rotate 3 Collect 3


– The subjects used all of the modalities=> Usefulness of multimodality

– Individual preferences leading in some cases tospecialization

• Subject 4: Tactile modality for rotating a puzzle piece• Subject 9: Speech for rotating and collecting a puzzle piece=> Usefulness of multimodality


• Engineering for HCI– Models and processes establishing links between

psychology-ergonomics and software engineering

– Interaction plasticity– Post-WIMP interaction– Modality and multimodality

– http://iihm.imag.fr/en/


• EHCI group• Mobility

Mobility: a broad concept

(1) moving between devices - hot-desking(2) moving within an instrumented

environment - intelligent buildings andintelligent appliances

(3) devices within moving vehicles-computers in cars

(4) small devices that move with you -PDAs, laptop computers etc

Alan Dix (Lancaster University)









Mobility:moving between devices

• Nomadic computing



• In the 1990s, the Internet– a global transport for e-mail

and data exchange,– but mostly for people

at fixed locations

• More recently– the office extended to employees on-the-go

("nomadic workers")=> similar to the case 4 « small devices that move

with you »







Moving within an instrumentedenvironment

• Examples:– Smart spaces– Augmented classroom / Augmented museum– Wireless supermarket


What is thenearest availableconference room?

How manypeople are inthe gym? Isthe treadmill

free?

Motion Sensor:Room 3201 is

Empty What is theaverage

temperatureon the second

floor?

Smart spaces



• Olivetti Research Laboratory inCambridge, UK, in 1990-92

• Active Badge: location-aware systems

Active Badge

Active Badge

...Active Badge

Badge Sensor

Badge Sensor

Server/Applicationnetwork

wirelessIR



• Active Badge– Experimented with 32 staffs in ORL company in 2 weeks– All staff wearing badge that emits IR signal every 15 seconds– Intended to aid telephone receptionist

– FIND(name)» Provides current location of the named badge

– WITH(name)» Locates a named badge and other badges around the badge

– LOOK(location)» Provides badges near the specified location

– HISTORY(name)» Generates report of the location history for the named badge


• Server• Network Control

– Polling all sensors on the network• Representation

– Builds linked list of (ID, location, time) in time order• Data Processing

– Process large amount of data from the active badge network• Display Interface

– Showing textual information


• ATT Lab Cambridge: Active Bat– Sentient Computing: a form of ubiquitous

computing which uses sensors to perceive itsenvironment- A "follow-me phone" which would cause the

telephone nearest the recipient to ring.- Teleporting desktops via VNC just by clicking their

Active Bat near the computer.

Video Sentient.mpg


• ParcTab (Rank Xerox)1995– http://sandbox.parc.xerox.com/parctab/

10,5 cm

7,8 cm

2,4 cm


• Projet EasyLiving Microsoft

• VideoEasyLiving.mpeg




• Location-aware system (within a buidling)• Context-triggered actions are simple IF-

THEN rules used to specify how context-aware systems should adapt

Like living in a rule-based expert system




• Augmented classroom: AR (video)

geometry and mathematics education




• Augmented surpermarket– Based on RFID for products– WLan for communication inside the

supermarket– Intelligent chariots (with PC tablet)

Wireless supermarket concepts

• The Shopping Buddywireless touch-screen device attached to ashopping cart– scans in items placed in the cart by shoppers– delivers personalized services and incentives

when activated with a frequent-shopper card– RFID which triggers certain offers and can

help shoppers find anything in the store anddraw a path to find it.



Wireless supermarket Operation• As you shop, you can scan in each item

– keeps a running total of how much you are spending– eliminates the need to wait in line at the check-out– you can also check the price of each item before you buy

• As you walk down the aisles– promotions and paperless coupons “pop-up” on the screen– Reminders of articles you search– Directions to articles

• You can place an order at the deli (ex. Half a kilo of ham)from anywhere– notification to pick it up (No more taking a number and

standing in line)

Wireless supermarket Operation

Wireless supermarket Operation

• The Everywhere Display (IBM)• Beamed

from the supermarket ceiling• transforms any surface

into an interactive computer• On the floor• On the items

Wireless supermarket Operation• The Everywhere Display (IBM) transforms any surface

into an interactive computer

• A recent MIT project: Sixthsense– wearable gestural interface that augments the

physical world around us with digital infor


• Technological approach– 1. Augment the user

• The user wears or carries a device to obtaininformation about physical objects.

• => similar to the case 4 « small devices that movewith you »

– 2. Augment the physical object• The physical object is changed by embedding

input, output or computational devices on or withinit.

– 3. Augment the environment surrounding theuser









Devices within moving vehicles

micro

Keyboard


Tactile Surface

Devices within moving vehicles• FutureLab of Ars Electronica (Autria)• Augmented Reality

• Instar.mov video


• INTUITION DGA project• Augmented cockpit (Rafale)

Devices within moving vehicles• INRETS-Lille









Mobility: small devices Evolution of personal computers:a continuum

Desktops Laptops Palmtops Handhelds Wearables

Eye-pod, projection on contact lenses (univ. Washington)

Desktops Laptops Palmtops Handhelds

portability/mobility/maniability

Wearables

External support + hand(s) Hands-free1 hand (at least)

Weight

Evolution of personal computers:a continuum


Evolution of personal computers:a continuum

Wearable Computer• A small portable computer that is designed to be worn on

the body during use.• In this, wearable computers differ from PDAs, which are

designed for hand-held use.• Wearable computers are usually either integrated into

the user's clothing or can be attached to the bodythrough some other means, like a wristband.

• They may also be integrated into everyday objects thatare constantly worn on the body, like a wrist watch or ahands-free cell phone.

University of Waterloo CA

• To develop new interfaces thatmediate (augment, deliberatelydiminish, or otherwise modify) non-computer activities

• Without interfering with the user'severyday tasks

• The design of wearable computers isstill a topic of research, and a varietyof user interfaces are being proposed

Aim of Wearable Computing

University of Waterloo CA



Evolution ofWearable Computers

• Medical domain– Medical

monitoringdevice

– (blood pressure,temperature)

• Military domain Amon: ICT project

Application domains

GlucoWatch

• Mobile workers– Hands free

Application domains Wearable Computing

• Key characteristics• Constant: Always ready. It is always on and always

running.• Personal: Human and computer are inextricably

intertwined.

Mobility: Huge variability ofdevices


Mobility: small devices

• Usage spacesof wireless devices

Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002




• Information– Personal Information Management– Business information: specific professionals, such as a drug

interaction reference for doctors

• Self-Enhancement– Active assistance

• Ways in which the device extends one’s normal capabilities.(Finland: cell-phones "Kanny » extension of the hand »)

– Health and Safety• Contact with emergency response teams or family

– Education


• M-Commerce– Electronic money, e-coupons


• M-Commerce– Electronic money, e-coupons

• Relationships– Core competence of phones to connect people

• Entertainment– Music, games– The mobile game market tends to follow the more

mature console/PC game market– Game performance must match or exceed the users’

expectations


• Usage spacesof wireless devices

• Huge varietyof applications

Marcus A., Chen E “Designing the PDA of the Future” Interactions 2002

Variety of applications• Device: designed for a certain user task

• + Different devices are appropriate fordifferent tasks and this should also guide theapplication design

• - Impractical to have several different informationappliances for different purposes

Sony – Reader-ebook PRS-505Screen has no backlight or flicker to give the experience of reading a paper page

Variety of applications

• Device: designed for a certain user task– impractical to have several different information

appliances for different purposes• Device: « Swiss-army-knife » useful in many

different situations– HCI complicated / non intuituive

• the question of information appliance orSwiss-army knife still remains unanswered









Mobility: Huge variability ofdevices


• EHCI group• Mobility• Interaction with handheld devices

Objectives

From: MobileHCI 07 N. Savio J. Braiterman Design Sketch: The context of Mobile Interaction

Challenges• Key issue: Transparency/Usability of

mobile devices• Challenges for HCI

– Variability of platforms– Variability of usage context– Limited interactionnal resources

Variability of usage context

From E. Kaasinen: User acceptance of mobile services - value, ease of use, trustand ease of adoption



Challenges

• In conventional HCI: the problem space isbounded– The target user (e.g., an office clerk)– accomplishes a well-defined set of tasks (e.g., writing a report)– using a fixed class of devices (e.g., a workstation)– in a predefined set of environments (e.g., in the office)

User

PlatformEnv.

Task

• In mobile computing: the problem spaceis unbounded

User

PlatformEnv.

Task

Challenges

• Approach1: to develop the systems on a case per case basis (ad-hoc manner)– Development cost and maintenance: very high!– Consistency problem between the various versions

• Approach2: to improve the methods and tools in order to support– Portability– Reusability– Modifiability– Scalability in terms of

• Computing resources• Services• Interaction techniques (multimodality, UI for small and

large devices)=> Plastic UI

Introduction: how to address thisproblem?

Handhelds: diversity, but incommon …

• Small screens• Limited input capabilities• No standards yet

• Recommandation• Apply the foundations of HCI - e.g., the method• Create without re-inventing the wheel: draw your

design on existing solutions (in order to supportconsistency across applications/brands) and inventnew design solutions to overcome current limitations

Objectives

From: MobileHCI 07 N. Savio J. Braiterman Design Sketch: The context of Mobile Interaction

Focus on HCI

Mobile Device Goals

Gulf of Evaluation

Gulf of Execution



The design principle oftransparency

• NOT to be understood as literal• useful feedback• easy to understand• intuitive to use• clear & easy to follow instructions• context sensitive guidance of how to proceed when stuck

From Prof. Steve Whittaker

a design principle is to try to make systems transparent sothat people can understand them better and know what todo

The Design of Everyday Things

• How to design things that real people canuse: successfully, easily, pleasantly

• 7 stages of performing an action• Conceptual models – the device should

explain itself• Visibility and Affordances• Mappings• Feedback


Seven Stages of Action

Goals

Evaluation of the interpretations

Interpreting the perception

Perceiving the stateof the world

Intention to act

Sequence of actions

Execution of theaction sequence

The World


Gulfs of Execution & EvaluationGoals

Evaluation of the interpretations

Interpreting the perception

Perceiving the stateof the world

Intention to act

Sequence of actions

Execution of theaction sequence

The World

GULF OF EXECUTION

GULF OF EVALUATION


Focus on HCI

Mobile Device Goals

Gulf of Evaluation

Gulf of Execution

The Gulf of Execution

• Does the system provide actions thatcorrespond to the user’s intentions?

• The difference between intentions andallowable actions is the Gulf of Execution





The Gulf of Evaluation

• Does the system provide a physicalrepresentation that can be readilyperceived and interpreted in terms of theuser’s intentions and expectations?

• The Gulf of Evaluation reflects the amountof effort that the person must exert tointerpret the physical state of the systemand determine how well the intentionshave been met.

The Seven Stages as Design aids

Determine the functionof the system?

Tell if the system isin the desired state?

Determine a mapping fromsystem state to interpretation

Tell what state thesystem is in?

Tell what actionsare possible?

Determine a mapping fromintention to physical action?

Perform the action?

How easily can the user:


… principles for good design• Visibility – by looking, users can tell the state of the device and

possible actions• Conceptual model – operations are presented consistently; users

can easily understand “how the system works”• Mappings – it is possible (easy) to determine the relationship

between actions and results, between controls and their effects, andbetween the system state, and visible indicators

• Feedback – users get full and continuous feedback about theresults of actions


Mobile HCI: inputs/outputs

• A very active domain

• Approach1: to develop the systems on a case per case basis

• Approach2: to improve the methods and tools in order to support– Portability– Reusability– Modifiability– Scalability in terms of

• Computing resources• Services• Interaction techniques (multimodality, UI for small and

large devices)=> Plastic UI

Mobile HCI Mobile HCI


• Plasticity (approach 2)• Innovative interaction techniques

(approach 1)– Innovative interaction techniques– Multimodality



Plasticity

• Materials: capability to change their ownshape while preserving usage, they do notbreak (within some limits, of course)

• Animals, plants: they adapt to theenvironment to survive

• By analogy in HCI …

Plasticity of an interactivesystem

• Capacity of the system to adapt to thecontext of use while preserving their ownutility and usability

• Context of use …• Adaptation …• Utility and Usability …

• Context: it’s complex! Generally speaking, it is an information space– that is shared between actors, dynamic, structured, and– that serves interpretation (by the actors)

• There is “no context without context” -> context of use

• Context of use: an information space– that is shared (between the software components of the interactive system and

between the system and the users), dynamic, and structured into 3 sub-spaces:• The users• The platform (from an elementary platform such as a mobile phone to a

cluster of elementary devices)• The physical environment (location, light-sound-heat and social conditions)

– that supports the adaptation process to preserve the system utility and usability

Context of use Plasticity of an interactivesystem

• Capacity of the system to adapt to the contextof use while preserving their own utility andusability

• Context of use …

• Adaptation …

• Utility and Usability …

Two means for UI adaptation

• Remolding (reshaping)

• Redistribution

UI remolding (reshaping)• Example: Plastic Clock



UI remolding (reshaping)

PC

PDA

Nabaztag

• Physical Presentation level (PP)NSButton :

Button :

JButton :

Contrôleur dedialogue

ANF

NF

Présentation

Interaction

Interaction

Interaction

The classes of widgets are maintained but theirbehaviour (possibly, their spatial and temporallayout) has changed


• Logical presentation level (LP)

Label + T extField :

Label + ComboBox :


ANF

NF Interaction

Présentation

Présentation

Interaction

Substitution of widgets

LP Modification -> PP Modification

UI remolding (reshaping): maycover multiple levels of abstraction


ANF

NF Interaction

Présentation

Interaction



• Dialogue Control level (DC)Tasks are maintained but their ordering has changed

DC Modification -> LP modification -> PPmodification


• Functional Core Adaptator level (FCA)Contrôleur de

dialogue

ANF

NF Interaction

Présentation

Interaction


ANF

ANF

Book (workstation) - title - authors - summary - keywords …

Book(Handheld) - title - first author -keyword

Tasks and domain-dependent concepts may be added/suppressed

FCA Modification -> DC modification -> LP modification > PP modification


• Functional Core level (FC)Contrôleur de

dialogue

ANF

NF Interaction

Présentation

Interaction


ANF

ANF

The nature of the services of the functional core haschanged

FC modification ->FCA Modification -> DC modification-> LP modification > PP modification




UI Redistribution

• The UI uses interaction resources that are distributed over multiple elementaryplatform (distributed UI)

• UI redistribution may be static or dynamic

• Dynamic redistribution => on-the-fly migration of the UI

– Migration may be total: the UI releases (frees) all of the interaction resourcescurrently used and migrates entirely to other interaction resources

– Migration may be partial: only portions of the UI migrate -> what is thegranularity for UI migration?

• Rekimoto’s “pickand drop”– Distribution is static– Distribution at the

workspace level– No remolding

• i-LAND– Distribution is dynamic– Distribution at the pixel

level– No remolding

UI Redistribution

http://www.ipsi.fraunhofer.de/ambiente/english/projekte/projekte/i_land.html

• CamNote

3. Distributed UI (PC and PDA) +remolding

Remolding of thecontrol panelat the FCA level

2. Arrival of the PDA: UI partial migration

1. PC only: centralized UI

UI Redistribution Plasticity of an interactivesystem

• Capacity of the system to adapt to the contextof use while preserving their own utility andusability

• Context of use …

• Adaptation …

• Utility and Usability …

Utility of an interactive system• Capacity of the system to provide the target users with

the appropriate functions/services (not less, not more)

• Appropriate = in conformity with users’need (not withdesigners’ and developers’ needs!)

• How to elicit the appropriate services?– Talking to users is not a luxury, it is mandatory!– Questionnaires, focus group, evaluation, rapid

prototyping (low-fidelity), observation of users intheir daily activities, etc.

– In short: user-centered design

Usability of an interactivesystem: many faces!

• To make it simple: Capacity of the system to allow thetarget users to access and manipulate the servicesprovided by the interactive system in conformity withtheir cognitive, motor and perceptual capabilities– Cognitive is the hard part ! Knowledge, pleasure,

emotion, mood come into play!

• How to develop a usable system?– Use theory-based knowledge (e.g., Theory of action,

Fitts law)– Apply ergonomics rules– Design, evaluate with the end-user and iterate



• Utility and usability depend on the “values”that the user associates to the system

• The “values” depend on the context of use

• Example: possible values for a heating controlsystem that may depend on the context of use

– Money saving– Comfort

Utility and Usability are not intrinsic to a product!It depends on the context of use Mobile HCI


• Plasticity• Innovative interaction techniques

– Innovative interaction techniques– Multimodality

Interaction techniques• Small screens• Limited input capabilities

Interaction techniques:small screen

• Fisheye technique– Allows the user to see

the detailswithout loosingthe context ofthe details(Furnas:seminal work)

– Video– datelens-video-web-server.wmv


• A peep hole on a large information space [Ka-Ping Yee, CHI2003]• Problem: loss of the context -• Halo tends to alleviate this problem

Halo to make observable the information that is of interest to the user


• Microsoft Research Lab• Projet Halo

• Problem +




Video Halo.avi

Solution

• Hybrid technique: thumbnail+editing• Collapse-to-Zoom

– Pages that are to big to fit the screen are presented as a “summarythumbnail”

– Thumbnails are editable (can be suppressed/enlarged, etc.)

[Baudisch CHI 2004]


collapse

expand

• Hybrid technique: thumbnail+editing• Collapse-to-Zoom

– Pages that are to big to fit the screen are presented as a “summarythumbnail”

– Thumbnails are editable (can be suppressed/enlarged, etc.)


(advertisement)

Grey lines indicate the existence of collapsed areas

video collapsetozoom

• Menu techniques– Space on screen– No keyboard for shortcuts (novice mode)– The best way to interact is to use only one-hand


Circular menus

• Marking menus


• Advantages– Circular design– Fluid transition– Scale independance

• Limitations– Screen space requirement– Number of commands

• Error rate in expert mode• Ambiguous marks in expert mode

• Marking menus• Multi-stroke marking menus• Instead of considering a spatial compound stroke,• Multi-Stroke menus introduce a serie of simple strokes

• Require less physical input spacein novice & expert modes– A submenu is displayed on top of

its parent menu– Overlapped marks

• Increase accuracy in expert mode

• Increase the number of items– No ambiguous gestures in expert mode


video `Multistroke




Wave menus• Few physical input space• A submenu is displayed at the

center of the menu system. Inorder to remain visible, its parentmenu is then enlarged to surroundthis submenu

• In expert mode, Wave Menuswork exactly the same way asMulti-Stroke Menus

(1)

(2)

Novice Mode

Expert Mode


Wave menus

video WaveMenu


Wavelet menus: extension on Iphone



•The Wavelet menu appears centered around thecontact point.•By drawing a stroke towards the desired item, thefirst level is enlarged permitting progressiveappearance of the submenu.•A second stroke selects an item in the submenu.



Long lists management: the linear list appears inthe center of the Wavelet menu and is surroundedby its parent menus. ( video Wavelet.mov)

• Menu techniques– Space on screen– The best way to interact is to use only one-hand


video Leafmenunovice Leafmenuexpert



• Menu techniques– Space on screen– The best way to interact is to use only one-hand– ArchMenu, ThumbMenu


• Non graphical techniques => Eye-Free interaction– Tactile & Sound feedback

• Spatialized sound


Meeting at 10

SoundbeamNeckset


Vibro-Tactile feedbackShoogle senses data (and makes sense of them!) [ Univ.Glasgow 07]Sensors: the Shake includes an accelerometer, gyroscope,magnetometer, vibro-tactile display communicating viabluetoothExpresses the existence of messages, remaining resources(e.g., the battery level) using a dynamic model inspired frommechanics (balls anchored with springs bouncing against walls)


Vibro-tactile feedbackShape-change feedback: the dynamic knob [Deutsche Telecomet Univ. Postdam]

Interaction techniques• Small screens• Limited input capabilities

Text entry



Text entry Keyboard: Text entry

• LucidTouch:a see-through mobile device, pseudo-tranparency

Text entry

• LucidTouch:a see-through mobile device, pseudo-tranparency

Text entry

Keyboard: entry• Oulu University (Finland)

Text entry

• Hand writting + Movements



Pointing Techniques• The best way to interact is to use only

one-hand• Using the thumb

Gestures of the thumb MicroRolls

• “Offset Cursor” [Potter, Shneiderman 1988)]: how to replace the styluswith the finger without hiding the target

• Selection technique: finger touches the screen, display the cursor at 1/2inch over the finger, move the finger to bring the cursor over the target,remove finger

Target Finger hides the targetCursor is offset from the target

Finger moves to bring the cursor over the target

Pointing Techniques

• Shift [P. Baudisch 07, MicroSoft]: improves the “offset cursor” technique

Pointing Techniques

Video: shiftMini.m4v

Pointing Techniques

Occlusion

Border accessibility

Accuracy

Occlusion of targets Occlusion of the interface

Areas more difficult to reach [Parhi 06]Thickness of screen

edges

Larger than the stylus Prediction hard

Pointing Techniques

Target to select 1st tap Magnified popup 2nd tap

• Tap-Tap Anne Roudaut (AVI 2008)• Temporal multiplexing strategy

• 1ST tap: Specify the focus area• 2nd tap: Select the target at a larger scale

Video: taptapMini.m4v

Uni. of California, Berkeley& IBMVideo: Phone3DMvt-UIST06.mov

• Camera phone based motion sensing

New input Techniques



• Movement-based UI – 3D gesture• Tilting using accelerometers [Rekimoto 1996, Harrison&Fishkin 1998]

Photos album


Video: RocknScroll.mov

• Movement-based UI – 3D gesture• Embodied UI: Rank Xerox


New input Techniques• Embodied UI: Rank Xerox

• Embodied UI• Prototyping tool

for embodied UI– Stanford HCI group

–Video: MapPDADesignplatform-UIST06.mov


New input Techniques• RodDirect: the stylus as a peripheral device (2 degrees of

freedom) + the second hand• Two types of stylus movement in the stylus holder are

associated with parameters– (1) Twist (Rotation) (2) Push/Pull (Sliding)

Application Switching SchedulerMap Browser

JAIST: Japan Advanced Institute of Science and Technology, JAPANVideo: RodDirectDemo.mpg

• Distorsion: bendable display• The Gummi interface: just bend it! (Sony, Schwesig CHI 2004)




• Gummi: zoom-in a subway station

• Alpha-blending of 2 graphics layers

New input Techniques Mobile HCI


• Plasticity• Innovative interaction techniques

– Innovative interaction techniques– Multimodality

Multimodal Interaction• Multimodal systems

– Multi-Sensori-Motor Systems– extend the sensori-motor capabilities of

computer systems = mobile devices

• Multimodal interaction makes use of severalinput and/or output (feedback) modalities ininteracting with a computer system.– Examples of modalities: manual gestures, gaze,

touch, speech, head & body movements– Modality: human sensory channel, different

representation modality, or different input method

Selection of one or several modalities Expression

MultimodalInformationto be

conveyed

Actor of the selection

UserDesigner System

Which actor performs modalitiesselection ?

No adaptation

Selection bythe designer

Adaptability

Selection bythe user

Adaptivity Selection bythe system

Multimodality selectionGo to the middle of the message

Gestural modality Speech

Embodied modality Direct manipulation

Multimodality Adaptability



Speech

Direct manipulation

Gesture

Embodied

MultimodalityAdaptability

• Usage of the modalities• All sessions / All subjects

Multimodality Adaptativity

• Selection of the modalities by the system• Context-aware systems

Tata tata Tata tata ta ta ta ta ta ta

taaaa tatataaaa tata

RingRing VibrationVibration

Multimodality: combiningmodalities

• “Put that there” paradigmR. Bolt MIT

„Play this soundlogo”

User selects a sound logoby clicking on the title witha stylus and speaking inorder to hear it

MATCH

• Multimodal Access to City Help• A Multimode Portable Device that accepts

speech and pen gestures created byATT&T

MATCH

• Part of a multi-year contract from DARPA• Enables users to interact using speech, pen, or

synchronized combinations of speech and pen• Essentially a testbed for designing portable

multimodal applications

• Video: DEMOMULTI/CityPlannerATT.mpeg

MATCH Testing Statistics

• Exchanges 338– Speech only 171 (51 %)– Multimodal 93 (28%)– Pen only 66 (19%)– GUI actions 8 (2%)



Multimodality: combiningmodalities

• KIRUSA : Multimodal Interaction– speech, text, and touch input– graphics, text and audio output

• CARE properties• C : Complementarity• A : Assignment• R : Redundancy• E : Equivalence

Multimodality: a tool forexploring multimodal interaction• OpenInterface (OI)

– Assembling components

OI: a tool for exploringmultimodal interaction

• Construction tool: Direct manipulation andassembling of components in order tospecify a "pipeline” defining a multimodalinteraction

• Focus on multimodal interaction: The"pipeline” of components is sendingelementary tasks (i.e., "commands") to therest of the interactive system (for example,a game)

OI Example:Assembling components

Transformationprocesses

Devices

CARE CARE combinationcombination

Task:Center and Zoom Point (x,y)Center and Zoom Point (x,y)Tasks

Speechcommand

Designationcommands

Camera Microphone

Complementarity

Go there

Iterative user-centered design

InitialRequirement

s

DesignErgonomicevaluation

SoftwareprototypeTest-bed

Design


Ergonomicevaluation

Design


Ergonomicevaluation

Design

Software prototypeTest-bed

Ergonomicevaluation

Design


Ergonomicevaluation

Iterative design

Final productrunning on mobile

devices

Code

Final design:Detailed

specifications

Development

User-centered design

– To support an iterative user-centered design– To allow the rapid prototyping of the mobile

interaction for conducting test with users



Game Test-Bed on MobileDevices

• Mobile game: Running on mobile phonesor PDAs

• Test-bed: Running on PC

PC: Multimodal interaction Mobile device: Game

Bluetooth /TCP-IPWiFi

Game testbed on mobile platform



Levels ofabstraction

of theexchangedmessages

?


• 1: Raw input from the devices (low level)• 2: Application-dependent messages (high level)• 3: Modality-independent and application-independent events



• Examples: “ESC”, “UP”, “START” => virtual gamepad

• 3: Modality-independent and application-independent events


Funny Rabbit

• 3D game on mobile phones• Goal: Find the rabbit that is hidden in a

chest



Funny Rabbit

InputMultimodalinteraction

Funny Rabbit:game andgraphicaloutputs(N93 and N95)

Bluetooth

Funny Rabbit• Goal: Find the rabbit that is hidden in a chest• Tasks: <Move the character> <Select the chest> <Examine objects>• Sub-tasks: <Left> <Right> <Up> <Down>

Multimodal interaction Funny Rabbit

Funny Rabbit

OI component pipeline Funny Rabbit

ARToolkit Adapter Evt Adapter MultiBT

<Left> <Right> <Up> <Down>

Funny Rabbit


Shake Evt Adapter MultiBT

<Left> <Right> <Up> <Down>Shakedevice

Funny Rabbit


Speech Evt Adapter MultiBT

<Left> <Right> <Up> <Down>Left

Funny Rabbit


Speech Evt Adapter MultiBT

<Open the chest>

Open

Shake Openinggesture



OpenInterface• OI: Running on PC• Connection to the mobile phone

PC: Multimodal interaction Mobile device

Bluetooth /TCP-IPWiFi

http://www.oi-project.org/

• EHCI group• Mobility• Interaction with handheld devices

• EHCI group• Mobility• Interaction with handheld devices• Conclusion






Input/Output Interaction Conclusion:From WIMP to Post-WIMP

Integration into the env.

[Lyytinen & Yoo 2002]

Mobility

Traditional interaction



Conclusion:From WIMP to Post-WIMP

Mobilecomputing


Mobility



Pervasivecomputing


Mobility

Traditional interaction Mobilecomputing



Mobility

Ambiant intelligence


Pervasivecomputing

Mobilecomputing ?

Thank you

Readings

• Books– S. Weiss, Handheld usability, Wiley, 2002. ISBN 0

470 84446 9.– C. Lindholm, T. Keinonen, H. Kiljander, Mobile

usability, McGraw-Hill, 2003. ISBN 0 07 138514 2.

• Conferences– Mobile HCI ACM-Press– UBIMOB Mobilité et Ubiquité AICPS ACM-Press– CHI ACM-Press– UIST ACM-Press– AVI ACM-Press