1 Direct Manipulation Interfaces Positive acceptance of an application Mastery of the interface Competence in performing tasks Ease in learning

1

Direct Manipulation Interfaces

Positive acceptance of an application

Mastery of the interface

Competence in performing tasks

Ease in learning originally and in assimilating advanced features

Confidence in the capacity to retain mastery over time

Enjoyment in using the interface

Eagerness to show off interfaces to novices

Desire to explore more powerful aspects

2


Attributes of direct-manipulation interfaces

Visibility of the objects and actions of interest

• Example: Driving an automobile Rapid, reversible, incremental actions

Replacement of types commands by a pointing action on the object of interest

Example: Dragging a file to a trash can versus “rm file.doc”

• What about: “rm file*.doc”?

• What about “rm *.*” with no undo?

3


Extensions of direct manipulation

Virtual reality – users are in an immersed environment

• Reality is blocked out via a head-mounted display

• Hand gestures (via a data glove) allow users to point,

select, grasp and navigate

Augmented reality – user remains in normal surroundings, but adds a transparent overlay.

• Examples – labeled buildings, hidden plumbing

• Google: Project Glass

• http://www.youtube.com/watch?v=W2E2zcFt9Xo

• http://www.youtube.com/watch?v=tnRJaHZH9lo&feature=related

Tangible user interfaces – users manipulate physical objects

• Example – putting several plastic blocks near each other to

create an office floor plan

4


Command-line versus display editors versus word processors

The Tubeless Interface

5



Single-line and Multi-line Editors (e.g., IBM MVS, VM, TSO, JCL)

6


Command-line versus display editors versus word processors Single-line and Multi-line Editors versus WYSIWYG (what you see is what you get) editors (e.g.,

Microsoft Word – early 1990s)

Example: Three basic modes of vi

• Command mode (Telling the computer what to do: Low level commands, e.g., move the cursor

to the right one character)

Default when you enter vi.

Most letters, or short sequences of letters, that you type will be interpreted as commands

Pressing Esc when you're in command mode, your terminal will beep at you. This is a

very good way to tell when you're in command mode

• Insert mode (Entering the content)

Whatever you type is inserted in the file at the cursor position

Press Esc to end insert mode, and return to command mode.

• Line mode (Telling the computer what to: High level commands, e.g. Save)

To enter line mode from command mode, type a colon ( : )

Your cursor moves to the bottom of the screen, by a colon prompt.

Type a line mode command, then press Enter.

7



Single-line and Multi-line Editors (e.g., vi)

Starting vi and Saving Files

Starting vi: vi filename (start editing filename, create it if necessary)

Saving the file you're working on and/or leaving vi: :wq (write the file to disk and quit)

Quit without saving any changes: :q!

:w! newfile (write all lines from the entire current file into the file 'newfile', overwriting any existing newfile)

:n,m w! newfile (write the lines from n to m, inclusive, into the file newfile, overwriting any existing newfile)

8



Single-line and Multi-line Editors

Type To Move To

h one space to the left (also try left arrow)

j one line down (also try down arrow)

k one line up (also try up arrow)

l one space to the right (also try right arrow)

$ end of current line

^ beginning of current line

Enter beginning first word on the next line

G end of file

:n line n; use :0 to move the beginning of the file

w beginning of next word; 5w moves to the beginning of the 5th word to the right

e end of next word

b beginning of previous word

Ctrl-b one page up

Ctrl-f one page down

% the matching (, ), [, ], {, or } (Press % with your cursor on one of these characters to move your cursor its mate.)

9




• Searching for text

• Inserting text

Type To

/string search down for string

?string search up for string

n repeat last search from present position

Type To

a append starting right of cursor

A append at the end of the current line

i insert starting left of cursor

I insert at beginning of the current line

o open line below cursor, then enter insert mode

O open line above cursor, then enter insert mode

:r newfile add the contents of the file newfile starting below the current line

10




• Deleting text

Type To

x delete single character; 5x deletes 5 characters

dw delete word; 5dw deletes 5 words

dd delete line; 5dd deletes 5 rows

cw delete word, leaves you in insert mode (i.e. change word)

cc change line -- delete line and start insert mode

s change character -- delete character and start insert mode

D delete from cursor to end of line

C change from cursor to end of line -- delete and start insert mode

u undo last change

U undo all changes to current line

J join current line with line that follows (press Enter in insert mode to split line)

11




• Cutting and Pasting

• Miscellaneous Commands

Type To

xp transpose two characters (two commands, x followed by p)

yy yank (i.e. copy) one line into a general buffer (5yy to yank 5 lines)

"ayy yank into the buffer named a

P put the general buffer back before the current line

"aP put from buffer a before current line

p put the general buffer back after the current line

"ap put from buffer a after the current line

Type To

Ctrl-g show line number of current line

Ctrl-l redraw the entire display

12



Early 1980s – Text editing was done with line-oriented command languages

Nroff/troff: Unix based word processor

.nf = no-fill, you use for graphs or text that you don't want spaces to be ignored

.ce # = centers by the # of sentences you enter

.ti # = .ti indents but only for one line, so if you have one sentence that needs to be indented 8 spaces but the rest of the page is indented 3 you can use .ti 8 for that single sentence

.fi = fill-in, extra space will be ignored and text that is entered like this, the fill-in command will continue until you enter a .nf command and vice-versa

.in # = # is the number of spaces you want the text indented, this command will be in place until you put in a new .in command...if you want something to not be indented you put in .in 0

13


Command-line editors

Who would use these interfaces?

Why?

14




Why?

• Mastery of the interface

• Competence in performing tasks

• Ease in learning originally and in assimilating advanced features

• Confidence in the capacity to retain mastery over time

• Enjoyment in using the interface

• Eagerness to show off interfaces to novices

• Desire to explore more powerful aspects

15




Why?

• BETTER THAN A TYPEWRITER

• BETTER THAN A SLIDE RULE

• BETTER THAN A CALCULATOR

16


Advantages of WYSIWYG Editors Users see a full page of text

• 20 to 60 lines provides a context for each sentence The document is seen as it will appear when printed

• Eliminating the clutter of formatting commands Cursor action is visible

• Indicates where to focus attention and apply action Cursor motion is natural

• Arrow keys or mouse provide natural physical mechanisms for moving the cursor Labeled icons make frequent actions rapid

• Toolbar for frequent actions Immediate display of the results of an action

• Example: Clicking a button to center text provides immediate result Rapid response and display

• Full page of text in a fraction of a second Easily reversible actions

• Example: Undo, backspace

17


Technology advancements evolving from word processing

Integration of graphics, spreadsheets, animations, photographs, etc.

Desktop publishing software

Presentation software

Hypermedia environments and the World Wide Web (hyperlinks to documents)

Improved macro facilities (e.g., construct, save and edit sequences of frequently used actions)

Spell checkers and thesauri

Grammar checkers

• Use of passive voice

• Excessive use of certain words

• Lack of parallel construction Document assemblers

• Contracts

• Wills

18


Spreadsheets

1979 – VisiCalc from a Harvard Business School student

254 rows and 63 columns

Functions within a cell as it relates to other cells

Simulation of an accountants spreadsheet

Lotus 1-2-3 dominated the market in the 1980s

Today Excel dominates

• Graphics displays

• Multiple windows

• Statistical routines

• Database access (e.g., Price List to Service Catalog)

19


Spatial Data Management

Geographic applications – Nicholas Negroponte at MIT

Display of the world and able to zoom in on the Pacific Ocean to see markers for convoys of military ship

20


Spatial Data Management

ArcView – ESRI

• Global Information Systems (Demo)

• Select type of information to display (roads, population, rainfall, topography,

political boundaries

21


Video Games

The most exciting, well-engineered, commercially successful application of direct-manipulation concepts?

Pong

Pacman (http://www.activitypad.com/online-games/pacman/)

Field of action is visual and compelling

Button presses, joystick motions and knob rotations produce rapid response on the screen

No syntax to remember

Error messages are rare – the results of the action are obvious and easily reversed

Often there is continuous display of the score (competition between others and the player themselves)

• Positive reinforcement that encourages mastery

22


Video Games

Educational Video Games – direct manipulation

• SimCity – education on urban planning

• The Sims – stronger attraction to women then men

23


Video Games

Computer Role Playing Games (CRPGs)

• Players assume the role of a fictional character

• Activity takes place in a fictional world

• User’s control many of their character's actions

• Myst Massively Multiplayer On-Line Role Playing Games (MMORPGs)

• A large number of players interact with one another in a virtual world

• Interaction is in a persistent world

– hosted by the game's publisher

– continues to evolve while the player is away from the game

• Worldwide MMORPGs revenues exceeded half a billion dollars in 2005

• World of Warcraft

24


Video Games versus Business Applications

Game players

• Engaged in competition with a system or other players

• Seek entertainment and focus on challenge

• May prefer random events Application users

• Prefer a strong internal locus of control

• Focus on their tasks and may resent too many playful distractions

• Do not prefer random events

25


Computer-aided Design

Automobiles, electronic circuitry, aircraft, mechanical engineering

Structural engineering, floorplans, interiors, landscaping, plumbing, electrical installation, etc.

When the design is complete, the program can provide information regarding:

• Current

• Voltage drops

• Fabrication costs

• Manufacturing problems

26


Computer – aided Manufacturing and Process Control

Honeywell’s Experion Process Knowledge System

• Provides the manager of a oil refinery or power

utility plant with a colorized schematic of the

plant

• Can indicate with a red line a sensor value that

is out of range

• With a single click the operator can get a more

detailed view of the troubling component

• A second click can provide more detailed

information the sensor, or reset a value or

circuit

• Basic strategy: eliminate the need for complex

commands that the operator might only need to

recall during a once-a-year emergency

27


Direct Manipulation in Office Automation

Xerox Star

• Sophisticated Text Formatting

• Graphics

• Multiple Fonts

• High Resolution

• Cursor Based Interface Apple Lisa

• Precursor to the Macintosh

• Hardware and software designs supported

– Pull-down menus

– Multi-window manipulation

– Editing of graphics and text

– Dragging of icons

28


Direct Manipulation in Office Automation

MS-DOS Commands vs. Macintosh Direct Manipulation

• Tasks: Creating, copying, renaming, erasing files

• After training and practice, average task times:

– MS-DOS is 5.8 minutes

– Macintosh is 4.8 minutes

• After training and practice, average errors:

– MS-DOS is 2.0

– Macintosh is 0.8

29


Continuing evolution of direct manipulation

Quicken

Home Automation

• Direct manipulation on a floor plan of:

– Burglar alarms

– Heat sensors

– Smoke detectors

– Opening/closing curtains or screens

– Air conditioning and heating

– Audio/video speakers or screens

» E.g., users can route sound from a MP3 player located in the living room to the kitchen by dragging the MP3 icon into the kitchen

Virtual Worlds

• Travel through the human body

• Ride an electron cloud as it spins around a nucleus

30


Continuing evolution of direct manipulation

Problems with direct manipulation

• Spatial or visual representation are not necessarily an improvement over text

– Especially for blind or visually impaired users

• Direct manipulation designs may consume considerable screen space

– May result in scrolling or multiple actions

• Users must learn the meanings of visual representations

– Icon interpretation

• For experienced typists, taking a hand off the keyboard to move a mouse may take more time then typing the relevant command

• Users may not share the same understanding of the metaphor, analogy, or conceptual model with the designer (testing is required)

• Browser based applications limit direct manipulation (e.g., drag & drop)

– Require Dynamic HTML, Java or Flash

31


Advantages of direct manipulation

Continuous representation of the objects and actions of interest with meaningful visual metaphors

Physical actions or presses of labeled buttons, instead of complex syntax

Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

32


Advantages of direct manipulation

Design systems with the following benefits

• Novices can learn basic functionality quickly

• Experts can work rapidly to carry out a wide range of tasks, even defining new functions and features

• Knowledgeable intermittent users can retain operational concepts

• Error messages are rarely needed

• Users can immediately see whether their actions are furthering their goals, and if the actions are counterproductive, they can change the direction of their activity

• Users experience less anxiety because the interface is comprehensible and because actions are easily reversed

• Users gain confidence and mastery because they are the initiators of action, they feel in control, and they predict the interface’s responses

33


The OAI Model and Direct Manipulation

The object of interest is displayed so that interface actions are close to the high-level task domain

Little need for mental decomposition of tasks into multiple interface commands with complex syntactic forms (e.g., vi: go to line, go to word, go to character)

Each action produces a comprehensible result in the task domain that is visible in the interface immediately

The closeness of the task domain to the interface domain reduces operator problem-solving load and stress

Compared to textual descriptors, visual representations of objects may be more natural and closer to human innate capabilities

• Action and visual skills emerged well before language in human evolution

34


Visual Thinking and Icons

Semiotics – the study of signs and symbols

Icon – an image, picture or symbol representing a concept

• In computer systems usually less than one inch square (64x64 pixels)

• Smaller icons are often integrated with a window border or toolbar Task dependency

• When working on a visual task (e.g., painting program), icons may be useful

• When working on a text-based task, it may be better to stay text based Icons with words (or mouse-overs) are useful

35


Icon related guidelines

Represent the object or action in a familiar and recognizable manner

Limit the number of different icons

Make the icon standout from its background

3-d icons can be visually distracting

Ensure that a single selected icon is clearly visible when surrounded by unselected icons

Make each icon distinctive from every other icon

Ensure harmoniousness of each icon as a member of a family of icons

Design the movement animation (e.g., grayed-out ghost image on a drag)

Detail information

• Larger shadowing for a larger file

• Color to show the age of a document

• Animation to show how much of a file has printed (document icon absorbed progressively into the printer icon)

Explore use of combination of icons

• (E.g., drag a document to a printer icon)

36


Direct Manipulation Programming

Example: programming a radio to a set of stations by pressing/holding a channel selection button

Phone Services

37



Programming in the User Interface

• Sufficient computational generality

– Conditionals (if, then, else)

– Iteration (repeat/while)

• Access to appropriate data structures and operators

– File structures for directories

– Addition, subtraction, etc.

• Ease in programming

– By specification or by demonstration (Flash animation)

– Argument passing

• Simplicity in invocation and assignment of arguments

• Low risk

– High probability of bug free programs

– Halt and resume

38



Viscosity – the difficulty of making changes to a program

Progressive evaluation – the capacity for execution of partial programs

39


3-Dimensional Interfaces

Some applications are designed as 2-D to be simpler than real-world systems

• Constrain movement

• Limit interface actions

• Ensure visibility of interface objects Enhanced 3D may be better than 3D

• Flying through objects

• Multiple simultaneous views of objects

• X-ray vision

• Shrink/expand objects

• Group/ungroup components

• Going back in time Less than successful 3D interfaces

• Air-traffic control

• Showing altitude by perspective drawing only adds clutter when compared to an overview from directly above

http://www.youtube.com/watch?v=Hw-O4zX8qRY

40


4-Dimensional Interfaces 4-D Anyone?

http://www.youtube.com/watch?v=bIj0oW-tTF4&feature=fvwrel

Tesseract

41


Second Life

Multi-user environment where users interact

Users can choose avatars (fantasy images, desirable characteristics)

42


Relationship to:

Personality Theory

Social Psychology

Spatial Cognition

Applicable to business meetings, community discussion groups, political forums?

Blaxxun envision

43


3D Desktops and Workplaces

Microsoft’s Task Gallery

Intel’s Grand Canyon

Xerox PARC’s Information Visualizer

No successful products yet

44


Tips for effective 3D interfaces

Use occlusion, shadows, perspective and other 3D techniques carefully

Minimize the number of navigation steps for users to accomplish their tasks

Keep text readable (better rendering, good contrast with background, an no more than 30-degree tilt)

Avoid unnecessary visual clutter, distraction, contrast shifts and reflections

Simplify user movement (keep movements planar, avoid surprises like going through walls)

Organize groups of items in aligned structures to allow rapid visual search

Enable users to construct visual groups to support spatial recall (e.g., placing items in corners)

45


Guidelines for inclusion of 3D features

Provide users overviews so they can see the big picture

Allow teleportation (rapid context shifts by selecting destination in an overview)

Offer x-ray vision

Provide history keeping (recording, undoing, replaying, editing)

Permit rich user actions on objects (save, copy, annotate, share, send)

Give users control over explanatory text (pop-up, floating, screen tips)

Offer tools to select, mark and measure

Implement dynamic queries to rapidly filter out unneeded items

Enable landmarks to show themselves even at a distance

Allow multiple coordinated views (users can be in more than one place at a time)

46


Teleoperation

Derived from direct manipulation and process control

Physical processes taking place in a remote location

• Clean-up in a nuclear reactor Need adequate feedback in sufficient time to permit effective decision

making

• Manufacturing

• Medicine (consultation, radiology)

• Military operations (drones) Home automation

• Answering machines

• Security systems

• Energy control

• Appliances

Mahru Humanoid Robot Real-Time Teleoperationhttp://www.youtube.com/watch?v=TJmQqC1nHTU&feature=fvwrel

47


Design to accommodate teleoperation issues

Slow response times and time delays

• Transmission delay (time for command to reach the microscope)

• Operation delay (time until the microscope responds) Incomplete feedback

• The microscope can transmit its current position, but operates so slowly it cannot indicate the exact current position

Unanticipated interferences

• The slide is accidentally moved by a person at the local site May be better for the user to specify a destination (rather than a motion)

and wait until the action is completed

http://www.youtube.com/watch?v=aFPQfhUArjI&feature=fvwrel

48


Telemedicine

Remote examination

Remote surgery

Telepathology

• Magnification

• Focus

• Illumination

• Position

49


Virtual and Augmented Reality

“Being in” as opposed to “Looking at”

Architectural applications

• Wall-sized image to give perspective

• Animation to simulate movement (left to right)

• Treadmill to simulate walking toward, walk through doors, stairs

• Replace projector with a head-mounted display Some applications are better when “looked at”

• Air-traffic control

• To surgeons want to “Be in” the patients body? Training using virtual reality

• Fifth Dimension Technologies www.5DT.com

50


Virtual and Augmented Reality

The CAVE (National Center for Supercomputing Applications)

An immersive virtual reality facility designed for the exploration of and interaction with spatially engaging environments.

The stereoscopic capabilities, coupled with its uniquely immersive design, enable scientists and researchers to interact with their data

• An atmospheric scientist can actually "climb inside" of a hurricane and visualize its complex and chaotic elements from any angle or visual perspective

• A biological researcher, examining a tightly coiled strand of DNA, can virtually "unravel" this strand and manipulate it in an environment that preserves the critical depth information of the data.

• Teach a child to cross a street

• http://www.youtube.com/watch?v=C5jSSKwBbVM

51


Artificial Reality

VideoPlace – Myron Krueger

• Surround the user with an artificial reality which responded to their movements and

actions.

• The users were able to visually see the results of their actions on screen, through the use

of colored silhouettes.

• The users had a sense of presence while interacting with onscreen objects and other

users.

• The sense of presence was enough that users pulled away when their silhouettes

intersected with those of other users.

http://www.youtube.com/watch?v=0MsySesdisE

eHarmony 2.0?

52


Applications of virtual environments

Phobia treatment

• Acrophobia Pain Control

• Immersive environments provide distractions for patients Interior Design

• OAI model

– Click, drag, enlarge objects

– Room painting tool

http://www.youtube.com/watch?v=jNIqyyypojg

53


Augmented Reality

See the real world with an overlay of additional information

• See wires or plumbing behind walls

• Tourist glasses – label buildings in a historic town

• Molecular biology

http://www.youtube.com/watch?v=ZczX6qleV4Q

54


Virtual environments dependent on integration of multiple technologies

Visual Display

• Normal Display

– 12 to 17 inches diagonally at a normal viewing distance of 70 cm subtends a visual angle of about 5-degrees

• Large Screen

– 17 to 30 inches can cover 20 to 30-degrees

• Head Mounted Displays

– 100 degrees horizontally and 60-degrees vertically

– Head motion produces new images so users perceive 360-degrees

– Displays must approach 100-millisecond delay in presenting images to approach real time

55



Head-position sensing

• Head-mounted displays can provide differing views depending on head position

Hand-position sensing

• DataGlove

56



Force feedback and haptics

• Hand-operated remote-control devices for performing chemistry experiments or for handling nuclear materials

• Gives users a sense of grasp Sound input and output

• Training of Army tank crews while using realistic sounds of battle resulted in:

– Elevated heart rates, more rapid breathing, and increased perspiration

• Speech recognition for initiating actions and making menu selection

– Keyboard and mouse use is restricted Other sensations

• Tilting and vibration of flight simulators Collaborative and competitive virtual environments

• Two people at remote sites working together while seeing each others actions and the object of interest

57


Definition, benefits, and drawbacks of direct manipulation

Definition

• Visual representation (metaphor) of the “world of action”

• Objects are always shown

• Rapid, incremental, and reversible actions

• Replacement of typing with pointing/selecting

• Immediate visibility of results of actions Benefits over commands

• Less syntax reduces error rates

• Errors are more preventable

• Faster learning and higher retention

• Encourages exploration

58


Definition, benefits, and drawbacks of direct manipulation

Concerns

• Increased system resources (possibly)

• Some actions may be cumbersome (e.g., form fillin forcing use of mouse)

• History and other tracing may be difficult

• Visually impaired users may have more difficulty

59


Piaget’s Four Stages of Development

Sensorimotor (birth to 2 years)

Preoperational (2 to 7 years)

Concrete operational (7 to 11 years)

• Physical actions on an object are comprehensible

• Children acquire the concept of conservation or invariance

• Direct manipulation brings activity to this stage Formal operational (begins at 11 years)

• Symbol manipulation to represent actions on objects

Documents

1 Direct Manipulation Interfaces Positive acceptance of an application Mastery of the interface Competence in performing tasks Ease in learning