Human Gesture Vocabulary for 3D Computer Environments

George Mudrak

Department of Computer Science

University of Colorado at Colorado Springs

Research paper for Dr. Chow

CS 525 Multimedia Communications and Computing

Abstract. As computing power and capabilities increase new paradigms in interaction

will naturally evolve that better enable the human computer interaction experience.

This has been demonstrated through the invention of the keyboard, the mouse and

tablet input devices. While these devices moved computer interaction forward they

require that the human user adapt to the technology and its limitation. Therefore these

devices represent a bridge between the flat 2 dimensional nature of the computers’

operational space and the need for humans to interact. More importantly, each device

represents a more natural state of progression through the introduction of a

“vocabulary” or a standard series of predefined inputs that would equate to specific

computer based actions. These vocabularies are extremely critical for optimal flow1 of

the human computer experience and logically evolve as computer interaction aligns

with the naturalistic mechanism that humans employ when interaction with their

environment. The next natural and logical evolution in computer interaction guides

technology to more correctly adapt to the human user through the use of human

gestures as inputs. For this new input means to provide lasting success a proper

1 M. Csikszentmihalyi, "Flow, the psychology of optimal experience", Harper Perennial Modern Classics. Presents and discusses the concept of "flow". How optimal experience can be achieved and benefit the user through a deep enjoyment, creativity and total involvement with life.

vocabulary must be defined. A vocabulary in which core gestures exist and represent

functionality we expect today but also extends the interface model beyond the current

flat 2 dimensional space today's devices support. A gesture based language with the

proper core gestures will easily map to a 3 dimensional computing environment. In this

paper, I will assume that such a 3 dimensional computer environment already exists and

will assume that technology already exists to properly read the human hand gestures.

With this groundwork, I focus on the necessary core gesture vocabulary for interacting

with such a 3d environment.

1 Introduction

Computer progress continues to drive innovation in the growing number of applications that computer

systems can perform on behalf of people today. In the vast majority of these proxy duties, humans

interact with machine in a hoped for ideal in which the method of interaction are (1) understood with

minimal ramp up time, (2) present efficient means of interaction and (3) are optimized for natural

human usage. In order to achieve these human computer interaction (HCI) goals, a working vocabulary

must be established. According to Webster’s Dictionary, a vocabulary is defined as follows.

Main Entry: vo · cab · u · lary 2

1: a list or collection of words or of words and phrases usually alphabetically arranged

and explained or defined.

2a: a sum or stock of words employed by a language, group, individual, or a work or

in a field of knowledge 2b: a list or collection of terms or codes available for use (as in

an indexing system)

3: a supply of expressive techniques or devices (as of an art form)

Definition 2b “a list or collection of terms or codes available for use” and 3 “a supply of expressive techniques”

provides the right basis for vocabularies defined for human interaction. In essence, I defined a gesture vocabulary

as:

Gesture Vocabulary 3

2 Merriam-Webster.com, http://www.merriam-webster.com/dictionary/vocabulary3 This is my definition from combining Merriam-Websters into the appropriate definition reflecting the core code plus expressive human action.

1: a list of expressive hand techniques available for use in communicating.

This definition defines the scope of my efforts in describing core human gestures in 3d computing

interaction.

2 Current Vocabulary and Devices

To understand what we take for granted, let’s look at a current input devices in use today and the

defined vocabularies for those devices.

2.1 Keyboard

The modern keyboard represents a device with a core vocabulary built around the written language.

Each letter of a language has one vocabulary keyboard key associated with it. This provides a direct

mapping between what the vocabulary states and accomplishes. Additional vocabulary keys exist that

serve for specific states / actions, such as the “Caps Lock”, “Tab”, “Control”, “Shift” and function keys.

These keys represent an extension of the core letter vocabulary presented to users by the keyboard.

Figure 1. Standard QWERTY Keyboard Layout4

With the keyboard input remains restricted to a direct mapping of vocabulary “word” to computer

action within a standard 2 dimensional operating plane. The affordances 5 for this design support direct

textual / data entry with weaker graphical and information manipulation.

4 Keyboard image found on the internet via a http://images.google.com search.5 D. Norman, “Things that Make Us Smart”, "Technologies have affordances", p.105.

2.2 Mouse

The modern mouse represents a device with a core vocabulary of essentially four states per mouse-

button,

Button Position State Code Description

Neutral N No button pressed or held down.

Clicked BC A button has been clicked / depressed.

Clicked & Held H Button has been clicked / depressed and is being held down.

Released BR Button has been released from a depressed state.

Figure 2. Standard Mouse Vocabulary

These four button states represent a portion of the mouse's vocabulary as four directional states

implemented for movement, (1) push up (2) pull down (3) push left (4) push right, exist. This represents

the extent of the vocabulary defined for the mouse as an input device. Functional extensions have been

achieved as a result of combining the various vocabulary words described above to achieve greater

features for applications. This includes capabilities such as drag-and-drop, object-selection, etc. While

this input device provides a greater degree of affordances6 for direct manipulation of the computer

operating system, applications, data, graphics, etc. it does not support such abilities as text entry, and as

such does not directly replace the keyboard. Further, this device still forces the user to operate within a

2 dimensional plane.

Figure 3. Button States Figure 4. Directional Vocabulary

6 D. Norman, “Things that Make Us Smart”, "Technologies have affordances", p. 105

Today, the mouse vocabulary has been extended to fit a more natural human-gesture based mode of

interaction by extension using “mouse gestures”. This augmentation was popularized by Norway’s

Opera Software and if slowly gaining converts in the programming community.7 In essence the new

functionality makes use of the above core vocabulary words of a mouse and ascribes new behaviors the

combinations, yielding the functional concept of “mouse gestures”.

2.3 Tablet

The third most common but less mainstream input device used today takes computer input to a more

natural state that better supports cross domain knowledge transfer skills8 from how users interact with

their world. Tablets afford users the ability to use a pen styled device that matches the core vocabulary

of a mouse in a more natural fashion. Tablets provide a canvas on which a user “writes” their actions.

Tablets do not introduce new vocabulary they simply support the mouse vocabulary. The affordances

for tablets primary provide greater degree of natural control and precision of mouse based input. This is

of significant benefit in the graphics arts and architecture domains. Additionally, tablets become the

first input device of these three that supports cross domain knowledge transfer as everybody the world

over uses (or has used) a pencil or pen to write. The writing skill readily transfers with no additional

training. Vocabulary understanding is implicit.

Figure 5. Computer Tablet.9

7 E. Hansen, “Buidling a better computer mouse”, http://news.cnet.com/2100-1023-960408.html8 Cross-domain knowledge transfer is a psychological process in which ones knowledge and skills within a known domain can more readily transfer into an unknown domain thereby giving the individual a starting skill set and knowledge base. This process reduces the necessary time an individual needs to become 'functional' within the new domain. 9 Tablet image found on the internet via a http://images.google.com search.

3 Future Vocabulary and Devices

Ongoing research continues to refine and prove out the viability of computer systems recognizing and

converting hand gestures into actionable computer tasks10. As most of the research I found revolves

around reading, interpreting and responding to gestures, this does not lead to the establishment of a

common language. The success of current input devices required a common vocabulary of known and

predictable elements. In the same way, the current generation of coarse gestures will need a common

interactive vocabulary to provide the core gesture actions for computer interaction in the operating

system and application domains. These gestures would provide the standard minimal vocabulary

needed to support core human interaction. From there the operating system and various applications

could augment gesture based interactivity by the introduction of new gestures in the same way that the

keyboard augmented its vocabulary with the introduction of function keys.

3.1 Optimal Environment

While a gesture vocabulary can facilitate natural interaction within a 2 dimensional computing system11,

the real flow of self-discoverable interaction from using gestures would come from a 3 dimensional

computing system. As such, my theories on gestures are targeted towards that space. To that end, the

optimal computing environment would require a visual display system of one or more monitors, a

system to recognize gestures with enough granularity to determine finger positions in relation to one

another along with hand/finger direction and speed. Speed must include constant measure along with

acceleration and deceleration measurements12. For my purposes, no text or data entry will be

considered, would be covered by today's' keyboard.

Additionally, I do not focus on the need for a gesture calibration stage to account for variations in user

physiology.13

3.2 Gestures

Gestures can be performed with either the left or right hand and at times may involve the use of both

hands to act upon the environment. These core gestures do not replace data entry or text based input

from a keyboard. Gestures must be as natural as possible given a non-physically challenged physiology.

10 C. Keskin, A. Erkan, L. Akarun, "Real Time Hand Tracking and 3D Gesture Recognition For Interactive Interfaces Using HMM"11 Macnn, "Apple patents depict touch gestures, 'invisible' buttons", http://www.macnn.com/articles/10/04/29/keys.could.illuminate.on.demand/12 My focus was not on 13 See the "Further Research" section at the end of this paper for more details.

In order for a gesture based language to provide value it must take into account the primary object

manipulation input device - the mouse, and provide matching vocabulary.

3.2.1 Mouse Friendly

Gestures can easily match the primary and secondary input vocabulary of a mouse. The chart below

highlights standard mouse vocabulary and provides the corresponding gesture equivalent. This

equivalency ensures that no standard of interactivity experience is lost.

Mouse Vocabulary Gesture Vocabulary

Primary button neutral Primary hand at rest

Primary button clicked Primary hand select gesture

Primary button held Primary hand pinch gesture

Primary button release Primary hand release gesture

Secondary button neutral Secondary hand at rest

Secondary button clicked Secondary hand select gesture

Secondary button held Secondary hand pinch gesture

Secondary button release Secondary hand release gesture

Figure 6. Mouse to Gesture Vocabulary

Here is the Hidden Markov Model (HMM) for the gesture states described above for the mouse

compatible vocabulary.

Figure 7. HMM for Gestures to Support Mouse Vocabulary

In the following descriptions of the gestures, I will make no distinction between primary or secondary

hand. Each gesture can be performed with either hand, unless otherwise specified.

These core gestures represent the minimum of what is necessary to support an alternative to mouse

usage within the operation system or application.

Select Gesture

Select A

Select B

Action The user reaches forward with one or two fingers and "selects" the object

they wish to interact with.

Performed By Primary, secondary or both hands simultaneously

Core Purpose Selection is a key concept as a users input and interaction must be directed

to either the operation system or an application.

Gesture Select A - single finger select

The hand is curled into a relaxed fist and the pointer finger extended and

used to select the desired object for focus and interaction.

Select B - two finger select

The hand is curled into a relaxed fist and both the pointer and middle fingers

are extended and used to selected the desired object for focus and

interaction.

Dimensional Front plane. 2 dimensional.

Start State gesture neutral.

Pinch Gesture

Pinch A

Pinch B

Action The user reaches forward with pinch a gesture to "select and hold" the

object they wish to interact with.

Performed By Primary, secondary or both hands simultaneously

Core Purpose There are many times where usage of an object requires the movement and

repositioning of that object. A pinch gesture meets this user need and

functions similarly to a mouse button click and hold action.

Gesture Pinch A - two finger pinch

The hand is curled into a relaxed fist and the thumb and pointer finger

extends to grasp the desired object for interaction.

Pinch B - three finger pinch

The hand is curled into a relaxed fist and the thumb, pointer and middle

fingers are extended and used to selected the desired object for interaction.

Dimensional Front plane. 2 dimensional.

Start State gesture neutral

Release Gesture

Release A

Release B

Action The user releases (opens) their fingers, thereby "dropping" the currently held

object.

Performed By Primary, secondary or both hands simultaneously

Core Purpose Releasing an object from a "hold" state is critical in direct interaction with

objects. The release gesture allows a user that has "picked up" an object to

"let it go" when they are positional placed the object. This allows for shaping

and setting up their desktop, environment, move objects, etc.

Gesture Release A - two finger release

From the pinch-A gesture, the user quickly opens their thumb and pointer

fingers in excess of 3 cm from a closed position. This movement indicates a

release or drop for the corresponding object to be placed.

Release B - three finger release

From the pinch-B gesture, the user quickly opens their thumb, pointer and

middle fingers in excess of 3 cm from a closed position with the pointer and

middle fingers acting as a single unit and the thumb representing the

opposing unit of the gesture. This movement indicates a release or drop for

the corresponding object to be placed.

Dimensional Front plane. 2 dimensional.

Start State A pinch-A or pinch-B gesture

A common mouse gesture that utilizes the existing mouse vocabulary is a "double-click". A double-click

represents an extension of the mouse vocabulary to increase functionality and while not directly

mapped above in the gesture vocabulary, can be easily met by simply repeating the select gesture twice

on the same object within a specified amount of time. Thus no loss of functionality occurs.

3.2.2 New Gestures for Expanded Interactivity

With the core gesture vocabulary meeting the expected mouse vocabulary, we can begin extending the

gestures vocabulary to increase the range of activity that can be accomplished. As with the mouse

gestures these gestures will be compatible with 2 dimensional and 3 dimensional usage.

Swat Gesture

Swat

Action The user utilizes an open hand, finger extended gesture moving from left to

right or right to left "brushing" against an object to initiate an unguided but

directed move. Much like swatting at a fly, gnat or mosquito.

Performed By Primary, secondary or both hands.

Core Purpose This gesture offers a quicker way to move objects from the interaction path

when the final resting position is less important than moving the object

quickly. A swat gesture moves the object in a given direction with a given

velocity and momentum.

Gesture Swat A - full hand

With an open hand and all fingers closed but extended. All four fingers must

be within local proximity of each other ~ 2 cm from center to center with the

thumb in any arbitrary position. For this gesture the thumb is irrelevant.

Dimensional Front plane, 2 dimensional moves in the same plane only.

Z-plan, 3 dimensional if the swat gesture occurs within a multi-z-plane of

interaction.

Start State Gesture neutral, no selected object.

Move Gesture

Move

Action The user moves the currently selected object anywhere within the current z-

plane or across (forward and back) the existing z-planes.

Performed By Primary, secondary or both hands

Core Purpose Allow a user to manage objects within their environment relative to each

other.

Gesture From gesture pinch-A or pinch-B, the user moves the hand that has the

selected object, either within the current z-plane to simply move the object

or across z-planes to move the object deeper or closer relative to the user.

Dimensional Front plane, 2 dimensional move within the same z-plane.

Z-plan, 3 dimensional move within the same z-plane and across z-planes.

Start State Pinch state, i.e. selected and held.

Rotate Gesture

Rotate A

Rotate B

Action The rotate gesture allows a user to rotate the selected object clockwise or

counter-clockwise an arbitrary number of degrees.

Performed By Primary, secondary or both hands

Core Purpose This gesture allows a user to rotate any given object or its contents in either

a clockwise or counter-clockwise direction. The gesture is analogous to

rotating a door knob except with the hand kept in an open position.

Gesture Rotate A - clockwise

With either hand, the thumb, pointer finger and middle finger will form an

inverted "L" with the thumb on the bottom of the long leg. The thumb and

pointer finger should be approximately 5 cm apart with the pointer finger

and middle finger approximately 3 cm apart. From this gesture, the user

"touches" the object in question and rotates their hand along their wrist in a

clockwise motion. If further rotation is desired, the gesture is performed

again.

Rotate B - counter-clockwise

With either hand, the thumb, pointer finger and middle finger will form an

inverted "L" with the thumb on the bottom of the long leg. The thumb and

pointer finger should be approximately 5 cm apart with the pointer finger

and middle finger approximately 3 cm apart. From this gesture, the user

"touches" the object in question and rotates their hand along their wrist in a

counter-clockwise motion. If further rotation is desired, the gesture is

performed again.

Dimensional Front plane, 2 dimensional.

Start State Neutral state to start but as a precondition, the object must be in a selected

state.

Delete Gesture

Delete

Action The user "grabs" an object with their open hand and closes it into a loose fist

to crumple the object for deletion. This may also be referred to as a

"crumple" gesture and for the delete action an optional "throw" may be

allowed.

Performed By Primary, secondary or both hands

Core Purpose A method to delete an object must exist. This gesture is analogous to

crumpling a piece of paper prior to throwing it away.

Gesture With either the left or right open hand and fingers extended, with no specific

minimum / maximum distance, the user will close their hand upon the

desired object and continue closing into a loose fist, thereby "crumpling" the

object. This crumple constitutes a delete and may be followed by an

optional "throw" to further indicate deleting.

Dimensional Front plane, 2 dimensional.

Start State Selected state or a neutral state.

Horizontal Scroll Gesture

Horizontal Scroll

Action Employing either select-A or select-B, "touch" and swipe either to the left or

right along the horizontal axis. While the hand-gesture motion will naturally

occur along a 3 dimensional path, only 2 dimensional input will be acted

upon.

Performed By Primary, secondary or both hands

Core Purpose Allow a user to horizontally scroll the contents of the selected object,

including the operating system, to view contents "off screen".

Gesture From a select-A or select-B gesture, the user "touches" the active object and

while in contact, swipes their hand to the left or right for a horizontal scroll

action. If the user wants to scroll more, then they "lift" their hand and begin

the process again.

Dimensional Front plane, 2 dimensional.

Start State Selected state, an object must be selected.

Vertical Scroll Gesture

Vertical Scroll

Action Employing either select-A or select-B, "touch" and swipe either up or down

along the vertical axis. While the hand-gesture motion will naturally occur

along a 3 dimensional path, only 2 dimensional input will be acted upon.

Performed By Primary, secondary or both hands

Core Purpose Allow a user to vertically scroll the contents of the selected object, including

the operating system, to view contents "off screen".

Gesture From a select-A or select-B gesture, the user "touches" the active object and

while in contact, swipes their hand up or down for a vertical scroll action. If

the user wants to scroll more, then they "left" their hand and begin the

process again.

Dimensional Front plane, 2 dimensional.

Start State Selected state, an object must be selected.

Z-Plan Change Gesture

Z-Plane

Action Employing either hand, the users reaches for a bottom z-plane "anchor" and

pulls it towards them, thereby bringing it to the front.

Performed By Primary, secondary or both hands

Core Purpose Allows the user to deliberately choose another z-plane as the current top-

most z-plane, effectively bringing it to the front.

Gesture Use a select-A or a select-B on the z-plane bottom anchor to select and hold.

Then pull / move the z-plane anchor towards the user thereby bringing it

higher in the z-plane stack and closer to the user. If the user desire this

plane as the top-most interactive plane then the user would bring the anchor

forward all the way.

Dimensional Front plane, 2 dimensional.

Start State Neutral state

4 Summary

In this paper, we covered that for the next generation of human computer interaction, based on

gestures, to move forward and provide greater user flow and affordances, a core vocabulary must exist

that is both natural and intuitive. This vocabulary must take into account current input technologies

and their vocabularies to provide corresponding alternatives for the same domain spaces and does so by

providing a more natural mouse based input for potential interaction with the operating system and

various applications. While not all input device vocabularies, such as keyboards, are met by a gesture

based vocabulary, both tablets and mice have corresponding words.

With core vocabulary words defined for the gesture space applications can build on them through

combinations or introduction of new words to create an optimal user flow experience.

The gesture language functions well in a 2 dimensional space and really shines within a 3 dimensional

computing space.

Further ResearchThese are just ideas that I have captured for further research and thought but time and my focus did not

allow for additional details.

Gesture CalibrationDue to the variations in user physiology, it would be necessary for a calibration process to occur when a

user setups their system for gesture based input. Calibration must include two primary steps. First, the

selection of the primary and secondary hands, then second, calibration of the operational "bounding

box" space that the hands function within.

The calibration needs to create baseline values for the bounding box that corresponds to the valid

gesture input field. This box is represented below.

Figure 8. Operational space top view. Figure 9. Operational space side view.

Primary and Secondary hand specification is simple and most naturally fits the users "right or left hand" dominancy.

Hand Selection DescriptionPrimary The primary hand represents the more natural and dominant hand that the user will

employ when interacting with the system.Secondary The secondary hand represents the alternate hand less likely to be employed when

interaction with the system.

Calibration of the operational space must occur for the following reference points.

Coordinate DescriptionX-min X-min represents the left most "edge" of the working gesture area comfortably reachable

by the users' left hand with arm extended.

X-max X-max represents the right most "edge" of the working gesture area comfortably reachable by the users' right hand with arm extended.

Y-min Y-min represents the bottom most "edge" of the working gesture area reachable by either the users' left or right hand, roughly parallel to the solar plexus.

Y-max Y-max represents the top most "edge" of the working gesture area reachable by either the users' left or right hand, roughly parallel to the top of the users head.

Z-min Z-min represents the closed "edge" of the working gesture area comfortably reachable by either the users' left or right hand and roughly 44 cm in front of the user.

Z-max Z-max represents the furthest "edge" of the working gesture area comfortably reachable by either the users' left or right hand and roughly 80 cm in front of the user.

It is important to note that activity outside of the corresponding "min" and "max" ranges will not be considered valid input. This further allows the user to place their hands in a natural "rest" position below the Y-min range.

Z-Plane OptimizationBecause of the natural support for working within the 3 dimensional space, further work must be done to determine the optimal number of z-planes.

Z-plane optimization needs to factor in large enough "gaps" before crossing from one plane to the next to prevent accidental movement and interactivity with objects in other planes aside from the primary plane of focus. Additionally the number of planes must be such that any arbitrary image could still be quickly recognizable from within any plane. For example, an image at a full resolution of 1900 x 1200 pixels would be easily recognizable, while an image at 12 x 12 pixels at a 32nd z-plane would not be recognizable.

To many planes reduce the flow of the UI experience, therefore yielding towards a minimum number of planes such as 8 may well yield the best results for rapid user acquisition and identification.

More research would be needed in this space.

Figure 10. Z-Planes for Optimal User Experience

Table of Figures

Figure 1. Standard QWERTY Keyboard Layout............................................................................................3Figure 2. Standard Mouse Vocabulary........................................................................................................4Figure 3. Button States Figure 4. Directional Vocabulary............................................................4Figure 5. Computer Tablet..........................................................................................................................5Figure 6. Mouse to Gesture Vocabulary.....................................................................................................7Figure 7. HMM for Gestures to Support Mouse Vocabulary.......................................................................7Figure 8. Operational space top view. Figure 9. Operational space side view....................................17Figure 10. Z-Planes for Optimal User Experience......................................................................................19

Reference Material

1. Optimoz Project, http://optimoz.mozdev.org. Mouse Gestures in Mozilla and Firefox.

2. E. Hansen, “Building a better computer mouse”, http://news.cnet.com/2100-1023-960408.html.

3. A. Argyros, M. Lourakis, “Vision-Based Interpretation of Hand Gestures for Remote Control of a

Computer Mouse”, HCI/ECCV 2006, LNCS 3979, pp. 40-51, 2006.

4. M. Csikszentmihalyi, "Flow, the psychology of optimal experience", © 1990, reprint 2008, Harper

Perennial Modern Classics.

5. D. Norman, "The Design of Everyday Things", © 2002, Basic Books.

6. D. Norman, "Things that Make Us Smart", © 1993, Basic Books.

7. C. Keskin, A. Erkan, L. Akarun, "Real Time Hand Tracking and 3D Gesture Recognition For Interactive

Interfaces Using HMM".

8. Apple, Inc., N. King, D. Kerr, P. Herbst, S. Hotelling, "Electronic device having display and surrounding

touch sensitive bezel for user interface and control". U.S.Patent Office:

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml

%2FPTO%2Fsearch-adv.htm&r=1&f=G&l=50&d=PALL&S1=07656393&OS=PN/07656393&RS=PN/

07656393

9. R. Miller, "10/GUI", http://10gui.com/

10. A. Kim, "Apple Working on 3D 'Hyper-Reality' Displays",

http://www.macrumors.com/2009/12/17/apple-working-on-3d-hyper-reality-displays/, U.S.Patent

Office: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=

%2Fnetahtml%2FPTO%2Fsearch-

bool.html&r=1&f=G&l=50&co1=AND&d=PG01&s1=20090313584&OS=20090313584&RS=20090313

584

11. WikiPedia.Org, "Gesture Recognition", http://en.wikipedia.org/wiki/Gesture_recognition

12. Macnn, "Apple patents depict touch gestures, 'invisible' buttons",

http://www.macnn.com/articles/10/04/29/keys.could.illuminate.on.demand/