HapticsHapticsME7960, Sect. 007Lect. 1: Overview

Spring 2011Prof William ProvancherProf. William Provancher

University of UtahSalt Lake City, UT USA

We would like to acknowledge the many colleagues whose course materials were borrowed and adapted in putting together this course, namely: Drs. Allison Okamura (JHU), Katherine Kuchenbecker (U Penn), Francois Conti, Federico Barbagli, and Kenneth Salisbury (Stanford), Ed Colgate (Northwestern), Hong Tan (Purdue), Blake Hannaford and Ganesh Sankaranarayanan (U. Washington), and Karon MacLean (UBC).

T d ’ ClToday’s Class

Haptics: definition Haptics: definition Course overview and mechanics Introduction to Haptics

Haptics overviewHi t History

High-level taxonomy of haptic devices Applications/Motivations Applications/Motivations Current challenges

Readings

2

Readings

Haptics

Haptics comes from Greek word ἅπτεσθαι Haptics comes from Greek word ἅπτεσθαι (haptesthai) which means “touch”

Haptics is to touch as Optics is to sight

3-3-

hap·tic ('hap-tik)hap tic ( hap tik)adj.

Of or relating to the sense of touch; tactile.[G k h tik f h t th i t t h (1890)][Greek haptikos, from haptesthai, to grasp, touch. (1890)]

• Location/configuration• Motion

Kinesthesia

• Temperature

• Force• ComplianceCutaneous

• Texture • Slip• Vibration• Force

4

• Force• Pain

© J. Edward Colgate 2006

Obj ti f thiObjectives of this course Gain broad perspective of haptics p p p Haptic perception Psychophysics

Design and control of haptic interfaces Design and control of haptic interfaces Stability Haptic Rendering Teleoperation

Hands-on experience with haptics Homework/Labs and Project Homework/Labs and Project

Gain some familiarity with current research and literature

5

Identify opportunities for research in haptics

Acknowledgmentsg

Katherine Kuchenbecker, U. Penn

Allison M. Okamura, JHU

J. Edward Colgate, Northwestern

Ken Salisbury, Stanford

Blake Hannaford, UW

6

Ganesh Sankaranarayanan, UW Federico Barbagli, Stanford and Hansen Medical

Francois Conti, Stanford

© K. Kuchenbecker + W. Provancher 2008

Ad i i t ti D t ilAdministrative Details

Class time is T Th 10:45 12:05 pm Class time is T Th 10:45-12:05 pm Dr. Provancher’s office hours:

TBA @ 2136 MEB TBA @ 2136 MEB Class Location: 3147 MEB Lab Location: 2172 MEB Readings: To be posted on class website Course Website:

http://mech.utah.edu/haptics

7

ttp // ec uta edu/ apt cs

Course Website:htt // h t h d /h tihttp://mech.utah.edu/haptics Readings password protected (to avoid the wrath of publishers)g p p

8

G diGrading 10% Class participation/contribution

A k ti ! Ask questions! Offer opinions, insights, during class and discussions

35% Homework/Labs 7-10 assignments

10% Individual paper presentationsE h t d t t i 10 i t t lk h Each student team gives a 10 minute talk on research paper

35% Project 33% of this for the final report

10% Project presentations and DemoEach student team gives a 15 minute presentation of

9

Each student team gives a 15 minute presentation of their project

C St tCourse Structure

First Half to Two Thirds First Half to Two-Thirds Lectures, readings, homework/labs

Last Half or ThirdP t ti d di i Paper presentations and discussions

ProjectP j t t ti d d t Project presentations, demo, and report

10

C t t f th CContent of the Course Human haptic sensing neurophysiology and Human haptic sensing, neurophysiology and

psychophysics Device design - haptic devices and tactile devices Haptic rendering Issues in device design for humans

St bilit l i Stability analysis Fundamentals of teleoperation and control Improvements in haptics Improvements in haptics Modeling humans (limbs and system ID) Event-based haptics

11

Event based haptics

Wh I?Who am I? Mechanical Engineering Mechanical Engineering Adjunct appointment in Computer Science

4 primary research topics 4 primary research topics Design of tactile displays and tactile perception Haptic guidance systems (e.g., active handrest)p g y ( g , ) Embedded manufacturing (embedded sensing

and actuation in multi-material structures) Climbing Robots

Haptics and Embedded Mechatronics LabR l t d R h

12

Related Research

Fingertip Shear Feedback for Haptic Guidance

Tactile feedback provides a new means for communication and provides possible safety advantages due to reduced cognitive load.

13-13-

Active Handrest for Precision Active Handrest for Precision Manipulation and Ergonomic SupportManipulation and Ergonomic SupportManipulation and Ergonomic SupportManipulation and Ergonomic Support

14-14-

Device Control Inputs

30 mm/s rapid 10 mm/s while drawing

15

30 mm/s rapid, 10 mm/s while drawing

Wh ?Who are you? Grads/Undergrads Grads/Undergrads

ME CS BME ECE ? ME, CS, BME, ECE, ?

Research interests Research interests

T k d i d d b k d i Take and review ungraded background quiz Hand in at end of class (at latest by next class) Those who need to get into this class must complete

16

Those who need to get into this class must complete this today

© A. Okamura, 2006

G tti th t t f lGetting the most out of class

Learning Method % RetentionLearning Method % Retention What one reads 10% What one hears 26% What one sees 30% What one sees and hears 50% What one speaks 70% p

J.E. Stice, Engineering Education, pp. 291-296, 1987

17© A. Okamura, 2006Any Questions?

Origins of HapticsMechanical Teleoperation

Ray Goertz Argonne National Lab 1940s Ray Goertz, Argonne National Lab, 1940s

18

Modern Teleoperation

JASON, e.g. Sayers (1999)

JPL

19© K. Kuchenbecker 2008

Haptic Interaction with Virtual Obj tObjects

Information and power flows

Courtesy Mandayam Srinivasan MIT

20

Courtesy Mandayam Srinivasan, MIT

© Barbagli, Conti, Salisbury 2008

S t i lSome terminology “Haptic” refers to the perceptual system that draws Haptic refers to the perceptual system that draws

information from the skin and kinesthesis “Proprioception” is the unconscious perception of Proprioception is the unconscious perception of

movement and spatial orientation arising from stimuli within the body itself

“Kinesthesia” is the sense that detects bodily position, weight, or movement of the muscles, tendons and jointstendons, and joints

“Tactual Stereognosis” is the perception of the form of an object by means of touch

21

of an object by means of touch

Types of Haptic Sensing Jointyp p g

Kinesthetic / Proprioceptive

Joint

Stems from body movement

Skin Surface

Cutaneous / TactileCutaneous / Tactile Stems from skin contact

22Illustration by A. M. Okamura.

© K. Kuchenbecker 2008 + W. Provancher 2009

Haptic Interfaces

Kinesthetic (F F db k) Displa sKinesthetic (Force Feedback) Displays Devices that provide forces or motions

TactileDisplay

Hybrid Haptic DisplaysAttempt to combine kinesthetic and tactile feedback

T til Di l

23

Tactile Displays Devices that stimulate skin to create touch sensation

© K. Kuchenbecker + W. Provancher, 2008

Ki th ti Di lKinesthetic Displays

Devices that provide forces or motionsDevices that provide forces or motions

Device TaxonomyDevice Taxonomy Grounded devices Body-based devices Inertial reaction devices

Transmission Types Impedance (Low inertia, backdriveable)

Ad itt ( b kd i bl )

24

Admittance (non-backdriveable)

Grounded kinesthetic devicesI d TImpedance Type

SensAble Phantom PremiumsI i I l E i SensAble Phantom Premiums Immersion Impulse Engine

Haptic Paddle

SensAble Omni MPB Freedom6S

Magnetic Joystick, Hollis g y ,

25Force Dimension Omega

Novint Falcon

© Colgate, Kuchenbecker, Provancher 2008/2009Exoskeleton, Rosen

Grounded kinesthetic devicesAd itt TAdmittance Type

Cobot Hand Controller[F l i /C l t ]

26© J. Edward Colgate 2006

[Faulring/Colgate]http://lims.mech.northwestern.edu/projects/handcontroller

Haptic Master (Moog Inc.)

B d b d d iBody-based devices

Cybergrasp Cybergrasp Rutgers Hand Master Arm Exoskeleton

Immersion CyberGrasp Glove

EXOS EAM Arm Exoskeleton

27© Colgate, Kuchenbecker, Provancher 2006-2009

Inertial Reaction Devices(Vibrotactile feedback)

Game controllers Game controllers

motorsshafts Vibrotactors are also

28

eccentric masses

© J. Edward Colgate 2006 + W. Provancher 2010

in cell phones

Quick Demo• Grounded kinesthetic device – Impedance Type• Grounded kinesthetic device – Impedance Type• Reaction Devices (Vibrotactile feedback)

Haptic PaddleVibrotactorMounted on

29© Provancher 2011

Haptic Paddle Mounted on Plastic disk

T til Di lTactile Displays Devices that stimulate skin to create touch sensation

Device Taxonomy Pin Arrays

Arrays of vertically moving pins Arrays will laterally moving pins (Hayward and others)

Slip Display (Colgate, Caldwell, and others) Shear Display

Variable friction display (Colgate and others)Variable friction display (Colgate and others) Shear motion (Provancher and others)

Contact Location Display (Provancher)

Other Technologies Other Technologies Vibrotactile: Pager motors and voice coils Pneumatic Electrotactile

30

Suction Ultrasonic Pressure Etc…

Pin Arrayswith vertically moving pins

Burdea & Coiffet (1994)

31

Kontarinis, et al. (1995)

Wagner, et al. (2002) Kaczmarek, et al. (1995)

© K. Kuchenbecker 2008 + W. Provancher 2009

Pin Arrayswith laterally moving pins

Hayward, et al. (2003-7)

32

Wang, et al. (2006) Fritschi, et al. (2006)

© K. Kuchenbecker 2008 + W. Provancher 2009

Sli Di lSlip DisplaysHole in outer handle to contact inner rotating mandrel

Salada et al (2002 5)Salada, et al. (2002-5)

Northwestern Haptic Knob.Salada, Lipsey, Colgate (early 2000’s)

http://lims.mech.northwestern.edu/projects/rotaryknob/

33© W. Provancher 2009

Combined tactile and kinesthetic display

V i bl F i ti Sh Di lVariable Friction Shear Display

Glassmire thesis (2006)

TPaD: Tactile Pattern Display.

34

Winfield & Colgate (2007)http://lims.mech.northwestern.edu/projects/TPaD/index.htm

© K. Kuchenbecker 2008 + W. Provancher 2009Biet et. al (2008)

Sh M ti Di lShear Motion Display Fingertip Shear Feedback & Skin Stretch for Fingertip Shear Feedback & Skin Stretch for

Haptic (Directional) Guidance Currently Currently

miniaturizing

Directional Shear Cues

35© W. Provancher 2009

Gleeson, Horschel, Provancher (2009)

Contact Display Prototypecombined tactile and kinesthetic displayy

2-D Contact Display Concept

ThimbleThimblePushPush--Pull WiresPull Wires

1-D Contact Display Concept

GimbalGimbal

36

Provancher et. al (2005)

© W. Provancher 2009

Pneumatic and vibrotactile f db kfeedback

Immersion CyberTouch Glove Sato, et al. (1991)

37

Stone (1992)

Burdea (1996)

© K. Kuchenbecker 2008Add Moy & Fearing, and Cahn & Fearing

T til D i Ch llTactile Device Challenges Stimulation density needs to be high Stimulation density needs to be high

Multimodal sensations are hard to overlay

High complexity and weight diminishes portability practicalityportability, practicality

Passive touch does not feel real

Tactile device design is difficult!38

Tactile device design is difficult!© K. Kuchenbecker 2008

H ti h li tiHaptics has many applications Blind Persons Entertainment Blind Persons

Programmable Braille Access to GUIs

Entertainment Arcade (steering wheels) Home (game controllers)

Training Medical Procedures Astronauts

Automotive BMW “iDrive” Haptic Touchscreens

Education Computer-Aided Design

p Mobile Phones

Immersion “Vibetonz”A i ti /M d li Assembly-Disassembly

Human Factors Animation/Modeling Art Material Handling

39

Material Handling Virtual Surfaces

© J. Edward Colgate 2006

Training Visual display alone is not sufficient for

certain types of virtual environments. To learn physical skills, such as using complicated hand tools, haptic information is

i ta requirement

40Applications

© J. Edward Colgate 2006

Virtual Prototyping

41

Applications McNeeley et al. (Boeing Corp.)© J. Edward Colgate 2006

Rehabilitation

42

Applications© J. Edward Colgate 2006

Teleoperation

43

Applications© J. Edward Colgate 2006

Telemedicine Future application for tactile

sensing/displayg p y Telemedicine/Telesurgery

44-44-www.intuitivesurgical.com

Computer interface for blind users Text based computers can easily be enhanced to Text-based computers can easily be enhanced to

include a speech synthesizer Graphical user interfaces are inherently visual Graphical user interfaces are inherently visual A haptic display can help a blind computer user

interact with graphics-based operating systems

45© J. Edward Colgate 2006

Entertainment

Microsoft Sidewinder Steering Wheel

46

Applications© Colgate, Kuchenbecker,

Provancher 2006-2009Logitech Wingman

Joystick

Underlying motivations for h tihaptics Looking across applications we find common Looking across applications, we find common

motivations: Haptics is required to solve the problem

I t f f th bli d Interfaces for the blind Phlebotomy training (task is mainly “feel”) Vibetonz – a private communication channel Improve communication (e g provide direction cue) Improve communication (e.g., provide direction cue)

Haptics improves realism and sense of immersion Entertainment Animation/modeling Animation/modeling Diagnosis (e.g., Medical)

Haptics provides constraint Assembly/disassembly

47

Assembly/disassembly Virtual surfaces

© J. Edward Colgate 2006 + W. Provancher 2009

How to design effective haptic i t finterfaces

A simple three step program A simple three-step program…

U d d h h h dA. Understand how the human sensory and perceptual systems work

B Use this information to develop performanceB. Use this information to develop performance metrics

C. Understand how to build/control machines thatC. Understand how to build/control machines that display haptic percepts and meet performance metrics

48© J. Edward Colgate 2006

Some current challenges in h tihaptics

Low power embedded (miniaturize) haptics Low power, embedded (miniaturize) haptics Mobile electronicsE l iti t t l t i ( f Exploiting tactual stereognosis (e.g., for automobile instrument panels)

E k l t l d i h ’t b th Exoskeletal devices haven’t been the answer Haptics over the internet (e.g., for

teles rger )telesurgery) Latencies are a big issueH ti f db k f t

49

Haptic feedback for amputees© J. Edward Colgate 2006

H ti f th tiHaptics for prosthetics

“Sensory reinnervation” provides a possible Sensory reinnervation provides a possible means for restoring the sense of touch to amputeesamputees

Kinesthetic and tactileand tactile sensorsHaptic

display

50© J. Edward Colgate 2006

R di f t tiReading for next time Overview of hapticsp

B. Hannaford and A. Okamura, Haptics, In Handbook of Robotics, Springer Publishing, 2008.

Overview of reception and perception (neurophysiology and psychophysics) Read P. 149-161 of

Karon E. MacLean, Haptic Interaction Design for Everyday Interfaces (CHAPTER 5), In Reviews of Human Factors and Ergonomics, V l 4Volume 4

Read Synopsis and Figs. 1 and 2 of Johansson R.S. and Flanagan, J R. (2007) Tactile sensory control of

object manipulation in human In Handbook of the Senses Vol :

51

object manipulation in human. In Handbook of the Senses. Vol.: Somatosensation. Edited by Kaas J. and Gardner E. Elsevier