Interaction DevicesHuman Computer InteractionCIS 6930/4930Section 4188/4186

Interaction Performance60s vs. TodayPerformanceHz -> GHzMemoryk -> GBStoragek -> TBInputpunch cards -> Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web camsOutput10 character/secMegapixel displays, color laser, surround sound, force feedback, VRSubstantial bandwidth increase!

Interaction PerformanceFuture?Gestural inputTwo-handed input3D I/OOthers: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedbackEngineering research!Entire companies created around one single technologyCurrent trend: Multimodal (using car navigation via buttons or voice)Helps disabled (esp. those w/ different levels of disability)

Keyboard and KeypadsQWERTY keyboards been around for a long time(1870s Christopher Sholes)Cons: Not easy to learnPros: FamiliarityStats:Beginners: 1 keystroke per secAverage office worker: 5 keystrokes (50 wpm)Experts: 15 keystrokes per sec (150 wpm)Is it possible to do better? Suggestions?

Keyboard and KeypadsLook at the piano for possible inspirationCourt reporter keyboards (one keypress = multiple letters or a word) 300 wpm, requires extensive training and useKeyboard properties that matterSize large - imposing for novices, appears more complexmobile devicesAdjustableReduces RSI, better performance and comfortMobile phone keyboards, blackberry devices, etc.

Keyboard LayoutsQWERTYFrequently used pairs far apartFewer typewriter jamsElectronic approaches dont jam.. why use it?DVOARK (1920s)150 wpm->200 wpmReducing errorsTakes about one week to switchStops most from tryingABCDE styleEasier for non-typistsStudies show no improvement vs. QWERTYNumber padsWhats in the top row? Look at phones (slight faster), then look at calculators, keypadsThose for disabledSplit keyboardsKeyBowls orbiTouch (screenshot)Eyetrackers, miceDasher - 2d motion with word prediction

KeysCurrent keyboards have been extensively testedSizeShapeRequired forceSpacingSpeed vs. error rates for majority of usersDistinctive click gives audio feedbackWhy membrane keyboards are slow (Atari 400?)Environment hazards might necessitate Usually speed is not a factor

Keys GuidelinesSpecial keys should be denotedState keys (such as caps, etc.) should have easily noted statesSpecial curves or dots for home keys for touch typistsInverted T Cursor movement keys are important (though cross is easier for novices)Auto-repeat featureImproves performance, but only if repeat is customizable (motor impaired, young, old)Two thinking points:Why are home keys fastest to type?Why are certain keys larger? (Enter, Shift, Space bar)This is called Fitts Law

Keypads for small devicesPDAs, Cellphones, Game consolesFold out keyboardsVirtual keyboardCloth keyboards (ElekSen)Haptic feedback?Mobile phonesCombine static keys with dynamic soft keysMulti-tap a key to get to a characterStudy: Predictive techniques greatly improve performanceEx. LetterWise = 20 wpm vs 15 wpm multitapDraw keyboard on screen and tap w/ penSpeed: 20 to 30 wpm (Sears 93)Handwriting recognition (still hard)Subset: Graffiti2 (uses unistrokes)

Pointing DevicesDirect manipulation needs some pointing deviceFactors:Size of deviceAccuracyDimensionalityInteraction Tasks:Select menu selection, from a listPosition 1D, 2D, 3D (ex. paint)Orientation Control orientation or provide direct 3D orientation inputPath Multiple poses are recorded ex. to draw a lineQuantify control widgets that affect variablesText move textFaster w/ less error than keyboardTwo types (Box 9.1)Direct control device is on the screen surface (touchscreen, stylus)Indirect control mouse, trackball, joystick, touchpad

Direct-control pointingFirst device lightpenPoint to a place on screen and press a buttonPros: Easy to understand and useVery fast for some operations (e.g. drawing)Cons: Hand gets tired fast! Hand and pen blocks view of screenFragileEvolved into the touchscreen Pros: Very robust, no moving partsCons: Depending on app, accuracy could be an issue 1600x1600 res with acoustic waveMust be careful about software design for selection (land-on strategy).If you dont show a cursor of where you are selecting, users get confusedUser confidence is improved with a good lift-off strategy

Direct-control pointingPrimarily for novice users or large user baseCase study: Disney WorldNeed to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

Indirect-Control PointingPros: Reduces hand-fatigue Reduces obscuration problemsCons: Increases cognitive load Spatial ability comes more into playMousePros: FamiliarityWide availabilityLow costEasy to useAccurateCons:Time to grab mouseDesk spaceEncumbrance (wire), dirtLong motions arent easy or obvious (pick up and replace)Consider, weight, size, style, # of buttons, force feedback

Indirect-Control PointingTrackballPros:Small physical footprintGood for kiosksJoystickEasy to use, lots of buttonsGood for tracking (guide or follow an on screen object)Does it map well to your app?TouchpointPressure-sensitive nubbin on laptopsKeep fingers on the home position

Indirect-Control PointingTouchpadLaptop mouse deviceLack of moving parts, and low profileAccuracy, esp. those w/ motor disabilitiesGraphics TabletScreen shotcomfortgood for cad, artistsLimited data entry

Comparing pointing devicesDirect pointingStudy: Faster but less accurate than indirect (Haller 84)Lots of studies confirm mouse is best for most tasks for speed and accuracyTrackpoint < Trackballs & Touchpads < MouseShort distances cursor keys are betterDisabled prefer joysticks and trackballs If force application is a problem, then touch sensitive is preferredVision impaired have problems with most pointing devices Use multimodal approach or customizable cursorsRead Vanderheiden 04 for a case studyDesigners should smooth out trajectoriesLarge targets reduce time and frustration

ExampleFive fastest places to click on for a right-handed user?

ExampleWhat affects time?

Fittss LawPaul Fitts (1954) developed a model of human hand movementUsed to predict time to point at an objectWhat are the factors to determine the time to point to an object?D distance to targetW size of targetJust from your own experience, is this function linear?No, since if Target A is D distance and Target B is 2D distance, it doesnt take twice as longWhat about target size? Not linear there eitherMT = a + b log2(D/W + 1)a = time to start/stop in seconds (empirically measured per device)b = inherent speed of the device (empirically measured per device)Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmAns: 300 + 200 log2(14/2 + 1) = 900 msReally a slope-intercept model

Fittss LawMT = a + b log2(D/W + 1)a = time to start/stop in seconds (empirically measured per device)b = inherent speed of the device (empirically measured per device)Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cmAns: 300 + 200 log2(14/2 + 1) = 900 msQuestion: If I wanted to half the pointing time (on average), how much do I change the size?Proven to provide good timings for most age groupsNewer versions taken into account Direction (we are faster horizontally than vertically)Device weightTarget shapeArm position (resting or midair)2D and 3D (Zhai 96)

Very Successfully StudiedApplies toFeet, eye gaze, head mounted sightsMany types of input devicesPhysical environments (underwater!)User populations (even retarded and drugged)Drag & Drop and Point & ClickLimitationsDimensionalitySoftware accelerated pointer motionTrainingTrajectory Tasks (Accot-Zhai Steering Law)Decision Making (Hicks Law)Results (what does it say about)Buttons and widget size?Edges?Popup vs. pull-down menusPie vs. Linear menusiPhone/web pages (real borders) vs. monitor+mouse (virtual borders)Interesting readings:http://particletree.com/features/visualizing-fittss-law/http://www.asktog.com/columns/022DesignedToGiveFitts.htmlhttp://www.yorku.ca/mack/GI92.html

Precision Pointing Movement TimeStudy: Sears and Shneiderman 91 Broke down task into gross and fine components for small targetsPPMT = a + b log2(D/W+1) + c log2(d/W)c speed for short distance movementd minor distanceNotice how the overall time changes with a smaller target.Other factorsAge (Pg. 369)Research: How can we design devices that produce smaller constants for the predictive equationTwo handedZooming

Novel DevicesThemes:Make device more diverseUsersTaskImprove match between task and deviceImprove affordanceRefine inputFeedback strategiesFoot controlsAlready used in music where hands might be busyCarsFoot mouse was twice as slow as hand mouseCould specify modes

Novel DevicesEye-trackingAccuracy 1-2 degreesselections are by constant stare for 200-600 msHow do you distinguish w/ a selection and a gaze?Combine w/ manual inputMultiple degree of freedom devicesLogitech Spaceball and SpaceMouseAscension BirdPolhemus Liberty and IsoTrack

Novel DevicesBoom ChameleonPros: Natural, good spatial understandingCons: limited applications, hard to interact (very passive) DataGlovePinch gloveGesture recognitionAmerican Sign Language, musical directorPros: NaturalCons: Size, hygiene, accuracy, durability

Novel DevicesHaptic FeedbackWhy is resistance useful?SensAble Technologys PhantomCons: limited applicationsSound and vibration are easier and can be a good approximationRumble packTwo-Handed inputDifferent hands have different precisionNon-dominant hand selects fill, the other selects objectsUbiquitous Computing and Tangible User InterfaceActive Badges allows you to move about the house w/ your profileWhich sensors could you use?Elderly, disabledResearch: Smart HouseMyron Kruger novel user participation in art (Lots of exhibit art at siggraph)

Novel DevicesPaper/WhiteboardsVideo capture of annotationsRecord notes (special tracked pens Logitech digital pen)Handheld DevicesPDAUniversal remoteHelp disabledRead LCD screensRooms in buildingMapsInteresting body-context-sensitive. Ex. hold PDA by ear = phone call answer.

Novel DevicesMiscellaneousShapetape reports 3D shape. Tracks limbsEngineer for specific app (like a gun trigger connected to serial port)Pros: good affordanceCons: Limited general use, time

Speech and Auditory InterfacesTheres the dreamThen theres realityPractical apps dont really require freeform discussions with a computerGoals: Low cognitive loadLow error ratesSmaller goals:Speech Store and Forward (voice mail)Speech GenerationCurrently not too bad, low cost, available

Speech and Auditory InterfacesBandwidth is much lower than visual displaysEphemeral nature of speech (tone, etc.)Difficulty in parsing/searching (Box 9.2)TypesDiscrete-word recognitionContinuous speechVoice informationSpeech generationNon-speech auditoryIf you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand

Discrete-Word RecognitionIndividual words spoken by a specific personCommand and control90-98% for 100-10000 word vocabulariesTrainingSpeaker speaks the vocabularySpeaker-independentStill requiresLow noise operating environmentMicrophonesVocabulary choiceClear voice (language disabled are hampered, stressed)Reduce most questions to very distinct answers (yes/no)

Discrete-Word RecognitionHelps:DisabledElderlyCognitive challengedUser is visually distractedMobility or space restrictionsApps:Telephone-based infoStudy: much slower for cursor movement than mouse or keyboard (Christian 00)Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch 91) and word processing (Karl 93)However acoustic memory requires high cognitive load (> than hand/eye)Toys are successful (dolls, robots). Accuracy isnt as importantFeedback is difficult

Continuous Speech RecognitionDictationError rates and error repair are still poorHigher cognitive load, could lower overall qualityWhy is it hard?Recognize boundaries (normal speech blurs them)Context sensitivityHow to wreck a nice beachMuch trainingSpecialized vocabularies (like medical or legal)Apps:Dictate reports, notes, lettersCommunication skills practice (virtual patient)Automatic retrieval/transcription of audio content (like radio, CC)Security/user ID

Voice Information SystemsUse human voice as a source of infoApps:Tourist infoMuseum audio toursVoice menus (Interactive Voice Response IVR systems)Use speech recognition to also cut through menusIf menus are too long, users get frustratedCheaper than hiring 24 hr/day repsVoice mail systemsInterface isnt the bestGet email in your carAlso helps with non-tech savvy like the elderlyPotentially aides withLearning (engage more senses)Cognitive load (hypothesize each sense has a limited bandwidth)Think ER, or fighter jets

Speech GenerationPlay back speech (games)Combine text (navigation systems)Careful evaluation!Speech isnt always greatDoor is ajar now just a toneUse flashSupermarket scannersOften times a simple tone is betterWhy? Cognitive loadThus cockpits and control rooms need speechCompetes w/ human-human communication

Speech GenerationEx: Text-to-Speech (TTS)Latest TTS uses multiple syllabi to make generated speech sound betterRobotic speech could be desirable to get attentionAll depends on appThus dont assume one way is the best, you should user testApps: TTS for blind, JAWSWeb-based voice apps: VoiceXML and SALT (tagged web pages). Good for disabled, and also for mobile devicesUse ifMessage is shortRequires dynamic responsesEvents in timeGood when visual displays arent that useful. When?Bad lighting, vibrations (say liftoff)

Non-speech Auditory InterfaceAudio tones that provide informationMajor Research AreaSonification converting information into audioAudiolizationAuditory InterfacesBrowsers produced a click when you clicked on a linkIncreases confidenceCan do tasks without visual cognitive loadHelps figure out when things are wrongGreatly helps visually impaired

Non-speech Auditory InterfaceTerms:Auditory icons familiar sounds (record real world sound and play it in your app)Earcons new learned sounds (door ajar)Role in video games is hugeEmotions, Tension, set moodTo create 3D soundNeed to do more than stereoTake into account Head-related transfer function (HRTF)Ear and head shapeNew musical instrumentsThereminNew ways to arrange music

DisplaysPrimary Source of feedbackProperties:Physical DimensionResolutionColor Depth and correctnessBrightness, contrast, glarePowerRefresh rateCostReliability# of users

Display TechnologyMonochrome displays (single color)Low costGreater intensity range (medical)ColorRaster Scan CRTLCD thin, brightPlasma very bright, thinLED large public displaysElectronic Ink new product w/ tiny capsules of negative black particles and positive whiteBraille refreshable cells with dots that rise up

Large DisplaysWall displaysInformationalControl rooms, military, flight control rooms, emergency responseProvidesSystem overview Increases situational awarenessEffective team reviewOld: Array of CRTsInteractiveRequire new interaction methods (freehand sketch, PDAs)Local and remote collaborationArt, engineering

Large DisplaysMultiple Desktop DisplaysMultiple CRTs or Flat panels for large desktopsCheapFamiliarSpatial divide up tasksComparison tasks are easierToo much info?HMDEventually -> Every surface a pixel

Mobile device displaysApplicationsPersonalReprogrammable picture framesDigital family portrait (GaTech)BusinessPDAs, cellphonesMedicalMonitor patientsResearch: Modality Translation Services (Trace Center University of Wisconsin)As you move about it auto converts data, info, etc. for you

Mobile device displaysActions on mobile devicesMonitor information and alert (calendar)Gather then spread out information (phone)Participate in groups and relate to individual (networked devices)Locate services and identify objects (GPS car system)Capture and then share info (phone)

Mobile device displaysGuidelines for designBergman 00, Weiss, 02Industry led research and design case studies (Lindholm 03)Typically short in time usage (except handheld games)Optimize for repetitive tasks (rank functions by frequency)Research: new ways to organize large amounts of info on a small screenStudy: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist 03)Searching and web browsing still very poor performancePromising: Hierarchical representation (show full document and allow user to select where to zoom into)

Animation, Image, and VideoContent quality has also greatly increased3D rendering is near life-likeDigital Photography is commonScanned documentsVideo compression Multimedia considerations for the disabledPrinters3D Printers create custom objects from 3D models