PERVASIVE COMPUTINGPERVASIVE COMPUTINGAT NISTAT NIST

Martin Herman, ChiefINFORMATION ACCESS & USER INTERFACES DIVISION

INFORMATION TECHNOLOGY LABORATORYNATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY

VISIONVISION• Small inexpensive computers and sensors

• in every device, appliance and piece of equipment• in buildings, offices, classrooms, homes, cars, factories• people carrying portable and wearable computers • networked to each other and the Internet

• “A trillion devices connected to the Net!"• sensing and reacting intelligently to the environment

• Information instantly accessible anywhere and anytime

• An emerging new paradigm for IT

• Enormous new global market• transforming the way U.S. industry does business• transforming the way people live and learn• requiring the development of a measurement and standards• infrastructure ensuring that U.S. manufacturers are not at a• disadvantage

NIST’s RoleNIST’s Role

Pervasive computing is in infancyPervasive computing is in infancy Many required technologies are immature & have Many required technologies are immature & have

high riskhigh risk NIST’s unique roleNIST’s unique role

– develop metrics, tests and standards to push forward develop metrics, tests and standards to push forward underlying technologies and help pervasive computing underlying technologies and help pervasive computing mature and growmature and grow

– develop test data and test scenarios/tasksdevelop test data and test scenarios/tasks– provide impartial, large-scale evaluations across many provide impartial, large-scale evaluations across many

different systems and companiesdifferent systems and companies– develop interoperability specifications and test methodsdevelop interoperability specifications and test methods– perform & provide tools for conformance testingperform & provide tools for conformance testing

Current Industry EffortsCurrent Industry Efforts Information appliancesInformation appliances

– Web TVs, screen phones, net-enabled smart handheld computers, internet Web TVs, screen phones, net-enabled smart handheld computers, internet gaming consoles, internet automotive systems, electronic booksgaming consoles, internet automotive systems, electronic books

Integration into InternetIntegration into Internet– wireline high-speed access (high-speed data over phone lines; cable)wireline high-speed access (high-speed data over phone lines; cable)– wireless high-speed access (terrestrial/LMDS; satellite services)wireless high-speed access (terrestrial/LMDS; satellite services)– short-range wireless LANs (HomeRF, Bluetooth)short-range wireless LANs (HomeRF, Bluetooth)– networking/communication protocols for information appliances (Jini, Universal networking/communication protocols for information appliances (Jini, Universal

Plug & Play, Inferno, Odyssey, T spaces, JetSend)Plug & Play, Inferno, Odyssey, T spaces, JetSend)

ApplicationsApplications– networked household (TV set-top boxes, handheld communicators, PDAs, PCs, networked household (TV set-top boxes, handheld communicators, PDAs, PCs,

laptops, printers/scanners, digital cameras)laptops, printers/scanners, digital cameras)– networked automobile (GPS, PDAs, cell phones, video games, e-mail, voice networked automobile (GPS, PDAs, cell phones, video games, e-mail, voice

mail, web surfing, navigation assistance)mail, web surfing, navigation assistance)– BodyLANs (e.g., FedEx integration of handheld computers, scanners & printers BodyLANs (e.g., FedEx integration of handheld computers, scanners & printers

to determine status of packages)to determine status of packages)

Proposed NIST EffortsProposed NIST Efforts Measurements and standards for long-term as Measurements and standards for long-term as

well as short-term needs.well as short-term needs. Rich, natural human-computer interactionRich, natural human-computer interaction

Extensive information presentation & visualizationExtensive information presentation & visualization

Dynamic networkingDynamic networking

Sophisticated multi-media information extractionSophisticated multi-media information extraction

Collaborative working environmentsCollaborative working environments

Security & reliabilitySecurity & reliability

System integration and interoperabilitySystem integration and interoperability

Tight integration of IT with electronic, semiconductor, optical, Tight integration of IT with electronic, semiconductor, optical,

manufacturing, chemical, materials, building and fire technologiesmanufacturing, chemical, materials, building and fire technologies

Worldwide Worldwide Information Information

Appliance ForecastAppliance ForecastUnit Shipments (000)Unit Shipments (000)

0

10000

20000

30000

40000

50000

60000

1997 1998 1999 2000 2001 2002

Appliances vs PCsAppliances vs PCsUS Shipments, Consumer Devices,US Shipments, Consumer Devices,

In MillionsIn Millions

0

5

10

15

20

25

1997 1998 1999 2000 2001 2002

PCs

IAs

Applications & Scenarios Personal Assistants

• Extension of today’s pocket organizers, smart cards, smart mobile phones• New functions: electronic money clip, locator, visual communicator• Provide access to personal data, monitor offices & homes, transfer funds, make secure transactions• Increase effectiveness of people’s time; extend the range of their actions

Personal assistant for youth/elderlyProvide ID informationVital dataGPS-based locationRemote transmission of location“Help me” communication functionAccess to medical informationFunds accessible for emergenciesVoice I/OEmbedded camera for visual interaction

Personal assistant for adultSecure access to informationEmbedded biometric sensorsCoordinate user’s scheduleRetrieve & store phone, fax, e-mailPersonal e-business

Electronic purchasing of ticketsElectronic toll paymentsElectronic banking

Integration with office computer, printer, in-car digital maps, WWWVoice I/O, voice-to-text, text-to-voiceEmbedded camera for visual interaction

Applications & Scenarios Smart Spaces

Smart Spaces are work spaces embedded with computers, information appliances, and multi-modal sensors that allow people to work efficiently, together or individually, through unprecedented access to information and help from computers.

Smart Space ExamplesBusiness meeting roomMedical consultation roomTraining/education facilityMilitary command centerCrisis management center

• Identify and perceive users and their actions and goals

• Speech, natural language, computer vision input

• Interaction with information-rich sources

• Provide for the use of mobile devices and receivers

• Provide extensive information-presentation capabilities

• Understand and anticipate user needs during task performance

• Distributed and local collaboration, including collaboration with field personnel & mobile workers

• Provide improved memory and summaries of activities and deliberations for later use

Technologies for ITL Technologies for ITL InvolvementInvolvement

Rich, natural HCIRich, natural HCISpeech & natural language Speech & natural language

understandingunderstandingInteractive dialogueInteractive dialogueText-to-speech generationText-to-speech generationDetecting, localizing, & Detecting, localizing, & tracking tracking peoplepeopleRecognizing peopleRecognizing peopleActivity & gesture recognitionActivity & gesture recognitionSituation awarenessSituation awarenessSummarizing activities & Summarizing activities & events events recorded by sensorsrecorded by sensorsComputer vision processingComputer vision processingDistributed multi-sensor Distributed multi-sensor fusion & fusion & integrationintegration

Integrated information Integrated information presentation & visualizationpresentation & visualization

Dynamic networkingDynamic networkingMobile/wireless communicationsMobile/wireless communicationsReal-time system & Real-time system & communications protocolcommunications protocolData compressionData compressionDistributed databasesDistributed databases

Multi-media information extractionMulti-media information extractionDocument retrievalDocument retrievalQuestion answeringQuestion answering

Collaborative working Collaborative working environmentsenvironmentsDistributed & local collaborationDistributed & local collaboration

SecuritySecurityComputer/Network SecurityComputer/Network SecurityBiometric authenticationBiometric authentication

System integration technologies & System integration technologies & interoperabilityinteroperability

Perceptive Interface Perceptive Interface OverviewOverview

VideoVideoDSPDSP

WorldWorldModelModel

PatternPatternClassifiersClassifiers

AcousticAcousticDSPDSP

DisplayDisplayServicesServices

Smart Space Test BedSmart Space Test Bed

Camera Elements

Large Screen Display

MicrophonePhased Array Array

Beams

EquipmentRoom

64 ADCs/Microphones64 ADCs/Microphones TI 320C32 60MHz with TI 320C32 60MHz with

512k of RAM512k of RAM 3 cameras/video 3 cameras/video

capturecapture Gigabit ethernetGigabit ethernet Servers for dataServers for data Pervasive devicesPervasive devices

– Palm topsPalm tops– TabletsTablets– Wireless LANWireless LAN

One-dimensional Microphone One-dimensional Microphone Array GeometryArray Geometry

Direction ofDirection ofpropagationpropagation

k

SensorSensorLocations Locations

ZZmm

-kIAUI PrototypeIAUI PrototypeAnalogAnalog ArraysArrays

59 Microphones at59 Microphones at15 mm Inter-sensor15 mm Inter-sensor

distancesdistances45 Microphones at45 Microphones at20 mm20 mm

Inter-sensorInter-sensordistancesdistances

Frontal Face Frontal Face DetectionDetection

Skin pixel extraction

Face localization

Template matching

Largest cluster of skin pixel coordinate

Template

CroppingCropping Edge detectorEdge detector

Hough TransformHough Transform

Face feature extractionFace feature extraction

Crop largest skin pixel cluster from frameCrop largest skin pixel cluster from frame

Apply: Apply:

– Edge detector to emphasize circular shapesEdge detector to emphasize circular shapes

– Hough Transform for eye-extractionHough Transform for eye-extraction

– Face Normalization to gallery conditionsFace Normalization to gallery conditions