Designing Apps for Intel RealSense Technology

1

Developed By You

2

Designing Apps for Intel® RealSense™ Technology: User Experience Guidelines with Examples for Windows®

Kevin Arthur – Senior User Experience Researcher, Intel Corporation

Meghana Rao – Developer Evangelist, Intel Corporation

SFTS018

3

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

4

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

5

F200 Short Range Camera

• Integrated into Laptop and All-In-One Devices

• Front, user-facing configuration

F200

6

F200 Features

• Features available with the Intel® RealSense™ SDK

- Hand tracking and gestures

- Face recognition and animation

- Background replacement

- 3D scanning

- Speech

F200

7

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

8

UX Guidelines Learned from F200 Apps

• Dozens of rounds of 1-on-1 usability testing with prototypes

• Co-design with select software vendors

Selected apps from the Intel® RealSense™ App Showcasehttps://appshowcase.intel.com/realsense/

F200

9

What the Camera Sees:Know the Effective Range and Capture Volume

Effective Range for Gestures 0.2 - 0.6 m

Effective Range for Face Tracking 0.35 - 1.2 m

RGB Resolution Up to 1080p at 30 fps

Depth ResolutionUp to 640x480 at 60 fps

F200

10

Provide Visual Feedback About Range

Distance Prompts

World Diagrams

F200

11

Provide Continuous Feedback When Needed

User Viewport

Move It! by Umajin

F200

12

Provide Continuous Feedback When Needed

User Overlay

Kagura by Shikumi Design

F200

13

User Input: Choose the Right Technology

Hand Gestures

• Fun and engaging for short experiences

• Natural and unencumbered

Mouse

• Precise, fast, and efficient

• Comfortable for long-term use

Understand the strengths of input technologies. Use gestures to enhance, not replace.

Touch

• Simple and direct

F200

14

Follow Gesture Best Practices to Minimize Fatigue

• Design for left-right or arc gesture motion

• Avoid actions that require your users to lift their hand above the height of their shoulder

• Design for breaks, and break up activities into small, short actions

• Do not require many repeating gestures

• Do not require precise input, like entering text or selecting small items

• Continuous and low-risk actions are best for gesture

F200

15

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

16

APIs To Detect Capture Volume

• Developers should use device neutral functions to obtain Field of View (FOV) frustrum information to help guide users through visual feedback

• Device interface – QueryColorFieldofView and QueryDepthFieldofViewfunctions allow you to obtain the Depth and Color capture volumes

Code:

F200

17

APIs To Detect Capture Volume

Other Device interface functions of interest:

• PXCMCapture.Device. QueryDepthSensorRange – returns depth sensor range in mm

• PXCMCapture.Device. QueryColorFocalLengthMM – returns color sensor focal length in mm

• PXCMCapture.Device. QueryDepthFocalLengthMM – returns depth sensor focal length in mm

F200

18

Alerts to Indicate Face Out of the Field of View

• PXCMFaceData.AlertData.AlertType

• AlertType enumerator itemizes alert events

Alert Type Description

ALERT_NEW_FACE_DETECTED A new face is detected

ALERT_FACE_NOT_DETECTED There is no face in the scene.

ALERT_FACE_OUT_OF_FOV The face is out of camera field of view.

ALERT_FACE_BACK_TO_FOV The face is back to field of view.

ALERT_FACE_LOST Face tracking is lost

F200

19

Coding Face Alerts in Your AppF200

20

Contd…Alert Handling

}

}

F200

21

Alerts to Indicate Hands Out of FOV

Use PXCMHandData.TrackingStatusType

Alert Name Description

ALERT_HAND_OUT_OF_BORDERS A tracked hand is outside of a 2D bounding box or 3D bounding cube defined by the user

ALERT_HAND_INSIDE_BORDERS A tracked hand has moved back inside the 2D bounding box or 3D bounding cube defined by the user.

ALERT_HAND_TOO_FAR A tracked hand is too far to the camera

ALERT_HAND_TOO_CLOSE A tracked Hand is too close to the camera.

ALERT_HAND_DETECTED A tracked hand is identified and its mask is available

ALERT_HAND_NOT_DETECTED A previously detected hand is lost, either because it left the field of view or because it is occluded.

And more… Refer to the documentation

F200

22

Coding the Hand Alerts in Your AppF200

23

Contd…Alert HandlingF200

24

World Space to Screen Space Mapping –Projection Interface

• The Projection interface provides functions to map or project among color, depth and world coordinates

- MapColorToDepth

- MapDepthToColor

- ProjectCameraToColor

- ProjectCameraToDepth

- ProjectColorToCamera

- ProjectDepthToCamera

F200

25

Adjusting UI to Different Screen Resolutions – Unity

To make the App go Full-screen in windowed/unwindowed mode, whatever the resolution of the display:

• Pre-compilation, in player’s settings, make sure the "Use Direct3D 11" box is checked

• Set it to use ONLY specific aspect ratios (16:9 and 16:10 and matching children)

Result - when the ‘resolution selection screen’ appears:

• If the “Windowed” check box is UNchecked, it will always start at Full Screen

• If it’s checked, it will start windowed but can be resized/maximized

F200

26

APIs to Detect Speed and Precision

• Joint Speed – Enabling joint speed allows developers to obtain the absolute/average speed of motion of each joint

- PXCMHandConfiguration.EnableJointSpeed

- Disabled by default to preserve CPU and memory resources. Only enable as necessary.

• Hand jitters can hamper tracking. Use PXC[M]Smoother to reduce jitter

F200

27

APIs to Detect Speed and Precision

• Use PXCMHandData.TrackingStatusType to obtain tracking status for speed

- TRACKING_STATUS_HIGH_SPEED

• If using hand gestures to control the OS using Touchless Controller, use the PXC[M]TouchlessController member functions to set pointer and scroll sensitivity:

- SetPointerSensitivity

- SetScrollSensitivity

• For face detection, fast movements may lead to lost tracking. Use PXCMFaceData.AlertData.AlertType – ALERT_FACE_LOST to determine if tracking is lost

F200

28

Bounding Boxes

• Provide clear visual cues to the user on the source and destination areas for object of interaction

• Bounding boxes provide a rough boundary for user action thus alleviating the need for high precision

• Hand bounding boxes:

- PXCMHandData.IHand.QueryBoundingBoxImage

- Use PXCMHandData.AlertType – ALERT_HAND_OUT_OF_BORDERS to detect if hand is out of the bounding box

• Face bounding boxes:

- PXCMFaceData.DetectionData.QueryBoundingRect

F200

29

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

30

R200 Long Range Camera

• Integrated into Tablets and Two-In-One Detachable Devices

• Rear, world-facing configuration

R200

31

R200 Features

• Features available with the Intel® RealSense™ SDK

- 3D scanning

- Depth-enabled photo and video

- Measurement

- Gaming and play with mixed and augmented reality

Scene perception module

Camera tracking and localization

Mesh reconstruction

- Face detection and tracking

- Speech (Windows® SDK only)

R200

32

Video – ToyZ Game Prototype

• Shows real-time scene perception for collision and occlusion (no pre-scan)

• By Shachar Oz, OmekStudio at Intel

R200

33

Video – Procedural Island Prototype

• Shows scan as part of “capture and play”

• Illustrates procedural shaders and set dressing

• By Eddy Ortega, Garrett Stevens, Perceptual Computing at Intel

R200

34

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

35

R200 Range and Capture VolumeR200

Effective Range

0.6 - 3.5 m indoors

RGB Resolution

Up to 1080pat 30 fps

Depth Resolution

Up to 640x480at 60 fps

36 36

3D Scanning: Plan for the Scene

Small object capture: Instruct users to place object on flat surface and move around the object to capture it from side and top views

Person capture: The subject must remain still while the user walks around them and captures side views

Scene capture: User will scan the tablet around the room as if taking a panoramic photo

Focus 3D scanning applications on one of these 3 scenarios, as the interaction flow will differ according to the size and surfaces being captured

R200

37

User Prompts

• Give simple instructions to start

• Preview the subject or area that will be captured

R200

38

User Prompts

• Tell user if tracking is lost

• Give distance cues

R200

39

Augmented Reality:Give People a Reason to Move, or They Won’t

• Tablet as window into a virtual space vs. tablet as fixed screen

• Address with motion hints –explicitly with instructions or implicitly with content

Google Spotlight Stories “Windy Day”

R200

40

Minimize Fatigue from Holding the Device

Active Camera Mode: Tiring Inactive Camera Mode: Less tiring

Support both active and inactive camera modes

R200

41

Choose the Right Mode for the Activity

Active Camera Mode

Touch interaction less comfortable, less precise. Best for short capture interactions.

Inactive Camera Mode

Touch interaction more comfortable, more precise. Best for extended play interactions.

Touch Zones Touch Zones

R200

42

Two Styles of Mixed-Reality Games

R200

43

Plan for the Scene and Camera Qualities

• Understand the camera limitations. Depth data is less accurate on

- Very bright areas

- Clear glass

- Black surfaces

• Give relevant feedback when problems are detected

• Fail gracefully and don’t prevent play

R200

44

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

45

APIs to Obtain R200 Camera Device Information• The Device interface provides the following functions:

API Description

QueryDSLeftRightCroppingSetDSLeftRightCropping

Access to the left/right image cropping mode.

QueryDSLeftRightExposureQueryDSLeftRightExposureInfoSetDSLeftRightAutoExposureSetDSLeftRightExposureSetDSLeftRightExposure

Access to the left/right stream exposure value.

QueryDSLeftRightGainQueryDSLeftRightGainInfoSetDSLeftRightGain

Access to the left/right stream gain adjustment.

QueryDSMinMaxZSetDSMinMaxZ

Access to the distance value range.

R200

46

Checking Scene Quality

• Use PXC[M]ScenePerception:CheckSceneQuality to check whether the stream quality is suitable to start scene perception

• Returns a value between 0 and 1. Higher the return value, better the scene

• Code:

R200

47

Checking Tracking Accuracy

• Use PXCScenePerception::TrackingAccuracy to obtain tracking accuracy data

- The TrackingAccuracy enumerator itemizes the tracking accuracy definitions

Name Description

HIGH High tracking accuracy

LOW Low tracking accuracy

MED Median tracking accuracy

FAILED Tracking failed

R200

48

Setting Voxel Resolution in Your Application

• Use PXC[M]ScenePerception:SetVoxelResolution/QueryVoxelResolution

• The PXC[M]VoxelResolution enumerator itemizes the supported voxel resolutions

Name Description

LOW_RESOLUTION The low voxel resolution. Use this resolution in a room-sized scenario (4/256m).

MED_RESOLUTION The median voxel resolution. Use this resolution in a table-top-sized scenario (2/256m).

HIGH_RESOLUTION The high voxel resolution. Use this resolution in a object-sized scenario (1/256m).

R200

49

3D Scanning Tracking Alerts

• Use PXC3DScan::AlertEvent for 3D Scanning tracking alerts

Name Description

ALERT_IN_RANGE The scanning object is in the right range

ALERT_TOO_CLOSE The scanning object is too close to the camera. Prompt the user to move the object away from the camera

ALERT_TOO_FAR The scanning object is too far away from the camera. Prompt the user to move the object closer

ALERT_TRACKING The scanning object is in good tracking

ALERT_LOST_TRACKING Lost tracking on the scanning object

R200

50

Agenda

• F200 Short Range Camera

• F200 UX Guidelines for Laptop and All-In-One Experiences

• F200 SDK tips

• R200 Long Range Camera

• R200 UX Guidelines for Tablet and Two-In-One Experiences

• R200 SDK tips

• Summary and Q&A

F200

R200

51

Summary and Next Steps

• Understand what the camera sees and give the user effective feedback

• Choose the right technology: Know when and when not to use gestures

• Understand your users and do iterative usability testing!

• Check out the UX guidelines and videos and send us feedback

- software.intel.com/articles/realsense-ux-design-guidelines

R200F200

52

Day 1 Prize

Completing an Online Session Evaluation by 10am Tomorrow Automatically Enters You in a Drawing to Win!

Day 2 PrizeIntel® Compute Stick (20)

Copies of the complete sweepstakes rules are available at the Info Desk

Winners will be notified by email

You will receive an email with a link to the online evaluation prior to the end of this session.

Day 3 PrizeMicrosoft* Surface* 3 (6) Dell Venue 10 7000 Series (4)

53

Q&A

54

What will you develop?

55

Legal Notices and DisclaimersIntel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.

No computer system can be absolutely secure.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.

Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties. A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

Intel, RealSense, and the Intel logo are trademarks of Intel Corporation in the United States and other countries.

*Other names and brands may be claimed as the property of others.

© 2015 Intel Corporation.

56

Risk FactorsThe above statements and any others in this document that refer to plans and expectations for the second quarter, the year and the future are forward-looking statements that involve a number of risks and uncertainties. Words such as "anticipates," "expects," "intends," "plans," "believes," "seeks," "estimates," "may," "will," "should" and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel's actual results, and variances from Intel's current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations. Demand for Intel's products is highly variable and could differ from expectations due to factors including changes in business and economic conditions; consumer confidence or income levels; the introduction, availability and market acceptance of Intel's products, products used together with Intel products and competitors' products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Intel's gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing quality/yields. Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing expenses, and Intel's ability to respond quickly to technological developments and to introduce new products or incorporate new features into existing products, which may result in restructuring and asset impairment charges. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Results may also be affected by the formal or informal imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice. Intel operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term. The amount, timing and execution of Intel's stock repurchase program could be affected by changes in Intel's priorities for the use of cash, such as operational spending, capital spending, acquisitions, and as a result of changes to Intel's cash flows or changes in tax laws. Product defects or errata (deviations from published specifications) may adversely impact our expenses, revenues and reputation. Intel's results could be affected by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel's ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. Intel's results may be affected by the timing of closing of acquisitions, divestitures and other significant transactions. A detailed discussion of these and other factors that could affect Intel's results is included in Intel's SEC filings, including the company's most recent reports on Form 10-Q, Form 10-K and earnings release.

Rev. 4/14/15