Kinect for Windows Quickstart Series

KINECT FOR WINDOWS

QUICKSTART SERIES

KINECT FOR WINDOWS

INSTALLING &USING KINECT

KINECT FOR WINDOWS

INTRODUCTION

• Designed to get you started• Assume you have some programming

experience• Managed API - Concepts work in both

VB/C#

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

KINECT SENSORSIR Emitter Color Sensor

IR Depth Sensor

Tilt Motor

Microphone Array

KINECT FOR WINDOWS

GET STARTED

• http://kinectforwindows.org • Order Kinect Hardware• Download Kinect SDK

KINECT FOR WINDOWS

INSTALLING AND USING KINECT

DEMO

KINECT FOR WINDOWS

SETTING UP YOUR DEVELOPMENTENVIRONMENT

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development

Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

KINECT API BASICS

• Manage Kinect state • Connected• Enable Color, Depth, Skeleton• Start Kinect

• Get Data• Events - AllFramesReady• Polling – OpenNextFrame

KINECT FOR WINDOWS

RESOURCES

• Install Kinect Explorer• KinectWpfViewers

• Coding4Fun Toolkit• Skeletal scaling

KINECT FOR WINDOWS

SETTING UP YOUR DEV ENVIRONMENT

DEMO

KINECT FOR WINDOWS

CAMERA FUNDAMENTALS

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

CAMERA DATA

• Events return ImageFrame• PixelDataLength• FrameNumber• Timestamp• Dimensions: Height, Width

• Use AllFramesReady event to synchronize

KINECT FOR WINDOWS

RESOLUTIONS

• Color• 12 FPS: 1280X960 RGB• 15 FPS: Raw YUV 640x480• 30 FPS: 640x480

• Depth • 30 FPS: 80x60, 320x240, 640x480

KINECT FOR WINDOWS

CAMERA FUNDAMENTALS

DEMO

KINECT FOR WINDOWS

WORKING WITHDEPTH DATA

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

DEPTH DATA• Returns the distance and player for every

pixel• Ex: 320x240 = 76,800 pixels

• Distance• Distance in mm from Kinect ex: 2,000mm (6.56

feet)

• Player• 1-6 players

KINECT FOR WINDOWS

MODE

Mode Depth & Player Center Hip Joint Other 19 Joints

Default Yes Yes Yes

Near Yes Yes No, for v1.0

KINECT FOR WINDOWS

DISTANCES1.3’ 2.6’ 9.8’ 13.1’ 26.2’

.4 .8 3 4 8

DefaultMode

Near Mode

Feet

Meters

KINECT FOR WINDOWS

FORMULAS

• Distance Formulaint depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth;

• Player Formulaint player = depthPoint & DepthImageFrame.PlayerIndexBitmask;

KINECT FOR WINDOWS

WORKING WITH DEPTH DATA

DEMO

KINECT FOR WINDOWS

SKELETAL TRACKINGFUNDAMENTALS

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

SKELETON DATA

KINECT FOR WINDOWS

SKELETAL JOINTS• Each player with set of <x, y, z> joints in meters• Each joint has associated state

• Tracked, Not tracked, or Inferred

• Inferred - Occluded, clipped, or low confidence joints• Use TransformSmoothParameters to smooth joint data to

reduce jitter

KINECT FOR WINDOWS

SKELETAL TRACKING• Maximum two players tracked at once

• Six player proposals per Kinect

• Each skeleton has a unique identifier - TrackingID • SkeletonStream

• Set AppChoosesSkeletons to false• Use ChooseSkeletons method to choose which skeletons to

track

KINECT FOR WINDOWS

SKELETAL TRACKING

DEMO

KINECT FOR WINDOWS

AUDIOFUNDAMENTALS

KINECT FOR WINDOWS

QUICKSTART SERIES

• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals

KINECT FOR WINDOWS

SOUND POSITION• Sound Source Angle – the angle and confidence level of

where audio is coming from• Beam Angle – The angle used to record audio that you can

set as a “directional microphone”

-50 0 +50

KINECT FOR WINDOWS

RECORDING

• KinectAudioSource audio buffer• Use Echo Cancellation to not record PC

sound

KINECT FOR WINDOWS

SPEECH RECOGNITION

• Grammar – What we are listening for• Code – GrammarBuilder, Choices

• Note: Ensure AutomaticGainControl = false

KINECT FOR WINDOWS

GRAMMAR<!-- Confirmation_Yes._value: string ["Yes"] --><rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes please </example> <one-of> <item> yes </item> <item> yeah </item> <item> yep </item> <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag>

var grammar = new Choices();grammar.Add(“yes please");grammar.Add(“yes");grammar.Add(“yeah");grammar.Add(“ok");

KINECT FOR WINDOWS

MULTI-MODAL FEEDBACK• Engagement model–Mic indicator for speech-enabled menus– Keyword to engage

• Feedback and confirmation

KINECT FOR WINDOWS

AUDIO

DEMO