1

EEC-492/592EEC-492/592Kinect Application Kinect Application

DevelopmentDevelopmentLecture 2Lecture 2

Wenbing ZhaoWenbing Zhao

wenbing@ieee.orgwenbing@ieee.org

22

Outline

Reminder No class next Monday: Martin Luther King Day

Components of Kinect sensor Kinect for Windows SDK Building first Kinect application

33

Components of Kinect for Windows Color camera Infrared (IR) emitter IR depth sensor Tilt motor Microphone array LED

44

The Color Camera

Red, blue, green color images (RGB)

Frame rate: 30 FPS at 640x480 12 FPS at 1280x960

Viewing range: 43 degrees vertical by

57 degrees horizontal

55

IR Emitter & IR Depth Sensor IR emitter emits IR light in a pseudo-random dot pattern Dotted light reflects off different objects IR depth sensor reads reflected light => depth info Depth stream resolution

640x480 320x240 80x60

66

Depth Data Processing

77

Tilt Motor Used to change the camera and sensor’s angles to

get the correct position of the human skeleton Can tilt vertically (upwards/downwards) up to 27

degrees

88

Microphone Array 4 different microphones: locate the direction of

the audio wave, not just capture the sound Enhanced noise suppression, echo cancellation,

beam-forming

99

LED

Placed between camera and IR emitter Used for indicating the status of the Kinect Green indicates Kinect drivers loaded

properly

1010

Kinect for Xbox vs. Kinect for Windows Kinect for Windows

Near mode: as close as 40cm away from Kinect Can be used for commercial applications Can be used in a Win7 virtual machine: HyperV,

VMWare, Parallels For both

Track motion up to 12 feet (4 meters) away Identical resolution

1111

Kinect for Windows SDK Supported operating systems

Windows 7, Windows 8, Windows Embedded 7 Hardware requirements

Dual core 2.66 GHz or faster CPU Dedicated USB 2.0 bus 2GB or higher RAM

Software requirement Microsoft Visual Studio 2010 or higher Kinect for Windows SDK: current version 1.8

1212

Kinect for Windows SDK

1313

Kinect for Windows SDK

1414

How Applications Interact with Kinect

1515

Classification of Kinect SDK APIs

1616

Kinect Driver The Kinect driver controls the camera, depth

sensor, audio microphone array, and motion Data passes between the sensor and the app in

the form of data streams Color data stream Depth data stream Audio data stream Infrared data stream (for low light environment) Accelerameter data

1717

The Near Mode Track a human upper body as close as 40 cm Only Kinect for Windows supports the near mode

1818

Tracking Human Skeleton and Joint Movement Kinect SDK does the magic to

extract the skeleton and joint positions from the data streams received

Can track up to two users at the same time with full joint positions

Each skeleton consists of 20 joints

1919

Building First Kinect App Modified KinectInfoBox app Steps

Create a new project: C# WPF application Adding Kinect reference Draw the GUI Modify MainWindow.xaml Adding code

2020

Creating a New Project

2121

Adding Kinect Libraries

2222

Create User Interface

2323

Modify MainWindow.xaml

2424

Adding Code (MainWindow.xaml.cs) Setting up name space

using Microsoft.Kinect;

Setting up state variable for the Kinect sensor An instance of Microsoft.Kinect.KinectSensor class It represents the complete runtime pipeline for the Kinect sensor

during life cycle of the apppublic partial class MainWindow : Window

{KinectSensor sensor;

// remaining code goes here

}

2525

Adding Code private void WindowLoaded(object sender, RoutedEventArgs e) {

if (KinectSensor.KinectSensors.Count > 0) {

this.sensor = KinectSensor.KinectSensors[0];

if (this.sensor != null && !this.sensor.IsRunning) {

this.sensor.Start();

displayInfo();

}

}

else {

MessageBox.Show("No device is connected with system!");

this.Close();

}

}

private void WindowClosing(object sender, System.ComponentModel.CancelEventArgs e) {

if (this.sensor != null && this.sensor.IsRunning) {

this.sensor.Stop();

}

2626

Adding Code

Double click “Increase” button, the template for the button1_Click() method will be automatically created

Double click “Decrease” button, the template for the button2_Click() method will be automatically created

private void displayInfo() { this.textBlock1.Text = this.sensor.DeviceConnectionId; this.textBlock2.Text = this.sensor.Status.ToString(); this.textBlock3.Text = this.sensor.ElevationAngle.ToString(); }

private void button1_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle + 1; displayInfo(); }

private void button2_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle - 1; displayInfo(); }

2727

More In-depth Stuff Starting Kinect sensor: sensor.Start() In the Start() method

First check the status of Kinect, if no sensor connected, runtime will throw an InvalidOperationException object with the KinectNotReady message

Init the sensor if connected

Init options None: default option UseDepthAndPlayerIndex UseColor UseSkeletonTracking UseDepth UseAudio

2828

More In-depth Stuff Open data streams

ColorImageStream: this.sensor.ColorStream.Enable(); DepthImageStream: this.sensor.ColorStream.Enable(); SkeletonStream: this.sensor.SkeletonStream.Enable();

Stopping the Kinect sensor this.sensor.Stop(); Stops the color, depth, and skeleton data stream Stops audio source, if open Kills all threads spawned Shuts down Kinect sensor and sets init option to None