Kinect on Android Pandaboard

ARDIC 2010ARDIC 2010KURUMSAL SUNUMKURUMSAL SUNUM

KINECT ON ANDROID KINECT ON ANDROID PANDABOARDPANDABOARD

ÜMİT UZUNÜMİT UZUNSenior Software Engineer

1ARDIC A.S CONFIDENTIAL

Agenda

2© ARDIC, Inc. 2013

• Introduce most known Kinect sensor device• Introduce PrimeSense technology behind

Kinect• Introduce OpenNI library and architecture

leaded by PrimeSense• Introduce NITE middleware library developed

by PrimeSense• Show Demo

Kinect Sensor


Kinect Sensor


The Kinect Sensor was originally

intended to be a motion sensing input

device for the Xbox 360,

allowing the user to control games via

gestures and spoken commands.

Kinect Teardown


Kinect IR Depth Sensor


• The IR light source is a laser that projects an irregular pattern of dots.

• The monochrome CMOS IR sensor measures the dots' “time of flight” after they reflect off objects in the scene.

• In the same way as sonar, knowing how long the light takes to return, the Kinect can calculate how far away an object is located.

• IR is used to reduce the problem of ambient light affecting the calculations.

IR Projected Dots


Kinect Development Software


There are four main Kinect development libraries:

• OpenKinect's libfreenect (http://openkinect.org/wiki/Main_Page)

• CLNUI (http://codelaboratories.com/nui/)• OpenNI (http://www.openni.org/)• Microsoft's Kinect for Windows SDK (

http://www.microsoft.com/en-us/kinectforwindows/)

http://openkinect.org/wiki/Main_Page

http://codelaboratories.com/nui/

http://www.openni.org/

http://www.microsoft.com/en-us/kinectforwindows/

Kinect Development Software


• OpenKinect's libfreenect is derived from a reverse-engineered/hacked Kinect driver, which works across Windows, Linux and MacOS X.

• CLNUI (http://codelaboratories.com/nui/) is aimed at Windows only, but allows multiple Kinects to work together.

• It's really not fair to compare libfreenect or CLNUI with OpenNI, which is intended to act as an interface between a variety of sensor drivers (including a Kinect driver) and middleware for manipulating the sensor data in a higher-level manner (e.g. for recognizing user gestures as commands).

PrimeSense - Technology


• PrimeSense 3D sensing technology gives digital devices the ability to observe a scene in three dimensions.

• It translates these observations into a synchronized image stream (depth and color) – just like humans do.

• It then takes those synchronized images and translates them into information.

• We call this depth sensing, which is made possible through the cutting-edge technology embedded in primesense sensors and middleware.

PrimeSense Reference Design


PrimeSense - Sensors


• PrimeSense 3D Sensors give digital devices the gift of sight, enabling Natural Interaction between people and devices and between devices and their surroundings.

• Using cutting-edge technology, sensors are robust, accurate and affordable.

• They are used in mass-market commercial devices. With more options than ever – mid-range, close range, a peripheral and an embedded sensor.

PrimeSense - NITE


• PrimeSense 3D sensing technology is a layered solution, combining both hardware and software.

• The PrimeSense NiTE™ perception algorithms layer is the most robust and advanced 3D middleware available in the market.

• It provides the application with a clear user-control API, whether it is hand-based control or a full-body control.

• The algorithms utilize the depth, color, IR and audio information received from the hardware device, which enable them to perform functions such as hand locating and tracking; user segmentation; user skeleton joint tracking; and more.

PrimeSense – Ecosystem


PrimeSense - Capri


PrimeSense – Capri


• Take the 3D sensor inside the Microsoft Kinect, shrink it down to a tenth of its original size and add a bunch of mobile capabilities, and you have yourself PrimeSense’s latest conquest, better known as Capri.

• The company, which is the brains behind the Kinect, has been openly working on bringing a tiny-yet-advanced 3D experience to tablets, televisions and smartphones for quite some time now.

OpenNI Layers


OpenNI Kinect Driver


• OpenNI's Kinect driver is called SensorKinect, and is derived from open sourced (LGPL) code from PrimeSense, the company behind Kinect's depth sensor technology.

• One advantage of libfreenect is API support for controlling the Kinect's tilt motor, status light, and accelerometer, which is missing from the SensorKinect driver.

KinectSDK vs OpenNI


• The main area where the SDK wins over OpenNI is audio. Although OpenNI contains audio-related APIs, not fully implemented.

• Other pluses for Microsoft's SDK are its extensive documentation and examples, and its ease of installation on Windows 7.

• The main drawback of the SDK is that it only works on Windows 7, not XP or any non-windows platform.

• The KinectSDK is free, but limited to non-commercial purposes.

NITE – Natural Interaction Middleware


• An important reason for using OpenNI is its support for middleware.

• The NITE library understands different hand movements as gesture types based on how 'hand points' change over time.

• NITE gestures include pushing, swiping, holding steady (i.e. no hand movement), waving, and hand circling.

NITE – SkeletonTracker


OpenNI SDK – Production Nodes


OpenNI SDK – Production Nodes


• The main functional element in OpenNI is the production node.

• A node may represent a sensor's functionality (e.g. depth detection, infrared, audio) or a higher-level abstraction (e.g. user positioning, hand tracking).

• An OpenNI application always starts by initializing the nodes it needs for its particular task.

OpenNI SDK – Generators




The MapGenerator subclasses typically output streams of images which are called maps in OpenNI.

The sensor-related Generator subclasses are:

• AudioGenerator: for producing an audio stream.

• DepthGenerator: for creating a stream of depth maps.

• ImageGenerator: for creating colored image maps.

• IRGenerator: for creating infrared image maps.



The middleware-related classes are:

• GesturesGenerator: for recognizing hand gestures, such as waving and swiping.

• HandsGenerator: for hand detection and tracking;

• SceneAnalyzer: for separating the foreground from the background in a scene, labeling the figures, and detecting the floor. The main output is a stream of labeled depth maps.

• UserGenerator: generates a representation of a (full or partial) body in the scene.

OpenNI SDK - Data and Metadata


OpenNI SDK - Data and Metadata


• The map generators (the depth, image, IR and scene generators) make their data available as subclasses of Map and MapMetaData.

• Metadata are data properties (as opposed to the data itself), such as map resolution.

OpenNI SDK - Listeners


OpenNI SDK - Listeners


• Generators can also interact with user programs via listeners (called callbacks in OpenNI).

• The MapGenerators tend to use StateChangedObservable listeners, while other generators (e.g. for hands and gesture recognition) use Observable<EventArgs> listeners, with different EventArgs subclasses.

OpenNI SDK - Capabilities


OpenNI SDK - Capabilities


• OpenNI is intended to work with different sensors, so the capabilities of an instantiated production node may vary. For instance, some webcams may allow an image map to be cropped.

• Each capability is represented by a subclass of CapabilityBase.

NiSimpleViewer


NiUserTracker




TEŞEKKÜRLERTHANK YOU

www.ardictech.com

© ARDIC, Inc. 2013

http://www.ardictech.com/