Kinect Open Source Programming Secrets
Kinect Open Source Programming SecretsAndrew DavisonPrince of
Songkla [email protected]
Hacking the Kinect with OpenNI, NITE, and Java
Overviewusing OpenNIusing NITEas an input sourceto other
librariesKOPS Talk (JavaOne
2012)2http://fivedots.coe.psu.ac.th/~ad/kinectEight examples(if I
have time)1. The Kinect Sensor2. Maps3. Skeletons4. Hand Points and
Gestures5. Computer Vision6. Augmented Reality7. Summary
1. The Kinect SensorKOPS Talk (JavaOne 2012)3used for 3D depth
sensingCMOS color sensor (for RGB imaging)motorized tilting
base(up/down ~30)three microphones(downward facing)IR light
sourceCMOS IR sensorstatus LEDthree-axisaccelerometerone
microphone(downward facing)2. Depth, Image, IR MapsKOPS Talk
(JavaOne 2012)4
The IR light emits a fixed pattern of spots (randomly
distributed)Map GenerationKOPS Talk (JavaOne 2012)5Depth
GeneratorDepthMaprepaintread mapwait on context updateinitialize
contextand
generator(s)ImageGeneratorImageMapIRGeneratorIRMapORORcameraimageInitializationContext
context = new Context(); License license = new
License("PrimeSense",
"0KOIk2JeIBYClPWVnMoRKn5cdY4=");context.addLicense(license);
DepthGenerator depthGen =
DepthGenerator.create(context);MapOutputMode mapMode = new
MapOutputMode(640, 480, 30); // xRes, yRes,
FPSdepthGen.setMapOutputMode(mapMode);
context.setGlobalMirror(true);context.startGeneratingAll(); KOPS
Talk (JavaOne 2012)6Wait, Read, Repaint Loop while (isRunning) {
try { context.waitAnyUpdateAll(); } catch(StatusException e) {
System.out.println(e); System.exit(1); } updateDepthImage(); // my
code repaint(); } try { // close down context.stopGeneratingAll();
} catch (StatusException e) {} context.release();
KOPS Talk (JavaOne 2012)7
OpenNI GeneratorsKOPS Talk (JavaOne 2012)8primarily used by
middleware(e.g. NITE)2.1. Transforming the UserKOPS Talk (JavaOne
2012)9
laterThe transformation is applied to all of the camera
image,but transparent parts of the changed image aren't
visible.KOPS Talk (JavaOne 2012)10
cycle through blurringTime-based changesCenter-of-mass (CoM)
based changes
change stayscentered onuser
Building the SceneKOPS Talk (JavaOne 2012)11Depth
GeneratorImageGeneratorset alternativeviewpointfor real
worldcoordinates to 2D(projective) mapping
UserGeneratorSceneMetaData
scene map(user ID map)
removebackgroundcameraimage
CoMs ofuser IDs(optional)Java 2D
filterOphttp://www.jhlabs.com/ip/filters/Panel ExecutionKOPS Talk
(JavaOne 2012)12repaintupdatewait on context updateinitialize
contextand generator(s)remove backgroundof camera image;apply
filterOp (optionally based ontime and CoM of user)draw background
imagedraw camera image;
3. OpenNI SkeletonThere are 15 joints (and 9 more that are not
currently initialized by the skeleton capability).KOPS Talk
(JavaOne 2012)13On-screen view,with mirroringenabled3.1. Heads and
BackgroundsKOPS Talk (JavaOne 2012)14
switch heads and backgrounds(via keyboard or dialog
selection)
Generators (and Capabilities)KOPS Talk (JavaOne 2012)15Depth
GeneratorImageGeneratorset alternativeviewpointfor real
worldcoordinates to 2D(projective) mapping
UserGeneratorSkeletonCapabilityPoseCapabilitySceneMetaDatalistenersfor
varioususer events::
camera imagescene map(user ID map)
Rotating the Head ImageKOPS Talk (JavaOne 2012)16Calculated
using the head and neckcoordinates.3.2. Viewing Users in 3DKOPS
Talk (JavaOne 2012)17
joints and limbsfor eachuserrendered using Java 3DCreating a
SkeletonKOPS Talk (JavaOne 2012)18Depth Generatorfor real
worldcoord to 2D(projective) mapping
UserGeneratorSkeletonCapabilityPoseCapabilitypsi
posedetectedcalibrationcompletednew userlost useruser has
exiteduser hasreenteredcalibrationstartingadd aSkeletonobject tothe
3D sceneOverviewusing OpenNIusing NITEas an input sourceto other
librariesKOPS Talk (JavaOne 2012)191. The Kinect Sensor2. Maps3.
Skeletons4. Hand Points and Gestures5. Computer Vision6. Augmented
Reality7. Summary4. NITE Hand Points and GesturesNITE gestures are
derived from a stream of hand points which record how a hand moves
through space over time.
Gesture detectors are sometimes called point listeners (or point
controls) since they analyze the hand points stream looking for
gestures.
KOPS Talk (JavaOne 2012)20NITE Gesture ClassesKOPS Talk (JavaOne
2012)21
4.1. Gesture GUIs (GGUIs)KOPS Talk (JavaOne 2012)22
Controlling the GGUIsKOPS Talk (JavaOne 2012)23Depth
GeneratorImageGeneratorfor coords mapping PointControlnew hand
point (P1)for camera
imageHandsGeneratorGestureGeneratorSessionManagerhand point moved
(P2)no active hand points (P3)session ended (S)update
GGUIdeactivate GGUIOverviewusing OpenNIusing NITEas an input
sourceto other librariesKOPS Talk (JavaOne 2012)241. The Kinect
Sensor2. Maps3. Skeletons4. Hand Points and Gestures5. Computer
Vision6. Augmented Reality7. Summary
5. Computer Vision and the KinectUse the Kinect sensor as input
to JavaCV (a Java binding for OpenCV)input can be RGB and/or depth
reading
Current and past (student) projects:motion detectionface
detection & recognitioneye trackingKOPS Talk (JavaOne
2012)25
http://opencv.willowgarage.com5.1. Hand RecognitionKOPS Talk
(JavaOne 2012)26
From OpenNI to OpenCVKOPS Talk (JavaOne 2012)27Depth
GeneratorDepthMapthreshold
smooth
largest contour
KOPS Talk (JavaOne 2012)28
start ptend ptdepth ptdepthconvex hulllargest contourconvexity
defectseach defect is a tuple:(start pt, end pt,depth pt,
depth)KOPS Talk (JavaOne 2012)29
start ptend ptdepth ptdepthsimplifiedconvexity defectsremove
defects with:shallow depthssmall angles betweenthe start and end
ptsFinger LabelingKOPS Talk (JavaOne 2012)30
center of gravity (cog)angle tothe horizontal
rotate hand image to matchpredefined hand
'parameters'thumbindexmiddleringlittlefragile, hacky
Uses?KOPS Talk (JavaOne 2012)31movefinger rotatezoom/pinchdouble
tapmulti slidewaveholdpick & dropcluster & movezoom
spothold & taprotate twocoverhide6. Augmented Reality (AR)A
combination of a real scene and virtual elements generated by the
computer.
Uses the Kinect camera as input to NyARToolkitKOPS Talk (JavaOne
2012)32
http://nyatla.jp/nyartoolkit/wp/?page_id=198Using NyARToolkit
and Java 3DKOPS Talk (JavaOne 2012)33
search forHiro markerusing NyARToolkitcalculateits position and
orientationapply position and orientationto marker modelrender
modelin front of thevideo frameusing Java 3Dvideo streamfrom the
Kinect
ImageGenerator
The User's ViewpointKOPS Talk (JavaOne 2012)34
background
+ Z+ Y+ Xmodel7. SummaryOpenNIRGB, depth, IR mapsskeletal
tracking, scene/user IDs
NITEhand points, hand gestures
7 Examplesaligned mapsuser transformationheads &
backgrounds3D usersgesture GUIshand recognitionaugmented reality
(AR)KOPS Talk (JavaOne 2012)35Topics not DiscussedTilt motor, LED,
accelerometereasy to add using LibusbJava
Audio source input, speech recognition, beamformingusing MS
Kinect audio driver, JSAPI, TalkingJava,
Motion detection, face recognition, eye trackingMultiple-marker
ARGames Kinect breakout
KOPS Talk (JavaOne 2012)36
rightleftpositivenegativeAll the code, draft chapters,and extra
examples,are available at my website:
KOPS Talk (JavaOne 2012)37
http://fivedots.coe.psu.ac.th/~ad/kinect/More Information
One last example...KOPS Talk (JavaOne 2012)38
8. Where I work...I'm an Ajarn (lecturer) in the Department of
Computer Engineering (CoE), Faculty of Engineering,Prince of
Songkla University (PSU)
I'm based at the Hat Yai campus, Songkhla province, in the south
of Thailand.KOPS Talk (JavaOne 2012)39
1st university in Southern Thailand, est. 19675 campuses39,000
students (2012)Hat YaiTrangPattaniPhuketSurat Thani
PSU at a Glance...
KOPS Talk (JavaOne 2012)41Prince of Songkla University(PSU), Hat
Yai campusDepartment of Computer Engineering (CoE)Faculty of
Engineering
KOPS Talk (JavaOne 2012)42
KOPS Talk (JavaOne 2012)43
Department of Computer Engineering (CoE)PSU Academically...A
National Research University (NRU)Ranked 4th in the Nation in terms
of publicationsResearch strong points include:Natural
rubberBiodiesel and energySea food and Halal foodMarine
sciencesNanotechnologyPeace studies
KOPS Talk (JavaOne 2012)44Center for Network Research
(CNR)Wireless Sensor Network (WSN) ANW Informatics Laboratory
Intelligent System (iSys) Research Team Computer Control and
Robotics Computer System Design and VLSI Design KOPS Talk (JavaOne
2012)45CoE Research Groups and Interests
