Embed Size (px)
Citation preview
LECTURE 5: OPENFRAMEWORKS AND SOLI
COMP 4026 – Advanced HCI Semester 5 - 2016
Mark Billinghurst University of South Australia
August 25th 2016
RECAP
Advanced Interface Technology • Wearable Computers • Augmented Reality • Virtual Reality • Invisible Interfaces • Environment Sensing • Physiological Sensing
Class Project 1. Pick Advanced Technology 2. Brainstorm use case 3. Develop conceptual design 4. Prototype interface/experience design 5. Conduct user evaluation 6. Repeat steps 3-5 7. Write report
Wearable Computing ▪ Computer on the body that is: ▪ Always on ▪ Always accessible ▪ Always connected
▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
Wearable Attributes
▪ fafds
Google Glass
View Through Google Glass
1977 – Star Wars
Augmented Reality Definition
• Defining Characteristics [Azuma 97] • Combines Real and Virtual Images
• Both can be seen at the same time • Interactive in real-time
• The virtual content can be interacted with • Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
Virtual Reality
• Immersive VR • Head mounted display, gloves • Separation from the real world
AR vs VR
Early Examples
• Interaction without devices: • BodySpace [Strachan 2007]: Functions to body position • Abracadabra [Harrison 2007]: Magnets on finger tips • GesturePad [Rekimoto 2001]: Capacitive sensing in clothing
• Palm-based Interaction • Haptic Hand [Kohli 2005]: Using non-dominant hand in VR • Sixth Sense [Mistry 2009]: Projection on hand • Brainy Hand [Tamaki 2009]: Head worn projector/camera
ImaginaryPhone
• Gustafson, S., Holz, C., & Baudisch, P. [2011]
Transfer Learning
Invisible Interfaces – Gestures in Space
• Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] • Using a non-dominant hand stabilized interface.
Project Soli
• Using Radar to support free-hand spatial input
Google Tango • Tablet based system
• Android OS • Multiple sensors
• RGBD Sensor • IR Structured light • Inertial sensors
• High end graphics • Nvidia tegra chip
Physiological Sensors • Sensing user state
• Body worn devices
• Multiple possible sensors • Physical activity • Eye tracking, gaze • Heart rate • GSR • Breathing • Etc
Tobii Eye Tracker
• Wearable eye tracking system • Natural data capture • Scene camera capture • Recording/streaming eye gaze, 60 Hz sampling
OPENFRAMEWORKS
OpenFrameworks (www.openframeworks.cc)
• Open source toolkit designed for creative coding • Developed by Z. Lieberman, T. Watson and A. Castro
• Framework – collection of libraries • Written in C++
• More powerful than Processing, but more complicated
• Must use IDE for development • Xcode, Visual Studio, Code::Blocks
• Runs on Mac, Windows, Linux platforms
Why use oF instead of Processing
• Speed • Accessibility of low level information • Debugger • C++ • Version control • Cross Platform
OpenFrameworks vs. Processing • Making project visible on Internet - Processing • Make a project with lots of 3D graphics - OpenFrameworks • Make a project for lots of different computers/OS – Processing • Make a project using an external library like the OpenCV
computer vision library – OpenFrameworks • Make a project that interfaces with the Arduino board - Either
OpenFrameworks Installation
• addons: added libraries from user community. Must be explicitly included in programs using them
• apps: store your programs here. Also contains example code. • libs: where the core libraries of OpenFrameworks are stored.
Also contains core openFrameworks folder
Building an Application
testApp.cpp
Application Structure
Typical Function Types • setup( )
• load assets • Initialize values • Initialize addons or components
• update( ) • calculations • increment video frames
• draw( ) • draw shapes/images/videos • use GLSL Shaders
Classes in C++ • C++ classes comprise of two files. It helps to think of these two
files as a recipe. • The header file (.h) is like the list of ingredients, and contains:
• Any preprocessor statements there to prevent multiple header definitions • Any include statements to other classes • Any class extension statements • Any variables local to the class • Prototypes of any functions to be contained in the class • Security settings of these functions and variables (e.g. public, private,
protected, etc).
• and a body file (.cpp) which is like the instructions on what to do with the ingredients and contains: • An include statement that references the .h file • All of the code to fill in the function prototypes.
Class Extending • Take one class and add functionality to it with a new class • Eg enemy class for video game !class Enemy {!! !int x, y; //position!! !.. .. !! !public void draw() {!! !//draw my picture to the screen at the proper location }!!}!
• Want to draw enemy twice – create new class !//on a "DoubleEnemy.h" file!!class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}!!{!! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file!!}; // note the ";" at the end of the class statement!
Pass by Value vs. by Reference
• void functn(int num) – pass by value • void functn(<class> test) – pass by reference
• sends address of where class stored
• use pointers to pass arrays back and forth through functions
int num = 5; value
stores address of variable value
void setup() { int num = 1; addOne(num); print(num); } void addOne(int num) { num++; }
class Test { int num=0; } void setup() { Test test = new Test(); test.num=1; addOne(test); print(test.num); } void addOne(Test test) { test.num++; }
Pass by Value Pass by Reference
& and * • In C++ you need to explicitly state whether you are passing
something by value or by reference. • Use & (referencing) and * (dereferencing) symbols
• the & symbol is used to acquire the memory address of a variable or function b=1;!a = &b; // a now equal to memory address of b!a++; // memory address of b + 1!*a++; // value a +1 (increments b as well)!
Example • What does this code do? ! ! !int x;!! ! !int *ptr;!!! ! !x=5;!! ! !ptr = &x;!! ! !*ptr = 10;!
2D Image Functions
• Colors • ofFill(); • ofCircle(100,400,80); • ofSetHexColor(0x000000); ofSetColor(255,0,0,127);
• Primitives • ofCircle(100,400,80); • ofRect(400,350,100,100); • ofLine(600,300,800, 250); • ofDrawBitmapString("rectangles", 275,500);
OpenFrameworks vs. Processing
OpenFrameworks Processing
Circle Grid
• Setting the size of the window. • Processing:
• size(800, 600, OPENGL);
• openFrameworks: • ofSetupOpenGL(&window, 800, 600, OF_WINDOW); • function is called in main() in the file main.cpp.
Circle Grid
• Declaring Variables • Processing:
• Declare the variables you need right after you import the libraries you need.
• openFrameworks: • Declare variables in the file testApp.h, after the line void windowResized(int w, int h);.
Circle Grid • Background Color • Processing:
• background(0); will set the background of your sketch to black. You need to call the function inside draw() to draw the background each frame.
• openFrameworks: • Call ofBackground(0, 0, 0); once inside the setup() method.
openFrameworks will draw the background automatically each frame. You can disable this by calling ofSetBackgroundAuto(false) within setup() in the file testApp.cpp.
Circle Grid • Drawing Circles • Processing:
• after you have set the stroke and fill, use ellipse(50, 50, 20, 20); to draw a circle with a diameter of 20 at (50, 50).
• openFrameworks: • you can use ofCircle(50, 50, 10); to draw the same circle. You could also use
ofEllipse(50, 50, 20, 20);. If you want to draw a circle with a stroke you will need to call the function to draw the circle two times. Once for the fill and once for the stroke.
ofSetColor(255, 255, 255);!! !ofFill();!! !ofCircle(50, 50, 20);!
Graphics Demo - graphicsExample
• setup( ) method • draw( ) method
Drawing Polygons • Must begin and end a shape
• ofVertex, ofCurveVertex, ofBezierVertex
!ofBeginShape();!! ! !ofVertex(200,135);!! ! !ofVertex(15,135);!! ! !ofVertex(165,25);!! ! !ofVertex(105,200);!! ! !ofVertex(50,25);!!ofEndShape();!•
polygonExample
ofBoxDemo
Importing Libraries • Large set of oF addon libraries (> 450)
• http://ofxaddons.com/
• Just download library to addons directory, then include library #include “myLibrary.h”!
• Sample libraries • ofxOpenCv • ofxVectorGraphics • ofxVectorMath • ofxNetwork • ofvOsc
Examples
• VectorGraphicsExample • 3DModelLoaderExample
• Loading 3D models
• assimpleExample • 3D animation
• openCVExample • Hand segmentation
Projects
OpenFrameworks Showcase
https://www.youtube.com/watch?v=6u6IDorMKAs
Piano Stairs +openFrameworks
Nike + Paint With Your Feet +openFrameworks +GPS
Resources
• Main website • http://www.openframeworks.cc/
• Forum • http://forum.openframeworks.cc/
• Addons • http://ofxaddons.com/
• Roxlu’s website • http://www.roxlu.com/
PROJECT SOLI
Overview
• Soli uses radar to detect fine scale finger motion
Project Soli Overview
https://www.youtube.com/watch?v=0QNiZfSsPc0
Sensing Modalities
• asdfas
Radar Fundamentals
• Radar tracks moving objects • Measures response to Radar waves sent from transmitter
Radar Reflections from Hand
• Multiple reflection points
Signal Processing
• Signal received combination of slow and fast time
Signal Processing
Processing Pipeline
• From raw hand motion to recognized gestures
Soli Hardware
• Miniaturized Radar
Signs vs. Actions
• Soli recognizes hand actions
Virtual Tools
• Use virtual tool metaphor • Change with distance
Types of Virtual Tools
• asdasf
Basic Gesture Movement
• Easily recognize distinctive gesture motions
• > 90% accuracy on filtered results (Bayesian Filter)
Recognition Results
Applications
• Gesture interaction with objects • Smart watch, car console
• Gesture interaction with environment • Furniture, walls
• Other applications • Material recognition, Gaming, Object scanning
Developers Showcase
https://www.youtube.com/watch?v=H41A_IWZwZI
Soli Enabled Watch
https://www.youtube.com/watch?v=pagDaQw-Tcw
Soli Enhanced Environment
https://www.youtube.com/watch?v=jNxvugxAoaY
Future Research • Radar Sensing
• Radar clutter, multi-path reflections, occlusion, etc
• Machine Learning • New gesture recognition approaches
• Human Factors • Measuring human performance abilities, requirements
• Interaction Design • New interaction modalities, metaphors
Background Reading
Lien, Jaime, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. "Soli: ubiquitous gesture sensing with millimeter wave radar." ACM Transactions on Graphics (TOG) 35, no. 4 (2016): 142.
