Download pptx - Google Glass, the GDK, and HTML5

Google Glass, the GDK, and HTML5-Based Mobile Apps

SVCC (10/11/2014)

Oswald Campesato

[email protected]

Topics in This Session

Google Glass and Glassware Glass Features (condensed summary) What is the Mirror API Some HTML5 Technologies What is the GDK PhoneGap and Google Glass What about Android Wear

Some Glass Features (briefly)

Directions (“ok glass…get directions”) Video (“ok glass…take a video”) Camera (“ok glass…take a picture”) Translate (scan text and translate) Answer gmail (voice input) Voice-activated Android apps Pictures/videos are backed up on G+ Touch pad (gesture detection in GDK):

Swipe left/right/down; tap; scroll

How to Create Glassware

Use the Glass Mirror API (lightweight/server-side)

Use Eclipse (or Android Studio) with the GDK

Use PhoneGap and add Glass-related plug-ins

Use the GDK + Mirror API to create “hybrid” apps:

https://github.com/mimming/kitty-compass

Note: Glass hybrid apps refer to GDK + Mirror API, which is different from HTML5 hybrid apps

Glass Pattern Building Blocks

UI elements:

+ Static cards (Card class is deprecated)

+ Live cards (Android service): low & high frequency

+ Immersions

Voice commands:

+ Focus on the action, not the agent

+ Decrease time from intent to action

+ Is colloquial and easy to say

+ Is general enough to apply to multiple Glassware

https://developers.google.com/glass/design/patterns



Internet Access for Glass

Go to https://glass.google.com/myglass

Enter the name of a WiFi and the password

Scan the QR code with Glass

“Okay Glass…get directions” (or whatever)

Note: GG remains tethered to your Android device even when you switch it off

https://glass.google.com/myglass

https://glass.google.com/myglass

Screencasting GG via MyGlass

1) Launch MyGlass on your Android phone

2) Tap “MyGlass” (top of device screen)

3) Tap “Screencast” (bottom of drop-down list)

4) “Swipe” left or right on your Android device

+ Swipe gestures update GG (and vice versa)

+ swipe left/forward: displays the present/future

+ swipe right/backward: displays the past

NB: Screencasting Glass-to-iOS-device works as well, but not from an iOS device to Glass

The Glass Mirror API (1)

“lightweight”: processes only on server-side Reduces battery consumption (versus GDK) Provides access to server-side Glass APIs Easy to “install” Supports static cards and timeline mgmt Support for various languages:

Python/JavaScript/Node/Java A ‘sandbox’ is available

Access requires a registered Glass device


Static cards can contain:

+ Text, HTML, media (images and videos)

+ Standard and custom menu items

+ multiple “pages” (articles)

+ auto or manual pagination

Timeline management:

+ Subscribe to timeline notifications

+ LBS (location-based services)


Disadvantages of using the Mirror API:

1) Internet access is required:

+ apps only work when in WiFi range

2) limited number of API calls (1000/day)

3) no voice trigger

4) No real-time (no animation)

5) Time delay receiving server response(s)

6) No device access:

+ Camera/Compass/GPS

NB: infrequent location updates

Glass Mirror API: what happens?

Step #1: user authenticates Step #2: Glass stores credentials (server-

side) Step #3: Glass inserts static card in

timeline Step #4: Glass synchronizes timeline Step #5: user sees static card on Glass

Glass Mirror API Quick Start Samples:

https://developers.google.com/glass/develop/mirror/quickstart/index






The GDK: Glass Development Kit

An “add-on” to the Android SDK The GDK provides Android-based APIs provides Glass-specific APIs GDK-based Glassware runs on Glass itself Use standard Android development tools ICS (4.0.3) or higher Simple installation: adb install abc.apk

Supports Timeline, cards, and immersions Support for sensors, Menu, and UI Support for touch pad and gestures Support for media (sound/camera/voice)

https://developers.google.com/glass

The +/- of the Glass GDK Advantages:

+ GDK-based apps work offline (Internet not required)

+ Unlimited access to GDK-based apps

+ Support for voice trigger

+ Real-time support (smoother experience)

+ support for animation effects

+ No delay (no server access required)

+ Device access (camera/compass/etc)

Disadvantages:

+ apps can drain the battery

+ more difficult to install GDK-based apps

Glass GDK Set-up

Option #1 software installation: Eclipse (3.5 or higher?) JDK 1.6 (or higher) ADT installation

Option #2 software installation: Android Studio (complete zip file)

Enable Debug Mode on Glass

+ swipe to "Settings" -> "Device info”

+ tap to open device menu

+ select "Turn on debug"

The Android ‘adb’ Utility

Install apks on devices: adb install abc.apk

List available devices: adb devices

List apks on Glass: adb shell pm list packages –f

Launch the Glass home card:

adb shell am start -n com.google.glass.home/.timeline.MainTimelineActivity

Create/Deploy “Hello World” in Eclipse

CREATE the application:

1) Launch Eclipse or Android Studio

2) File > New > Android Application Application

3) specify “HelloWorld” for “Application Name”

4) API 15 (or API 19) for all SDK-related options

5) Click “Next” 4 times and click “Finish” once

DEPLOY the app to a device: Run > Run > Android Application > OK

ELAPSED TIME: 60 seconds

Android Project Structure

The top-level project directory contains the following:

./src: Java code (usually in a package)

./res/values: XML files with text strings and layout styles

./res/layout: XML files for layouts (such as activity classes)

./res/drawable: static images (*)

./res/assets/www: HTML-related assets

./libs: JAR files

./gen: generated code

./bin: “apk” file (and other stuff)

AndroidManifest.xml is the main “control” file

NB: assets are “mapped” to properties in R.java

(*) drawable-hdpi, drawable-mdpi, and so forth

Android App Components Activity: essentially a screen in an application

Intent: launch Activity (provides other “stuff”)

Service: background operations (no UI), such as downloads

Broadcast receiver (listens to system-wide events)

Content provider: manages shared app data, such as contacts

Also defined in AndroidManifest.xml

The PhoneGap Way (HTML5)

Install node and npm

Install PhoneGap: npm install –g phonegap

Create an app: cordova create test1

Go to root directory: cd test1

Add Android: cordova platform add android

cordova plugin add https://github.com/aphex/cordova-glass-core

cordova plugin add https://github.com/aphex/cordova-glass-touch

cordova plugin add https://github.com/aphex/cordova-glass-gesture

Deploy Android apk: cordova run

https://github.com/aphex/cordova-glass-gesture







What is CardBuilder?

Replaces deprecated Card class with new layouts TEXT and TEXT_FIXED: text content and a footer COLUMNS and COLUMNS_FIXED: display a mosaic

or icon on the left side and text on the right side AUTHOR: an avatar, primary+secondary headings,

and body text TITLE: an image or mosaic with a name and icon ALERT and MENU: for dialogs with warning

messages and action confirmation flows improvements to GestureDetector’s scroll handling GDK samples fully support Android Studio

What is a Glass Card?

A Card is “a unit of work” (one screen)

A Card contains:

+ Main body text

+ Left-aligned footer

+ Right-aligned footer for a timestamp

+ One image for the background

Creating a (deprecated) Card (native code):

+ set the properties of the card

+ invoke Card.toView() to convert to an Android view

Glass Live Cards Native Glassware “sits” on Timeline (left-

side)

display real-time info (weather/news/etc)

frequent updates (running in background)

Similar to Android Widgets (or Service)

Display text and images (and 3D graphics)

a 'stop' option is required for live cards

Live Card Project (1) Create Android Project in Eclipse (or Android

Studio)

Specify version 15 or higher

Rename MainActivity to MenuActivity

Remove launcher/main intent filters from manifest

Add Service as the main component, which handles Live Card creation/management

Live Card Project (2) Implement Live Card Service (override 2

methods)

onStartCommand() method: called when service is started Create a live card and publish it If live card already exists, then display it

onDestroy() method: Called when service is destroyed Unpublish the live card

Glass Immersions

They run outside the Timeline

Typically consist of Android Activities

fully-customizable screens

you can design your own UI

Access to all user input

Better for interactive experiences

Touch Gestures on Glass

GDK provides a gesture detector for the touchpad

Gestures include: tapping/swiping/scrolling

Working with gestures in the GDK:

+ create a GestureDetector

+ implement one or more listeners

+ override onGenericMotionEvent()

+ pass motion events to onMotionEvent()

Glass and Voice Activation (1)

OLD contents of <intent-filter> in AndroidManifest.xml:

<intent-filter>

<action android:name="android.intent.action.MAIN" />

<category android:name="android.intent.category.LAUNCHER" />

</intent-filter>

NEW contents of <intent-filter> in AndroidManifest.xml:

<intent-filter>

<action android:name="com.google.android.glass.action.VOICE_TRIGGER" />

<meta-data

android:name="com.google.android.glass.VoiceTrigger"

android:resource="@xml/voice_trigger" />

</intent-filter>

Glass and Voice Activation (2)

Contents of $APP_TOP/res/xml/voice_trigger.xml:

<?xml version="1.0" encoding="utf-8"?>

<trigger keyword="@string/glass_voice_trigger">

<input prompt="@string/glass_voice_prompt" />

</trigger>

Contents of $APP_TOP/res/values/strings.xml:

<?xml version="1.0" encoding="utf-8"?>

<resources>

<string name="glass_voice_trigger">take note</string>

<string name="glass_voice_prompt">tell me something</string>

</resources>

</intent-filter>

Sensors on Google Glass Supported sensors:

TYPE_ACCELEROMETER

TYPE_GRAVITY

TYPE_GYROSCOPE

TYPE_LIGHT

TYPE_LINEAR_ACCELERATION

TYPE_MAGNETIC_FIELD

TYPE_ROTATION_VECTOR

Unsupported sensors:

TYPE_AMBIENT_TEMPERATURE

TYPE_PRESSURE

TYPE_PROXIMITY

TYPE_RELATIVE_HUMIDITY

TYPE_TEMPERATURE

Glass Mirror API versus GDK Use Mirror API for:

+ platform independence

+ common infrastructure

+ built-in functionality

Use GDK for:

+ real-time interaction

+ access to hardware

+ offline functionality

Use the GDK and the Mirror API

What is HTML5?

“Anything that you’re working on right now that’s cool…that’s HTML5” (PPK)

HTML5 means different things to different people

Multiple Technologies (see W3C)

W3C REC status: MathML, SVG, and 2 others

Technologies “in” HTML5 Canvas 2D

CSS3

Drag-and-Drop (DnD)

File API

Geolocation

Microdata

Offline Applications

Server Sent Events (SSE)

SVG

Web Intents

Web Messaging

Web Storage

Web Sockets

Web Workers

HTML5 WAI ARIA

Web Accessibility Initiative: Accessible Rich Internet Applications

W3C specification for increasing the accessibility of

web pages and user interface components developed with Ajax, HTML, JavaScript and related technologies.

WAI-ARIA predates HTML5

WAI ARIA specification defines support for accessible web applications, which involves markup extensions that are often attributes of HTML5 elements

CSS3 Features (condensed) (Perf) Gradients (linear and radial) (Perf) Rounded corners/drop shadows CSS3 filters (merged with SVG filters) Simple shapes (circle/ellipse/rectangle) (Perf) 2D and 3D animation Multi-column support Exclusions and regions Flexbox (like “grid bag” in Java AWT)

CSS3 2D/3D Transforms

CSS3 2D transforms:

rotate, scale, translate, matrix, and perspective, and skew (deprecated)

CSS3 3D transforms (invoke the GPU):

1) similar to 2D versions (but no ‘skew’)

2) also three axis-specific methods:

rotateX, rotateY, rotateZ (and so forth)

• CSS3 animation via a keyframes rule

What “works” with CSS3?

Combine these with CSS3 in a Web Page:

+ SVG and D3.js (=JavaScript+SVG)

+ HTML5 Canvas (bitmap and no DOM)

For Web applications: jQuery (cross-browser + cross-platform) BackboneJS (reduces spaghetti code) AngularJS (open source from Google) Other JavaScript-based toolkits

What is HTML5 Canvas?

HTML5 <canvas> element a rectangular bitmap in a web page Direct mode (SVG is “deferred” mode) Draw-and-forget (“Etch-a-Sketch”) JavaScript APIs for rendering shapes Support for 2D but not for 3D Available on iOS5+ and Android ICS+ Hardware accelerated (RIM Playbook)

Basic Shapes in HTML5 Canvas

Line segments Circles and arcs Ellipses (with the scale transform) Bezier curves (quadratic/cubic) Polygons Paths (for arbitrary shapes)

What You can Do with 2D Shapes

Linear and radial gradients

Rotate, scale, skew, translate methods

Perspective and matrix() method

2D Animation effects

“Template” for HTML5 Canvas (Part 1)

<!DOCTYPE html>

<html lang="en">

<head>

<meta charset="utf-8">

<title>Graphics with HTML5 Canvas</title>

<script type="text/javascript"></script>

</head>

<body>

<canvas id="myCanvas" width=“800" height=“500">

</canvas>

</body>

</html>

What is D3.js?

D3.js: an open source project (circa 2010) Created by Mike Bostock based on JavaScript a layer of "abstraction" over SVG also support for HTML5 Canvas github.com/mbostock/d3 https://github.com/mbostock/d3/wiki/Gallery

Why/When use D3.js?

For data visualization If you need “extreme versatility” leverage JavaScript skills leverage SVG HTML5 Web pages with D3.js and:

HTML5 Canvas, CSS3, SVG, jQuery, …

What Can D3.js Do?

All the stuff you can do in SVG graphics/animation (2D only) filters/gradients mouse/keyboard events custom charts/graphs Support for Ajax, JSON, XML, CSV files

Simple D3.js Example<html>

<head>

<script src="d3.min.js"></script>

<script>

d3.select("body")

.append("p")

.text("Hello From D3");

</script>

</head>

<body>

<p>Hello From D3</p>

</body>

</html>

D3 and Other Technologies

You can create HTML5 web pages with:

+ D3 & Bootstrap + Firebase (or Parse)

+ D3 & BackboneJS

+ D3 & AngularJS

+ D3 & HTML5 Web Sockets

+ D3 & NodeJS (or Meteor)

What about SVG?

XML-based vocabulary for 2D shapes Gradients/Bezier curves/polygons/animation DOM-based and declarative syntax Supported in iOS5+ and Android ICS+ Partial hardware acceleration No support for 3D effects

SVG versus Canvas:

http://msdn.microsoft.com/en-us/hh552482.aspx



Useful Features of SVG

render circles/ellipses/elliptic arcs squares/rectangles/parallelograms cubic/quadratic Bezier curves arbitrary polygons linear/radial gradients and filters mouse events and animation support (*) good for charts/graphs Works with CSS3 (reference SVG via “url()” (*) consider using D3.js

Other Aspects of SVG

SVG elements are inserted in the DOM so you can track/manage groups of SVG elements

no blurred/jagged edges when zooming in

Convenient format for import/export between tools

Can apply XSL stylesheets to SVG documents

Blog by Patrick Dengler: SVG versus Canvas

Disadvantages of SVG

• Verbose (what do you expect? It’s XML)

• Can be difficult/incomprehensible (SVG tiger)

Animation code can be cumbersome/tedious

Consider using D3 instead of “pure” SVG

Strengths of CSS3/SVG/Canvas

CSS3: a) 2D/3D graphics/animation, b) GPU support, c) embeddable in SVG <defs> element, d) “easing functions” for animation, e) animate HTML elements

Canvas: a) 2D graphics/animation, b) GPU support, c) good for updating many small objects (games), d) works with video (use ThreeJS/WebGL for 3D animation)

SVG: a) 2D graphics/animation, b) some GPU support, c) create arbitrary 2D shapes, d) custom <pattern>s, and e) grouping via the <g> element, and “easing functions in D3 (but SVG cannot animate HTML)

Combining CSS3 and SVG CSS3 can leverage the power of SVG:

+ reference SVG documents via “url()”

• SVG can leverage CSS3 by:

embedding CSS selectors in an SVG doc

use jQuery methods:

+ the css() method for updating properties

• Use jQuery Mobile features:

use bindings to handle mouse/touch events

CSS3+SVG or CSS3+D3.js?

CSS3+SVG: useful when 1) you already have SVG-based data and 2) you must support IE6 (can do with Raphael toolkit but not D3)

• CSS3+D3: for modern browsers, and also easier for defining event handlers and animation (probably also easier to maintain/enhance)

• Notes:• CSS3+SVG might not have GPU support• Consider D3 with BackboneJS/AngularJS/etc

What about CSS3 and HTML5 Canvas?

Insert a <canvas> element in a Web page• JavaScript APIs for 2D shapes (similar to Skia) All shapes are rendered in a <canvas> element Modern browsers support <canvas> element “write-and-forget” (no DOM-based “history”)• CSS3 selectors can match <canvas> element• Canvas often renders polygons faster than SVG • Tracing Canvas calls:

http://www.html5rocks.com/en/tutorials/canvas/inspection/

Some Useful CSS3 Resources

• caniuse.com

online forums (Yahoo group)

Various meetup groups in BA

stackoverflow.com

Impressive HTML5 Canvas Demos

• Various HTML5 Canvas demos:

http://www.craftymind.com/factory/html5video/CanvasVideo.html

http://codepen.io/stuffit/pen/KrAwx

http://davidwalsh.name/canvas-demos







PhoneGap and Glass

Perform the following steps:

+ Install PhoneGap: npm install –g phonegap

+ Create a PhoneGap mobile app (CLI)

+ Add HTML/CSS/JavaScript assets

+ Deploy to Google Glass via an IDE

+ command line installation:

adb install –r myapp.apk

CSS3 and Glass Demo

CSS3 2D skew (for the three cube faces) CSS3 3D linear/radial gradients CSS3 2D and 3D transforms CSS3 3D animation effects (keyframes)

Demos: Canvas and Glass

Demo #1 uses Android Canvas: touch-enabled bar chart uses the GDK with Android Canvas

Demo #2 uses HTML5 Canvas: PhoneGap-based application

What about Android Wear

Android Wear provides a separate SDK

Not compatible with the GDK(?)

Different paradigm from Glass

Google Glass Official Sites

Currently available Glassware: https://glass.google.com/glassware

Purchasing Google Glass:

https://glass.google.com/getglass/shop/glass

Google Glass community:

https://www.glass-community.com/?auth_token=y8WXiHyf_TbV-_0aDamI9Q1UFE5tTkEUyWCHoPPAy10&authuser=0&hl=en_US&ts=1405039468196

https://glass.google.com/glassware



Bonus: NodeJS and Quadcopters var arDrone = require('ar-drone'); var client = arDrone.createClient();

client.takeoff();

client .after(5000, function() { this.clockwise(0.5); }) .after(3000, function() { this.animate('flipLeft', 15); }) .after(1000, function() { this.stop(); this.land(); });

Open Source Projects (graphics)

• Graphics Projects on http://code.google.com/p:

+ css3-graphics and html5-canvas-graphics

+ css3-jquery-graphics and d3-graphics

+ svg-graphics and svg-filters-graphics

+ easel-graphics, fabric-graphics, paper-graphics

+ ThreeJS, jQuery, Raphael, Google Go, Android

+ Dart, Dojo, JSF, Two.js, JavaFX 2.0

+ Lua, PHP, Perl, Python, Ruby, SWT graphics

Graphics Projects on github (ocampesato):

+ Famo.us, Polymer, AngularJS

http://code.google.com/p

NodeJS Course (10/19-10/21)

Adam CrabTree

Enerspace

2225 East Bayshore Road

Suite 100

Palo Alto, CA 94303

http://www.eventbrite.com/e/noderietytm-nodejs-bootcamp-october-tickets-12539314427

Recent/Upcoming Books

1) HTML5 Canvas and CSS3 Graphics (2012)

2) jQuery, CSS3, and HTML5 for Mobile (2013)

3) HTML5 Pocket Primer (2013)

4) jQuery Pocket Primer (2013)

5) HTML5 Mobile Pocket Primer (2014)

6) D3 Pocket Primer (2014)

7) Google Glass Pocket Primer (2015)

8) AngularJS Pocket Primer (2015)

• “WebGL: Up and Running” (Tony Parisi)

Co-creator of VRML and X3D