Developing TouchActive with GAE and GWT - Google DevFest Barcelona 2012

Andrew MackenzieDaniel RuizBCNTouchhttp://ta.bcntouch.com

Google App Engine & Google Web Toolkitour experience developing TouchActive

The purpose of this talk

● Share our experiences using GAE and GWT

● Help guide you if you are considering similar technologies

● Provide ideas if you are already developing with these technologies

● Provide feedback to Google on things to maintain, improve and renew/drop

Contents

● Solution● PaaS choice● GAE overview● GWT overview● GAE + GWT debugging● TouchActive

implementation● Addit● Optimization tools● Phonegap experiment● Avoiding lock-in● What would we do

differently● Feedback to Google

Our Project

GAE + GWT

Implementation

Summary

Feedback

Our Project

TouchActive Platform- what we built

A platform to put consumers and

brands in touch with each other

Brand

Consumer

Using tagged products that the consumers were

interested in

Mobile apps to read tagged products- NFC, QR, barcode

Cloud service and data storage

Mobile apps read products tagged

with various technologies

Scalable store in the cloud with product

data and synchronization of

user data

Get product information, help

remember, organize in lists, etc.

Focus on easy and intuitive interaction with no data entry. NFC the best.

Get information about productsfrom tagged objects

Management console for brand- multi-tenant, groups, roles


Access control on changes

Products findable in open queries from

mobile devices

Consumer Web Interface- Lists, Products, History, etc


Mobile web version for mobile platforms without a native app

User data synchronized with native mobile app

Addit "Web Clipper" to capture products from web pages

Social networks, Identity and Sharing

Leverage existing user identities for

easy sign-up (web and mobile)

- OpenId and OAuth

Sharing of Products, Lists via:

● Social Networks● EMail● NFC Tags● NFC peer-to-peer● QR codes● Bump

TouchActive Solution Overview- what we built

TouchActive Solution Overview- what we built

Mobile Apps

Cloud backend

Desktop and Mobile consumer web

Addit

Tagged products

Management console

Identity Providers

Sharing (web and mobile) over net and face-to-face

Context for our choices2+ years ago

● Start Small○ pre-start-up: build prototype○ uncertain financing (probably small)○ small team of engineers

● Think big, or the possibility of big!

● Uncertainties○ Mobile platforms (especially NFC)○ Cloud computing platforms

Cooling

Power / UPS

Security

Buildings

Cooling

Power / UPS

Security

Buildings

LAMP stack on owned or hosted HW

Amazon Web Services(AWS)

Google App Engine(GAE)

Newer options:● OpenStack (Open SW, multiple service

providers)● Windows Azure● Amazon features for auto scaling, etc.

small start-up, low capital investment.

Pay as you go, don't buy servers.

But we didn't want to get "locked-in" to a single service provider

don't spend people time and money on managing servers, runtimes, etc. Focus on the application.

unknown/unexpected demand

we chose GAE

Applications developed with GWT

And many more, see:http://gwtreferencelist.appspot.comhttps://developers.google.com/web-toolkit/developer_spotlighthttp://gwtgallery.appspot.com/http://www.quora.com/What-web-applications-use-Google-Web-Toolkit-(GWT)

AdWords AdSense Orkut Wave Health Blogger

http://en.wikipedia.org/wiki/AdSense

http://en.wikipedia.org/wiki/Orkut

http://en.wikipedia.org/wiki/Google_Wave

http://en.wikipedia.org/wiki/Google_health

http://en.wikipedia.org/wiki/Blogger_(service)

Google Web ToolkitComplex UI

Complex UI with graphical Data representation

GAE + GWT

10

Google App Engine Features

● Free developer / Pay-as-you go costs● Platform not Infrastructure

○ No need to architect the SW stack for scalability○ No need to manage the SW stack○ No machine administration○ Reasonable set of platform services

● Scalability○ Instances created automatically based on demand○ Scalable, distributed, datastore○ Content Distribution Network (CDN)○ High-performance Image service

Google App EngineTools and Languages

Language choices at the time: Java and Python○ Now "GO" is also available now

We chose Java● Developing J2ME and Android mobile apps in Java

at the same time

Eclipse IDE○ Local development server○ Java debugging of server code

Appstats for server performance analysis (RPC)Management dashboardCommand line tools and ANT tasks for automation

Google App EngineDatastore

Advantages● Scalable● Time vary with result set

size, not search set size● Compatible with existing

frameworks○ Java Data Objects (JDO)○ Java Persistence API (JPA)

● Distributed, replicated across data centers

Drawbacks● Pay-per-use● NoSql & pre-built

indexes○ No JOIN operations

○ No inequality filters on more than one "field"

○ No filtering of data based on subqueries

○ No Grouping of results

● Write rate limited

Datastore

Megatable

BigTable

GFS2

Google App EnginePlatform Services

Standard Implementations● Mail (JMail)● Datastore (JPA/JDO)● MemCache (JSR107)● Logging (Java Logging)

Non-Standard● Cron Tasks● Task Queues● Back-ends● Users (Identity)● Blob Store● Image Service● Chat (XMPP/Jabber)

with Presence● Channels API (Push)

Web Client DevelopmentGoogle Web Toolkit

● Develop Javascript AJAX client with Java○ Strongly, statically, typed language. ○ Many errors caught at compile time○ Full IDE support (completion, etc)○ Internationalization (i18n) built-in○ Generates optimized JavaScript, client bundles, code splitting, etc.○ Hides browser differences

■ Compiles a "permutation" for each language and browser you support

● Integrated Server and Client development project○ Java across Server and Client

○ Shared classes between Server and Client

○ "Transparent" RPCs between server and client (takes care of serialization and exception transport)


● Deferred binding with "properties"○ Used for loading script variants based on browser, language, form

factor, etc

● Reasonable Widget set○ Has improved a lot

○ Other widget sets available on top (GXT from Sencha, SmartGWT, Vadim, etc)

● "DevMode"○ Java debug experience in IDE for client side code○ Combined Client/Server/RPC debug experience○ No-compile, fast turn-around, hot code replace○ Memory intensive! :-(


Encourages reusable modular code development● Custom widgets● Code in defined "modules"

(server/client/shared code, RPCs, html, css, images, etc)

● Bundle modules into GWT libraries (.jar)

image


Ease of implementing Design Patterns● Model-View-Presenter with

late-binding of views● Same Presenter, different

Views● Detect Form Factor from

UA and serve the most appropriate view at run-time

Google Web ToolkitLate-Binding of views with ClientFactory

TouchActive.java

TouchActive.gwt.xml

Google Web ToolkitLate-Binding of views with ClientFactory

TouchActive.java

Desktop Mobile

Google Web ToolkitMVP, Activities & Places

Event-Bus across client code for subscribe/broadcast event programming

Application Activity and Place management via browser history

History Token

Google Web ToolkitUiBinder (XML) for declarative layout

With standard or custom widgets

GAE/GWT Debug experience:An example

Click Event on Facebook sign in button

GAE/GWT Debug experience:Client debugging

GAE/GWT Debug experience:Server debugging

GAE/GWT Debug experience:Client debugging

Implementation

20

Our implementation of TouchActive with GAE + GWT

● Authentication / Identity management

● Architecture overview

● Data Persistence Layer

● Platform API

● Portability

● CMS Integration

● Addit

Authentication:Own, portable, Identity management

● Created own User Object and related to standard external accounts○ OAuth (FB, Twitter, LikedIn)○ OpenID (Google, Yahoo, MySpace, ...)○ Google Apps domains for company users

● Username + Password also● Avoid forcing users having to have a Google account● Use OAuth & OpenId on mobile apps transparently for

account creation and sign-up

Avoided lock-in, didn't use Google App Engine Users Service

Architecture Overview

Servlet Filters

Request● Headers, cookies, ...

RequestContext

Platform Specific Services

Platform API

Data Persistence Layer

App Logic

Servlet Filters

● Flexible, just map filters to servlets with a config file using paths○ /api○ /rpc○ /cms○ /static

● Keeps servlet code clean and modular● Used for:

○ Language selection○ Desktop/Mobile pages while maintaining constant URL○ Authentication + assignment user to their Multi-Tenancy namespace○ Query logging for analytics

Servlet Filters:EntryPoint Filter (The Special One)

● Different types of EntryPoint filters that set different permissions and tokens○ Initializes the Request Context used in your app and DataManager○ Use of Thread variables to make available to all code but multi-threaded

● Authenticates the user and sets the correct "Namespace" in the request context

● Defined the appropriate filter for each request based on Access Control rules in web.xml

Servlet Filters

Request● Headers, cookies, ...

RequestContext

EntryPoint Filter


DataManager I/F to rest of app

RequestContext

Data Manager

JDO portable library across multiple datastore implementations

Google Datastore MySQL

Google AppEngine AWS MySQL Server


DataManager Interface● Created an API to manage data

objects for reading and writing● DataManager layer is used by all

app, app is kept 100% portable

Data Objects● ACL, Protected, Global, etc...● Enforce visibility and

permissions according to filters used on request and object type

Multi-Tenancy● Create data areas in the same

datastore● Avoided using Google Datastore

Namespaces

DataManager I/F to rest of app

Data Manager

JDO

Google Datastore MySQL

Google AppEngine AWS MySQL Server

Platform Specific Services

Custom build process outside IDE● ANT tasks to build, deploy, test,

release● Run manually from Command

Line or scripts / cron● Significant effort

App Logic

Defined a Platform API for platform dependant services

Cron Tasks

Task Queues

Blobstore

Image Service

Chat (XMPP)

Channels (Push)

GAE Implementation

Cron Tasks

Task Queues

Blobstore

Image Service

Chat (XMPP)

Channels (Push)

Other Implementation

Additional Need:Web content beyond the appEditable by non-programmersNo need for application release on each changeLocalized to multiple languages

Use a CMS

CMS Content

GWT

Integrating a CMS

Chose to serve content, scripts and RPCs from same domain○ Deploy CMS to same GAE "appID" as our app○ This could maybe be avoided with subdomains...

Chose Vosao GAE/J based CMS○ A bit immature at the time....still < 1.0○ Uses Apache Velocity (Java based) scripting language○ Java plugin can integrate tightly with your app and call app code directly

from within CMS web pages○ Has locked us in to GAE at CMS level, but our code is still portable○ Manual integration of config files, with help from Vosao team it could be

automated

Internationalization (i18N)

● Client GWT UI○ Built-in i18N support in GWT, strings in property files○ Permutation of script compiled for each language and correct

language loaded at run time

● Server messages○ We have avoided as much as possible○ Java Message Bundle + Properties files

● CMS content○ Core feature of CMS. Individual strings or entire pages can be

localized with common template

Other Features

30

Other features implemented

● Chat functionality with server (XMPP)○ used for testing at the moment

● "Presence" detection of users (XMPP)

● Push messages to browser using the GAE Channel API○ GWT Client JSNI Wrapper and EventBus connection

● "Addit" Bookmarklet and "web-clipper"

Addit -Objectives

● Help users create products in TouchActive by capturing product from any web page

● Reduce need for data entry by capturing as much data as possible automatically○ Use eCommerce/Search meta data when available

● Relatively simple and easy to use● Broad usage

○ Support all main web browsers○ Support as many web sites as possible○ Desktop, Tablet and Mobile versions

Product meta-data

meta and micro-formats used by eCommerce sites to provide data to search engines or crawlers:

Various Standards Exist:

● Facebook○ OpenGraph "product"

● Schema.org● GoodRelations● GoogleProducts

○ Google Merchant Center / Google Product Search

● hProduct

Google Product Search

Addit Installation

Drag and drop bookmarklet from web-page to bookmarks bar or folder

Install extension directly● From Chrome Web Store● Chrome, Safari, Firefox from web page

Addit UseUser clicks either the

Bookmarklet or the Extension button (FF, Safari, Chrome)

AdditScript loading

Keeping script out of the bookmarklet or

extention allows us to update the script

continually, without any re-install or

redistribution to users

Load localized script from cloud server and injected

into the page

Script NOT from Same-Origin as page and

security concerns and barriers to overcome

Detect language from page contents

AdditFetch user lists and detect currency

Fetches users lists from their TouchActive

account using a Cross-Domain "JSONP" RPC

call

Select default currency based on page language, page

contents and user settings retrieved with

lists

Additproduct meta-data parsing

Parse DOM and detect micro-format product

markup elements.Prepares products for

user automatically

If no product detected, uses page-level markup (meta, OpenGraph) or

general page information (Title, etc)

If multiple products detected, a simple

product selector UI is shown initially

AdditPersonalize and save to account

Select one of their own lists they would like to add the

product to

Will be automatically synced with their mobile app

Users take the suggested texts, or customize them.Select image, price, etc.

... and "Addit" to their TouchActive account

Add a personal note

AdditDifferent layouts for different form factors

DesktopWider, no vertical scrolling, smaller mouse targets

MobileNarrower (adjust to Viewport),

vertical scrolling, larger touch targets

All Developed in GWT

● GWT XSI Cross-Site linker for CORS script loading● Wrote own JSONP library for Cross-Site RPC

○ in Java with some JSNI

● GWT "late run-time binding" of scripts for Localization and Form Factor permutations using custom "Property Providers"○ Detect "best guess" language from host page using multiple methods

(poor standards implementations)○ Detect form factor using a combination of User Agent and display size

● Avoid older Layout Widgets that compile down to HTML Tables (Box Model and CSS problems...)

● Fast product meta-data parsing from DOM using "querySelectorAll()" and CSS3 selectors

Difficulties - mainly due to injecting a script into a page you don't control

● Unknown, uncontrolled CSS / DOCTYPE / QuirksMode○ Can't just do a "CSS Reset"○ CSS properties influence complex layouts○ DOCTYPES affect browser behavior (a nightmare on IE 7/8/9)

● "Difficult" sites○ Don't follow W3C standard for language selection○ iFrames/Frames/Frameset based○ Flash sites!○ Broken image links

● Different implementations of product markup standards○ Especially prices "," vs "." EUR vs € before/after/middle (1€25)

● JavaScript image galleries and tricks○ Images loaded but not visible○ Images being pre-cached (not always in DOM as image)○ Images adding / removed from the DOM

IE and DOCTYPE and QuirksFor masochists only!http://blogs.msdn.com/b/askie/archive/2009/03/23/understanding-compatibility-modes-in-internet-explorer-8.aspx

Phonegap Experiment

Can you take a GWT compiled app (HTML, CSS, JS, Images etc) and create a "native" app for multiple platforms?

Kind-of...● GWT RPCs from Phonegap client to server!

○ Must be able to go cross-domain○ We wrapped the RPC class so that it remaps the RPC endpoint to

default server when running "offline"○ Version problems expected with serialization in the future!!

● Phonegap has problems with relative file paths○ Processed GWT comiled output to use absolute paths

● Prototype produced from existing GWT app○ Automated with ANT tasks

Q

A

Optimization

40

Optimization Tools Available

Chrome Developer Tools "Audit"

PageSpeed

AppStats for server GAE internal RPC

SpeedTracer

SpeedTracer with AppStats

Chrome Audit

Page Speed

GAE AppStatsunderstand AppEngine RPC usage

AppStatsdrill-down to calls, duration and latency

SpeedTracerUnderstanding client side performance

SpeedTracerNetwork resource usage from client

Resource Loads● HTML● CSS● JavaScript

SpeedTracer (client) integrated with AppStats (server)

RPC request to server

Server side trace and timing from AppStats on server

What's new in GWT2.5

● Super Dev Mode (experimental)○ Replaces DevMode, uses "SourceMaps" in Chrome

to enable debugging in language JS was written in (Java, Dart...)

● New compiler optimizations○ Smaller, faster JavaScript code○ With slightly longer compile times

● Elemental○ Lightweight JavaScript library

● Updated ARIA support● UIBinder enhancements● Validation enhancements

GWT Compiled codesize reductions

GWT 2.4

100%

GWT 2.5

93%

GWT 2.5Closure Compiler

87%

Summary

40

Did we avoid provider "Lock-in"?

Portable data layer built on JDO● Maybe not the most optimal when running on GAE● Have to work with limitations of JDO/GAE subset● Over time, actually moving the data becomes a problem

Platform API has kept rest of code portable● needs work to get running on additional platforms

GAE based CMS has increased it● outside of the code, but still important for the project● decouple by moving to multiple domain approach

Things we would do differently if we started again

New competitors to GAE

Analytics

Phonegap app

API and directory structure

CMS

● New AWS services bring it closer to PaaS

● OpenStack from multiple providers

● Analyze more closely and prototype if time permits



Analytics

Phonegap app


CMS

● Avoid programming a lot of that, and falling short

● Expose analytics data and use external tools

● Give business users top-class tools with flexibility of analysis, drill-down etc



Analytics

Phonegap app


CMS

● Design and structure web-app for that from day one



Analytics

Phonegap app


CMS

● Assume you will have to version your external API

● Design app structure early on○ statics, resources, API,

CMS, RPC,...



Analytics

Phonegap app


CMS

● Explore using other CMS outside of GAE

● Sub-domains for "Single-Origin" content, RPCs

Possible Future Work

● Optimize○ Using tools to focus our efforts○ More use of memcache for DS accesses

● Explore use JDO3 (Polymorphism)● Prototype alternative CMS

○ Run on sub-domain in parallel

● Write platform code for other platforms○ Test on AWS or OpenStack?

● Phonegap app for other platforms○ Blackberry, Windows Phone...

● Investigate Continuous Integration Support○ Jenkins on GAE

GAE + GWTThe Good

● Scalable, without knowing about or managing the stack● Free for developers, our costs have been pretty low● Java across mobile, backend and client● Programming large app "all together" as client-server

○ But avoid that permitting spaghetti, structure APIs and code○ Hierarchy of reusable modules of client, server and shared code○ Drop-in GTW libraries with client, server and shared code

● Excellent debug experience when it works!○ including hot code replacement (server + client)

● Model-View-Presenter architecture combined with late-binding○ extend your app to mobile, tablet views more easily

● UI Binder for declarative UI layout, code for behavior

GAE + GWT The Bad

DevMode problems● Lack of maintenance on browser plug-ins● Memory intensive

You have to put in effort to avoid "lock-in"

JDO● Some people reporting long start-up times (meta-data related)

○ Has worked pretty well for us● Some features missing: e.g. No Async writes

Feedback

44

Feedback for GoogleApp Engine

● Don't introduce features or ways to use features that increase lock-in. ○ If you do, we will go to "standard" stacks on AWS or OpenStack...○ e.g. threads, etc

● More transparency on Billing and scheduler behaviour

● Need better attention to developers in forums○ Paying customers!○ Especially around scheduling and billing - invest people hours in this

● Channels API○ Support multiple listeners per page and broadcast○ Google supported JSNI GWT library

Feedback for GoogleGWT Debugging

Don't break the debug experience!

● Combined server/RPC/client debug experience in a single IDE is fantastic and unique (?) in developing an AJAX client - cloud server application○ Make it better! More maintenance of plug-in○ Support at least on Chrome / WebKit browser○ Less of a memory hog

● GWT 2.5 SuperDevMode does not replace it IMHO○ list of limitations of seems to outweigh the advantages○ main advantages (plug-in maintenance) are for Google

Feedback for GoogleGWT - others

● Allow client-only code and server-only code in same file○ E.g. Via @annotations○ Avoid having to complicate the class hierarchy to

have shared/server-only/client-only code for a class

● Java Calendar emulation in GWT and allow for shared-code date handling

Thank You!Questions?

Andrew [email protected]@bcntouch.com

http://www.linkedin.com/in/andrewdmackenzie

Daniel Ruiz Gimé[email protected]@bcntouch.comhttp://www.linkedin.com/in/danielruizgimenez

50

Technology

Developing TouchActive with GAE and GWT - Google DevFest Barcelona 2012