Glynn Bird – Cloudant – Building applications for success.- NoSQL matters Barcelona 2014

© 2014 IBM Corporation

Building Applications for SuccessGlynn Bird, Developer Advocate

Agenda

• Application Development

• Scaling

• Real-world examples & how can Cloudant help?

Introduction

@glynn_bird glynn@cloudant.com

Glynn BirdDeveloper AdvocateCloudant

http://glynnbird.com

Development - Go-to tools

Basic

Pascal

C

C++

Forth

PHP4Java

PHP5

ObjC

Node.js

1980s 1990s 2000s 2010s

File System

NoSQL

SQL

Scalability?

• Pick your tools of choice• PHP + MySQL, Ruby + Mongo

• Build on your machine

• Deploy!

• Scalability is an ops problem?

• Hint: “no”

Scaling tip #1 - Keep things separate

• Separate • load balancing• app• database• cache

• Separation of layers allows independent scaling

http://bit.ly/10xGe2YGoogle’s first server rack

Scaling tip #2 - Keep app servers stateless

• Store all persistent data in a database

• Don’t be tempted to store sessions on local disk

• Stateless app servers can be created and destroyed at will

http://bit.ly/10xGe2Y

IBM 305 RAMAC 5Mb Disk - 1957

Scaling tip #3 - queue-driven workflow

• Store tasks in queues

• Workers servers consume the queue

• Queues allow workers to be scaled at will

http://bit.ly/1Aycjse

Scaling tip #4 - Automate everything

• No need to manually configure servers anymore

• Workers servers consume the queue

• It should be as easy to deploy N servers as 1 server

http://bit.ly/1Aycjse

What does failure to scale look like?

http://bit.ly/1Aycjse

What does failure to scale look like?

• increased latency

• downtime

• high maintenance costs

• data loss

• disgruntled customers

http://bit.ly/1Aycjse

Cloudant Highlights• Founded in 2009 by data scientists:

Hoffman, Miller & Kocoloski

• 20,000+ freemium and 225+ paying customers

• Key collaboration & investment agreement with Samsung Electronics

• HQ’d in Boston with offices/presence in Seattle, San Francisco, Bristol, UK & Seoul, SK

• 34 global locations on 4 hosting providers

• Acquired by IBM in February 2014 into the Information Management software group

Cloudant NoSQL DBaaS

Indexed JSON

Storage

Geo-Load Balancing

App Layer / Browser / Mobile App

Physical Storage @ IBM SoftLayer, Rackspace, Microsoft Azure, AWS

Cloudant JSON API

JSON CRUD

Multi- Master

Replication

Device Sync

Lucene Search

2D, 3D Geospatial

Incremental MapReduce Monitoring Security

IO Control

Managed for you 24x7 by Cloudant developer-operators

REST (HTTPS)

How can Cloudant help you to scale?

• Scales massively & elastically• Handle millions of daily active users

• Guaranteed performance & up time• Removes risk of project & SLA delivery failure

• More agile development for web & mobile• No rigid schemas to slow development

• Managed for you 24x7• Stay focused on new development, no DB administration

Cloudant empowers clients to get their applications to market faster in a cost-effective, hassle-free delivery model with guaranteed performance and support.

Example 1 - IoT Data Logging

• Temperature sensor £1.50

• Raspberry Pi £25

• Network attachedthermometer

DS18B20

DS18B20

Software

• Bash script

• Where should we put the data ?

sensorid=`find /sys/bus/w1/devices/* -type l | grep -o '[0-9]*-.*$' | head -n 1`filename="/sys/bus/w1/devices/$sensorid/w1_slave"T=`tail -n 1 "$filename" | grep -o "[0-9]*$"`t=${T:0:2}.${T:2:3}unix=`date "+%s"`d=`date "+%Y-%m-%d %H:%M:%S %Z"`json="{ \"ts\": $unix, \"date\": \"$d\", \"temperature\": $t, \"sensor_id\": \"$sensorid\" }"

DS18B20

Storage

• Local CouchDB

• Installed with one line

sensorid=`find /sys/bus/w1/devices/* -type l | grep -o '[0-9]*-.*$' | head -n 1`filename="/sys/bus/w1/devices/$sensorid/w1_slave"T=`tail -n 1 "$filename" | grep -o "[0-9]*$"`t=${T:0:2}.${T:2:3}unix=`date "+%s"`d=`date "+%Y-%m-%d %H:%M:%S %Z"`json="{ \"ts\": $unix, \"date\": \"$d\", \"temperature\": $t, \"sensor_id\": \"$sensorid\" }"curl -X POST -H "Content-Type: application/json" -d "$json" "http://localhost:5984/logger/"

Where does Cloudant fit in?

• Replicate to Cloudant

• Scale by adding more sensors

• Data collection can be offline

• Sync when connected

Demo

• http://celsius.mybluemix.net

• Replication• collect data at the edge, replicate when online

• Cloudant can store data at massive scale with high availability

• Time-series data is a great fit for Cloudant

Why is Cloudant a good fit?

• MapReduce indexes on date field

• Allows range querying with grouping e.g. find min/max/avg between two dates, grouped by year/month/day/hour etc.

Time-series data in Cloudant

{ "_id": "00339bc7", "_rev": "1-0ef01b53", "ts": 1411723490, "date": "2014-09-26 09:24:50 UTC", "temperature": 20.937, "sensor_id": "28-000006746cde" }

[2014,9,26,9,24,50] ---> 20.937

• Mobile game

• Stores games state, usage stats, achievements etc. locally

• Needs to be cross-platform

Example #2 - Mobile applications

• In-browser, cross-platform datastore

• CouchDB compatible

• Syncs with Cloudant

Option 1 - PouchDB + HTML5

http://pouchdb.com/

• iOS and Android native libraries

• Local storage + query

• Syncs to and from Cloudant

Option 2 - CloudantSync

https://cloudant.com/product/cloudant-features/sync/

One database per user

• Each user’s data has it’s own access control

• Each database is relatively small; small enough to fit on phone

• Scales massively; hundreds of thousands of concurrent users

One database per user

https://cloudant.com/product/cloudant-features/sync/

• Business Card collection app

• Native Android App

• Companion Web App

Demo

https://cloudant.com/product/cloudant-features/sync/

How did that work?

pouchdb

Android Phone App

IBM BlueMix

Web App

var db = new PouchDB("myfirstdatabase");

db.post( { a:1, b: "2", c: true } );

db.sync(URL, { live: true });

What about the code?

https://cloudant.com/product/cloudant-features/sync/

• Replication is a hard problem. PouchDB + Cloudant solve it

• One database per-user is a great way to scale mobile apps

Why is Cloudant a good fit?

• Building with Cloudant as your data layer has become a “go to” development solution

• Building with Cloudant builds in scale from day one• large data sets• lots of smaller data sets

• Cloudant syncs!

Conclusion

https://cloudant.com/product/cloudant-features/sync/