Upload
sander-van-de-graaf
View
1.961
Download
0
Tags:
Embed Size (px)
DESCRIPTION
A talk I did at phpday.it in Verona, Italy. Using some more advanced CouchDB functionality, like filters, changes feed, shows, geolocation (GeoCouch), and other interesting stuff. It also covers some real use cases.
Citation preview
CouchDBrelax
CouchDBrelax
Sander van de Graaf@svdgraaf
RELAX
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff (GeoCouch!)
CouchDBrelax
NOSQL
IT’S A MOVEMENT
1998
Carlo Strozzi
NOSQL == Not Only SQL
“[The NoSQL movement] departs from the relational model altogether, it should therefore have been called more appropriately ‘NoREL’, or something to that effect.”
- Carlo Strozzi
CouchDBrelax
NUTSHELL
SPEED
APPEND ONLY
NO REPAIR NEEDED
COMPACTING
HTTP SERVER
CAP
CAP
CouchDB
EVENTUALLY CONSISTENT
CouchDB
FULL REST API
HTTP METHODS
• GET
• PUT
• POST
•DELETE
• COPY
• SELECT
• UPDATE
• INSERT
•DELETE
• ...
{ "total_rows": 2, "offset": 0, "rows": [ { "id": "abc089b91fadfcf67a438e73f426f71d", "key": [ "2011-05-07 13:52:15", "abc089b91fadfcf67a438e73f426f71d" ], "value": { "_id": "abc089b91fadfcf67a438e73f426f71d", "_rev": "2-a38e08996586cd774b0b5c3006659de0", "message": "A2 HUISARTSENPOST BARNEVELD NIJKERKERWEG 119 3771LA 119" } }, { "id": "abc089b91fadfcf67a438e73f4286fd5", "key": [ "2011-05-07 14:00:16", "abc089b91fadfcf67a438e73f4286fd5" ], "value": { "_id": "abc089b91fadfcf67a438e73f4286fd5", "_rev": "2-691929b5f3d6cde58f27d62681c7b599", "message": "A1 LEEUWERIKSWEIDE 183 WAGENINGEN 6708LH 183" } } ] }
REPLICATION
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff
RELAX
PHP USAGE
PHP LIBRARIES
• PHPillow (LGPL)
• PHP Object Freezer (BSD)
• PHP On Couch (GPL 2 / 3)
• PHP CouchDB Extension (PHP license)
• SAG for CouchDB (apache)
•Doctrine 2 CouchDB ODM
<?PHP// setup connection for couchdb$client = new Couchdb_Client('http://ponies.couchone.com:5984','rainbows');
// fetch a document$doc = $client->getDoc('awesome_pony');
// updating document$doc->type = "awesome";
try { $client->storeDoc($doc);}catch (Exception $e) { echo "Document storage failed : " . $e->getMessage();}
PHP ON COUCH
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff
REPLICATION
DEFINITION
“Replication is the process of sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.”
Source: wikipedia
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
US NL
BE
USE CASEIDG Publishers
IDG NL IDG DE
IDG US IDG AUSTRALIA
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
CouchDBrelax
IDG NL IDG US
IDG DE IDG Italy
IDG Brazil
IDG Romania
IDG Poland
IDG SpaniaIDG UK
IDG Ghana
IDG Hungary
IDG Sweden
IDG Portugal
IDG Japan IDG Thailand IDG Vietnam
IMPLICATIONEvery single bit of IDG content published
worldwide, is available anywhere, anytime in msec.
P2P WEB
“World Domination”
HOW DO I DO THAT?
CLUSTERING“The fun stuff ”
CouchDBrelax
CouchDBrelax
loadbalancer
...n
CHALLENGES
• Large amounts of data
• Large views (with big/long map/reduce queries)
• LOTS of traffic
• Location based partitions
• For fun and profit
MAP/REDUCE
INPUT
IP Bytes
212.122.174.13 18271
212.122.174.13 191726
212.122.174.13 198
74.119.8.111 91272
74.119.8.111 8371
212.122.174.13 43
MAPPER => REDUCER
IP Bytes
212.122.174.13
18271
212.122.174.13191726
212.122.174.13198
212.122.174.13
43
74.119.8.11191272
74.119.8.1118371
AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643
PARTITION INPUT
Partition IP Bytes
0 212.122.174.13 18271
0 212.122.174.13 191726
0 212.122.174.13 198
1 74.119.8.111 91272
1 74.119.8.111 8371
0 212.122.174.13 43
MAPPER => REDUCER
Partition IP Bytes
0 212.122.174.13
18271
0 212.122.174.13191726
0 212.122.174.13198
0 212.122.174.13
43
1 74.119.8.11191272
1 74.119.8.1118371
AFTER REDUCE
IP Bytes
212.122.174.13 210238
74.119.8.111 99643
• CouchDB Lounge
• Pillow
• BigCouch
CLUSTERING OPTIONS
LOUNGE
•partitioning/clustering
•Nginx module
•meebo.com
• ‘easy’
•http://tilgovi.github.com/couchdb-lounge/
LOUNGE
• dumb_proxy => proxy for simple PUT/GET’s
• smart_proxy => proxy for map/reduce over shards
• replicator => updates all copies, redundantly
CouchDBrelax
CouchDBrelax
nginx
...n
dumb_proxy
CouchDBrelax
CouchDBrelax
nginx
...n
smart_proxy
Bonus:
other nginx modules work too
PILLOW
•Erlang based
• router/rereducer (map/reduce over multiple systems)
• In development (but promising!)
•https://github.com/khellan/Pillow
BIGCOUCH
•Fork
•100% api compatible
•Open Source/Commercial
•https://cloudant.com/#!/solutions/bigcouch
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff
BACKEND USAGE
PROXIED
CouchDBrelax
DIRECT
CouchDBrelax
NOSQL && SQL HYBRID
• onSave, onCommit hooks available in every major framework
• onSave -> make a JSON representation of your object, and PUT it to couchdb (#protip: only ‘public’ data)
• sql db is leading, you don’t care about versioning in couchdb
• you can use your data directly from couchdb within your frontend javascript
<?php
class Pony extends Application_models{ public function toArray() { $data = $this->_getData();
// we don't want any private data pushed unset($data['created_on']); unset($data['created_by']); unset($data['access_level']); unset($data['private_data']);
$data['tags'] = $this->getTags(); $data['categories'] = $this->getCategories(); $data['rainbows'] = 'double'; return $data; }}
MODEL
AFTER_SAVE<?php
class article_module extends admin_module{ public function after_save() { parent::after_save(); $data = $this->toJson();
// store the document in couchdb $res = CouchDB::put($data); // set the local id and revision, so we can // use it the next time $this->_id = $res->_id; $this->_rev = $res->_rev; }}
RewriteEngine OnRewriteRule /data/(.*) http://127.0.0.1:5984/db/$1 [P,L]
PROXY
?callback=foobar
JSONP
foobar(...)
JAVASCRIPT
<script type="text/javascript">$.getJSON("/db/ponies/_design/ponies/_view/best-ponies?include_docs=true",function(res){ for(i in res.rows) { doc = res.rows[i].doc; // do stuff }});</script>
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff
COUCHAPP
“Distributed, scalable, web applications you say?
omgwtfbbq!?!1!!!one1!1!eleven”
_attachments
CouchDBrelax
CouchDBrelax
CouchDBrelax
INSTALLATION
Couchapp 0.7.5
$ couchapp init
LAYOUT
$ couchapp push http://ponies.couchone.com:5984/rainbows
get your own copy:
$ couchapp clone http://svdgraaf.couchone.com/couchappdemo/_design/↩ phpday phpday
https://github.com/brandon-beacher/couchapp-tmbundle
CONTENTS
• Introduction
• PHP Usage
• Replication/Scalability
• Backend usage
• Couchapps
•Other stuff
OTHER STUFF
REWRITES
_REWRITE
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-ponies?descending=true&limit=5&key=”foobar”"
STEP 1
{ .... "rewrites": [{ "from": "/best-5-ponies", "to": "ponies/_view/best-ponies", "method": "GET", "query": { "descending": true, "limit": 5, "key": "foobar" } }]}
STEP 1
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_view/best-ponies?descending=true&limit=5&key=”foobar”"
STEP 1
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
STEP 1
[vhosts]awesomeponies.com = /rainbows/_design/ponies/_rewrite
STEP 2
$ curl "http://ponies.couchone.com/rainbows/_design/ponies/_rewrite/best-5-ponies"
STEP 2
$ curl "http://awesomeponies.com/best-5-ponies"
STEP 2
_CHANGES
RELAX
$ curl -X GET "http://ponies.couchone.com/rainbows/_changes"
{"results":[
], "last_seq":0}
curl -X PUT http://ponies.couchone.come/rainbows/foobar -d '{"type":"awesome"}'
{"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }], "last_seq":1}
{"results":[ { "seq":1, "id":"foobar", "changes":[{"rev":"1-aaa8e2a031bca334f50b48b6682fb486"}] }, { "seq":2, "id":"foobar2", "changes":[{"rev":"1-e18422e6a82d0f2157d74b5dcf457997"}] }], "last_seq":2}
_CHANGES OPTIONS
• ?since
• Longpolling
• Continuous
$ curl -X GET "http://ponies.couchone.com/rainbows/_changes?since=20"
SINCE
curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=longpoll&since=2"
LONGPOLLING
curl -X GET "http://ponies.couchone.com/rainbows/_changes?feed=continuous&since=2"
CONTINOUS
FILTERS
function(doc, req) { if(doc.priority == 'high') { return true; } return false;}
FILTERS/IMPORTANT.JS
$ curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/important
function(doc, req) { if(doc.city.name == req.query.name) { return true; } return false;}
FILTERS/CITY.JS
curl -X GET"http://ponies.couchone.com/rainbows/_changes?feed=continuous&filter=app/name&name=verona
SHOWS
function(doc, req) { return { body: "Hello World" }}
curl -X"http://db.example.com/db/_design/foobar/_show/showfunction/docid"
function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
USE CASECDN
From public hash to private file
/m/m1gz4l4a1clb.jpg
a2d85de382c8672925174e731a1d2a89
Varnish middleware
storage(http based)
proprietarywebserver
mysql
1 2 3
4
5
middleware
storage(http based)
nginx + mod_cacheVarnish
1 2 3
4
function(doc) { return { "code": 302, "body": "See other", "headers": { "Location": doc.target } };}
{ "_id": "m1gz4l4a1clb", "_rev": "2-d7855bba9445f446c0b71f044e796c28", "content-type": "image/jpeg", "target": "http://10.16.10.1/a2d85de382c8672925174e731a1d2a89",}
LUCENE
[external]fti=/path/to/python /path/to/couchdb-lucene/tools/couchdb-external-hook.py
[httpd_db_handlers]_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
function(doc) { var ret=new Document(); ret.add(doc.message); ret.add(new Date(doc.datetime)); return ret;}
FULLTEXT/BY_MESSAGE/BY_QUERY.JS
curl -X GET"http://db.example.com/db/_fti/_design/app/by-query?q=foobar"
GEOCOUCHhttps://github.com/vmx/couchdb
GEOCOUCH
• Supports bbox, polygon, lines, etc.
• fork (for now)
• outputs via lists, georss possible
• directly useable by google maps
• can read GIS data
• combined with _changes makes interesting use cases possible
SPATIAL INDEXin spatial/points.js
function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }
SPATIAL INDEXin spatial/points.js
function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }
SPATIAL INDEXin spatial/points.js
function(doc) { if (doc.geo && doc.geo.latitude != '' && doc.geo.longitude != '') { emit( { type: "Point", coordinates: [ parseFloat(doc.geo.latitude), parseFloat(doc.geo.longitude) ] }, [ doc._id, doc ] ); } }
http://db.example.com/couchappdemo/_design/phpday/_spatial/points?bbox=0,0,180,90
Worldwide search
{"update_seq":3,"rows":[ { "id":"Verona", "bbox":[10.991748,45.438367,10.991748,45.438367], "value":[ "Verona", [10.991748,45.438367] ] }]}
bbox=west,south,east,north
https://github.com/couchbaselabs/iOS-Couchbase
COUCHDB FOR IOS
http://www.couchbase.com/downloads
Q?