Upload
dangthuy
View
228
Download
0
Embed Size (px)
Citation preview
… not Vice Versa
Your Datastore Should Serve Your Application's Architecture
Tuesday, November 1, 11
•Web Apps need lightweight, high performance, highly scalable data stores.
•The cloud changes the expectations and performance profile of data stores.
•Focus of the new "NoSQL" paradigm of lots of inexpensive cloud servers and horizontal scalability; slow disks and random failures not as huge an impact
Data Access in the Right Context
Tuesday, November 1, 11
So We’ve Built an Application with a Database
How do we integrate that database with our application’s object hierarchy?
Tuesday, November 1, 11
I know! Let’s use an ORM!
Congratulations: Now we’ve got 2 problems!(or is it n+1?)
Tuesday, November 1, 11
Stuffing an object graph into a relational model is like fitting a square
peg into a round hole.
Tuesday, November 1, 11
Sure, we can use an ORM. But who are we really fooling?
... and who/what are we going to wake up next to in the morning?
Tuesday, November 1, 11
•Object Graphs are often orthogonal to relational models.
Make Your Data Work for You
Tuesday, November 1, 11
•Object Graphs are often orthogonal to relational models.
•Lifetimes of blood, sweat, tears and late night plots of homicide surround ORMS. They simply defer the pain (and make debugging harder)
Make Your Data Work for You
Tuesday, November 1, 11
•Object Graphs are often orthogonal to relational models.
•Lifetimes of blood, sweat, tears and late night plots of homicide surround ORMS. They simply defer the pain (and make debugging harder)
•… Is a generic query really the right answer for your app?)
Make Your Data Work for You
Tuesday, November 1, 11
This is a SQL Modelmysql> select * from book;+----+----------------------------------------------------------+| id | title |+----+----------------------------------------------------------+| 1 | The Demon-Haunted World: Science as a Candle in the Dark || 2 | Cosmos || 3 | Programming in Scala |+----+----------------------------------------------------------+3 rows in set (0.00 sec)
mysql> select * from bookauthor;+---------+-----------+| book_id | author_id |+---------+-----------+| 1 | 1 || 2 | 1 || 3 | 2 || 3 | 3 || 3 | 4 |+---------+-----------+5 rows in set (0.00 sec)
mysql> select * from author;+----+-----------+------------+-------------+-------------+---------------+| id | last_name | first_name | middle_name | nationality | year_of_birth |+----+-----------+------------+-------------+-------------+---------------+| 1 | Sagan | Carl | Edward | NULL | 1934 || 2 | Odersky | Martin | NULL | DE | 1958 || 3 | Spoon | Lex | NULL | NULL | NULL || 4 | Venners | Bill | NULL | NULL | NULL |+----+-----------+------------+-------------+-------------+---------------+4 rows in set (0.00 sec)
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data• We have to flatten our object out into three tables
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data• We have to flatten our object out into three tables
• 7 separate inserts just to add “Programming in Scala”
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data• We have to flatten our object out into three tables
• 7 separate inserts just to add “Programming in Scala”
• Once we turn the relational data back into objects ...
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data• We have to flatten our object out into three tables
• 7 separate inserts just to add “Programming in Scala”
• Once we turn the relational data back into objects ...• We still need to convert it to data for our frontend
Tuesday, November 1, 11
Joins are great and all ...
• Potentially organizationally messy• Structure of a single object is NOT immediately clear to someone glancing at the shell data• We have to flatten our object out into three tables
• 7 separate inserts just to add “Programming in Scala”
• Once we turn the relational data back into objects ...• We still need to convert it to data for our frontend• I don’t know about you, but I have better things to do with my time.
Tuesday, November 1, 11
The Same Data in MongoDB> db.books.find().forEach(printjson){ "_id" : ObjectId("4dfa6baa9c65dae09a4bbda3"), "title" : "The Demon-Haunted World: Science as a Candle in the Dark", "author" : [ { "first_name" : "Carl", "last_name" : "Sagan", "middle_name" : "Edward", "year_of_birth" : 1934 } ]}{ "_id" : ObjectId("4dfa6baa9c65dae09a4bbda4"), "title" : "Cosmos", "author" : [ { "first_name" : "Carl", "last_name" : "Sagan", "middle_name" : "Edward", "year_of_birth" : 1934 } ]}
Tuesday, November 1, 11
The Same Data in MongoDB(Part 2)
{ "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" } ]}
Tuesday, November 1, 11
Access to the embedded objects is integral> db.books.find({"author.first_name": "Martin", "author.last_name": "Odersky"}){ "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "title" : "Programming in Scala", "author" : [ { "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "first_name" : "Bill", "last_name" : "Venners" }] }
Tuesday, November 1, 11
As is manipulation of the embedded data> db.books.update({"author.first_name": "Bill", "author.last_name": "Venners"},... {$set: {"author.$.company": "Artima, Inc."}})> db.books.update({"author.first_name": "Martin", "author.last_name": "Odersky"},... {$set: {"author.$.company": "Typesafe, Inc."}})> db.books.findOne({"title": /Scala$/}){ "_id" : ObjectId("4dfa6baa9c65dae09a4bbda5"), "author" : [ { "company" : "Typesafe, Inc.", "first_name" : "Martin", "last_name" : "Odersky", "nationality" : "DE", "year_of_birth" : 1958 }, { "first_name" : "Lex", "last_name" : "Spoon" }, { "company" : "Artima, Inc.", "first_name" : "Bill", "last_name" : "Venners" } ], "title" : "Programming in Scala"}
Tuesday, November 1, 11
NoSQL Really Means...non-relational, next-generation
operational datastores and databases
Tuesday, November 1, 11
NoSQL Really Means...non-relational, next-generation
operational datastores and databases... Let’s focus on the “non-relational” bit.
Tuesday, November 1, 11
Less Suited For
highly transactional applications
problems which require SQL
ad-hoc business intelligence
Tuesday, November 1, 11
•(200 gigs of MongoDB files creates 200 gigs of virtual memory)
•OS controls what data in RAM
•When a piece of data isn't found, a page fault occurs (Expensive + Locking!)
•OS goes to disk to fetch the data
•Compare this to the normal trick of sticking a poorly managed memcached cluster in front of MySQL
Operating System map files on the Filesystem to Virtual Memory
Tuesday, November 1, 11
_idif not specified drivers will add default:
ObjectId("4bface1a2231316e04f3c434")timestamp
machine idprocess id
counter
http://www.mongodb.org/display/DOCS/Object+IDs
Tuesday, November 1, 11
BSON Encoding{ _id: ObjectId(XXXXXXXXXXXX), hello: “world”}
\x27\x00\x00\x00\x07_id\x00X X X X X X X X X X X X X X\x02 h e l l o \x00\x06\x00\x00\x00 w o r l d \x00\x00
http://bsonspec.org
Tuesday, November 1, 11
Scaling
•Operations/sec go up
•Storage needs go up
•Capacity
•IOPs
•Complexity goes up
•Caching
Tuesday, November 1, 11
• Optimization & Tuning• Schema & Index Design• O/S tuning• Hardware configuration
• Vertical scaling• Hardware is expensive• Hard to scale in cloud
How do you scale now?
$$$
throughput
Tuesday, November 1, 11
Write scaling - add Shards
write
read
shard1
node_c1
node_b1
node_a1
shard2
node_c2
node_b2
node_a2
Tuesday, November 1, 11
Write scaling - add Shards
write
read
shard1
node_c1
node_b1
node_a1
shard2
node_c2
node_b2
node_a2
shard3
node_c3
node_b3
node_a3
Tuesday, November 1, 11
How MongoDB Sharding works> db.runCommand( { addshard : "shard1" } );> db.runCommand( { shardCollection : “mydb.blogs”, key : { age : 1} } )
-∞ +∞
•Range keys from -∞ to +∞ •Ranges are stored as “chunks”
Tuesday, November 1, 11
How MongoDB Sharding works
> db.posts.save( {age:40} )
-∞ +∞
-∞ 40 41 +∞
•Data in inserted•Ranges are split into more “chunks”
Tuesday, November 1, 11
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )
-∞ +∞
-∞ 40 41 +∞
41 50 51 +∞ •More Data in inserted•Ranges are split into more“chunks”
Tuesday, November 1, 11
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )
-∞ +∞
-∞ 40 41 +∞
41 50 51 +∞
61 +∞ 51 60
Tuesday, November 1, 11
-∞ +∞
41 +∞
51 +∞
How MongoDB Sharding works
> db.posts.save( {age:40} )> db.posts.save( {age:50} )> db.posts.save( {age:60} )
-∞ 40
41 50
61 +∞ 51 60
Tuesday, November 1, 11
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
-∞ 40
41 50
61 +∞
51 60
Tuesday, November 1, 11
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
-∞ 40
41 50
61 +∞
51 60
shard1
Tuesday, November 1, 11
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
-∞ 40
41 50
61 +∞
51 60
shard1 shard2
Tuesday, November 1, 11
How MongoDB Sharding works
> db.runCommand( { addshard : "shard2" } );
-∞ 40
41 50
61 +∞
51 60
shard1 shard2
> db.runCommand( { addshard : "shard3" } );
shard3
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2
Member 3
•Set is made up of 2 or more nodes
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2PRIMARY
Member 3
•Election establishes the PRIMARY•Data replication from PRIMARY to SECONDARY
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2DOWN
Member 3
negotiate new master
•PRIMARY may fail•Automatic election of new PRIMARY
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2DOWN
Member 3PRIMARY
•New PRIMARY elected•Replication Set re-established
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2RECOVERING
Member 3PRIMARY
•Automatic recovery
Tuesday, November 1, 11
How MongoDB Replication works
Member 1
Member 2
Member 3PRIMARY
•Replication Set re-established
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python• Dynamic language with flexible syntax; Dynamic database with flexible schemas
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python• Dynamic language with flexible syntax; Dynamic database with flexible schemas• Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together.
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python• Dynamic language with flexible syntax; Dynamic database with flexible schemas• Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together.
• Java Driver ... No Scala sugar or tricks
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python• Dynamic language with flexible syntax; Dynamic database with flexible schemas• Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together.
• Java Driver ... No Scala sugar or tricks• scamongo (pre-lift): ODM (ORMey) or JSON tools
Tuesday, November 1, 11
“ This s%#! doesn’t work the way I want it to”‘mongo-scala-wrappers’ Is Born
• Learned MongoDB from Python• Dynamic language with flexible syntax; Dynamic database with flexible schemas• Tooling for MongoDB + Scala was limited or unsuited. Mostly focused on ODM. None of what I loved about Scala or MongoDB possible together.
• Java Driver ... No Scala sugar or tricks• scamongo (pre-lift): ODM (ORMey) or JSON tools• mongo-scala-driver: A little syntactic sugar but mostly ODM; didn’t “get” it
Tuesday, November 1, 11
MongoDB from Python 1 doc = { 2 "name": { 3 "first": "Brendan", 4 "last": "McAdams" 5 }, 6 "email": "[email protected]", 7 "twitter": "@rit", 8 "age": 32, 9 "interests": ["scala", "python", "akka", "mongodb"]10 }11 12 age = doc['age']13 14 type(age) # <type 'int'>15 16 doc['interests'][1] # 'python'17 type(doc['interests']) # <type 'list'>
Tuesday, November 1, 11
MongoDB from Java (or Scala, pre-Casbah)
1 val b = BasicDBObjectBuilder.start() 2 3 b.add("name", new BasicDBObject("first", "Brendan").append("last", "McAdams")) 4 b.add("email", "[email protected]") 5 b.add("twitter", "@rit") 6 b.add("age", 32) 7 8 val interests = new BasicDBList() 9 interests.add("scala")10 interests.add("python")11 interests.add("akka")12 interests.add("mongodb")13 14 b.add("interests", interests)15 16 val doc = b.get()17 18 val age = doc("age") // AnyRef = 3219 20 doc("interests")(1)21 /* error: AnyRef does not take parameters22 doc("interests")(1) 23 */24 25 doc("interests").asInstanceOf[BasicDBList](1) // java.lang.Object = python26
Tuesday, November 1, 11
Type Safety and CompilationShouldn’t Necessitate Syntactic
Suicide
• There’s absolutely nothing wrong with that Syntax... For Java.
Tuesday, November 1, 11
Type Safety and CompilationShouldn’t Necessitate Syntactic
Suicide
• There’s absolutely nothing wrong with that Syntax... For Java.• Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB.
Tuesday, November 1, 11
Type Safety and CompilationShouldn’t Necessitate Syntactic
Suicide
• There’s absolutely nothing wrong with that Syntax... For Java.• Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible
Tuesday, November 1, 11
Type Safety and CompilationShouldn’t Necessitate Syntactic
Suicide
• There’s absolutely nothing wrong with that Syntax... For Java.• Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible• Self Imposed Limitation: Don’t reinvent the wheel. Java Driver’s Network Layers, BSON Encoding, etc work great.
Tuesday, November 1, 11
Type Safety and CompilationShouldn’t Necessitate Syntactic
Suicide
• There’s absolutely nothing wrong with that Syntax... For Java.• Scala is expressive, fluid and beautiful; so is (IMHO) MongoDB. • My goal: Teach Scala to be as close to Python / Mongo Shell as possible• Self Imposed Limitation: Don’t reinvent the wheel. Java Driver’s Network Layers, BSON Encoding, etc work great. • Just add Syntactic Sugar!
Tuesday, November 1, 11
Today ... 1 val doc = MongoDBObject( 2 "name" -> MongoDBObject("first" -> "Brendan", "last" -> "McAdams"), 3 "email" -> "[email protected]", 4 "twitter" -> "@rit", 5 "age" -> 32, 6 "interests"-> Seq("scala", "python", "akka", "mongodb") 7 ) 8 // Full support also for Scala 2.8 Collections' Factory / Builders 9 10 val age = doc.getAs[Int]("age") // Option[Int] = Some(32)11 12 val interests = doc.as[Seq[_]]("interests") // Seq[java.lang.String] = List(scala, python, akka, mongodb)13 14 interests(2) // akka15 16 // Experimental Dynamic support in 2.9 lets you do doc.age.typed[Int]
Tuesday, November 1, 11
It’s All About Implicits
• Casbah’s core tenet is to reuse what works, and improve (not replace) what doesn’t...
• Casbah provides a series of wrapper classes (and in some cases, companion objects) which proxy the “core” Java driver classes to provide scala functionality.
• Generally provide “Scala-esque” wrappers to the MongoDB Java objects whereever possible. • These make sure to make iterable things Iterable, Cursors implement Iterator, DBObjects act like Scala Maps, etc.
Tuesday, November 1, 11
Connecting to MongoDB// Connect to default - localhost, 27017
val mongoConn = MongoConnection()// mongoConn: com.mongodb.casbah.MongoConnection
// connect to "mongodb01" host, default portval mongoConn = MongoConnection("mongodb01")// mongoConn: com.mongodb.casbah.MongoConnection
// connect to "mongodb02" host, port 42017val mongoConn = MongoConnection("mongodb02", 42017)// mongoConn: com.mongodb.casbah.MongoConnection
// connect to a replica setval mongoConn = MongoConnection(List(
new ServerAddress(“rs01”, 27017), new ServerAddress(“rs02”, 27017)
))// mongoConn: com.mongodb.casbah.MongoConnection
Tuesday, November 1, 11
Getting a Collection
val mongoDB = mongoConn("casbah_test")// mongoDB: com.mongodb.casbah.MongoDB = casbah_test
val mongoColl = mongoConn("casbah_test")("test_data")// mongoColl: com.mongodb.casbah.MongoCollection = MongoCollection()
Tuesday, November 1, 11
Builders for DBObject
val builder = MongoDBObject.newBuilderbuilder += "foo" -> "bar"builder += "x" -> "y"builder += ("pie" -> 3.14)builder += ("spam" -> "eggs", "mmm" -> "bacon")val newObj = builder.result// newObj: com.mongodb.DBObject =/* { "foo" : "bar" , "x" : "y" , "pie" : 3.14 , "spam" : "eggs" , "mmm" : "bacon"} */
Tuesday, November 1, 11
Queryingval mongoColl = MongoConnection()("casbah_test")("test_data")
val user1 = MongoDBObject("user" -> "bwmcadams", "email" -> "~~brendan~~<AT>10genDOTcom")val user2 = MongoDBObject("user" -> "someOtherUser")mongoColl += user1mongoColl += user2mongoColl.find()// com.mongodb.casbah.MongoCursor =// MongoCursor{Iterator[DBObject] with 2 objects.}
for { x <- mongoColl} yield x/* Iterable[com.mongodb.DBObject] = List( { "_id" : { "$oid" : "4c3e2bec521142c87cc10fff"} , "user" : "bwmcadams" , "email" : "~~brendan~~<AT>10genDOTcom"}, { "_id" : { "$oid" : "4c3e2bec521142c87dc10fff"} , "user" : "someOtherUser"} ) */
Tuesday, November 1, 11
Querying
val q = MongoDBObject("user" -> "someOtherUser")val cursor = mongoColl.find(q)// cursor: com.mongodb.casbah.MongoCursor =// MongoCursor{Iterator[DBObject] with 1 objects.}
val user = mongoColl.findOne(q)// user: Option[com.mongodb.DBObject] =/* Some({ "_id" : { "$oid" : "4c3e2bec521142c87dc10fff"} , "user" : "someOtherUser"}) */
Tuesday, November 1, 11
Query DSLval q = "email" $exists true
// q: (String, com.mongodb.DBObject) =// (email,{ "$exists" : true})val users = for (x <- mongoColl.find(q)) yield xassert(users.size == 1)
val rangeColl = mongoConn("casbah_test")("rangeTests")rangeColl += MongoDBObject("foo" -> 5)rangeColl += MongoDBObject("foo" -> 30)rangeColl += MongoDBObject("foo" -> 35)rangeColl += MongoDBObject("foo" -> 50)rangeColl += MongoDBObject("foo" -> 60)rangeColl += MongoDBObject("foo" -> 75)
rangeColl.find("foo" $lt 50 $gt 5)// com.mongodb.casbah.MongoCursor =// MongoCursor{Iterator[DBObject] with 2 objects.}
for (x <- rangeColl.find("foo" $lt 50 $gt 5) ) println(x)// { "_id" : { "$oid" : "4c42426f30daeca8efe48de8"} , "foo" : 30}// { "_id" : { "$oid" : "4c424bca9130daeca8f0e48de8"} , "foo" : 35}for (x <- rangeColl.find("foo" $lte 50 $gt 5) ) println(x)// { "_id" : { "$oid" : "4c42426f30daeca8efe48de8"} , "foo" : 30}// { "_id" : { "$oid" : "4c4249e0a130daeca8f0e48de8"} , "foo" : 35}// { "_id" : { "$oid" : "4c42427330daeca8f1e48de8"} , "foo" : 50}
Tuesday, November 1, 11
Query DSL
val q: DBObject = ("foo" $lt 50 $gt 5) ++ ("bar" $gte 9)/* com.mongodb.DBObject = { "foo" : { "$lt" : 50 , "$gt" : 5} , "bar" : { "$gte" : 9}} */
val qMix = ("foo" $gte 5) ++ ("baz" -> 5) ++ ("x" -> "y")/* qMix: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : { "$gte" : 5} , "baz" : 5 , "x" : "y"} */
val qMix2 = ("foo" $gte 5 $lte 10) ++ ("baz" -> 5) ++ ("x" -> "y") ++ ("n" -> "r")/* qMix2: com.mongodb.casbah.commons.Imports.DBObject = { "foo" : { "$gte" : 5 , "$lte" : 10} , "baz" : 5 , "n" : "r" , "x" : "y"} */
Tuesday, November 1, 11
Briefly, GridFS
import com.mongodb.casbah.gridfs.Imports._
val gridfs = GridFS(mongoConn) // creates a GridFS handle on ``fs``
val logo = new FileInputStream("casbah-gridfs/src/test/resources/powered_by_mongo.png")gridfs(logo) { fh => fh.filename = "powered_by_mongo.png" fh.contentType = "image/png"}
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO• NOT (contrary to popular panic/confusion) a replacement for Casbah
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO• NOT (contrary to popular panic/confusion) a replacement for Casbah
• Focused more on framework support than userspace
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO• NOT (contrary to popular panic/confusion) a replacement for Casbah
• Focused more on framework support than userspace
• Will likely offer optional synchronous and asynchronous hammersmith module for casbah-core, with Java driver as casbah-core-classic
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO• NOT (contrary to popular panic/confusion) a replacement for Casbah
• Focused more on framework support than userspace
• Will likely offer optional synchronous and asynchronous hammersmith module for casbah-core, with Java driver as casbah-core-classic
• Working on sharing as much code as possible between Hammersmith & Casbah for MongoDBObject, etc.
Tuesday, November 1, 11
epilogue• Casbah lives on and will continue to evolve, but it also has a younger brother/cousin• “Hammersmith”, purely asynchronous, purely Scala and a distillation of ~2 years of MongoDB knowledge• Only Java is the BSON serialization; still no excuse for reinventing the wheel• Netty for now, but probably will end up as pure NIO• NOT (contrary to popular panic/confusion) a replacement for Casbah
• Focused more on framework support than userspace
• Will likely offer optional synchronous and asynchronous hammersmith module for casbah-core, with Java driver as casbah-core-classic
• Working on sharing as much code as possible between Hammersmith & Casbah for MongoDBObject, etc.
• Porting casbah-query to target Hammersmith (as well as Lift)
Tuesday, November 1, 11
hammersmith 1 def iterateSimpleCursor(conn: MongoConnection) = { 2 var x = 0 3 conn("bookstore").find("inventory")(Document.empty, Document.empty)((cursor: Cursor) => { 4 for (doc <- cursor) { 5 x += 1 6 } 7 }) 8 9 x must eventually (be_==(336))10 }11 12 def iterateComplexCursor(conn: MongoConnection) = {13 var x = 014 conn("bookstore").find("inventory")(Document.empty, Document.empty)((cursor: Cursor) => {15 def next(op: Cursor.IterState): Cursor.IterCmd = op match {16 case Cursor.Entry(doc) => {17 x += 118 if (x < 100) Cursor.Next(next) else Cursor.Done19 }20 case Cursor.Empty => {21 if (x < 100) Cursor.NextBatch(next) else Cursor.Done22 }23 case Cursor.EOF => {24 Cursor.Done25 }26 }27 Cursor.iterate(cursor)(next)28 })29 30 x must eventually(5, 5.seconds) (be_==(100))32 }32
Tuesday, November 1, 11
33 def insertWithSafeImplicitWriteConcern(conn: MongoConnection) = {34 val mongo = conn("testHammersmith")("test_insert")35 implicit val safeWrite = WriteConcern.Safe36 mongo.dropCollection()(success => {37 log.info("Dropped collection... Success? " + success)38 })39 var id: Option[AnyRef] = null40 var ok: Option[Boolean] = None41 42 val handler = RequestFutures.write((result: Either[Throwable, (Option[AnyRef], WriteResult)]) => {43 result match {44 case Right((oid, wr)) => {45 ok = Some(true)46 id = oid }47 case Left(t) => {48 ok = Some(false)49 log.error(t, "Command Failed.")50 }51 }52 })53 mongo.insert(Document("foo" -> "bar", "bar" -> "baz"))(handler)54 ok must eventually { beSome(true) }55 id must not (beNull.eventually)56 // TODO - Implement 'count'57 var doc: BSONDocument = null58 mongo.findOne(Document("foo" -> "bar"))((_doc: BSONDocument) => {59 doc = _doc60 })61 doc must not (beNull.eventually)62 doc must eventually (havePairs("foo" -> "bar", "bar" -> "baz"))63 }64 65 // vim: set ts=2 sw=2 sts=2 et:
hammersmith
Tuesday, November 1, 11
type classes, anyone?/** * Type class base for anything you want to be serialized or deserialized */trait SerializableBSONObject[T] {
def encode(doc: T, out: OutputBuffer)
def encode(doc: T): Array[Byte]
def decode(in: InputStream): T
def decode(bytes: Seq[Array[Byte]]): Seq[T] = for (b <- bytes) yield decode(b)
def decode(b: Array[Byte]): T = decode(new ByteArrayInputStream(b))
/** * These methods are used to validate documents in certain cases. * They will be invoked by the system at the appropriate times and you must * implement them in a manner appropriate for your object to ensure proper mongo saving. */ def checkObject(doc: T, isQuery: Boolean = false): Unit
def checkKeys(doc: T): Unit
/** * Checks for an ID and generates one, returning a new doc with the id. * The new doc may be a mutation of the old doc, OR a new object * if the old doc was immutable. */ def checkID(doc: T): T
def _id(doc: T): Option[AnyRef]
}
Tuesday, November 1, 11
type classes, anyone?
def insert[T](doc: T, validate: Boolean = true)(callback: WriteRequestFuture)(implicit concern: WriteConcern = this.writeConcern, m: SerializableBSONObject[T]) { db.insert(name)(doc, validate)(callback) }
def batchInsert[T](docs: T*)(callback: WriteRequestFuture)(implicit concern: WriteConcern = this.writeConcern, m: SerializableBSONObject[T]) { db.batchInsert(name)(docs: _*)(callback) }
def update[Upd](query: BSONDocument, update: Upd, upsert: Boolean = false, multi: Boolean = false)(callback: WriteRequestFuture)(implicit concern: WriteConcern = this.writeConcern, uM: SerializableBSONObject[Upd]) { db.update(name)(query, update, upsert, multi)(callback) }
def save[T](obj: T)(callback: WriteRequestFuture)(implicit concern: WriteConcern = this.writeConcern, m: SerializableBSONObject[T]) { db.save(name)(obj)(callback) }
Tuesday, November 1, 11
type classes, anyone? def find[Qry <: BSONDocument, Flds <: BSONDocument](query: Qry = Document.empty, fields: Flds = Document.empty, numToSkip: Int = 0, batchSize: Int = 0)(callback: CursorQueryRequestFuture)(implicit concern: WriteConcern = this.writeConcern) { db.find(name)(query, fields, numToSkip, batchSize)(callback) }
sealed trait RequestFuture { type T
val body: Either[Throwable, T] => Unit
def apply(error: Throwable) = body(Left(error))
def apply[A <% T](result: A) = body(Right(result.asInstanceOf[T]))
protected[futures] var completed = false}
sealed trait QueryRequestFuture extends RequestFuture { type DocType val decoder: SerializableBSONObject[DocType]}
trait CursorQueryRequestFuture extends QueryRequestFuture { type T <: Cursor[DocType]}
def query[A: SerializableBSONObject](f: Either[Throwable, Cursor[A]] => Unit) = new CursorQueryRequestFuture { type DocType = A type T = Cursor[A] val body = f val decoder = implicitly[SerializableBSONObject[A]] override def toString = "{CursorQueryRequestFuture}" }
Tuesday, November 1, 11
Context Bounds for fun and profit
• Context Boundaries let you skip over certain implicit arguments:
def $type[A: BSONType: Manifest]
Tuesday, November 1, 11
Context Bounds for fun and profit
• Context Boundaries let you skip over certain implicit arguments:
def $type[A: BSONType: Manifest]
• Is the equivalent of coding:
def $type[A](implicit evidence$1: BSONType[A],implicit evidence$2: Manifest[A])
Tuesday, November 1, 11
Pop Quiz!• What the hell does this code do?
def insert[A <% DBObject](docs: Traversable[A], writeConcern: WriteConcern) = { val b = new scala.collection.mutable.ArrayBuilder.ofRef[DBObject] b.sizeHint(docs.size) for (x <- docs) b += x underlying.insert(b.result, writeConcern) }
Tuesday, November 1, 11
Pop Quiz!• What the hell does this code do?
def insert[A <% DBObject](docs: Traversable[A], writeConcern: WriteConcern) = { val b = new scala.collection.mutable.ArrayBuilder.ofRef[DBObject] b.sizeHint(docs.size) for (x <- docs) b += x underlying.insert(b.result, writeConcern) }
• Answer: This is a “View Boundary”• Code “flattens” at compile time to something like this:def insert[A](docs: Traversable[A], writeConcern: WriteConcern)(implicit ev: A => DBObject) = { val b = new scala.collection.mutable.ArrayBuilder.ofRef[DBObject] b.sizeHint(docs.size) for (x <- docs) b += ev(x) underlying.insert(b.result, writeConcern)}
Tuesday, November 1, 11
@mongodb
conferences, appearances, and meetupshttp://www.10gen.com/events
http://bit.ly/mongoO Facebook | Twitter | LinkedIn
http://linkd.in/joinmongo
download at mongodb.org
mms.10gen.com (MongoDB monitoring! free! Awesome!)
We’re Hiring [email protected]
(twitter: @rit)
Tuesday, November 1, 11