1. #MongoDBdays #askAsya @asya999Building a Social Platformwith MongoDBAsya KamskyMongoDB Inc

2. Solutions Engineering Identify Popular Use Cases Directly from MongoDB Users Addressing "limitations" Go beyond documentation and blogs Create open source project Run it! 3. Social Status Feed 4. Status Feed 5. Status Feed 6. Socialitehttps://github.com/10gen-labs/socialite Open Source Reference Implementation Various Fanout Feed Models User Graph Implementation Content storage Configurable models and options REST API in Dropwizard (Yammer) https://dropwizard.github.io/dropwizard/ Built-in benchmarking 7. ArchitectureContent ProxyGraph Service Proxy 8. Pluggable Services Major components each have an interface see com.mongodb.socialite.services Configuration selects implementation to use ServiceManager organizes : Default implementations Lifecycle Binding configuration Wiring dependencies see com.mongodb.socialite.ServiceManager 9. Simple Interfacehttps://github.com/10gen-labs/socialiteGET /users/{user_id} Get a User by their IDDELETE /users/{user_id} Remove a user by their IDPOST /users/{user_id}/posts Send a message from this userGET /users/{user_id}/followers Get a list of followers of a userGET /users/{user_id}/followers_count Get the number of followers of a userGET /users/{user_id}/following Get the list of users this user is followingGET /users/{user_id}/following count Get the number of users this user followsGET /users/{user_id}/posts Get the messages sent by a userGET /users/{user_id}/timeline Get the timeline for this userPUT /users/{user_id} Create a new userPUT /users/{user_id}/following/{target} Follow a userDELETE /users/{user_id}/following/{target} Unfollow a user 10. Technical DecisionsUsertimelinecacheSchemaIndexing Horizontal Scaling 11. Operational TestingReal life validation of our choices.Most important criteria?User facing latencyLinear scaling of resources 12. Scaling Goals Realistic real-life-scale workload compared to Twitter, etc. Understanding of HW required containing costs Confirm architecture scales linearly without loss of responsiveness 13. ArchitectureContent ProxyGraph Service Proxy 14. Operational Testing All hosts in AWS Each service used its own DB, cluster or shards All benchmarks through `mongos` (sharded config) Used MMS monitoring for measuring throughput Used internal benchmarks for measuring latency Based volume tested on real life social metrics 15. Scaling for Infinite Content 16. ArchitectureContent ProxyGraph Service Proxy 17. Socialite Content Service System of record for all user content Initially very simple (no search) Mainly designed to support feed Lookup/indexed by _id and userid Time based anchors/pagination 18. Social Data Ages Fast Half life of most content is 1 day ! Popular content usually < 1 month Access to old data is rare 19. Content Service Index by userId, _id Shard by userId (or userId, _id) Supports user data as pass-through{"_id" : ObjectId("52aaaa14a0ee0d44323e623a"),"_a" : "user1","_m" : "this is a post,"_d" : {"geohash" : "6gkzwgjzn820"}} 20. Benchmarks 21. ArchitectureContent ProxyGraph Service Proxy 22. Graph Data - SocialJohn KatefollowsBobPeteRecommendation ? 23. Graph Data - PromotionalJohn KatefollowsBobPeteMentionAcmeSodaRecommendation ? 24. Graph Data - Everywhere Retail Complex product catalogues Product recommendation engines Manufacturing and Logistics Tracing failures to faulty component batches Determining fallout from supply interruption Healthcare Patient/Physician interactions 25. Design Considerations 26. The Tale of Two BiebersVS 27. Follower Churn Tempting to focus on scaling content Follow requests rival message send rates Twitter enforces per day follow limits 28. Edge Metadata Models friends/followers Requirements typically start simple Add Groups, Favorites, Relationships 29. Storing Graphs in MongoDB 30. Option One Embedding Edges 31. Embedded Edge Arrays Storing connections with user (popular choice)Most compact formEfficient for reads However. User documents grow Upper limit on degree (document size) Difficult to annotate (and index) edge{"_id" : "djw","fullname" : "Darren Wood","country" : "Australia","followers" : [ "jsr", "ian"],"following" : [ "jsr", "pete"]} 32. Embedded Edge Arrays Creating Rich Graph Information Can become cumbersome{"_id" : "djw","fullname" : "Darren Wood","country" : "Australia","friends" : [{"uid" : "jsr", "grp" : "school"},{"uid" : "ian", "grp" : "work"} ]}{"_id" : "djw","fullname" : "Darren Wood","country" : "Australia","friends" : [ "jsr", "ian"],"group" : [ school", work"]} 33. Option Two Edge Collection 34. Edge Collections Document per edge> db.followers.findOne(){"_id" : ObjectId(),"from" : "djw","to" : "jsr"} Very flexible for adding edge data> db.friends.findOne(){"_id" : ObjectId(),"from" : "djw","to" : "jsr","grp" : "work","ts" : Date("2013-07-10")} 35. Operational comparison Updates of embedded arrays grow non-linearly with number of indexed array elements Updating edge collection => inserts grows close to linearly with existing number of edges/user 36. Edge Insert Rate 37. Edge CollectionIndexing Strategies 38. Finding FollowersConsider our single follower collection :> db.followers.find({from : "djw"}, {_id:0, to:1}){"to" : "jsr"}Using index :{"v" : 1,"key" : { "from" : 1, "to" : 1 },"unique" : true,"ns" : "socialite.followers","name" : "from_1_to_1"}Covered indexwhen searching on"from" for allfollowersSpecify only ifmultiple edgescannot exist 39. Finding FollowingWhat about who a user is following?Can use a reverse covered index :{"v" : 1,"key" : { "from" : 1, "to" : 1 },"unique" : true,"ns" : "socialite.followers","name" : "from_1_to_1"}{"v" : 1,"key" : { "to" : 1, "from" : 1 },"unique" : true,"ns" : "socialite.followers","name" : "to_1_from_1"}Notice the flippedfield order here 40. Finding FollowingWait ! There is an issue with the reverse index..SHARDING !{"v" : 1,"key" : { "from" : 1, "to" : 1 },"unique" : true,"ns" : "socialite.followers","name" : "from_1_to_1"}{"v" : 1,"key" : { "to" : 1, "from" : 1 },"unique" : true,"ns" : "socialite.followers","name" : "to_1_from_1"}If we shard this collectionby "from", looking upfollowers for a specificuser is "targeted" to ashardTo find who the user isfollowing however, it mustscatter-gather the query toall shards 41. Dual Edge Collections 42. Dual Edge CollectionsWhen "following" queries are common Not always the case Consider overhead carefullyCan use dual collections storing One for each direction Edges are duplicated reversed Can be sharded independently 43. Edge Query Rate ComparisonNumber of shardsvsNumber of queriesFollowers collectionwith forward andreverse indexesTwo collections,followers, followingone index each1 10,000 10,0003 90,000 30,0006 360,000 60,00012 1,440,000 120,000 44. ArchitectureContent ProxyGraph Service Proxy 45. Feed Service Two main functions : Aggregating followed content for a user Forwarding users content to followers Common implementation models : Fanout on read Query content of all followed users on fly Fanout on write Add to cache of each users timeline for every post Various storage models for the timeline 46. Fanout On Read 47. Fanout On ReadProsSimple implementationNo extra storage for timelinesCons Timeline reads (typically) hit all shards Often involves reading more data than required May require additional indexing on Content 48. Fanout On Write 49. Fanout On WriteProsTimeline can be single document readDormant users easily excludedWorking set minimizedCons Fanout for large follower lists can be expensive Additional storage for materialized timelines 50. Fanout On Write Three different approaches Time buckets Size buckets Cache Each has different pros & cons 51. Timeline Buckets - TimeUpsert to time range buckets for each user> db.timed_buckets.find().pretty(){"_id" : {"_u" : "jsr", "_t" : 516935},"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"},{"_id" : ObjectId("...dd2"), "_a" : "ian", "_m" : "message from ian"}]}{"_id" : {"_u" : "ian", "_t" : 516935},"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"}]}{"_id" : {"_u" : "jsr", "_t" : 516934 },"_c" : [{"_id" : ObjectId("...da7"), "_a" : "ian", "_m" : "earlier from ian"}]} 52. Timeline Buckets - SizeMore complex, but more consistently sized> db.sized_buckets.find().pretty(){"_id" : ObjectId("...122"),"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"},{"_id" : ObjectId("...dd2"), "_a" : "ian", "_m" : "message from ian"},{"_id" : ObjectId("...da7"), "_a" : "ian", "_m" : "earlier from ian"}],"_s" : 3,"_u" : "jsr"}{"_id" : ObjectId("...011"),"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"}],"_s" : 1,"_u" : "ian"} 53. Timeline - CacheStore a limited cache, fall back to "fanout on read" Create single cache doc on demand with upsert Limit size of cache with $slice Timeout docs with TTL for inactive users> db.timeline_cache.find().pretty(){"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"},{"_id" : ObjectId("...dd2"), "_a" : "ian", "_m" : "message from ian"},{"_id" : ObjectId("...da7"), "_a" : "ian", "_m" : "earlier from ian"}],"_u" : "jsr"}{"_c" : [{"_id" : ObjectId("...dc1"), "_a" : "djw", "_m" : "message from daz"}],"_u" : "ian"} 54. Embedding vs Linking ContentEmbedded content for direct access Great when it is small, predictable in sizeLink to content, store only metadata Read only desired content on demand Further stabilizes cache document sizes> db.timeline_cache.findOne({_id" : "jsr"}){"_c" : [{"_id" : ObjectId("...dc1)},{"_id" : ObjectId("...dd2)},{"_id" : ObjectId("...da7)}],_id" : "jsr"} 55. Socialite Feed Service Implemented four models as plugins FanoutOnRead FanoutOnWrite Buckets (size) FanoutOnWrite Buckets (time) FanoutOnWrite - Cache Switchable by config Store content by reference or value Benchmark-able back to back 56. Benchmark by feed type 57. Benchmarking the Feed Biggest challenge: scaling the feed High cost of "fanout on write" Popular user posts => # operations: Content collection insert: 1 Timeline Cache: on average, 130+ cache documentupdates SCATTER GATHER (slowest shard determineslatency) 58. Benchmarking the Feed Timeline is different from content! "It's a Cache"IT CAN BE REBUILT! 59. Benchmarking the FeedIT CAN BE REBUILT! 60. Benchmarking the Feed Results over two weeks ran load with one million users ran load with ten million users used avg send rate 1K/s; 2K/s; reads 10K-20k/s 22 AWS c3.2xlarge servers (7.5GB RAM) 18 across six shards (3 content, 3 user graph) 4 mongos and app machines 2 c2x4xlarge servers (30GB RAM) timeline feed cache (six shards) 61. Summary 62. Socialitehttps://github.com/10gen-labs/socialite Real Working Implementation Implements All Components Configurable models and options Built-in benchmarking Questions? I will be at "Ask The Experts" this afternoon!https://github.com/10gen-labs/socialite 63. Thank You!https://github.com/10gen-labs/socialite