Embed Size (px)
DESCRIPTION
Squire is an app for iPad that provides suggestions of movies and TV shows, based on the user’s preferences. There are also some social features, for instance a user can suggest a movie or TV show to one of his/her contacts. We have developed the backend and web services that support the Squire iPad app. The recommendations system was written in Scala and everything else was developed in Ruby. This architecture tries to combine the best features of the different languages. The Ruby API is easy to maintain, while the recommendations engine written in Scala and Neo4j is powerful, fast and able to handle heavy computations in a reasonable time.
Citation preview
Squire: A polyglot application combining Neo4j, MongoDB, Ruby and Scala
Alberto Perdomo@albertoperdomo
FOSDEM 2013Graph Processing DevRoom
About me
About me
Co-Founder of Aentos
About me
Co-Founder of Aentos
Living my dream* (CNY)
About me
Co-Founder of Aentos
Living my dream* (CNY)
Web developer
d
Features
Features
★ Extensive movies & TV-shows data
Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings
Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings
★ Suggestions by users
Features
★ Extensive movies & TV-shows data
★ Info & price on streaming & rent services:
iTunes, Amazon, Hulu, Netflix...
★ Social features (contacts, comments)
★ Ratings
★ Suggestions by users
★ System recommendations
UnconventionalSystem
Architecture*
Ingredients
✔ MongoDB
Ingredients
✔ MongoDB
✔ Neo4j
Ingredients
✔ MongoDB
✔ Neo4j
✔ Ruby
Ingredients
✔ MongoDB
✔ Neo4j
✔ Ruby
✔ Scala
Ingredients
⚛ System Overview ⚛iPad
Client
Main API(Ruby)
Recommend. Engine(Scala)
MongoDB Neo4jUsersRatingsMovies and TV shows
All data
API: Ruby + MongoDB
API: Ruby + MongoDB
⚛ REST+JSON
API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts
API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts
⚛ Ratings
API: Ruby + MongoDB
⚛ REST+JSON
⚛ Accounts & sessions
⚛ Movies & TV-Shows
⚛ Contacts
⚛ Ratings
⚛ User suggestions & recommendations
Recommendations: Scala + Neo4j
Recommendations: Scala + Neo4j
⚛ Movies & TV-shows (IDs & rel.)
Recommendations: Scala + Neo4j
⚛ Movies & TV-shows (IDs & rel.)
⚛ Users
Recommendations: Scala + Neo4j
⚛ Movies & TV-shows (IDs & rel.)
⚛ Users
⚛ Ratings
Communication (RabbitMQ)
iPad Client
Main API(Ruby)
Recommend. Engine(Scala)
MongoDB Neo4j
RabbitMQ + JSON
JSON array of IDs
GET /suggestions
Query params in JSON
UsersRatingsMovies and TV shows
JSON array of data
Retrieve datafor response Collect IDs of suggestions
All dataGraph reset*
Issues
Issues
☠ API server waiting for query in Neo4j
Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data
Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data
☠ Transform data
Issues
☠ API server waiting for query in Neo4j
☠ Query MongoDB for add. data
☠ Transform data
☠ Ruby server is blocking
Communication (DB)
iPad Client
Main API(Ruby)
Recommend. Engine(Scala)
MongoDB Neo4j
GET /suggestions
UsersRatingsMovies and TV shows
Redirect to Recomm.Eng. w/ hashed params
Retrieve datafor query params Collect IDs of suggestions
All data
JSON array of suggested results
Retrieve data for response
Follow redirect link
Graph reset*
Improvements
Improvements
✌ No more long blocking requests in Ruby server
Improvements
✌ No more long blocking requests in Ruby server
✌ Move heavy lifting to Spray/Scala (async)
Improvements
✌ No more long blocking requests in Ruby server
✌ Move heavy lifting to Spray/Scala (async)
✌ Add cache layer
Improvements
✌ No more long blocking requests in Ruby server
✌ Move heavy lifting to Spray/Scala (async)
✌ Add cache layer
Additional Cache Layer
Additional Cache Layer
* Read-Through
Additional Cache Layer
* Read-Through
* Stores transformed data
Additional Cache Layer
* Read-Through
* Stores transformed data
* Implemented using spray-caching
Total Response Time
40s → <1s
Delta Updates
Delta Updates
* API: New ratings in queue collection (MongoDB)
Delta Updates
* API: New ratings in queue collection (MongoDB)
* Recommendations:Backend checks for updates in queue and applies them in graph
Delta Updates
iPad Client
Main API(Ruby)
Recommend. Engine(Scala)
MongoDB Neo4j
POST /rating
UsersRatingsMovies and TV shows
Store rating +Put rating in queue Collect IDs of suggestions
Check if updatesin queue
GET /suggestions
Apply updatesif pending
Retrieve data for response
(Re)loading data in graph
(Re)loading data in graph
1. Create new Neo4j instance
(Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
(Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap
(Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap
★ Allows for massive updates in the primary
database
(Re)loading data in graph
1. Create new Neo4j instance
2. Import data from MongoDB
3. Hot swap
★ Allows for massive updates in the primary
database
★ Used only for content-related data and user
Graph DB as an index for relations*
Graph DB as an index for relations*powerful queries on relations
Graph DB as an index for relations*powerful queries on relations
Graph DB as an index for relations*powerful queries on relations
⁂ like full text search for strings
Design decisions: Graph
Design decisions: Graph
✍ Copy subsets of data in Neo4j
Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no downtime*
Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no downtime*
✍ Massive updates blazingly fast
Design decisions: Graph
✍ Copy subsets of data in Neo4j
✍ Rebuild graph with no downtime*
✍ Massive updates blazingly fast
✍ Less critical pieces
The right tools, for us! ✌
API in Ruby & MongoDB
The right tools, for us! ✌
API in Ruby & MongoDBMaintanability, flexibility, libraries,
The right tools, for us! ✌
API in Ruby & MongoDBMaintanability, flexibility, libraries,
schema-free!, development speed
The right tools, for us! ✌
Recommendations
The right tools, for us! ✌
RecommendationsScala: Performance + good interface for Neo4j
The right tools, for us! ✌
RecommendationsScala: Performance + good interface for Neo4j
Neo4j: Graph problem
Thanks! ☃
Alberto Perdomo@albertoperdomo
❤ Credits ❤
Entering Hyperspace by Éole Wind
