(PFC308) How Dropbox Scales Massive Workloads Using Amazon SQS | AWS re:Invent 2014

  • View
    1.224

  • Download
    1

Embed Size (px)

DESCRIPTION

In this session, learn how Dropbox scales to provide one of the largest cloud storage and file sharing services in the world. Hear how Dropbox leverages Amazon EC2 to run varied workloads including thumbnail generation and document prevent, as well as document indexing to support full-text search. Dropbox presents ''Livefill'' - a generic framework built on top of Amazon SQS. Livefill enables them to trigger customizable data-processing workloads on data stored in Amazon S3 and helps them support more than 200,000 workload requests per second, spread across thousands of machines.

Text of (PFC308) How Dropbox Scales Massive Workloads Using Amazon SQS | AWS re:Invent 2014

  • 1. November 12, 2014 Akhil Gupta, Head of Infrastructure, Dropbox

2. web app server Dropbox data center Databases S3 Bucket web app server 3. Dropbox data center Databases web app server web app server S3 Bucket 4. Dropbox data center Databases web app server web app server S3 Bucket 5. Dropbox data center Databases web app server web app server S3 Bucket 6. Dropbox data center Databases web app server web app server S3 Bucket 7. Dropbox data center Databases web app server web app server S3 Bucket 8. Dropbox data center Databases web app server web app server S3 Bucket 9. Dropbox data center Databases web app server web app server S3 Bucket 10. Dropbox data center Databases web app server web app server S3 Bucket 11. Dropbox data center Databases web app server web app server S3 Bucket 12. Dropbox data center Databases web app server web app server S3 Bucket 13. Dropbox data center Databases web app server web app server S3 Bucket 14. Dropbox data center Databases web app server web app server S3 Bucket 15. Dropbox data center Databases web app server web app server S3 Bucket 16. Dropbox data center Databases S3 Bucket web app server Livefill Server Livefill Extractor Livefill Extractor Virtual Cluster #1 Virtual Cluster #3 Virtual Cluster #2 In Queue Out Queue 17. Dropbox data center Databases S3 Bucket web app server Livefill Server Livefill Extractor Livefill Extractor Virtual Cluster #1 Virtual Cluster #3 Virtual Cluster #2 Process File A InQueue Out Queue 18. S3 Bucket Livefill Extractor Virtual Cluster #1 InQueue Out Queue 19. S3 Bucket Virtual Cluster #1 Read File A 20. S3 Bucket Fetch File A Handler 1Handler 2 21. Dropbox data center Databases S3 Bucket web app server Livefill Server Livefill Extractor Livefill Extractor Virtual Cluster #1 Virtual Cluster #3 Virtual Cluster #2 Store extracted metadata InQueue Out Queue 22. http://bit.ly/awsevals