Upload
yoav-avrahami
View
820
Download
1
Tags:
Embed Size (px)
Citation preview
Managing 100TB of small files…
Prospero Media Storage
IGT –
July 2011
Event
Numbers
• 70TB used space• 700 million files• 200GB and 250,000 files uploaded every day• 1200Mbps bandwidth throughput in peak• 180TB of data is being served out monthly• 3700 Hits per second in peak • 40 storage node servers – 300TB raw space• $0.13 per GB
Motivation
• Web 2.0 content serving paradigm shift– Too many files
• 12M users x 1 file = very long tail– Too many connections
• 1M users + keepalive = 1M connections– Living with modern content in web 2.0
• 1 file x (thumbnail + iPhone + Mac) = 3 file copies
Traditional Architecture
Centralized Storage (NAS, SAN, DAS etc.)
HT TP
IO IO IO IO
Traditional Architecture
Centralized Storage (NAS, SAN, DAS etc.)
HT TP – TOO MANY CONNECTIONS
IO IO IO IO
Traditional Architecture
Centralized Storage (NAS, SAN, DAS etc.)
HT TP
IO IO IO IOIO IOIO
Traditional Architecture
To o m u c h I O
HT TP
IO IO IOIO IOIOIO
Traditional Architecture
Centralized Storage (NAS, SAN, DAS etc.)
HT TP
IO IO IO IOIO IOIO
Cache
“There are only two hard things in Computer Science: cache invalidation and naming things”.
-- Tim Bray quoting Phil Karlton
Architecture goals
• Symmetric identical server nodes– Simplified management and scaling– Linear scaling out
• No functional / role servers– No single point of failure– No performance bottlenecks
• Multiple datacenters support– DRP support– Geo load distribution
Meet Prospero
• Distributed Web content storage system
• Full blown HTTP support
• Runs on low cost commodity hardware
• Adjustable file level replication controls redundancy policy for every content type
• Provides dynamic image manipulation
How do we do it?
Designed to fail
• Fallback for every operation– Geographical, machine, storage medium
• Write never fails– All files will reach their destination
• Journaling– Tracking all uploaded files
• Pending jobs – Guaranteed file distribution
How do we achieve this
• Control the input– define the only unified API
• Functional process isolation– every function deserves its own process by default– watchdogs– monitors– alerts
5.static 7.static3.static1.static
0.static
00-1f
2.static
20-3f
6.static
60-7f
4.static
40-5f
HTTP HTTP HTTP
HTTP HTTP HTTP
get 37D815B5.jpg Go to 37 range servers Fallback if not found
Fallback Example
Node Architecture
Output – Front• lighttpd forkHTTP handler• Dynamic image processor• lighttpd forkVan Gogh• customCross Datacenter Distributer• customLocal Datacenter Distributer• Tornado web framework
applicationSupervisor HTTP handler
Input – Supervisor
Real Life
It’s all about performance
• Non blocking IO, readiness notification (epoll)• Asynchronous file IO (AIO)• Zero copy (sendfile)• Memory maps• Inter-process binary protocols• UNIX socket• Minimize dynamic memory allocation• lighttpd memory footprint: 50MB
Lessons learnt
• Be symmetric• Control the input• Design to failure• Performance matters again• Simple is hard but a must