Upload
scylladb
View
369
Download
1
Embed Size (px)
Citation preview
Cassandra & Scylla at Kenshoo
About Me
• Wrote “Basic” code when I was a kid
• 17 years in the internet industry
• Big data fanatic for the last 6 years
• Big data team leader At Kenshoo
• Our team is Big data DBA
• Programming: ETL & Administration tools
Kenshoo
• 10 years, Tel-Aviv based Startup
• Industry Leader in Digital Marketing
• 500+ employees
• Heavy data shop
Kenshoo Legacy Architecture
Bigdata at Kenshoo
Cassandra at Kenshoo
• Datastax Community
• 5 clusters
• 70 nodes
• Biggest cluster, ver 1.2.19▪ 40 physical nodes▪ 4TB compressed to 1TB per node▪ 14 Billion records▪ 1500 bytes values, IOPS: 5K avg, 30K burst▪ Processing clicks and conversion for user behavior
Cassandra Cost
• Cassandra is great at writes (At first)
• Postponing the “cost” to background processes (no free meals)▪ Compact▪ Repair▪ Cleanup▪ Add / Remove node
War & PeaceKnowing your Cassandra
Day to day
• Requires a lot of in depth knowledge
• Lack of documentation
• Tuning per application
• Lot’s of custom maintenance scripts
• Don’t run repair , only rebuild
• Compaction didn’t complete during maintenance window
• Leveled CS is a ColumnFamily cluster wide configuration
• Found a jmx “hack” for a per server config
Migration to Leveled Compaction Strategy
• It took us a month to manually switch
• Needed to reconfigure each server after restart
GC Hell
• Lot’s of GC under load
• GC causing 95th percentile performance▪ One problematic node affects the cluster performance
• Full restart to the cluster each week
• Change rpc server type to hsha
• Tuning is black magic, takes days to see effects
Maintenance is delicate
• Need to wait between adding and removing nodes for
cluster to rebalance
• Turn off a node thrift service
• Running partial cleanup
• Tuning of params▪ compaction_throughput_mb_per_sec▪ concurrent_compactors
Scylla Evaluation
Self tuning features
Lab setup
• i2.8xlarge
• Cassandra 2.1.15▪ compaction_throughput_mb_per_sec = 0 (16)▪ stream_throughput_outbound_megabits_per_sec = 10000 (200)▪ inter_dc_stream_throughput_outbound_megabits_per_sec = 10000 (200)
• Scylla 1.3▪ out of the box
GC at Scylla
Repair
• 3 x i2.8xlarge
• RF 3
• 72GB data per node• 10M rows
▪ 5 columns▪ 1500 bytes value
• Delete all the data from one node
Repair results
Cleanup at Scylla
• 4 x i2.8xlarge
• RF 3
• 270GB data per node• 50M rows
▪ 5 columns▪ 1500 bytes value
• Decommission a node and join it back
Cleanup results
Compact at Scylla
• 3 x i2.8xlarge
• RF 3
• 30 minute stress• 10M rows
▪ 5 columns▪ 1500 bytes value
Compaction result
Latency & Throughput
• 3 x i2.8xlarge
• RF 3
• 72GB data per node• 10M rows
▪ 5 columns▪ 1.5K value▪ 3 writes, 2 reads mixed
• 4 x cassandra-stress nodes▪ 30 minutes
• Scylla has lower cost▪ Compaction, repair & cleanup are much more efficient▪ Consistent in latency under much higher load
• Moving forward▪ Integrate it in an inner production monitoring system
Conclusion
Q&A
Thank You!
Contact: [email protected]