(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second

Jim Plush Sr Director of Engineering, CrowdStrike

Dennis Opacki, Sr Cloud Systems Architect, CrowdStrike

October 2015

BDT323

Amazon EBS and Cassandra1 Million Writes Per Second on 60 Nodes

An Introduction

to CrowdStrike

We Are CyberSecurity Technology Company

We Detect, Prevent And Respond To All Attack Types In Real Time,

Protecting Organizations From Catastrophic Breaches

We Provide Next Generation Endpoint Protection, Threat Intelligence & Pre &Post

IR Services

http://www.crowdstrike.com/introduction-to-crowdstrike-falcon-host/

CrowdStrike Scale

• Cloud-based endpoint protection

• Single customer can generate > 2 TB daily

• 500K+ events per second

• Multi-petabytes of managed data

Truisms???

• HTTPS is too slow to run everywhere

• All you need is anti-virus

• Never run Cassandra on Amazon EBS

What is Amazon EBS?

EBS data volume

/mnt/foo

/mnt/bar

EC2 Instance

Network mounted hard drive

Ability to snapshot data

Data encryption at rest & in flight

Existing Amazon EBS Assumptions

• Jittery I/O a.k.a: Noisy neighbors

• Single point of failure in a region

• Cost is too damn high

A Recent Project: Initial Requirements

• 1PB of incoming event data from millions of devices

• Modeled as a graph

• 1 million writes per second (burst)

• Age data out after x days

• 95% write 5% read

We Tried

• Cassandra + Titan

• Sharding?

• Neo4J

• PostgreSQL, MySQL, SQLite

• LevelDB/RocksDB

We Have to Make This Work

Cassandra had the properties we needed

Time for a new approach?

Number of Machines for 1PB

I2.xlarge c4.2XL EBS

Yearly Cost for 1PB Cluster

I2.xlarge-on demand I2.xlarge-reserved c4.2xl - on demand c4.2xl - reserved

ions o

With Amazon EBS

Initial Launch

Date Tiered Compaction

…more details by Jeff Jirsa, CrowdStrike

Cassandra Summit 2015 - DTCS

http://www.slideshare.net/JeffJirsa1/cassandra-summit-2015-real-world-dtcs-for-operators

Initial Launch

• Cassandra 2.0.12 (DSE)

• m3.2xlarge 8 core

• Single 4TB EBS GP2 ~10,000 IOPS

• Default tunings

Performance Was Terrible

• 12 node cluster

• ~60K writes per second RF2

• ~10K writes per 8 core box

• We went to the experts

Cassandra Summit 2014

Family Search asked the

same question:

Where’s the bottleneck?

https://www.youtube.com/watch?v=Qfzg7gcSK-g

IOPS Available

12500.

25000.

37500.

50000.

I2.xlarge c4.2xlarge

1.3K IOPS?

I see you there,

but I can’t reach you!

The magic gates

opened…

We hit 1 million

writes per second

RF3 on 60 nodes

Testing Setup

Testing Methodology

• Each test run

• clean C* instances

• old test keyspaces dropped

• 13+TBs of data loaded during read testing

• 20 C4.4XL Stress Writers each with their own 1BB sequence

Cluster Topology

Stress Node

10 Instances

AZ: 1A

Stress Nodes

10 Instances

AZ: 1B

20 C* Nodes

AZ: 1A

20 C* Nodes

AZ: 1B

20 C* Nodes

AZ: 1C

OpsCenter

Amazon EBS

Cassandra Stress 2.1.x

bin/cassandra-stress user duration=100000m cl=ONE profile=/home/ubuntu/summit_stress.yaml ops$insert=1$ no-warmup -pop

seq=1..1000000000 -mode native cql3 -node 10.10.10.XX -rate threads=1000 -errors ignore

PCSTAT - Al Tobey

http://www.datastax.com/dev/blog/compaction-improvements-in-cassandra-21

https://github.com/tobert/pcstat

Netflix Test - What is C* capable of?

Netflix Test

1+ million writes per second RF:3 3+ million local writes per second

Netflix Test

No dropped mutations, system healthy at 1.1M after 50 mins

Netflix Test

I/O util is not peggedCommit disk = steady!

Netflix Test

Low I/O wait

Netflix Test

95th Latency = Reasonable

Netflix Test - Read Fail

compression={'chunk_length_kb': '64', 'sstable_compression': 'LZ4Compressor'}

https://issues.apache.org/jira/browse/CASSANDRA-10249

Data Drive Pegged

Reading Data

• 24-hour read test

• over 10 TBs of data in the CF

• sustained > 350K reads per

second over 24 hours

• 1M reads/per sec peak

• CL ONE

• 12 C4.4XL stress boxes

Reading Data

Not Pegged

Reading Data

7.2ms 95th latency

180 less cores (45 less i2.xlarge instances)

• C4.4XL vs. i2.XLarge

24 hour test (sans data transfer cost)

• Netflix cluster/stress

• Cost: ~$6300

• 285 i2.xlarge $0.85 per hour

• CrowdStrike cluster/stress with Amazon EBS cost

• Cost: ~$2600

• 60 C4.4XL $0.88 per hour

VS Netflix Blog Post

• Our test was a single 10K IOPS volume

• More/bigger reads?

• PIOPS gives you as much throughput as you need

• RAID0 multiple Amazon EBS volumes

Read Notes with Amazon EBS

EBS Data

Volume

EBS Data

Volume

/mnt/bar

EC2 Instance

What Unlocked Performance

Major Tweaks

• Ubuntu HVM types

• Enhanced networking

• Now faster than PV

• Ubuntu distro tuned for cloud workloads

• XFS Filesystem

Major Tweaks

• Cassandra 2.1

• Java 8

• G1 Garbage Collector

Major Tweaks

• C4.4XL 16 core, EBS Optimized

• 4TB, 10,000 IOPS EBS GP2 Encrypted Data Drive

• 160MB/s throughput

• 1TB 3000 IOPS EBS GP2 Encrypted Commit Log Drive

Major Tweaks

cassandra-env.sh

• MAX_HEAP_SIZE=8G

• JVM_OPTS=“$JVM_OPTS —XX:+UseG1GC”

• Lots of other minor tweaks in crowdstrike-tools

cassandra-env.sh

Put PID in batch mode

Mask CPU0 from the process to reduce context switching

Magic From Al Tobey

YAML Settings

cassandra.yaml (based on 16 core)

• concurrent_reads: 32

• concurrent_writes: 64

• memtable_flush_writers: 8

• trickle_fsync: true

• trickle_fsync_interval_in_kb: 1000

• native_transport_max_threads: 256

• concurrent_compactors: 4

cassandra.yaml

We found a good portion of the CPU load was

being used for internode compression which

reduced write throughput

internode_compression: none

Lessons Learned

• Amazon EBS was never the bottleneck during testing, GP2 is legit

• Built-in types like list and map come at a performance penalty

• 30% hit on our writes using Map type

• DTCS is very young (see Jeff Jirsa’s talk)

• 2.1 Stress Tool is tricky but great for modeling workloads

• How will compression affect your read path?

Test Your Own!

https://github.com/CrowdStrike/cassandra-tools

It’s Just Python

launch 20 nodes in us-east-1

• python launch.py launch --nodes=20 —config=c4-ebs-hvm

—az=us-east-1a

bootstrap the new nodes with C*, RAID/Format disks, etc…

• fab -u ubuntu bootstrapcass21:config=c4-highperf

run arbitrary commands

• fab -u ubuntu cmd:config=c4-highperf,cmd="sudo rm -rf

/mnt/cassandra/data/summit_stress"

Run Custom Stress Profiles… Multi-Node Support

ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 —seednode=10.10.10.XX —-threads=50

Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m cl=ONE

profile=/home/ubuntu/summit_stress.yaml ops$insert=1,simple=9$ no-warmup -pop seq=1..1000000000 -mode native cql3 -node

10.10.10.XX -rate threads=50 -errors ignore

ubuntu@ip-10-10-10.XX:~$ python runstress.py --profile=stress10 --seednode=10.10.10.XX --threads=50

Going to run: /home/ubuntu/apache-cassandra-2.1.5/tools/bin/cassandra-stress user duration=100000m cl=ONE

profile=/home/ubuntu/summit_stress.yaml ops$insert=1,simple=9$ no-warmup -pop seq=1000000001..2000000000 -mode native cql3

-node 10.10.10.XX -rate threads=50 -errors ignore

export NODENUM=1

export NODENUM=2

• ~3 months on our Amazon EBS–based cluster

• Hundreds of TBs of graph data and growing in C*

• Billions of vertices/edges

• Changing perceptions?

• DataStax - Planning an Amazon EC2 cluster

Where Are We Today?

Al Tobey’s Tuning Guide for Cassandra 2.1

https://tobert.github.io/pages/als-cassandra-21-tuning-

guide.html

Resources

Special Thanks To

Leif Jackson

Marcus King

Alan Hannan

Jeff Jirsa

• Al Tobey

• Nick Panahi

• J.B. Langston

• Marcus Eriksson

• Iian Finlayson

• Dani Traphagen

Amazon EBS Heading Into 2016

4TB (10k IOPS) GP2

I/O Hit? Not enough to phase C*

So why the hate for

Amazon EBS?

• Used instance-store image and

ephemeral drives

• Painful to stop/start instances, resize

• Couldn’t avoid scheduled maintenance

(i.e., Reboot-a-palooza)

• Encryption required shenanigans

Following the Crowd – Trust Issues

• We still had failures

• Now we get to rebuild from scratch

Guess What

What do you mean my volume is “stuck”?

• April 2011 – Netflix, Reddit, and Quora

• October 2012 – Reddit, Imgur, Heroku

• August 2013 – Vine, Airbnb

Amazon EBS’s Troubled Childhood

http://techblog.netflix.com/2011/04/lessons-

netflix-learned-from-aws-outage.html

Spread services across multiple regions

Test failure scenarios regularly (Chaos Monkey)

Make Cassandra databases more resilient by

avoiding Amazon EBS

Kiss of Death

Amazon moves quickly and quietly:

• March 2011 – New Amazon EBS GM

• July 2012 – Provisioned IOPs

• May 2014 – Native encryption

• Jun 2014 – GP2 (game changer)

• Mar 2015 – 16TB / 10K GP2/ 20K PIOPS

Redemption

• Prioritized Amazon EBS availability and consistency beyond features and functionality

• Compartmentalized the control plane – removed cross-AZ dependencies for running volumes

• Simplified workflows to favor sustained operation

• Tested and simulated via TLA+/PlusCal - better understood corner cases

• Dedicated a large fraction of engineering resources to reliability and performance

Redemption

Amazon EBS team targets 99.999%

availability

exceeding expectations

Reliability

• In past 12 months, zero Amazon EBS–

related failures

• Thousands of GP2 data volumes (~2PB

• Transitioning all systems to Amazon EBS

root drives

• Moved all data stores to Amazon EBS

(C*, Kafka, Elasticsearch, Postgres, etc.)

CrowdStrike Today

• Select a region with >2 AZs (e.g.,

us-east-1 or us-west-2)

• Use Amazon EBS GP2 or PIOPs storage

• Separate volumes for data and commit

Staying Safe - Architecture

• Use Amazon EBS volume monitoring

• Pre-warm Amazon EBS volumes?

• Schedule snapshots for consistent backups

Staying Safe - Ops

• Challenge assumptions

• Stay current on AWS blog

• Talk with your peers

Most Importantly

http://aws.amazon.com/ebs/nosql/

Remember to complete

your evaluations!

BDT323

Thank you!

@jimplush@opacki

(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per Second

Technology

Running Cassandra on Amazon’s ECS - Meetupfiles.meetup.com/7439192/Cassandra-ECS.pdf · • Cassandra • ECS • Cassandra on Docker best practices • Cassandra on ECS. Motivation

Cassandra Summit EU 2014 - Testing Cassandra Applications

Associate)Professor)Cassandra)L.Atherton) Deakin ...cassandra-atherton.com/wp-content/uploads/2018/01/Cassandra-Athert… · Associate)Professor)Cassandra)L.Atherton) Deakin’University’

Cassandra Training Session 2svn.wso2.org/repos/wso2/people/kasunw/BAM/Cassandra/Cassandra... · Configuring Cassandra Contd ... replicationStrategy, replicationFactor, cfs); cluster.addKeyspace(definition);

SOPRADORAS E ENCHEDORAS - Smigroup...EBS 3 HC EBS 4 HC EBS 6 HC EBS 2 K EBS 3 K EBS 4 K EBS 4 EBS 6 EBS 8 EBS 10 EBS 12 EBS 14 EBS 16 3 Os valores apresentados não são vinculativos,

Cassandra Freeman - Thoughtful Inspirationsthoughtfulinspirations.com/.../2017/12/Cassandra-Freeman-Final-Bio.pdf · Cassandra Freeman Cassandra Freeman ... Jim Rohn, Zig Ziglar Leadership

LA Cassandra Day 2015 - Testing Cassandra

Apache Cassandra at Target - Cassandra Summit 2014

Apache Cassandra™ Documentationcourses.physics.illinois.edu/cs425/fa2017/cassandra10.pdfApache Cassandra 1.0 Documentation Introduction to Apache Cassandra Apache Cassandra is a

Cassandra writes-reads-compaction

CADWELD und Erdungsverbinder€¦ · 2 Ebs 02.05.19 3 Ebs 02.05.34 2 Ebs 15.01.09 4 Ebs 15.03.17 3 Ebs 20.01.01 Bezugsquelle für Erdungsbrücken: ERICO GmbH, 66851 Schwanenmühle

Chicago Cassandra - Cassandra from Python

Cassandra Day Atlanta 2016 - Monitoring Cassandra

State of Cassandra, 2012 - NoSQL | Apache Cassandra · State of Cassandra, 2012 Jonathan Ellis Project Chair, Apache Cassandra CTO, DataStax @spyced ©2012 DataStax Some Cassandra

Cassandra @ Yahoo Japan | Cassandra Summit 2016

Upon completion you will be able to: Describe EBS Basics of EBS Types of EBS Sets EBS Requirements Types of Directory Numbers Describe EBS

Cassandra Community Webinar: Apache Cassandra Internals

Cassandra Summit 2014: Cassandra Compute Cloud: An elastic Cassandra Infrastructure

Cassandra Day Atlanta 2015: Python & Cassandra

1 Million Writes per second on 60 nodes with Cassandra and EBS