There and back again, ING’s Cassandra Tale

Cassandra Summit

Gary Stewart, @Gaz_GandAChristopher Reedijk, @creedijk

San José, September 7, 2016

Market leaders Benelux

Growth markets

Commercial Banking

Challengers

2

European map

Full-service bankVery strong European baseIT is something ING doesLargest bank of NL150+ Squads20+ use cases in production

Engineer / Architect Engineer / Chapter Lead

3

a little bit about us

Gary Stewart Christopher Reedijk

#CassandraSummit

4

the contract … #CassandraSummit

5

an unexpected party #CassandraSummit

De

vO

ps

En

gin

ee

rs

Ma

na

ge

rs

Infr

a E

ng

ine

ers

6

an unexpected journey #CassandraSummit

Financial Fit

7

#CassandraSummit

the trolls of data modeling

8

#CassandraSummit

Think about the queries first!

Sometimes storing JSON in text is best

saved by …

9

#CassandraSummit

Lonely Planet

10

#CassandraSummit

11

the goblins of data querying #CassandraSummit

Don’t do table scans

Think about Read pathi.e. avoid increasing sequential reads as data grows

Do performance tests even for small use-cases!

12

saved by … #CassandraSummit

iDeal

13

#CassandraSummit

14

the riddles of Gollum #CassandraSummit

What could go wrong this time ….

NTP killed the happy flowCustomers couldn’t go to next page

the riddles of time

15

#CassandraSummit

NTP killed the happy flowCustomers couldn’t go to next page

Versioning to the rescue

saved by …

16

#CassandraSummit

Experiments API

17

#CassandraSummit

Aggressive distributedRead-after-Write, causing read-repair-blocking

18

the spiders of 99 percentile #CassandraSummit

Be careful with read-after-write

Always check the functionality of the framework you’re using

19

saved by … #CassandraSummit

Cached Data

20

#CassandraSummit

SizeTieredCompactionStrategy

Default, compact after 4 sstables

20GB data needs 100GB+ storage

the loss of storage

21

SST1 SST1 SST2 SST1 SST2 SST3 SST1 SST2 SST3 SST4 SST5

FLUSHFLUSH

FLUSH FLUSH

COMPACT

#CassandraSummit

Data space required is now around 50% less!

Attention to details can save i/o and disk space

saved by …

22

SST1 SST1 SST2 SST3

FLUSHFLUSH

COMPACT

#CassandraSummit

23

and actually also helped us with session use cases …

Fraud Investigation

24

#CassandraSummit

Bad performance and nodes dying constantly

Network and GC issuesHinted handoffs going crazyVery long GC-pauses causing nodes to freeze

Never ending compactions

25

the wood elves of network #CassandraSummit

Take a step back

Upgraded to 4.8.5with G1GC (default when using Java 8)4.7.3 is way too chatty

Increased heap and new to 16GB and 8GB

for coordinator nodes, which in our case are all

26

saved by … #CassandraSummit

Incoming Traffic

Take a step back

Upgraded to 4.8.5with G1GC (default when using Java 8)4.7.3 is way too chatty

Increased heap and new to 16GB and 8GB

for coordinator nodes, which in our case are all

27

saved by … #CassandraSummit

Incoming Traffic

28

the desolation of smaug #CassandraSummit

Think top-down – queries first

Talk about failuresExperimentation – SandboxEnvironment that is safe to fail

Take a step backunderstand the architecture

flow of dataensure you are understanding the problem

29

desolated by … #CassandraSummit

30

the battle of 5 armies #CassandraSummit

31

the battle of 5 armies #CassandraSummit

Infr

a E

ng

ine

ers

De

vO

ps

En

gin

ee

rs

Arc

hit

ect

s

???

Ma

na

ge

rs

La

ten

cy

Th

rou

gh

tpu

t

Re

sou

rcin

g

…

Each time we ventured out therewith new insights ready to take on new epic challenges

And each time we came backBit bruised but never broken More willing to shareAnd rolled out into PRD

32

there and back … #CassandraSummit

again and again and again …

33

#CassandraSummit

Thank you

christopher.reedijk@ing.nl@creedijk

gary.stewart@ing.nl@Gaz_GandA

visuals by Isabelle Hörl:www.isabellehorl.com | @IsabelleHorl

we are hiring!

Follow us to stay a step ahead

ING.com

YouTube.com/ING

SlideShare.net/ING@ING_News LinkedIn.com/company/ING

Flickr.com/INGGroupFacebook.com/ING

ING Group’s Annual Accounts are prepared in accordance with International Financial Reporting Standards as adopted by the European Union (‘IFRS-EU’).

In preparing the financial information in this document, the same accounting principles are applied as in the 2014 ING Group Annual Accounts. All figures in this document are unaudited. Small differences are possible in the tables due to rounding.

Certain of the statements contained herein are not historical facts, including, without limitation, certain statements made of future expectations and other forward-looking statements that are based on management’s current views and assumptions and involve known and unknown risks and uncertainties that could cause actual results, performance or events to differ materially from those expressed or implied in such statements. Actual results, performance or events may differ materially from those in such statements due to, without limitation: (1) changes in general economic conditions, in particular economic conditions in ING’s core markets, (2) changes in performance of financial markets, including developing markets, (3) consequences of a potential (partial) break-up of the euro, (4) the implementation of ING’s restructuring plan to separate banking and insurance operations, (5) changes in the availability of, and costs associated with, sources of liquidity such as interbank funding, as well as conditions in the credit markets generally, including changes in borrower and counterparty creditworthiness, (6) the frequency and severity of insured loss events, (7) changes affecting mortality and

morbidity levels and trends,(8) changes affecting persistency levels, (9) changes affecting interest rate levels, (10) changes affecting currency exchange rates, (11) changes in investor, customer and policyholder behaviour, (12) changes in general competitive factors, (13) changes in laws and regulations, (14) changes in the policies of governments and/or regulatory authorities, (15) conclusions with regard to purchase accounting assumptions and methodologies, (16) changes in ownership that could affect the future availability to us of net operating loss, net capital and built-in loss carry forwards, (17) changes in credit ratings, (18) ING’s ability to achieve projected operational synergies and (19) the other risks and uncertainties detailed in the Risk Factors section contained in the most recent annual report of ING Groep N.V. Any forward-looking statements made by or on behalf of ING speak only as of the date they are made, and, ING assumes no obligation to publicly update or revise any forward-looking statements, whether as a result of new information or for any other reason.

This document does not constitute an offer to sell, or a solicitation of an offer to purchase, any securities in the United States or any other jurisdiction. The securities of NN Group have not been and will not be registered under the U.S. Securities Act of 1933, as amended (the “Securities Act”), and may not be offered or sold within the United States absent registration or an applicable exemption from the registration requirements of the Securities Act.

www.ing.com

Disclaimer

36