Building Scalable, Real Time Applications for Financial Services with DataStax

Building Scalable, Real Time Applications for Financial Services

Simon Webster SVP & General Manager of EMEA

© DataStax, All Rights Reserved.

Feedback• Please use the Event App to give us your

feedback • Everyone who submits feedback will be entered

in the prize draw for a UE BOOM 2 by Ultimate Ears Bluetooth Waterproof Portable Speaker

• Winner will be notified via email

3

DataStax Enterprise Foundations for FinanceDaniel Cohen Solutions Engineer


But Enough About Me…• Solutions Engineer at DataStax • LA ➜ SF ➜ NYC ➜ SF ➜ London • Previously at JP Morgan in London • Morgan Stanley, UBS, Merrill Lynch…

5



5



5

Sandy

7

Sandy

8

Sandy


DataStax Enterprise

9

The database for cloud applications


1 Introductions

2 Technology Overview

3 Architectures

4 Use Cases

5 Questions?

10


What is Apache Cassandra?• A distributed NoSQL database – Google BigTable ✖ Amazon Dynamo

• Scale out with linear performance – Just add nodes

• Continuously available – Disaster avoidance, not disaster recovery

• Run on commodity hardware – In the cloud, on premise, or hybrid

11


Distributed Architecture• Fully distributed – Data spread over multiple nodes – All nodes participate in a cluster – Configurable data replication

• Masterless – All nodes are equal – Read from or write to any node

• No Single Point of Failure

12

Node 11st copy

Node 4

Node 5Node 2

2nd copy

Node 33rd copy


Scale Out Linearly• Need more storage? – Add more nodes.

• Need greater throughput? – Add more nodes.

• Predictable, linear performance gains

13





13





13


Linear Scale Out at Netflix

14

Source: The Netflix Tech Blog http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html

http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html


A Quick Vocabulary Lesson

15



15

Replication Factor – RF• Specifies how many copies of a datum C* distributes across nodes • Highly configurable for SLA & network topology (racks, data centers)

CREATE KEYSPACE "Excalibur" WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'CPH1' : 3, 'CPH2' : 3};



15



• Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation

… ONE … LOCAL_QUORUM … QUORUM … EACH_QUORUM …

Consistency Level – CL



15



• Specifies how synchronized a row of data must be across C* nodes • Tunable consistency: client application decides CL for each operation

… ONE … LOCAL_QUORUM … QUORUM … EACH_QUORUM …

Consistency Level – CL


Node 1 1st copy

Tunable Consistency

16

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Write Consistency Level = QUORUM Replication Factor = 3 Node 2

2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write

Write Consistency Level = QUORUM Replication Factor = 3 Node 2

2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write

Write Consistency Level = QUORUM Replication Factor = 3

5 μs ack

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write


5 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write


5 μs ack

12 μs ack12 μs ack

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write


5 μs ack

12 μs ack

500 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Node 1 1st copy

Tunable Consistency

16

Parallel Write


5 μs ack

12 μs ack

500 μs ack

12 μs ack

Node 2 2nd copy

Node 3 3rd copy

Node 4

Node 5


Continuous Availability

17

Node 1 1st copy

Node 4

Node 5 Node 2 2nd copy

Node 4

Node 2 2nd copy

Node 1 1st copy

Node 3 3rd copyNode 3 3rd copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Node 4

Node 2 2nd copy

Node 1 1st copy


Node 4

Node 5



17

Node 1 1st copy

Node 4


Read Consistency Level = QUORUM Replication Factor = 3

Node 4

Node 2 2nd copy

Node 1 1st copy


Node 4

Node 5



17

Node 1 1st copy

Node 4



Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Node 4

Node 2 2nd copy

Node 1 1st copy

Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Node 4

Node 2 2nd copy

Node 1 1st copy


Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Hints

Node 4

Node 2 2nd copy

Node 1 1st copy


Node 4

Node 5



17

Node 1 1st copy

Node 4


Parallel Read


Hints

Node 4

Node 2 2nd copy

Node 1 1st copy


Node 4

Node 5


Multiple Data Centers

18

Node 11st copy

Node 4

Node 5 Node 22nd copy

Node 33rd copy

Node 11st copy

Node 4

Node 5 Node 22nd copy

Node 3

London DCNew York DC


Disaster Avoidance

19

New York

London

Singapore


Disaster Avoidance

19

New York

London

Singapore


Disaster Avoidance

19

New York

London

Singapore


Disaster Avoidance

19

New York

London

Singapore


Mixed Workload DSE Cluster

Cassandra Only DC

Transactions

Workload Isolation

20

Cassandra+ Spark DCAnalytics

Analytical Application

Operational Application


• Syntax similar to RDBMS SQL • Create objects via DDL – e.g. CREATE…

• INSERT, UPDATE, DELETE • GRANT, REVOKE • SELECT…WHERE • Command line and GUI tools

CQL: Cassandra Query Language

21

CQL Example

CREATE TABLE market_prices ( symbol TEXT, date TIMESTAMP, price DECIMAL, side INT, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);


Drivers & Connectors• Certified by DataStax: – Java, C# – Node.js, Python – Ruby, C / C++ – PHP, ODBC – Connector for Apache Spark

• Many community drivers – Go, Clojure – Erlang, Rust

22


1 Introductions


3 Architectures

4 Use Cases

5 Questions?

23


Application Tier Resilience• Stateless ➜ fault tolerant • Horizontally scalable • Store sessions in Cassandra – Sessions replicate naturally – Recover on disaster / failure

• UI ➜ intelligently stateful – Active failover to app server

24

AppServerCache

C* C*

AppServer Cache

DC LDN1 DC LDN2




24

AppServerCache

C* C*

AppServer Cache

DC LDN1 DC LDN2




24

AppServerCache

C* C*

AppServer Cache

DC LDN1 DC LDN2




24

AppServerCache

C* C*

AppServer Cache

session

DC LDN1 DC LDN2




24

C* C*

AppServer Cache

session

DC LDN1 DC LDN2




24

C* C*

AppServer Cache

session

DC LDN1 DC LDN2




24

C* C*

AppServer Cache

session

DC LDN1 DC LDN2


…A Resilient Enterprise Architecture

DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25

❓



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25

❌



DC LDN2

App App

App App

DC NY1

App App

App App

DC LDN2

C*

DC LDN1

C*

DC NY2

C*DC NY1

C*

25

❌


Unify & Scale Legacy Infrastructure

26

…USA Equities

UK FX

UK Bonds

Global Users

Legacy Systems

USA FX

DataStax Enterprise ClusterC*

User Interface / Application Services


Microservices on Legacy

27

Users µServices

DC NY1A E

F T

DC LDN1A B

E F

Messages

DC NY1

DC LDN1

DC NY1

DC LDN1

C*

C*

USA Equities

DSE

UK FX

Legacy


DSE Real-time Analytics Reference Architecture

HTTP Application Message Queue

Streaming Analytics

Batch Analytics

Real-time

28


1 Introductions


3 Architectures

4 Use Cases

5 Questions?

29


DataStax Use Cases• Customer 360° • Master data management • Customer profile management • Authentication & identity management • Product personalization • Anti-fraud & money laundering • Payments & transactions • Risk reporting / capital adequacy • Market data capture / replay

30


Customer 360° / User Profile• Consolidate data, reduce complexity • Alleviate strain on legacy systems • Integrated data sovereignty • Eliminate downtime • Recommend higher margin products • Customer personalization • Audit and regulatory compliance • Full text indexing and document search

31

“DataStax has allowed us to break the complexities of our legacy systems, empowering us to leverage data to innovate and make more informed decisions so we can provide a truly personalized and premium experience to our customers.”


Payments & Transactions• Transaction history tagging, search and

budgeting, cash flow prediction • Product recommendation based on

purchasing history • Data safe against loss of node, rack, or

data center • Shed load from legacy systems to reduce

cost and provide better user experience

“We had reached the limits of our scalability. We wanted to remove all single points of failure. We had to be active-active. That's why we chose Cassandra.”

32


Regulatory Compliance“The platform enables traders to capture vast amounts of versioned market data which is then used for risk management purposes in real time.”

33

• Capture and analyze every trade, every position for every trader

• Real-time risk analysis • Capital adequacy • MiFID II • Globally replicated, 100% uptime • Low latency, high ingest


“DataStax Enterprise gives us the power to change information in our supporting systems into actionable insights… we can not only scale but give our users a real-time, engaging customer experience.”

• Financial product recommendation • Provide the “small bank” feel through

personalization • Real-time recommendation using transaction

data, search, and browsing history • Measure engagement and satisfaction to

prevent flight • Website and mobile metrics • Prevent customer churn

Recommendations & Engagement

34


Market Tick & Time Series Data• Capture high fidelity time series

information at high ingest rates and low latencies

• 100% uptime even in the event of data center loss

35

“We primarily use Cassandra as a distributed data store for Financial Instrument and Time-Series data, where the ‘master’ data stores cannot scale globally and provide the high availability needed to support a global application with heavy user demand.”


Mission Critical Applications

36

“DSE is a perfect match… easing availability challenges by being active-active and

having an always-on architecture”

Source: Exploiting Hotel Cassandra ➜ http://bit.ly/1mf0KRE

http://bit.ly/1mf0KRE

Thank you!Daniel Cohen Solutions Engineer

[email protected] @CodaAzzurra