Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
www.techarchday.fiwww.techarchday.fi
Modern Data Platforms
(in the age of microservices)
Dominik Wagenknecht
Senior Technology Architect
Accenture
www.techarchday.fi
About Dominik
Dominik Wagenknecht
Accenture Vienna / Austria
Senior Technology Architect
Emerging Technology
www.techarchday.fi
Accenture Vienna
www.techarchday.fi
SO YOU ADOPTED MICROSERVICES
Chapter 1
www.techarchday.fi
So you adopted microservices
• CI/CD pipeline is pure magic!
• Super-motivated Teams
• Business feels progress
• Great Monitoring
• Data?
www.techarchday.fi
What changed?
Browser
API Gateway
Service Service
Service Service Service
DB DB DB
Stateless
Model
API + Services!
www.techarchday.fi
What hasn’t changed?
Browser
API Gateway
Service Service
Service Service Service
DB DB DBBasically CRUD
Synchronous
www.techarchday.fi
BUSINESS WANTS MORE
Chapter 2
www.techarchday.fi
Business wants more
• More Interaction!
– Mobile Push with content
– Personalized eMail,…
• More Smartness!
– Fraud Detection
– Next-best-action
www.techarchday.fi
Mobile Push “OldSchool”
• We need to de-couple!
– Events need to reach a Queue
– Logic listens (MDB?), then push
• Issues?
– Composes really badly
– Force people to testing Queues + MDB’s
www.techarchday.fi
Data Layer is holding you back
Browser
API Gateway
Service Service
Service Service Service
DB DB DB
ServiceQueue
AppleGoogle
Data flow
async
www.techarchday.fi
MODERN DATA IN S, M & L(+ ONE DARK HORSE)
Chapter 3
www.techarchday.fi
Detour: Every op an event
• Service initiates calls
• Modern DB’s separate
– Ops first go to a (binary) log
– Then data is written into core store async
Application(s)
Database
INSERT INTO...INSERT INTO...COMMIT
(Commit)Log
Data
www.techarchday.fi
Modern Data in Small
• Idea: Don’t do it yourself
• Native
– Firebase, RethinkDB
– PipelineDB (kind of)
• Attached
– Meteor’s LiveQuery
S
www.techarchday.fi
Modern Data in Medium
• Idea: Stay lean, keep control
• Messaging
– Something simple like RabbitMQ
– Kafka if you want (but not more)
• Fill from app (or there’s an easy hook)
• Just process, add leader-election
M
www.techarchday.fi
Modern Data in Large
• Idea: Establish a big data Platform
• Really only two choices
– Hadoop with HBase, Kafka, Spark Streaming / Storm / Samza / Flink, etc…
– Cassandra (mostly used with Spark)
L
www.techarchday.fi
Detour: Hook up to Legacy
Application(s)
Database
INSERT INTO...INSERT INTO...COMMIT
(Commit)Log
Data Log Replicator e.g. Kafka
Async starts here!
www.techarchday.fi
PRAGMATISM
Chapter 4
www.techarchday.fi
Some tips
• Physical != Logical
• Stay small
• Be smart about moving parts
www.techarchday.fi
Dark Horse: In-Memory Grid
• Not “micro”, “stateless” or anything
• Think Cache + Logic + Streams
• Great (OpenSource) choice, in Java-world alone:
– Apache Ignite (GridGain)
– Apache Geode (Incubating, Pivotal Gemfire)
– Hazelcast
– …
www.techarchday.fiwww.techarchday.fi
THANKS