Upload
tom-michiels
View
609
Download
1
Embed Size (px)
Citation preview
@_tommichiels_#Devoxx #Streambased
Stream Based Architecture punching and crunching
@_tommichiels_#Devoxx #Streambased
During this talk I will try to convince you that we need to evolve to stream based architectures
@_tommichiels_#Devoxx #Streambased
Setting the stageOur actors
Punchers
Crunchers
@_tommichiels_#Devoxx #Streambased
Birth, First Encounter, The LoveStory
@_tommichiels_#Devoxx #Streambased
Crunchers life
@_tommichiels_#Devoxx #Streambased
1950 1970 1990
Machine Code
Imperative
Crunchers life
Declarative
@_tommichiels_#Devoxx #Streambased
@_tommichiels_#Devoxx #Streambased
Punchers live
@_tommichiels_#Devoxx #Streambased
Data models
Flat
Hierarchical
Network
RelationalNavigational
@_tommichiels_#Devoxx #Streambased
Normalisation
Isolate data so that additions, deletions, and modifications can be made in just
one table and then propagated through the rest of the database using the defined foreign keys
@_tommichiels_#Devoxx #Streambased
1950 1970 1990
Flat
Navigational
Punchers life
Relational
@_tommichiels_#Devoxx #Streambased
The marriage, thx Edgar
@_tommichiels_#Devoxx #Streambased
Normalisation
Isolate data so that additions, deletions, and modifications can be made in just
one table and then propagated through the rest of the database using the defined foreign keys
Wedding vows
SQL +
@_tommichiels_#Devoxx #Streambased
Crunch(calculate)
Punch(store)
Logic
One happy family
@_tommichiels_#Devoxx #Streambased
And then it started to get complicate
Perf
orm
ance
Complexity
@_tommichiels_#Devoxx #Streambased
PerformanceConcurrent users/operations
@_tommichiels_#Devoxx #Streambased
Performance
@_tommichiels_#Devoxx #Streambased
Complexity(Re)act to changing world
@_tommichiels_#Devoxx #Streambased
ComplexityHistorical reasons
@_tommichiels_#Devoxx #Streambased
ComplexityLegacy
@_tommichiels_#Devoxx #Streambased
Puncher problems
Perf
orm
ance
• Persistence / IO ?•Model ?
@_tommichiels_#Devoxx #Streambased
Cruncher problems
Complexity
•Abstractions ?•Reuse?
@_tommichiels_#Devoxx #Streambased
Crunch(calculate)
Punch(store)
Logic
Scaling up(centralised)
@_tommichiels_#Devoxx #Streambased
Crunch(calculate)
Punch(store)
Logic
Scaling out(distributed)
@_tommichiels_#Devoxx #Streambased
Crunch(calculate)
Punch(store)
Logic
Scaling out(distributed)
@_tommichiels_#Devoxx #Streambased
Propagating Through
•File•DB•RPC•Message
@_tommichiels_#Devoxx #Streambased
Centralised vs distributed
Perf
orm
ance
Complexity
Distributed
Centralised
@_tommichiels_#Devoxx #Streambased
Layers
Crunch(calculate)
Punch(store)
Logic
Punch(store)
UI
@_tommichiels_#Devoxx #Streambased
2 vs N tier
Perf
orm
ance
Complexity
Distributed
Centralised
2 tier N tier
@_tommichiels_#Devoxx #Streambased
DataDriven
Perf
orm
ance
Complexity
Distributed
Centralised
DataDriven
2 tier N tier
@_tommichiels_#Devoxx #Streambased
Query
OLTP
ETLExtract
Transform
Load
OLAP
@_tommichiels_#Devoxx #Streambased
Abstraction
Reuse
Crunchers midlife crisis
@_tommichiels_#Devoxx #Streambased
1950 1970 1990
Machine Code
ProceduralObject Oriented
Crunchers midlife crisis
@_tommichiels_#Devoxx #Streambased
AppDriven
Perf
orm
ance
Complexity
Distributed
Centralised
AppDriven
2 tier N tier
@_tommichiels_#Devoxx #Streambased
ORM Hell
Impedance Mismatch
@_tommichiels_#Devoxx #Streambased
Monolith Hell
@_tommichiels_#Devoxx #Streambased
Spaghetti integration
@_tommichiels_#Devoxx #Streambased
Service Oriented
Perf
orm
ance
Complexity
Distributed
CentralisedService Oriented
2 tier N tier
@_tommichiels_#Devoxx #Streambased
Satellite Services
client
Service interface
Data aggregates
O/R mapping
RDBMS
@_tommichiels_#Devoxx #Streambased
Punchers midlife crisis
©
@_tommichiels_#Devoxx #Streambased
Punchers midlife crisis
@_tommichiels_#Devoxx #Streambased
Punchers midlife crisis
•Partitions•Eventual Consistence
First Class Citizens
@_tommichiels_#Devoxx #Streambased
MicroServices
client
NOSql
Service
@_tommichiels_#Devoxx #Streambased
What about reports?
@_tommichiels_#Devoxx #Streambased
DataLake
@_tommichiels_#Devoxx #Streambased
Distributed Data Driven
Perf
orm
ance
Complexity
Distributed
Centralised
2 tier N tier
Distributed DataDriven
@_tommichiels_#Devoxx #Streambased
Distributed Data Driven
Perf
orm
ance
Complexity
Distributed
Centralised
2 tier N tier
Distributed DataDriven
DataDrivenAppDriven
Service Oriented
@_tommichiels_#Devoxx #Streambased
How did we end up here!!!!
@_tommichiels_#Devoxx #Streambased
How did we end up here!!!!
Shared DB
Messaging
RPC
File
@_tommichiels_#Devoxx #Streambased
The Divorce(ETL)
@_tommichiels_#Devoxx #Streambased
Remember our wedding vows
Isolate data so that additions, deletions, and modifications can be made in just
one table and then propagated through the rest of the database using the defined foreign keys
@_tommichiels_#Devoxx #Streambased
Get us out of this mess!
@_tommichiels_#Devoxx #Streambased
Immutable
an immutable object is an object whose state cannot be modified after it is created
10
@_tommichiels_#Devoxx #Streambased
Stream
10 10 - 10 10 8 vs
is a (ordered) sequence of immutable objects
@_tommichiels_#Devoxx #Streambased
Stream Computing
10 + 10+ (- 10) = 10
data
state is derivedintent
@_tommichiels_#Devoxx #Streambased
Back to Maths
•Associativity•Commutative
@_tommichiels_#Devoxx #Streambased
Back to Maths
10 + 10+ (- 10) = 10
Type
Binary Operation
@_tommichiels_#Devoxx #Streambased
Back to Maths
A ∪ B
Type
Binary Operation
@_tommichiels_#Devoxx #Streambased
Birth of a commutative monoid
(10 + 10)+ (- 10) = ((-10) +10) + 10
Calculation can be partitioned
How we combine intermediate results is not important
@_tommichiels_#Devoxx #Streambased
Programming with solid base in maths
@_tommichiels_#Devoxx #Streambased
Programming with solid base in maths
@_tommichiels_#Devoxx #Streambased
Examples of streams
•Logfiles•Journals/Ledgers•Atom Feeds
@_tommichiels_#Devoxx #Streambased
Propagating New Style
•Make stream available•Consumers keep track of consumed events
•Producers Schema on Write
@_tommichiels_#Devoxx #Streambased
Wedding vows renewed
Propagate immutable events by using streams and use stream crunching as local as possible.
@_tommichiels_#Devoxx #Streambased
Wedding vows renewed
Not Only ETL
@_tommichiels_#Devoxx #Streambased
Streams on Micro(Service) Level
•EventSourcing•CQRS
@_tommichiels_#Devoxx #Streambased
Streams on Micro Level
@_tommichiels_#Devoxx #Streambased
Streams on Macro level
@_tommichiels_#Devoxx #Streambased
Streams on Macro level
kappa architecture
@_tommichiels_#Devoxx #Streambased
Streams on Macro level
@_tommichiels_#Devoxx #Streambased
Reports vs Dashboards
@_tommichiels_#Devoxx #Streambased
Fast Data
@_tommichiels_#Devoxx #Streambased
• BloomFilter
• HyperLogLog
• Count-MinSketch
• MinHash
Stream processing
@_tommichiels_#Devoxx #Streambased
Smack Stack https://github.com/killrweather/killrweather
@_tommichiels_#Devoxx #Streambased
Happily Ever After