EVENT SOURCINGTHE WAYBACK MACHINE FOR YOUR DATA

EVENT SOURCING

WHO AM I?

▸ Chris Hansen

▸ Currently

▸ CTO @

▸ Formerly

▸ Board member @

▸ Senior Architect @

▸ Architect @

EVENT SOURCING

WHO ARE YOU?

▸ What’s event sourcing?

▸ I’ve heard of event sourcing

▸ I’ve implemented event sourcing

▸ I’m stuck maintaining Chris’ first-attempt at event sourcing

EVENT SOURCING

ACTIVE RECORD PATTERN

TABLES ARE NOT OBJECTS. THEY NEVER WERE; AND NEVER WILL BE.

Uncle Bob Martin

EVENT SOURCING

OBJECT/RELATIONAL MAPPING IS THE VIETNAM OF COMPUTER SCIENCE

Ted Neward

EVENT SOURCING

EVENT SOURCING

EVENT SOURCING

QUICK POLL

▸ In our app(s) we don’t delete user data

▸ We also don’t have any UPDATE statements (or the equivalent for NoSQL, etc.)

▸ Conceptually an UPDATE is an atomic DELETE + INSERT

YOUR USERS’ DATA IS MORE VALUABLE THAN THE DISK IT’S STORED ON.

Chris Hansen

EVENT SOURCING

EVENT SOURCING

WHAT IS EVENT SOURCING?

▸ Persist every event (i.e. state change) instead of persisting the current state

▸ Events are immutable

▸ Events are stored in an append-only event store

▸ Current state is calculated by processing each event in order

EVENT SOURCING

PERSIST EVENTS

EVENT SOURCING

CALCULATE CURRENT STATE

EVENT SOURCING

▸ Benefits + HR suite for small businesses

▸ Following examples are a bit simplified

EVENT SOURCING

PERSIST EVENTS

EVENT SOURCING

CALCULATE CURRENT STATE

O(n) QUERIES

EVENT SOURCING

SIDE-EFFECTS

EVENT SOURCING

PREVENTING DUPLICATE SIDE-EFFECTS

▸ Decouple state changes from side-effects

▸ Async processing of side-effects

▸ “Watch” the event store for new events

▸ Async means eventually consistent

EVENT SOURCING PROS

EVENT SOURCING

EVENT SOURCING PROS

▸ Automatic audit log

▸ Decouple persistence from domain model

▸ Ability to query state at any point in the past

▸ Async communication across service boundaries

▸ Derive value from past events

▸ Ability to replay past events

EVENT SOURCING

EMERGING REQUIREMENTS, PAST EVENTS

▸ Example: Identify VIPs

▸ An Employee is a VIP if they have received 2 promotions within 2 years

▸ They get special treatment and offers in-app

▸ Subscribe to EmployeePromoted events in the event store

▸ Replay past events to determine starting list of VIPs

EVENT SOURCING

DATA MINING

▸ Event store is a rich source of data for analytics

▸ Example: Predictive analytics

▸ Given what users of your system have done, try to predict what they might want to do next

▸ Use existing events as training data for ML algorithm

▸ Fine-tune the algorithm, re-train, repeat

EVENT SOURCING

FIXING CORRUPT DATA

▸ Scenario 1: Missing side-effects

▸ Fix the bug

▸ Identify and replay affected events for their side-effects

EVENT SOURCING

FIXING CORRUPT DATA

▸ Scenario 2: Incorrect side-effects

▸ Fix the bug

▸ Identify and undo incorrect side-effects

▸ Identify and replay affected events for their side-effects

EVENT SOURCING CHALLENGES

CONCURRENT WRITES

EVENT SOURCING

OPTIMISTIC LOCKING

▸ Add a version id to each event

▸ Invariant: (object id, version id) pair is unique

▸ Event store rejects all but one concurrent write based on duplicate (object id, version id) pair

SCALING QUERIES

EVENT SOURCING

SNAPSHOTTING

▸ Create a materialized view which is a snapshot of your data at a point in time

▸ Query for current state:

▸ Read the latest snapshot

▸ Apply any events newer than the snapshot

EVENT SOURCING

COMMAND QUERY RESPONSIBILITY SEPARATION (CQRS)

▸ Treating reads differently from writes in a software system

▸ Command: an operation that modifies data

▸ Query: an operation that reads data

EVENT SOURCING

WHEN TO USE CQRS

▸ Disparity between reads and writes

▸ Vastly differing requirements for throughput, scalability, security, availability, reliability, responsiveness, audit-ability, etc.

EVENT SOURCING

TWITTER TIMELINES

▸ Evolved from a monolithic Rails app (Monorail) to highly distributed, decoupled architecture

▸ Fully synchronous until 2010

▸ Write: 5K/s avg., 7K/s daily peak, 12K/s for large events

▸ Read: 300K/s for timelinesSource: Timelines at Scale by @raffi, QCon SF 2012

O(n) writes O(1) reads

EVENT SOURCING

EVENT SOURCING CHALLENGES

▸ Unfamiliarity / added complexity

▸ Forced eventual-consistency

▸ Lack of mature libraries

▸ Lack of mature event store implementations

▸ geteventstore.com

▸ Apache Kafka might work

EVENT SOURCING

WHEN SHOULD I USE EVENT SOURCING?

▸ Audit-ability or version control is critical

▸ You can derive business value out of event data

▸ Inherently non-CRUD (e.g. event-driven) domain

▸ Scalability forces you into CQRS

▸ You have rich domain models (e.g. DDD)Credit: Event sourcing in practice (Reitzammer & Seitz)

EVENT SOURCING

UTAH SOFTWARE ARCHITECTURE MEETUP

▸ Monthly

▸ Small group

▸ Presenter with roundtable discussion

QUESTIONS?