Microservices Manchester: Concursus - Event Sourcing Evolved By Domonic Fox

ConcursusEvent Sourcing Evolved

Introductions

Dominic Fox

Twitter: @dynamic_proxy

Email: dominic.fox@opencredo.com

Github: http://github.com/poetix

Concursus Github: http://github.com/opencredo/concursus

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Agenda

"I have told people over and over and over again, don't write a

CQRS framework…I can basically guarantee you that it will be

abandonware within one year, like every other one has become.”

- Greg Young, inventor of the term “CQRS”

Why Concursus?

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

From Presence to Presents

From Presence to Presents

Aggregate services together into a single container or integrate

them through a single enterprise service bus...

Manage all of our data through a single relational database

schema with global constraints...

Use distributed locks and transactions to make a distributed

system behave as if it were one single system with a global

transactional semantics...

From Presence to Presents

From Presence to Presents

From Presence to Presents

From Presence to Presents“Here the Microservice can become an escape route from reality. Within each Microservice, we can

live on a safe island of determinism and strong consistency — an island where we can live happily

under the illusion that time and the present is absolute.

However, as soon as we exit the boundary of the Microservice we enter a wild ocean of non-

determinism—the world of distributed systems, which is a very different world. You have probably

heard that building distributed systems is hard. It is. That being said it is also the world that gives

us solutions for resilience, elasticity, isolation amongst others. At this point, what we need to do is

not to run back to the monolith, but instead learn how to apply and use the right set of principles,

abstractions and tools in order to manage it.”

- Jonas Bonér, Reactive Microservices Architecture, p. 28-29

From Presence to Presents

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

“Write First, Reason Later”

“Write First, Reason Later”

“Write First, Reason Later”

Scene from Endgame, by Samuel Beckett

“Write First, Reason Later”

“Write First, Reason Later”

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Three Processing Schedules

1.Transient

2.Durable

3.Persistent

Three Processing Schedules

1.Transient

2.Durable

3.Persistent

Three Processing Schedules

1.Transient

2.Durable

3.Persistent

Three Processing Schedules

1.Transient - what happens

2.Durable - what’s happening

3.Persistent - what happened

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Domain Model: Events

aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

eventTimestamp: 2016-03-31T10:31:17.981Zparameters: { “wattage”: 60 }

Domain Model: Events

aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

eventTimestamp: 2016-03-31T10:36:42.171Zparameters: { “location”: “hallway”}

Domain Model: Events

aggregateType: lightbulbaggregateId: 69016fb5-1d69-4a34-910b-f8ff5c702ad9

eventTimestamp: 2016-03-31T10:36:42.171ZprocessingTimestamp: 2016-03-31T10:36:48.3904Zparameters: { “location”: “hallway”}

Domain Model: Events

Domain Model: Summary

Every Event occurs to an Aggregate, identified by its type and id.

Every Event has an eventTimestamp, generated by the source of the event.

An Event History is a log of Events, ordered by eventTimestamp, with an additional processingTimestampwhich records when the Event was captured.

Network

Event sources

Event processors

Events arrive:• Partitioned• Interleaved• Out-of-order

Processing Model: Ordering

Log is:• Partitioned by aggregate id• Ordered by event timestamp

Processing Model: Ordering

CREATE TABLE IF NOT EXISTS concursus.Event (aggregateType text,aggregateId text,eventTimestamp timestamp,streamId text,processingId timeuuid,name text,version text,parameters map<text, text>,characteristics int,PRIMARY KEY((aggregateType, aggregateId), eventTimestamp,

streamId)) WITH CLUSTERING ORDER BY (eventTimestamp DESC);

Cassandra Schema

CassandraEvent Store

RabbitMQ Topic

DownstreamprocessingLog

events

Publish events

Cassandra & AMQP

CassandraEvent Store

RabbitMQ Topic

Downstreamprocessing

out-of-order events

ordered query results

Cassandra & AMQP

CassandraEvent Store

Kafka Topic

Downstreamprocessing

Event store listener

Publish events

Log events

Cassandra & Kafka

Processing Model: Summary

Events arrive partitioned, interleaved and out-of-order.

Events are sorted into event histories by aggregate type and id.

Events are sorted within event histories by event timestamp, not processing timestamp.

Event consumers need to take into account the possibility that an event history may be incomplete at the time it is read – consider using a watermark to give incoming events time to “settle”.

Programming Model: Core Metaphor

Consumer<Event>

Programming Model: Core Metaphor

You give me a Consumer<Event>, and I send Events to it one at a time:

Emitting Events

I implement Consumer<Event>, and handle Events that are sent to me.

Handling Events

Event-handling middleware is a chain of Consumer<Event>s that transforms, routes, persists and dispatches events. A single event submitted to this chain may be:

■ Serialised to JSON

■ Written to a message queue topic

■ Retrieved from the topic and deserialised

■ Persisted to an event store (e.g. Cassandra)

■ Published to an event handler which maintains a query-optimised view of part of the system

■ Published to an event handler which maintains an index of aggregates by event property values (e.g. lightbulbs by wattage)

Event-Handling Middleware

Java 8 Mapping

Java 8 Mapping

Java 8 Mapping

Querying and Replaying

Querying and Collecting

Kotlin Mapping

Kotlin Mapping

Kotlin Mapping

■ Event type and middleware implementing Consumer<Event>provide basic mechanics for creating, processing and handling events.

■ Java 8 mappings provide a convenient, type-safe method-mapping wrapper around these mechanisms.

■ Kotlin mappings provide an alternative wrapper based on immutable classes.

■ Other mappings are possible – Scala, Clojure, Java with POJOs…

Programming Model: Summary

Agenda

1. From Presence to Presents

2. “Write First, Reason Later”

3. Three processing schedules

4. Concursus

5. Future Directions

Future Directions

1. Stable Kafka integration

Future Directions

1. Stable Kafka integration

2. Avro integration

Future Directions

1. Stable Kafka integration

2. Avro integration

3. End-to-end async programming model

Questions?