The CQRS diet

★ the CQRS diet ★Make your models lose

weight

By Luismi Cavallé

1

Hello, everyone! Tonight I’d like to talk you about CQRS…

★ the CQRS diet ★

COMMAND-QUERY RESPONSIBILITY

SEGREGATION

2

CQRS stands for Command-Query Responsibility Segregation which is a design pattern behind a really…


Simple Idea

3

…simple idea. Let’s take for instance a typical Rails model…


Project ReportProject

find

scopeddelete

save

4

In Rails any request regarding Projects is handled by the same ActiveRecord model Be that request a find, a filter, a save or a delete, the same model will be used..


Project Report

Project

find

scopeddelete

save

QUERIES

COMMANDS

5

Well, the idea behind CQRS is the creation of two objects were there was previously one. The separation occurs based upon whether the methods are a query, meaning that they ask to read data, or a command, meaning that they ask to change the state of the system.


Simple ideaenables

interesting things

architecturally

that

6

As you can see, the idea is quite simple and not especially interesting in itself What makes it interesting is the architectural possibilities that it opens.


Simple ideaenables

interesting things

architecturally

that

7

Some months ago, observing how these ideas were gaining traction in other development communities, particularly among developers that call themselves “architects” and work in the enterprise, which I thought it was really cool… or maybe not… well!, anyway, the thing is that I became interested and started a personal research about these topics. This talk is about sharing my findings. We will explore this pattern along with a few related concepts and discuss some different implementations with their benefits and pitfalls.


WHY SHOULD I CARE? ™

8

So, let’s start with the motivations…


Are you happy with your Rails app?

★ Models clean and straightforward

★ No need to deal with complex domain logic

★ No scalability or performance issues

★ No maintainability problems

9

The question to ask here is “Are you happy with your Rails app?”. Because if your models are clean and straightforward –probably because you don’t deal with complex business logic–, if you haven’t got important scalability or performance issues and you don’t feel like there’s too much technical debt in your code. Then you probably don’t need to care about separating reads and writes or any other architectural alternative. But…


Some smells…★ Denormalized database

★ Complex caching

★ Overweighted models

★ Presenter pattern

★ Increasingly complex infrastructure

★ Considering NoSQL

10

…if you’ve got a denormalized database, or there’s caching logic spread all around your app, or your models are way too big and complex, or you are using the presenter pattern, or your infrastructure is becoming increasingly complex, or you are considering NoSQL solutions. Then, chances are that you’ll find CQRS ideas interesting.


THE SINGLE

RESPONSIBILITY

PRINCIPLE should never be more

one reason class

change

There

than

for a

to

“

”

11

You know? There’s this Object-Oriented Design better practice that you may already know. It is called “The single responsibility principle” and it states that one object should only have one responsibility (or, put otherwise, just one reason to change). But if you look at an ActiveRecord model you see a totally different story…


★ Validation★ Domain Logic methods & callbacks

★ Structure associations

★ Filtering scopes

★ Presentation helper methods

★ Persistence

Model Responsibilities

12

In Rails, every of your models has to take care of: Validation, Domain logic (typically hooked to persistence callbacks), Structure (every model declares its associations), Filtering, Presentation (sometimes its convenient to put that presentation logic in the model instead of in a helper) and, last but not least, every model is aware of its own Persistence.

Wow! definitely our models got more than one reason to change. Which is ok, as we mentioned earlier, when there’s no much complexity to deal with. But when that’s not the case, this Rails’ monolithic approach makes hard to partition our problem.


the CQRS diet

13

I think it is clear that we need to put our models on a diet. The CQRS diet!


Project Report Project

COMMANDSQUERIES

14

Let’s take it from where we left it. The idea was quite simple: Commands & Queries are handled separately by different objects. The interesting part is the architectural decisions that this enables. So let’s explore those interesting things.


Project

COMMANDSQUERIES

ProjectList

Project Details

15

The first and one of the most interesting things you can do in a CQRS architecture is having two separated data models and making different choices on each of them regarding things like how data is structured or where and how it is stored. It turns out that, given the different purposes of each data model, these choices can be quite different.


DOMAIN DATA MODEL

READ DATA MODEL

★ Validation★ Business Logic

★ Filtering★ Data presentation

CO

NC

ER

NS

AT

RIB

UT

TE

SNORMALIZED

PERSISTENCE IGNORANT

TRANSACTIONALCONSISTENT

OBJECTORIENTED

ACID

DATA ORIENTED

DENORMALIZED

EVENTUALLY CONSISTENT

INDEXABLE

BASE

16

The main concerns of a domain model are mainly about business logic validation and enforcing business rules and domain invariants. So we might decide that the most important attributes for our domain model and its repository are like to be normalized, transactional or consistent.

As for the Read model, concerns are different It only cares about presenting data to the user so, in this case, other attributes could be more convenient, like to be denormalized (so data access is faster), indexable or eventual consistent (it’s not a big deal if data is stale some times)

These attributes may or may not the best choice for any given system, but the point is that different trade-offs apply and now we can make totally different choices for each model.

★ the CQRS diet ★Event Handler

COMMANDSQUERIES

EVENTSUPDATES

ProjectList

Project Details

DOMAIN MODELREAD

MODEL

TaskProject

17

So now we have two different models: A domain model, responsible of commands, and a read model, responsible of queries. Each of them with totally different attributes.

But there’s something missing here, as you’ve probably noticed. In order for this to work at all, we need to keep both models consistent and, preferably, uncoupled as well. So, in order to get that, the piece that we need to add to the system is some kind of syncing mechanism that listens to the changes happening in the domain model and updates the read model accordingly. This agent, ensures consistency and, since its based on events, it keeps both models unaware of each other.

Simple example using just the tools in the box

(i.e. Rails)

18

To better illustrate this, we’re gonna see a very simple implementation using anything else but Rails.


class Project < ActiveRecord::Base # t.string :name # t.text :notes # t.datetime :deadline # t.boolean :active has_many :tasks scope :active, where(:active => true) validates_presence_of :name def status active? ? "Active" : "Inactive" endend

INITIAL MODEL

19

We have this initial model. No CQRS here, yet, ok? It’s an ActiveRecord model which has four attributes: a name, some notes, a deadline, and a flag that indicates whether the project is active or not A project has many tasks It seems that it needs to be filtered by the active flag The presence of a name is necessary for a Project to be valid And finally, we’ve got this handy method that returns a user-friendly string denoting the status of the project.

How about our controller?…


CONTROLLER

class ProjectsController < ApplicationController

def index @projects = end

def create project = Project.create!(params[:project]) redirect_to(project, :notice => 'Project created.') end

def destroy project = Project.find(params[:id]) project.destroy

redirect_to(projects_url) end end

ProjectReport.activeProject.activeQUERY

COMMAND

COMMAND

20

…Nothing unexpected here either. We’ve got a nice, skinny controller which deals with two commands (create and destroy) and one query (the index action). All three actions are using the same Project model we saw in the previous slide. And that’s what we’re going to change. Queries and Commands should use different models so, for our index action we’re replacing Project with a new read model we call ProjectReport. Now let’s see how ProjectReport looks like…


class ProjectReport < ActiveRecord::Base # t.string :name # t.string :status # => “Active” or “Inactive” # t.integer :project_id scope :active, where(:status => "Active")end

READ MODEL

21

First thing to notice here is that we are using ActiveRecord, but we could be using anything else. From a more lightweight ORM like Sequel, to any NoSQL database, like Redis, CouchDB or MongoDB. Whatever we think that will work better for our read model which only purpose is, I remind you: populating views with data as fast and as conveniently as possible. So, it is just for the sake of the example that we stick to ActiveRecord here.

What have here? First thing are the attributes. We’ve got only those that are shown to the user, like the name, the status, which it’s handy that it stores the string to be shown to the user (not a boolean flag that we’d need to transform later) And a reference to the Project in the domain model, that will help us keep track of its changes.

Besides attributes, we’ve moved here the “active” scope. Filtering is part the read model so this scope belongs to here.

Let’s go back to our domain model now...


class Project < ActiveRecord::Base

DOMAIN MODEL

scope :active, where(:active => true)

validates_presence_of :name

def status active? ? "Active" : "Inactive"end

has_many :tasks

end

# t.string :name

# t.datetime :deadline# t.boolean :active

# t.text :notes

22

Our initial model used to look like this. But now, we have a Read Model which frees this one from a couple of its responsibilities Like filtering, this scope is no longer being used (controllers use the one in the read model) so, let’s get rid of it…




def status active? ? "Active" : "Inactive"end

has_many :tasks

end

DOMAIN MODEL

# t.string :name


# t.text :notes

23

What else? The status. Now it is stored in the read model and our domain logic doesn’t need it for anything. So let’s remove it too…




# t.string :name

has_many :tasks

end


DOMAIN MODEL

# t.text :notes

24

Nice! Removing code is one of the great joys of being a programmer, don’t you think?

So, we end up with this two models…


class ProjectReport < ActiveRecord::Base # t.string :name # t.string :status # t.integer :project_id scope :active, where(:status => "active")end

DOMAIN MODELREAD MODEL

class Project < ActiveRecord::Base # t.string :name # t.string :status # t.datetime :deadline # t.boolean :active has_many :tasks validates_presence_of :nameend

25

…that we need to keep consistent. We need something that listens to the Domain Model and updates the Read Model accordingly. For that, one of the simplest tool we can use, right out of Rails box, is…


OBSERVER

class ProjectObserver < ActiveRecord::Observer def after_save(project) report = ProjectReport.find_or_create_by_project_id(project.id) report.update_attributes! :name => project.name, :status => project.active ? "Active" : "Inactive" end def after_destroy(project) ProjectReport.destroy_all(:project_id => project.id) end end

26

…An observer. By definition, observers listen to changes in other objects and do something about them. So they seem exactly what we need Every time a Project is saved in the Domain, we will find or create a related ProjectReport in the Read Repository and update the name and the status accordingly. Note that for the status, we made here the transformation from the active flag in the domain model to the string we store in the read model.

Besides that when a Project is removed from the domain repository, we want any related ProjectReport to be removed from the read model as well.

And that’s all about the observer…


class ProjectReport < ActiveRecord::Base # t.string :name # t.string :status # t.integer :project_id scope :active, where(:status => "active")end

DOMAIN MODELREAD MODEL

class Project < ActiveRecord::Base # t.string :name # t.string :status # t.datetime :deadline # t.boolean :active has_many :tasks validates_presence_of :nameend

class ProjectObserver < ActiveRecord::Observer def after_save(project) report = ProjectReport.find_or_create_by_project_id(project.id) report.update_attributes! :name => project.name, :status => project.active ? "Active" : "Inactive" end def after_destroy(project) ProjectReport.destroy_all(:project_id => project.id) end end

OBSERVER

27

We are quite done with this example. This slide summarizes what we ended up with.

You know, the problem with examples is they have to be simple to avoid distraction, but, just because of that, often they don’t justify the technique you’re showing. Our initial model had no “overweight” problem to begin with. But, still, the point is that the example helps you understand the pattern. If that’s not the case just raise your hand and ask.


Similar, real-world example*

★ MongoDB for the Read Model (Presenters Silos)

★ Asynchronous Observers with RabbitMQ (Silovators)

Denormalizing Your Rails Applicationby Daniel Lucraft & Matt WynneScottish Ruby Conference 2010

*28

Ok, before moving on to something else, I just want to point out an interesting real-world example of this approach It was presented by Daniel Lucraft and Matt Wynne in the latest Scottish Ruby Conf and there’s a video available in the internet. In the talk they describe the architecture of songkick (a social website about music and concerts) which is very similar to what we’ve just seen /

They use MongoDB for the Read Model, using Presenters and what they call, “Silos”, and, the other interesting thing is that their observers work asynchronously (which is very typical) using RabbitMQ queues.

So, this is an interesting resource if you want to dig deeper into this basic approach to CQRS.


EVENTS

29

So, let’s move on to the next big topic of the talk: Events. And I don’t mean last night’s event which was awesome. I mean…


EVENTSDOMAIN

30

…Domain Events.


READ REPOSITORYDOMAINEVENTS

EVENT HANDLERS

31

Let’s say that we are sold to CQRS and now we have a whole system that segregates reads and writes. Even in colors! We’ll soon realize that domain events have become a critical part our system.


Why events are key?

★ Events keep models consistent

★ Events keep models loosely coupled

★ Events keep track of all changes

32

Why? Well they keep our data models consistent, we already know that. But there’s more.

They also keep our data models loosely coupled. This is very important. We could try to keep our models consistent in different ways like, for instance, by asking the domain model about its state. However, doing that we would be adding coupling into the system, making it brittle (you know, changes in one part of the system would cause other parts of the system to break). So instead of that, we use events. Events don’t expose the state of the domain but only the *changes* in the domain. That maintains the domain model isolated from the outside world, which is nice to make changes without affecting other parts of the system.

And there’s one more important thing about events They keep track of absolutely any change that happens in the domain. And that could make events no less than...


SourceTruth

The

of

33

…The Source of Truth!

And we would have new ways to add value to our system. Let’s see them.



EVENT HANDLERS

EVENT STORE

34

Ok, there we are. The first further step we can take is to store every event that the domain publishes. If domain events are going to be our source of truth, we definitely need to store them. This will give us some immediate value…


Event store

★ Comprehensive audit log

★ Ability to replay events

35

First of all we’ve got the most comprehensive audit log, which for some domains (like Accounting) is a must, and for the rest, it can provide a great business value. This is not the same as any other infrastructure log, this events has meaning to the domain, to the business. Think about diagnosing problems or giving customer support. The value of such a log could be huge. And with an event store, we got this value virtually for free.

Besides that we have now the ability to replay events. Let’s say that we find a bug in one of our event handlers that makes one of our views to show incorrect data. When we fix the bug, we also need to fix the corrupt data in the read repository. Normally you would do it kind of manually: you’d run a script over the read repository that fixes the data. But now, thanks to our shiny event store, we could do something else, we could just replay all the related events over the, now fixed, event handler so that the correct dataset will be generated.

Nice, but there’s more…


Domain Persistence

36

…and it has to do with Domain Persistence. We want our events to be our source of truth. So let’s say that we implement in our Domain objects the ability to apply events to them. With that, we have now the capacity to re-construct any Domain object up to its latest state just by applying its event stream to it.

And, if we have this capacity, we no longer need any persistence solution for our domain other than…


Domain Persistence

Event Store

37

The event store.



EVENT HANDLERS

EVENT STORE

38

That’s it. Every time we need a certain domain object to process a command the Domain Repository will get the stream of events related to that object from the Event Store. Then, it will instantiate a new object and it’ll apply the events to it. That way we get the object in its current state, ready to process the next command.

We won’t need any other persistence mechanism. We won’t ever “save” domain objects, we will just *publish* events.

We’re gonna see an example of this. But first let me just point out that all this approach around events has a name which is…


EVENT SOURCING

39

“Event Sourcing”, and it is a design pattern in its own right that happens to work very well with CQRS.

So now that we have a name, let’s see an implementation example of a CQRS system with Event Sourcing


github.com/cavalle/banksimplistic

40

The example is extracted from a sandbox application that I use to experiment with different implementations of these ideas. It is called BankSimplistic, it’s about banking (not surprisingly), and you can find the code in github if you want to check it out.


FEATURE

feature "Deposit cash", %q{ In order to have money to lend to other clients As a banker I want clients to deposit cash into their accounts}

41

In our bank we want our clients to be able to deposit cash into their accounts, so that we have money to invest in subprime mortgages, right?

So, “Deposit cash” is the functionality we’re looking into in this example.


class DepositsController < ApplicationController # POST /accounts/23/deposits def create account = Account.find(params[:account_id]) account.deposit(params[:deposit][:amount]) redirect_to account end end

CONTROLLER

42

We start with the controller. Let’s say that we receive a POST request to make a deposit into a given account.

First thing we do is to fetch the Account from the Domain Repository, finding it by id. Then we invoke its deposit method passing it through the amount of money. Finally we just redirect the user to the Account page.

And that’s all. Two things to note here: 1st, the deposit method sounds like a command (it is quite clear what we’re doing, we’re not updating attributes or anything like that). And the 2nd thing is that we’re not saving the account. And this is not because the save call is inside the deposit method as we’re going to see right now. It is because we don’t have to save anything.

So let’s see the model


DOMAIN MODELclass Account include AggregateRoot def initialize @balance = 0 end def deposit(amount) # Do some validation new_balance = @balance + amount publish :deposit_made, :amount => amount, :new_balance => new_balance, :account_uid => uid end

43

Ok, the Account model includes a module called AggregateRoot. Don’t worry very much about this name. All you need to know is that it provides the necessary infrastructure logic.

So our model has an initialize method that sets the balance of the account to zero. Makes sense that new accounts has no money, of course.

Then we have our deposit method. The first thing you put in a method like this is some kind of validation logic related to the command being processed. You would check here things like deposit limits or whether the account is open, for instance. Next, if validation is ok, the new balance is calculated, just by adding the amount to the current balance. That’s our domain logic. And, finally, the method publishes an event called “deposit_made” that includes the amount of the deposit, the new balance, and the id of the account where it has been made. And that’s all.

The interesting thing here is that we’re not updating the balance of the account inside this method. We’re just calculating the new balance and then publishing the event.


DOMAIN MODELclass Account include AggregateRoot def initialize @balance = 0 end def deposit(amount) # Do some validation new_balance = @balance + amount publish :deposit_made, :amount => amount, :new_balance => new_balance, :account_uid => uid end def on_deposit_made(event) @balance = event.data[:new_balance] end

44

To update the balance of the account we have this “on_deposit_made” method that will receive the event that the command generates and will update the balance.

And, why is that? you may ask. Well, good question. We need this “on” methods because, as I mentioned earlier, we need to be able to reconstruct a domain object just by applying its events. That’s why any logic that updates the state of the object needs to be inside one of these event handling methods. That’s what these methods are: internal event handlers.

To understand this better, let’s take a look one level down at the infrastructure logic. To begin with, what does the `publish` method do?


THE MAGIC

def publish(name, attributes) event = Event.new(:name => name, :data => attributes) do_apply event event.aggregate_uid = uid published_events << eventend

def do_apply(event) method_name = "on_#{event.name}" method(method_name).call(event)end

45

It starts by creating a new event with its name and data. Good Then it calls a method called “do_apply” and what that method does is precisely calling the internal event handler we saw earlier. Makes sense? Then the id of the object is added to the event data and finally the event is added to a published_events collection from where it will be finally published to the external world and, also, saved to the event store.


THE MAGIC

def build_from(events) object = self.new events.each do |event| object.send :do_apply, event end objectend

def find(klass, uid) events = Event.find(:aggregate_uid => uid) klass.build_from(events)end

46

Some other magic you might find interesting is the find method we used from the controller to get the Account object It first gets all the events related to the requested object, and then it calls the build_from method in the Account class, passing it through the event stream.

That build_from method will initialize a new instance of the class and then it will apply each event to the object using the same “do_apply” method we saw before. And then the object will be ready for the next command.

And that’s pretty much all about this example


DOMAIN MODELclass Account include AggregateRoot def initialize @balance = 0 end def deposit(amount) # Do some validation new_balance = @balance + amount publish :deposit_made, :amount => amount, :new_balance => new_balance, :account_uid => uid end def on_deposit_made(event) @balance = event.data[:new_balance] end

47

I’ve skipped the part of updating the read model since it would be quite similar to what I showed you in the first example.

Summarizing: In this example the command method does three things: first it validates that the command is ok to be processed (no domain invariant is broken, and so on and so forth). Second it applies the necessary business logic required by the command. And third it publishes an event. But it doesn’t change the state of the object. This is done in an internal handler for that event just published. That way, the domain repository will be able to restore domain objects just from their events.


TIME FOR A WEIGHT

CHECK

48

At the beginning of the talk we decided to put our models on a diet. Now I think it’s time for a weight check


★Validation

★Domain Logic methods & callbacks

★Structure associations

★Filtering scopes

★Presentation helper methods

★Persistence

49

We started with models which had all these responsibilities. The first we did was creating a separated read model. That freed our domain model from Presentation and Filtering. We can also say that those associations used only for filtering could be removed from the domain as well.

Then, in the second part of the talk, we saw the value of using an Event Store as the source of truth of the system and decided to use it also as the only persistence mechanism of the domain. So our models no longer need to worry about persistence.

We end up with a nice, thin Domain model that now, can focus in its most important responsibilities: validation and domain logic. It’ll be more about behaviours rather than structure.

By the way, talking about responsibility…


There will be (some) dragons

★ More complex architecture than Rails’ MVC

★ More “moving parts” in the infrastructure

★ Distributed and asynchronous world is chaotic

★ Problems that used to be trivial no longer are

50

It wouldn’t be very responsible of me to let you go from these talk thinking we’ve got a silver bullet. I’m sorry to say there is no such a thing as a silver bullet. Everything has trade-offs.

So, I should mention that with CQRS the simplest case, say the “Hello world” app, is more complex than with Rails MVC. We have two models instead of one and we have to coordinate them. We saw it in the first example: we finished with more lines of code than we started.

We’ll also have more “moving parts” in our infrastructure. You know? It is very nice to only have to worry about one database and one web server. That’s happiness! But with CQRS it is easy to end up with several different databases, one messaging queue, not to mention caching or text indexing. Be careful with that. More “moving parts” means more things that can stop working so more things to monitor, and it becomes more difficult to diagnose problems.

Also as soon as you decide to handle your events asynchronously, you will enter to the distributed world which could be kind of chaotic: messages arrive out-of-order or duplicated, everything is asynchronous, consistency in the read model is eventual so you have to deal with this in the User Interface, etc. This is a new world and it requires you to change your mindset, which is not always easy.

Related to this you’ll find that some problems that used to be trivial, no longer are. For instance validating the uniqueness of a value might not be as simple as it used to. Again, you have to change your mindset, think differently, constraints have changed.


You need a good reason

★ Complex domain logic

★ Complex existent infrastructure

★ Complex scalability needs

51

So, since there will be dragons you need a good reason to get into CQRS. I’d mention three: A complex domain logic (separating responsibilities makes things simpler), an existing complex infrastructure (CQRS would give you a framework to organize it better) or complex scalability needs (with CQRS, you are partitioning your system at application level and that makes it quite scalable).

In all, you need to have...


Complex Problem

52

a complex problem to be solved for CQRS to pay off


FINAL THOUGHTS

53

So, we’re almost done. Some final thoughts.


Separating reads & writes is not new

★ Databases

★ HTTP

★ Infrastructure level

★ Performance or Scalability

54

The idea of separating reads & writes in your system isn’t new at all. We’ve been doing that with databases for ages (you don’t need to look at NoSQL databases like CouchDB where this is pretty explicit, you just need to look at the classic master-slave schema with mysql, where one database is only used for reading)

One of the key points of HTTP (and the REST philosophy) is that it segregates reads and writes with its four verbs, so that any proxy can take advantage of knowing whether a request is a GET or not, and do interesting things about it (like caching)

However, the segregation in these cases is made at infrastructure level and because of performance or scalability reasons.


Separating by function is not new

★ Big systems (e.g. Amazon, Ebay) partition its functionality at application level

★ But again the motivation is scalability

55

Separating by function isn’t new either Big systems like Amazon’s or Ebay’s partition its functionality at application level. Although there’s only one user interface, they have separated systems for Orders, Billing or Shipping. That way it is easier for them to scale the system But again, that’s the motivation, scalability


Separate reads & writes at application level

56

In my opinion, what makes CQRS interesting is that it takes the idea of separating reads & writes and applies it at application level


Benefits in terms of maintainbility

57

And because of that you get benefits in terms of maintainability. It makes easier for you to write a beautiful, expressive domain model


Performance and scalability as a side effect

VERY NICE

Benefits in terms of maintainbility

58

And yes, as a nice side effect, your system will perform and scale quite well

Luismi Cavallé@cavalle

THANKS!★ the CQRS diet ★

59


Resources

★ github.com/cavalle/banksimplistic

★ cqrsinfo.com

★ groups.google.com/group/dddcqrs

60

http://groups.google.com/group/dddcqrs

http://groups.google.com/group/dddcqrs

This slide is intentionally left blank

61

Technology

The CQRS diet