IMDB Showdown@robertfriberg

robert@devrexlabs.comorigodb.com

Speed of light vs spinning metal

What Time ScaleL1 Cache 0.5 ns 0.008 2 mL2 Cache 7 ns 0.23RAM 60 ns 1 240 m 1 second1K over Gbit network 10 µs 167 2.5 minutes4K read SSD 150 µs 2500Rotating disk seek 10 ms 167000 40000 km 46 hours

Price/GB vs GB/Server 1980- 2015

8 USD

6.480.000 USD

0.001 GB

2000 GB

In-memory stores

• VoltDB• MemSQL

• Hazelcast• Aerospike• Memcached• Oracle Coherence• Redis

• Oracle Times Ten• SQL Server

In-memory OLTP• SAP Hana

• OrigoDBKey/value

New SQL Hybrid

?

1. SQL ServerIn-memory OLTP

B-trees and Transactions

LOG

DATA 64KB blocks w 8x8KB pagesLogical BTREE of 8kb data pagesIn the buffer pool (cache)

BufferManager

Transactions append inserted, deleted, original and modified pages to the LOG

CHECKPOINT

SQL Server In-memory OLTP

SP

Logging

Latches

Locks

Buffer Manager I/O

Native compiled SPs

Minimal Logging and checkpointing

Lock-free data structures

Multi-version Currency control

In Memory and Memory optimized data structures

Sql Server In-memory OLTP

• Heka = Greek for 100• Transparent for application• Integrates with Sql Server but with limitations• Enterprise license• 5-30x performance gain

Demo!

2. RedisREmote DIctionary Server

• Hybrid Key/Value store• Value -> String | List | Set | Hash | SortedSet• Predefined Commands -> SET | GET | HSET | ...• Persistence: Snapshots + AOF• Highly optimized C, fast algorithms

Twitter using redis

INCR next_user_id //returns 1000HMSET user:1000 name bart password ¤¤¤hash¤¤¤HSET users bart 1000ZADD followers:1000 1401267618 1234ZADD following:1234 1401267618 1000INCR next_post_id => 10343HMSET post:10343 user 1000 time $time body ’Ay Caramba’RPUSH posts:1000 10343

3. OrigoDBBuild faster systems faster

What’s the problem?

ServiceLayer

DomainLayer

Data AccessLayer

RelationalModel

Views/SP’s

CacheX

One simple idea...

Keep state in memoryPersist operations, not system state

s0 s1 s2op1 op2

Sn = apply(opn, Sn-1)

... with many names

• System prevalance – Prevalyer, java• MongoDB op log• Redis AOF• Memory Image – Martin Fowler• VoltDB – logical logging• Akka persistence – logging per actor• Event Sourcing

OrigoDB

Kernel

Engine

Model

Storage

App Code

Server

Command

Query

FileSqlEvent StoreCustom

ConsistencyIsolationconcurrency

Sends commands and queries

JournalingSnapshotsBinaryFormatterProtoBufJSON

tcpJSON/http

In-processcalls

Domain specificobject-graph

Domain specific operations

ReplicationAd-hoc queriesWeb uiConsole or win svc

Complete history of events

• Point in time • Debugging• Restore• Queries

• Audit trail• New interpretations

Example – the model[Serializable]public class CommerceModel : Model{ internal SortedDictionary<Guid, Customer> Customers { get; set; } internal SortedDictionary<Guid, Order> Orders { get; set; } internal SortedDictionary<Guid, Product> Products { get; set; }

public CommerceModel() { Customers = new SortedDictionary<Guid, Customer>(); Orders = new SortedDictionary<Guid, Order>(); Products = new SortedDictionary<Guid, Product>(); } }

Command[Serializable]public class AddCustomer : Command<CommerceModel>{ public readonly Guid Id; public readonly string Name;

public AddCustomer(Guid id, String name) { Id = id; Name = name; }

public override void Execute(CommerceModel model) { if (model.Customers.ContainsKey(Id)) Abort("Duplicate customer id"); var customer = new Customer {Id = Id, Name = Name}; model.Customers.Add(Id, customer); }}

Query[Serializable]

public class CustomerById : Query<CommerceModel, CustomerView>

{

public readonly Guid Id;

public CustomerById(Guid id)

{

Id = id;

}

public override CustomerView Execute(CommerceModel model)

{

if (!model.Customers.ContainsKey(Id)) throw new Exception("no such customer");

return new CustomerView(model.Customers[Id]);

}

}

Start your engines!static void Main(string[] args)

{

var engine = Engine.For<CommerceModel>();

Guid id = Guid.NewGuid();

var customerCommand = new AddCustomer(id, "Homer");

engine.Execute(customerCommand);

var customerView = engine.Execute(new CustomerById(id));

Console.WriteLine(customerView.Name);

Console.WriteLine("{0} orders", customerView.OrderIds.Count);

Console.ReadLine();

}

Demo!Geekstream

4. ConclusionThe old, the new and the ugly

100% ACID Out of the Box?

SQL Server Redis OrigoDBAtomicity NO NO YESConsistency NO NO YESIsolation NO YES YESDurability YES YES YES

Comparison Matrix

SQL REDIS ORIGO

License/Cost $$ OSS/Free MIT/Free (+$)

Language TSQL Commands + Lua C#/LINQ

OLTP YES YES YES

OLAP (indexing) YES NO YES

In-process NO NO YES

Througput/latency 3 1 2

Modeling Relational Fixed Multi

Maturity 1 2 3

Size 250GB/DB Available RAM Available RAM

Thank you!

• Try Origo!• Contribute, it’s open source• http://origodb.com• @robertfriberg, robert@devrexlabs.com