miguelinlas3@gmail.com !@miguelinlas3

Reactive applications Building concurrent and distributed apps using Akka

Miguel Ángel Pastor Olivar

Writing software for the Liferay infrastructure teamPhd student? on distributed and cloud systems

Scala enthusiast

Twitter: @miguelinlas3

About me

A little, very little, bit of experience with Erlang and OTP

Reactive apps: do we really need them? !

Crash course on basic Akka concepts !

Old known problems !

Bonus: A quick look into newer (and more fun tools)

Agenda

Applications has become extremely demanding !

New kind of systems !

Event driven, scalable , resilient, responsive !

This is what Reactive Applications are all about

Reactive applications

Blocking Caller blocked. Wait for the results !

Non blocking Caller not blocked. Need to pool

!

Asynchronous Caller not blocked. Select, epoll, signals

!

Event driven Message flowing through the system

Reactive applications: definitions

Basic pillars

Basic pillars

Event Driven

Basic pillars

Event Driven

Resilient

Basic pillars

Event Driven

Resilient Scalable

Basic pillars

Event Driven

Responsive

Resilient Scalable

Going event drivenOld known problems, The actor model

Going Event Driven

Going Event Driven

Shared Memory

Going Event Driven

Shared Memory

Going Event Driven

Shared Memory

Thread 1

Going Event Driven

Shared Memory

Thread 1 Thread 2

Going Event Driven

Shared Memory

Thread 1 Thread 2

Going Event Driven

Shared Memory

Thread 1 Thread 2

Going Event Driven

Shared Memory

Thread 1 Thread 2

R/W

Going Event Driven

Shared Memory

Thread 1 Thread 2

R/WR/W

Going Event Driven

Shared Memory

Thread 1 Thread 2

R/WR/W AVOID IT!!

Going event driven: problems with locks

Going event driven: problems with locks

Composition

Going event driven: problems with locks

CompositionLocks don´t compose

Going event driven: problems with locks

CompositionGranularity

Locks don´t compose

Going event driven: problems with locks

CompositionGranularity

Locks don´t compose

Have I taken too few locks? Too many?

Going event driven: problems with locks

CompositionGranularity

Encapsulation

Locks don´t compose

Have I taken too few locks? Too many?

Going event driven: problems with locks

CompositionGranularity

Encapsulation

Locks don´t compose

Have I taken too few locks? Too many?

Breaking abstractions

Going event driven: problems with locks

CompositionGranularity

EncapsulationCorrectness

Locks don´t compose

Have I taken too few locks? Too many?

Breaking abstractions

Going event driven: problems with locks

CompositionGranularity

EncapsulationCorrectness

Locks don´t compose

Have I taken too few locks? Too many?

Have I taken the correct lock?

Breaking abstractions

Going event driven: problems with locks

CompositionGranularity

EncapsulationCorrectness

Ordering

Locks don´t compose

Have I taken too few locks? Too many?

Have I taken the correct lock?

Breaking abstractions

Going event driven: problems with locks

CompositionGranularity

EncapsulationCorrectness

Ordering

Locks don´t compose

Have I taken too few locks? Too many?

Have I taken the correct lock?

Have I used the correct order?

Breaking abstractions

Asynchronous message/event passing !

Workflow of the events through your system !

Benefits: Better throughput Lower latencies Loosely coupled solutions

Going event driven: designing your system

Going event driven: Amdahl’s law

Going event driven: the actor model

Fundamental unit of computation that embodies: • Processing • Storage • Communication !

Three axioms. When an Actor receives a message it can: • Create new Actors • Send messages to Actors it knows • Designate how it should handle the next message it receives !Carl Hewitt

Going event driven: the actor model

Going event driven: the actor model

Going event driven: the actor model

Going event driven: the actor model

Zero sharing !

Isolated, lightweight event based processes !

Communication through asynchronous message passing !

Location transparent !

Built-in supervision

Actor model: Define a new actor

class Frontend extends Actor { … ! def receive = { case work => implicit val timeout = Timeout(5.seconds) ! (mediator ? Send("/user/master/active", work, localAffinity = false)) map { case Master.Ack(_) => Ok } recover { case _ => NotOk } pipeTo sender ! } !}

Actor model: Creating an actor

val system = ActorSystem(systemName) !Cluster(system).join(joinAddress) !val frontend = system.actorOf(Props[Frontend], “frontend") !system.actorOf(Props(classOf[WorkProducer], frontend), “producer") !system.actorOf(Props[WorkResultConsumer], "consumer")

Actor model: Getting a reference

system = ActorSystem(systemName) !val initialContacts = Set(system.actorSelection(RootActorPath(contactAddress) / "user" / "receptionist"))

Actor model: Sending a message

case Work(workId, job) => … workExecutor ! job …

Actor model: Replying to a message

case RegisterWorker(workerId) => if (workers.contains(workerId)) { workers += (workerId -> workers(workerId).copy(ref = sender)) } else { workers += (workerId -> WorkerState(sender, status = Idle)) if (pendingWork.nonEmpty) sender ! WorkIsReady }

Actor model: Switch implementation

def waitForWorkIsDoneAck(result: Any): Receive = { case Ack(id) if id == workId => sendToMaster(WorkerRequestsWork(workerId)) ! context.setReceiveTimeout(Duration.Undefined) context.become(idle) }

Actor model: Routing

val writer = system.actorOf( Props(new TweetWriter(cluster)).withRouter(RoundRobinRouter(nrOfInstances = 100)))

Actor model: Configuring a router

akka { actor { deployment { /TweetWriter { router = round-robin nr-of-instances = 100 } } } }

Actor model: Dispatchers

Default dispatcher !

PinnedDispatcher !

BalancingDispatcher !

CallingThreadDispatcher

Actor model: Dispatchers

my-pinned-dispatcher { executor = "thread-pool-executor" type = PinnedDispatcher } !val myActor = context.actorOf(Props[MyActor].withDispatcher("my-pinned-dispatcher"), "myactor2")

Going scalableDistributed computing, The actor model

Going scalable: scalability

I really do have an scalability problem ... !

if my system is fast for a single user but slow when the system is under heavy load

Going scalable: transparent distributed computing

Distributed shared mutable state !

Distributed objects !

Distributed transactions

Going scalable: transparent distributed computing

Synchronous method dispatching across the network is a bad idea !

Ignores partial failures !

Raises Latency !

General scalability and DC concerns

Going scalable: transparent distributed computing

EMBRACE THE NETWORK!!

Going scalable: remote actors

akka { actor { deployment { /metricsActor { provider = akka.remote.RemoteActorProvider

remote = "akka://MonitoringServiceSystem@hostname:5557"

} } } }

val monitoringServiceSystem = ActorSystem(“MonitoringServiceSystem");

!def metricsActor = monitoringServiceSystem.actorOf( new Props(MetricsProcessorActor.class), "metricsActor");

No changes are required in our existing code!!

Going scalable: clustering

Gossip cluster membership !

Failure detector !

Cluster death watch !

Cluster aware routers

Going scalable: cluster membership

Dynamo based (heavily influenced by Riak) !

Gossip protocol for state dissemination !

Cluster membership Configuration data Leader determination !

Vector clocks to detect convergence !

!

Going scalable: leaders

P2P approach !

Any node can be the leader !

Deterministically recognised by all nodes

Going scalable: cluster usageakka { actor { provider = "akka.cluster.ClusterActorRefProvider" } remote { log-remote-lifecycle-events = off netty.tcp { hostname = "127.0.0.1" port = 0 } } cluster { seed-nodes = [ "akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://ClusterSystem@127.0.0.1:2552"] auto-down-unreachable-after = 10s } }

Going resilientError handling

Going resilient: common problems

Single thread of control !

Thread blows up --> you are screwed !

Error handling within single thread !

Errors don´t get propagated !

Defensive programming Tangled within business logic Scattered along all our codebase

Going resilient: actor topology and supervision

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B5

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B5

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7

Supervision strategies: One for one

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B5

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B6

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

B7

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

Supervision strategies: One for all

Syste

A1

A2

A4

B1

B2

B4

B3

A3

/A1

/A1/A2

/A1/A2/A4

/A1/A3

/B1

/B1/B2

/B1/B2/B4

/B1/B3

/B1/B3/B5

/B1/B2/B6

/B1/B2/B7B5

B6

B7

Going resilient: configure supervision

override def supervisorStrategy = OneForOneStrategy() { case _: ActorInitializationException => Stop case _: DeathPactException => Stop case _: Exception => currentWorkId foreach { workId => sendToMaster(WorkFailed(workerId, workId)) } context.become(idle) Restart }

Supervision strategies are pluggable and you can develop and configure your own

Something more?Of course!

Something more

Akka Http (previously know as Spray.io) !

Akka.io !

Akka Reactive Streams !

Akka Event Sourced !

Dataflow

TakeawaysQuick summary of main ideas

Takeaways

Go reactive !

Avoid locking (share nothing, lock free algs) !

Already in the JVM? Akka could be good choice !

Distributed systems are hard !

Use the correct level of abstraction !

And remember ...

Takeaways

When all you have is a hammer everything looks like

a nail. !

Choose the

right tool for your job!

Questions?And let’s hope answers

BONUS!We deserve better and more fun tools

More tools: futures

•Spawn concurrent computations !

•Write once - Read many !

•Freely sharable !

•Non-blocking composition !

•Composable (monadic operations) !

•Managing failure

More tools: futures

val f: Future[List[String]] = future { session.getRecentPosts } !// handle both cases: Success and Failure f onComplete { case Success(posts) => for (post <- posts) println(post) case Failure(t) => println("An error has occured: " +

t.getMessage) } !// only handle Success f onSuccess { case posts => for (post <- posts) println(post) } !// only handle Failure f onFailure { case t => println("An error has occured: " + t.getMessage) }

More tools: agents

•Reactive memory cells !

•Send update function to an agent !

•Reads are “free” !

•Composable !

•Originally in Clojure, borrowed by Akka

More tools: agents

// declare the agent val counter = Agent(0) !// send the function: enqueued the change counter send { _ + 1} !// long running or blocking operations !implicit val ec = defineSomeExecutionContext() agent sendOff blockingOperation !!// read the agent’ s value val agentValue = counter.value

More tools: software transactional memory

Sometimes we will deal with shared state Heap + Stack: transactional dataset: !

Begin, commit, rollback !

Retry on collision !

Rollback on abort !

Transaction composability

More tools: reactive extensions

•Futures + Stream concept !

•Composables !

•Async and event based !

•Push collections !

•JVM available through the RxJava project

More tools: reactive extensions

!def simpleComposition() { customObservableNonBlocking() .skip(10) .take(5) .map({ stringValue -> return stringValue + "_transformed"}) .subscribe({ println "nextElement => " + it}) } // we should an output like this !nextElement => foo_10_transformed nextElement => foo_11_transformed nextElement => foo_12_transformed nextElement => foo_13_transformed nextElement => foo_14_transformed

More tools: reactive extensions!def Observable<T> getData(int id) { if(availableInMemory) { return Observable.create({ observer -> observer.onNext(valueFromMemory); observer.onCompleted(); }) } else { return Observable.create({ observer -> executor.submit({ try { T value = getValueFromRemoteService(id); observer.onNext(value); observer.onCompleted(); }catch(Exception e) { observer.onError(e); } }) }); } }

Useful approaches

Where should I use each tool?

Layers of complexity

Declarative & immutable core Logic or functional programming Futures and/or dataflow

!

Non-determinism when needed Actors, Agents or Rx

!

Shared mutability Protected by Software Transactional Memory

!

Finally, only if it is really needed Locks and explicit threads

Takeways

A quick summary

References

Interesting resources

References: slides, talks and books

High Performance Networking in the JVM by Eric Onnen !

Going reactive talk by Jonas Bonér !

Introducing Akka by Jonas Bonér !

The future I was promised by Viktor Klang !

Real world Akka recipes by Björn Antonsson et all

References: slides, talks and books

Programming Concurrency in the JVM: Mastering synchronization, STM and Actors by Venkat Subramaniam !

Systems than run forever Self-heal by Joe Amstrong !

The art of multiprocessor programming, Revised Reprint by Maurice Herlihy !

Reactive Manifesto

References: slides, talks and books

Akka Concurrency: Building reliable software in a multi-core world !

Reative Design Patterns (available as Manning MEAP at the time of this writing)

References: frameworks and toolkits

Erlang !

Akka Toolkit !

Scala language !

Vert.x !

Clojure language !

GPars

References: frameworks and toolkits

Cloud Haskell !

NodeJS !

Netty IO !

Finagle