Scale up your thinking

© 2014 VMware Inc. All rights reserved.

Scale Up Your Thinking

Reactive Programming with Scala and Akka

Lior Shapsa Yardena Meymann

February 24, 2014

2

About us• Lior Shapsa – Leads Benchmark Development (lshapsa@vmware.com)

• Yardena Meymann - Group Architect (ymeymann@vmware.com)

3

The Project

• New Generation Business Management Products

• Manage Financial Aspects of the Cloud• Help CFOs to optimize the cloud cost

• Benchmarking – Compare to Industry Best Practices

What Do We Need

• Web Scraping – naïve way isn’t good enough

• Crowd Sourcing

• Analyze Data– Build a reliable price list

– How efficient is my virtual infra vs. industry? (cost, capacity)

– How fast am I responding to business challenges

• Start With Single Instance and Scale Out Fast!

Why to Scale Out

Moore’s Law

• The Clock Speed Has

Reached Its Limit in 2006

• Free Lunch Is Over

Why to Scale Out

Amdahl’s Law• The New Reality

http://en.wikipedia.org/wiki/Amdahl's_law

The Challenge

Shared mutable state -> race conditions

Threads are expensive

Locks are hard (and expensive)

8 Fallacies of Distributed Computing:

– The network is reliable– Latency is zero– Bandwidth is infinite– The network is secure– Topology doesn't change– There is one administrator– Transport cost is zero– The network is homogeneous

The Challenge

9

Reactive Programming

• React to Events– Pushing rather than pulling (No Shared State)

• React to Load – Focus on scalability rather than single-user performance.

• React to Failure – Resilient systems with the ability to recover quickly at all levels.

• React to Users– Combine the above traits for an interactive user experience

Chosen Solution: Actor Model

• Formalized in 1973 by Carl Hewitt and refined by Gul Agha in mid 80s. – The first major adoption is done by Ericsson in mid 80s.

• Invented Erlang and later open-sourced in 90s.

• Built a distributed, concurrent, and fault-tolerant telecom system which has 99.9999999% uptime

ActorActor

Mailbox

state

behavior state

behaviorMailbox

Chosen Solution: Actor Model

• Actor Instead of Object, Asynch Message Instead of Method Call

• Keep Mutable State Internal, Share Nothing

• Communicate via Immutable Message Passing

• Messages are Kept in Mailbox and Processed Sequentially

• Asynchronous and Non-blocking

Actor Example (Scala)

• Definecase class Greeting(who: String) //messageclass GreetingActor extends Actor {

def receive = {case Greeting(who) => sender ! “Hello ” + who

} }

• Createval sys = ActorSystem("MySystem")val greeter = sys.actorOf(Props[GreetingActor], "greeter")

• Sendgreeter ! Greeting(“Akka”);

Actor Libs For the JVM

• Akka (Java/Scala)• Kilim (Java)

• Jetlang (Java)

• Actor’s Guild (Java)

• Actorom (Java)

• FunctionalJava (Java)

• GPar (Groovy)

Web Scraping - Example

Dispatch

Crawling library

Netty

Web Scraping Actors Design

Curiosity

Site …

Pages

Level1 Level1

Level2

…

Level2

File writer

Site

…

k/vk/v

Example Site

class Servers extends Site { override def start(): Unit = goto[ListPage] “…”}class ListPage extends Page { override def processResult(result: Response): Unit = { val links = listProductLinks(result.content) links foreach goto[ProductPage](_, pageType = Child) val next = getNextPage(result.content) next foreach goto[ListPage](_, pageType = Sibling) }}class ProductPage extends Page { override def processResult(result: Response): Unit = { val spec = productTitle(result.content) ++ generalSpec(result.content) ++ pricingInfo(result.content) writeToFile(spec) }}

Supervision

• Let It Crash!– Errors are encapsulated as messages and isolated from the BL

– The caller is not necessarily available

• The Supervisor Reacts Upon Failure– Resume, keeping its accumulated internal state

– Restart, clearing out its accumulated internal state

– Terminate

– Escalate

Supervision – Example (optional)

override val supervisorStrategy =

OneForOneStrategy(maxNrOfRetries = 10,

withinTimeRange = 1 minute) {

case _: ArithmeticException => Resume

case _: NullPointerException => Restart

case _: Exception => Escalate

}

Akka Distributable by Design

• Superior model for detecting and recovering from errors

• Location transparent - scale UP and OUT for free

• Adaptive load-balancing, cluster rebalancing & actor migration

• Build extremely loosely coupled and dynamic systems that can change and adapt at runtime

Under the hood

Receive Method

• Pattern match on the messageclass Site… extends Actor … {

override def receive: Receive = {

case Start =>

log.info(s"Starting $siteName ...")

start()

…

}

• Defined for some (but not necessarily all) messages

• Unhandled messages are sent to a special location

Splitting Behavior Between Methods

• Group related messages in a receive function

• Compose receive functions with orElse operator

private def process: Receive = {

case res: Response => …

case Status.Failure(e) => …

}

private def monitor: Receive = {

case Create… => …

case Terminated(actor) => …

}

protected override def receive: Receive = process orElse monitor

Curiosity

Crawler

Crawling library

Netty

Site …

Pages

Page

Page

Page

…

Page

File writer

Site

…

Using Traits to Compose a Behavior

trait Percolator { myself: Actor =>

protected def percolate: Receive = …

}

trait Composite { myself: Actor =>

protected def monitor: Receive = …

}

class Page extends Actor with Percolator with Composite {

private def process: Receive = …

protected override def receive =

process orElse percolate orElse monitor

}

Composite Percolator

Actor

Page

c

t t

c

Web Scraping Actors Design

Curiosity

Dispatch

Crawling library

Netty

Site …

Pages

Level1 Level1

Level2

…

Level2

File writer

Site

…

Propagate Messages

You can forward messages without understanding them, “methodMissing”

trait Percolator { myself: Actor =>

protected def percolate: Receive = {

case _ => parent forward msg

}

class Site… extends Actor {

protected def override receive: Receive = {

case msg: Goto => crawler forward msg

case msg: WriteFile => writer forward msg

}

Futures• Future holds a value which may become available at some point

– completed when the computation results in a value or an exception

• java.util.concurrent has futures, but we can do better…

• Scala’s futures are asynchronously compose-able– Callbacks: onSuccess, onFailure, onComplete

val f: Future[List[String]] = future { session.getRecentPosts }f onFailure { case e => println("An error has occurred: " + e.getMessage)}f onSuccess { case posts => for (post <- posts) println(post)}

• More and more API’s are non-blocking - returning futures

Tell vs Ask

• Tell, a.k.a. !– Fire & forget

– Best concurrency and scalability

• Ask, a.k.a. ?– sends and (optionally) wait for reply

– uses delayed computation (Futures)

– combining futures and actors – pipeTo trick

import akka.pattern.pipe

val page: ActorRef = …

val f: Future[xml.NodeSeq] = Http.browse(url(…) OK as.tagsoup.NodeSeq)

f pipeTo page

• Why we decided to stick with “bare actors”

A Closer Look at Akka Actor

Reference

Instance

Instance factory

Context

MailboxReceive function

Receive functionBehavior

Actor State

It’s Ok to have mutable state inside actor

trait Composite { myself: Actor =>

private var childrenCount = 0

override def receive: Receive = {

case Created… =>

childrenCount = childrenCount + 1

case Terminated… =>

childrenCount = childrenCount – 1

if (childrenCount == 0) context stop self

}

Lightweight State-Machine with become

class Curiosity extends Actor { private def free: Receive = { case Start => crawlers foreach (_ ! Start) context become busy case Status | Stop => sender ! Free } private def busy: Receive = { case Stop => crawlers foreach (_ ! Stop) context become awaitingTermination case Status | Start => sender ! Busy case AllFinished => context become free } private def awaitingTermination: Receive = { case AllFinished => context become free case Status | Start | Stop => sender ! Busy } override def receive: Receive = free}

FreeBusy

Awaiting Termination

Start

StopFinished

Finished

Actor Hierarchy DO’s and DON’Ts

• DO give actors meaningful names if you can

• DON’T be afraid to create lots of short-living actors

• DON’T nest actor classes in one another

• DO note that restart that results from failure is not the same as a graceful stop

• DON’T define public methods on Actors

• DO choose supervision strategy wisely, build your hierarchy according to the strategy

• DON’T confuse actor.Status.Failure with util.Failure

• DO make sure messages are immutable

• DON’T rely on using generics in messages, as they will be erased in bytecode (unless you reify with ClassTag/TypeTag)

Stopping Actors

• Watching another actor – Death Watch– Not same as supervision

– Use context.watch(actorReference)– Terminated(actorReference) events

• Stop vs. PoisonPill and how we used both– context.stop – terminate after current message

– context.kill – terminate now

– PoisonPill message – terminate after processing all messages

Web Scraping - Termination

Curiosity

Dispatch

Crawling library

Netty

Site …

Pages

Page Page

Page

…

Page

File writer

Site

…

k/vk/vk/vk/vk/vk/v

34

ETL – Decrypt Translate & Load (optional)

Router

Router

Load Manager

Decrypt Worker

Translate Worker

ArchiverMongo Loader

Decrypt Manager

Translate WorkerTranslate

Worker

Decrypt WorkerDecrypt

Worker

Directory Watcher

Decrypt WorkerDecrypt

Worker

Load ManagerDecrypt Manager

Directory Watcher

Exposing Actors with Spray

• Spray – lightweight HTTP server, non-blocking, 100% Akka

• Making your actors respond to HTTP requests

import akka.io.IOimport spray.can.Httpimport spray.routing.HttpService

val Handler = new Actor with HttpService { def actorRefFactory = context def receive = runRoute { path("start") { … { crawlers ! Start ; …}} ~ path("stop") { … { crawlers ! Stop ; …}} ~ … }}

IO(Http)(system) ! Http.Bind(handler, interface = … , port = …)

Testing Akka

• ScalaTest – BDD (Given-When-Then)

• ScalaMock & Mockito

• Measuring Scala code coverage with sbt (scct + cobertura + some tricks)

• Using Akka TestKit + Spray

• ScalaCheck

Testing Akka - Example

import org.scalatest._

import akka.testkit.{ImplicitSender , TestActorRef, TestKit}

class CuriosityTest extends TestKit(new ActorSystem(“…”) … {

val myTestedActor = TestActorRef[Curiosity]

“Curiosity" must {

"display free status when not working" in {

myTestedActor ! Status

expectMsg(Free)

}

}

…

}

Testing Akka – Running Tests

Debugging Akka

• Typesafe console

• Logs, watching for DeadLetter

• Using Scala clone of Yammer Metrics

Typesafe Console

Q&A

Our Amazing Team

If you want to join, find us at http://vmware.jobs/

Thank You!