Implementation of 'go-like' language constructions in scala [english version]

Go / Scala

Implementation of go language constructions in scala

Ruslan Shevchenko<ruslan@shevchenko.kiev.ua>

https://github.com/rssh/scala-gopher

Whats in go ?

scope ( defer/ panic / destroy )

parallelism channels select statement

Differences from 'native' scala approach How to implement go variant

Future directions...

Scope

def copy(inf: File, outf: File): Long = goScope { val in = new FileInputStream(inf) defer{ in.close() } val out = new FileOutputStream(outf); defer{ out.close() } out.getChannel() transferFrom(in.getChannel(), 0, Long.MaxValue) }

def copy(inf: File, outf: File): Long = { val in = new FileInputStream(inf) try { val out = new FileOutputStream(outf) try { out.getChannel() transferFrom(in.getChannel(), 0, Long.MaxValue) } finally { out.close(); } } finally { in.close(); }}

Old way:

Traditional way:

def copy(inf: File, outf: File): Long = { val in = new FileInputStream(inf) try { val out = new FileOutputStream(out) try { out.getChannel() transferFrom(in.getChannel(), 0,Long.MaxValue) } finally { out.close(); } } finally { in.close(); }}

def copy(inf: File, outf: File): Long = goScope { val in = new FileInputStream(inf) defer{ in.close() } val out = new FileOutputStream(outf); defer{ out.close() } out.getChannel() transferFrom(in.getChannel(), 0, Long.MaxValue) }

Go way

try

try

try

finally

finally

finally

op/defer

How this done is scala:

Macroses Call-by-name-params Traditional try/cath

goScope = .... (macro wich rewrite arg)

defer => @compileTimeOnly

def copy(inf: File, outf: File): Long = { val sc0 = new ScopeContext(); sc0.eval { val in = new FileInputStream(inf) sc0.pushDefer{ in.close() } val out = new FileOutputStream(outf); sc0.pushDefer{ out.close() } out.getChannel() transferFrom(in.getChannel(), 0, Long.MaxValue) } sc0.unwindDefers[Long]() }

Unsugared:

Scope: defer / panic / recover

panic => throw new PanicException() Recover => return

try { ....} catch { case ex: Exception => return x}

defer{ ......... recover(x)}

Channels

Read:

go: channel -> x

Scala: channel ~> x x = channel ? ?! ??

Writego: channel <- x

Scala: channel <~ x channel ! x !! (immediatly) !? (with timeout)

Select

Channels connect execution flows

go func() { for { select { case msg1 := <- c1: fmt.Println(msg1) case msg2 := <- c2: fmt.Println(msg2) } } }()

Go: typical pattern: select inside for inside goroutine

go for(s <- select) s match { case c1 ~> (msg1: Msg1Type) => println(msg1) case c2 ~>(msg2:Msg2Type) => println(msg2) }

Scala Analog:

- implemented as foreach macros- ~> pattern: require typing- active queue

Open question: native non-macro syntax

Unsugared (simplicified) [1]:

go { val s = new SelectContext(); s.addReader(c1) { (msg1:Msg1Type) => println(msg1) } s.addReader(c2) { (msg2:Msg2Type) => println(msg2) } s.run()}

Unsugared (simplicified) [2]:

val scope = new ScopeContext(); val s = new SelectContext();Future { s.addReader(c1) { (msg1:Msg1Type) => println(msg1) } s.addReader(c2) { (msg2:Msg2Type) => println(msg2) }} map { s.runUnblocked() } map { scope.finish() }

(Future work)

S

Just select:

for(s <- select.once) s match { case c1 ~> (msg1: Msg1Type) => println(msg1) case c2 ~>(msg2:Msg2Type) => println(msg2) }

select { case msg1 := <- c1: fmt.Println(msg1) case msg2 := <- c2: fmt.Println(msg2)}

GO

SCALA

Internal implementation:

Active channel Channel = [Blocked Query + Execution Context ]

Coordination primitives + wait for select Count down latch Memory barrier

Extensions over Go model async operations context shutdown.

val consumer = go { for(s <- select) { s match { case `channel` ~> (i:Int) => sum = sum + i if (i==1000) s.shutdown() } } sum }

Future[Int]

Akka (Erlang-like actors)

Channels (go-like)

Reactive No blocking operations

Request/Reply paradigm.

Mailbox overflow on overload

Handle something.

Flow oriented Wait is normal (switch during wait) One-direction pipe paradigm

wait on send On overload

Compute something

x1, x2, x3, ............

x1*y1, x2*y2, x3*y3, ............

y1, y2, y3, ............

go { while(;) { val x = channelX ? val y = channelY ? channelZ ! x*y }}

Example: Easy with channels, hard with Akka

bindWritebindRead

? [blocking operation]

Future directions

Redefining of base primitivies [select in loop in go] is too fundamental, to be combination of 3 constructions.

Native non-macro syntax Make scala type inference works

Pluggable implementations Optimize corner cases

Pluggable implementations.

Exists many strategies. (current implemented is not best)

Schedule/callback instead waits Play with own dispatcher .....

Performance tests, Implement/Prototype/Test loop,

Problems: Lack of import-based configuration in scala @exported import

Community help needed.

Rewriting blocking API into reactive

- like CPS (continuations passing style)

go { ... val x = channelX ? val y = channelY ? channelZ ! x*y ...} go { .....

channelX.read map { x => channelY.read map { y => channelZ.write map { ch => ch <~! x*y } } } }}

someCollection foldLeft { (s,x) => s + (x* channel ?) }

Problem – same as with CPS: - foldLeft – implemented not here.

- foreach

for(x <- something) channel <~~ x

Continuations not supported directly by JVM

Continuations in JVM

- Patched JVM - Byte-code rewriting. - Do nothing – thread pools now big (utilize wait by running Flow of non-blocking tasks here)

The Da Vinci machine

http://openjdk.java.net/projects/mlvm/index.html

AVIAN VM

http://oss.readytalk.com/avian/

Someday everything will be fine. For now - here is a cat

Continuations: Bytecode rewriting.

Frames:

Let's save frame during function end, instead deleting one.

foldLeft

? channel

+

*

Continuations: Bytecode rewriting.

Frames:

foldLeft

+

*

Continuations: Bytecode rewriting.

Frames:

Suspended

CPS transformation on bytecode level

foldLeft

+

*

Channel ?

Community help needed.

https://github.com/rssh/scala-gopher

Build examples of typical usage. Instrument ones for performance tests Provide better implementations.

Let's think together. Thanks for attention.

Ruslan Shevchenko <Ruslan@Shevchenko.Kiev.UA>