The Rise and Rise of Dataflow in the JavaVerse · Copyright © 2016 Russel Winder 71 The Rise and Rise of Dataflow in the JavaVerse. Author: Russel Winder Created Date: 6/14/2016

Copyright © 2016 Russel Winder 1

The Rise and Rise ofDataflow in the JavaVerse

Russel Winder

@[email protected]

https://www.russel.org.uk

mailto:[email protected]


The Plan for the Session● Stuff.● More stuff.● Even more stuff – possibly.● Summary and conclusions – maybe.● Q & A

Notice theflow goingon here.


Some Definitions● Dataflow — moving data from one transformation to another.● Javaverse —

– the Java Platform: the JVM the Java compiler, and the standard library;

– other languages such as Kotlin, Ceylon, Groovy, Scala, Clojure, etc;– other libraries from Bintray, Maven Central, etc.


More on the Term Dataflow● Wikipedia: https://en.wikipedia.org/wiki/Dataflow

– Dataflow can also be called stream processing or reactive programming.

– Dataflow programming is about pipelines.– …data flow programming promotes high-level functional style…

Streams, pipelines imply one-dimensional.

https://en.wikipedia.org/wiki/Dataflow


Some Libraries of Interest● GPars — http://www.gpars.org/● Akka — http://akka.io/● Quasar — http://docs.paralleluniverse.co/quasar/● RxJava — https://github.com/ReactiveX/RxJava● Apache Spark — http://spark.apache.org/


What is a Program?

n m


Example


Let’s be a bit abstractfor the moment.


Let Us Assume Java and…

public static Integer f1(final Integer i) { return i * 2; }



All this in a class.


Compute a Value Given an Input

public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3;}

Relatively declarative style.Quite dataflow oriented.


Diagrammatically



public static Integer statementSequence(final Integer i) { final Integer i1 = f1(i); final Integer i2 = f2(i1); final Integer i3 = f3(i2); return i3;}

Relatively declarative style.Quite dataflow oriented.



public static Integer statementSequence(final Integer i) { Integer x = f1(i); x = f2(x); x = f3(x); return x;}

Very traditionalImperative style.


State changevs.

Data flow



A more functional anddeclarative approach.

public static Integer functionApplication(final Integer i) { return f3(f2(f1(i)));}


Diagrammatically


Compilers deal in “dataflow analysis”,let us not forget this view of a program.


That is all very Java,how about doing it with

Frege?


The Frege Transforms

f1 = (* 2)

f2 = (* 3)

f3 = (* 4)

Pointfree definitionof functions usingpartial evaluation.Frege has top-level functions.



bindingSequenceScalar i = f3 i3 where i2 = f1 i i3 = f2 i2

An Imperative stylefunctional approach.


Diagrammatically



functionApplicationScalarExplicit i = f3 (f2 (f1 i))





functionApplicationScalar i = f3 $ f2 $ f1 $ i




functionCompositionScalar i = f3 . f2 . f1 $ i


Diagrammatically


Let’s do the maths…


Function Composition

f 3(f 2(f 1(x))) = (f 3∘ f 2∘ f 1)(x)

It’s only a bit of maths,do not be afraid.




functionSavedCompositionScalar i = f i where f = f3 . f2 . f1


This example isseriously unrealistic.


Make it a wee bit more realisticby switching to a

potentially infinite dataset…


…let’s stage this by doinga finite sequence first…


…let’s go(statically compiled) Groovy…

Do not have to have classes,top-level functions are allowed.


The fs but Groovy

Integer f1(final Integer i) { i * 2 }




Compute a Value Given an InputList<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { final i1 = f1(i) final i2 = f2(i1) final i3 = f3(i2) result.add(i3) } result}


Compute a Value Given an InputList<Integer> statementSequence(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i: l) { def x = f1(i) x = f2(x) x = f3(x) result.add(x) } result}



List<Integer> functionApplication(final List<Integer> l) { final result = new ArrayList<Integer>() for (final Integer i : l) { result.add(f3(f2(f1(i)))) } result}


But this is all about state,no real dataflow. So…



List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList())}

This is using Streams from the Java Platform library.


Diagrammatically


Diagrammatically



List<Integer> usingStream(final List<Integer> l) { l.stream() .map(this.&f1) .map(this.&f2) .map(this.&f3) .collect(Collectors.toList())}

This is using Streams from the Java Platform library.

Intermediate

Terminal


Do this with explicit composition?


Frege

functionCompositionSequence l = map (f3 . f2 . f1) l


Let’s introduce anew language:

Kotlin

Do not have to have classes,top-level functions are allowed.


The Three Functions

fun f1(i:Int):Int = i * 2



We can already tell thatKotlin will be lots of fun.


Using Streams

fun usingStream(l:List<Int>):List<Int> = l.stream() .map(::f1) .map(::f2) .map(::f3) .collect(Collectors.toList<Int>())


Kotlin version

fun usingMap(l:List<Int>):List<Int> = l.map(::f1).map(::f2).map(::f3)


Using composition

fun usingComposedMap(l:List<Int>):List<Int > = l.map(::f1 compose ::f2 compose ::f3)


Kotlin Compose

infix fun<V, T, R> Function1<T, R>.compose(before: (V) -> T): (V) -> R { return { v: V -> this(before(v)) }}

t ∘b (i) = t (b(i))


But this is still finite, whatabout potentially infinite?


Cannot do collect.

Map is another name forcollect in most circumstances.


Infinite Data● With an infinite data sequence you can:

– Do some form of reduction Or windowing.– Perform a side-effect, e.g. output.


New (more realistic) problem…


Cumulative mean andStandard deviation.


Equation warning:please do not be afraid.


x̄=1n∑i=0

nxi

s=√ 1n−1∑i=0

n(x i− x̄)2


x̄=1n∑i=0

nxi

s=√ 1n−1

((∑i=0

nx i

2)−n x̄2)



I only do equations after being fed…


Code?


A little architecture first.



Code.


What’s the Message?● Small, single threaded, communicating processes are easy to

program. (Communicating Sequential Processes, CSP)● Threadpools and processpools make parallelism easy to

realize, without manual locks.● Most calculations and dataset are now very big, hence “Big

Data”.


Parallelism is mandatoryFor “Big Data”.


Dataflow not stateis required for parallelism.



Russel Winder

@[email protected]




But before I go…



Q & A



Russel Winder

@[email protected]



Documents

The Rise and Rise of Dataflow in the JavaVerse · Copyright © 2016 Russel Winder 71 The Rise and Rise of Dataflow in the JavaVerse. Author: Russel Winder Created Date: 6/14/2016