Upload
spring-io
View
537
Download
1
Embed Size (px)
DESCRIPTION
Speaker: Paul King Groovy doesn't claim to be a fully-fledged functional programming language but it does provide the Java or Groovy developer with a whole toolbox of features for doing functional style programs. This talk looks at the key Groovy features which support a functional style. Topics covered include using closures, currying and partial evaluation, closure composition, useful functional-centric AST macros, useful functional-centric runtime meta-programming tricks, trampolining, using Java functional libraries, immutable data structures, lazy and infinite lists, using Groovy 2's static typing and approaches for moving beyond Java's type system. There are many advantages to using a functional style in your programs. Learn what can be done to leverage functional style while retaining many of the productivity gains of the Groovy programming language.
Citation preview
© A
SE
RT
2006-2
013
Dr Paul King
@paulk_asert
http:/slideshare.net/paulk_asert/functional-groovy
https://github.com/paulk-asert/functional-groovy
Functional Groov
Topics
Intro
• Functional Style
• Design Patterns
• Immutability
• Laziness
• GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
Introduction
• What is functional programming? – Favour evaluation of composable
expressions over execution of commands
– Encourage particular idioms such as side-
effect free functions & immutability
• And why should I care?
– Declarative understandable code
– Reduction of errors
– Better patterns and approaches to design
– Improved reusability
– Leverage concurrency
© A
SE
RT
2006-2
013
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Using Closures...
• Code deserves to be free
© A
SE
RT
2006-2
013
public class Main { public static void main(String[] args) { print(reverse("Hello")); } public static String reverse(String arg) { return arg.reverse(); } public void static print(String arg) { System.out.println(arg); } } Main.main();
...Using Closures...
• Code deserves to be free
© A
SE
RT
2006-2
013
def codeListInG = [ { println it }, { it.reverse() } ] def main(first, second, arg) { def third = second >> first third(arg) } // def main = { Closure first, Closure second, String arg -> // def mainClosure = this.&main main(codeListInG[0], codeListInG[1], "Hello") // => olleH
def code = { arg -> println "Hello $arg" } code.call('Washington') code('Washington') // => Hello Washington
...Using Closures...
• Used for many things in Groovy: • Iterators
• Callbacks
• Higher-order functions
• Specialized control structures
• Dynamic method definition
• Resource allocation
• Threads
• Continuation-like coding
© A
SE
RT
2006-2
013
def houston(Closure doit) { (10..1).each { count -> doit(count) } } houston { println it }
new File('/x.txt').eachLine { println it }
3.times { println 'Hi' } [0, 1, 2].each { number -> println number } [0, 1, 2].each { println it} def printit = { println it } [0, 1, 2].each printit
...Using Closures
© A
SE
RT
2006-2
013
import static Math.* piA = { 22 / 7 } piB = { 333/106 } piC = { 355/113 } piD = { 0.6 * (3 + sqrt(5)) } piE = { 22/17 + 37/47 + 88/83 } piF = { sqrt(sqrt(2143/22)) } howCloseToPI = { abs(it.value() - PI) } algorithms = [piA:piA, piB:piB, piC:piC, piD:piD, piE:piE, piF:piF] findBestPI(algorithms) def findBestPI(map) { map.entrySet().sort(howCloseToPI).each { entry -> def diff = howCloseToPI(entry) println "Algorithm $entry.key differs by $diff" } }
Algorithm piE differs by 1.0206946399193839E-11 Algorithm piF differs by 1.0070735356748628E-9 Algorithm piC differs by 2.668102068170697E-7 Algorithm piD differs by 4.813291008076703E-5 Algorithm piB differs by 8.321958979307098E-5 Algorithm piA differs by 0.001264489310206951
© A
SE
RT
2006-2
013
Better Design Patterns: Builder
<html> <head> <title>Hello</title> </head> <body> <ul> <li>world 1</li> <li>world 2</li> <li>world 3</li> <li>world 4</li> <li>world 5</li> </ul> </body> </html>
import groovy.xml.* def page = new MarkupBuilder() page.html { head { title 'Hello' } body { ul { for (count in 1..5) { li "world $count" } } } }
• Markup Builder
© A
SE
RT
2006-2
013
SwingBuilder import java.awt.FlowLayout builder = new groovy.swing.SwingBuilder() langs = ["Groovy", "Ruby", "Python", "Pnuts"] gui = builder.frame(size: [290, 100], title: 'Swinging with Groovy!') { panel(layout: new FlowLayout()) { panel(layout: new FlowLayout()) { for (lang in langs) { checkBox(text: lang) } } button(text: 'Groovy Button', actionPerformed: { builder.optionPane(message: 'Indubitably Groovy!'). createDialog(null, 'Zen Message').show() }) button(text: 'Groovy Quit', actionPerformed: {System.exit(0)}) } } gui.show()
Source: http://www.ibm.com/developerworks/java/library/j-pg04125/
Better File Manipulation...
© A
SE
RT
2006-2
013
import java.util.List; import java.util.ArrayList; import java.util.Arrays; import java.io.File; import java.io.FileOutputStream; import java.io.PrintWriter; import java.io.FileNotFoundException; import java.io.IOException; import java.io.BufferedReader; import java.io.InputStreamReader; import java.io.FileInputStream;
public class PrintJavaSourceFileLinesThatContainTheWordJava { public static void main(String[] args) { File outfile = new File("result.txt"); outfile.delete(); File basedir = new File(".."); List<File> files = new ArrayList<File>(); files.add(basedir); FileOutputStream fos = null; PrintWriter out = null; try { fos = new FileOutputStream(outfile); out = new PrintWriter(fos); while (!files.isEmpty()) { File file = files.remove(0); if (file.isDirectory()) { files.addAll(Arrays.asList(file.listFiles())); } else { if (file.getName().endsWith(".java")) { processFile(file, out); } } } } catch (FileNotFoundException e) { e.printStackTrace(); } finally { if (out != null) { out.close(); } if (fos != null) { try { fos.close(); } catch (IOException e) { e.printStackTrace(); } } } } // ...
// ... private static void processFile(File file, PrintWriter out) { FileInputStream fis = null; InputStreamReader isr = null; BufferedReader reader = null; try { fis = new FileInputStream(file); isr = new InputStreamReader(fis); reader = new BufferedReader(isr); String nextline; int count = 0; while ((nextline = reader.readLine()) != null) { count++; if (nextline.toLowerCase().contains("java")) { out.println("File '" + file + "' on line " + count); out.println(nextline); out.println(); } } } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally { if (reader != null) { try { reader.close(); } catch (IOException e) { e.printStackTrace(); } } if (isr != null) { try { isr.close(); } catch (IOException e) { e.printStackTrace(); } } if (fis != null) { try { fis.close(); } catch (IOException e) { e.printStackTrace(); } } } } }
...Better File Manipulation...
© A
SE
RT
2006-2
013
import java.util.List; import java.util.ArrayList; import java.util.Arrays; import java.io.File; import java.io.FileOutputStream; import java.io.PrintWriter; import java.io.FileNotFoundException; import java.io.IOException; import java.io.BufferedReader; import java.io.InputStreamReader; import java.io.FileInputStream;
public class PrintJavaSourceFileLinesThatContainTheWordJava { public static void main(String[] args) { File outfile = new File("result.txt"); outfile.delete(); File basedir = new File(".."); List<File> files = new ArrayList<File>(); files.add(basedir); FileOutputStream fos = null; PrintWriter out = null; try { fos = new FileOutputStream(outfile); out = new PrintWriter(fos); while (!files.isEmpty()) { File file = files.remove(0); if (file.isDirectory()) { files.addAll(Arrays.asList(file.listFiles())); } else { if (file.getName().endsWith(".java")) { processFile(file, out); } } } } catch (FileNotFoundException e) { e.printStackTrace(); } finally { if (out != null) { out.close(); } if (fos != null) { try { fos.close(); } catch (IOException e) { e.printStackTrace(); } } } } // ...
// ... private static void processFile(File file, PrintWriter out) { FileInputStream fis = null; InputStreamReader isr = null; BufferedReader reader = null; try { fis = new FileInputStream(file); isr = new InputStreamReader(fis); reader = new BufferedReader(isr); String nextline; int count = 0; while ((nextline = reader.readLine()) != null) { count++; if (nextline.toLowerCase().contains("java")) { out.println("File '" + file + "' on line " + count); out.println(nextline); out.println(); } } } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } finally { if (reader != null) { try { reader.close(); } catch (IOException e) { e.printStackTrace(); } } if (isr != null) { try { isr.close(); } catch (IOException e) { e.printStackTrace(); } } if (fis != null) { try { fis.close(); } catch (IOException e) { e.printStackTrace(); } } } } }
boilerplate
...Better File Manipulation
© A
SE
RT
2006-2
013
def out = new File('result.txt') out.delete() new File('..').eachFileRecurse { file -> if (file.name.endsWith('.groovy')) { file.eachLine { line, num -> if (line.toLowerCase().contains('groovy')) out << "File '$file' on line $num\n$line\n\n" } } }
File '..\files\src\PrintGroovySourceLines.groovy' on line 4 if (file.name.endsWith('.groovy')) { File '..\files\src\PrintGroovySourceLines.groovy' on line 6 if (line.toLowerCase().contains('groovy')) File '..\jdbc\src\JdbcGroovy.groovy' on line 1 import groovy.sql.Sql …
DSL example...
© A
SE
RT
2006-2
013
Object.metaClass.please = { clos -> clos(delegate) } Object.metaClass.the = { clos -> delegate[1](clos(delegate[0])) } show = { thing -> [thing, { println it }] } square_root = { Math.sqrt(it) } given = { it } given 100 please show the square_root // ==> 10.0
...DSL example...
© A
SE
RT
2006-2
013
Object.metaClass.of = { delegate[0](delegate[1](it)) } Object.metaClass.the = { clos -> [delegate[0], clos] } show = [{ println it }] square_root = { Math.sqrt(it) } please = { it } please show the square_root of 100 // ==> 10.0
...DSL example...
© A
SE
RT
2006-2
013
show = { println it } square_root = { Math.sqrt(it) } def please(action) { [the: { what -> [of: { n -> action(what(n)) }] }] } please show the square_root of 100 // ==> 10.0
Inspiration for this example came from …
...DSL example
© A
SE
RT
2006-2
013
// Japanese DSL using GEP3 rules Object.metaClass.を = Object.metaClass.の = { clos -> clos(delegate) } まず = { it } 表示する = { println it } 平方根 = { Math.sqrt(it) } まず 100 の 平方根 を 表示する // First, show the square root of 100 // => 10.0
// source: http://d.hatena.ne.jp/uehaj/20100919/1284906117
// http://groovyconsole.appspot.com/edit/241001
Topics
• Intro
Functional Style
• Design Patterns
• Immutability
• Laziness
• GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Show me the code
Example1.groovy
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Show me the code
Example2.groovy
Topics
• Intro
• Functional Style
Design Patterns
• Immutability
• Laziness
• GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
interface Calc { def execute(n, m) } class CalcByMult implements Calc { def execute(n, m) { n * m } } class CalcByManyAdds implements Calc { def execute(n, m) { def result = 0 n.times { result += m } return result } } def sampleData = [ [3, 4, 12], [5, -5, -25] ] Calc[] multiplicationStrategies = [ new CalcByMult(), new CalcByManyAdds() ] sampleData.each {data -> multiplicationStrategies.each {calc -> assert data[2] == calc.execute(data[0], data[1]) } }
def multiplicationStrategies = [ { n, m -> n * m }, { n, m -> def total = 0; n.times{ total += m }; total }, { n, m -> ([m] * n).sum() } ] def sampleData = [ [3, 4, 12], [5, -5, -25] ] sampleData.each{ data -> multiplicationStrategies.each{ calc -> assert data[2] == calc(data[0], data[1]) } }
Language features instead of Patterns
(c) A
SE
RT
2006-2
013
Strategy Pattern
with interfaces
with closures
Adapter Pattern… class RoundPeg { def radius String toString() { "RoundPeg with radius $radius" } } class RoundHole { def radius def pegFits(peg) { peg.radius <= radius } String toString() { "RoundHole with radius $radius" } } def pretty(hole, peg) { if (hole.pegFits(peg)) println "$peg fits in $hole" else println "$peg does not fit in $hole" } def hole = new RoundHole(radius:4.0) (3..6).each { w -> pretty(hole, new RoundPeg(radius:w)) }
(c) A
SE
RT
2006-2
013
RoundPeg with radius 3 fits in RoundHole with radius 4.0
RoundPeg with radius 4 fits in RoundHole with radius 4.0
RoundPeg with radius 5 does not fit in RoundHole with radius 4.0
RoundPeg with radius 6 does not fit in RoundHole with radius 4.0
…Adapter Pattern… class SquarePeg { def width String toString() { "SquarePeg with width $width" } } class SquarePegAdapter { def peg def getRadius() { Math.sqrt(((peg.width/2) ** 2)*2) } String toString() { "SquarePegAdapter with width $peg.width (and notional radius $radius)" } } def hole = new RoundHole(radius:4.0) (4..7).each { w -> pretty(hole, new SquarePegAdapter(peg: new SquarePeg(width: w))) }
(c) A
SE
RT
2006-2
013
SquarePegAdapter with width 4 (and notional radius 2.8284271247461903)
fits in RoundHole with radius 4.0
SquarePegAdapter with width 5 (and notional radius 3.5355339059327378)
fits in RoundHole with radius 4.0
SquarePegAdapter with width 6 (and notional radius 4.242640687119285)
does not fit in RoundHole with radius 4.0
SquarePegAdapter with width 7 (and notional radius 4.949747468305833)
does not fit in RoundHole with radius 4.0
…Adapter Pattern
SquarePeg.metaClass.getRadius = { Math.sqrt(((delegate.width/2)**2)*2) } (4..7).each { w -> pretty(hole, new SquarePeg(width:w)) }
SquarePeg with width 4 fits in RoundHole with radius 4.0
SquarePeg with width 5 fits in RoundHole with radius 4.0
SquarePeg with width 6 does not fit in RoundHole with radius 4.0
SquarePeg with width 7 does not fit in RoundHole with radius 4.0
Adapter Pattern
Do I create a whole new class
or just add the method I need
on the fly?
Consider the Pros and Cons!
(c) A
SE
RT
2006-2
013
Further reading: James Lyndsay, Agile is Groovy, Testing is Square
Topics
• Intro
• Functional Style
• Design Patterns
Immutability
• Laziness
• GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Immutability options
• Built-in
• Google Collections – Numerous improved immutable collection types
• Groovy run-time metaprogramming
• Groovy compile-time metaprogramming
– @Immutable can help us create such classes
– Also gives us @Synchronized and @Lazy
import com.google.common.collect.* List<String> animals = ImmutableList.of("cat", "dog", "horse") animals << 'fish' // => java.lang.UnsupportedOperationException
def animals = ['cat', 'dog', 'horse'].asImmutable() animals << 'fish' // => java.lang.UnsupportedOperationException
def animals = ['cat', 'dog', 'horse'] ArrayList.metaClass.leftShift = { throw new UnsupportedOperationException() } animals << 'fish' // => java.lang.UnsupportedOperationException
@Immutable...
• Java Immutable Class – As per Joshua Bloch
Effective Java
© A
SE
RT
2006-2
013
public final class Person { private final String first; private final String last; public String getFirst() { return first; } public String getLast() { return last; } @Override public int hashCode() { final int prime = 31; int result = 1; result = prime * result + ((first == null) ? 0 : first.hashCode()); result = prime * result + ((last == null) ? 0 : last.hashCode()); return result; } public Person(String first, String last) { this.first = first; this.last = last; } // ...
// ... @Override public boolean equals(Object obj) { if (this == obj) return true; if (obj == null) return false; if (getClass() != obj.getClass()) return false; Person other = (Person) obj; if (first == null) { if (other.first != null) return false; } else if (!first.equals(other.first)) return false; if (last == null) { if (other.last != null) return false; } else if (!last.equals(other.last)) return false; return true; } @Override public String toString() { return "Person(first:" + first + ", last:" + last + ")"; } }
...@Immutable...
• Java Immutable Class – As per Joshua Bloch
Effective Java
© A
SE
RT
2006-2
013
public final class Person { private final String first; private final String last; public String getFirst() { return first; } public String getLast() { return last; } @Override public int hashCode() { final int prime = 31; int result = 1; result = prime * result + ((first == null) ? 0 : first.hashCode()); result = prime * result + ((last == null) ? 0 : last.hashCode()); return result; } public Person(String first, String last) { this.first = first; this.last = last; } // ...
// ... @Override public boolean equals(Object obj) { if (this == obj) return true; if (obj == null) return false; if (getClass() != obj.getClass()) return false; Person other = (Person) obj; if (first == null) { if (other.first != null) return false; } else if (!first.equals(other.first)) return false; if (last == null) { if (other.last != null) return false; } else if (!last.equals(other.last)) return false; return true; } @Override public String toString() { return "Person(first:" + first + ", last:" + last + ")"; } }
boilerplate
...@Immutable
© A
SE
RT
2006-2
013
@Immutable class Person { String first, last }
Approaches to managing collection storage
• Mutable • Persistent
© A
SE
RT
2006-2
013
• Immutable
‘c’ ‘a’ ‘c’ ‘a’ ‘c’ ‘a’
Add ‘t’ Add ‘t’ Add ‘t’
‘c’ ‘a’ ‘t’ ‘c’ ‘a’
‘c’ ‘a’ ‘t’
X ‘c’ ‘a’
‘t’
Persistent collections
• See also – Clojure (next)
– TotallyLazy (soon)
– Functional Java (tomorrow)
© A
SE
RT
2006-2
013
@Grab('org.pcollections:pcollections:2.1.2') import org.pcollections.* PSet<String> set = HashTreePSet.empty() set += "something" println set println set + "something else" assert set.size() == 1 // => [something] // => [something, something else]
Clojure Libraries
© A
SE
RT
2006-2
013
@Grab('org.clojure:clojure:1.0.0') import clojure.lang.* def ss = StringSeq.create('The quick brown fox') def done = false while (!done) { println ss.first() ss = ss.next() done = !ss }
'The quick brown fox'.each{ println it } <= Plain Groovy equivalent
Topics
• Intro
• Functional Style
• Design Patterns
• Immutability
Laziness
• GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Show me the code
TotallyLazyScript.groovy
GPars and TotallyLazy library
© A
SE
RT
2006-2
013
@GrabResolver('http://repo.bodar.com') @Grab('com.googlecode.totallylazy:totallylazy:808') import static groovyx.gpars.GParsExecutorsPool.withPool import static com.googlecode.totallylazy.Callables.asString import static com.googlecode.totallylazy.Sequences.sequence withPool { pool -> assert ['5', '6'] == sequence(4, 5, 6) .drop(1) .mapConcurrently(asString(), pool) .toList() }
withPool { assert ['5', '6'] == [4, 5, 6] .drop(1) .collectParallel{ it.toString() } }
<= Plain GPars equivalent
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Memoization
© A
SE
RT
2006-2
013
def plus = { a, b -> sleep 1000; a + b }.memoize() assert plus(1, 2) == 3 // after 1000ms assert plus(1, 2) == 3 // return immediately assert plus(2, 2) == 4 // after 1000ms assert plus(2, 2) == 4 // return immediately // other forms: // at least 10 invocations cached def plusAtLeast = { ... }.memoizeAtLeast(10) // at most 10 invocations cached def plusAtMost = { ... }.memoizeAtMost(10) // between 10 and 20 invocations cached def plusAtLeast = { ... }.memoizeBetween(10, 20)
Show me the code
Memoize.groovy, Factorial.groovy
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Show me the code
JScience, SPrintfChecker, GenericStackTest
Topics
Intro
• Functional Style
• Design Patterns
• Immutability
• Laziness
GPars
• Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
What makes up functional style?
• Functions, Closures, Lambda, Blocks as
first-class citizens
• Higher order functions
• Mutable vs Immutable data structures
• Recursion
• Lazy vs Eager evaluation
• Declarative vs Imperative style
• Advanced Techniques – Memoization, Trampolines, Composition and Curry
• Compile-time safety
• Concurrency
Ralph Johnson: Parallel Programming
• Styles of parallel programming – Threads and locks
• Nondeterministic, low-level, rumored humans can do this
– Asynchronous messages e.g. Actors –
no or limited shared memory • Nondeterministic, ok for I/O but be careful with side-effects
– Sharing with deterministic restrictions
e.g. Fork-join • Hopefully deterministic semantics, not designed for I/O
– Data parallelism • Deterministic semantics, easy, efficient, not designed for I/O
© A
SE
RT
2006-2
013
http://strangeloop2010.com/talk/presentation_file/14485/Johnson-DataParallelism.pdf
Each approach has some caveats
GPars • http://gpars.codehaus.org/
• Library classes and DSL sugar providing
intuitive ways for Groovy developers to
handle tasks concurrently. Logical parts:
– Data Parallelism features use JSR-166y Parallel Arrays
to enable multi-threaded collection processing
– Asynchronous functions extend the Java 1.5 built-in
support for executor services to enable multi-threaded
closure processing
– Dataflow Concurrency supports natural shared-memory
concurrency model, using single-assignment variables
– Actors provide an implementation of Erlang/Scala-like
actors including "remote" actors on other machines
– Safe Agents provide a non-blocking mt-safe reference to
mutable state; inspired by "agents" in Clojure
© A
SE
RT
2006-2
013
Coordination approaches S
ou
rce
: R
eG
inA
– G
roovy in
Actio
n, 2
nd e
ditio
n
Data Parallelism:
Fork/Join
Map/Reduce
Fixed coordination
(for collections)
Actors Explicit coordination
Safe Agents Delegated coordination
Dataflow Implicit coordination
GPars: Choosing approaches F
or
mo
re d
eta
ils s
ee: h
ttp://g
pars
.co
de
ha
us.o
rg/C
once
pts
+co
mp
are
d
Parallel
Collections
Data Parallelism
Task
Parallelism
Streamed Data
Parallelism
Fork/
Join
Dataflow
operators
CSP
Actors
Dataflow tasks
Actors
Asynch fun’s
CSP
Fork/
Join
Immutable
Stm, Agents
Special collections
Synchronization
Linear Recursive
Linear
Recursive
Shared
Data
Irregular Regular
Groovy Sequential Collection
© A
SE
RT
2006-2
013
def oneStarters = (1..30) .collect { it ** 2 } .findAll { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.max() == 196 assert oneStarters.sum() == 747
GPars Parallel Collections…
© A
SE
RT
2006-2
013
import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30) .collectParallel { it ** 2 } .findAllParallel { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.maxParallel() == 196 assert oneStarters.sumParallel() == 747 }
…GPars Parallel Collections
• Suitable when – Each iteration is independent, i.e. not:
fact[index] = index * fact[index - 1]
– Iteration logic doesn’t use non-thread safe code
– Size and indexing of iteration are important
© A
SE
RT
2006-2
013
import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30) .collectParallel { it ** 2 } .findAllParallel { it ==~ '1.*' } assert oneStarters == [1, 16, 100, 121, 144, 169, 196] assert oneStarters.maxParallel() == 196 assert oneStarters.sumParallel() == 747 }
Parallel Collection Variations
• Apply some Groovy metaprogramming
© A
SE
RT
2006-2
013
import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30).makeConcurrent() .collect { it ** 2 } .findAll { it ==~ '1.*' } .findAll { it ==~ '...' } assert oneStarters == [100, 121, 144, 169, 196] }
import groovyx.gpars.ParallelEnhancer def nums = 1..5 ParallelEnhancer.enhanceInstance(nums) assert [1, 4, 9, 16, 25] == nums.collectParallel{ it * it }
GPars parallel methods for collections Transparent Transitive? Parallel Lazy?
any { ... } anyParallel { ... } yes
collect { ... } yes collectParallel { ... }
count(filter) countParallel(filter)
each { ... } eachParallel { ... }
eachWithIndex { ... } eachWithIndexParallel { ... }
every { ... } everyParallel { ... } yes
find { ... } findParallel { ... }
findAll { ... } yes findAllParallel { ... }
findAny { ... } findAnyParallel { ... }
fold { ... } foldParallel { ... }
fold(seed) { ... } foldParallel(seed) { ... }
grep(filter) yes grepParallel(filter)
groupBy { ... } groupByParallel { ... }
max { ... } maxParallel { ... }
max() maxParallel()
min { ... } minParallel { ... }
min() minParallel()
split { ... } yes splitParallel { ... }
sum sumParallel // foldParallel +
Transitive means result is automatically transparent; Lazy means fails fast
For
mo
re d
eta
ils s
ee R
eG
inA
or
the G
Pa
rs d
ocu
me
nta
tion
GPars: Map-Reduce
© A
SE
RT
2006-2
013
import static groovyx.gpars.GParsPool.withPool withPool { def oneStarters = (1..30).parallel .map { it ** 2 } .filter { it ==~ '1.*' } assert oneStarters.collection == [1, 16, 100, 121, 144, 169, 196] // aggregations/reductions assert oneStarters.max() == 196 assert oneStarters.reduce { a, b -> a + b } == 747 assert oneStarters.sum() == 747 }
GPars parallel array methods
Method Return Type
combine(initValue) { ... } Map
filter { ... } Parallel array
collection Collection
groupBy { ... } Map
map { ... } Parallel array
max() T
max { ... } T
min() T
min { ... } T
reduce { ... } T
reduce(seed) { ... } T
size() int
sort { ... } Parallel array
sum() T
parallel // on a Collection Parallel array
For
mo
re d
eta
ils s
ee R
eG
inA
or
the G
Pa
rs d
ocu
me
nta
tion
Parallel Collections vs Map-Reduce
Fork Fork
Join Join
Map
Map
Reduce
Map
Map
Reduce
Reduce
Map
Filter
Filter Map
Concurrency challenge…
• Suppose we have the following
calculation involving several functions:
• And we want to use our available cores …
© A
SE
RT
2006-2
013
// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] def a = 5 def b = f1(a) def c = f2(a) def d = f3(c) def f = f4(b, d) assert f == 10
…Concurrency challenge…
• We can analyse the example’s task graph:
© A
SE
RT
2006-2
013
// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] def a = 5 def b = f1(a) def c = f2(a) def d = f3(c) def f = f4(b, d) assert f == 10
f2
f3
f1
f4
a a
b
c
d
f
…Concurrency challenge…
• Manually using asynchronous functions:
© A
SE
RT
2006-2
013
// example adapted from Parallel Programming with .Net def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import static groovyx.gpars.GParsPool.withPool withPool(2) { def a = 5 def futureB = f1.callAsync(a) def c = f2(a) def d = f3(c) def f = f4(futureB.get(), d) assert f == 10 }
f2
f3
f1
f4
a a
b
c
d
f
…Concurrency challenge
• And with GPars Dataflows:
© A
SE
RT
2006-2
013
def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task new Dataflows().with { task { a = 5 } task { b = f1(a) } task { c = f2(a) } task { d = f3(c) } task { f = f4(b, d) } assert f == 10 }
f2
f3
f1
f4
a a
b
c
d
f
…Concurrency challenge
• And with GPars Dataflows:
© A
SE
RT
2006-2
013
def (f1, f2, f3, f4) = [{ sleep 1000; it }] * 3 + [{ x, y -> x + y }] import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task new Dataflows().with { task { f = f4(b, d) } task { d = f3(c) } task { c = f2(a) } task { b = f1(a) } task { a = 5 } assert f == 10 }
f2
f3
f1
f4
a a
b
c
d
f
GPars: Dataflows...
© A
SE
RT
2006-2
013
import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.result = flow.x + flow.y } task { flow.x = 10 } task { flow.y = 5 } assert 15 == flow.result
new DataFlows().with { task { result = x * y } task { x = 10 } task { y = 5 } assert 50 == result }
5 10
y x
*
...GPars: Dataflows...
• Evaluating:
© A
SE
RT
2006-2
013
import groovyx.gpars.dataflow.DataFlows import static groovyx.gpars.dataflow.DataFlow.task final flow = new DataFlows() task { flow.a = 10 } task { flow.b = 5 } task { flow.x = flow.a - flow.b } task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } assert flow.result == 75
b
10 5
a
+ -
*
result = (a – b) * (a + b)
x y
Question: what happens if I change the order of the task statements here?
...GPars: Dataflows...
• Naive attempt for loops
© A
SE
RT
2006-2
013
import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task final flow = new Dataflows() [10, 20].each { thisA -> [4, 5].each { thisB -> task { flow.a = thisA } task { flow.b = thisB } task { flow.x = flow.a - flow.b } task { flow.y = flow.a + flow.b } task { flow.result = flow.x * flow.y } println flow.result } } // => java.lang.IllegalStateException: A DataflowVariable can only be assigned once.
... task { flow.a = 10 } ... task { flow.a = 20 }
Don’t do this!
X
...GPars: Dataflows...
© A
SE
RT
2006-2
013
import groovyx.gpars.dataflow.DataflowStream import static groovyx.gpars.dataflow.Dataflow.* final streamA = new DataflowStream() final streamB = new DataflowStream() final streamX = new DataflowStream() final streamY = new DataflowStream() final results = new DataflowStream() operator(inputs: [streamA, streamB], outputs: [streamX, streamY]) { a, b -> streamX << a - b; streamY << a + b } operator(inputs: [streamX, streamY], outputs: [results]) { x, y -> results << x * y } [[10, 20], [4, 5]].combinations().each{ thisA, thisB -> task { streamA << thisA } task { streamB << thisB } } 4.times { println results.val }
b
10
10
20
20
4
5
4
5
a
+ -
*
84
75
384
375
...GPars: Dataflows
• Suitable when: – Your algorithms can be expressed as mutually-
independent logical tasks
• Properties: – Inherently safe and robust (no race conditions or
livelocks)
– Amenable to static analysis
– Deadlocks “typically” become repeatable
– “Beautiful” (declarative) code
© A
SE
RT
2006-2
013
import groovyx.gpars.dataflow.Dataflows import static groovyx.gpars.dataflow.Dataflow.task final flow = new Dataflows() task { flow.x = flow.y } task { flow.y = flow.x }
…GPars: Actors...
© A
SE
RT
2006-2
013
import static groovyx.gpars.actor.Actors.* def votes = reactor { it.endsWith('y') ? "You voted for $it" : "Sorry, please try again" } println votes.sendAndWait('Groovy') println votes.sendAndWait('JRuby') println votes.sendAndWait('Go') def languages = ['Groovy', 'Dart', 'C++'] def booth = actor { languages.each{ votes << it } loop { languages.size().times { react { println it } } stop() } } booth.join(); votes.stop(); votes.join()
You voted for Groovy
You voted for JRuby
Sorry, please try again
You voted for Groovy
Sorry, please try again
Sorry, please try again
Software Transactional Memory…
© A
SE
RT
2006-2
013
@Grab('org.multiverse:multiverse-beta:0.7-RC-1') import org.multiverse.api.references.LongRef import static groovyx.gpars.stm.GParsStm.atomic import static org.multiverse.api.StmUtils.newLongRef class Account { private final LongRef balance Account(long initial) { balance = newLongRef(initial) } void setBalance(long newBalance) { if (newBalance < 0) throw new RuntimeException("not enough money") balance.set newBalance } long getBalance() { balance.get() } } // ...
…Software Transactional Memory
© A
SE
RT
2006-2
013
// ... def from = new Account(20) def to = new Account(20) def amount = 10 def watcher = Thread.start { 15.times { atomic { println "from: ${from.balance}, to: ${to.balance}" } sleep 100 } } sleep 150 try { atomic { from.balance -= amount to.balance += amount sleep 500 } println 'transfer success' } catch(all) { println all.message } atomic { println "from: $from.balance, to: $to.balance" } watcher.join()
Topics
• Intro
• Functional Style
• Design Patterns
• Immutability
• Laziness
• GPars
Word Split (bonus material)
• More Info
© A
SE
RT
2006-2
013
Word Split with Fortress
© A
SE
RT
2006-2
013
Guy Steele’s StrangeLoop keynote (from slide 52 onwards for several slides):
http://strangeloop2010.com/talk/presentation_file/14299/GuySteele-parallel.pdf
Word Split…
© A
SE
RT
2006-2
013
def swords = { s -> def result = [] def word = '' s.each{ ch -> if (ch == ' ') { if (word) result += word word = '' } else word += ch } if (word) result += word result }
assert swords("This is a sample") == ['This', 'is', 'a', 'sample'] assert swords("Here is a sesquipedalian string of words") == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words']
Word Split…
© A
SE
RT
2006-2
013
def swords = { s -> def result = [] def word = '' s.each{ ch -> if (ch == ' ') { if (word) result += word word = '' } else word += ch } if (word) result += word result }
Word Split…
© A
SE
RT
2006-2
013
def swords = { s -> def result = [] def word = '' s.each{ ch -> if (ch == ' ') { if (word) result += word word = '' } else word += ch } if (word) result += word result }
…Word Split…
© A
SE
RT
2006-2
013
…Word Split…
© A
SE
RT
2006-2
013
Segment(left1, m1, right1) Segment(left2, m2, right2)
Segment(left1, m1 + [ ? ] + m2, right2)
…Word Split…
© A
SE
RT
2006-2
013
…Word Split…
© A
SE
RT
2006-2
013
class Util { static maybeWord(s) { s ? [s] : [] } } import static Util.* @Immutable class Chunk { String s public static final ZERO = new Chunk('') def plus(Chunk other) { new Chunk(s + other.s) } def plus(Segment other) { new Segment(s + other.l, other.m, other.r) } def flatten() { maybeWord(s) } } @Immutable class Segment { String l; List m; String r public static final ZERO = new Segment('', [], '') def plus(Chunk other) { new Segment(l, m, r + other.s) } def plus(Segment other) { new Segment(l, m + maybeWord(r + other.l) + other.m, other.r) } def flatten() { maybeWord(l) + m + maybeWord(r) } }
…Word Split…
© A
SE
RT
2006-2
013
def processChar(ch) { ch == ' ' ? new Segment('', [], '') : new Chunk(ch) }
def swords(s) { s.inject(Chunk.ZERO) { result, ch -> result + processChar(ch) } }
assert swords("Here is a sesquipedalian string of words").flatten() == ['Here', 'is', 'a', 'sesquipedalian', 'string', 'of', 'words']
…Word Split…
© A
SE
RT
2006-2
013
…Word Split…
© A
SE
RT
2006-2
013
…Word Split…
© A
SE
RT
2006-2
013
THREADS = 4 def pwords(s) { int n = (s.size() + THREADS - 1) / THREADS def map = new ConcurrentHashMap() (0..<THREADS).collect { i -> Thread.start { def (min, max) = [ [s.size(), i * n].min(), [s.size(), (i + 1) * n].min() ] map[i] = swords(s[min..<max]) } }*.join() (0..<THREADS).collect { i -> map[i] }.sum().flatten() }
…Word Split…
© A
SE
RT
2006-2
013
import static groovyx.gpars.GParsPool.withPool THRESHHOLD = 10 def partition(piece) { piece.size() <= THRESHHOLD ? piece : [piece[0..<THRESHHOLD]] + partition(piece.substring(THRESHHOLD)) } def pwords = { input -> withPool(THREADS) { partition(input).parallel.map(swords).reduce{ a, b -> a + b }.flatten() } }
…Guy Steele example in Groovy…
© A
SE
RT
2006-2
013
def words = { s -> int n = (s.size() + THREADS - 1) / THREADS def min = (0..<THREADS).collectEntries{ [it, [s.size(),it*n].min()] } def max = (0..<THREADS).collectEntries{ [it, [s.size(),(it+1)*n].min()] } def result = new DataFlows().with { task { a = swords(s[min[0]..<max[0]]) } task { b = swords(s[min[1]..<max[1]]) } task { c = swords(s[min[2]..<max[2]]) } task { d = swords(s[min[3]..<max[3]]) } task { sum1 = a + b } task { sum2 = c + d } task { sum = sum1 + sum2 } println 'Tasks ahoy!' sum } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } }
DataFlow version: partially hard-coded to 4 partitions for easier reading
…Guy Steele example in Groovy…
© A
SE
RT
2006-2
013
GRANULARITY_THRESHHOLD = 10 THREADS = 4 println GParsPool.withPool(THREADS) { def result = runForkJoin(0, input.size(), input){ first, last, s -> def size = last - first if (size <= GRANULARITY_THRESHHOLD) { swords(s[first..<last]) } else { // divide and conquer def mid = first + ((last - first) >> 1) forkOffChild(first, mid, s) forkOffChild(mid, last, s) childrenResults.sum() } } switch(result) { case Chunk: return maybeWord(result.s) case Segment: return result.with{ maybeWord(l) + m + maybeWord(r) } } }
Fork/Join version
…Guy Steele example in Groovy
© A
SE
RT
2006-2
013
println GParsPool.withPool(THREADS) { def ans = input.collectParallel{ processChar(it) }.sum() switch(ans) { case Chunk: return maybeWord(ans.s) case Segment: return ans.with{ maybeWord(l) + m + maybeWord(r) } } }
Just leveraging the algorithm’s parallel nature
Topics
• Intro
• Functional Style
• Design Patterns
• Immutability
• Laziness
• GPars
• Word Split (bonus material)
More Info
© A
SE
RT
2006-2
013
More Information: Groovy in Action