Scala Next

SF Scala meetup Dec 8th, 2011

2

Scala Today

3

Some adoption vectors:

•  Web platforms

•  Trading platforms

•  Financial modeling

•  Simulation

Fast to first product, scalable afterwards

Github vs. Stack Overflow

RedMonk: “Revisiting the Dataists Programming Language Rankings”

4 Typesafe Confidential

Commercial Adoption

•  Scala jobs tripled in last year

•  Now at estimated 100,000 developers

5 Typesafe Confidential

6

Scala 2.8:

(Only 17 months ago!)

New collections

Package objects

Context bounds

Better implicits

...

7

Scala 2.9:

Parallel collections

DelayedInit and App

Faster REPL

Progress on IDEs: Eclipse, IntelliJ, Neatbeans, ENSIME

Better docs

Lots of bug fixes

 

Parallel Collections •  Use Java 7 Fork Join framework •  Split work by number of Processors •  Each Thread has a work queue that is split

exponentially. Largest on end of queue •  Granularity balance against scheduling overhead •  On completion threads “work steals” from end of other

thread queues

8

9

... and its usage import  java.util.ArrayList;  ...  Person[]  people;  Person[]  minors;  Person[]  adults;  {    ArrayList<Person>  minorsList  =  new  ArrayList<Person>();        ArrayList<Person>  adultsList  =  new  ArrayList<Person>();        for  (int  i  =  0;  i  <  people.length;  i++)                (people[i].age  <  18  ?  minorsList  :  adultsList)  

         .add(people[i]);        minors  =  minorsList.toArray(people);        adults  =  adultsList.toArray(people);  }  

   

... in Java:

... in Scala: val  people:  Array[Person]  val  (minors,  adults)  =  people  partition  (_.age  <  18)  

A simple pattern match

An infix method call

A function value

10

Going Parallel

 ?  ... in Java:

... in Scala: val  people:  Array[Person]  val  (minors,  adults)  =  people.par  partition  (_.age  <  18)  

General Collection Hierarchy

GenTraversable

GenIterable

GenSeq

Traversable

Iterable

Seq

ParIterable

ParSeq

11

Remove this layer in 2.10?

Going Distributed

•  Can we get the power of parallel collections to work on 10’000s of computers?

•  Hot technologies: MapReduce (Google’s and Hadoop) •  But not everything is easy to fit into that mold •  Sometimes 100’s of map-reduce steps are needed. •  Distributed collections retain most operations, provide a

powerful frontend for MapReduce computations. •  Scala’s uniform collection model is designed to also

accommodate parallel and distributed. •  Projects at Google (Cascade), Berkeley (Spark), EPFL.

12

13

Scala next:

Eclipse IDE Play web framework 2.0 Akka 2.0 Scala 2.10

14

Scala Eclipse IDE

Now in RC2 Final expected before the end of

the year.

Goals

reliable (no crashes/lock ups) responsive (never wait when typing) work with large projects/files

–  Scala compiler (80k LOC), 4-5000 LOC/file –  advanced use of the type system:

path-dependent types, self-types, mix-ins

Features

Keep it simple –  highlight errors as you type –  completions (including implicits) –  hyperlinking –  project builder (+ dependent projects)

Support mixed Java-Scala projects –  all features should work between Java/Scala sources

JUnit Test Runner should pick up tests More stuff based on external libraries

–  (some) refactoring, code formatter, mark occurrences, structured selections, show inferred semi-colons

Features (3)

based on external libraries –  (some) refactoring –  code formatter –  mark occurrences –  structured selections –  show inferred semi-colons

@jonifreeman Joni Freeman Latest Scala Eclipse plugin works surprisingly well! Even manages our mixed Java/Scala project. Kudos to the team! #scala

@esorribas Eduardo Sorribas The latest beta of the Scala IDE for eclipse is much better. I'm starting to like it.

@jannehietamaki Janne Hietamäki After years of misery, the Eclipse Scala plugin actually seems to work quite well.

Architecture

Use the full-blown Scala compiler for: –  interactive error highlight, completion, hyperlinking –  turning Scala symbols into Java model elements

Weave the JDT compiler when it needs help –  JDT was NOT meant to be extended

Why rely on scalac?

–  reuse (type-checker == 1-2 person years)

–  consistency

–  compiler plugins

Why not?

–  SPEED

–  (very) tight dependency on the Scala version

Presentation Compiler

asynchronous interruptible targeted stop after type-checking

Result is communicated through a SyncVar

•  All compiler activity happens on PC thread •  compile loaded files when work queue is empty (in the

background) •  Check work queue when type checker reaches “safe-points” in

the AST •  Drop everything when a file is changed (AskReload)

26

Implementation

1 type-checker run / instance --> 100s of type-check runs / minute –  memory leaks –  side-effects/state –  out-of-order and targeted type-checking

needed to improve the compiler –  2.9.x, 2.10 (trunk) –  what about 2.8?

2.8.2, 2.8.3-SNAPSHOT

New: Play Framework 2.0 •  Play Framework is an open source web application

framework, inspired by Ruby on Rails, for Java and Scala •  Play Framework 2.0 retains full Java support while moving

to a Scala core and builds on key pieces of the Typesafe Stack, including Akka middleware and SBT

•  Play will be integrated in TypeSafe stack 2.0 •  Typesafe will contribute to development and provide

commercial support and maintenance.

Roadmap

Typesafe Stack 1.0

Typesafe Stack 1.1

Typesafe Stack 2.0

Typesafe Stack 2.x

May 2011 Oct 2011 Q1 2012 Q3 2012

Scala 2.9.0 Akka 1.1

Scala 2.9.1 Akka 1.2

Scala 2.9.x Akka 2.0 Play 2.0

Scala 2.10 Akka 2.x Play 2.x

Slick (DB)

30

Scala 2.10:

1.  New reflection framework 2.  Reification 3.  type Dynamic  4.  More IDE improvements: find-

references, debugger, worksheet.

5.  Faster builds 6.  SIPs: string interpolation,

simpler implicits.

ETA: Early 2012.

New in Scala 2.10: Dynamic

Type Dynamic bridges the gap between static and dynamic typing. Method calls get translated to applyDynamic Great for interfacing with dynamic languages (e.g. JavaScript)

31

class JS extends Dynamic { def applyDynamic(methName: String, args: Any*): Any = { println("apply dynamic "+methName+args.mkString("(", ",", ")")) } } val x = new JS x.foo(1) // à x.applyDynamic(“foo”, 1) x.bar // à x.applyDynamic(“bar”)  

Proposed for Scala 2.10: SIP 11: String interpolation

Idea: Instead of

 “Bob  is  ”  +  n  +  “years  old”  

write:

 s“Bob  is  $n  years  old”  

which gets translated to

 new  StringContext(“Bob  is”,  “years  old”).s(n)  

Here, s is a library-defined method for string interpolation.  

 

       

 

32

This can be generalized to other string processors besides s: xml”””        <body>      <a  href  =  “some  link”>  ${linktext}  </a>        </body>    ”””  

   

 scala”””        scala.concurrent.transaction.withinTransaction  {            (implicit  currentTransaction:  Transaction)  =>      $expr        }    ”””  

33

Proposed for Scala 2.10: SIP 12: Uncluttering control

Should be able to write:

 

 if  x  <  0  then  –x  else  x  

 

 while  x  >  0  do  {  println(x);  x  -­‐=  1  }  

 

 for  x  <-­‐  xs  do  println(x)  

 

 for  x  <-­‐  xs  yield  x  *  x  

       

 

34

Proposed for Scala 2.10: SIP 13: Implicit classes

Variation: Add @inline  to class def to get speed of extension methods.

35

New in Scala 2.10: Reflection

Previously: Needed to use Java reflection, no runtime info available on Scala’s types. Now you can do:

36

(Bare-Bones) Reflection in Java

37

Want to know whether type A conforms to B?

Write your own Java compiler!

Why not add some meaningful operations?

Need to write essential parts of a compiler (hard).

Need to ensure that both compilers agree (almost impossible).

 

How to do Better?

•  Problem is managing dependencies between compiler and reflection.

•  Time to look at DI again.

38

Dependency Injection

•  Idea: Avoid hard dependencies to specific classes. •  Instead of calling specific classes with new, have someone else do

the wiring.

Using Guice for Dependency Injection

39

(Example by Jan Kriesten)

... plus some Boilerplate

40

Dependency Injection in Scala

41

Components are classes or traits

Requirements are abstract values

Wiring by implementing requirement values

But what about cyclic dependencies?

The Cake Pattern

42

Requirements are types of this  

Components are traits

Wiring by mixin composition

Cake Pattern in the Compiler

The Scala compiler uses the cake pattern for everything Here’s a schema: (In reality there are about ~20 slices in the cake.)

43

Towards Better Reflection

Can we unify the core parts of the compiler and reflection? Compiler Reflection Different requirements: Error diagnostics, file access, classpath

handling - but we are close! 44

Compiler Architecture

45

reflect.internal.Universe  

nsc.Global  (scalac) reflect.runtime.Mirror  

Problem: This exposes way too much detail!

Complete Reflection Architecture

Cleaned-up facade:

Full implementation:

46

reflect.internal.Universe  

nsc.Global  (scalac) reflect.runtime.Mirror  

reflect.api.Universe  /  reflect.mirror  

How to Make a Facade

47

The Facade

The Implementation

Interfaces are not enough!

Conclusion

Scala is a very regular language when it comes to composition:

1.  Everything can be nested: –  classes, methods, objects, types

2.  Everything can be abstract: –  methods, values, types

3.  The type of this can be declared freely, can thus express dependencies

4.  This gives great flexibility for SW architecture, allows us to attack previously unsolvable problems.

48

Going further: Parallel DSLs

Mid term, research project: How do we keep tomorrow’s computers loaded? –  How to find and deal with 10000+ threads in an

application? –  Parallel collections and actors are necessary but not

sufficient for this. Our bet for the mid term future: parallel embedded DSLs.

–  Find parallelism in domains: physics simulation, machine learning, statistics, ...

Joint work with Kunle Olukuton, Pat Hanrahan @ Stanford. EPFL side funded by ERC.

49

EPFL / Stanford Research

Domain Embedding Language (Scala)

Virtual Worlds

Personal Robotics

Data informatics

Scientific Engineering

Physics (Liszt) Scripting Probabilistic

(RandomT) Machine Learning (OptiML)

Rendering

Parallel Runtime (Delite, Sequoia, GRAMPS)

Dynamic Domain Spec. Opt. Locality Aware Scheduling

Staging Polymorphic Embedding

Applications

Domain Specific

Languages

Heterogeneous Hardware

DSL Infrastructure

Task & Data Parallelism

Hardware Architecture

OOO Cores SIMD Cores Threaded Cores Specialized Cores

Static Domain Specific Opt.

Programmable Hierarchies

Scalable Coherence

Isolation & Atomicity

On-chip Networks

Pervasive Monitoring

50

Fuel injection Transition Thermal

Turbulence

Turbulence

Combustion

Example: Liszt - A DSL for Physics Simulation

•  Mesh-based •  Numeric Simulation •  Huge domains

–  millions of cells •  Example: Unstructured Reynolds-averaged Navier

Stokes (RANS) solver

51

Liszt as Virtualized Scala val // calculating scalar convection (Liszt) val Flux = new Field[Cell,Float] val Phi = new Field[Cell,Float] val cell_volume = new Field[Cell,Float] val deltat = .001 ... untilconverged { for(f <- interior_faces) { val flux = calc_flux(f) Flux(inside(f)) -= flux Flux(outside(f)) += flux } for(f <- inlet_faces) { Flux(outside(f)) += calc_boundary_flux(f) } for(c <- cells(mesh)) { Phi(c) += deltat * Flux(c) /cell_volume

(c) } for(f <- faces(mesh)) Flux(f) = 0.f }

AST

Hardware

DSL Library

Optimisers Generators …

… Schedulers

GPU, Multi-Core, etc

52

53

Follow us on twitter: @typesafe scala-lang.org

typesafe.com

akka.io

scala-lang.org