What’s in Store for Scala?

Martin Odersky

DEVOXX 2011

What’s in Store for Scala?

Martin Odersky

Typesafe and EPFL

3

Scala Today

4

Some adoption vectors:

• Web platforms

• Trading platforms

• Financial modeling

• Simulation

Fast to first product, scalable afterwards

5

Scala 2.9:

1. Parallel and concurrent computing libraries

2. DelayedInit and App

3. Faster REPL

4. Progress on IDEs: Eclipse, IntelliJ, Neatbeans, ENSIME

5. Better docs

Play Framework 2.0

• Play Framework is an open source web application framework, heavily inspired by Ruby on Rails, for Java and Scala

• Play Framework 2.0 retains full Java support while moving to a Scala core and builds on key pieces of the Typesafe Stack, including Akka middleware and SBT

• Play will be integrated in TypeSafe stack 2.0

• Typesafe will contribute to development and provide commercial support and maintenance.

Scala Eclipse IDE 2.0

• The Scala Eclipse IDE has been completely reworked.

• First Release Candidate for IDE 2.0 is available now.

• Goals:

reliable (no crashes/lock ups)

responsive (never wait when typing)

work with large projects/files

8

Scala 2.10:

1. New reflection framework

2. Reification

3. type Dynamic4. More IDE improvements: find-

references, debugger, worksheet.

5. Faster builds

6. SIPs: string interpolation, simpler implicits.

ETA: Early 2012.

“Scala” comes from “Scalable”

Scalability: Powerful concepts that hold up from small to large

At JavaOne - Scala in the small:

Winner of the Script Bowl

(Thank you, Dick!)

This talk - Scala in the large:

Scale to many cores

Scale to large systems

9

Scaling to Many Cores

The world of mainstream software is changing:

– Moore’s law now achieved by increasing # of cores not clock cycles

– Huge volume workloads that require horizontalscaling

– “PPP” Grand Challenge

10

Data from Kunle Olukotun, Lance Hammond, Herb Sutter, Burton Smith, Chris Batten, and Krste

Asanovic

“The free lunch is over”

Concurrency and Parallelism

Parallel programming Execute programs faster on parallel hardware.

Concurrent programming Manage concurrent execution threads explicitly.

Both are too hard!

11

The Root of The Problem

• Non-determinism caused by concurrent threads accessing shared mutable state.

• It helps to encapsulate state in actors or transactions, but the fundamental problem stays the same.

• So,

non-determinism = parallel processing + mutable state

• To get deterministic processing, avoid the mutable state!• Avoiding mutable state means programming functionally.

12

var x = 0

async { x = x + 1 }

async { x = x * 2 }

// can give 0, 1, 2

Space vs Time

13

Time (imperative/concurrent)

Space (functional/parallel)

14

Scala is a Unifier

Agile, with lightweight syntax

Object-Oriented Scala Functional

Safe and performant, with strong static tpying

15

Scala is a Unifier

Agile, with lightweight syntax

Parallel

Object-Oriented Scala Functional

Sequential

Safe and performant, with strong static tpying

Scala’s Toolbox

16

17

Different Tools for Different Purposes

Parallelism:

Parallel CollectionsCollections

Distributed Collections

Parallel DSLs

Concurrency:

Actors Software transactional memory AkkaFutures

18

Let’s see an example:

19

A class ...public class Person {

public final String name;

public final int age;

Person(String name, int age) {

this.name = name;

this.age = age;

}

}

class Person(val name: String, val age: Int)

... in Java:

... in Scala:

20

... and its usageimport java.util.ArrayList;

...

Person[] people;

Person[] minors;

Person[] adults;

{ ArrayList<Person> minorsList = new ArrayList<Person>();

ArrayList<Person> adultsList = new ArrayList<Person>();

for (int i = 0; i < people.length; i++)

(people[i].age < 18 ? minorsList : adultsList)

.add(people[i]);

minors = minorsList.toArray(people);

adults = adultsList.toArray(people);

}

... in Java:

... in Scala: val people: Array[Person]val (minors, adults) = people partition (_.age < 18)

A simple pattern match

An infix method call

A function value

21

Going Parallel

?(for now)... in Java:

... in Scala: val people: Array[Person]val (minors, adults) = people.par partition (_.age < 18)

Parallel Collections• Use Java 7 Fork Join framework• Split work by number of Processors• Each Thread has a work queue that is split

exponentially. Largest on end of queue• Granularity balance against scheduling overhead• On completion threads “work steals” from end of other

thread queues

22

General Collection Hierarchy

GenTraversable

GenIterable

GenSeq

Traversable

Iterable

Seq

ParIterable

ParSeq

23

Going Distributed

• Can we get the power of parallel collections to work on 10’000s of computers?

• Hot technologies: MapReduce (Google’s and Hadoop)• But not everything is easy to fit into that mold• Sometimes 100’s of map-reduce steps are needed.• Distributed collections retain most operations, provide a

powerful frontend for MapReduce computations.• Scala’s uniform collection model is designed to also

accommodate parallel and distributed.• Projects at Google (Cascade), Berkeley (Spark), EPFL.

24

25

The Future

Scala’s persistent collections are

• easy to use:

• concise:

• safe:

• fast:

• scalable:

We see them play a rapidly increasing role in software development.

few steps to do the job

one word replaces a whole loop

type checker is really good at catching errors

collection ops are tuned, can be parallelized

one vocabulary to work on all kinds of collections: sequential, parallel, or distributed.

Going further: Parallel DSLs

But how do we keep a tomorrow’s hardware loaded?– How to find and deal with 10000+ threads in an

application?

– Parallel collections and actors are necessary but not sufficient for this.

Our bet for the mid term future: parallel embedded DSLs.– Find parallelism in domains: physics simulation, machine

learning, statistics, ...

Joint work with Kunle Olukuton, Pat Hanrahan @ Stanford.

EPFL side funded by ERC.

26

EPFL / Stanford ResearchEPFL / Stanford Research

Domain Embedding Language (Scala)

Virtual Worlds

Personal Robotics

Datainformatics

ScientificEngineering

Physics(Liszt)

ScriptingProbabilistic(RandomT)

Machine Learning(OptiML)

Rendering

Parallel Runtime (Delite, Sequoia, GRAMPS)

Dynamic Domain Spec. Opt. Locality Aware Scheduling

StagingPolymorphic Embedding

Applications

DomainSpecific

Languages

HeterogeneousHardware

DSLInfrastructure

Task & Data Parallelism

Hardware Architecture

OOO CoresOOO Cores SIMD CoresSIMD Cores Threaded CoresThreaded Cores Specialized CoresSpecialized Cores

Static Domain Specific Opt.

ProgrammableHierarchies

ProgrammableHierarchies

Scalable CoherenceScalable

CoherenceIsolation & Atomicity

Isolation & Atomicity

On-chipNetworksOn-chip

NetworksPervasive MonitoringPervasive Monitoring

27

Fuel injection

Transition Thermal

Turbulence

Turbulence

Combustion

Example: Liszt - A DSL for Physics Simulation

• Mesh-based• Numeric Simulation• Huge domains

– millions of cells

• Example: Unstructured Reynolds-averaged Navier Stokes (RANS) solver

28

Liszt as Virtualized Scala

val // calculating scalar convection (Liszt)

val Flux = new Field[Cell,Float]val Phi = new Field[Cell,Float]val cell_volume = new Field[Cell,Float]val deltat = .001...untilconverged { for(f <- interior_faces) { val flux = calc_flux(f) Flux(inside(f)) -= flux Flux(outside(f)) += flux } for(f <- inlet_faces) { Flux(outside(f)) += calc_boundary_flux(f) } for(c <- cells(mesh)) { Phi(c) += deltat * Flux(c)

/cell_volume(c) } for(f <- faces(mesh)) Flux(f) = 0.f}

AST

Hardware

DSL Library

Optimisers Generators

…

…

Schedulers

GPU, Multi-Core, etc

29

New in Scala 2.10: Reflection

Previously: Needed to use Java reflection,

no runtime info available on Scala’s types.

Now you can do:

30

(Bare-Bones) Reflection in Java

31

Want to know whether type A conforms to B?

Write your own Java compiler!

Why not add some meaningful operations?

Need to write essential parts of a compiler (hard).

Need to ensure that both compilers agree (almost impossible).

How to do Better?

• Problem is managing dependencies between compiler and reflection.

• Time to look at DI again.

32

Dependency Injection

• Idea: Avoid hard dependencies to specific classes.• Instead of calling specific classes with new, have someone else do

the wiring.

Using Guice for Dependency Injection

33

(Example by Jan Kriesten)

... plus some Boilerplate

34

Dependency Injection in Scala

35

Components are classes or traits

Requirements are abstract values

Wiring by implementing

requirement valuesBut what about cyclic dependencies?

The Cake Pattern

36

Requirements are types of this

Components are traits

Wiring by mixin composition

Cake Pattern in the Compiler

The Scala compiler uses the cake pattern for everything

Here’s a schema:

(In reality there are about ~20 slices in the cake.)

37

Towards Better Reflection

Can we unify the core parts of the compiler and reflection?

Compiler Reflection

Different requirements: Error diagnostics, file access, classpath handling - but we are close!

38

Compiler Architecture

39

reflect.internal.Universe

nsc.Global (scalac) reflect.runtime.Mirror

Problem: This exposes way too much detail!

Complete Reflection Architecture

Cleaned-up facade:

Full implementation:

40

reflect.internal.Universe

nsc.Global (scalac) reflect.runtime.Mirror

reflect.api.Universe /reflect.mirror

How to Make a Facade

41

The Facade

The Implementation

Interfaces are not enough!

Conclusion

Scala is a very regular language when it comes to composition:

1. Everything can be nested:– classes, methods, objects, types

2. Everything can be abstract:– methods, values, types

3. The type of this can be declared freely, can thus express dependencies

4. This gives great flexibility for SW architecture, allows us to attack previously unsolvable problems.

42

43

Follow us on twitter: @typesafe

scala-lang.org

typesafe.com

akka.io

scala-lang.org