FunctionalConf '16 Robert Virding Erlang Ecosystem

The Erlang Ecosystem

Robert Virding

www.erlang-solutions.com

Background: the problem

‣Ericsson’s “best seller” AXE telephone exchanges (switches) required large effort to develop and maintain software.

‣The problem to solve was how to make programming these types of applications easier, but keeping the same characteristics.


3

‣Handle a very large numbers of concurrent activities.

‣Actions must be performed at a certain point in time or within a certain time.

‣System distributed over several computers.

‣Interaction with hardware.

‣Very large software systems.

‣Complex functionality such as feature interaction.

‣Continuous operation over many years.

‣Software maintenance (reconfiguration etc.) without stopping the system.

‣Stringent quality and reliability requirements.

‣Fault tolerance both to hardware failures and software errors.Bjarne Däcker, November 2000 - Licentiate Thesis

Background: problem domain

‣Handle a very large numbers of concurrent activities.

‣Actions must be performed at a certain point in time or within a certain time.

‣System distributed over several computers.

‣Interaction with hardware.

‣Very large software systems.

‣Complex functionality such as feature interaction.

‣Continuous operation over many years.

‣Software maintenance (reconfiguration etc.) without stopping the system.

‣Stringent quality and reliability requirements.

‣Fault tolerance both to hardware failures and software errors.Bjarne Däcker, November 2000 - Licentiate Thesis

Not just telecom


4

Background: some reflections

WE WERE TRYING TO SOLVE THEPROBLEM

We were not out to implement the actor model

We were not out to implement a functional language


5

Background: some reflections

The language/systemevolved to solve the problem

‣We had a clear set of criteria for what should go into the language/system▹Was it useful?▹ Did it or did it not help build the system?

‣This made the development of the language/system very focused


6

Erlang and the system around it was designed to solve this type of problem

Erlang/OTP provides direct support for these issues

Background


7

▸ Lightweight concurrency▹ Must handle a large number of processes▹ Process creation, context switching and inter-process communication must be

cheap and fast.

▸Asynchronous communication▸ Process isolation▹ What happens in one process must not affect any other process.

▸ Error handling▹ The system must be able to detect and handle errors.

▸Continuous evolution of the system▹ We want to upgrade the system while running and with no loss of service.

First Principles


8

Also▸High level language to get real benefits.▸The language/system should be simple▹ Simple in the sense that there should be a small number of basic principles, if

these are right then the language will be powerful but easy to comprehend and use. Small is good.

▹ The language should be simple to understand and program.

▸ Provide tools for building systems, not solutions▹ We would provide the basic operations needed for building communication

protocols and error handling

First Principles


9

Hard at work developing Erlang

10

▸Simple functional language▹ With a “different” syntax▸ It is safe!▹ For example no pointer errors▸ It is reasonably high-level▹ At least then it was▹ Still is in many ways▸Dynamically typed!▹ No user defined data-types!

Sequential Language

11

▸Typical features of functional languages▹ Immutable data▹ Immutable variables▹ Extensive use of pattern matching▹ Recursion rules!

Sequential Language

12

<< IpVers:4, HdrLen:4, SrvcType:8, TotLen:16, ID:16, Flags:3, FragOff:13, TTL:8, Proto:8, HdrChkSum:16, SrcIP:32, DestIP:32, RestDgram/binary >>

▸ IP datagram of IP protocol version 4

Sequential Language: high-level binaries

13

Mike Williams:“three properties of a programming language are central to the efficient operation of a concurrent language or operating system. These are: 1) the time to create a process. 2) the time to perform a context switch between two different processes and 3) the time to copy a message between two processes. The performance of any highly-concurrent system is dominated by these three times.”

Concurrency

14

▸Light-weight processes▹ Millions of Erlang processes possible on one machine▸Asynchronous message passing▹ Only method of communication between processes▹ Necessary for non-blocking systems▹ Provide basic mechanism▹ Very cheap▸Selective receive mechanism▹ Allows us to ignore messages which are uninteresting now ▸Processes are isolated!▸NO GLOBAL DATA!

Concurrency: core ideas


15

Error Handling: basic premise

ERRORS WILL ALWAYS OCCUR!


16

Error Handling: basic goal

THE SYSTEM MUST NEVER GO DOWN!

Parts may crash and burnBUT

The system must never go down!

17

Robust systems must always be aware of errorsBUT

Want to avoid writing error checking code everywhere

Want to be able to handle process crashes among cooperating processes

Interact well with process communication

Error Handling

18

We just want to

Error Handling

Let it crash!

19

▸Process based▸ If one process crashes then all cooperating processes should

crash▹ Cooperating processes are linked together▹ Process crashes propagate exit signals along links▹ A process receiving an exit signal crashes▸ ”System” processes can monitor them and restart them when

necessary by trapping exits▹ exit signals are converted to messages in the process message queue▹ the process does not crash▸But sometimes we do need to handle errors locally

Error Handling

20

How do you build robust systems?

▸At least you need to ensure▹ Necessary functionality is always available▹ System cleans up when things go wrong

▸Must have at least two machines!▹ Need distribution

Robust Systems

21

Supervision trees

▸Supervisors will start child processes▹ Workers▹ Supervisors▸Supervisors will monitor

their children▹ Through links and trapping exits▸Supervisors can restart the

children when they terminate

Robust Systems: necessary functionality

Supervisors

Workers

22

Monitor processes

▸Servers monitoring clients▹ Clean-up after then if they crash▸Processes monitoring co-workers▸Groups of co-workers dying together

Robust Systems: system cleanup

23

▸A set of design patterns for building concurrent, fault tolerant systems▸Generic behaviours▹ Implement the design patterns▹ Extensible to support new patterns▸Libraries▸Support tools for building systems/releases

OTP (Open Telecom Platform)

There is nothing about telecomsin OTP!

24

Systems built with Erlang tend to be very OS like

▸Provides services▸Very seldom a central thread of execution▹ At most something which starts “tasks”

Systems

25

What is the BEAM?

A virtual machine to run Erlang

26

▸ Lightweight, massive concurrency▸Asynchronous communication▸ Process isolation▸ Error handling▸Continuous evolution of the system▸ Soft real-time▸Transparent SMP/multi-core support▸ Interfaces to the outside world

These we seldom have to worry about directly in a language, except for receiving messages

Properties of the BEAM

27

▸ Immutable data▸ Predefined set of data types▸ Pattern matching▸ Functional language▸Modules/code▸No global data

These are what we mainly "see" directly in our languages

Properties of the BEAM

Erlang Ecosystem

28

Languages built/running on top of the BEAM, Erlang and OTP.

By following “the rules” the languages openly interact with, and support, each other making the whole system more powerful than any individual language can ever be.

Erlang Ecosystem

29

The whole system can interact with other

systems

30

Extending the system

31

▸ Languages which keep the basic Erlang execution model and data types▹ New syntax▹ Different “packaging”

▸ Elixir▸ LFE (Lisp Flavoured Erlang

▸ Languages which extend the Erlang execution model and data types▸ Lua▸ Prolog

Extending the system: new skin for the old ceremony

32

The thickness of the skin affects how efficiently the new language can be implemented and how seamlessly it can interact

Extending the system: new skin for the old ceremony

ERLANG BEAM

OTP

OTP

Erlang

New Language

New Language libraries

33

▸ “Elixir is a dynamic, functional language designed for building scalable and maintainable applications.”▸ “Elixir is influenced by Ruby”▹ “Elixir is NOT Ruby on the Erlang VM”▸Elixir has meta programming capabilities using macros▸Many libraries and interfaces standardised, rewritten and

extended▸Comes with extensive set of build tools

Elixir

Thin skin

34

▸ "LFE, Lisp Flavoured Erlang, is a lisp syntax front-end to the Erlang system"▸ It's a real Lisp▸Provides lots of lisp goodies▹ Real homoiconicity and macros (yay!)▸Seamlessly interacts with vanilla Erlang/OTP▹ Be able to freely mix vanilla code and LFE code▸Small core language

LFE (Lisp Flavoured Erlang)

Thin skin

35

▸ "Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description."▸ Implements Lua 5.2▹ All of it!▸Shared, mutable, global data▸Lua handling of code▸ ...

Lua

Thick skin

Robert [email protected]@erlang-solutions.com@rvirding

Internet

FunctionalConf '16 Robert Virding Erlang Ecosystem