Flow-based programming with Elixir

3rd Kyiv Elixir meetup, October 1, 2016

Hello!

I am Anton Mishchuk

Ruby developer at Matic

Elixir fan

ESpec’s author and maintainer

Github: antonmi

2

Introduction◈ Elixir new awesome feature - GenStage◈ Elixir Flow-Based Programming (Peter C Marks)◈ Flow-Based Programming, 2nd Edition (J. Paul Morrison)

3

What I’d tell you

◈ “Conventional programming”◈ Flow-Based Programming◈ “Telegram problem” example◈ Elixir GenStage◈ “Telegram problem” redesign◈ Data-flow routing with GenStage

4

1.Software design in conventional programming

5

Warning! Too general philosophy!

Micro level

6

◈ Basic components:○ Objects in OOP comprising both

information and behaviour○ Functions and data-structures in FP

◈ Hierarchical organization◈ Sequential communication or

evaluation

DSLs and Frameworksas “coordination languages”

◈ Define common abstractions◈ Initialize your components◈ Use components in right order

7

Visualization

8

Concurrency& parallelism

9

◈ Hierarchic structure of program◈ Code is procedural and sequential◈ Visualisation is more about structure◈ Concurrency is not native

What is wrong with conventional programming?

10

Basic concepts

2.Flow-Based Programming

11

“FBP - a programming paradigm that defines applications as networks of "black box" processes,

which exchange data across predefined connections by message passing, where the connections are specified externally to the processes. These black box processes

can be reconnected endlessly to form different applications without having to be changed internally.

https://en.wikipedia.org/wiki/Flow-based_programming

12

FBP diagram

A

B

D

C

IN 1

IN 1

IN 1

IN 2

IN 1

IN 2

OUT 1

OUT 2

OUT 1

OUT 1

OUT 1

Processes, ports, connections

13

“... whereas the conventional approaches to programming start with process and view data as

secondary, business applications are usually designed starting with data and viewing process as

secondary – processes are just the way data is created, manipulated and destroyed.

J. Paul Morrison, Flow-Based Programming, 2nd Edition

14

Express a problem in terms of transforms on

streams of data15

Soft Drink Bottling Factory

◈ Independent well-defined components◈ Clean interfaces◈ Simple to reconfigure◈ Minimizes side-effects◈ Designer can sit at one “station”, or can

follow an item through system

http://www.jpaulmorrison.com/fbp/FBPnew.ppt16

FPB characteristics

◈ Asynchronous processes communicating via streams of data packets

◈ Data packets with a lifetime of their own◈ Definition of connections external to

components◈ Consistent view from macro to micro

http://www.jpaulmorrison.com/fbp/FBPnew.ppt17

Native parallelism

18

FBP is about “coordination

language”19

Everything new is actually well-forgotten old

◈ 1971, IBM, basic concepts◈ 1975, Bank of Montreal, on-line banking

system, still works!◈ 1994, Flow-Based Programming, 1st Ed◈ 2010, Flow-Based Programming, 2st Ed◈ 2012, NoFlo project by Henri Bergius,FBP in JavaScript◈ JavaFBP, C#FBP, CppFBP (C++ and Lua)

20

What about Erlang/Elixir?

21

FBP example with Elixir

3.Telegram problem

22

Telegram problem

A program which accepts lines of text and generates output lines of a different predefined length, without splitting any of the words in the text.The program accepts an input stream of lines of some length and produce an output stream of lines of another length.

23

Problem FBP design using GenServers

ReadSeq Recompose WriteSeqDecompose

Lines Words Lines

◈ Each component is a GenServer. ◈ Each component implements only

one function - “run” which transforms input to output

◈ Communication is asynchronous◈ Code is here:

https://github.com/antonmi/kyiv_meetup_324

ReadSeq (naive implementation)

25

Decompose (naive implementation)

26

Recompose (naive implementation)

27

WriteSeq (naive implementation)

28

The main Telegram module(naive implementation)

29

What is cool in this simple implementation

◈ Each component do its own part of work◈ The only API is “init” and “run”◈ Components works in parallel◈ Components are under supervisor

30

What is wrong with the implementation

◈ Components are not independent (each of them knows who is the next)

◈ Flow is controlled internally by each of the component

◈ There is no back-pressure mechanism (when some component lags it can be overflowed by data)

31

Experimental feature.

4.Elixir GenStage

32

GenStage abstractions

◈ GenStage is build on top of GenServer◈ Each of the stage can be producer,

consumer or both

producer producer consumer consumerproducer

consumer

33

GenStage communication

◈ Consumer subscribes to producer◈ Consumer asks for data (sends a demand)◈ Producer sends data

producer consumer

Subscribe

Ask

Events

34

GenStage API

◈ init(state) which must return:○ {:producer, state}○ {:producer_consumer, state}○ {:consumer, state}

◈ handle_demand(events, from, state)○ {:noreply, events, new_state}

◈ handle_events(events, from, state)○ {:noreply, events, new_state}

35

5.Telegram problem redesign

36

ReadSeq GenStage

37

Decompose GenStage

38

Recompose GenStage

39

WriteSeq GenStage

40

Main Telegram module

41

It is awesome!

◈ Each component is independent!◈ Data flow is coordinated externally!◈ There is a back-pressure!

42

5.Data-flow routingwith GenStage

43

GenStage Dispatchers

◈ GenStage.DemandDispatcher○ sends events to the highest demand

◈ GenStage.BroadcastDispatcher○ accumulates demand from all

consumers before broadcasting events to all of them

◈ GenStage.PartitionDispatcher○ sends events according to partitions

44

Duplicate file example

45

ReadSeq

WriteSeq

Broadcast

WriteSeq

Code is here: https://github.com/antonmi/kyiv_meetup_3

The only new component is Broadcast

46

Split file example

47

Code is here: https://github.com/antonmi/kyiv_meetup_3

ReadSeq

WriteSeq

Split

WriteSeq

The only new component is Split

48

6.Conclusion

49

Erlang VM+

Elixir GenStage=

FBP paradigm50

Thanks!