IQMULUS INFRASTRUCTURE

Michel KrämerFraunhofer IGD

IQmulus, Final WorkshopBergen, September 21, 2016

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 318787.

IQMULUS CODE CAMP JUNE 2013

2

IQMULUS ARCHIT ECT URE

3

IQMULUS ARCHIT ECT URE

4

IQMULUS ARCHIT ECT URE

5

MICROSERVICE ARCHIT ECTURE

„In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.“

- Martin Fowler

6

http://martinfowler.com/articles/microservices.html

IQMULUS ARCHIT ECT URE

7

Processingcloud

Virtualmachine Virtualmachine Virtualmachine

…

Service Service

Service Service

Service Service

Service

Service Service

Service

Service

Service Service

Service

Service Service

Service Service

Service

ServiceService

Service Service

D S L TOOLC H A IN , C ON TR OL C OMP ON ENTS

8

User:• Select/write workflow• Upload/select data to process

Infrastructure:• Select suitable processing services

and parameters• Generate executable process chains• Monitor execution and use of

resources

Workflow script(DSL)

Workflow Editor

Interpreter

Job Manager

Job TrackerJob TrackerJob Tracker

Processing serviceProcessing serviceProcessing service

US2: DOMAIN-SPECIFI C LANGUAGE

9

Unroll and parallelizefor loop

FOR LOOP UNROL L ING

10

[PointCloudCollection]

Placeholder:[File1,File2,File3,…]Interpreter

Loopiteration1

Loopiteration2

Loopiteration3 …

File1 File2 File3 …

JobManager

JobTrackerServices

Node1

JobTrackerServices

Node2

JobTrackerServices

Node3

…

…

FOR LOOP UNROL L ING

11

[PointCloud]

Placeholder:File1Interpreter

Workflow

File1

JobManager

JobTrackerServices

Node1

JobTrackerServices

Node2

JobTrackerServices

Node3

…

…File1File1Fileset

FOR LOOP UNROL L ING

12

[PointCloudCollection]

Placeholder:[File1,File2,File3,…]Interpreter

Loopiteration1

Loopiteration2

Loopiteration3 …

File1 File2 File3 …

JobManager

JobTrackerServices

Node1

JobTrackerServices

Node2

JobTrackerServices

Node3

…

…File1File1Fileset

F IFO SCHEDULING

13

N4

Nodes

N3 N2 N1 P1

Process chains

P2 P3 P4 P5 P6

F IFO SCHEDULING

14

N4

Nodes

N3 N2 N1 P1

Process chains

P2 P3 P4 P5 P6

F IFO SCHEDULING

15

N4

Nodes

N3 N2

N1 P1

Process chains

P2 P3 P4 P5 P6

F IFO SCHEDULING

16

N4

Nodes

N3

N2

N1 R1

Process chains

P2

P3 P4 P5 P6

F IFO SCHEDULING

17

N4

Nodes

N3

N2

N1

R1

Process chains

R2

P3

P4 P5 P6

F IFO SCHEDULING

18

N1

Nodes

N4

N3

N2

R1

Process chains

R3

P4

P5 P6

R2

F IFO SCHEDULING

19

N2

Nodes

N1

N4

N3

R1

Process chains

R4

P5

P6

R2R3

F IFO SCHEDULING

20

N3

Nodes

N2

N1

N4

R1

Process chains

R5

P6

R2R3R4

F IFO SCHEDULING

21

N4

Nodes

N3

N2

N1

R1

Process chains

R6 R2R3R4R5

F IFO SCHEDULING

22

N1

Nodes

N4 N3N2

R1

Process chains

R6 R2R3R4R5

INFRASTRUCTURE: VMS

23

INFRASTRUCTURE: VMS

24

OVERVIEW

25

SOME STATIST ICS

26

• 69 virtual machines• about 70 Docker images• about 570 Docker containers• 4 Spark clusters (24 VMs)• 1 Hadoop cluster (4 VMs)• HDFS (30 TB, 15 TB used, 61 VMs)

TECHNOLOGIES

27

• Docker• Spark• Hadoop• Ansible• Angular JS + Bootstrap• MongoDB/PostgreSQL• Elasticsearch + Logstash + Kibana (ELK)• Artifactory• C/C++/Java/Scala/Groovy/Python/TypeScript/JavaScript

28