ISPA 2008 APDCT Workshop 1

Reinforcement Learning applied to Meta-scheduling in grid environments

Bernardo Costa

Inês Dutra

Marta Mattoso

ISPA 2008 APDCT Workshop 2

Outline

Introduction Algorithms Experiments Conclusions and Future work

ISPA 2008 APDCT Workshop 3

Introduction Algorithms Experiments Conclusions and Future work

ISPA 2008 APDCT Workshop 4

Introduction

Relevance: Available grid schedulers usually do not employ a

strategy that may benefit a single or multiple users.

Some strategies employ performance information dependent algorithms (pida).

Most works are simulated.

Difficulty: monitoring information not reliable due to network latency.

ISPA 2008 APDCT Workshop 5

Introduction Algorithms Experiments Conclusions and Future work

ISPA 2008 APDCT Workshop 6

Study of 2 Algorithms (AG) A. Galstyan, K. Czajkowski, and K.

Lerman. Resource allocation in the grid using reinforcement learning. In AAMAS, pages 1314–1315. IEEE, 2004.

(MQD) Y. C. Lee and A. Y. Zomaya. A grid scheduling algorithm for bag-of-tasks applications using multiple queues with duplication. 5th IEEE/ACIS International Conference on Computer and Information Science and 1st IEEE/ACIS International Workshop on Component-Based Software Engineering, Software Architecture and Reuse. ICIS-COMSAR, pages 5–10, 2006.

ISPA 2008 APDCT Workshop 7

What is reinforcement learning?

Machine learning technique used to learn behaviours given a series of temporal events.

Non-supervised learning. Based on the idea of rewards and

punishments.

ISPA 2008 APDCT Workshop 8

Algorithms

AG and MQD use reinforcement learning to associate an efficiency rank to an RMS.

Reinforcement learning native to AG. MQD was modified to use this technique to

estimate computational power of an RMS. AG allocates RMS in a greedy and probabilistic

way. MQD allocates RMS associatively and

deterministically.

ISPA 2008 APDCT Workshop 9

Algorithms

Calculating efficiency: Reward is assigned to RMS that has performance

better than average. Reward can be negative (punishment). RMS may not change its efficiency value.

ISPA 2008 APDCT Workshop 10

Algorithms

Calculating efficiency: parameters: and l is the importance of the time spent executing a

task affects rewarding.

l is a learning parameter

ISPA 2008 APDCT Workshop 11

Algorithms

AG: With high prob, associates job to the best

available RMS, otherwise, selects randomly. MQD:

Groups of jobs sorted according execution time are associated to an RMS. Most efficient executes the heaviest jobs. Initial allocation to estimate RMS´ efficiency

ISPA 2008 APDCT Workshop 12

Algorithm AG

ISPA 2008 APDCT Workshop 13

J1 J3J2 J4 J5

J6 J7 J8 J9

R1E = 0

R2E = 0

R3E = 0

ISPA 2008 APDCT Workshop 14

J4 J5

J6 J7 J8 J9

R1E = 0

R2E = 0,3

R3E = -0,3

ISPA 2008 APDCT Workshop 15

J7 J8 J9

R1E = 0,3

R2E = 0,057

R3E = 0,51

ISPA 2008 APDCT Workshop 16

Algorithm MQD

ISPA 2008 APDCT Workshop 17

J140

J350

J215

J430

J510

J670

J720

J820

J940

R1E = 0

R2E = 0

R3E = 0

ISPA 2008 APDCT Workshop 18

J140

J350

J215

J430

J510

J670

J720

J820

J940

R1E = 0

R2E = 0

R3E = 0

ISPA 2008 APDCT Workshop 19

J140

J350

J430

J670

J820

J940

R1E = 0,3

R2E = -0,3

R3E = 0

ISPA 2008 APDCT Workshop 20

J140

J350

J820

R1E = 0,09

R2E = -0,09

R3E = -0,3

ISPA 2008 APDCT Workshop

Avg per proc

Global Avg

ISPA 2008 APDCT Workshop

ISPA 2008 APDCT Workshop

ISPA 2008 APDCT Workshop

ISPA 2008 APDCT Workshop 25

Introduction Algorithms Experiments Conclusions and Future work

ISPA 2008 APDCT Workshop 26

Experiments

GridbusBroker:No need to install it in other grid sitesOnly requirement: ssh access to a grid

nodeRound-robin scheduler (RR)

Limitations:Does not support job duplication Imposes a limit on the number of active

jobs per RMS

ISPA 2008 APDCT Workshop 27

Experiments

Resources in 6 grid sites:LabIA: 24 (Torque/Maui)LCP: 28 (SGE)Nacad: 16 (PBS PRO)UERJ: 144 (Condor)UFRGS: 4 (Torque)LCC: 44 (Torque)

ISPA 2008 APDCT Workshop 28

Experiments

Objective: study performance of algorithms in a real grid environment.

Application: bag-of-tasks. CPU intensive.

Duration between 3 and 8 minutes.

ISPA 2008 APDCT Workshop 29

Experiments

Evaluation criteria: makespan.

Makespan was normalized with respect to RR

ISPA 2008 APDCT Workshop 30

Experiments

Phase I: Tuning of parameters and l 500 jobs.

Phase II: Performance of re-scheduling. Later load increased to 1000 jobs.

ISPA 2008 APDCT Workshop 31

Experiments

One experiment is a run of consecutive executions of RR, AG and MQD.

A scenario is a set of experiments with fixed parameters.

For each scenario: 15 runs. T-tests to verify statistical difference

beteween AG/MQD e RR, with 95% confidence (the results have a normal distribution).

ISPA 2008 APDCT Workshop

ISPA 2008 APDCT Workshop 33

Experiments (Phase I)

ISPA 2008 APDCT Workshop 34

Experiments (Phase II)

ISPA 2008 APDCT Workshop 35

Introduction Algorithms Experiments Conclusions and Future work

ISPA 2008 APDCT Workshop 36

Conclusions and Future work

Results showed that was possible to achieve optimizations with both AG and MQD wrt RR

Experiments validate MQD simulation results found in the literature.

Reinforcement learning is a promising technique to classify resources in real grid environments.

ISPA 2008 APDCT Workshop 37

Conclusions and Future work

Study the behavior of AG and MQD with other kinds of applications, e.g., data intensive, with dependencies.

ISPA 2008 APDCT Workshop 38

Questions?

ISPA 2008 APDCT Workshop 39

Annex

ISPA 2008 APDCT Workshop 40

Definições

Gerenciador de recursos: sistema que gerencia a submissão e execução de jobs dentro de um domínio específico.

Resource Management System (RMS): sinônimo para gerenciador de recursos.

Batch job scheduler: escalonador típico de um RMS. Ex: SGE, PBS/Torque.

ISPA 2008 APDCT Workshop 41

Definições

Meta-escalonador: um escalonador que não tem acesso direto aos recursos, mas apenas aos RMS que os gerenciam.

Aprendizado por reforço: técnica que induz um agente a tomar decisões por meio de recompensas oferecidas.

Makespan: tempo total gasto por um meta-escalonador para finalizar a execução de um conjunto de jobs a ele designado.

ISPA 2008 APDCT Workshop 42

Definições

Job: aplicativo submetido ao grid por um usuário, executado em geral por um RMS. Exemplos de tipos de jobs: Bag-of-Tasks: jobs que não possuem relação de

dependência ou precedência explícita entre si. Troca de parâmetros (APST): jobs de um mesmo

executável que diferenciam-se por um valor de entrada que varia entre as execuções.

ISPA 2008 APDCT Workshop 43

ISPA 2008 APDCT Workshop 44