32
1 Exploring Magnum and Senlin integration for autoscaling containers Hongbin Lu (Huawei) Jay Lau, Ton Ngo (IBM) Qiming Teng, Julio Ruano (IBM) Exploring Magnum and Senlin integration

Autoscaling with magnum and senlin

Embed Size (px)

Citation preview

Page 1: Autoscaling with magnum and senlin

1

Exploring Magnum and Senlin integration

for autoscaling containers

Hongbin Lu (Huawei)

Jay Lau, Ton Ngo (IBM)

Qiming Teng, Julio Ruano (IBM)

Exploring Magnum and Senlin integration

Page 2: Autoscaling with magnum and senlin

Contributors

Exploring Magnum and Senlin integration 2

Zilian Ji, Linjuan Xia, Shijia E, Peiyu Lin, Tianyuan Zhuang, Ji Jiang, Yongkui Wang

Page 3: Autoscaling with magnum and senlin

Talk Outline

1. Use cases

2. Magnum

3. Senlin

4. How it works

5. Demo

Exploring Magnum and Senlin integration 3

Page 4: Autoscaling with magnum and senlin

Use cases

Exploring Magnum and Senlin integration 4

Page 5: Autoscaling with magnum and senlin

Use case

• Shared COE cluster

– Autoscale on some policies (utilization, etc.)

– Scale number of nodes

• Multiple apps on cluster

– Autoscale on some policies (request rate, etc.)

– Scale number of containers

• Example:

– 2, 3-tiered web apps: handle requests

– SLA determines policy

Exploring Magnum and Senlin integration 5

Page 6: Autoscaling with magnum and senlin

Scaling scenarios (1)

Exploring Magnum and Senlin integration 6

node

c c c

node

c cc

node

c scale app

scale cluster

Single app on cluster

• scale out containers as needed

• scale cluster by utilization

• coordination = utilization

Page 7: Autoscaling with magnum and senlin

Scaling scenarios (2)

Exploring Magnum and Senlin integration 7

node

c

node

c c

node

c c c c scale 2 apps

Two apps on cluster

• scale out containers as needed

• scale cluster by utilization

• coordination = utilization

scale cluster

unlimited

scale cluster

limited

Page 8: Autoscaling with magnum and senlin

Scaling scenarios (3)

Exploring Magnum and Senlin integration 8

node

c c

node

scale 2 apps

scale cluster

limited

c

node

c c c c c c

Two apps on cluster, limited cluster size

• blue scales first, retains containers

• red scales later, fails

• maybe acceptable

• what if red is critical and blue is not

• inverted priority: blue < red

Page 9: Autoscaling with magnum and senlin

Scaling scenarios (4)

Exploring Magnum and Senlin integration 9

node

c c scale 2 apps

Limit on apps

• Place max on blue

• blue scales, red does not scale

• cannot use full resources

scale cluster

limited

c

nodenode

c

limited

Page 10: Autoscaling with magnum and senlin

Scaling scenarios (5)

Exploring Magnum and Senlin integration 10

node

c c

node

scale 2 apps

scale cluster

limited

c

node

c c c c c cc c c c

Priority on apps

• blue scales first, red scales later

• blue scales back to free up resource for red

• 2 ways linkage: container cluster host cluster

Page 11: Autoscaling with magnum and senlin

Magnum

Exploring Magnum and Senlin integration 11

Page 12: Autoscaling with magnum and senlin

Instance

Magnum Overview

Exploring Magnum and Senlin integration 12

• Provision

• Kubernetes

• Docker Swarm

• Mesos

• Scale

• Add instances

• Remove instances

• Manage

• Pod

• Service

• Replication Controller

• Container

Nova Instance

containercontainer

Magnum CLI

Magnum

Heat

Page 13: Autoscaling with magnum and senlin

Magnum Architecture

Exploring Magnum and Senlin integration 13

Page 14: Autoscaling with magnum and senlin

Magnum: Autoscale Bay

1. Raw data: Magnum pulls raw

data from Bays.

2. Metrics: Metrics, like memory

utilization, are computed and sent

them to Ceilometer.

3. Alarm: Ceilometer triggers alarms

based on user-specified scenario

and metrics.

4. Scale: Heat scales the bays.

Exploring Magnum and Senlin integration 14

Ceilometer

Magnum Heat

Bay1) Raw data

2) Metrics 3) Alarm

4) Scale

Page 15: Autoscaling with magnum and senlin

Kubernetes: Autoscale Pod

1. Metrics: The autoscaler

periodically queries pods’ metrics.

2. Scale: Based on collected

metrics, the autoscaler uses a

built-in algorithm to trigger the

scaling.

3. Resize: Replication Controller

resize the pods.

Exploring Magnum and Senlin integration 15

Horizontal

Pod

Autoscaler

Pods

1) Metrics3) Resize

Replication

Controller

2) Scale

Page 16: Autoscaling with magnum and senlin

Future direction

• Magnum AutoScaler

– A new component in Magnum

• Scaling Bays

• Scaling Applications

– Interacts with AutoScaler from COEs

• Kubernetes AutoScaler as a driver for container-level scaling

• Magnum responsible for scaling RCs without Kubernetes autoscaler

– Customizability

• data sources

• metrics and thresholds

• standby bay nodes (i.e. resource pool)

Exploring Magnum and Senlin integration 16

Page 17: Autoscaling with magnum and senlin

Two-Levels of AutoScaling

Exploring Magnum and Senlin integration 17

Working

K8S Bay

NodeNode

K8S Master

Heapster

Kublet

cAdvisor

Kublet

cAdvisor

Storage Backend

RC AutoScaler

Pod

Magnum AutoScaler Service

K8S Driver

Collector

Analyzer

OperatorSwarm Driver

MesosDriver

Senlin

Magnum Conductor

Standby

Page 18: Autoscaling with magnum and senlin

Senlin

Exploring Magnum and Senlin integration 18

Page 19: Autoscaling with magnum and senlin

Clustering - A Missing Service in OpenStack

Exploring Magnum and Senlin integration 19

Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects

Page 20: Autoscaling with magnum and senlin

Senlin Architecture

Exploring Magnum and Senlin integration 20

REST RPC

Senlin

Database

Policies

Profiles

Page 21: Autoscaling with magnum and senlin

Clustering/AutoScaling/LoadBalancing

Exploring Magnum and Senlin integration 21

Nova

Kubernetes

Heat

Ironic BareMetal

VMs

Stacks

Containers

Senlin

placement

deletion

scaling

health

load-balance

batching

Page 22: Autoscaling with magnum and senlin

Scaling A Cluster using Senlin

Exploring Magnum and Senlin integration 22

type: os.nova.serverversion: 1.0properties:

image: fedora-21flavor: 3networks:

- network: privateuserdata:

...

Profile

Cluster

type: senlin.policy.scalingversion: 1.0properties:

event: CLUSTER_SCALE_INadjustment:

type: CHANGE_IN_CAPACITYnumber: 1min_step: 1best_effort: True

attach

Policy

type: senlin.policy.scalingversion: 1.0properties:

event: CLUSTER_SCALE_OUTadjustment:

. . .

attach

Policy

webhook

RESIZE

Page 23: Autoscaling with magnum and senlin

Policies

Exploring Magnum and Senlin integration 23

type: senlin.policy.scalingversion: 1.0properties:adjustment:min_step: 1

best_effort: true

number: 1type: CHANGE_IN_CAPACITY

event: CLUSTER_SCALE_OUT

• Declarative

• Can be mapped to from a standard

• Example: SUR_Cluster

(Add k8s Minion / Add Spark worker node)

Page 24: Autoscaling with magnum and senlin

Trigger

Exploring Magnum and Senlin integration 24

type: CeilometerThresholdAlarmversion: 1.0rule:meter_name: memory_utilcomparison_operator: gtthreshold: 70period: 60evaluation_periods: 1statistic: avgquery:- field: resource_metadata.clusterop: eqvalue: SUR_Cluster

• Generic abstraction

• alarms in OpenStack

monitoring services

• e.g. Ceilometer Alarm

Page 25: Autoscaling with magnum and senlin

How it works

Exploring Magnum and Senlin integration 25

Page 26: Autoscaling with magnum and senlin

Design goals

1. Coordinate scaling at apps and cluster level

• Control flows in both direction

• Apps -> Cluster: utilization, other methods

• Cluster -> Apps: priority, other methods

2. Integrate scaling support from COE if available

• Kubernetes autoscaler

3. Policy driven

• Easy to express, manage complex cases

Exploring Magnum and Senlin integration 26

Page 27: Autoscaling with magnum and senlin

template

Outer Stack

Autoscaling: OpenStack-Only Solution (Today)

27 Exploring Magnum and Senlin integration

AutoScalingGroup Stack

Heat

Ceilometer

AH

PUp PDn

AL

ASGS1 S2 S3

updateASG

Page 28: Autoscaling with magnum and senlin

Senlin

profile

Autoscaling: OpenStack-Only Solution (New)

28 Exploring Magnum and Senlin integration

Cluster

Ceilometer

AH

PUp

PDn

AL

SCALE_IN

SCALE_OUT

Node1 Node2 Node3

webhookspolicies triggers

Page 29: Autoscaling with magnum and senlin

Demo

Exploring Magnum and Senlin integration 29

Page 30: Autoscaling with magnum and senlin

Autoscaling with Kubernetes

30 Exploring Magnum and Senlin integration

VMs

K8S Bay

NodeNode

Heapster

Kublet

cAdvisor

Kublet

cAdvisor

Pod

Senlin

Magnum

profile

CeilometerWorkload

policy

Page 31: Autoscaling with magnum and senlin

Questions

Exploring Magnum and Senlin integration 31

Page 32: Autoscaling with magnum and senlin

Thank You

Exploring Magnum and Senlin integration 32