Online Model-Based Performance and Resource … · Towards self-aware performance and resource management in modern service-oriented systems. In ... Deployment topology, resource

KIT – University of the State of Baden-Wuerttemberg and

National Research Center of the Helmholtz Association

DESCARTES RESEARCH GROUP

INSTITUTE FOR PROGRAM STRUCTURES AND DATA ORGANIZATION

www.kit.edu

Samuel Kounev

VMware Academic Research Symposium, Palo Alto, July 17, 2013

Online Model-Based Performance and Resource

Management in Virtualized Application Environments

2

Elastic Capacity Management / Online Workload Forecasting

N. Herbst, S. Kounev, and R. Reussner. Elasticity in Cloud Computing: What it is, and What it is Not. In Proc. of the 10th Intl. Conference

on Autonomic Computing (ICAC 2013), San Jose, CA, June 24-28. USENIX. 2013. [ bib | slides | .pdf ]

N. Herbst, N. Huber, S. Kounev, and E. Amrehn. Self-Adaptive Workload Classification and Forecasting for Proactive Resource

Provisioning. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), Prague, Czech

Republic, April 21-24. 2013. [ bib | slides | .pdf ]

N. Huber, F. Brosig, and S. Kounev. Model-based Self-Adaptive Resource Allocation in Virtualized Environments. In 6th International

Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2011), Waikiki, Honolulu, HI, USA. May 23-24,

2011. [ bib | http | .pdf ]

Automatic Model Extraction based on Benchmarking or Online System Monitoring

Q. Noorshams, D. Bruhn, S. Kounev, and R. Reussner. Predictive Performance Modeling of Virtualized Storage Systems using Optimized

Statistical Regression Techniques. In Proc. of the 4th ACM/SPEC International Conference on Performance Engineering, Prague, Czech

Republic, ICPE '13, pages 283-294, New York, NY, USA. ACM. 2013. [ bib | DOI | http | .pdf ]

F. Brosig, N. Huber, and S. Kounev. Automated Extraction of Architecture-Level Performance Models of Distributed Component-Based

Systems. In 26th IEEE/ACM International Conference On Automated Software Engineering (ASE 2011), Oread, Lawrence, Kansas.

November 2011. [ bib | .pdf ]

S. Kounev, K. Bender, F. Brosig, N. Huber, and R. Okamoto. Automated Simulation-Based Capacity Planning for Enterprise Data Fabrics.

In 4th International ICST Conference on Simulation Tools and Techniques (SIMUTools 2011), Barcelona, Spain. March 21-25, 2011.

Best Paper Award. [ bib | slides | .pdf ]

Q. Noorshams, K. Rostami, S. Kounev, P. Tůma, and R. Reussner. I/O Performance Modeling of Virtualized Storage Systems. In

Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication

Systems (MASCOTS 2013), San Francisco, USA. 2013. [ bib | .pdf ]

Performance Modeling and Prediction in Virtualized Environments

F. Brosig, F. Gorsler, N. Huber, and S. Kounev. Evaluating Approaches for Performance Prediction in Virtualized Environments.

In Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication


N. Huber, M. von Quast, M. Hauck, and S. Kounev. Evaluating and Modeling Virtualization Performance Overhead for Cloud

Environments. In Proceedings of the International Conference on Cloud Computing and Services Science (CLOSER 2011),

Noordwijkerhout, The Netherlands, pages 563 - 573. SciTePress. May 7-9, 2011. Best Paper Award. [ bib | http | .pdf ]

References

© S. Kounev Online Model-Based Performance and Resource Management in Virtualized Application Environments

http://sdqweb.ipd.kit.edu/publications/kounev_bib.html

http://sdqweb.ipd.kit.edu/publications/pdfs/HeKoRe2013-ICAC-Elasticity_Slides.pdf

http://sdqweb.ipd.kit.edu/publications/pdfs/HeKoRe2013-ICAC-Elasticity.pdf



http://sdqweb.ipd.kit.edu/publications/pdfs/HeHuKoAm2013-ICPE-WorkloadClassificationAndForecasting_Slides.pdf

http://sdqweb.ipd.kit.edu/publications/pdfs/HeHuKoAm2013-ICPE-WorkloadClassificationAndForecasting.pdf



http://dl.acm.org/authorize?425581

http://descartes.ipd.kit.edu/descartes-pdfs/HuBrKo2011-SEAMS-ResAlloc.pdf



http://dx.doi.org/10.1145/2479871.2479910

http://doi.acm.org/10.1145/2479871.2479910

http://sdqweb.ipd.kit.edu/publications/pdfs/noorshams2013a.pdf



http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/BrHuKo2011-ASE-AutomExtraction.pdf



http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/KoBeBrHuOk2011-ICST-DataFabrics_Slides.pdf

http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/KoBeBrHuOk2011-ICST-DataFabrics.pdf



http://sdqweb.ipd.kit.edu/publications/pdfs/noorshams2013c.pdf



http://sdqweb.ipd.kit.edu/publications/pdfs/BrGoHuKo2013-MASCOTS-EvaluationApproachesForPerformancePredictionInVirtualizedEnvironments.pdf



http://closer.scitevents.org/

http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/HuQuHaKo2011-CLOSER-ModelVirtOverhead.pdf


3

References

Descartes Meta-Model (DMM) - Online Models for Architecting Self-Aware Systems

http://www.descartes-research.net/research_and_profile/descartes_meta_model/

F. Brosig, N. Huber, and S. Kounev. Architecture-Level Software Performance Abstractions for Online Performance Prediction. Elsevier

Science of Computer Programming Journal (SciCo), 2013. [ bib | DOI | .pdf ]

N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. Modeling Run-Time Adaptation at the System Architecture Level in Dynamic

Service-Oriented Environments. Service Oriented Computing and Applications (SOCA), 2013, Springer. In print. [ bib | .pdf ]

N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System

Architectures. In 9th IEEE Intl. Conf. on e-Business Engineering (ICEBE 2012), Hangzhou, China, September 9-11, 2012. [ bib | http | .pdf ]

F. Brosig, N. Huber, and S. Kounev. Modeling Parameter and Context Dependencies in Online Architecture-Level Performance Models. In

Proc. of the 15th ACM SIGSOFT Intl. Symposium on Component Based Software Engineering (CBSE 2012), June 26-28, 2012. [ bib | .pdf ]

N. Huber, F. Brosig, and S. Kounev. Modeling Dynamic Virtualized Resource Landscapes. In Proceedings of the 8th ACM SIGSOFT

International Conference on the Quality of Software Architectures (QoSA 2012), Bertinoro, Italy, June 25-28, 2012. [ bib| .pdf ]

S. Kounev, F. Brosig, and N. Huber. Descartes Meta-Model (DMM). Technical report, Karlsruhe Institute of Technology (KIT), 2013. To

appear. [ bib | http ]

Vision of Self-Aware Computing Systems

“Model-driven Algorithms and Architectures for Self-Aware Computing Systems” Dagstuhl Seminar scheduled to take place in

October 2014 organized by Samuel Kounev, Jeff Kephart, Marta Kwiatkowska and Xiaoyun Zhu.

S. Kounev. Engineering of Self-Aware IT Systems and Services: State-of-the-Art and Research Challenges. In Proc. of the 8th European

Performance Engineering Workshop (EPEW'11), Borrowdale, The English Lake District, October 12-13. 2011. (Keynote Talk). [ bib | .pdf ]

S. Kounev. Self-Aware Software and Systems Engineering: A Vision and Research Roadmap. In GI Softwaretechnik-Trends, 31(4),

November 2011, ISSN 0720-8928, Karlsruhe, Germany, 2011. [ bib | .html | .pdf ]

S. Kounev, F. Brosig, and N. Huber. Towards self-aware performance and resource management in modern service-oriented systems. In

Proc. of the 7th IEEE Intl. Conference on Services Computing (SCC 2010), July 5-10, Miami, Florida, USA. IEEE, 2010. [ bib | .pdf ]


http://sdqweb.ipd.kit.edu/publications/article_brosig_bib.html

http://dx.doi.org/10.1016/j.scico.2013.06.004

http://sdqweb.ipd.kit.edu/publications/pdfs/BrHuKo2013-SciCo-SoftwarePerformanceAbstractions.pdf


http://sdqweb.ipd.kit.edu/publications/descartes_2013_bib.html

http://sdqweb.ipd.uka.de/publications/descartes-pdfs/HuHoKoBrKo2013-SOCA-ModelingRuntimeAdaptation.pdf



http://conferences.computer.org/icebe/2012/index.htm

http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/HuHoKoBrKo2012-ICEBE-AdaptationLanguage.pdf



http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/BrHuKo2012-CBSE-ParamDep.pdf



http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/HuBrKo2012-QoSA-ModelingVirtResLandscapes.pdf



http://www.descartes-research.net/metamodel/


http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/Ko2011-EPEW-Keynote.pdf



http://pi.informatik.uni-siegen.de/stt/31_4/index.html

http://sdqweb.ipd.uka.de/publications/descartes-pdfs/Ko2011-SE-DescartesResearch.pdf



http://sdqweb.ipd.uka.de/publications/descartes-pdfs/KoBrHuRe2010-SCC-Towards.pdf


4

Cloud Usage Scenarios, Challenges and Opportunities

A. Milenkoski, A. Iosup, S. Kounev, K. Sachs, P. Rygielski, J. Ding, W. Cirne, and F. Rosenberg. Cloud Usage Patterns: A Formalism for

Description of Cloud Usage Scenarios. Technical Report SPEC-RG-2013-001 v.1.0.1, SPEC Research Group - Cloud Working Group,

Standard Performance Evaluation Corporation (SPEC), April 2013. [ bib | .pdf ]

S. Kounev, P. Reinecke, F. Brosig, J. T. Bradley, K. Joshi, V. Babka, A. Stefanek, and S. Gilmore. Providing dependability and resilience

in the cloud: Challenges and opportunities. In K. Wolter, A. Avritzer, M. Vieira, and A. van Moorsel, editors, Resilience Assessment and

Evaluation of Computing Systems, XVIII. Springer-Verlag, Berlin, Heidelberg, 2012. ISBN: 978-3-642-29031-2. [ bib | http | .pdf]

Performance Isolation in Shared Execution Environments (e.g., Multi-Tenant SaaS)

R. Krebs, C. Momm, and S. Kounev. Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments. Elsevier

Science of Computer Programming Journal (SciCo), 2013. To appear. [ bib ]

R. Krebs, C. Momm, and S. Kounev. Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments. In Barbora

Buhnova and Antonio Vallecillo, editors, Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software

Architectures (QoSA 2012), Bertinoro, Italy, pages 91-100, New York, USA. ACM Press. June 25-28, 2012. [ bib | http | .pdf ]

Intrusion Detection and Prevention in Virtualized Environments

A. Milenkoski, S. Kounev, A. Avritzer, N. Antunes, and M. Vieira. On Benchmarking Intrusion Detection Systems in Virtualized

Environments. Technical Report SPEC-RG-2013-002 v.1.0, SPEC Research Group - IDS Benchmarking Working Group, Standard

Performance Evaluation Corporation (SPEC), June 2013. [ bib | .pdf ]

References



http://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC-RG-2013-001_CloudUsagePatterns.pdf



http://www.springer.com/computer/communication+networks/book/978-3-642-29031-2

http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/KoReBrBrJoBaStGi2012-ResBook-CloudChallenges.pdf




http://qosa.ipd.kit.edu/qosa_2012/

http://sdqweb.ipd.kit.edu/publications/pdfs/KrMoKo2012-QoSA-QuantifyingPerfIsoMetrics.pdf



http://research.spec.org/fileadmin/user_upload/documents/wg_ids/endorsed_publications/SPEC-RG-2013-002-BenchmarkingVMMBIDSes.pdf


5

Agenda

Motivation

Goals & Approach

Case Studies

Workload Forecasting

Performance Modeling

Resource Management

Outlook


6

Motivation

Traffic Monitoring System Inventory Management System


Goals & ApproachMOTIVATION OutlookCase Studies

7

Motivation

Varying Workloads

vs.

Varying Workloads

vs.vs.




8

Motivation

Varying Workloads

vs.

System Evolution

• New streets / bus lines

• New features and services

• Upgraded cameras

vs.

Varying Workloads

vs.vs.

System Evolution

• New supermarket stores

• New features and services

• Upgraded RFID readers

vs.




9

Motivation


Software systems increasingly complex and dynamic

Must be reconfigured at run-time more and more frequently

Component instances, application configuration

Deployment topology, resource allocations

Two issues:

Determine WHEN exactly reconfigurations are necessary?

Determine WHAT exactly each reconfiguration should do?



10

Example: Elastic Resource Provisioning

Load

Fluctuations

SLAs



11

Example: Elastic Resource Provisioning

Load

Fluctuations

SLAs

Expand / shrink resources on-the-fly

• When exactly should a reconfiguration be triggered?

• Which particular resources should be scaled?

• How quickly and at what granularity?



12

Semantic Gap Problem

VMM

Host

VM VM VM

VMM

Host

VM VM VM

VMM

Host

VM VM VM

OS

JVM

Java EE

EAR EAR

Complex Software Stacks

• Multiple layers

• Heterogeneous

Applications

• Multiple tiers

• Multiple resource types

Resource

Allocation

End-to-end QoS metrics

Application SLAs

Resource Allocations in

Each Tier & Each Layer?



13

State-of-the-Art

Hard to predict the effect of dynamic changes on the system

performance and resource demands

Rely on simple trigger-based “best effort” adaptation mechanisms

OR

Avoid need for adaptation by over-provisioning resources

Consequences: Poor resource efficiency

Rising energy costs for IT systems

1600% increase by 2025 [Gartner]

Rising global CO2 emissions of ICT sector

Today: ca 3%, Increase to 10% expected in 10 years [EU]



14

Descartes Research Group

Modeling methods for predicting at run-time the effect of

dynamic changes on the system Quality-of-Service (QoS)

Current focus: availability and performance

(response time, throughput and resource/energy efficiency)

Model-based algorithms and techniques for autonomic

system adaptation during operation

Goal:

End-to-end QoS guarantees

High resource/energy efficiency

Low operating costs



15

Descartes Research Group



16

“I think,therefore I am…”

-- René Descartes

Self-Reflective

Aware of their software architecture, execution environment and hardware

infrastructure, as well as of their operational goals (e.g., QoS and efficiency)

Self-Predictive

Able to anticipate and predict the effect of dynamic changes in the environment, as

well as the effect of possible adaptation actions

Self-Adaptive

Proactively adapting as the environment evolves to ensure that their operational

goals are continuously met



17

“I think,therefore I am…”

-- René Descartes

Self-Reflective

Aware of their software architecture, execution environment and hardware

infrastructure, as well as of their operational goals (e.g., QoS and efficiency)

Self-Predictive

Able to anticipate and predict the effect of dynamic changes in the environment, as

well as the effect of possible adaptation actions

Self-Adaptive

Proactively adapting as the environment evolves to ensure that their operational

goals are continuously met



S. Kounev. Engineering of Self-Aware IT Systems and Services: State-of-the-Art and Research Challenges. In Proc. of the 8th

European Performance Engineering Workshop (EPEW'11), Borrowdale, The English Lake District, October 12-13. 2011.

(Keynote Talk). [ bib | .pdf ]

S. Kounev. Self-Aware Software and Systems Engineering: A Vision and Research Roadmap. In GI Softwaretechnik-Trends,

31(4), November 2011, ISSN 0720-8928, Karlsruhe, Germany, 2011. [ bib | .html | .pdf ]

S. Kounev, F. Brosig, and N. Huber. Towards self-aware performance and resource management in modern service-oriented

systems. In Proc. of the 7th IEEE Intl. Conference on Services Computing (SCC 2010), July 5-10, Miami, Florida, USA.

IEEE, 2010. [ bib | .pdf ]













18

Vision: Self-Aware Computing Systems


GOALS & APPROACHMotivation OutlookCase Studies

19


PHASE 1

Online QoS Prediction for Problem Anticipation



20


“Online”-Vorhersage zur

Analyse der Auswirkung

möglicher Rekonfigurationen

PHASE 2

Online QoS Prediction for Reconfiguration Impact Analysis

Online reconfiguration impact prediction

for trade-off analysis

Service A

VM1

VM replication/

cloning

Service A

VM1*

Scaling up/

Improving dependability

Online prediction

Dependability/

Responsiveness

OK

Service A

VM1

Service B

VM1

Service C

VM1

Dynamic server

consolidation

LiveVM

migration

Online prediction

Efficiency

OK



21


PHASE 3

Autonomic System Adaptation



22




23

Examples of Performance-Influencing Factors

System workload and usage profile Number and type of clients Input parameters and input data Data formats used Service workflow

Software architecture Connections between components Flow of control and data Component resource demands Component usage profiles

Execution environment Number of component instances Server execution threads Amount of Java heap memory Size of database connection pools

Virtualization layer Physical resources allocated to VMs

number of physical CPUsamount of physical memorysecondary storage devices

Network bandwidth between system nodes



24

Technical

Report

+

IEEE TSE

Article

Under Review

Descartes Meta-Model (DMM)

Architecture-level modeling language for modeling QoS and resource

management related aspects of IT systems

Prediction of the impact of dynamic changes at run-time

Current version focused on performance (incl. capacity and efficiency)

[Springer SOCA 2013]

[IEEE ICEBE 2012]

[AC

M Q

oS

A2012]

[Elsevier SciCo 2013]

[ACM CBSE 2012]

[IEEE/ACM ASE 2011]

[ACM QoSA 2012]

[CLOSER 2011]

[DOA 2010]

Adaptation Points

Application Architecture

Resource Landscape

Aaptation Process

<<Container>>Node1

<<Container>>Node3

<<Container>>Node2

<<InternalAction>>

ResourceDemandX

Degrees-of-

Freedom



25

N. Huber, F. Brosig, and S. Kounev. Modeling Dynamic Virtualized Resource Landscapes. In

Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures

(QoSA 2012), Bertinoro, Italy, June 25-28, 2012. [ bib| .pdf ]

Example: Resource Environment






26

Example: Resource Environment

Influence Factors of the Virtualization Layer

Virtualization Platform

Binary Translation

Para-Virtualization

Full Virtualization

Virtualization Type

exclusive OR

Legend

inclusive OR

Resource Management

Configuration

CPU Scheduling

CPU Allocation

Core Affinity

Workload Profile

I/O

CPU

Memory

Network

Disk

CPU Priority Memory Allocation

Number of VMs

Resource

Overcommitment

VMM Architecture

Dom0

Monolitic

e.g. cap=50e.g. mask=1,2

e.g. vcpu=4

N. Huber, M. von Quast, M. Hauck, and S. Kounev. Evaluating and Modeling Virtualization Performance

Overhead for Cloud Environments. In Proceedings of the International Conference on Cloud Computing

and Services Science (CLOSER 2011), Noordwijkerhout, The Netherlands, pages 563 - 573. SciTePress.

May 7-9, 2011. Best Paper Award. [ bib | http | .pdf ]







27

Example: Application Architecture

<<UsageScenario>>

DealerDriver.Manage

<<SystemCallAction>>

showInventory<<SystemCallAction>>

showInventory


home

<<BranchAction>>

<<BranchTransition>>

Probability: 0.6<<BranchTransition>>

Probability: 0.4


cancelOrder

<<BranchAction>>

<<BranchTransition>>

Probability: 0.4<<BranchTransition>>

Probability: 0.6

<<LoopAction>>

Loop Iteration Number =

[ (1;0.55) (2;0.11)... ]


sellInventory

Control flow and data flow, service resource demands

Parameter and context dependencies

F. Brosig, N. Huber, and S. Kounev. Architecture-Level Software Performance Abstractions for Online Performance

Prediction. Elsevier Science of Computer Programming Journal (SciCo), 2013. [ bib | DOI | http | .pdf ]

F. Brosig, N. Huber, and S. Kounev. Modeling Parameter and Context Dependencies in Online Architecture-Level

Performance Models. 15th ACM SIGSOFT Intl. Symposium on Component Based Software Engineering (CBSE 2012),

June 26-28, 2012. [ bib | http | .pdf | Abstract ]





http://authors.elsevier.com/sd/article/S0167642313001421




http://cbse-conferences.org/2012/



28

Impact Prediction through Automatic Generation of

Tailored Predictive Models at Run-Time

Infrastructure

Sub-model

Virtualization

Sub-model

Middleware

Sub-model

Soft. Arch.

Sub-model

Operational

Analysis

Analytical Sol.

Queueing

Network Models

Analytical Sol.

Simulation

Full-Blown

Simulation Simulation

Queueing

Petri Nets

Analytical Sol.

Simulation

Stochastic

Process Alg.

Analytical Sol.

Dynamically

Composed

Model

Instance

Usage

Sub-model



30

Case Study: Process Control System (ABB)

P. Meier, S. Kounev, and H. Koziolek. Automated Transformation of Component-based Software

Architecture Models to Queueing Petri Nets. In 19th IEEE/ACM Intl. Symp. on Modeling, Analysis and

Simulation of Computer and Telecommunication Systems (MASCOTS), Singapore, July 2011. [ bib | .pdf ]




http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/MeKoKo2011-MASCOTS-PCMtoQPN.pdf

http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/MeKoKo2011-MASCOTS-PCMtoQPN.pdf

31

Overview of Applied Modeling Techniques

Workload Forecasting

AR(I)MA

Extended exp.

smoothing

tBATS

Croston‟smethod

Cubic smoothing

splines

Neural network-based

Resource Demand

Estimation

Regression-based

techniques

Kalman

filter

Nonlinear optimization

Maximum likelihood estimation

Independent component

analysis

Regression Analysis

MARS

CART

M5 trees

Cubist forests

Quantileregression

forests

Support vector

machines

• OMG Meta Object Facility (MOF)

• MOF-based meta-models

• (UML MARTE)

• (UML SPT)

Descriptive Architecture-level Models

• Bounding techniques

• Operational analysis

• Statistical regression models

• Stochastic process algebras

• (Extended) queueing networks

• Layered queueing networks

• Queueing Petri nets

• Machine learning-based models

• Detailed simulation models

Predictive Performance Models



32

Case Study 1: Workload Forecasting

history now near future

work

load

inte

nsity

N. R. Herbst, N. Huber, S. Kounev, and E. Amrehn. Self-Adaptive Workload Classification

and Forecasting for Proactive Resource Provisioning. In Proceedings of the

4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013),

Prague, Czech Republic, April 21-24. 2013. [ bib | slides | .pdf ]

Use multiple alternative forecasting

methods in parallel

Select which method to trust based on its

accuracy in recent time horizons


Goals & ApproachMotivation OutlookCASE STUDIES





33

Time Series Analysis

[BFAST]



34

Forecasting Methods Used

Basic Methods (initial)

Naïve, Moving Averages, Random Walk

Estimation and Modelling of Seasonal Pattern (complex)

Extended Exponential Smoothing (ETS) [Hynd08, Hyn08]

ARIMA framework with automatic model selection [Box08, Hynd08]

tBATS for complex seasonal patterns [Live11]

Trend Interpolation (fast)

Simple Exponential Smoothing (SES) [Hynd08]

Cubic Smoothing Splines [Hynd02]

Croston„s method for intermittent time series [Shen05]

Autoregressive Moving Averages (ARMA11) [Box08]



35

WCF Technique

Self-Adaptive Workload Classification & Forecasting

Overhead Constraint

Forecast Frequency

Forecast Horizon

ConfidenceLevel

Forecast Objectives

(I) Select Method

Forecast Objectives

WorkloadIntensity

Trace

INITIAL

Naïve,

Smoothing,

Random

Walk

Forecast MethodsOverhead Groups

(II) Forecast

& Evaluate

Accuracy

Accuracy

Feedback

FAST

Trend

Inter-

polation

COMPLEX

Decom-

position &

Seasonal

Patterns



36

Workload Intensity Characterization

Workload Intensity Trace:

Time Series of Request Arrival Rates

Length

GradientMono-tonicity

Positivity

Variance

Frequency

QuartileDispersion

Burstiness

High level data analysis to gain information on:

• Noise level & occurrences of unpredictable bursts

• Influence of trends and seasonal patterns

Data

Attributes

Metrics



37

Decision Tree for Classification



38

Process of Classification & Forecasting

classification:

(I) select strategies

forecast

execution &

result output

forecast horizon/

frequency

(need not to be equal)

classification:

(II) evaluate selection

…

forecast

execution &

result output

new forecast

horizon

new classification:

(I) select strategies

forecast

execution &

result output

classification frequency (# forecast executions)

forecast

execution &

result output

forecast

execution &

result output

forecast

execution &

result output



39

Forecast Accuracy Improvement

• Real-world workload intensity trace: IBM CICS transactions on System z

• Comparison of WCF approach to ETS forecast



40

Forecast Accuracy Improvement

WCF

ETS

Naïve

WC

F |

ETS

| N

aive

percentage error



41

Case Study:

Example for Using Forecast Results

• Scenario: Additional server instances at certain thresholds, 3 weeks

• Real-world workload intensity trace (Wikipedia DE page requests per hour)



42

Proactive Resource Provisioning

Resource provisioning:

(I) Without forecasting (solely reactive):

Resource provisioning actions triggered by

76 SLA violations

(II) Using WCF approach (proactive):

Reduction to 34 or less SLA violations

No significant change in resource usage observed

(server instances per hour)



43

Case Study 2: Automated Derivation of

Statistical Regression Models

Q. Noorshams, D. Bruhn, S. Kounev, and R. Reussner. Predictive Performance Modeling of Virtualized

Storage Systems using Optimized Statistical Regression Techniques. In Proc. of the 4th ACM/SPEC

International Conference on Performance Engineering, Prague, Czech Republic, ICPE '13, pages 283-

294. ACM. 2013. [ bib | DOI | http | .pdf ]

1. Systematic

Experiments

2. Regression

Optimization

3. Regression

Model Derivation

Measurement Points

Optimal Parameters of

Techniques

Measurement Points

Action Information Flow




http://dx.doi.org/10.1145/2479871.2479910

http://doi.acm.org/10.1145/2479871.2479910



44

Example Statistical Regression Models

LRM

MARS

CART

M5

Cubist

Piecewiselinearity

TreestructureStep function

CombinationBoosting

Inst.-based

Figure 2: Relat ion between the Regression Techniques

3.5 Cubist ForestsCubist forests [26, 18] are an extension of M5 trees. Thus,Cubist forests are rule-based model trees with linear models

in the leaves. Compared to M5, Cubist introduces twoextensions. First, it follows a boosting-like approach, i.e.,

it creates a set of trees instead of a single tree. To obtaina single value, the tree predictions are aggregated using

their arithmetic mean. Second, it combines model-basedand instance-based learning (cf. [24]), i.e., it can adjust theprediction of unseen points by the values of their nearest

t raining points.Model Derivation. Initially, themaximum number of trees

in Cubist is defined to construct a forest. The first tree iscreated using the M5 algorithm. The following trees are

created to correct thepredictionsof the training pointsbytheprevious tree f t (x). Each valueof a training point yi is

modified by yi := 2yi − f t (x) to compensate for over- andunder-estimations. Then, the treecreation is repeated. In

contrast to, e.g., RandomForests[4] combiningthepredictiontreeswith themodeoperator, Cubist aggregates thevalues

predicted by each treeusing arithmetic mean. Finally, theprediction of unseen points can beadjusted by thevalues of

a possibly dynamically determined number of training points(so-called instance-based correction), cf. [24]. Theprediction

of a new point x is adjusted by the weighted mean of thenearest training points (so-called neighbors) with weight

wn := 1/ (m(x,n) + 0.5) for every neighbor n, wherem(x,n)is theManhatten distanceof x and n. M5 isa special case

of Cubist with one t ree and no instance-based correct ion.

Parameters.

− nspl i t s: Maximum number of splits in the forward step.

− nt r ees : Number of t rees.

− ni nst ances : Size of instance-based correct ion.

3.6 SummaryFigure 2 shows an overview of the considered regression

techniques illustrating the relationships among them. Asdescribed above, the MARS algorithm can be seen as an

extension of LRM by allowing piecewise linear models. How-ever, MARS regulates the number of linear terms. While

MARSand CART seem relatively different, the forward stepof MARScan betransformed into theoneof CART by using

a tree-based structure with step functions, cf. [12]. M5 inturn can beseen asa combination of LRM and CART. How-ever, M5 differs from CART in thecomplexity criterion and

thepruning procedure. Finally, Cubist extends M5 by intro-ducing a boosting likeschemecreating several trees that are

aggregated by their mean. Furthermore, Cubist introducesan instance-based correction to include the training data in

the predict ion of unseen data.

IBM System z

IBM DS8700

CPU, RAM

Processors,

Memory

PR/ SM (Hypervisor)

z/ VM (Hypervisor)

z/ Linuxz/ OS

z/ Linux

LPAR1 LPAR2

RAID ArraysSSD/

HDD

Storage ServerVC

NVC

Fibre

Channel

SwitchedFibre Channel

Figure 3: IBM System z and IBM DS8700

4. METHODOLOGYIn our approach, weapply statistical regression techniques

to createperformancemodels based on systematic measure-ments. In this section, wepresent our experimental environ-

ment and setup as well as our measurement methodologyand performance modeling approach.

4.1 System Under StudyA typical virtualized environment in a data center consistsof servers providing computational resources connected to

a set of storage systems. Such storage systems typicallydiffer significantly from traditional hard disks and native

storage systems due to the complexity of modern storagevirtualizat ion plat forms.

In thispaper, weconsider a representativevirtualized en-vironment based on the IBM mainframeSystem z and the

storagesystemDS8700. They arestate-of-the-art high-perfor-mancevirtualized systems with redundant or hot swappable

resources for high availability. The System z combined withtheDS8700 represent a typical virtualized environment that

can beused asabuildingblock of cloud computing infrastruc-tures. It supportson-demand elasticity of pooled resources

with a pay-per-use accounting system (cf. [22]). The Sys-tem z provides processors and memory, whereas theDS8700provides storagespace. Thestructureof this environment is

illust rated in Figure 3.TheProcessor Resource/ System Manager (PR/ SM) is a

hypervisor managing logical partitions (LPARs) of thema-chine and enabling CPU and storage virtualization. For

memory virtualization and administration purposes, IBMintroducesanother hypervisor, z/ VM. TheSystemzsupports

theclassical mainframeoperating system z/ OS and specialLinux ports for System z commonly denoted as z/ Linux.

The System z isconnected to theDS8700 via fibrechannel.Storage requests are handled by a storage server having a

volatile cache (VC) and a non-volatile cache (NVC). Thestorageserver isconnected via switched fibrechannel with

SSD or HDD RAID arrays. Asexplained in [8], thestorageserver applies several pre-fetching and destaging algorithms

for optimal performance. When possible, read-requestsareserved from the volatile cache, otherwise they are served

from the RAID arrays and stored in the volatile cache forfuture requests. Write-requests arewritten to thevolatileas

well as thenon-volatilecache, but they aredestaged to theRAID arrays asynchronously.

In such a virtualized storage environment, a wide vari-

ety of heterogeneousperformance-influencing factors exists,cf. [23]. In many cases, the factorshavea significantly differ-

ent effect compared to traditional nativestoragesystems. AsFigure1b illustrates, e.g., thestandard Linux I/ O scheduler

CFQ performs significantly worse than theNOOP scheduler

Simulta

neous Requests

0

20

40

60

80

100

Request Size (KB)0

20

40

60

Response

Tim

e (m

s) 5

10

15

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Simulta

neous Requests

0

20

40

60

80

100

Request Size (KB)0

20

40

60

Respon

se T

ime (m

s) 5

10

15

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Simultaneous Requests

0

20

40

60

80

100

Request Size (KB)0

20

40

60

Re

spon

se T

ime (m

s)

0

5

10

15

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Simulta

neous Requests

0

20

40

60

80

100

Request Size (KB)0

20

40

60

Resp

onse T

ime (m

s)

0

5

10

15

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

Simulta

neous Requests

0

20

40

60

80

100

Request Size (KB)0

20

40

60

Response T

ime (m

s)

0

5

10

15

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

f (x1, x2 ) = c1 x1x2 +c2 x1 +c3 x2 +c4

interaction term

LRM - Linear Regression Models

MARS - Multivariate Adaptive Regression Splines

CART - Classification and Regression Trees

Parameters: # of terms, … Parameters: # of nodes, …

Parameters: # of nodes, …Parameters: # of trees,

# of nodes, …



45

F. Brosig, N. Huber, and S. Kounev. Automated Extraction of Architecture-Level Performance Models

of Distributed Component-Based Systems. In 26th IEEE/ACM International Conference On Automated

Software Engineering (ASE 2011), Oread, Lawrence, Kansas. November 2011. [ bib | .pdf ]



Server 1

AppServer AAppServer N

Server 2

Experimental environment at KIT High-level architecture model overview

20 AppServer nodes (20 x 8 core)

Database server: 24 CPU cores

28 software components

63 behavior specifications

Case Study 3: Automated Extraction of Architecture-level Models




46

Integration of online performance models into virtualization

platforms and virtual appliances

Model-based performance and resource management

Project with VMware



47

Case Study 4: End-to-End Proof-of-Concept

N. Huber, F. Brosig, and S. Kounev. Model-based Self-Adaptive Resource Allocation in Virtualized

Environments. In 6th International Symposium on Software Engineering for Adaptive and Self-Managing

Systems (SEAMS 2011), Waikiki, Honolulu, HI, USA. May 23-24, 2011. [ bib | http | .pdf ]

N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. Modeling Run-Time Adaptation at the

System Architecture Level in Dynamic Service-Oriented Environments. Service Oriented Computing

and Applications (SOCA), 2013, Springer. In print. [ bib | .pdf ]










48

Modeling Run-time Adaptation Policies

Separate

Logical view, high-level

process

Technical view, low-level

operations

StrategyX

TacticA

Action1

reconfigure

execute

use

trigger / guide

Events / Objectives

System Model /Real System

StrategyY

TacticB TacticC

Action2

Action3

Action4ActionN

[ICEBE 2012, SOCA 2013]



49

Actions refer to configuration points

Tactics execute Actions in Reconfiguration Plans

Strategies use weighted Tactics

S/T/A Meta-Model

Event

Strategy

description : String

OverallGoal

specification : String

Objective

Repository

weight : Double

WeightedTactic

Tactic ReconfigurationPlan

name : String

type : Type

Parameter

AbstractControlFlowElement

Action ActionReference

Start StopiterationCount : Integer

Loop

condition : String

Branch

1..*

hasObjectivestriggeringEvents

0..1

objective

1

tactics

1..*

uses1

implemented

Plan

1

strategies

1..*

tactics

1..*

steps

0..*

successor0..1

predecessor0..1

parameters

0..*

actions

1..*

outputParam0..1

inputParams0..*

outputParam0..1

inputParams0..*

referredAction

1

branches1..2

body1

Action

Tactic

StrategyWeightingFunction

1



50

Further Topics

Modeling virtualization performance overhead (CLOSER„11,

MASCOTS„13)

Performance isolation in shared execution environments with

focus on multi-tenant SaaS (QoSA‟12, SciCo‟13)

Modeling of virtualized storage systems using optimized

statistical regression techniques (ICPE‟13, MASCOTS‟ 13)

Cloud usage patterns: A formalism for description of cloud

usage scenarios (SPEC Research Group)

Intrusion detection and prevention in virtualized environments

(SPEC Research Group)


Goals & ApproachMotivation OUTLOOKCase Studies

51

Elastic Capacity Management / Online Workload Forecasting

N. Herbst, S. Kounev, and R. Reussner. Elasticity in Cloud Computing: What it is, and What it is Not. In Proc. of the 10th Intl. Conference

on Autonomic Computing (ICAC 2013), San Jose, CA, June 24-28. USENIX. 2013. [ bib | slides | .pdf ]

N. Herbst, N. Huber, S. Kounev, and E. Amrehn. Self-Adaptive Workload Classification and Forecasting for Proactive Resource

Provisioning. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013), Prague, Czech

Republic, April 21-24. 2013. [ bib | slides | .pdf ]

N. Huber, F. Brosig, and S. Kounev. Model-based Self-Adaptive Resource Allocation in Virtualized Environments. In 6th International

Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2011), Waikiki, Honolulu, HI, USA. May 23-24,

2011. [ bib | http | .pdf ]

Automatic Model Extraction based on Benchmarking or Online System Monitoring

Q. Noorshams, D. Bruhn, S. Kounev, and R. Reussner. Predictive Performance Modeling of Virtualized Storage Systems using Optimized

Statistical Regression Techniques. In Proc. of the 4th ACM/SPEC International Conference on Performance Engineering, Prague, Czech

Republic, ICPE '13, pages 283-294, New York, NY, USA. ACM. 2013. [ bib | DOI | http | .pdf ]

F. Brosig, N. Huber, and S. Kounev. Automated Extraction of Architecture-Level Performance Models of Distributed Component-Based

Systems. In 26th IEEE/ACM International Conference On Automated Software Engineering (ASE 2011), Oread, Lawrence, Kansas.

November 2011. [ bib | .pdf ]

S. Kounev, K. Bender, F. Brosig, N. Huber, and R. Okamoto. Automated Simulation-Based Capacity Planning for Enterprise Data Fabrics.

In 4th International ICST Conference on Simulation Tools and Techniques (SIMUTools 2011), Barcelona, Spain. March 21-25, 2011.

Best Paper Award. [ bib | slides | .pdf ]

Q. Noorshams, K. Rostami, S. Kounev, P. Tůma, and R. Reussner. I/O Performance Modeling of Virtualized Storage Systems. In

Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication


Performance Modeling and Prediction in Virtualized Environments

F. Brosig, F. Gorsler, N. Huber, and S. Kounev. Evaluating Approaches for Performance Prediction in Virtualized Environments.

In Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication


N. Huber, M. von Quast, M. Hauck, and S. Kounev. Evaluating and Modeling Virtualization Performance Overhead for Cloud

Environments. In Proceedings of the International Conference on Cloud Computing and Services Science (CLOSER 2011),

Noordwijkerhout, The Netherlands, pages 563 - 573. SciTePress. May 7-9, 2011. Best Paper Award. [ bib | http | .pdf ]

References



http://sdqweb.ipd.kit.edu/publications/pdfs/HeKoRe2013-ICAC-Elasticity_Slides.pdf












http://dx.doi.org/10.1145/2479871.2479910

http://doi.acm.org/10.1145/2479871.2479910







http://sdqweb.ipd.kit.edu/publications/descartes-pdfs/KoBeBrHuOk2011-ICST-DataFabrics_Slides.pdf













52

References

Descartes Meta-Model (DMM) - Online Models for Architecting Self-Aware Systems

http://www.descartes-research.net/research_and_profile/descartes_meta_model/

F. Brosig, N. Huber, and S. Kounev. Architecture-Level Software Performance Abstractions for Online Performance Prediction. Elsevier

Science of Computer Programming Journal (SciCo), 2013. [ bib | DOI | .pdf ]

N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. Modeling Run-Time Adaptation at the System Architecture Level in Dynamic

Service-Oriented Environments. Service Oriented Computing and Applications (SOCA), 2013, Springer. In print. [ bib | .pdf ]

N. Huber, A. van Hoorn, A. Koziolek, F. Brosig, and S. Kounev. S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System

Architectures. In 9th IEEE Intl. Conf. on e-Business Engineering (ICEBE 2012), Hangzhou, China, September 9-11, 2012. [ bib | http | .pdf ]

F. Brosig, N. Huber, and S. Kounev. Modeling Parameter and Context Dependencies in Online Architecture-Level Performance Models. In

Proc. of the 15th ACM SIGSOFT Intl. Symposium on Component Based Software Engineering (CBSE 2012), June 26-28, 2012. [ bib | .pdf ]

N. Huber, F. Brosig, and S. Kounev. Modeling Dynamic Virtualized Resource Landscapes. In Proceedings of the 8th ACM SIGSOFT

International Conference on the Quality of Software Architectures (QoSA 2012), Bertinoro, Italy, June 25-28, 2012. [ bib| .pdf ]

S. Kounev, F. Brosig, and N. Huber. Descartes Meta-Model (DMM). Technical report, Karlsruhe Institute of Technology (KIT), 2013. To

appear. [ bib | http ]

Vision of Self-Aware Computing Systems



S. Kounev. Engineering of Self-Aware IT Systems and Services: State-of-the-Art and Research Challenges. In Proc. of the 8th European

Performance Engineering Workshop (EPEW'11), Borrowdale, The English Lake District, October 12-13. 2011. (Keynote Talk). [ bib | .pdf ]

S. Kounev. Self-Aware Software and Systems Engineering: A Vision and Research Roadmap. In GI Softwaretechnik-Trends, 31(4),

November 2011, ISSN 0720-8928, Karlsruhe, Germany, 2011. [ bib | .html | .pdf ]

S. Kounev, F. Brosig, and N. Huber. Towards self-aware performance and resource management in modern service-oriented systems. In

Proc. of the 7th IEEE Intl. Conference on Services Computing (SCC 2010), July 5-10, Miami, Florida, USA. IEEE, 2010. [ bib | .pdf ]


http://sdqweb.ipd.kit.edu/publications/article_brosig_bib.html








http://conferences.computer.org/icebe/2012/index.htm










http://www.descartes-research.net/metamodel/











53

Cloud Usage Scenarios, Challenges and Opportunities

A. Milenkoski, A. Iosup, S. Kounev, K. Sachs, P. Rygielski, J. Ding, W. Cirne, and F. Rosenberg. Cloud Usage Patterns: A Formalism for

Description of Cloud Usage Scenarios. Technical Report SPEC-RG-2013-001 v.1.0.1, SPEC Research Group - Cloud Working Group,

Standard Performance Evaluation Corporation (SPEC), April 2013. [ bib | .pdf ]

S. Kounev, P. Reinecke, F. Brosig, J. T. Bradley, K. Joshi, V. Babka, A. Stefanek, and S. Gilmore. Providing dependability and resilience

in the cloud: Challenges and opportunities. In K. Wolter, A. Avritzer, M. Vieira, and A. van Moorsel, editors, Resilience Assessment and

Evaluation of Computing Systems, XVIII. Springer-Verlag, Berlin, Heidelberg, 2012. ISBN: 978-3-642-29031-2. [ bib | http | .pdf]

Performance Isolation in Shared Execution Environments (e.g., Multi-Tenant SaaS)

R. Krebs, C. Momm, and S. Kounev. Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments. Elsevier

Science of Computer Programming Journal (SciCo), 2013. To appear. [ bib ]

R. Krebs, C. Momm, and S. Kounev. Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments. In Barbora

Buhnova and Antonio Vallecillo, editors, Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software

Architectures (QoSA 2012), Bertinoro, Italy, pages 91-100, New York, USA. ACM Press. June 25-28, 2012. [ bib | http | .pdf ]

Intrusion Detection and Prevention in Virtualized Environments

A. Milenkoski, S. Kounev, A. Avritzer, N. Antunes, and M. Vieira. On Benchmarking Intrusion Detection Systems in Virtualized

Environments. Technical Report SPEC-RG-2013-002 v.1.0, SPEC Research Group - IDS Benchmarking Working Group, Standard

Performance Evaluation Corporation (SPEC), June 2013. [ bib | .pdf ]

References






http://www.springer.com/computer/communication+networks/book/978-3-642-29031-2





http://qosa.ipd.kit.edu/qosa_2012/






54

Thank You!

http://www.descartes-research.net

http://www.linkedin.com/groups/SelfAware-Computing-5103054

Dagstuhl Seminar “Model-driven Algorithms and Architectures for Self-Aware Computing

Systems” (2014) co-organized by S. Kounev (KIT), J. Kephart (IBM),

M. Kwiatkowska (Oxford) and X. Zhu (VMware).


Documents

Online Model-Based Performance and Resource … · Towards self-aware performance and resource management in modern service-oriented systems. In ... Deployment topology, resource