Trace-Context Sensitive Performance Proﬁling for ...oceanrep.geomar.de/14488/2/Sipew-MatthiasRohr.pdf · Enterprise Software Applications Presentation at SPEC International Performance

Trace-Context Sensitive Performance Profiling forEnterprise Software Applications

Presentation at SPEC International Performance Evaluation Workshop 2008

Matthias Rohr1, Andre van Hoorn1, Simon Giesecke2,Jasminka Matevska1, Wilhelm Hasselbring1, and Sergej Alekseev3

1Graduate School TrustSoft, Software Engineering Group, University of Oldenburg, Germany2OFFIS Institute for Information Technology, Oldenburg, Germany

3Nokia Siemens Networks GmbH & Co KG, Berlin, Germany

Contact: rohr(at)informatik.uni-oldenburg.de, http://www.matthias-rohr.com,

http://rohrrohr.googlepages.com/index.html

July 2, 2008

Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 1 / 29

http://www.trustsoft.org

http://www.matthias-rohr.com

http://rohrrohr.googlepages.com/index.html

Foundations

Outline

1 Motivation

2 Foundations

3 Approach to Calling-context Sensitive Timing Behavior Modeling

4 Related work

5 Evaluation

6 Conclusions and Future Work


Foundations

Monitoring

:Bookshop :CRM :Catalog

:Bookshop

<<Component>>

:CRM

<<Component>>

:Catalog

<<Component>>

M

M

MM

M

M

M

M

M

M

M

Monitoring of

Response times: Time between start and end of software operation executionsExecution sequences of operations for each thread

Monitoring tool and trace analysis tool:

Kieker1 [Rohr et al., 2008b]Uses Aspect-oriented Programming (AspectJ)

1http://kieker.sourceforge.net, http://www.matthias-rohr.comMatthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 6 / 29

http://kieker.sourceforge.net

http://www.matthias-rohr.com

Foundations

Monitoring of traces

Operation TraceID tin tout rt = tout − tina 1 000 150 150c 1 030 050 20b 1 060 140 80...

......

......

c 2 340 358 18c 2 400 437 37


Foundations

Monitoring of traces

Operation TraceID tin tout rt = tout − tina 1 000 150 150c 1 030 050 20b 1 060 140 80...

......

......

c 2 340 358 18c 2 400 437 37

Trace reconstruction

Traces are sequences of executions that correspond to system service requests

Sequence diagrams and dynamic call trees represent (classes) of traces


Foundations

Construction of dynamic call trees

Dynamic call trees:

Dynamic call trees [Ammons et al., 1997] represent execution sequences by anordered tree.

A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Foundations


Dynamic call trees:


A

a

C D E F

c

d

e

f

c

d

e

A.a

C.c D.d E.e F.f

C.c D.d E.e


Approach to Calling-context Sensitive Timing Behavior Modeling

Outline

1 Motivation

2 Foundations

3 Approach to Calling-context Sensitive Timing Behavior ModelingCalling-context Equivalence ClassesOptimization of the Trace-context Sensitive Timing Behavior Model

4 Related work

5 Evaluation




Approach – Overview

Goal

Derivation of a timing behavior model with lower variance and less multi-modalityin timing behavior distributions




Goal


Definition: Calling-context of an operation call

Set of circumstances or facts that surround an operation call, in particular thesequence of surrounding operation executions.




Goal


Definition: Calling-context of an operation call

Set of circumstances or facts that surround an operation call, in particular thesequence of surrounding operation executions.

Steps of the approach

1 Integrate calling-context information into timing behavior model

2 Optimize timing behavior model; e.g. model size reduction


Approach to Calling-context Sensitive Timing Behavior Modeling Calling-context Equivalence Classes

Calling-context equivalence

Three levels of abstraction for calling-context information:

Equivalence relations on software operation executions

Two executions of the same operation are

caller-context equivalent (cp. Graham et al. [1982]):= called from operations with the same name.

stack-context equivalent (cp. Ammons et al. [1997]):= equal paths from their tree nodes to root

trace-context equivalent :=1) corresponding trees are equal2) tree nodes have same position in tree



Calling-context equivalence

Three levels of abstraction for calling-context information:

Equivalence relations on software operation executions

Two executions of the same operation are

caller-context equivalent (cp. Graham et al. [1982]):= called from operations with the same name.

stack-context equivalent (cp. Ammons et al. [1997]):= equal paths from their tree nodes to root

trace-context equivalent :=1) corresponding trees are equal2) tree nodes have same position in tree

Stack-context equivalence ⇒ caller-context equivalence

Trace-context equivalence ⇒ stack-context equivalence



Caller-context equivalence (cp. Graham et al. [1982])

Two executions of the same operation are caller-context equivalent if they arecalled from operations with the same name.

A.a

C.c D.d E.e F.f

C.c D.d E.e

A.a

C.c D.d E.e B.b

C.c D.d E.e

B.b F.f

B.b

Dynamic call tree for trace 1 Dynamic call tree for trace 2

Caller−context

equivalence for E.e

Figure: Example: Caller-context equivalence for operation E.e (3 eq. classes)



Stack-context equivalence (cp. Ammons et al. [1997])

Two executions of the same operation are stack-context equivalent if the pathsfrom the corresponding nodes to its root are equal.

A.a

C.c D.d E.e F.f

C.c D.d E.e

A.a

C.c D.d E.e B.b

C.c D.d E.e

B.b F.f

B.b


Stack−context

equivalence for E.e

Figure: Example: Stack-context equivalence for operation E.e (4 eq. classes)



Trace-context equivalence – Example

Two executions of the same operation are trace-context equivalent if thecorresponding trees are equal and the both executions correspond to dynamic calltree nodes with the same position within the tree.

A.a

C.c D.d E.e F.f

C.c D.d E.e

A.a

C.c D.d E.e B.b

C.c D.d E.e

B.b F.f

B.b


Trace−context

equivalence for A.a

Figure: Example: Trace-context equivalence for operation A.a (2 eq. classes)

For both traces together, number of trace-contexts = total number of nodes innon-equal trees = 19.



Calling-Context Sensitive Timing Behavior Model

Monitored response timesof all instrumented

software operations

RT = [ ... ]

Partitioning basedon operation name equality

Partitioning basedon caller−context equivalence

Partitioning basedon stack−context equivalence

Partitioning basedon trace−context equivalence

Calling-context sensitive timing behavior model derived from monitoring data:

Complete partitioning of all response times based on trace-context,stack-context, or caller-context equivalence


Approach to Calling-context Sensitive Timing Behavior Modeling Optimization of the Trace-context Sensitive Timing Behavior Model

Additional Model Optimization using Tree Operators

Possibly new problems resulting from calling-context analysis:

Efficiency:

Too many calling-contextsCalling-contexts do not differ in timing behavior distributions

Applicability and robustness of statistical methods

Calling-contexts with an insufficient number of measurements


Approach to Calling-context Sensitive Timing Behavior Modeling Optimization of the Trace-context Sensitive Timing Behavior Model

Additional Model Optimization using Tree Operators

Possibly new problems resulting from calling-context analysis:

Efficiency:

Too many calling-contextsCalling-contexts do not differ in timing behavior distributions

Applicability and robustness of statistical methods

Calling-contexts with an insufficient number of measurements

Leaf nodes with similar distribution

characteristics are merged

Leaf nodes without a sufficient

amount of observations are

linked to an ancestor node

Leaf nodes without

siblings are removed


Related work

Outline

1 Motivation

2 Foundations


4 Related work

5 Evaluation



Related work

Related work:

Correlating control-flow information and performance measurements:

Graham et al. [1982] introduced the profiler gprof that can evaluatemeasurements in the context of the caller

Ammons et al. [1997] introduced evaluation in the context of the stack ofcallers; Implemented by commerical profilling tools (e.g., Intel’s VTuneProfiler)

Handling clusters in response time distributions:

Bulej et al. [2005] describe “clusters” in timing behavior measurements andapply k-means clustering on response time data


Evaluation

Outline

1 Motivation

2 Foundations


4 Related work

5 EvaluationField StudyCase Study



Evaluation Field Study

Field Study

Setting

System at Nokia Siemens Networks

Commercial software platform for implementing telecommunication networkservices (e.g., signaling)

Workload: Test driver of Nokia Siemens Networks (5,000 - 500,000Calls/Hour)

Goals

Demonstration of applicability in a complex system

Demonstration of resolving multi-modality and standard deviation

reduction



Results100

300

500

700

Calendar time (hour:minute)

Re

sp

on

se

tim

e in

mic

rose

co

nd

s

14:18 14:20 14:22 14:24

(a) Response times measured for operation f .

0.0

00

0.0

04

0.0

08

0 100 200 300 400 500 600

Response time in microseconds

Pro

ba

bili

ty d

en

sity(b) Probability density distribution for opera-tion f .

Figure: “Clusters” and multi-modality for operation f .



Results – Removal of multi-modality

0.0

00

0.0

04

0.0

08

0 100 200 300 400 500 600


Pro

ba

bili

ty d

en

sity


Pro

babili

ty d

ensity

0.0

00.1

00.2

0

65 70 75

(a) PDF for trace-context 2.


Pro

babili

ty d

ensity

0.0

00

0.0

04

0.0

08

0.0

12

200 300 400 500

Trace−context 1

Trace−context 3

(b) PDF for trace-context 1 and 3.



Results – Removal of multi-modality


Pro

babili

ty d

ensity

0.0

00.1

00.2

0

65 70 75

(c) PDF for trace-context 2.


Pro

babili

ty d

ensity

0.0

00

0.0

04

0.0

08

0.0

12

200 300 400 500

Trace−context 1

Trace−context 3

(d) PDF for trace-context 1 and 3.

Total average decrease in standard deviation

136.47 → 53.83 ; 2.20; 49.74 → 35.94 → 73.66% reduction of standard deviation


Evaluation Case Study

Outline

1 Motivation

2 Foundations


4 Related work

5 EvaluationField StudyCase Study




Case Study

Goals and research questions

Quantitative evaluation of standard deviation reduction.

Is trace-context analysis more effective than caller- and stack-contextanalysis?

How does the number of monitoring points relate to the number ofcalling-contexts?

Setting

System: iBATIS JPetStore 5 Web application deployed on Apache Tomcat

Workload:

Probabilistic workload generated by Markov4JMetera[van Hoorn et al., 2008]10 concurrent users

Experiment run statistics:

Random instrumentation: 1-199 of 199 possible monitoring pointsExperiment runs last 18 minutes2–2,032,572 monitored operation executions per run

ahttp://markov4jmeter.sourceforge.net/Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 24 / 29

http://markov4jmeter.sourceforge.net/


Case study: Standard deviation reduction for two particular instrumentations

Average st.dev. decrease in %18 mon.pts. scenario Full instrumentation

Caller-context analysis 0.2 6.8Stack-context analysis 0.6 11.0Trace-context analysis 3.3 42.2

010

20

30

40

Caller Stack Trace

De

cre

ase

of

std

ev in

%

Calling−context type

(e) 18 mon.pts. scenario

CallerContext StackContext TraceContext0

20

40

60

80

De

cre

ase

of

std

ev in

%

Calling context type

(f) Full instrumentation



Case study: Standard deviation reduction for two particular instrumentations

Average st.dev. decrease in %18 mon.pts. scenario Full instrumentation

Caller-context analysis 0.2 6.8Stack-context analysis 0.6 11.0Trace-context analysis 3.3 42.2

010

20

30

40

Caller Stack Trace

De

cre

ase

of

std

ev in

%

Calling−context type

(g) 18 mon.pts. scenario

CallerContext StackContext TraceContext0

20

40

60

80

De

cre

ase

of

std

ev in

%

Calling context type

(h) Full instrumentation

A significant part of the variance is connected to trace-context information!



Case study: Number of monitoring points vs. standard deviation reduction

Trace−context analysisStack−context analysisCaller−context analysis1st and 3rd quartile

25

10

20

50

0 50 100 150 200

Number of monitoring points

Ave

rag

e r

ela

tive

st.

de

v.

de

cre

ase

in

%

Figure: Average decrease in standard deviation for different numbers of monitoringpoints using calling-context information compared to standard deviation using nocalling-context information.



Case study: Number of monitoring points vs. number of calling-contexts

0100

200

300

0 50 100 150 200


Num

ber

of conte

xts

Stack−contexts

Caller−contexts

(a)

02000

4000

6000

8000

0 50 100 150 200


Num

ber

of conte

xts

Trace−contexts

Stack− and Caller−contexts

(b)

The number of stack-contexts and number of caller-contexts both growlinearly with a similar rate (cp. [Rohr et al., 2008a]).

The number of trace-contexts increases faster than the number ofstack-contexts and number of caller-contexts.


Conclusions and Future Work

Summary

Contribution:

New approach to consider trace-context information in timing behavior modelsTrace-context equivalence extends the concepts of caller-context equivalenceand stack-context equivalence

Results of the field study:

Trace-context analysis is applicable in large complex software systemDemonstration of multi-modality removal and standard deviation reduction

Results of the case study:

Significant reduction of standard deviation in timing behavior modelTrace-context analysis is more effective than stack- and caller-context analysisThe number of trace-contexts grows linearly


Conclusions and Future Work

Outlook and Future work

Current limitations / problems:

“Explosion” of the model size is still possible

Multi-modality and high variance may not be an issue in a performanceanalysis

Outlook and possibilities for future work:

Efficiency evaluation: Benefit vs. monitoring costs

Evaluation for execution times (instead response times)

Demonstration of effectiveness in the context of anomaly detection ([Rohret al., 2007])

Considering also asynchronous communication within traces

Acknowledgement

We would like to acknowledge Nokia Siemens Networks Berlin, Business ServiceSolution for supporting this project.


References

G. Ammons, T. Ball, and J. R. Larus. Exploiting hardware performance counterswith flow and context sensitive profiling. In Proceedings of the Conference onProgramming language design and implementation (PLDI’97), pages 85–96.ACM, 1997. ISBN 0-89791-907-6. doi:10.1145/258915.258924.

L. Bulej, T. Kalibera, and P. Tuma. Repeated results analysis for middlewareregression benchmarking. Performance Evaluation, 60(1-4):345–358, 2005.ISSN 0166-5316. doi:10.1016/j.peva.2004.10.013.

S. L. Graham, P. B. Kessler, and M. K. McKusick. gprof: a call graph executionprofiler. SIGPLAN Notes, 17(6):120–126, 1982. doi:10.1145/872726.806987.

M. Rohr, S. Giesecke, and W. Hasselbring. Timing Behavior Anomaly Detection inEnterprise Information Systems. In J. Cardoso, J. Cordeiro, and J. Filipe,editors, Proceedings of the Ninth International Conference on EnterpriseInformation Systems (ICEIS’07), volume DISI, pages 494–497. INSTICC Press,June 2007. ISBN 978-972-8865-88-7.

M. Rohr, A. van Hoorn, S. Giesecke, J. Matevska, and W. Hasselbring.Trace-context sensitive performance models from monitoring data of softwaresystems. In C. Lebsack, editor, Proceedings of the Workshop on ToolsInfrastructures and Methodologies for the Evaluation of Research Systems(TIMERS’08) at IEEE International Symposium on Performance Analysis ofSystems and Software 2008, pages 37–44, Apr. 2008a.


http://dx.doi.org/10.1145/258915.258924

http://dx.doi.org/10.1016/j.peva.2004.10.013

http://dx.doi.org/10.1145/872726.806987

Additional slides

M. Rohr, A. van Hoorn, J. Matevska, N. Sommer, L. Stoever, S. Giesecke, andW. Hasselbring. Kieker: Continuous monitoring and on demand visualization ofJava software behavior. In Proceedings of the IASTED International Conferenceon Software Engineering 2008, pages 80–85. ACTA Press, Feb. 2008b. ISBN978-0-88986-715-4.

A. Sabetta and H. Koziolek. Measuring performance metrics: Techniques andtools. In I. Eusgeld, F. Freiling, and R. Reussner, editors, Dependability Metrics,volume 4909 of Lecture Notes in Computer Science (LNCS). Springer, 2008.ISBN 978-3-540-68946-1.

A. van Hoorn. Workload-sensitive timing behavior anomaly detection in largesoftware systems, Sept. 2007. Master’s thesis (Diplomarbeit), Department ofComputing Science, University of Oldenburg, Germany.

A. van Hoorn, M. Rohr, and W. Hasselbring. Generating probabilistic andintensity-varying workload for web-based software systems. In S. Kounev,I. Gorton, and K. Sachs, editors, Performance Evaluation – Metrics, Models andBenchmarks: Proceedings of the SPEC International Performance EvaluationWorkshop (SIPEW ’08), volume 5119 of Lecture Notes in Computer Science(LNCS), pages 124–143, Heidelberg, June 2008. SPEC, Springer. ISBN978-3-540-69813-5. doi:10.1007/978-3-540-69814-2 9.


http://dx.doi.org/10.1007/978-3-540-69814-2_9

Additional slides

Application scenario: Anomaly Detection

Experiment time in seconds

Re

sp

on

se

tim

e in

mill

ise

co

nd

s

51

02

05

01

00

20

05

00

10

00

0 200 400 600 800 1000 1200 1400 1600 1800

WISAD: Anomaly

WISAD: no anomaly

Period of anomaly injection

Excluded excution


Additional slides

Standard deviation reduction

0 5 10 15 20 25 30

0.0

00.0

50.1

00.1

5

Response time in milliseconds

Pro

babili

ty d

ensity

0.00 0.02 0.04 0.06 0.08 0.10

050

100

150


Pro

babili

ty d

ensity

5 10 15 20 25 30 35 40

0.0

00.0

20.0

4


Pro

babili

ty d

ensity

Quantitative evaluation metric:

Standard deviation of trace-contexts weighted by invokation frequency

compared to original standard deviation: 1 −

P

n

i=1 |TCi |·stdev(TCi )

|Original|·stdev(Original)

Standard variation relates to uncertainty and confidence in statistical analysis

For operation insertOrder() example:

Standard deviation original distribution: 100%

Standard deviation trace-context sensitive model: 61.27% = 38.73% less


Additional slides

Resolving multi-modality 1/3

Operation newOrder() has a multi-modal response time distribution:


Pro

babili

ty d

ensity

0 5 10 15 20 25 30

0.0

00.0

50.1

00.1

5

Figure: Kernel density estimate for all response times of operation newOrder()


Additional slides


Trace-context analysis replaces the original distribution by two uni-modaldistributions:

0 5 10 15 20 25 30

0.0

00.0

50.1

00.1

5Response time in milliseconds

Pro

babili

ty d

ensity

0.00 0.02 0.04 0.06 0.08 0.10

050

100

150


Pro

babili

ty d

ensity

5 10 15 20 25 30 35 40

0.0

00.0

20.0

4


Pro

babili

ty d

ensity


Additional slides


Identified by trace analysis:

Two trace-contexts

One caller-context and one stack-context

The two trace-contexts correspond to two call scenarios:

$ ActionServlet OrderBean

process(..)

newOrder()

(a)

$ ActionServlet OrderBean OrderService

process(..)

newOrder()

insertOrder(..)

(b)

The sequence diagrams only distinguish in the call action to insertOrder().


Additional slides

Calling-Context Tree

Calling-Context Tree

Monitored response timesof all instrumented

software operations

RT = [ ... ]

Partitioning basedon operation name equality

Partitioning basedon caller−context equivalence

Partitioning basedon stack−context equivalence

Partitioning basedon trace−context equivalence

Complete calling-context sensitive timing behavior model

A complete timing behavior model consists of any node subset of the CCT thatcorresponds to a complete partitioning of all monitored observations.


Additional slides

Influences to Software Timing Behavior

System architecture:

Hardware designSoftware design

System usage [cp. Sabetta and Koziolek, 2008]:Workload intensity

Concurrent service requestsNumber of active users [van Hoorn, 2007]

Service demand characteristics

Parameter valuesParameter value sizeCaller identity / stack content / history of calls

System state:

Cache contentLoad balancer stateOther active processes on same platformDatabase content


Additional slides

Nokia Siemens Field Study

Figure: UML Sequence Diagrams from a module of a partially instrumentedtelecommunication signaling system of Nokia Siemens Networks derived from monitoringdata. (Operation names changed, operations omitted).


Additional slides

Nokia Siemens Field Study

Figure: Tree representation of each of the traces.

Pro

babili

ty d

ensity

0.0

00

0.0

04

0.0

08

0.0

12

200 300 400 500

Trace−context 1

Trace−context 3P

robabili

ty d

ensity

0.0

00.1

00.2

0

65 70 75Matthias Rohr et al. (TrustSoft, Univ. Oldenburg) Trace-Context Sensitive Performance Profiling June 27, 2008, Darmstadt Germany 29 / 29

Documents

Trace-Context Sensitive Performance Proﬁling for ...oceanrep.geomar.de/14488/2/Sipew-MatthiasRohr.pdf · Enterprise Software Applications Presentation at SPEC International Performance