Upload
sherman-nelson
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Software Architecture based Software Architecture based Performance and Reliability Performance and Reliability EvaluationEvaluation
Vibhu S. SharmaVibhu S. Sharma
Ph.D. Scholar Ph.D. Scholar
CSE, IITkCSE, IITk
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Software architecture is the fundamental Software architecture is the fundamental organizationorganization of a system, embodied in its of a system, embodied in its componentscomponents, their , their relationshipsrelationships to each other and to each other and the the environmentenvironment, and the principles governing its , and the principles governing its design and evolution. - design and evolution. - IEEE 1471-2000.IEEE 1471-2000.
Software architecture styles are like templates Software architecture styles are like templates which are followed by commonly used systems:which are followed by commonly used systems:• Layered style : Web-based systemsLayered style : Web-based systems• Pipe and Filter style : Compilers Pipe and Filter style : Compilers • . . .. . .
Software ArchitectureSoftware Architecture
Client 1
Web-server
Application Server
Client 2
Client nc
A web-based software system
DB
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Modeling the workflow of the Software Modeling the workflow of the Software ArchitectureArchitecture
We use finite absorbing Discrete Time Markov We use finite absorbing Discrete Time Markov Chains (DTMC) for representing the software Chains (DTMC) for representing the software architecture workflow.architecture workflow.
Examples:Examples:
These DTMCs can be used to find the average These DTMCs can be used to find the average visit counts to various components for a typical visit counts to various components for a typical request as well as the limiting probabilities.request as well as the limiting probabilities.
Very useful for reliability as well as performance Very useful for reliability as well as performance prediction.prediction.
1 2 3 Cp12 p23 1
1-p12
1-p23
ESA Antenna array configuration tool architecture
S0
S1 S2 Snl-1 Snl
S2nl+1
1
1p12 p23 p(nl-2)(nl-1) p(nl-1)nl
. . . . . . . . . .
Snl+1 Snl+2S2nl-1 S2nl
11111
. . . . . . . . . .
1- p(nl-1)nl1-p231-p12
1A typical layered architecture
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Architecture based Reliability predictionArchitecture based Reliability prediction Issues/Questions:Issues/Questions:
• What is the reliability of a software system What is the reliability of a software system composed of unreliable components?composed of unreliable components?
• How does increasing the reliability of a How does increasing the reliability of a particular component affects overall particular component affects overall reliability ?reliability ?
• How does changing the architecture affect How does changing the architecture affect reliability ?reliability ?
Reliability modeling using DTMCsReliability modeling using DTMCs • Model the component failure as a state in Model the component failure as a state in
the DTMC and associate probabilities of the DTMC and associate probabilities of transition (unreliabilities). transition (unreliabilities).
• Overall reliability is the probability of Overall reliability is the probability of ending in the correct state.ending in the correct state.
• Another option is using a reward based Another option is using a reward based method method
1
23
4
5
S
F
pf1
pf2
pf3
pf4
pf5
(1-pf1)p12
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Architecture based performance Architecture based performance predictionprediction
Issues/Questions:Issues/Questions:• What is the maximum number of clients the
system can able to handle before it saturates ?• What effect does varying the number of clients
have on the throughput and the average response time ?
• How should software components be allocated to the hardware nodes ?
• What changes should be made in the system to improve performance ?
We take an example of layered software We take an example of layered software architecture.architecture.• A Discrete Time Markov Chain (DTMC) is used
to characterize the control flow in the layered software architecture.
• The DTMC is used to calculate the visit counts to various layers and then a closed queueing network (QN) model is constructed and solved.
Client 1
Layer 1 Layer 2 Layer nl Client 2
Client nc The Software System
S0
S1 S2 Snl-
1
Snl
S2nl+
1
1
1 p12 p23 p(nl-2)(nl-1) p(nl-1)nl
. . . . . .
. . . .
Snl+
1
Snl+
2
S2nl-
1
S2nl
11111
. . . . . .
. . . .
1- p(nl-1)nl1-p231-p12
1
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Some Analysis ResultsSome Analysis Results
Average response time and throughput and bottleneck analysis
Effect of change in individual reliabilities & workload on overall reliability
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Reliability and Performance TradeoffReliability and Performance Tradeoff
Given a set of components and a set of machines to deploy these onto• What are the performance & reliability characteristics of the setup
under a given load ?• Which architecture should be chosen for optimal performance and
reliability ?• What is the tradeoff ?
Software failures of different types cause different effects on the performance of the system.
Hardware availability also needs to be taken into account.Hardware availability also needs to be taken into account. Some configurations will result in high throughput with Some configurations will result in high throughput with
decreased reliability and vice-versa.decreased reliability and vice-versa.
Combining Static Analysis and Combining Static Analysis and Testing Frameworks Testing Frameworks
to find Software Defectsto find Software Defects
Vipindeep VVipindeep V
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Defect Prevention during Software Defect Prevention during Software Development Development
Write Code Compile/Build Customer
Evaluate
ProgramSpecification
Static analysis Test Suite
Testing
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Testing + Static Analysis – How?Testing + Static Analysis – How? Static Analysis Vs TestingStatic Analysis Vs Testing
• Different areas altogether, but motive – identify defectsDifferent areas altogether, but motive – identify defects• Combining the approaches… improved program checkingCombining the approaches… improved program checking
Motivations for the Proposed ApproachMotivations for the Proposed Approach• Static analysis concernsStatic analysis concerns• Testing is done.. Bugs are revealedTesting is done.. Bugs are revealed• Ideal test suite => identify all bugsIdeal test suite => identify all bugs• Non-ideal scenario of TestingNon-ideal scenario of Testing
Identify defects, partiallyIdentify defects, partially Some paths/blocks are “not (properly) tested” .. Why??Some paths/blocks are “not (properly) tested” .. Why??
• Static analysis effort on unexplored parts of the codeStatic analysis effort on unexplored parts of the code
Our proposed approach Our proposed approach • Focused Static Analysis with Path Pruning using Coverage Focused Static Analysis with Path Pruning using Coverage
Data Data
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Overview of the approachOverview of the approach
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Pros and ConsPros and Cons
• Advantages:Advantages:1.1. New real errors can be identified which escaped New real errors can be identified which escaped
in _all_paths analysisin _all_paths analysis
2.2. Rigorous testing effort counted for better static Rigorous testing effort counted for better static analysisanalysis
3.3. Reduce execution timeReduce execution time
4.4. Reduced warnings => Reduced NoiseReduced warnings => Reduced Noise
• Disadvantages:Disadvantages:1.1. Some errors escape in the pruned path analysisSome errors escape in the pruned path analysis
2.2. Depends on testing frameworkDepends on testing framework
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Applied the approach with FindBugs, Jlint for JAVAApplied the approach with FindBugs, Jlint for JAVA Results from testing were encouragingResults from testing were encouraging Tested the applicability using Random Path Selection Tested the applicability using Random Path Selection
% of Blocks Pruned VS Percentage of new errors
0
2
4
6
8
10
12
14
0 5 10 15 20 25 30 35 40 45 50
% of Blocks Pruned
% o
f n
ew e
rro
rs
Series1
Preliminary ResultsPreliminary Results
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
ConclusionsConclusions Works well in practiceWorks well in practice
• Finds enough real defects to be usefulFinds enough real defects to be useful• Noise is low enough that people can use itNoise is low enough that people can use it• Scales well, so works on large very large code Scales well, so works on large very large code
basesbases More issues to explore..More issues to explore..
• Better algorithm to use testing dataBetter algorithm to use testing data• Path profiles, Crash dumps – focused static analysisPath profiles, Crash dumps – focused static analysis• Use static analysis results for focusing the testing Use static analysis results for focusing the testing
efforteffort• Combining static and dynamic analysisCombining static and dynamic analysis• Usability related issuesUsability related issues
State Based Testing State Based Testing AutomationAutomation
Atul GuptaAtul Gupta
Ph.D. ScholarPh.D. Scholar
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
State Based TestingState Based Testing
Program behavior can be modeled as a Program behavior can be modeled as a state graphstate graph
A formal testing approach at all levels-unit, A formal testing approach at all levels-unit, component or system level. component or system level.
Effective testing strategy to O-O softwareEffective testing strategy to O-O software Test Cases and Test Oracles can be Test Cases and Test Oracles can be
automatically generated and evaluatedautomatically generated and evaluated
Atul Gupta PhD Scholar
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Class CoinBox {unsigned totalQtrs;unsigned curQtrs;unsigned allowVend;
public:CoinBox( ) {
totalQtrs = 0;curQtrs = 0;allowVend = 0;
}void retQtrs( ) {
curQtrs = 0;}void addQtr( ) {
curQtrs = curQtrs +1;if (curQtrs > 1)allowVend = 1;
}void vend( ) {
if (allowVend) {totalQtrs = totalQtrs + curQtrs;curQtrs = 0;allowVend = 0;}
}};
An Example : A CoinBox ClassAn Example : A CoinBox Class
WAIT Q1 Q2
allowVend = 0curQtrs = 0
allowVend = 0curQtrs = 1
allowVend = 1curQtrs >= 2
addQtr( ) addQtr( )addQtr( )
retQtrs( )
retQtrs( )
vend( )
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
State Based Testing CriteriaState Based Testing Criteria
All-Transition Coverage (AT)All-Transition Coverage (AT) All-Transition-pair Coverage (ATP)All-Transition-pair Coverage (ATP) Full Predicate coverage (FP)Full Predicate coverage (FP) Transition-Tree Coverage (TT)Transition-Tree Coverage (TT) All-Round-Trip path coverage (ART)All-Round-Trip path coverage (ART) Complete Sequence (CSComplete Sequence (CS))
WAIT Q1 Q2
allowVend = 0curQtrs = 0
allowVend = 0curQtrs = 1
allowVend = 1curQtrs >= 2
addQtr( ) addQtr( )addQtr( )
retQtrs( )
retQtrs( )
vend( )
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Generating Test Generating Test SequencesSequences
CriterionCriterion PrefixPrefix(State)(State)
Test SequenceTest Sequence Test OracleTest Oracle(State)(State)
All-Transition (AT):All-Transition (AT): WAITWAITQ1Q1Q2Q2
addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )-addQtr()- retQtrs( )addQtr( )-addQtr()- retQtrs( )vend( )vend( )
WAITWAITWAITWAITWAITWAIT
All-Transition-Pair All-Transition-Pair (ATP)(ATP)
At Q1At Q1 WAITWAITWAITWAIT
addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )
WAITWAITQ2Q2
At Q2At Q2 Q1Q1Q1Q1Q1Q1Q2Q2Q2Q2Q2Q2
addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )addQtr( )- vend( )addQtr( )- vend( )addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )addQtr( )- addQtr( )addQtr( )- vend( )addQtr( )- vend( )
WAITWAITQ2Q2WAITWAITWAITWAITQ2Q2WAITWAIT
At WAITAt WAIT Q2Q2Q1Q1Q2Q2
retQtrs( )- addQtr( )retQtrs( )- addQtr( )retQtrs( )- addQtr( )retQtrs( )- addQtr( )vend ( )- addQtr( )vend ( )- addQtr( )
Q1Q1Q1Q1Q1Q1
Transition-Tree (TT)Transition-Tree (TT)
WAITWAITWAITWAITWAITWAITWAITWAIT
addQtr( )- retQtrs( )addQtr( )- retQtrs( )addQtr( )- addQtr( )-retQtrs( )addQtr( )- addQtr( )-retQtrs( )addQtr( )- addQtr( )-vend( )addQtr( )- addQtr( )-vend( )addQtr( )- addQtr( )- addQtr( )addQtr( )- addQtr( )- addQtr( )
WAITWAITWAITWAITWAITWAITQ2Q2
WAIT Q1 Q2
allowVend = 0curQtrs = 0
allowVend = 0curQtrs = 1
allowVend = 1curQtrs >= 2
addQtr( ) addQtr( )addQtr( )
retQtrs( )
retQtrs( )
vend( )
WAIT
Q1
Q2
addQtr( )
addQtr( )
addQtr( )
retQtrs( )
retQtrs( )
vend( )
WAIT
WAIT Q2WAIT
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Research IssuesResearch Issues
How to generate test cases How to generate test cases automatically from a state modelautomatically from a state model
How to execute them in an How to execute them in an automated mannerautomated manner
How to evaluate the results How to evaluate the results automaticallyautomatically
Cutting Edge 2005Cutting Edge 2005IIT KanpurIIT Kanpur
Our ApproachOur Approach
Identify State Model of a Class Identify State Model of a Class Generate test sequences based on Generate test sequences based on
some Coverage Criteriasome Coverage Criteria Convert these sequences into JUnit Convert these sequences into JUnit
format for Automatic Execution and format for Automatic Execution and Result evaluationResult evaluation
AIM : Complete end-to-end AutomationAIM : Complete end-to-end Automation