Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
1
The Role of Quality Assurers In Software Assurers In Software
Engineering’s Fourth WaveDr. Bill Curtis
SVP & Chief ScientistCAST Software
© CAST 2008
Rethinking Quality
The degree to which a product meets its specified requirementsQuality
Problem⎯Customers struggle to state functional requirements. They do not understand non-functional requirements.
“…a failure to satisfy a non-functional requirement can be critical, even
t t hi f ti l i t
2© CAST 2008
catastrophic…non-functional requirements are sometimes difficult to verify. We cannot
write a test case to verify a system’s reliability…The ability to associate code to
non-functional properties can be a powerful weapon in a software engineer’s arsenal.”
2
The 4th Wave in Software EngineeringWhat: Architecture, Quality characteristics, ReuseWhen: 2005Why: Ensure software is constructed to standards
that meet the lifetime demands placed on itProduct
4
What: Design methods, CASE toolsWhen: 1980-19902
What: CMM, ITIL, PMBOK, AgileWhen: 1990-2005Why: Provide a more disciplined environment for
professional work incorporating best practicesProcess
3
3© CAST 2008
What: 3rd & 4th generation languages, structured programmingWhen: 1965-1980Why: Give developers greater power for expressing their
programsLanguages
1
Why: Give developers better tools and aids for constructing software systemsMethods
Expanding Technical Diversity in Apps
ASP/JSP/VB/.NET
User Interface Tier
WebServices
Middle-ware
Enterprise Applications
Application Logic TierJava, C++, …
Frameworks Struts MVC, Spring
Legacy Applications
CICS Monitor (Cobol) Tuxedo Monitor (C)
CICS Connector
Data Management Tier
4© CAST 2008
Preventing application level defects requires analysis of all the interactions between components of heterogeneous technologies
DatabasesFiles
COBOLBatch
Shell Scripts
Database
Data Management TierEJB – Hibernate
3
Godot’s Gotta Be in There Somewhere
Product Catalogue
? hosts
Resource pools?Connection pools?
Error handling?Ti t ?
RetailWebsite? hosts
?K threads
Credit Card Application
? hosts? threads
? threads
Order EntryApplication
? hosts? threads
Timeouts?
5© CAST 2008
Express Service Application
? hosts? threads
Shipping Application
? hosts? threads
Ever have an application hang
waiting for a response that will never come?
Even Worse, Dispersed Development
6© CAST 2008
4
What Is the Role for Quality Assurers?
Old answer: “I test and manage releases”
“I assure the process”“I assure the process”
New answer: “I ensure dependable & secure IT services to the business”
7© CAST 2008
“I provide data to quantify risk, the business value of quality, and IT decisions”
Ensure Dependability and Security
National Research CouncilSoftware for Dependable Systems
f“As higher levels of assurance are demanded…testing cannot deliver the level of confidence required at
a reasonable cost.”
“The cost of preventing all failures will usually be prohibitively
expensive, so a dependable system
8© CAST 2008
“The correctness of the code is rarely the weakest link.”
will not offer uniform levels of confidence across all functions.”
Jackson, D. (2009). Communications of the ACM, 52 (4)
5
Application quality measures how well the individual components
Application Quality
Application Quality vs. Code Quality
e t e d dua co po e tswork together to make up the
overall business system
Code quality is the measure of individual components for
li ith t d d
Code Quality
9© CAST 2008
Good code quality ≠ Good application quality
compliance with standards and best practices in the
context of a specific language
Failure to Provide Limits
TIO
N
User Interface Tier Logic Tier Data Tier
AP
PLI
CA
T
User enters No pagination Eager fetch−all Large table
10© CAST 2008
Big problems are often the result several interacting weaknesses in the code, none of which caused the problem by itself
User enters large range
of data
No pagination or cache
mechanism
Eager fetch all records from
dependent tables are returned
Large table yields huge data volume
6
Bypassing Architectural Elements
TIO
NUser Interface Tier Logic Tier Data Tier
AP
PLI
CA
T
11© CAST 2008
Preventing application level defects requires analysis of all the interactions between components of heterogeneous technologies
Passing Test Is Not the Same As Smooth Operations
GUI LOGIC DATA
Queries developed in a loopFunctions used in WHERE clause
AP
PLI
CA
TIO
N
DATA
F
12© CAST 2008
Queries do not use indexes Table size grew by 10X in a month
Coding weaknesses that passed test are exposed as the size of the data or frequency of use grows
7
Stability Anti-PatternsScaling effects
SLA Inversion
Attacks of self denial
Blocked
Usersleads to
exacerbatescounters
Decoupling middleware
reducesimpact
counters
mitigates
and Patterns
Blocked threads
Chain reaction Integration
points
Slow CascadingUnbalanced
found near
leads to
leads to
leads to
mutualaggravation
Bulkheads
counters
Steady state
results from violating
avoids
Test harnessDetects
problems
counters
13© CAST 2008
Slow responses
Cascading failures
Unbounded result sets
Unbalanced capacities
Michael Nygard (2007). Release It!
leads to
leads to
Handshaking
Fail fast
counters
Timeouts
Circuit breaker
counters
can avoid works with
counterscounters
prevents
Presenting Dependability Cases
Provide direct evidence that a system satisfies its dependability requirementsObjective
Dependabilitygoal
Dependabilityclaims
Dependabilityproperties
stated as about
argues provides
14© CAST 2008
Dependabilitycase
Dependabilityevidence
comprises• Auditable• Complete
• Sound
• Tests• Proofs
• Analyses
8
The Business Value of Application Quality
TRANSFERABILITYallows new teams to quickly
begin working with an application
Application Health Factors
Improve software readability
Reduce vendor lock-in
Reduce cost of ownership
Maximize standards compliance
Tactical Objectives
Minimize business risks
Business Value
CHANGEABILITYmakes an application easier and
quicker to modify
ROBUSTNESSimproves application stability & reduces injecting new defects
Reduce learning curves
Ease team handoffs
Reduce application rework
Maximize application availabilityMinimize liquidated damages
Minimize degraded service
Reduce injected defects
Reduce application mistakes
Maximize business agility
Minimize IT costs
15© CAST 2008
PERFORMANCEReduces degraded response
times and increases scalability
SECURITYaffects an application’s ability to prevent unauthorized intrusions
Reduce modification effortAccelerate new function delivery
Maximize application scalability
Maximize speed of response
Maximize information retrieval
Maximize information protection
Maximize customer confidence
Minimize unwanted breaches
Optimize work productivity
Maximize customer loyalty
Anti-patterns and Defects
0.015
0.02025Different Applications in a Bank
rns/
BFP D
efec erns
/BFP D
efec
15
20
25
30
0.40
0.45Application Versions in Telecom
0.000
0.005
0.010
15
20
App A App B App C
Anti-
patte
r cts/BFPAnti-patterns
Defects
0.070.08
10
12
FP
Same Application in a Bank
Anti-patternsD f t 12
14
0 025
0.03
BFP A
Application Versions in System Integrator
Anti-
patte
cts/BFP
0
5
10
15
R4 R5 R6 R70.30
0.35
Release
Anti-patternsDefects
16© CAST 2008
00.010.020.030.040.050.06
0
2
4
6
8
5.3.07 5.7.0.1 5.11.0.7
Defects/B
FP
Anti-
patte
rns/
BF
App C Versions
Defects
0
2
4
6
8
10
0
0.005
0.01
0.015
0.02
0.025
3.2 3.3 3.4 3.6App Versions
Actua
l Defects/B
Anti‐patterns/BFPAnti-patterns
Defects
9
Allianz Austria’s Maintainability Results
Business Need• Reduce maintenance costs• 700,000 LOC insurance mgt. system• 10 million customers and processes
SolutionStatically analyze application quality and stabilize maintainability scoreProvide reports to help development teams remediate problems faster
12LOC
10 million customers and processes • 8 million claims a year
Business Benefit
17© CAST 2008
0
2
4
6
8
10
Def
ects
/ K
L2002 2003 2004 2005 2006
56% reduction in defects in 4 years
• Maintainability stabilized despite 40% increase in code over 4 years
• Reduced delivery time by 60%
• Reduced costs 20% over 3 years
• 56% reduction in defects in 4 years
Deutsche Telecom⎯Managing SuppliersT-Systems Active Billing & Multimedia Solutions
• 120 billion call records and 1 billion invoices per year• SAP, Siebel, all front end apps that power www.tcom.de
Internal quality analysis run one or two times per quarterly release• Penalties in contracts based on internal quality metrics• Aggregation of metrics into C-level management dashboards
Project #nProject #3
Project #2Project #1
150+ app in 4 silos Dashboard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1.1 2.0 2.5
18© CAST 2008
Parser Agent
T-Systems Multimedia Solutions GmbH
Team #1
Team #2
Team #3
Team #4
10
Consortium for IT Software Quality
CISQIT organizations, Outsourcers, Agencies, Experts
IT & AD Executives
Technical experts
19© CAST 2008
Define industry issuesDrive standards adoptionBuild appraiser program
Create quality standardDeveloper certification
Integrate with standards
What Does This Mean for QA Professionals
1. New opportunities as QA role upscales
2. Develop application analysis skills
3. Manage IT risk on behalf of business
4. Drive data collection and analysis
20© CAST 2008
The closer to the business, the stabler the job!