Embed Size (px)
Citation preview
Conventional & Unconventional Dimensions of Metrics in an
Outsourced Engagement
September 21st, 2007
Dr Om Prakash JainHead - Quality Consulting Practice
Confidential © L&T Infotech | 2 | L&T Infotech
Company Overview
Common IV&V Services and
Solutions
Test Outsourcing
Software Testing Trends
Agenda
Customer Expectations
Conventional Measures & Metrics
Unconventional Measures &
Metrics
Confidential © L&T Infotech | 3 | L&T Infotech
Larsen & Toubro InfotechLarsen & Toubro Limited
Established in 1938, European
roots
Technology focused diversified
business portfolio
Revenues: $5 billion
35,000+ employees world-wide
40 factories, 30 offices
150+ International Collaborations
& JVs
Forbes 2000 Company
Global IT services company
15 International Offices
14+ yrs of international
experience
NASSCOM – Top 9 Indian IT
Exporters
Strong Insurance domain
experience
18 out of Fortune 100 clients
8000+ IT Professionals
Corporate Certifications
Company Overview
Confidential © L&T Infotech | 4 | L&T Infotech
Software Testing Trends
Testing Trends ‘V Model’,’W’ Model, Agile TestingRisk Based Testing, Test Driven Design
Focus on early defect detection
Independent Verification and Validation (IV &V)
Increasing trend of an independent testing provider, akin to.. FOX TAKING CARE OF THE HEN - Forrester
V &V of development life cycle
Confidential © L&T Infotech | 5 | L&T Infotech
Frequently Not OutsourcedFrequently Outsourced
Test Outsourcing
Key FactsSoftware bugs cost $59.5 billion to the US economy- US Department of Commerce45-50 % getting outsourced - Gartner
Testing services growth rate - more than 50 % Offshore outsourced testing business in India ~ $1 billion by end of 2007 – META Group study
Functional, ETL, DW, Backend
Test Automation, SOA
Performance, Load, Penetration (Security)
Product, Package ImplementationTest & Process Consultancy
UAT Testing
White Box Testing
Business Process TestingRisk based TestingTest Driven Design
Confidential © L&T Infotech| 6 | L&T Infotech
Common IV&V Services and Solutions
Services
Functional
Non-Functional
ProductTesting Process Consultancy
Solutions
Automation Frameworks
Test Harness
Test BedsMetrics
Confidential © L&T Infotech| 7 | L&T Infotech
Expectations
Customer
Senior Mgmt Middle Mgmt
Product/Project Mgmt
IT-Business Alignment
Vendor
Growth Opportunities
Outsourcing Maturity
Quality Process Compliance
Testing Process Compliance
Thought Partnership
Cost ReductionROI
OptimizationEngagement Benefit
IQRCost of Quality
Domain Savvy
Quality ProcessTools
Confidential © L&T Infotech| 8 | L&T Infotech
Conventional Measures
Common SDLC Metrics
Project Effort
Project Cost
Project Risk
Project Schedule
Project Productivity
Defect related Metrics
Confidential © L&T Infotech | 9 | L&T Infotech
Conventional Measures
Testing Measures
Test Process Efficiency
Test Productivity
Cost of Testing
Test CoverageTest Effectiveness
Common Measurement Frameworks
Goal/Question/Metric Paradigm
Quality Function Deployment
Software Quality Metrics
Quality of Software
Confidential © L&T Infotech | 10 | L&T Infotech
Conventional Metrics
Measure Metrics
Test Process Efficiency
• Process Compliance• Resource Allocation and
Utilization
Test Productivity• Schedule Variance• Effort Variance
Cost of Testing• Direct Cost• Indirect Cost
Test Coverage• Requirement Coverage• Code Coverage
Test Effectiveness
• Residual Defect Density• Defect Distribution
(Severity)• Defect Rejection
Test Transformation• Automation Ratio• Onsite-Offshore Ratio
Testing Measures & Metrics
Confidential © L&T Infotech | 11 | L&T Infotech
Application Defect Trend Analysis
0
5
10
15
20
25
30
35
40
Cycle1 Cycle1 Cycle2 Cycle3 Cycle1 Cycle1 Cycle1
Oct-06 Nov-06 Nov-06 Nov-06 Jan-07 Feb-07 Mar-07
Month/Cycle
Def
ects
Critical Severity High Severity Medium Severity
Low Severity Undecided Total
Defect Trend Metrics
Conventional Metrics
Confidential © L&T Infotech | 12 | L&T Infotech
Quality Process Compliance
0
1
2
3
4
5
Cycle1 Cycle2 Cycle3 Cycle4
Reviews Audits Goals
Quality Process Compliance
Conventional Metrics
Confidential © L&T Infotech | 13 | L&T Infotech
Unconventional Measures
Do we Measure?
Testability of SoftwareQuality of Test cases
Testing ROI
Reliability
Domain Knowledge
User Satisfaction
IT-Business alignment
Business process optimization levelBusiness process fragmentation across applications
Confidential © L&T Infotech | 14 | L&T Infotech
Unconventional Metrics
Measure Metrics
Testability of Software• Controllability• Observability
Quality of Test cases
• Test case Impact• Test case Reusability• Test case Conformity
Engagement Efficiency • Engagement Cost Benefit
User Satisfaction • User Satisfaction Survey
Domain Knowledge • Domain Certification
Business Impact• Risk based Testing• Test Driven Development
Unconventional Measures & Metrics
Confidential © L&T Infotech | 15 | L&T Infotech
Unconventional Metrics
Units
Controllability
ControlsControlled Partially Not
Controlled Controlled
Units
Observability
Project Timeline
Dir
ec
tly
Vis
ible
O/P
Ind
ire
ctl
y V
isib
le O
/P
O/P
no
t v
isib
le
Vis
ible
Sta
tes
No
t V
isib
le S
tate
s
Testability Metrics
Unconventional Metrics
Confidential © L&T Infotech | 16 | L&T Infotech
Risk Based Testing Metrics
Unconventional Metrics
Confidential © L&T Infotech | 17 | L&T Infotech
References
Measuring the Effectiveness of Test, by Harry M. Sneed
Heuristics of Software Testability, by James Bach
Risk based Testing, by Hans Schaefer
Confidential © L&T Infotech | 18 | L&T Infotech
Makarand Hiralikar
Head IV & V Practice - Insurance, L&T Infotech
Rajesh Bharathan
Head Technology Solutions - Insurance, L&T Infotech
Acknowledgement
Deepesh BelaniLead, IV & V practice
This presentation contains contributions from my following colleagues in terms of their thoughts, experience, expertise and time.
Thank You
