Disclaimer: All the documentation and other material contained herein is the property of Thinksoft Global Services and all intellectual property rights in and to the same are owned by Thinksoft Global Services. You shall not, unless previously authorized by Thinksoft Global Services in writing, copy, reproduce, market, license, lease or in any other way, dispose of, or utilize for profit, or exercise any ownership rights over the same. In no event, unless required by applicable law or agreed to in writing, shall Thinksoft Global Services, or any person be liable for any loss, expense or damage, of any type or nature arising out of the use of, or inability to use any material contained herein. Any such material is provided “as is”, without warranty of any type or nature, either express or implied. All names, logos are used for identification purposes only and are trademarks or registered trademarks of their respective companies.

For more details visit, www.thinksoftglobal.com

Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies

Initiative Yea $ Cost % reduction

Initial state 0 38000 0%

RPM repository 1-p1 32300 15% consolidation

Automation 1-p2 19380 25%

Productivity 2 10659 5% improvement

Productivity 3 5330 5% improvement

Total 50%

Using this framework, the specialist testing partner:

• Created Centers of Excellence around the cards business and

related test process

• Built and organized structured test repositories

• Designed Test Automation frameworks custom made to the

cards business

• Implemented Risk Prioritization Matrices to ‘Triage’ the most

business critical functionalities that are most likely to fail

• Developed tools to integrate the repositories and automation

frameworks

Net result - a sustainable savings of 50% on annual regression test costs - $1.2 m!

Epilogue:

From Dr Robert Harris’ presentation it was clear that collaborating with a reliable and worthy partner with the right set of capabilities is very critical to success!

What was not made explicitly clear at his presentation was that in both cases the reliable and worthy testing partner was none other than Thinksoft Global Services Ltd.

T r u s t t h e E x p e r t s

IS SOFTWARE TESTING A“ZERO SUM GAME”– YES OR NO?

Case Studies

The Context

• How does one optimize management decisions in the software testing arena? • Are business users to entrust ‘requirement planning’, product fitment, process mapping and customization to the product

developer? • Should the developer also be entrusted with all interfacing and data migration tasks?• In parallel, should the business user also build in-house teams of testers and test managers to interface with the product

developer, and do the “acceptance testing” themselves? • Can the expertise required to sync with ever changing customer needs be developed and retained in-house• Alternatively should the entire gamut of testing functions be off loaded to specialist testing houses that are springing up,

independent of software product developers? • Would such independent expert services be more reliable and flexible over time in highly competitive sectors based on

“deep domain “knowledge? • How do you balance on the one hand the high cost of failure and on the other, the need to control costs to retain margins

under competitive pressures?

These were some of the important questions occupying Dr. Robert Harris, who was to make a presentation at an industry seminar.

Being an academic he knew the answers in theory, but he still needed to validate his assumptions using “field data”.

The following case studies illustrate his findings.

Case 1

Testing center of excellence for the Treasury arm of a global conglomerate

The Program Manager (PM) of the global conglomerate had to execute a multiyear program to roll out a sophisticated reporting tool and a data ware-housing engine.

The existing globally reputed system integrator who was expected to manage requirement planning and testing was not delivering satisfactorily due to obvious limitations in domain experience.

Business requirements were ill articulated and poorly documented. They existed only as spreadsheets or only in the minds of the business user.

There was a severe shortage of business users to carry out even mandatory tests.

Existing test practices being weak, defective software was leaking into production resulting in significant rework. T r u s t t h e E x p e r t s

Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies

What were the Product Manager’s (PM’s) options?

At that juncture, a colleague suggested the name of a reliable specialist testing house that focused on the financial services industry from end to end. The PM took the plunge and the results were immediate with efficiencies improving and costs shrinking significantly.

The benefits across the first 2 years were:

Quicker Knowledge Transfer Being industry focused, the testing team was 40% faster than the earlier generic IT partner in getting up to speed on mapping processes and applications. This enabled a faster ramp-up and a quicker user buy-in.

Static Testing reduces avoidable reworkBy identifying gaps between data mapping specs and design specs upfront, close to 900 errors were un-earthed. Almost the same numbers of defects were identified during the testing phase. Thus, 50% of total defects, related rework and repair costs were avoided, even before a single piece of code was written. Compared to earlier times, development rework fell by 12%. Testing schedules were compressed by 10%.

Right ShoringAs the testing transitioned from business users to the specialist partner up to 68% and 74% of the work was off-shored in Y1 and Y2 respectively, resulting in significant cost savings. Defect PreventionAs’ Functional defect’ leakage to production reduced to less than 1%, the business experienced a near-defect free application delivery

Domain Value AddsGiven the short windows for daily validation in production, the domain focused test team identified business critical fields. The team also identified a representative data sample covering all instruments such that the shorter production validation strategy covered all the business critical areas.

Benefits across Years 3 and 4

As the relationship matured, the specialist partner brought in benefits from re-usable repositories, automation and productivity improvement initiatives.

Productivity GrowthWhile the specialist test team size grew only 30% in 3 years, the output increased by 143% and 224%

AutomationThe creation of a robust automation framework for data warehousing cum business analytics engine started delivering savings of about $100,000 for the first application stream and was expected to save $740,000 as it is implemented across application streams.

Re-usable repositoriesCreating structured repositories resulted in over 69% of the test cases being re-used. The re-use is bringing in annual savings of $200k.

Cost-out initiativesThe specialist implemented various cost-out initiatives such as ‘prototype testing’, ‘QA batch load ownership’ and ‘offshore shifts’ for further cost savings to the tune of $400k per annum.

Case 2

Global Regression Test Center for an Industry leader in Cards

The Vice President Product Engineering (VP-PE) of a globally renowned cards software maker found that the product QA was becoming overwhelmingly complex due global custom implementations and frequent product releases.

The top Issues of concern

• Customer demand for high product flexibility and wide parameterization.

• Intense competition meant no second chance on new product introductions.

• Spiraling maintenance costs due to field releases of in-adequately tested products

• Large customers invoking penalty clauses due to defective deliveries ($6m for every month of delay!)

On enquiry, one of the clients recommended a specialist testing house that is focused on the financial services industry.

On engaging the firm the VP-PE found striking differences in performance!

As he attempted to quantify the benefits from the specialist, he wondered why he didn’t do this earlier.

The Short and medium term benefits across first 2 years were: -

ZERO Knowledge TransferThe product developer had no bandwidth for any knowledge transfer. Yet the specialist successfully ramped up, thanks to their financial industry focus, knowledge and experience.

Avoiding Penalties in Customer RolloutsRigorous ‘cards industry focused’ test methodologies helped the software maker avoid client penalties (which were in some cases as high as $6 million for every month’s of delay)

Defect Leakage less than 5%Defect leakage was either not measured and was 20% in some cases. It was uniformly brought down to less than 5% thus improving the quality of applications delivered and reducing the maintenance costs for the product vendor.

88% right shoringBuilding the QA organization, the specialist started off with an offshore leverage of 83% and raised it to 88% within the first year.

Long term benefits over Years 3 and 4 were: -With the specialist, continuous improvement was not an afterthought. To deliver a 50% saving on regression test costs, the specialist implemented a ‘continuous improvement’ framework

Release # of static # of defects % of defects # Testing identified Total avoided Issues in execution (A/C*100) identified upfront

(A) (B) C = (A+B)

1 427 592 1019 42%

2 164 90 254 64%

3 24 17 41 59%

4 28 8 36 78%

4.2 245 173 418 59%

All 888 880 1768 50%

Initiative Annual Cost Savings

1 Prototype Testing $50,000

2 QA Batch Load Ownership $60,000

3 Offshore Shifts $40,000

4 Improve offshore leverage $50,000

5 Productivity Improvements $200,000

Result Specialist General Root Cause Area Testing IT service Analysis Partner provider

Defects in Customer

Environment

6% 20% Functional Coverage in

TestingTest Strategy

Time to Market

On time 7 days for P1 (SLA = 2

days)

Delays in Defect Resolution turnaround

Requirements Coverage

in Testing

>95% Not Measured,

60%

Lack of Traceability,

Test Strategy and Inadequate

coverage

130%

Team

100%

123%

130%

Year 1

Productivity

Year 2

Year 3

Test Created

100%

268%

243%

Executed

100%

230%

324%

243%

324%

pe

rce

nta

ge

gro

wth

350%300%250%200%150%100%

50%0%

Domain COE

• Business Processes• Functional Artifacts• Tracking of Global Regulatory Changes

• Process Maintenance• Process Definition• Metric Program• Process Training• Continuous Process Improvement• Best Practice implementation

Process COECompetency

Building, Recruitment,

Training

Repositories Ready-to-use

Artifacts

Cards Processor

Competency Center

Resource Management

Quality Assurance

ResourceManagement

Automation using Tools

Metrics & Reporting

Standard Processes & Methodologies

KnowledgeRepositories Best Practices

Continuous Improvement Framework

Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies

What were the Product Manager’s (PM’s) options?

At that juncture, a colleague suggested the name of a reliable specialist testing house that focused on the financial services industry from end to end. The PM took the plunge and the results were immediate with efficiencies improving and costs shrinking significantly.

The benefits across the first 2 years were:

Quicker Knowledge Transfer Being industry focused, the testing team was 40% faster than the earlier generic IT partner in getting up to speed on mapping processes and applications. This enabled a faster ramp-up and a quicker user buy-in.

Static Testing reduces avoidable reworkBy identifying gaps between data mapping specs and design specs upfront, close to 900 errors were un-earthed. Almost the same numbers of defects were identified during the testing phase. Thus, 50% of total defects, related rework and repair costs were avoided, even before a single piece of code was written. Compared to earlier times, development rework fell by 12%. Testing schedules were compressed by 10%.

Right ShoringAs the testing transitioned from business users to the specialist partner up to 68% and 74% of the work was off-shored in Y1 and Y2 respectively, resulting in significant cost savings. Defect PreventionAs’ Functional defect’ leakage to production reduced to less than 1%, the business experienced a near-defect free application delivery

Domain Value AddsGiven the short windows for daily validation in production, the domain focused test team identified business critical fields. The team also identified a representative data sample covering all instruments such that the shorter production validation strategy covered all the business critical areas.

Benefits across Years 3 and 4

As the relationship matured, the specialist partner brought in benefits from re-usable repositories, automation and productivity improvement initiatives.

Productivity GrowthWhile the specialist test team size grew only 30% in 3 years, the output increased by 143% and 224%

AutomationThe creation of a robust automation framework for data warehousing cum business analytics engine started delivering savings of about $100,000 for the first application stream and was expected to save $740,000 as it is implemented across application streams.

Re-usable repositoriesCreating structured repositories resulted in over 69% of the test cases being re-used. The re-use is bringing in annual savings of $200k.

Cost-out initiativesThe specialist implemented various cost-out initiatives such as ‘prototype testing’, ‘QA batch load ownership’ and ‘offshore shifts’ for further cost savings to the tune of $400k per annum.

Case 2

Global Regression Test Center for an Industry leader in Cards

The Vice President Product Engineering (VP-PE) of a globally renowned cards software maker found that the product QA was becoming overwhelmingly complex due global custom implementations and frequent product releases.

The top Issues of concern

• Customer demand for high product flexibility and wide parameterization.

• Intense competition meant no second chance on new product introductions.

• Spiraling maintenance costs due to field releases of in-adequately tested products

• Large customers invoking penalty clauses due to defective deliveries ($6m for every month of delay!)

On enquiry, one of the clients recommended a specialist testing house that is focused on the financial services industry.

On engaging the firm the VP-PE found striking differences in performance!

As he attempted to quantify the benefits from the specialist, he wondered why he didn’t do this earlier.

The Short and medium term benefits across first 2 years were: -

ZERO Knowledge TransferThe product developer had no bandwidth for any knowledge transfer. Yet the specialist successfully ramped up, thanks to their financial industry focus, knowledge and experience.

Avoiding Penalties in Customer RolloutsRigorous ‘cards industry focused’ test methodologies helped the software maker avoid client penalties (which were in some cases as high as $6 million for every month’s of delay)

Defect Leakage less than 5%Defect leakage was either not measured and was 20% in some cases. It was uniformly brought down to less than 5% thus improving the quality of applications delivered and reducing the maintenance costs for the product vendor.

88% right shoringBuilding the QA organization, the specialist started off with an offshore leverage of 83% and raised it to 88% within the first year.

Long term benefits over Years 3 and 4 were: -With the specialist, continuous improvement was not an afterthought. To deliver a 50% saving on regression test costs, the specialist implemented a ‘continuous improvement’ framework

Release # of static # of defects % of defects # Testing identified Total avoided Issues in execution (A/C*100) identified upfront

(A) (B) C = (A+B)

1 427 592 1019 42%

2 164 90 254 64%

3 24 17 41 59%

4 28 8 36 78%

4.2 245 173 418 59%

All 888 880 1768 50%

Initiative Annual Cost Savings

1 Prototype Testing $50,000

2 QA Batch Load Ownership $60,000

3 Offshore Shifts $40,000

4 Improve offshore leverage $50,000

5 Productivity Improvements $200,000

Result Specialist General Root Cause Area Testing IT service Analysis Partner provider

Defects in Customer

Environment

6% 20% Functional Coverage in

TestingTest Strategy

Time to Market

On time 7 days for P1 (SLA = 2

days)

Delays in Defect Resolution turnaround

Requirements Coverage

in Testing

>95% Not Measured,

60%

Lack of Traceability,

Test Strategy and Inadequate

coverage

130%

Team

100%

123%

130%

Year 1

Productivity

Year 2

Year 3

Test Created

100%

268%

243%

Executed

100%

230%

324%

243%

324%

pe

rce

nta

ge

gro

wth

350%300%250%200%150%100%

50%0%

Domain COE

• Business Processes• Functional Artifacts• Tracking of Global Regulatory Changes

• Process Maintenance• Process Definition• Metric Program• Process Training• Continuous Process Improvement• Best Practice implementation

Process COECompetency

Building, Recruitment,

Training

Repositories Ready-to-use

Artifacts

Cards Processor

Competency Center

Resource Management

Quality Assurance

ResourceManagement

Automation using Tools

Metrics & Reporting

Standard Processes & Methodologies

KnowledgeRepositories Best Practices

Continuous Improvement Framework

Disclaimer: All the documentation and other material contained herein is the property of Thinksoft Global Services and all intellectual property rights in and to the same are owned by Thinksoft Global Services. You shall not, unless previously authorized by Thinksoft Global Services in writing, copy, reproduce, market, license, lease or in any other way, dispose of, or utilize for profit, or exercise any ownership rights over the same. In no event, unless required by applicable law or agreed to in writing, shall Thinksoft Global Services, or any person be liable for any loss, expense or damage, of any type or nature arising out of the use of, or inability to use any material contained herein. Any such material is provided “as is”, without warranty of any type or nature, either express or implied. All names, logos are used for identification purposes only and are trademarks or registered trademarks of their respective companies.

For more details visit, www.thinksoftglobal.com

Is Software Testing a “Zero Sum Game”– Yes or No? Case Studies

Initiative Yea $ Cost % reduction

Initial state 0 38000 0%

RPM repository 1-p1 32300 15% consolidation

Automation 1-p2 19380 25%

Productivity 2 10659 5% improvement

Productivity 3 5330 5% improvement

Total 50%

Using this framework, the specialist testing partner:

• Created Centers of Excellence around the cards business and

related test process

• Built and organized structured test repositories

• Designed Test Automation frameworks custom made to the

cards business

• Implemented Risk Prioritization Matrices to ‘Triage’ the most

business critical functionalities that are most likely to fail

• Developed tools to integrate the repositories and automation

frameworks

Net result - a sustainable savings of 50% on annual regression test costs - $1.2 m!

Epilogue:

From Dr Robert Harris’ presentation it was clear that collaborating with a reliable and worthy partner with the right set of capabilities is very critical to success!

What was not made explicitly clear at his presentation was that in both cases the reliable and worthy testing partner was none other than Thinksoft Global Services Ltd.

T r u s t t h e E x p e r t s

IS SOFTWARE TESTING A“ZERO SUM GAME”– YES OR NO?

Case Studies

The Context

• How does one optimize management decisions in the software testing arena? • Are business users to entrust ‘requirement planning’, product fitment, process mapping and customization to the product

developer? • Should the developer also be entrusted with all interfacing and data migration tasks?• In parallel, should the business user also build in-house teams of testers and test managers to interface with the product

developer, and do the “acceptance testing” themselves? • Can the expertise required to sync with ever changing customer needs be developed and retained in-house• Alternatively should the entire gamut of testing functions be off loaded to specialist testing houses that are springing up,

independent of software product developers? • Would such independent expert services be more reliable and flexible over time in highly competitive sectors based on

“deep domain “knowledge? • How do you balance on the one hand the high cost of failure and on the other, the need to control costs to retain margins

under competitive pressures?

These were some of the important questions occupying Dr. Robert Harris, who was to make a presentation at an industry seminar.

Being an academic he knew the answers in theory, but he still needed to validate his assumptions using “field data”.

The following case studies illustrate his findings.

Case 1

Testing center of excellence for the Treasury arm of a global conglomerate

The Program Manager (PM) of the global conglomerate had to execute a multiyear program to roll out a sophisticated reporting tool and a data ware-housing engine.

The existing globally reputed system integrator who was expected to manage requirement planning and testing was not delivering satisfactorily due to obvious limitations in domain experience.

Business requirements were ill articulated and poorly documented. They existed only as spreadsheets or only in the minds of the business user.

There was a severe shortage of business users to carry out even mandatory tests.

Existing test practices being weak, defective software was leaking into production resulting in significant rework. T r u s t t h e E x p e r t s