Test and Verification Solutions129 April 2009 Managing Outsourcing - Case Studies Presentation to TMF April 2009 Mike Bartley, TVS

Test and Verification Solutions 129 April 2009

Managing Outsourcing - Case Studies

Presentation to TMF

April 2009

Mike Bartley, TVS

2Test and Verification Solutions 229 April 2009

What is “successful” software testing?

Outsourcing Success

Insource Onshore Near Shore Offshore

Budget L Schedule M Quality H

Onsite contractor

Offsite UK contractor

Offsite non-UK contractor (e.g. Ireland)

Offsite non-UK

contractor (e.g. India)

Minimise budget

Hit schedule

Appropriate quality

Low impact on minimising budget

Outsourcing Success


Budget L L - M Schedule M M Quality H H

Outsourcing Success


Budget L L - M M Schedule M M M Quality H H H

Outsourcing Success


Budget L L - M M H Schedule M M M H Quality H H H H

Outsourcing Success


Budget L L - M M H Schedule M M M H Quality H H H H Risk L L - M M H?

Outsourcing Success


Budget Schedule Quality


The main risks

• Supplier corporate governance– “$1 billion of the cash reported being on Satyam's books

was utterly "fictitious" ” Times Newspapers Online, 13th Jan `09

• Over-inflated expectations– “executives assume that labor arbitrage will yield

savings comparable to person-to-person comparison” META Group Inc, 16th Feb `09

• The usual project risks are even greater– “Project managers must recognize that offshore projects

will likely require even greater management of risk due to the unique challenges posed by geographical, cultural, and other differences”Ralph Kliem, "Managing the Risks Of Offshore IT Development Projects“, Information Systems Management, 2004, Vol. 21 Issue 3


The main risks

• Problems of a distributed team– Communication– Knowledge management– Failure to deliver required deliverables

• Stability of requirements and specifications• Security• Transition and change risk “I often have to caution

them not to be swayed by the “economics” without understanding all of the

“costs” involved”Rajiv Dholakia, VP Product Development, Cenzic Inc.,

“Happy About Outsourcing, 2005

• Do you know your tolerance for risk?

• How well do you manage risks?


Agenda

• Agenda

– Some background

– How to successfully manage software testing offshore• Setting up the relationship

• Managing the ongoing relationship

– Q&AMaximise the benefits

Minimise the risks


Outsourcing – some facts!

• Market size and share– India’s share of the global

offshore outsourcing market for software and back-office services is 44%

• Preferred destinations– Trend to look for cities

rather than countries

Global Services’ Top Outsourcing Cities for IT and BPO

Top Outsourcing Cities

Bangalore India

Chennai India

Delhi NCR India

Dublin Ireland

Hyderabad India

Makati City The Philippines

Mumbai India

Pune India

Source: Global Services October 2008

USD billion 2005-06 2006-07 2007-08

I T Software and Services Exports

17.3 22.9 28-29

Source: Nasscom


Why do companies outsource?

• Access to flexible resource– Ramp-up and ramp-down

• Time to market• Concentrate on core skills• Process improvements

– Outsource suppliers tend to have a strong focus on process

• Quality improvement– Access to specialist resources

• Cost– Reduction– Variabilisation!

Outsourcing allows a small company to grow

aggressively without taking on too much fixed cost

Outsourcing allows a large company to variabilise its fixed costs and potentially

reduce overall costs


The outsourcing process

• Define the strategic goals• Decide what to outsource• Review your readiness to outsource• Select your preferred outsource model• Select your outsource partner• Agree the

– Master Services Agreement – Service Level Agreement

• Outsource execution– Management– Governance


Effective outsourcing - Defined strategic goals

• Quality improvement– Independent test team– Get access to specialist skills– Getting access to higher quality staff

• Improve company focus• Reduce time to market

– Through improved company resource flexibility • Cost variablization• Cost reduction

• Should be agreed by the appropriate stakeholders– the software manager– the testing manager– the engineering manager– the chief financial officer– the outsourcing manager.

“Successful outsourcing activities are ones which are derived from strategic

objectives of the

companies involved”Bill Widmer, CEO g8solutions,

“Happy About Outsourcing, 2005

Having agreed strategic goals allows low level

decision making without continuous approval from higher level management







Deciding what to outsource

Control

Input OutputAlgorithm

New interactive “pad and mouse” device for computer interaction by disabled users

• The “special sauce” is in the algorithm• Develop algorithm unit tests and user

acceptance tests in-house• Outsource IO test development and

running system tests

Silicon

Driver

Libraries

ApplicationsSilicon manufacturer delivering

applications to run on their chipsNew applications being developed

• The variability and specialist domain knowledge is at the “application level”

• But the lower levels have stable APIs• Driver and library testing offshored• Application testing offshored as they stabilise


Deciding what to outsource ctd.

Interactive e-learning package using iterative development

• The “special sauce” is in the domain knowledge and pedagogy

• Outsource the GUI testing and “platform” testing

Bespoke application development using V-model• Detailed component level

specifications• Component testing must be

independent and achieve high structural coverage targets

• Outsource the component testing – black-box first, then white-box

Important considerations• Intellectual property issues• Proximity to product development• Complexity of interface• Minimise learning curve/knowledge transfer• How stable are the requirements/spec/design?• Is there a standard available?• Dev process: V-model vs. Agile


What development process do you use?

The V model• A very formal

development process

• Specifications are (in theory) written at the various levels

• Allowing early test spec and dev

• This lends itself well to outsourced testing


What development process do you use?

• Do agile methods work with outsourcing?• Quality of communication

– Bandwidth– Latency – Reliability – Availability

• There are a few examples of multi-site agile projects– Practical issues of daily scrum meetings, sharing burn down charts,

information radiators, story cards, …

• & fewer examples of offshored testing in agile projects• But …

– Testing the output of each time-boxed iteration– Regression testing to ensure older functionality doesn’t rust







Effective outsourcing - Readiness for Outsourcing

• The Capability Maturity Model – Level 1 - Ad hoc (Chaotic) – Level 2 - Repeatable – Level 3 - Defined – Level 4 - Managed – Level 5 - Optimizing

• Less formally– Requirements management– Change tracking and management– Project planning, monitoring and control– Configuration management– Build, test and release processes (and level of automation)– Validation and verification processes

• There is often a mismatch in process maturity between supplier & client!

There is a real risk that you will simply outsource your internal chaos and

get double the chaos







Effective outsourcing - Outsource model selection

• Commercial model– Cost-plus pricing– Fee-for-service pricing (or T&M)– Fixed price– Shared-risk/reward pricing– Business outcome achievement

• Location– Onsite– Near-shore– Off-shore “Multisourcing: the disciplined provisioning

and blending of business and IT services from the optimal set of internal and external

providers in the pursuit of business goals”Linda Cohen and Allie Young, Gartner, INC., “Multisourcing”







Effective outsourcing - Outsource partner selection

• Criteria– People (academic profile, training policy)– Retention– Process maturity– Quality Management System– Project and risk management– Corporate and financial profile– Technology and domain knowledge– Cost

• Process– Identify potential partners– Assess them against selection criteria to generate a short-list– Visit short-list to undertake a more detailed analysis– Have potential partners undertake a pilot project

Ensure you consider the costs of testers at various

levels of seniority and experience







Effective outsourcing – MSA & SLA

• The MSA is a complex legal agreement

– Confidentiality, ownership of deliverables, assignment of rights, indemnification, warranties, liability, changes in personnel, subcontracting, termination, penalties for not meeting agreed service and delivery targets, etc

• The SLA should cover the service and support goals

– Targets for responsiveness and quality of deliverables– The metrics & remedies should the targets be missed – Fixed-price vs. T&M– Sharing in success– Setting behavioral expectations

I find it useful to separate the MSA and SLA. They have different objectives and the MSA should be

more stable than the SLA







Outsourcing Execution: Management and Governance

• Management is about responsibility for specific decisions• Governance covers decision-making processes: who, how, when

• Good governance involves at least the following– Established standard decision-making procedures – Well-understood processes for handling exceptions – Decisions made quickly at the correct level against well-defined criteria– Decisions get recorded

• And good governance is helped by the following– having agreed strategic objectives– formal communication methods which record project status and decisions– controls & records to ensure governance procedures are followed– regular meeting of the stakeholder group

• Good management means– The usual project management practises PLUS

You need to give the outsourcer the “right” level of independence!


Good management

• Communication– Who can communicate with who?– How often?– Formal vs. informal communication– How does the communication get recorded?– What happens on miscommunication?

• Knowledge management– Getting up the initial learning curve– On-going knowledge management

• What is the process for asking a question?• What is the process for recording an answer?

Perform regular audits of the knowledge

database!


Good management ctd.

• Audit trails – What trails do you need?

• Decision making

• Execution history

• Timesheets

– You need to save this data as quickly and as cheaply as possible

• What about access time?

– How strict does your audit need to be?

• Well-defined incoming QA proceduresYes – QA of the QA!


Case Study – ClearSpeedBackground

• ClearSpeed’s product range includes chips, accelerator cards, and software

• All work with industry-standard x86-based systems

• 3 outsourcers• 2 fixed-price• 1 T&M team-

based


Case Study – ClearSpeedSome details

• Readiness– Mostly V-model, some iterative– Use of MRD– cfg mgt system supporting regular builds, automated tests, ftp– Software releases followed alpha, beta, GA

• Complex MSA• SLA

– Fixed-price – simple SOW and end-of-project reviews– T&M

• Predicted test execution completion dates• Daily and weekly status reports• Direct access to the defect tracking system• Audit trail of test results, decisions, changes to plans

We had a matrix of tests

& platforms with expected

execution times

We could access the results of any test ever run


Case Study – ClearSpeedMain results• Improvement in main KPIs (key performance

indicators)– Quality (Defect Containment Effectiveness)– The time-to-market– Plus reduced costs

• Other benefits– Able to access specialist resource (e.g. GUI testing)– Team size matched required level of testing within

financial constraints– Undertake one-off projects that would not have been

possible otherwise– The use of buffer resources– Use of multi-sourcing to help control price increases

• Problems– The quality on one of the fixed-price projects

deteriorated over time– Costs for lower qualified staff– Active use of the knowledge database– Metrics fatigue!

Improved time-to-market helped byEngineers able to work on rel “X+1” in parallel to release “X” being testedEngineers focused on their core competencyReduced context switches for engineersReduced test cycle time and fewer spins (esp. close to the release)

Improved quality came throughIndependence of the test functionImproved test processesSpecialist test skillsImproved audit trails of tests

Could we have gained the benefits by building an internal team?


In conclusion

• Minimise risk of offshoring – Understand your strategic objectives for offshoring

• Cost, schedule, quality, …– Careful partner selection

• Domain expertise; People; Retention; Process; Quality; Project and risk management; Company profile; Technology; Cost

– Process maturity• Internally and at the offshore supplier

– Commercial contract• T&M vs. fixed-price

– Select what to offshore and the development process– Execution and governance

• Expect an increase in your management overhead• It requires specialist management and testing expertise

– Prepare your incoming QA well in advance


In conclusionMaximise impact of offshoring on your success factors

– Budget• Offshore SW development resource costs depend on

– Location– Supplier – Experience– Volume

– Schedule• Resource flexibility

– Rapid ramp-up (domain knowledge!) and ramp-down– Buffer resource

• Partners with a large workforce and skills-base (?)– Test techniques and specialism– Vertical markets– Tools

– Quality• Independence of the test function• Use of specialist services in development and testing• Careful partner selection


The potential benefits

• Revenue growth without fixed cost growth• Improve the 3 critical success factors

– Budget, schedule, quality

• Independent V&V– Studies show independent V&V can have a significant

impact on quality

• Process improvement(?)• Employee motivation

– Outsource the boring bits!


Questions?

Dr Mike Bartley MBA – [email protected]

07796 307958

• Thank you for your time – any questions?

Documents

Test and Verification Solutions129 April 2009 Managing Outsourcing - Case Studies Presentation to TMF April 2009 Mike Bartley, TVS