Moving Quality upstream - HCL Technologies · quality control. This reactive process of inspection entailed receiving each unit into the Quality Control Department off the line and

Business Assurance & Testing

WHITEPAPER JunE 2015

www.hcltech.com

Moving Quality upstreamHow shifting quality to day one of the delivery lifecycle using HCL’s Business Assurance Framework can transform the traditional testing function from value destroyer to value creator

AuthOr:

Robb La VelleGlobal Leader of Business Assurance

& Testing Services

QEx Whitepaper

2MOVING QUALITY UPSTREAM

The day US Marine walked into the Oldsmobile showroom and the keys were handed over to him was a momentous one.

The year was 1952, World War II was over and a bright future with his growing family lay ahead. What better means of

propelling him into that new, wide open future than a new Super 88. But was the 88 really all that super, much less the

state of quality in the entire American automotive industry?

Edward Deming had arrived in Japan only a few years earlier and used his military assignment to assist a Japanese census

to begin training engineers in statistical process control and concepts of quality, lessons that would ultimately transform

their industry and the world.

But Flint, Michigan was 9,000 miles and a paradigm away from Tokyo and the burgeoning quality revolution. The old Olds

was subject to the same approach to quality that had been in place since the inception of mass production: post-production

quality control. This reactive process of inspection entailed receiving each unit into the Quality Control Department off the

line and running through a checklist designed to identify defects in the manufacturing process. The defective vehicles

would then be staged for re-work and ultimately released for delivery.

Today’s automotive manufacturing environment looks nothing like that of 1952. Developments in statistical quality control,

quality in design and engineering, six sigma manufacturing processes, lean manufacturing, design simulation and other

methods and tools now provide us with products that have quality ‘built in’ from the beginning of their lifecycle rather than

‘hammered on’ at its tail end. Yet despite the huge advances realized in the car industry, it is sad to say that the way quality

is managed in today’s enterprise software development projects has not progressed far beyond the methods used for the

old ’88.

THE SOLDIER’S

OLDSMOBILE


How familiar is this scenario to you? The project kicks off with the fanfare and enthusiasm of a New Year’s celebration. The

teams are energized and full of excitement as they jump off of the starting line and quickly get into high gear. But as those

individual gears start to mesh, misalignments take root and a breeding ground for defects emerges. These misalignments,

‘special cause’ variability as termed by Deming, should be the target of an end-to-end quality effort.

To give an example, ‘normal’ variability exists when business requirements are still in a state of flux while teams collaborate

on a solution. Ultimately, a process of iteration and agreement stabilizes requirements and ambiguity is purged. ‘Special

cause’ variability, however, emerges when the changes to these requirements are not accurately propagated across

project teams and defects are allowed to take root.

As this story continues, requirements completion lags as limited access to business users constrains progress and their

definitions become murkier in the sprint to meet deadlines. Design teams hamstrung by the same resource bottleneck

do their utmost to turn requirements into cohesive functional designs, often without vital insight required to fit all of the

pieces of the puzzle together. Design review sessions are held as often as possible, but the difficulty of getting the right

representation when needed compromises the end-to-end integrity of the solution.

Developers now get into the picture translating completed functional designs into technical specifications. But ongoing

modification of requirements sends a wave of volatility through the solution landscape even though efforts are made to

keep the whole entity in synch. Code and configuration commences on finalized technical specifications without the benefit

of clarity on adjacent technical components. This would be akin to installing a kitchen without the luxury of knowing how

the plumbing will run.

HOW FAR

WE’VE COME


Hours that had been dedicated for initial unit and assembly testing are now consumed to complete objects held hostage

by upstream deliverables. Structural misalignments creep into the source code and are undetected by development leads.

And while this great machine is being specified, designed and built like fitting new shocks to a car moving at high speed,

quality control begins in the form of functional, performance and security testing. If all goes to plan, testing can commence

against a relatively complete code base. But if stage containment violations persist, causing a ‘smear’ of deliverables

across project phases rather than a clean hand off as planned, test leads are usually asked to make due and manage ‘test

phase squeeze’ as best they can. Sometimes, the team can effectively risk-assess what needs to be done and deployment

can happen without a grand explosion. But more often than not, the churn of late code against a previously stable system,

compression of the testing timeline, and compromises of its scope lead to poor test effectiveness and unacceptable defect

leakage into production.

The most unacceptable aspect of this very typical scenario is that quality is purely of the reactive ‘inspective’ variety as

outlined in the Olds 88 example. As the project nears the finish line, resources that would have been rolled off are extended

and assigned to the ‘all hands on deck’ effort to identify and rectify as many of these embedded defects as possible. This

firefighting approach to defect purging invariably adds unforeseen costs to the project budget.

Deming outlined 14 key principles for

transforming business effectiveness in

his book Out of the Crisis. The second of

these principles is directly applicable to this

problem and is the basis for this paper:

“To effectively build quality into a finished

good, teams must ‘cease dependence on

inspection to achieve quality” but rather,

they must “eliminate the need for massive

inspection by building quality into the

product in the first place.”

The arguments against transcending this

state or even considering a better way

to mitigate its inevitable occurrence are

frequent and varied. As indicated before,

an often-heard viewpoint in the field is that

quality is something the combined team will

‘hammer through’ at the end of the project.

Again, sometimes the strongarm approach

does work, but often it does not. Another

view is that the cost of implementing

lifecycle-spanning quality measures is more

expensive than the cost of fixing errors later in production. This may be true depending on the applications in question and,

of course, the nature of the production incidents.

But the arguments that ‘consistency drives quality’ and ‘quality drives productivity’ are difficult for anyone with a background

in manufacturing to dispute. Even more difficult to ignore is the simple and industry-accepted fact that the longer a defect

becomes ‘nested’ in a solution, the more difficult (and expensive) it is to resolve.


A recent example from the automotive industry (actually, motorcycle industry) that affected the author directly involved some incorrectly manufactured connecting rods for a high performance Italian sport motorcycle. A missed design specification led to the incorrect manufacture of these components that were installed into engine blocks, bolted to their transmissions and ultimately fixed to the frames of completed motorcycles. These then left the factory and were shipped to dealers worldwide.

As value was added in the creation of the finished good, so too was the complexity of rectifying the issue as the source became increasingly nested in its core. The company made the correct move in simply replacing the engines of all affected bikes but at a staggering cost that severely impacted profitability.

The message for managers of enterprise software projects should be resoundingly clear: By identifying errors at the point where they occur, we can deliver better software at a lower overall cost.

HCL, a global thought leader in Application QA Services, has understood this problem for some time. To address it, the company has researched and developed a quality construct designed to shift quality upstream in the software development continuum right up to the point where projects have their inception. HCL calls this construct Business Assurance Framework.

The HCL has been an industry leader in the application quality assurance space for over 20 years and this experience has culminated in our ‘quality from day one’ approach to software delivery. The essence of Business Assurance Framework is that quality should be a primary focus from the very start of every project and that doing so requires an orchestrated quality strategy and the people, processes and tools to enable it. But more than anything, it requires a change in thinking from a design/build/test model to one based on an end-to-end quality stream.

Business Assurance Framework offers this quality stream in the form of an umbrella framework of components aligned for this purpose. It defines a holistic quality strategy, it enables an overarching quality organization and provides the individual tools and processes required to execute the quality program from day one. Its ultimate goal is to allow businesses to deliver

higher quality software products for less overall cost.

ANOTHER LESSON FROM

MANUFACTURING


As argued earlier, it is commonly accepted that the cost of resolving an undetected defect increases as the project unfolds

and it becomes increasingly integrated into the end solution. To illustrate, consider a meeting where the project team

for a petro-chemical company is collaborating on the requirements for a new order transaction type that must meet the

needs of customers requesting that orders be split across multiple shipments. Absent from the meeting on that day is a

representative from the transportation group. The team tables and defines the split shipment requirement along with many

others that busy day and makes the final input available to the business analyst team for design.

The vital piece of information that never made it into the requirement was the compartment capacity of individual tanker

trucks types. The lifecycle continues with the requirements defect now nested in the solution, technical specifications are

drafted, the new order type is coded, interfaces are modified, data objects are defined and mapped, test cases are written

and executed.

Unfortunately, not until User Acceptance Test is the design flaw identified. At that point, it is estimated that the effort to

resolve the defect, including modifying impacted technical and data objects, changing supporting designs and re-writing

test cases will run into thousands of man-hours. In fact, several studies have indicated that the cost of fixing this defect

during UAT will cost about 50 times more than what it would have cost to resolve during the design phase.

Scenarios like this are anything but rare on large, enterprise projects. The cost avoidance offered by Business Assurance

Framework strategies can be easily quantified by considering the average cost of fixing the same defect at various stages

during the software development lifecycle and an average distribution of defects through the phases of a typical project.

THE ECONOMIC

ARGUMENT FOR BUSINESS ASSURANCE

FRAMEWORK


In a study conducted by Caper Jones,

a leading researcher in software

engineering methodologies and

Chief Scientist Emeritus of Software

Productivity Research, defect data

collected from 675 companies

representing 13,500 individual projects

indicates that 36% of defects have their

origins in Requirements and Design

phases.

Adding a further dimension to this

analysis, a study by TRW Emeritus

Professor of Software Engineering

Barry Boehm assessed that the average

cost of fixing a defect increases 100

fold between the initial Design Phase

of a project and the deployment to

Production. So where a defect found in

that initial phase will cost $140 to fix, it will increase to $500 during Build, $1,000 during Unit Test, $2,500 during Integration

Test, $4,500 during System Test, $7,000 during Operational Readiness Testing/UAT and ultimately $14,000 in Production.

1.Designdefects

2. Codedefects

3. UnitTest

defects

4. DataDefects

5.Require

mentdefects

6. WebSite

defects

7.Sequiritydefects

8. Bad fixdefects

9. Testcase

defects

10.Docume

ntdefects

11.Architect

uredefects

TotalDefects

SDLC Distribution 17.00% 15.00% 13.00% 11.00% 19.00% 8.00% 7.00% 4.00% 2.00% 2.00% 2.00% 100.00%

Defects 170 150 130 110 190 80 70 40 20 20 20 1000

Cost to fix at Source $23,800 $21,000 $18,200 $15,400 $26,600 $11,200 $9,800 $5,600 $2,800 $2,800 $2,800 $140,000

Addl cost to fix 50% during SIT $358,700 $316,500 $274,300 $232,100 $400,900 $168,800 $147,700 $84,400 $42,200 $42,200 $42,200 $2,110,00

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

Axis

Titl

e

Incremental Cost of Defect Resolution

Total Potential savings of $1,970,000

The data points of these two studies enable the estimation of very realistic scenarios. Figure 2 shows the distribution of

1,000 Severity 1 & 2 defects to each of the offending phases using the Caper Jones study. If we assume that 50% of these

defects are not found until System Test, applying the Boehm defect resolution costs noted above yields a resolution cost

that is almost $2m higher than it would have been had they been detected and resolved in the phase of their origins. The

defects from the early Requirements and Design Phases contribute nearly 40% to these increased fix costs.

Just as in the example of the nested engine component in the expensive Italian motorcycle, the argument to identify and

resolve defects early, or better, prevent them altogether, is a compelling one that all IT executives should consider.

Figure 1: Distribution of software development project defects

17.00%

15.00%

13.00%

11.00%

19.00%

8.00%7.00%

4.00%

2.00% 2.00% 2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

20.00%

36% of defects originate in Requirements &

Design phases Defect Distribution

Source: (150 clients in Fortune 500 set); About 35 government/military groups;

About 13,500 total projects; New data = about 50-75 projects per month; Data

collected from 24 countries

Figure 2: Model estimating cost to resolve 1,000 Severity 1 & 2 defects at their origin vs cost to resolve same defects during System Test


As anyone who has spent years managing enterprise software projects knows, the business case presented above may

seem realistic but developing a cost effective, manageable approach to tackle it is undeniably challenging. The Business

Assurance Framework is designed to do just that.

HCL has developed and perfected an array of organizational change requirements, methods and tools, both internally

developed and sourced through strategic partnerships, which can help better manage the flow of quality deliverables

through the development lifecycle and achieve the delivery of better business solutions at a lower total cost.

The implementation of an effective Business Assurance Framework strategy can take many forms, but the pillars of success

will be based on three core tracks:

1. Deploy Quality-Driving Tools

2. Enable Quality through the Empowerment of the QA Team

3. Arm a Quality Program with the Industry-Leading Processes

AN APPROACH FOR IMPLEMENTING A

BUSINESS ASSURANCE FRAMEWORK STRATEGY


1. DEPLOY QUALITY-DRIVING TOOLS

The tools that support quality programs are many and varied but to be sure that they support

a Business Assurance Framework strategy, they should serve three clear objectives:

1. Support the capture and management of deliverables created during the project lifecycle,

including Requirements, Designs, Development Objects, Test Cases and Defects.

2. Support the generation of efficiencies, which enable more comprehensive test coverage.

3. Support the validation of quality in design and structure.

At their most basic are simple Excel-based tools for test design and management. But larger

programs should rely on purpose-built business process design tools and test management

tools to enable the integrated management of requirements, test design & execution and

defects as well as specialty tools designed to automate testing and compress testing cycles.

During the Build Phase, tools by vendors provide an automated approach to capture and

quantify the quality, complexity and size of business applications by analyzing their structural

quality. Structural Quality Analyzing applications perfectly complement a Business Assurance

Framework quality strategy by providing analysis of all tiers of a complex application at the

source code level and measuring adherence to architectural and coding standards, while

providing a bottom-up view of development quality and technical debt and software engineering

advice to Application Development teams.

And during the Testing Phase more emphasis should be exercised on the automated creation

of standardized test cases, checking the non-functional requirements, and executing full end

to end tests provide a thorough insight in the application quality and still focus on the Business

Assurance Framework strategy. In particular, the creation of standardized test cases provides

faster execution of the full Test Phase. By using models for designing standard test cases, an

enabler of better, faster and cheaper testing is created.

2. ENABLE QUALITY THROUGH THE EMPOWERMENT OF THE QA TEAM

No quality program will be successful without program-wide support and the firepower to back

it up. As explained above, current approaches to quality are heavily weighted toward the tail

end of the delivery lifecycle. Activities during these phases fall on the shoulders of a test lead

with the usual heavy migration of business analysis and development resources to the test

team.

A Business Assurance Framework quality environment foresees quality activities kicking off and

running in parallel to the entire development lifecycle. So in order to be successful, it should

be orchestrated by a single organizational entity. This could be the PMO or Project Manager,

but for larger projects a better solution would be a dedicated resource with a background in

testing and quality management as well experience in upstream project activities. This elevated

role would have responsibility for defining the overall quality strategy, assembling the resources

and assets required to implement it, and have the clout to stand up to project leadership on

implementation, particularly at Quality Gates as collaboration points.


3. ARM A QUALITY PROGRAM WITH THE INDUSTRY- LEADING PROCESSES

Process additions against the status quo need not be many, they just need to be better defined,

aligned and managed. For instance, every project has a process to define, capture and manage

requirements. The issue tends to be the robustness of these processes and the effectiveness

of their execution. In order to be effective, requirements should be documented, actionable,

measurable, testable, traceable, related to identified business needs or opportunities, and

defined to a level of detail sufficient for system design.

The reality is that the output of the requirements phase is often incomplete, ambiguous,

poorly documented and seldom testable, let alone traceable. Business Assurance Framework

provides specific assets in the form of processes and tools designed to sharpen project

component outputs so they are efficiently usable in the next stage of the lifecycle, in this case

the development of functional designs.

Likewise in other lifecycle phases, co-ordinated and structured use case development, model-

based testing, design reviews and other quality measures drive to ensure the integrity of work

in progress and better position these outputs as inputs into the phase that follows.

While the quality asset library of each project may be different, a core of measures by

development phase could consist of the following:

1. Quality in Requirements

a. Testability Specifications – provide the business teams with a clearly defined format

for each requirement to ensure that the required attributes are covered.

b. Requirements Validation – sessions constructed to enable the review of requirements

across the solution landscape and include leaders from design, build and test teams.

c. Bi-directional Requirements Traceability - In order to validate coverage of all

requirements in testing, these must be systematically mapped to test cases prior to

execution.

d. A effective Requirements Management Process

e. Requirements Ambiguity Testing (RAT) – Workflow-driven tools that use logic and

approvals to ensure requirements are clear and testable.

f. Defect Prediction – Processes and tools designed to statistically analyse past

development cycles and apply probability theory to predict type, severity and

complexity of potential defects. This enables mitigating strategies and sufficient

capacity to detect and resolve.


2. Quality in Design

a. User Story/Use Case Authoring - The creation of detailed, requirements-driven user

stories enable the development of functional requirements as well as test scenarios.

b. Evaluation (Reviews & Inspections) – The determining of the difference between

the actual properties and the required properties of an intermediate product and/or

processes in the development process.

c. Model-Based Testing - testing based on some form of a computer-readable model

that describes some aspects of the tested system in a way that enables automatic or

semi-automatic test generation.

d. Design Review – Well structured review sessions to validate design integrity,

completeness and integration.

e. Bi-Directional Design Traceability – creating systematic linkages between designs,

requirements and supporting test cases such that if any node changes, corresponding

changes of interconnecting solution elements are flagged.

f. Prototype Testing – Usage of use cases or modeling tools to validate designs prior to

commitment to code.

3. Quality in Build

a. Test-Driven Development - The practice of developing test cases immediately after

requirements have been defined and validated. These test cases then form a primary

input into the coding process such that code is written to pass the test case.

b. Code Structure & Test Coverage Validation - The usage of code analysis tools, which

validate all application tiers at the source code level and measure adherence to

architectural and coding standards.

c. Code Review – Peer review sessions to validate coding structures as well as validation

of technical architecture.

d. Code Profiling for performance and security – Early architectural reviews to assess

potential flaws in end state security and performance.

e. Release Management – Processes and tools designed to ensure that the code

releases to upper and lower environments is complete and synchronized.

f. Test Environment Management - Processes and tools designed to ensure that lower

environments are configured properly and are synchronized with the latest test data

sets.

g. Continuous Integration – the merging and testing of all working branches of code

daily.

h. Test Data Management – processes and tools designed to ensure that test data is

of high quality, is available and loaded to target environments, is consumed per plan

and is secure.

i. Services Virtualization – the usage of tool designed to virtualize unavailable technical

objects to remove constraint of early development and QA.


4. Quality in Test

a. End-to-End Testing – An approach that tests key end to end process streams to

ensure that the full solution functions per requirements across application, integration,

UI, reporting and database layers.

b. Common Test Tool Platform - Utilization of a common test tool for all projects in order

to maximize efficiency.

c. Test Automation - Usage of tools, which automate the test design and execution

process enabling efficiencies over manual methods and increased quality through

more frequent regression cycles.

d. Model-Based Test Design - The creation of a test model that describes some of the

expected behavior (usually functional) of the test object. The purpose is to review the

requirements thoroughly and/or to derive test cases in whole or in part from the test

model. The test models are derived from the requirements or designs.

e. Test Infrastructure in the Cloud - A ‘pay per use’ cloud-enabled Test Infrastructure

Service that can be accessed on demand, without the need of capital investment and

large-scale testing resources. Providing a comprehensive, easily accessible, flexible

and low-cost service for test execution.

5. Quality in Deployment

a. Operational Readiness Assessment – procedure to ensure to be deployed solution

is production ready. This entails all solution components including applications,

integration, network and data as well as non-solution areas like support and training.

b. User Experience Testing – An approach that validates the user experience is per

specifications in a Production environment.

c. Release Management - Processes and tools designed to ensure that the code

releases to production environments is complete and synchronized.

HCL’s concerted Business Assurance Framework quality program aggregates these methods

and many others under one umbrella providing enterprise projects with the tools, processes

and metrics to implement end-to-end quality without re-inventing the wheel.

Most critical in this orchestration is the standardization of quality measures and the alignment

with a Cost of Quality approach. Defined as the delta between what a defect cost to fix at

source versus the cost to resolve in Production, it is this single measure that should drive all

quality measures in the software delivery process because it demonstrates hard value driven

back into the business.


Resource Management Test Management Service Management Transformation Management

Managed Quality Services

Business Assurance Framework

Quality in Requirements Quality in Design Quality in Build

Quality in Test(Services &

Accelerators)Quality in Deploy

Bi-directional Requirements Traceability

Requirement Completeness Index

Requirements Review Non-functional risk

assessment Requirements

Ambiguity Testing (RAT)

Defect Prediction Requirements Stability Requirements

Completeness

Design Review Design Traceability Model based testing Application

understanding document

Model based design Prototype testing

Code Quality Analysis Code Review Configuration

Management Test Environment

Management Continuous Integration Unit Test Automation

Framework Cloud-based Testing Service Virtualization In-Process Automation Change based Testing

Functional Testing NexGen Automation Perf Engineering Service Virtualization Mobility Testing Cloud Testing Infrastructure Testing Model-Based Testing Test Accelerators Compliance Testing Validated Testing Risk-Based Testing Orthogonal Array UAT

Operational Readiness Assessment

User Experience Testing

OAT Release Management Resilience Testing

MQS with BAF enables at-source defect detection and avoidance. As a result, detection during testing declines and ultimately, testing becomes redundant.

Cost of Quality Defect-Free Deployments

Figure 4: Framework for Business Assurance Framework supports

HCL’s Managed Quality Services Solution


As consumers, we have grown accustomed to a high level of quality in

the products and services we buy. What started a half a century ago with

Deming has now evolved and permeated both manufacturing and services

sectors of industry. We live in an era in which vendors of inferior quality

products simply do not survive. Why should the products of enterprise

software development projects be any different? For those businesses

with the insight to survey the total picture, quality as a weapon to reduce

total cost is an opportunity definitely worth pursuing.

CONCLUSION


Robb La Velle Mr. La Velle is the Global Leader of the Business Assurance & Testing Service group at HCL Technologies and is based

in London. His 20 years of IT consulting have included clients and project locations as diverse as a global energy concern

in Cape Town, a multinational telecommunications company in Manila, a mining company on Borneo, a global bank in

Frankfurt, a major semiconductor manufacturer in Shanghai and the United States Army at the Pentagon.

He has managed major application quality assurance programs for many multinational clients including Deutsche Bank,

Chevron, AT&T, T-Mobile and Best Buy.

Mr La Velle holds a bachelor’s degree in Economics and German Literature from Universität Hohenheim (Stuttgart,

Germany) and an M.B.A. from the Université Libre de Bruxelles (Brussels, Belgium).

ABOUT THE AUTHOR

Hello there! I am an Ideapreneur. I believe that sustainable business outcomes are driven by relationships nurtured through values like trust, transparency and flexibility. I respect the contract, but believe in going beyond through collaboration, applied innovation and new generation partnership models that put your interest above everything else. Right now 105,000 Ideapreneurs are in a Relationship Beyond the Contract™ with 500 customers in 31 countries. How can I help you?

ABOUT HCL

About HCL Technologies

HCL Technologies is a leading global IT services company working with clients in the areas that

impact and redefine the core of their businesses. Since its emergence on the global landscape,

and after its IPO in 1999, HCL has focused on ‘transformational outsourcing’, underlined by

innovation and value creation, offering an integrated portfolio of services including software-led

IT solutions, remote infrastructure management, engineering and R&D services and business

services. HCL leverages its extensive global offshore infrastructure and network of offices

in 31 countries to provide holistic, multi-service delivery in key industry verticals including

Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare &

Life sciences. HCL takes pride in its philosophy of ‘Employees First,

Customers Second’ which empowers its 104,184 transformers to create real value for

customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$

5.8 billion, for the Financial Year ended as on 31st March 2015 (on LTM basis). For more

information, please visit www.hcltech.com

About HCL Enterprise

HCL is a $6.8 billion leading global technology and IT enterprise comprising two companies

listed in India – HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one

of India’s original IT garage start-ups. A pioneer of modern computing, HCL is a global

transformational enterprise today. Its range of offerings includes product engineering, custom

& package applications, BPO, IT infrastructure services, IT hardware, systems integration,

and distribution of information and communications technology (ICT) products across a wide

range of focused industry verticals. The HCL team consists of over 109,643 professionals of

diverse nationalities, who operate from 31 countries including over 505 points of presence in

India. HCL has partnerships with several leading global 1000 firms, including leading IT and

technology firms. For more information, please visit www.hcl.com

Documents

Moving Quality upstream - HCL Technologies · quality control. This reactive process of inspection entailed receiving each unit into the Quality Control Department off the line and