Embed Size (px)
Citation preview
Authors
Supriya S Nayak ([email protected])
Prashant Rajgopal Katti ([email protected])
Divya Murthy ([email protected])
Title:Applying Lean Techniques in Software Testing
13th Annual International
Software Testing Conference
India 2013
1 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
1. Abstract Lean methodology - one of the main principles of Toyota Production System (TPS) continue to garner
interest in the field of program management when it comes to software service industry. Lean
methodologies are mainly found to be applied in the areas of manufacturing industries leading to
improved performance in many cases yet failed in some implementations, the general notion of Lean
methodologies and its principles are mixed.
Software testing in general does not follow a defined assembly line processing as evident in
manufacturing industry. A simple task such as test plan creation or for that matter designing test cases
based on the business requirements may get hampered due to changing customer requirements, which
becomes difficult to determine the actual “Work In Progress” activities in Software Testing Life Cycle due
to its intangible nature. The success of Lean principles in software service industry in particular is mostly
an unexplored area and that is why most of the processes and the work culture in manufacturing industry
cannot be implemented directly in software service industry.
This paper will discuss the literature around Lean principles and investigates various challenges in
employing lean principles and its methodologies to software testing life cycle and propose a framework to
overcome the challenges and discuss the various opportunities of reducing the Non Value Added (NVA)
activities leading to reduced effort, time and a better return on investment.
2. Introduction to Lean Over the past century there has been tremendous progress in technology. Changing customer
requirements are expected to be fulfilled at much faster pace and hence time to market becomes crucial
for any product to be launched in the market. It’s the customers who like to see the real value in any
product that has been developed and Lean is a way to determine those values sought by the customers,
provide value through an end to end process by minimizing waste. The core principle of Lean is to
minimize waste by maximizing customer value.
Lean principles help to evaluate the most important processes in business and focus on minimizing
unnecessary costs, minimizing waste and improving the overall procedures to develop a product. Few
benefits of Lean are listed below
Problem identification
Increases business efficiencies by ensuring the employee time is spent on value adding activities
Reduce Costs and unnecessary overheads
Simplify the overall processes
Lean started out with manufacturing industry and now is slowly getting recognized in other industries as
well. Toyota Production System (TPS) and Lean principles helped resolve most of the issues in
manufacturing domain. Companies like GM, Starbucks, GM etc. have employed lean practices and are the
best example of Lean Organization.
2 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
James P Womack along-with Daniel Roos and Daniel T. Jones thoroughly described the thought process of
lean in the book “The Machine that changed the world” in 1990. Later seeing the success of TOYOTA,
there have been many articles, books, papers and various literatures that created an enormous demand
for lean knowledge. This resulted in leaders from other industries such as logistics and transportation,
healthcare, retail to take a serious note of lean benefits and started adapting the tools and techniques.
Work culture and processes to develop a product in Software service industry is different from that of
manufacturing industry, due to which these lean practices don’t seem to adapt in the software service
industry. However on the contrary the literature suggests that there are various Lean concepts that have
been applied and used for improving software development and not software testing in particular.
This white paper focuses on studying the challenges in implementing lean tools and practices in software
testing. For the benefit of the study, the factors that lead to lean's success in manufacturing domain will
be analyzed.
2.1 Journey of Lean Henry Ford was the first person to really understand the entire production system and in 1913, created an
innovative assembly line for which was called as Flow Production. Ford’s revolutionary breakthrough lined
up fabrication steps in the process sequence by installing Go/NO GO gauges to ensure no faulty
components flow in the production. This innovative assembly line was a great success in mass production,
however was limited to a single model of the car Model-T and could not handle different variations.
Ford lost its way when the world was looking for variety and during this period, other auto makers
stepped in and responded to the demand by providing many options for each models. However the
throughput time to get the models to market was still huge and only improvement were seen in the
fabrication shops with larger and larger machines that ran faster and faster resulting in reduced costs per
process steps. During 1930, Taiichi Ohno and Shigeo Shingo at Toyota visited Ford’s original thinking and
recognized that the ford system had contradictions and various shortcomings particularly with the
employees, they created Toyota Production System (TPS).
Ohno and Shingo soon discovered that that factory employee had more to contribute than any other
which resulted in the discovery of Quality Circle movement that cultivates team development and cell
manufacturing. In similar fashion a series of simple innovations helped built the continuity in the process
flow and easily offer wide variety in products. Toyota could not have achieved success if they tried
implementing what American companies implemented. Toyota's excellent thought leadership that
realized the need to tweak certain aspects before implementing the assembly line process in Japan as the
Japanese work culture was very different from American market and industries.
In the book “The Lean Thinking” published by James P. Womack and Daniel T. Jones, mentioned the
following lean principles
Value: Have a clear understanding of what value is to the customer and specify it
Value Stream: In a process, evaluate the value stream for a process and identify various non-
value added activities
Continuous Flow: Ensure the product flow is continuous in line with value added activities
Pull: Ensuring the components are built only when there is a demand
Reduce time : Manage lean towards perfection so that the time taken to serve the customer nee
reduces continuously
3 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
3. LEANWIRE framework Scope of lean concepts is carried from manufacturing firms to various other domain and software firms
too are inclined towards going lean with no exception. However, the difference in work culture and
process between these two domains makes it difficult to envision applying lean concepts to software
firms. This section proposes a framework to apply lean in any software cycle and this paper deals with
software testing cycle in particular.
Figure 1: LEANWIRE Framework
LEANWIRE framework provides a step wise approach for implementing lean in software projects. By
combining lean concepts with software process, it not only makes the lean implementation process
simpler but also aims to keep the change process to be effective andefficient. Different phases of
LEANWIRE framework and its relevance to software testing is discussed in subsequent sections.
3.1 LOOKOUT
As the first step, lookout for pain points in the system and process or activities that do not add any value
(waste). 3Ms concept of Lean help us identify the pain points and waste in the system.
Muda or waste is any activity that consumes time/effort/money but do not add any value to
customer
Mura or variation in operation or process
Muri or strain resulting in overburden of resources and facilities
LOOKOUT for painpoints and wastes (non value add) activities and processes in testing life cycle
ESTABLISH the list of activities to be considered to increase the overall value add and waste removal
APPRAISE and assess the system and address the painpoints to build the future state
NOTIFY the team about the proposed changes through suitable training and collect feedback
WORKOUT the possible solutions and identify the metrics to be measured
IMPLEMENT the various tools defined in WORKOUT phase and implement the future state
Review and collect feedback on the new system and review the metrics
EVOLVE to achieve continous improvement by working on the feedback
TIMWOOD,
Fishbone, VSM
Pareto Analysis,
SWOT Analysis
FSM
Andon, Kanban,
Poka Yoke
Kaizen
Training
Measure Metrics
Implementation
4 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
In terms of Lean, waste can be of 7 types, commonly known as TIMWOOD – Transportation, Inventory,
Motion, Waiting, Over processing, Over production and Defects
In a similar fashion, Mary and Tom Poppendieck translated these wastes in to “The seven wastes of
Software Development” - Partially Done work, Extra Features, Re-learning, Hand-offs, Delays, Task
Switching and Defects.
Below are the various wastes in terms of software testing:
Transportation
In testing, this can be termed as some information that is supplied to the testing team, be it requirements
document and any test specific information. If a crucial information is not passed to the testing team in
time, the amount of time spent later to gather the same information is a wasteful activity.
Inventory
From testing perspective, inventory can be considered as our valuable testing resources/individuals.
Testing resources that are not utilized to its full extent are a waste to the organization. It’s important to
have right people at the right time of the project timelines to ensure seamless project delivery.
Motion
It is often seen that the documents hand-offs that happen from the business to developers / testers tend
to lose some amount of knowledge in transition. Gaining the lost knowledge involves re-work; discussion
with the stakeholders etc. is invariably a waste in the testing world. And also another analogy that can be
brought in here is the number of releases that are planned to achieve the desired quality for a product.
Waiting
Much of the information that is provided to the test team moves across various teams before being
provided to the test team. Most often proper version controlling is not followed and the test team
continues working on the older version of the design doc/ requirements doc only to realize it later and
rework on all the scripts created thus far. Also, critical information never reaches the test team and the
time spent to get the information or re-clarify the same is wasted effort.
Quite often we have many of the test resources
- sitting idle due to lack of environment, waiting for defect fixes etc., - not having enough work to keep them busy for the entire day - waiting for tests to be completed by the tool used for completing the execution ( Automated/
Performance testing) - improper utilization of skillsets; niche skilled testers used for manual functional testing
Over processing
The rationale behind retesting all the test cases whenever there is a change to the system has never been
proven. Minimal or no thought is put into analyzing the impact of the change made during the testing
cycle. As a result, test teams either ends up executing ALL the test cases in the test pack adding no value
to the project or not creating the right test pack.
5 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
In one’s urge to showcase what is being done, much of the testing effort is lost in creating reports for
various stakeholders from the development team to the program anchors. Many of these reports are only
skimmed at the surface and very few add value to the testing team and the program.
Over production
Very often there is a tendency to create extensive test cases with an intent of being able to prove that no
stone has been left unturned as part of testing and also to some extent provide cover in case of defect
slippages. In this process, the testing team ends up creating an over exhaustive list of test cases many of
which are totally unproductive and are a waste of effort and time.
Defects
The general tendency is to involve the testing team very late in the SDLC lifecycle. Quality is most often
treated as a postscript and involved at the far end of the project. This results in:
- Lack of clear understanding of what needs to be tested and how it has to be tested - Improper understanding the requirements due to lack of time - Identification of defects late in the lifecycle impacting the cost and time. - Limited proactivity form the testing team resulting in creation of voluminous test cases packs.
3.2 ESTABLISH
As a next step, once the team is clear about the various wastes that can happen in the software testing
life-cycle, identifyand prioritize the list of pain points pertaining to the specific testing project. All the pain
points identified in the LOOKOUT phase need not be addressed in one go while implementing lean.Lean
should be implemented in phased approach and hence as part of ESTABLISH phase, establish the list of
pain points you need to address using Lean concepts. This will probably require detailed analysis with
respect to the pain points in the testing process, then identify and prioritize suitable pain points
applicable in the testing process that can help reduce the waste. Once the list of activities are identified,
the team is expected to work and mine more data for that particular pain out and gather the required
information so that could help in implementing the lean concepts (discussed in later sections) to reduce
waste and maximize the customer value.
Tools such as Pareto analysis and SWOT analysis can be extensively used to have a clear understanding of
the pain points in each area.
3.3 APPRAISE/ASSESS The identified and shortlisted pain points will have an impact on the value stream of the testing process
inducing delay, wait and other wastes. A value stream map helps in analyzing and assessing the flow of
material and information in the process of building the product or providing a service. In Appraise/Assess
phase of LEANWIRE, Lean implementation team needs to appraise and assess the current system and
come up with a Current State Map (CSM) depicting the process and highlighting the wastes.
A typical current stream map for testing process is shown in the below diagram which depicts the flow of
testing process. The purpose of LEANWIRE will be to transform the current state to a desired future state
by reducing non value add activities and waste.
6 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
Figure 2: Current State Map
3.4 NOTIFY In this phase, the lean implementation team should be looking for initiatives to involve people, team and
the complete workforce in the knowledge sessions explaining the benefits of lean implementation and
how this could help everyone to minimize waste in their day to day activities. Lean transformation
obviously provides sustainable results by increase in productivity, innovation and helps the organization
learn the flow of problem solving as well as understand the process flow to minimize wastes.
In order to ensure the employees understand the importance of lean implementation, a proper change
management has to be put in place that takes care of revealing the actual problems to the staff using
standards/practices, involve employees in every decision, share the clear vision of the project or
organization goal in understandable terms.
John Kotter (1995) devised the 8 steps to change which could be easily employed in case of lean
implementation in software testing
Create a sense of need
Form a support team
Develop a vision for Change
Use proper communication channels to spread the idea of the Change
Remove Obstacles
Create Short-Term goals and targets
Start building on the Change
Make the process as a habit and keep on improving continuously
3.5 WORKOUT In this phase, the lean implementation team should look for optimal solutions to implement the desired
future state, which would address the established pain points. Lean concepts suggest various tools to
eliminate the waste or pain points like 5S, Andon, Kanban, Poka Yoke, Jidoka etc Implementing these tools
7 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
in software firms need careful analysis of the pain point and thought process on the suitable tool to
address the issue effectively. The selected tool should always aim not to impose additional overhead on
any of the stakeholders from employees to customer and effectively remove the non-value add from the
system. The following discussion provides an overview of various lean tools and how these can be used to
address software testing pain points.
3.5.1 5S From a manufacturing point of view 5S is a method of organizing the workplace or a manufacturing unit. It
ensures that the member who is working on a component or a unit in manufacturing plant is well
organized and neat. From software testing perspective, any method that is well established can be really
challenging to apply in software testing field.
Sort: This can be achieved by going through the list of requirements and mapping them to the identified
test cases and removing the unnecessary ones which do not add any value. Remember, it’s not about
quantity of test cases, it’s about quality of test cases
Straighten: Now that the test cases of utmost quality are identified, it’s time to ensure they are properly
organized in a systematic way. It could be in the form of a test management tool or any in house tool that
can help maintain and organize the test bed in an effective way.
Shine: Ensuring that the test bed is easily manageable and searchable. If at any given point in time, a
need arises to find a test case, it should not take more than 5 minutes to locate it. For this to happen,
proper naming conventions, standards and templates are to be followed to maintain order of multiple
test cases across releases.
Standardize: Most of the organizations work on the standards, if not commonly known ones. And it’s not
about the test cases; it’s about every document related to testing that follows a specific standard to
minimize the mistakes.
Sustain: Once the above 4S have been established, they become the new way to operate the same old job
in a much effective manner. Now it’s all about sustaining these activities and ensuring that there is no
slightest inclination towards the old way of doing things.
3.5.2 Andon
Andon in Japanese means light. Toyota implemented andon signals in their car assembly lines to notify
issue in any of their system or equipment and raise a flag, so that all the related or associated processes
could be stopped to avoid defective production and grab supervisors’ attention to the right spot on the
shop floor. Andon could be as simpler as a simple light signal to an andon board highlighting the issue in
the big picture. A similar concept can be implemented in software testing life cycle to address few of the
pain points.
One of the pain point in any software testing is the amount of effort put into creating various reports. A
typical testing team would have various reports like daily status report, weekly progress report, monthly
report, quarterly report, defect report etc If we take a deeper look in all these reports, we would see that
90% of the data would be the same and only change would be the way they are published, keeping the
intended audience in mind.
On similar lines to andon, we can visualize a dashboard which can be a one point stop for all the data
available from these reports. This dashboard can be plugged with the test management tool to retrieve
8 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
the data with the help of a macro or a piece of code. A single click can get the necessary data to the
stakeholders.
Figure 3: Andon Dashboard
Environment availability is another pain point in testing where testing might get impacted due to
environmental issues like test service being down or issues in network connectivity. Such issues would
need immediate attention from the environment support team which in most of the cases consumes lot
of time in communication and follow-up.
An ANDON dashboard can be used for quick alert and attention to avoid rework and faster issue
resolution. Below is an example of ANDON dashboard for the test environment when there are no
environment issues. This dashboard displays all the test applications and their connectivity to backend
systems through middleware components.
9 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
Figure 4: Environment Andon with no ALERTS
Whenever there is an issue in any of the components, the ANDON dashboard should be designed to raise
an alert highlighting the impacted systems as shown in the below diagram.
Figure 5: Environment Andon with ALERTS
Highlighting the impacted system through andon dashboard will
• Help testers stop testing the impacted system to avoid any invalid executions and rework
• Alert the respective environment support team to measure the impact and hence set right priority in
fixing the issue
3.5.3 Kanban Kanban or demand scheduling plays a major role in implementation of ‘Just In Time’ production. A kanban
system is “a production control system for just-in-time production and making full use of workers’
capabilities” (Sugimori et al., 1977). Kanban means ‘signboard’ in Japanese. In Toyota Production System,
the kanban board would provide visual signals to operators as to the amount of job they need to run or
when they need to stop (Gross and McInnis, 2003). In Kanban system, work will be pulled from upstream
system when downstream is ready for a task. Kanban is proved to improve the flow by reducing inventory
and prevents overproduction (Sugimori Y et al., 1997).
The kanban concept of demand scheduling can be used to address few testing pain points like
Wait time between various test processes functional planning and automation script preparation
Resource management
Consider the below process within test planning phase
10 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
Doing Done Doing Done Doing Done
HLTP DLTP Scripting
KANBAN BOARD (Before)
Doing Done Doing Done Doing Done
KANBAN BOARD (After)HLTP DLTP Scripting
Improper planning could lead to overburden on Automation scripting resources which can impact quality
of the scripts. By implementing demand scheduling, we can plan for effective resource utilization in
advance to meet the timelines.
Figure 6: Kanban Board
3.5.4 Poka Yoke Poka-Yoke is a quality assurance process introduced by Shigeo Shingo. This term is nothing but a Mistake
Proofing technique. The main purpose of introducing Poka Yoke is to reduce mistakes that lead to defects
in the systems.
In Software Testing perspective, Defect prevention and Defect Detection can be considered two
categories for Poka Yoke. Defect prevention mechanism can be applied in early development stages to
ensure the defects are found early in the life-cycle and are not carried to the later stages of the cycle.
Defect Detection is common approach that the testing team employs using various strategies and
approaches. However to ensure test team do not make any mistakes during testing, one can think of using
standards and templates for test casedesign, defect management etc. Also Poka Yoke can be best applied
in case of everyday communication that we use in the form of email; A simple check/rule in the outlook
will ensure that the email that is sent do not have any mistakes in the form of grammar, attachment
missing, important recipients etc.
3.6 IMPLEMENT In this phase, it’s all about applying the LEAN concepts and implementing the Future State. Future State
Mapping is a term used in value stream to identify the various stages and the revised cycle time post
application of lean concepts. By working out the possible solutions to reduce non value add activities and
eliminating waste, a desired future state should be designed and implemented.
High Level Test
Planning
Detail Level
Test Planning
Automation script
preparation
10 cases per day 7 cases per day 3 cases per day
BOTTLENECK
11 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
Figure 6: Future State Map
In the sample future state map, there is an effort to have parallel manual and automation test planning. If
a firm wish to view this in their future systems, sufficient planning and approval need to be done in the
workout phase like attaining expertise in progressive automation, discussion with necessary stakeholders
like the customer project management team and the development team etc. In implementation phase,
team need to acquire proper tools and skillsets to implement the plan.
3.7 REVIEW This is one of the important phases, where the team has to sit down and review the Lean process
upgraded in the testing life-cycle. Each and every process area has to be reviewed for its benefits,
advantages with respect to the lean concepts applications and whether it is really worth by comparing it
with the previous processes. Collecting feedback from the team who worked on this change is also
important as it will be helpful to gauge the differences in understanding the lean implementation and
how best it can be improved further.
The whole ideas of this phase is to collect data, feedback, perform a peer review, and verify if applying
lean concepts to testing life-cycle really made sense.
3.8 EVOLVE The fruit of successful lean is not always evident immediately after implementation. In software industry,
there is a tendency to reflect the benefits of any new process in the quarterly results. But this do not hold
Scenario
Identification
CT – 30min
per scenario
= 1
Detail Test
Planning
CT – 45min
per test case
= 1
Automation
Scripting
CT – 3 hour
per test case
= 4
Functional
Execution
CT – 30 min
per test case
= 2
Regression
Execution
CT – 15 min
per test case
= 1
Weekly Weekly
Total CT
= 5 hrs
Total wait =
11 days
Future state map for
a typical testing life
cycle
Weekly
30 min 3 hr 45 min
1 day 10 days
45 days
5S
Report
dashboard
Kanban
board
12 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
good for lean concepts. According to “Toyota way”, it requires 6 years to experience any benefits of
implementing lean. Unlike other process improvements, lean builds a culture in any organization to
reduce wastes and increase the value. In Evolve phase, team need to work towards the feedback received
and look for continuous improvement by looking out for other pain points.
4. Conclusion Lean in itself builds a culture of reducing waste and increasing value-add in the process of delivering any
service. The success of Lean practices in service industry is mostly unexplored and attempts in various
areas to implement Lean in service industry have failed miserably. Work culture and processes in
Software service industry is very distinct and different from that of manufacturing industry, due to which
one cannot implement the lean practices from manufacturing industry as it is in software service
industries. Unlike manufacturing industry, tasks in software industry do not follow a defined assembly line
process. A particular task, may it be software development or software testing, may very well change in
the middle of the life cycle due to changing customer and business requirements. At any given point of
time, one cannot accurately measure the “work in progress” tasks in a software life cycle since most of
the intermittent materials are not tangible. This also makes it difficult to measure the scrap generated in a
particular software life cycle. Such issues in software industry pose a major challenge in implementing
lean best practices.LEANWIRE framework considers all the points necessary for a successful lean
implementation in software industry by setting the right mindset. Increasing value add in all the activities
in turn will also increase client value leading to higher customer satisfaction. Rather than performing the
various improvement activities in silo, LEANWIRE provides a platform to align the activities in an
organized way, making the change process easier.
13 Applying Lean Techniques to Software Testing
Date 16 Sep 2013
5. References
1. Fujimoto T, 1999, The evolution of manufacturing system at Toyota, Oxford university press, New
York
2. Gross J M, McInnis K R, 2003, Kanban Made Simple, AMACON
3. Hibbs, C., Jewett, S., Sullivan, M., 2009. The Art of Lean Software Development: A Practical and
Incremental Approach. O’Reilly Media, Inc.
4. Kerievsky J, 2012, Lean Startup: Why it rocks far more than agile development, GOTO Conference
Aarhus 2012, www.youtube.com/watch?v=V5p8m1ljJoA, Accessed on 15th
June 2013
5. Liker K, 2003, The Toyota Way, McGraw Hill Professional 2003
6. Poppendieck, M., Poppendieck, T., 2003. Lean Software Development: An Agile Toolkit. Addison
Wesley, Boston.
7. Robinson H, 1997, Applying mistake-proofig to software, paper presented at the Sixth
International conference on Software Testing Analysis and Review (STAR’97), Accessed from
http://www.mistakeproofing.com/software.html
8. Shingo S, 1985, A study of Toyota Production System from an Industrial Engineering viewpoint,
Productivity Press 1985
9. Womack P, Jones T, 2003, Lean Thinking, Free Press July 2003
