Agile Test Automation-GM4b-abridgedxunitpatterns.com/~gerard/NonTechTestAutomation.pdf · 2015-06-26 · • How do agile principles & values support automated testing? • What is

1

1 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial

Agile Test Automation StrategyFor the Non-technical

Gerard Meszaros

[email protected]


I.T.

EmbeddedTelecom

My Background

Gerard [email protected]

•Software developer

•Development manager

•Project Manager

•Software architect

•OOA/OOD Mentor

•Requirements (Use Case) Mentor

•XP/TDD Mentor

•Agile PM Mentor

•Test Automation Consultant

•Lean/Agile Coach/Consultant

2


Learning evolves; paper doesn’t

• Michael Bolton

– (the tester, not the singer!)

• Slides may be different from what you have printed off. Sorry!

– Latest version at:

– http://agile2010.xunitpatterns.com

– http://Testing2010.gerardm.com


Know Thy Audience• Who’s ... Already involved in an agile project?

• ... Just starting with Agile?

• ... a Product Owner / Manager?

• ... a Project Manager?

• ... a Tester or Test Manager?

3


Know Thy Audience (2)• Are you ... automating to find bugs?

• ... automating to prevent bugs?

• ... providing acceptance tests before the software is built?

• ... convinced you need automation?

• Do you have ... an automated func’l test suite?

• ... “complete” automated test coverage?

• ... “fragile tests”?


Common Questions

Why?

How Management?

How Technical?

• Why do I need to be concerned with Test Automation?

• Why do I need to be more concerned with Agile (than Waterfall?)

• Can all testing be automated?

• How does test automation fit within an overall test strategy?

• Why do I need to tell the development team I expect automated unit tests?

• What kinds of test automation do I need?

• Who needs be involved in preparing Functional tests?

• How does automated testing support agile principles and values?

• How do agile principles & values support automated testing?

• What is the Fragile Test problem and how do we avoid it?

• What does it take to be successful with agile test automation?

• What are the common approaches to test automation?

• What Tools should we be using?

• How do we choose which tool to use?

• What about legacy code? How do we test it?

4


Agenda

• Why Do We Need Test Automation?

– What’s Wrong with the status quo?

– How much is it costing?

• Automation and / or Traditional Testing?

– Changing the Role of Tests

– Automated vs. Manual Tests

– How is agile approach different?

• How to Automate Effectively?

– How can we avoid Fragile Tests?

– What are Common Approaches to Test Automation?

– What is the Role of Unit Testing?

• Agile Values & Test Automation

• Conclusion


Q: Why Do I need to be concerned with Test Automation?

• To keep software soft. To get the product you expected.

• To make your expectations about the product clear.

• Iterative & Incremental Development has large impact on the amount of testing to be done. Automation makes it manageable.

5


Product Owner Goal

• Goal: Maximize business value received

Features Money

Product Owner

Concept

Minim

ize

Maximize

Quality is Assumed; Not Managed


Why Quality Often Sucks

• Iron Triangle of Software Engineering:

You can fix any three; the fourth is the outcome

Resources

Time

Functionality

• What about Quality?

Quality

6


In Agile, we “Pin” quality

using automated tests

Why Quality Often Sucks

• Iron Triangle of Software Engineering:

You can fix any three; the fourth is the outcome

Functionality

Quality

• What about Quality?

Resources

Time


CHAOS 2004 – Resolution of Projects

7


CHAOS 2004 – Project Success Trend

Copyright © 2004 – The Standish Group International, Inc.

Challenged

Succeeded

Failed


CHAOS 2004 – Average % Cost Overrun

8


Is This Good Enough?

Who thinks this is good enough?

Who would let this doctor operate on you?


History of Medical Sterility• 1844 Ignaz Semmelweis noted midwife

wards had 1/3 the infection rate of doctors wards

• Deduced due to better cleanliness habits. Proposed rigorous hand washing before surgery.

• Actively resisted by doctors despite much better results.

• 1865 Joseph Lister requires hand-washing in his wards. Adopted after much controversy.

• 1920 Doctors still wear blood-spattered aprons in operating room.

• Today, sterile technique recognized as mandatory. Still requires reinforcement:

9


Would you ...

... ask your doctor to reduce the cost of the operation ...

... by skipping the sterile technique ?

Test Automation is like hand washing:Improved results but at a cost.


Minimizing Cost of Product

Total cost includes:

• developing the software

• verifying the newly built functionality

• verifying old functionality still works

• fixing any bugs found

• Verifying noting was broken by fixes

Agile Test Automation can reduce the cost of all of these activities.

10


Overall Strategy

• Avoid Defects by Providing Acceptance Tests Up Front

• Reduce Cost of Fixing Bugs by Finding Defects Earlier

• Find Defects Earlier by Testing Earlier

• Test Earlier by Developing Incrementally

• Pin the functionality using Automated Tests

• Reduce the Cost of ReTesting by Automating Acceptance Tests

• Reduce Cost of Fixing Bugs thru Automated Unit Testing

• Make Test Automation Cost-Effective thru Design for Testability


Preventing Defects

• Type NF bugs: New Func. is wrong

• Type RB bugs: New bugs in old func. (Regression Bugs)

Concept

Version 1.0

EvolvedConcept

Initial

NF

Version 1.xVersion 1.0

RB

NF

11


Test&Fix Cycles

DevelopmentVerification & Acceptance

Requirements

B6! Missed!

Requirements

Are you Playing Battleship™?


What is the cost of NF Bugs?

• Late discovery of bugs results in Delay

1. Missed Sales due to missing Market Windows

2. Delayed Revenue

3. Damaged Reputation

4. Increased development & testing costs

Calculating 1-3 depends heavily on your business

#4 can be calculated based on the delay

12


S/W Development Value Stream

What is this costing us?

• In delayed delivery?

• In extra effort?

Write Code

Insert Defect

Deploy to QA

Test Code

Log Defect

Defect List

Write Code (inserting defects)Fix -----


Wasted Time – Delayed Delivery

• Imagine it takes 2 test cycles to fix enough bugs to release– 2 weeks to fix bugs

– 1 week waiting to deploy

– 1 week retesting

• Delivery is delayed by 8 weeks!– Benefit: 20% cost reduction on $10m/year

– Cost = 8/52 * .2*10m = .15*2m = $307,700

Defect List

Write Code

Insert Defect

Deploy to QA

Test Code

Log Defect

Write Code (inserting defects)

Avg: 2 weeks

Avg: 1 week delay

Avg: 1 week

Fix -----

13


Wasted Effort – Per Bug

• Imagine we have 100 KLOCS of code delivered with a defect rate of 10 bugs per KLOC.

– 1000 bugs; 12 hours each to find, fix and retest

=12,000 hours of wasted effort

=1500 person days

= 7.5 person years @ 100K / person-year = $7.5m

Defect List

Write Code

Insert Defect

Deploy to QA

Test Code

Log Defect

10 per KLOC


1 hours per bug8 hours per bug

3 hours per bug


Wasted Effort – Test Cycles

• Imagine we have 100 KLOCS of code delivered with a defect rate of 10 bugs per KLOC.

– 30 person team

– Test&fix duration of 50 work days (5 cycles @ 10 days)

=1500 person days

= 7.5 person years @ 100K / person-year = $7.5m

Defect List

Write Code

Insert Defect

Deploy to QA

Test Code

Log Defect

10 per KLOC


1 hours per bug8 hours per bug

3 hours per bug

14


Exercise: How Much are NF Bugs Costing You?How much delay?

1. Current number of test&fix cycles: _____

2. Duration of typical test&fix cycle: _____

3. Current test&fix delay (1 x 2) : _____

4. Reduced number of test&fix cycles: _____

5. Saved test&fix cycles (1-4): _____

6. Total reduction in delay (2 x 4): _____

Saved cost of delivery

7. Cost of team per day: $ _____

8. Actual cost of delay (5 x 7): $ _____

Cost of delay:

– Lost Revenue/Benefit due to market window _____

– Delayed R/B due to delayed delivery _____


What is the cost of Regression Bugs?

• Cost of finding the bugs (regression testing)

• Cost of fixing the bugs

• Cost of fixing goes up the longer the time between introducing the bug and finding it

• Ways to reduce this interval:

– NF Bugs: Incremental Acceptance Testing

– Regression Bugs: Automation helps

15


Exercise: Cost of Regression1. How many times per year do you need to do

regression testing? _____

2. How much elapsed time? _____

3. How much effort is involved? _____

4. Total delay (1 x 2): days _____

5. Total cost (1 x 3): person-days _____

6. Suppose you reduced the iteration length (say, 2 weeks?) What would it cost now?

(n times 5) person-days _____

7. Could you include it in your daily build? ____


Agenda













16


Q: How does test automation fit within an overall test strategy?

• Acceptance Tests are used to “Pin” the functionality we want to preserve.

• Automation is used to reduce the repetitive work involved in verifying the functionality of each successive build of software.

It does not replace:

• Exploratory testing in which testers try using the software and look for unexpected problems.

• Usability testing, in which real users are asked to complete typical task

Use machines for what they are good at so people can focus on what they are good at!


Traditional Role of Testing

Thanks to Brian Marrick and Mary Poppendieck

Shigeo ShingoCo-inventor of

Toyota Production System

Inspection to find

defects is Waste

Inspection to preventdefects is essential

Acceptance Tests

Regression Tests

Usability Tests

Exploratory Tests

Unit Tests

Component Tests

Property Tests(Response Time,

Security, Scalability)

ReportCard

Critique Product

FunctionalityUsabilityScalabilityResponseAvailability

BCABC

Business

Facing

Technology

Facing

GGM37

Slide 32

GGM37 Add "Pin" to Regression Tests and Unit TestsGerard Meszaros, 07/08/2010

17


Changing the Role of Testing

Acceptance Tests

Regression Tests

Usability Tests

Exploratory Tests

Unit Tests

Component Tests



Business

Facing

Technology

Facing

Prevent anticipatable defects from happening

Find non-anticipatable Defects, ASAP!


Critique Product ReportCard


BCABC

Define ProductRequirements

SoftwareDesign


Prevention: - Building the Right Product

What the customer thought they wanted

What the customer actually asked for

What the customer realized they actually needed

What development thought the customer asked for

What development actually built

What testing thought the customer asked for

What testing actually tested for

18


Building the Right Product

• How do we eliminate the waste caused by building the wrong product?

– Missed requirements?

– Misunderstood requirements?

– Unneeded functionality?


Building the Right Product

• How do we eliminate the waste caused by building the wrong product?

– Missed requirements?

– Misunderstood requirements?

19


Example-Driven Development

• A.K.A.

– Acceptance Test Driven Development

– StoryTest-Driven Development

• Concrete examples flesh out requirements

• Testers flush out missed scenarios

• Examples/Tests are executed as

product is built


Life Cycle of an Example / Test

User Goal

Feature

StoryTitle

Story Narrative

Story Scenarios

Story Example

Executable Example

Satisfied Example

New!

FormalizationAutomation

ProductDevelopment

Make ConcreteAdd Data

DefineAcceptanceCriteria

20


Knowing the Score Earlier, Always

• Development needs feedback

– How are we doing?

– What works, what doesn’t

• The sooner development gets feedback, the less effort gets wasted:

– Building more software based on wrong assumptions

– Trying to remember how the software works

» Because it’s been weeks or months since I wrote it

– Debugging the software

» Because I can’t remember what/when I changed it to work that way

GGM34


Reducing the Cost to Fix Defects

• Why?

• We can remember where we put the newly inserted defect because

1. We know what code we were working on

2. The design of the code is still fresh in our minds

• We may have to change less code

– Because we wrote less code based on the defect

Cost to understand and fix a defect goes

up with the time it takes to discover it.

Slide 39

GGM34 Merge with previous slide?Gerard Meszaros, 06/08/2010

21


41

Continuous Acceptance Testing!

• Defines what “Done Looks Like”

– Several to many tests per User Story / Feature

Write StoryTest

Write StoryTest Build Code Test Code Test Story

Build Code Test Code Test Story

• Tests executed as soon developer says “It’s Ready”

– End-of-iteration: OK

– Mid-iteration : Better


Readiness

Assessment

42

Continuous Readiness Assessment!

• Defines what “Done Looks Like”

– Several to many tests per User Story / Feature

Write StoryTest

Write StoryTest Build Code Test Code Test Story

Build Code Test Code

• Executed by developers during development

– To make sure all cases are implemented

– To make sure it works before showing to business

Test Story

• Tests executed as soon developer says “It’s Ready”

– End-of-iteration: OK

– Mid-iteration : Better

Manual Tests: OKAutomated: Better

Reduces # of testCycles;

Saves your time!

22


What Do You Need to Do?

• Break work (functionality) down into small chunks that can be developed and tested incrementally

– Feature or Product Backlog

• Provided detailed examples of how it should work for each chunk

– Acceptance Tests

• Plan to verify each chunk as soon as it is finished

– May require more resources earlier in project

• Make it clear that Development is expected to run all the tests before delivery for testing

Ideally, automatable

Encourages automation


Q: How is the agile approach to test automation different?

• We automate the tests for a different reason

– Defect Prevention vs Detection

– To communicate requirements

– To “Pin” the functionality once it’s built

• We automate the tests a different way

– We don’t rely on GUI-based automation

– Many different kinds of tests

– Using tools that support collaboration & communication

» in addition to confirmation

• We plan the automation based on ROI

– Goal isn’t: 100% automation

– Goal is: To maximize benefit while minimizing cost

23


Success Factors

Business

Not Willing or Able

Willing & Able

Te

ch

nic

al

Not Willing or Able

Willing & Able

New!

Failure to Automate

Automate the Wrong Tests or Too Late

Success Possible

Won’t Even Try


Maximizing Test Automation ROI

• Need to Treat Automation as an Investment

• Need to Prioritize / Triage Which Tests to Automate

• At least 3 Approaches to Choose From:

– Traditional QA-Based Automation

– Collaborative Critical Path Automation

– Collaborative Selective Automation

New!

24


QA Automation After Dev Complete

A.K.A. Traditional Approach to Automation

Summary:

– Done by QA Department (i.e. Testers)

– After Product is Built

– Typically done using (C)OTS Record & Playback tools

Issues:

• Too Late for Defect Prevention

– Tests aren’t available to development team

• Too Late to Ensure Easy Automation

– System not Designed for Testability

• Tools Create Fragile Tests

– We’ll Discuss this in Detail Later

New!


Collaborative Automation on Critical Path

A.K.A. Dogmatic (A)TDD Approach

Summary:

–Goal: 100 % automation

–Automate Tests Before Building Functionality

» Test automation task in each Story

Issues:

• Some Tests are MUCH Harder to Automate

• May Increase Costs and Delay Benefits of Functionality

• May Cause EDD to be Abandoned

New!

25


Collaborative Automation based on ROI

A.K.A. Pragmatic Approach

Summary:

– Goal: Just Enough Automation

– Apply Agile Priniciples to Implementation of Automation

Issues:

– Won’t Have Complete Test Coverage

– Can Lead to Automation Being Dropped in Favour of More Functionality

– Requires a Disciplined Product Owner, or,

– A Fixed Budget for the Automation

New!


Life Cycle of an Example / Test

User Goal

Feature

StoryTitle

Story Narrative

Story Scenarios

Story Example

Satisfied Example

New!

Executable Example

FormalizationAutomation

ProductDevelopment

Make ConcreteAdd Data

DefineAcceptanceCriteria

26



• Ask Test Automaters for Estimates of Effort

– For each different kind of test

• Ask Testers for Estimates of Impact of Not Having Automation

– For each different kind of test

– Effort to Run Tests Repeatedly

– Risk / Impact if Tests Weren’t Run Often Enough

• Prioritize Automation Stories Based on ROI

– Highest Impact to Effort Ratio

• Ensure that High ROI Automation gets Done

– Don’t Let New Features Crowd Out Essential Automation

New!


Short ExerciseGGM19

Slide 52

GGM19 Devise an exercise.

Maybe a group exercise:Identify possible issues with doing ATDD.Discuss how you could surmount them.Gerard Meszaros, 03/08/2010

27


Agenda














Typical Test Strategy #1

• Large numbers of (possibly automated)

“customer” tests

• Very few if any automated component or unit tests

SystemTests

UnitTests

Manual or Record & Playback

Typical when testing (& automation) is “QA’s job”

QA Automation

ComponentTests

TypicallyManual

TypicallyManual

28


Typical Test Strategy #2

• Large numbers of automated unit tests

• Some component tests

– Business Rules

– Technical Components

• Much fewer automated “customer” tests

Unit Tests Hand-Coded

All must be runnable by developers

Manual or automated

SystemTests

Agile (A)TDD

Component TestsHand-Written


Anatomy of an Automated Test

Our System

Other System

Other System

InterfaceBusinessLogic Database

Container Services

Other System

Test

Test Scenario Name-----------------------Preconditions----------------------1. Do Something2. Check Something-----------------------Clean Up

1&2 May be repeated

Fixture Setup

FixtureTeardown

Fixture(state)

Fixture(adapter)

Given...

When ...Then ....

29


Q: What makes automated tests fragile?

What, when changed, may break our tests accidentally:

– Behavior Sensitivity

» Business logic

– Interface Sensitivity

» User or system

– Data Sensitivity

» Database contents

– Context Sensitivity

» Other system state

Our System

Other System

Other System


Container Services

Other System

Test


Our System

Other System

Other System


Container Services

Other System

Interface Sensitivity• Tests must interact with

the SUT through some interface

• Any changes to interface may cause tests to fail.– User Interfaces:

» Renamed/deleted windows or messages

» New/renamed/deleted fields

» New/renamed/deleted data values in lists

– Machine-Machine Interfaces:» Renamed/deleted functions in API

» Renamed/deleted messages

» New/changed/deleted function parameters or message fields

Window

Window

Field

Button

Title

Caption

Link

Data Grid

E.g.: Move tax field to new popup window

30


Interface Sensitivity Impact *

• Symptoms:

– Unrecognized interaction

» e.g. window or message

– New unrecognized field

– Wrong data

• Impact:

– Many of these will be fatal.

– Non-fatals ones need manual inspection

– Every test that uses modified interface may fail

» Large maintenance impact

“*” means “skipped during presentation”


Q: How to avoid Interface Sensitivity?

• Express the tests using Intention-Based language

– Describe steps in business domain terms (Customers, Accounts), not UI terms (Screens, Fields, Buttons)

• Bypass the UI to an Intention-Based API

– Don’t test business logic via the UI

– Hand-script tests against Intention-Based API

– Do Built-in R&PB recording at Intention-Based API

• Wrap User Interface with an Adapter

– Record, Refactor (Extract Adapter), Playback

» Adapter converts Intention-Based API to User Interface

– Hand-script tests against the Intention-Based Adapter

31


Keeping Tests Simple: Testing via API

• API’s need to be designed in

– Design for Testability

CoreBusinessLogic

Test Invoice Generation-New Customer

-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccoun NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….

UserInterface

What we want to write:SUT

�API

Intention-based Keywords


When There’s No API Available

• Large gap between:

– what we want to write & What can be executed

– Many tests to adjust when UI changes � High Maintenance Cost

CoreBusinessLogic



UserInterface

SUT


-----------------------------Goto Login creenEnter “Clerk” in UserName fieldEnter “Pw123Secret” in Password fieldEnter …..-----------------------------Goto Cust ScreenClick “New Customer”Enter “Acme” in Name fieldEnter “123 Main St.” in Addr fieldEnter …..GotoScreen( “Account” )Find customer “Acme”Click “Add Account”Enter “Credit” in Type fieldEnter …..

Without a test API we have to write:

IntentionObscuringCode

CodeDuplication

� No API

32

63 Copyright 2010 Gerard MeszarosAgile 2010 Tutorial�Adapter

Keeping Tests Simple: Testing via Adapters

• Adapters can be tacked on– Single place to adjust when UI changes– But may be complex and error prone

CoreBusinessLogic



UserInterface

� No API

What we want to write:

Code inAdapter SUT

Intention-based Keywords


Our System

Other System

Other System


Container Services

Other System

Behavior Sensitivity• Tests must verify the

behavior of the system.– Behavior also involved in test

set up & tear down

• Any changes to business logic may cause tests to fail.– New/renamed/deleted states

– New/changed/removed business rules

– Changes to business algorithms

– Additional data requirements

Object

Object

StateIdentity

AlgorithmRule

E.g.: Change from GST+PST to HST

33


Behavior Sensitivity Impact *

• Symptoms:

– Tests block due to invalid inputs

– More/fewer items found

• Impact:

– Tests that depend on this behavior to setup the test fixture may also fail unexpectedly (fragile test)


Avoiding Behavior Sensitivity• Limit the usage of SUT behaviors not specific to the

current test

– Minimize test fixture set up using other parts of SUT

» Use API’s or backdoors (e.g. database) to setup SUT

– Or, build that logic into reusable test assets (e.g. Keywords) that can be used from many tests but only maintained in one place (like an adapter)

• Minimize duplication of coverage between Test Scripts

– In part 2 of tests

– Use Data-driven tests instead

Four Parts to Every Test:1. Put SUT into known

state2. Exercise the SUT3. Compare the response

and end state with expected

4. Clean up

Here

Here

Behavior Sensitivity Strikes

Here

And sometimes

34


Our System

Other System

Other System


Container Services

Other System

Data Sensitivity• All tests depend on

“test data” which are:– Preconditions of test

– Often stored in databases

– May be in other systems

• Changing the contents of the database may cause tests to fail.– Added/changed/deleted

records

Table

RowRowRow

E.g.: Change customer’s billing terms


Data Sensitivity Impact *

• Symptoms:

– Too many records found

– Too few records found

– Unique key violation

• Impact:

– Any test that depends on database contents may fail

– Tests that add to database may fail

35


Erratic Tests – Interacting Tests

If many tests use same objects, tests can affect each other’s results.

–Test 2 failure may leave Object X in state that causes Test n to fail.

Test 1 read:5

TestRunner 1

Test n

read:99

Symptoms:

–Tests that work by themselves fail when run in a suite.

–Cascading errors caused by a single

bug failing a single test.

» Bug need not affect other tests directly but leaves fixture in wrong state for subsequent tests to succeed.

Fail! (expects 99)

Object X

a: 5

Test 2 update(99)

Object X

a: 99

Fail!

Object X

a: 5


Erratic Tests – Unrepeatable Tests

If many test runs use same objects, test runs can affect each other’s results.

– Test 2 update may leave Object X in state that causes Test 1 to fail on next run.

TestRunner 1

Symptoms:– First run after opening the TestRunner or re-

initializing Shared Fixture behaves differently» Succeed, Fail, Fail, Fail

» Fail, Succeed, Succeed, Succeed

– Resetting the fixture may “reset” things to square 1 (restarting the cycle)

» Closing and reopening the test runner for in-memory fixture

» Reinitializing the database

Fail! (expects 5)

Object X

a: 5

Test 1 read:5

Test n

Test 2 update(99)

Object X

a: 99Test 1

36


Erratic Tests – Test Run War

• If many test runners use the same objects (from Global Fixture), random results can occur.

– Interleaving of tests from parallel runners makes determining cause very difficult

Test 1

Test 2

Test n

Object Xupdate

read Test 1

Test 2

Test n

Object Xupdate

read

TestRunner 1 TestRunner 2

Fail!

Fail!

Fail!


Q: How do we avoid Data Sensitivity (or “Fragile Fixture”)?

• Avoid having tests share data

– With other tests in test suite

– Across test runs

» E.g. rebuild the environment between runs

– Across test runners

» E.g. separate environment per runner

• Each test should:

– Create any/all data it plans to modify

– Clean up after itself

• Test should not:

– Modify any data shared with other tests

» E.g. Reference data such as postal codes

37


Our System

Other System

Other System


Container Services

Other System

Context Sensitivity

• Tests may depend on another systems’ state

– Stored outside the application being tested

• Changing the state of the context may cause tests to fail.

– State of the container

» e.g. time/date

– State of related systems

» Availability, data contents

Security System

User: XPermissions: none

Container Services

TimeDate

Customer X

E.g.: Run test in a shorter/longer month


Context Sensitivity Impact *

• Symptoms:

– Same system “build” behaves differently during different test runs

• Impact:

– Unrepeatable tests

– Nondeterministic tests

38


Q: How to avoid Context Sensitivity?

• Design SUT so that external dependencies can be simulated:

• Time/date under test control

• Security requests redirected to a test stub

• Access to other databases redirected to controlled instances

Requires Design for Testability (DfT) by Development

Collaboration!

Our System

Other System

Other System


Container Services

Other SystemSecurity System

User: XPermissions: none

Container Services

TimeDate

Customer XTest Stub

Test Stub


Q: What are the common approaches to test automation?

Test Preparation

Test Language

Test Interface

Test Data

Recorded Code Raw UI Global, Static

Hand-written Keyword Adapter Per Run

Data API Per Test

Our System


Container Services

Test Scenario Name-----------------------

----------------------

-----------------------Clean Up

API

AdapterVia

Raw UI

How Set U

p?

Fixture(state)

Preconditions

1. Do Something2. Check Something

GGM26

Slide 76

GGM26 Animate the columns if possibleGerard Meszaros, 05/08/2010

39


executiondefinition

• User executes tests manually; tool records as tests

• Tool replays tests later without user intervention

Fixture

Test Result Repository

Test Script Repository

TestRecorder

SUT

TestScript 1

TestScript 2

TestScript n

ExpectedOutputs

Inputs

Inputs

Outputs

InputsTestRunner

Script nResult

TestResults

ExpectedOutputs

Recorded Tests

The tests are are code/data interpreted by the test runner.

2 Variations:1. External COTS 2. Built-in Custom


(C)OTS Record&Playback

Test Preparation

Test Language

Test Interface

Test Data

Recorded Code Raw UI # Global, Static

Hand-written Keyword* Adapter Per Run

Data API Per Test

Notes:* Keywords, if used, tend to be very low level:

•GotoWindowNamed: name•SelectFieldNamed: name•EnterText: text•(Not the same as true Keyword-Driven testing)

# Most COTS Tools operate at UI or HTTP interface; many open-source tools do so as well

40


Sample Recorded Test@@ GoToPageMaintainTaxonomy()

Browser("Inf").Page("Inf").WebButton("Login").Click

Browser("Inf").Page("Inf_2").Check CheckPoint("Inf_2")

Browser("Inf").Page("Inf_2").Link("TAXONOMY LINKING").Click


Browser("Inf").Page("Inf_3").Link("MAINTAIN TAXONOMY").Click


@@ AddTerm("A","Top Level", "Top Level Definition")

Browser("Inf").Page("Inf_4").Link("Add").Click

wait 4

Browser("Inf_2").Page("Inf").Check CheckPoint("Inf_5")

Browser("Inf_2").Page("Inf").WebEdit("childCodeSuffix").Set "A"

Browser("Inf_2").Page("Inf").WebEdit("taxonomyDto.descript").Set "Top Level"

Browser("Inf_2").Page("Inf").WebEdit("taxonomyDto.definiti").Set "Top Level Definition"

Browser("Inf_2").Page("Inf").WebButton("Save").Click

wait 4

Browser("Inf").Page("Inf_5").Check CheckPoint("Inf_5_2")

@@ SelectTerm("[A]-Top Level")

Browser("Inf").Page("Inf_5").WebList("selectedTaxonomyCode").Select "[A]-Top Level“

@@ AddTerm("B","Second Top Level", "Second Top Level Definition")

Browser("Inf").Page("Inf_5").Link("Add").Click

Manually Added

Manually Added

Manually Added

Manually Added

Manually Added


Record, Refactor, Playback

• Use Test Recording as a way to capture tests

• Remove duplication by replacing with calls to domain-specific Test Utility Methods

– using Extract Method refactorings

• Make Test Utility Methods reusable

– Replace Hard-Coded Literal Values with variables/parameters

• Effectively turns recorded tests into programmed or keyword-driven test scripts

– But, still through UI Adapter & original tool choice

Most appropriate with legacy systemsEspecially with many interfaces

41


Recorded Tests Can Work Well V

Satisfied Customer Says:

“T Not only did this save 10's of man-years of testing effort, but it even uncovered before unknown bugs in the legacy system which we considered to be the gold standard. T”

V in the right circumstances:

– We retrofitted the capability to record and playback right into the legacy application.

– We came up with strategies to mitigate each of the four sensitivities.

– See:


Built-In Record&Playback

• User executes tests manually; SUT records as tests

• Tool replays tests later without user intervention

The tests are data

interpreted by the test runner.

SUT


TestScript 1

TestScript 2

TestScript n

execution

TestRunner

InputsExpectedOutputs

TestRecorder

Inputs

ExpectedOutputs

GGM25

Slide 82

GGM25 Get Stickman / Actor symbolGerard Meszaros, 05/08/2010

42


Built-in Record&Playback

Test Preparation

Test Language

Test Interface

Test Data



API Per Test


Sample Built-in R&PB Test Recording

2. Supply Create

43


Prev. Rec. User Input

Actual choicesplus

test results

Previouslyrecordedchoices

Raw XML for “Designation” Field *

<field name="designation" type="selection"><used-value>DIRECTIONAL</used-value><expected>

<value>DIRECTIONAL</value><value>WORK</value><value>PSGR</value><value>PLOW</value><value>PLOW WORK</value><value>ENG</value>

</expected><actual>

<value status="ok">DIRECTIONAL</value><value status="ok">WORK</value><value status="ok">ENG</value><value status="ok">PSGR</value><value status="surplus">MIXED</value><value status="ok">PLOW</value><value status="ok">PLOW WORK</value>

</actual></field>

Record

ingPlayb

ack


• Test automater writes test code to exercise the software

• Runs test code whenever testing required

Hand-Coded Tests

executiondefinition

The tests are a program that tests the program

A.K.A. Scripted,Programmed

TestRunner



TestScript 1

TestScript 2

TestScript n

Fixture

S/W Devt.Tool

SUT

ExpectedOutputsInputs

Inputs

Outputs

Script nResult

TestResults

Inputs

ExpectedOutputs

TestScript n

44


Hand-Coded Tests

Test Preparation

Test Language

Test Interface

Test Data



Data API Per Test

• Hand-written code requires software development skills and test automation skills.

• Developers need training on writing clear tests!


Sample Hand-Coded Test public void testAddItemQuantity_severalQuantity () {

// Fixture set up:final int QUANTITY = 5 ;Product product = createAnonymousProduct();Invoice invoice = createAnonymousInvoice();

// Exercise SUTinvoice.addItemQuantity(product, QUANTITY);

// VerifyLineItem expectedLineItem = newLineItem( invoice,

product, QUANTITY, product.getPrice()*QUANTITY );

assertExactlyOneLineItem( invoice, expectedLineItem );}

45


Sample Hand-Coded Test - Poorpublic void testAddItemQuantity_severalQuantity () {

// Setup Fixturefinal int QUANTITY = 5;Address billingAddress = new Address("1222 1st St SW", "Calgary",

"Alberta", "T2N 2V2", "Canada");Address shippingAddress = new Address("1333 1st St SW", "Calgary",

"Alberta", "T2N 2V2", "Canada");Customer customer = new Customer(99, "John", "Doe", new

BigDecimal("30"), billingAddress, shippingAddress);Product product = new Product(88, "SomeWidget", new

BigDecimal("19.99"));Invoice invoice = new Invoice(customer);// Exercise SUTinvoice.addItemQuantity(product, QUANTITY);// Verify OutcomeList lineItems = invoice.getLineItems();if (lineItems.size() == 1) {

LineItem actualLineItem = (LineItem)lineItems.get(0);assertEquals(invoice, actualLineItem.getInvoice());assertEquals(product, actualLineItem.getProduct());assertEquals(quantity, actualLineItem.getQuantity()); assertEquals(new BigDecimal("30"),

actualLineItem.getPercentDiscount());assertEquals(new BigDecimal("19.99"),

actualLineItem.getUnitPrice());assertEquals(new BigDecimal(“69.96"),

actualLineItem.getExtendedPrice());} else {

assertTrue(“Invoice should have exactly one line item“, false);}

Developersneed

training on writing

clear tests!


executiondefinition

• The tests are expressed in domain-specific vocabulary.

• The tests are read & executed by a test interpreter written by techies.

Keyword-Driven Tests

Prepared like Hand-Coded Tests but with a much more limited vocabulary.

46


Keyword-Driven Tests

Test Preparation

Test Language

Test Interface

Test Data

Recorded Code Raw UI * Global, Static


Data API Per Test

Notes:• While the Keyword Interpreter may go against the Raw

UI, it is better to delegate to an adapter if no API is available.



-----------------------------Logged in as ClerkItem1, Item2 exist-----------------------------1. CreateCustomer “Acme” 2. CreateAccount NewCust3. AddPurchase Item14. AddPurchase Item25. GenerateInvoice NewAcct6. ….

SUT

Sample Keyword-Driven Test

ComponentUnder Test

ResultsDocument

Result of step

Result of step

Fit Library

Fit Test Runner

Keyword Interpreter

Keyword Interpreter

47


executiondefinition

• The tests are expressed as tabular data by users.

• The tests are read & executed by a test interpreter written by techies.

Data-Driven Tests

Runs the same test script many times; once per set of data.


Data-Driven Test

Test Preparation

Test Language

Test Interface

Test Data #

Recorded * Code Raw UI Global, Static


Data API Per Test

Notes:* The underlying script may be either hand-written or

recorded and parameterized. But the input values and expected outputs are almost always prepared by hand.

# The inputs/outputs are per test (per row) but there may be global or per-run data used as reference data by the underlying script.

48


Sample Data-Driven Test in FIT

• Same script is run for each row of table

• Avoids duplication of test script.

• Compact summary of input values & results

• Sometimes called “Business Unit Test” or “Business Rule Test”

PayrolFixtures.WeeklyCompensation

Standard Hours

Holiday Hours

Hourly Wage

Pay( )

40 0 10 $400

40 0 20 $800

41 0 20 $830

40 1 20 $840 expected

$800 actual

41 1 20 $870 expected

$830 actual

�---Inputs---� �Outputs�

SUT

ComponentUnder Test

ResultsDocument

Marked upTable

Fit Library

Fit Test Runner

Table Interpreter


Q: How to choose test automation tool?

1. Determine test automation goals

– Finding bugs?

– Preventing bugs?

– Frequent/Rapid Regression Testing?

2. Choose the test automation approach(es)

– Hand-written or Recorded?

– Code, Data-driven, or Keyword-driven?

3. Choose tool(s) that supports approach(es)

– E.g. xUnit for Hand-Coded Unit Tests

– E.g. RobotFramework for Hand-Written Acceptance Tests

– E.g. Fit for Hand-written Business Rule Tests

49


Q: How to choose which test prep. strategy?

Fixture



TestRecorder

SUT

TestScript 1

TestScript 2

TestScript n

ExpectedOutputs

Inputs

Inputs

Outputs

InputsTestRunner

Script nResult

Test

Results

Expected

Outputs


Fixture


DocumentEditor

SUT

TestScript 1

TestScript 2

TestScript n

ExpectedOutputs

Inputs

Inputs

Outputs

TestRunner

TestInterpretter

Script nResult

TestResults

InputsExpectedOutputs

?

GGM15


Which Test Preparation Strategy?

Recorded Tests:

• Fast to create– “No coding skills Required”

• Don’t support example-driven development– Only useful after the fact

• Unmaintainable– Re-record, or

– Coding skills required

• Done by Testers– Fairly non-technical

• Unintelligible to Business– Because it’s too detailed

– Doesn’t capture business intent

Hand-Coded Tests:

• Harder to create– Coding skills required

• Supports example-driven development – Can be done ahead of time

• Requires S/W Skills– Design-for- Testability

– Test s/w engineering

• Done by Techies– Development or Technical

testers

• Unintelligible to Business– Because it’s in code

GGM24

Slide 97

GGM15 Fix this up - make it coherentGerard Meszaros, 30/07/2010

Slide 98

GGM24 Need to add Data-Driven and Keyword-Diven Tests

Make this more comprehensiveGerard Meszaros, 06/08/2010

50


Why Build R&PB into your Application?

• Gives you complete control over what to pay attention to and what to ignore!– E.g. Ignore the time/date fields

• Tests will run much faster bypassing the User Interface

• May be less expensive than commercial tools, if:– Large numbers of users, or

– Low cost to build » I.e., simple design or easy-to-use

framework

Why Not:

• Can be a significant investment of time and money

– Expect it to cost twice what you first estimate

• Can require a fair bit of expertise


Why Build R&PB into your Application? *

Can make it much easier for your business people to use:• Domain-specific terminology

– Business concepts, not technical

– E.g. Select Direction South

vs. Select value “south” from dialog box with title=“Direction”

• Testing rules built in

– No need to fiddle with generic sensitivity settings in each recorded test

– E.g., Always ignoring time fields

• Simplified test setup

– Knows what starting state consists of and how to cause it to be

– E.g., Managing Database Snapshots

51


Why Use a (C)OTS R&PB Tool?

• Available Right Away

– Free trial downloads

– A lot of test automation expertise is built in

• May Be Lower Risk

– If you can address the 4 sensitivities

Why Not?

• Expensive to buy

– Per-user licensing cost can make it very expensive to deploy

– May have to limit who uses it

• May force you to work at too low a level

– E.g., screen widgets

– May cause you to loose focus on the big picture

• Giving up control of your destiny

– Bound to a tools vendor and their vision of the future• Open Source may address cost

• But not other issues.


Why Hand-Code Tests

• Most flexibility

• Tests will be automated by developers

• Technical product

– E.g. For developers

Why Not?

• Requires programming skills

• Business people won’t understand tests/examples

• More effort

• Less collaboration

52


Why Use Keyword-Driven Tests

• Tests/examples can be automated and reviewed by business experts

• Encourages collaboration between business, testers & developers

• Less effort per test script

Why Not?

• Less flexible

• Requires collaboration!


Why Use Data-Driven Tests

• Can test large numbers of data combinations easily

• Avoids duplicated test script code

• Less effort per test condition

Why Not?

• Less flexible; need separate test format for each business rule

• App. is not rules intensive

• Requires collaboration!

53


What You Need to Do

• Ensure Development and Testing understand that you expect them to:

– Build sustainable test automation

– Work together to ensure system is easily tested

• Ask for regular updates on how test automation will be built

– Ensure they have addressed all 4 sensitivities

– You will need to sustain it long after Dev’t & Test have moved on to other projects!

• Provide knowledgeable resources to specify the business scenarios to be tested

– Maybe even build the test cases


Exercise

• If you have done test automation in the past:

– Which approaches have you used?

– What problems did you run into?

– How did these relate to the Four Sensitivities?

• Which of these approaches look most promising for your current / next project?

– Recorded COTS

– Recorded Built-in

– Keyword-Driven

– Data-Driven

– Hand-Coded

54


Q: Who needs be involved in preparing Functional tests?

• Business subject matter experts are needed to specify the tests/examples to be automated.

• Testers should also be involved as they often think of scenarios that business people don’t think of (what could possibly go wrong?)

• Test automation engineers or software developers are required to build the automated scripts (Hand-scripted or Record-Refactor-Playback) or the test script interpreters (Keyword-Driven).

A Collaborative Approach Works Best!


ApplicationCollaborative Test Automation

What:

• Define test agreeing on document format

• Implement keyword interpreters

• Run tests using Test Runner

• Fit marks up Tabular Data with test results

ComponentUnder Test

TestsDocument

TabularData

TabularData

ResultsDocument

Marked upData

Marked upData

Fit Library

Keyword Interpreter

Keyword Interpreter

Fit Test Runner

Who:

Joint

Dev’t or TE

Anyone

Automatic

55


Collaborative Test AutomationPayrolFixtures.WeeklyCompensation

Standard Hours Holiday Hours Hourly Wage Pay( )

40 0 10 $400

40 0 20 $800

41 0 20 $830

40 1 20 $840

41 1 20 $870


Collaborative Test AutomationPayrolFixtures.WeeklyCompensation

Standard Hours Holiday Hours Hourly Wage Pay( )

40 0 10 $400

40 0 20 $800

41 0 20 $830

40 1 20 $840 expected

$800 actual

41 1 20 $870 expected

$830 actual

56


Sample Fit Column Fixture

public class WeeklyCompensation ColumnFixture {

public int StandardHours;

public int HolidayHours;

public Currency Wage;

public Currency Pay( ) {

WeeklyTimesheet timesheet = new WeeklyTimesheet( StandardHours, HolidayHours);

return timesheet.CalculatePay( Wage );

}

}


Requires access to software internals

Sample Fit Column Fixture

public class WeeklyCompensation ColumnFixture {

public int StandardHours;

public int HolidayHours;

public Currency Wage;

public Currency Pay( ) {

WeeklyTimesheet timesheet = new WeeklyTimesheet( StandardHours, HolidayHours);

return timesheet.CalculatePay( Wage );

}

}

57


Sample Multi-Table FIT Test• Load data into table using Column Fixture:

• Exercise system using Action Fixture

• Verify database contents using Row Fixture

PayrolFixtures.EmployeeTableLoad

Number Name Hours Add()

10001 Gerard Meszaros 45 OK

10002 John Smith 40 OK

10003 Jane Doe 420 OK expectedToo Big actual

PayrolFixtures.EmployeeCompensationThisWeek

Number Name Hours Pay

10001 Gerard Meszaros 45 $950

10002 John Smith 42 expected40 actual

$860 expected$800 actual

10003 surplus Jane Doe 20 $800

PayrolFixtures.RunPayrollActionFixture

RunJob Payroll


What You Need to Do

• Dedicate good (or better) business resources to specifying the requirements as examples/tests

– Requires good knowledge and communication skills.

– Don’t fob off the people you can spare

• Ensure you include positive & negative cases

– May require experienced tester to bring this perspective

• Ensure Development team is involved

– To understand testability requirements of application.

GGM17

Slide 114

GGM17 Fill inGerard Meszaros, 30/07/2010

58


Q: Why We Need Unit/Component Tests?

Acceptance Tests

Regression Tests

Usability Tests

Exploratory Tests

Unit Tests

Component Tests



Business

Facing

Technology

Facing

Prevent anticipatable defects from happening

Find non-anticipatable Defects, ASAP!


Critique Product

ReportCard


BCABC

Define ProductRequirements

SoftwareDesign


What are Unit/Component Tests?

• Lets us test code we cannot hit in func. tests

– Avoids nasty lurking bugs that are only found by users

• They run much faster than functional tests

– Makes retesting bugs cheaper

• Helps developers find their bugs much sooner

– Makes fixing them cheaper

Application

Customer (Functional) Testing

ComponentTesting

Component

Class

Unit Testing

59


Insert Defect

DetermineFix

Identify/ Debug

Run Unit Tests

Preventing Coding Defects

Write Req’ts

Review Req’ts

Update Req’ts

Sign Off on Req’ts

Test Application

Deploy to QA

Deploy System

DesignSoftwareDetection!

Write Code

Unit Test Code

Rework!Rework!


Preventing Coding Defects

Write Req’ts

Review Req’ts

Update Req’ts


Write Unit Tests

Run Unit Tests

Test Application

Deploy to QA

Deploy System

DesignSoftware

Write Code

Prevention!

DetermineFix

Identify/ Debug

60


Prevention: - Coding Defects

Write Req’ts

Review Req’ts

Update Req’ts


Write Unit Tests

Run Unit Tests

Test Application

Deploy to QA

Deploy System

DesignSoftware

Write Code

Test-Driven Development

Prevents bugs crawling back in

Prevent defects in new code



Make it clear to development team that:

• They are to do proper unit testing.

• All estimates must include unit test cost

• Ideally, automated tests and test harnesses

Like reminding the Doctor that Sterility is not optional

61


Q: Why do I need to tell the development team I expect automated unit tests?

• They may not feel empowered to impose this overhead on development because they don’t recognize the business value.

• They may have to been told in the past by Product Owners (or development managers) “not to waste their time” doing automated testing.


Q: What Tools should we be using?

• Won’t talk about specific tools per se but:

• Tools should make the job as easy as possible for the people doing the work:

• The people preparing the tests or examples

• The people running the tests or examples

• The people supporting the tests or examples

•

62


Q: How does automated testing support agile principles and values?

• AT provides rapid feedback for developers.

• AT reduces the waste involved in repeatedly retesting the same functionality.

• AT improves quality by preventing defects from being introduced (by providing concrete examples, and by avoiding regression.).

• AT enables cost-effective regression testing of each increment of functionality.


Q: How do agile principles & values support automated testing?

• Collaboration between test automaters and developers ensures the system is built to support testability.

• The preparation of automated functional tests/examples forces collaboration between subject matter experts (business-oriented people) and technical people (test automaters and developers.)

• Testers are motivated to help developers do better unit/component testing.

63


Test Automation Pyramid

• Large numbers of very small unit tests

• Smaller number of functional tests for major components

• Even fewer tests for the entire application & workflow

Unit Tests

ComponentTests

SystemTests

Various automation

tools(or manual)

Keyword or xUnit

xUnit


Application not Designed for Testability

Causes of Fragile Tests:

• Interface Sensitivity

• Behavior Sensitivity

• Data Sensitivity

• Context Sensitivity

MonolithicApplication

Test

UI Logic

Business Logic

Technical Logic

Data Access Logic

ControlPoint

Observation Point

See: http://agileregressiontestpaper.gerardmeszaros.com

64


Application with Database

MonolithicApplication

Test

Observation Point – Indirect Output

Control Point– Indirect Input

ControlPoint

Observation Point

UI Logic

Business Logic

Technical Logic

Data Access Logic


ControlPoint

Observation Point

Application Designed for Testability

UI LayerBusinessLogic Test

ControlPoint

Observation Point

Business Logic Layer

Technical Layer

Data Access Layer

TechnicalLayer Test

ControlPoint

Observation Point

Data AccessLayer Test

ControlPoint

Observation Point

UI Test

Observation Point – Indirect Output

Control Point– Indirect Input

65


Collaboration Required!

DevelopmentIndependent Verification

Requirements

B6! Missed!

Requirements



Requirements

B6! Missed!

Requirements

DevelopmentIndependent Verification

66



Requirements Requirements

This Software is Ready for Testing

Let’s Test it Together

This test is tedious but essential.

Let’s automate

it.

GGM11


Strategy Summary

• Avoid Defects by Providing Acceptance Tests Up Front

• Reduce Cost of Fixing Bugs by Finding Defects Earlier

• Find Defects Earlier by Testing Earlier

• Test Earlier by Developing Incrementally

• Pin the Functionality using Automated Tests

• Reduce the Cost of ReTesting by Automating Acceptance Tests

• Prioritize Automation by ROI = (Value / Cost)

• Improve Internal Quality and Reduce Cost of Fixing Bugs thru Automated Unit Testing

• Make Test Automation Cost-Effective thru Design for Testability

Slide 131

GGM11 Maybe omit this slide?Gerard Meszaros, 30/09/2008

67


Thank You!

Gerard [email protected]

http://www.xunitpatterns.com

http://www.gerardmeszaros.com

Slides: http:// Agile2010.xunitpatterns.com

Call me when you:• Want to transition to Agile or Lean

• Want to do Agile or Lean better

• Want to remove waste from your process

• Want to teach developers how to test

• Need help with test automation strategy

• Want to improve usability of your applications

Jolt Productivity Award winner - Technical Books

Coming Soon:


References

• For Success, Build Record/Playback into Your Application - StarEast 2008 Class

– http://agile2010.xunitpatterns.com/

Documents

Agile Test Automation-GM4b-abridgedxunitpatterns.com/~gerard/NonTechTestAutomation.pdf · 2015-06-26 · • How do agile principles & values support automated testing? • What is