An Introduction to Agile Software Testing

1

Course Objectives

After this seminar, you will:

Understand what an Agile project

Know the differences between Kanban and Scrum

Understand what and how a tester performs the testing in Scrum process

Understand what testing techniques required for a tester in Agile

Know what challenges of Agile Software Testing

2

Course Content

What is an Agile Project?

What are Scrum and Kanban?

What are Scrum & Kanban Processes?

What are testing Activities in Scrum?

What are Agile Testing Techniques?

3

In practice, “Agile” labeled

to projects that its delivery of

each increment taking no

longer 4 weeks

What is an Agile Project?

All projects are “agile”, but some agile projects are more agile than others

How to “agile” a project

An Agile purist would point to agile manifesto and its principles

Individuals and interactions over process and tools

Working SW over comprehensive documents

Customer collaboration over contract negotiation

Responding to change over following a plan

Is it met level of agility?

4

Agile Project

An

introduction

of Scrum and

Kanban

5

What is Scrum?

• Scrum is an agile process that allows us to focus on

delivering the highest business value in the shortest

time.

• It allows us to rapidly and repeatedly inspect actual

working software (every two weeks to one month).

• The business sets the priorities. Teams self-organize to

determine the best way to deliver the highest priority

features.

• Every two weeks to a month anyone can see real

working software and decide to release it as is or

continue to enhance it for another sprint.

Scrum in 100 words

6

9 Scrum principles

What is Scrum? - Principles

(1)Demo

(2)Sit together

(3)Daily Scrum

(4)Retrospective

(5)Sprint backlog

(6)Product Backlog

(7)Sprint planning meetings

(8)Time-boxed iterations

(9)Definition of Done

Scrum contains sets of principles (some of them

required while others are optional) and

predefined roles

7

What is Kanban?

Visualize the workflow Split the work into pieces, write each item on a card and put on the wall. Use named columns to illustrate where each item is in the workflow.. Limit Work In Progress (WIP) – assign explicit limits to how many items may be in progress at each workflow state Measure the lead time (average time to complete one item, sometimes called “cycle time”), optimize the process to make lead time as small and predictable as possible

Kanban in 100 words

8

Kanban vs Scrum Board Kanban and Scrum Board

V.S.

Scrum resists change within

an Iteration

Scrum and Kanban over time

V.S.

Kanban limits number of items

in workflow state V.S.

9

An

introduction

of Scrum

Process

10

What is Scrum process?

Product Owner

Scrum Master

Scrum Team

(max – 9)

Scrum Roles Key Artifacts

Product Backlog • Requirements – user stories • Desired work • Prioritized by Product Owner • Anybody can add to it

Sprint Goal • Summary of focused work in

sprint • Declared by Product Owner • Accepted by team

Sprint Backlog • Team signs up for work of

their own – work never assigned

• Owned/managed by the team • Estimated work remaining is

updated daily

Blocks List • List of blocks & unmade

decisions • Owned by Scrum Master • Updated daily

Burndown chart • Effort spent over period • Stories/ features completed

Ceremonies

Sprint planning

• Hosted by Scrum Master

• Pick highest priority items in

Product backlog and the team turns

the items into Sprint Backlog

• Estimate sprint backlog in hours

• Work breakdown

• Declare Sprint Goal

Daily Scrum

• Hosted by Scrum Master

• 15 minutes - same time every day

• Not for problem solving

• 3 questions in meeting: (1) What

did you do? (2) What will you do?

(3) What’s in your way?

• Team updates sprint backlog

Sprint Review

• Hosted by Scrum Master – 2-4 hours

• Accomplishments

• Whole team participants

• Take form of demo for features

Sprint Retrospective

• Hosted by SM – 15-30 minutes

• Discuss on “Start doing”, “Continue

doing” and “Stop doing”

Product

Backlog

Sprint – 2-4 weeks

Sprint Planning

Daily Work

Daily Scrum

Sprint Review/

Retrospective

Sprint Goal

Sprint

Backlog

Blocks

Product

Product

Backlog

Process

11

What are roles in Scrum?

Product Owner (PO) is a client’s representation

Define features of product

Decide Release Date and content

Prioritize features according to market value

Be responsible for the profitability of product

Accept or reject work item result

A practice for prioritizing features (or User Story)

Priority = Max (5 (or 6) x priority that assigned by PO, sum

(priority that assigned by team members)

Scrum Master represents Management

Enacting Scrum values

Ensure team’s productivity

Prioritize features according to market value

Corporate across all roles and functions

Shield team from external interferences

The Team

5-9 members

Including: developers, testers, designers,…

Full time

Work Self-organizing

Membership should be changed each sprint

12

What are roles of tester in Scrum?

Isn’t it really necessary tester position in Scrum?

What happens without tester?

No future for tester?

There is NO tester in formal Scrum Process

Testing is carried out by developer with Unit Test

•Test Driven Development means testing

executed before development and refactoring

performed to optimize development

•Testing coverage

Testing is carried out by Product Owner/ Client

•Frequently testing by Product Owner – each

sprint

•Testing Acceptance Criteria

13

Questions:

• Will developers do acceptance

testing well?

•Will developers test their work

product well?

•Is that enough with only one

kind of testing – Unit Test?

• How to test non-functional

requirements?

Benefits when have

dedicated test team:

•Focus on customer usage over

technical implementation

•Focus on uncovering flaws over

confirming completeness

What are roles of tester in Scrum ? (cont.) Are testers obsolete?

14

What are roles of tester in Scrum? (cont.) • Participant in Release/ Sprint

Planning

• Support developers in Unit Testing

– say them about testing view instead

of development view

• Test User Story when completed,

last gate to confirm testing

completeness

• Collaborate with customer and

Product Owner to define acceptance

criteria

• Provide positive feedback to

customer

• Develop automation testing

15

Testing Activity

in Scrum &

Kaban Process

16

What are Testing Activities in Scrum Process?

Product

Backlog

Sprint – 2-4 weeks

Sprint Planning

Daily Work

Daily Scrum

Sprint Review/

Retrospective

Sprint Goal

Sprint

Backlog

Blocks

Product

Product

Backlog

Process

In Sprint Planning:

• As a tester, he/she says what should be picked from

Product Backlog for this Sprint to bring value to customer

& to create high performance

•As a tester, he/she says how many hours should have

to complete testing for each of selected User Stories

•As a tester, he/she must understand what sprint goals

• As a tester, involves into the prioritizing process

Tester must know what model of cadence used

sin

gle

ca

de

nce

Thre

e

cad

en

ce

ev

en

t D

riv

en

17

What are Testing Activities …? – estimate effort

Effort Estimate techniques:

Heuristic (expert judgment based) Parametric (calculation based) Comparison to similar systems Function Points

Expert Judgment Use Case Points

Activity Based (top down) User Story Points

Task Based (bottom up)

Three-point estimate

50/90 estimate

In Agile, testers are empowered to make decision. It means they

are given authority to estimate effort for each work item that

stated in Story Card for Product Owner reviews

As a result, testers are responsible for all testing activities that

they estimated.

18

What are Testing Activities …? – 50/90 estimate

50/90 Estimation

Worst case: number of hours (cost) in the worse-case scenario

Most -likely: number of hours (cost) in the Most -likely scenario

Total Hours = Most Likely

Task variance = (Worst Case (W) – Most Likely (A))

Deviation = sqrt (sum((Wi-Ai)^2))

19

What are Testing Activities …? – 3-point estimate

3 Points Estimate (PERT)

Opt: indicates number of hours (cost) in the best-case scenario Pess: indicates number of hours (cost) in the worse-case scenario

Most -likely: indicates number of hours (cost) in the Most -likely scenario

PERT = (Opt + 4 x Most-Likely + Pess) / 6

Variance = (Pess – Opt) / 6

20

What are Testing Activities …? – User Story Points User Story points (USP)

Story points are a unit of measure for expressing the overall size of a

user story, feature, or other piece of work

The raw values assigned are unimportant

Image a story with 2 USP(s) will 2 times bigger than 1-USP one

Two estimation scales or or

Velocity is a measure of a team’s rate of progress – Ideal Time

The beauty of this is that estimating in story points completely separates

the estimation of effort from the estimation of duration

21

If we find that a particular work unit is twice the size we thought.

The estimate remains the same, but rate of progress through it

will be slower

What are Testing Activities in Scrum Process?

Product

Backlog

Sprint – 2-4 weeks

Sprint Planning

Daily Work

Daily Scrum

Sprint Review/

Retrospective

Sprint Goal

Sprint

Backlog

Blocks

Product

Product

Backlog

Process

In Daily Scrum:

• As a tester, he/she joins all daily meeting (standup)

to speak up

1. what was tested and did yesterday/ progress;

2. What will test and do today?

3. Any in testing progress

•As a tester, he/she can bring any backlog item that

cannot be completed in current sprint and put to next

sprint – EX: I don’t have enough time to execute

testing for item #4. it will be tested in next sprint.

22

What are Testing Activities in Scrum Process?

Product

Backlog

Sprint – 2-4 weeks

Sprint Planning

Daily Work

Daily Scrum

Sprint Review/

Retrospective

Sprint Goal

Sprint

Backlog

Blocks

Product

Product

Backlog

Process

In Daily Work:

• As a tester, he/she is responsible for develop

automation scripts (for automation testing

•As a tester, he/she schedules automation testing

with Continuous Integration (CI) System.

•As a tester, he/she reviews & reports test results to

stakeholders via CI System or by himself/herself.

•As a tester, he/she executes non-functional testing

with approved user stories

Some organizations, Product Owner does not have

enough energy to do acceptance testing each sprint.

Tester is hired for this case

•As a tester, in end of Sprint, he/she executes UAT

and confirms testing completeness for current sprint.

23

What are Testing Activities in Scrum Process? In Scrum, Software is integrated daily, build and test

each day. Therefore, CI System is deployed to most

of Agile Project. A CI System & test components can

be identified as below

• Hudson / CruiseControl – CI Framework

• Junit (for Java)/ Nunit (.Net) – Unit Test

• Cobertura (Java)/ NCoverage – Testing

Coverage

• Ant (for Java)/ Nant (.Net) – Automation Build

• SVN – Version Control

•Selenium integrated with Hudson - Functional

testing

In Daily Work, tester will responsible for develop

automation scripts, integrate testing automation

testing tools/scripts with CI for regression test and

UAT

As a part of duty, tester is playing role supporter in

helping developers create Unit Tests effectively and

efficient 24

What are Testing Activities in Scrum Process?

In Daily Work:

As a practice, “Done” of sprint can be considered when

Code checked in.

Builds ready

Unit Test complete successfully

80% code branch coverage on UT

Non-functional testing completed

(confirmed by testers)

Finish deploying to Production

Tester’s

supports

25

What are Testing Activities in Scrum Process?

Product

Backlog

Sprint – 2-4 weeks

Sprint Planning

Daily Work

Daily Scrum

Sprint Review/

Retrospective

Sprint Goal

Sprint

Backlog

Blocks

Product

Product

Backlog

Process

In Review & Retrospective:

• As a tester, he/she identifies what went right and

what went wrong in current sprint

•As a tester, he/she identifies lesson learned and

best practices.

•As a tester, he/she is encouraged to write user

stories that support testing.

EX: As a tester, I need interface with inputs – A, B, C

to execute testing of feature X.

•As a tester, he/she is encouraged to write user

stories that bring value to customer.

EX: As a user, I need to be able to sort items.

All these user stories are put to Product Backlog

for review and Approved

26

What are Testing Activities in Scrum Process?

In Review & Retrospective:

• Any items remaining will be

•Review why tester could NOT complete in

current sprint

Problem may be identified by lacking of

experience in estimate or any roadblock

prevented it completed. Whatever reason

identified will be noted as lesson learned

•Put to next sprint for continuing testing

• Any roadblock comes from lacking of support of

Product Owner, should be pointed out for Scrum

Master’s follow-up

To bring effectiveness and efficiency to review sessions,

Scrum Master should request participants to prepare list of

items/ issues at least 1 day prior to meeting date

27

What are Testing Activities in Kanban Process?

Kanban is less prescriptive than Scrum

All testing activities are similar with Scrum

No required time-boxed iteration

•Testing is not limited by time duration

•Accept changes at anytime – Scrum protects changing backlog items

in sprint while Kanban still accepts removing and adding new items

• Testing item in workflow state can be added or removed to get the

most effectiveness and efficiency.

Kanban limits number of items for each workflow state

•Testing items should be picked logically to put to workflow stated

28

What are Testing Activities in Kanban Process? An Example of Managing Tasks/ Work Item

1. Backlog: state all work items/ user stories translated into

tasks - As a characteristic of Backlog may be limited

number of items

2. Ready: List of work items that ready for executing

3. Specifics: Identify what criteria to confirm work item

completeness

4. Execute: Work Item in executing

Tuan completed work item - testing for “Sorting Items”. He reviews and confirms that it satisfied criteria state. He moves this item to “Done”

Next, he moves another work item – testing for “Filter Items” from ready to “execute” to start new work

He also moves another work item from Backlog as a pipeline item.

29

What are Testing Activities in Product Development?

Unit Testing

Integration Test

Non-functional

testing/ confirms

story is met

Testing against

user story

Testing in live

environment - UAT

In Agile, testing is focused on

1. Unit Test that developers need support of testers to

create effective set of tests and

2. Acceptance Test that the responsibility is belonged

to tester

Integration & regression test are executed daily by

automation.

Tester is also responsible for

testing user story that cannot be

verified by UT to confirm testing

completed before demo and

release to production.

30

What are Testing Activities in Product Development?

Unit Testing

Integration Test

Non-functional

testing/ confirms

story is met

Testing against

user story

Testing in live

environment - UAT

Story Exploration

Estimation

Story Planning

Story Progression

Story Acceptance

Communication within an agile team to explore story

understanding

Consider flows of between Product Owner and team on story

Testers also speak up with questions to clarify any vagueness

in the story

What are Testing Activities in Product Development?

Story Exploration

Estimation

Story Planning

Story Progression

Story Acceptance

Estimation is a part of iteration where tester is using appropriate

techniques

Story Planning occurs after a story has been estimated and be

placed in iteration. Details of test tasks defined as

1. Prepare test data 2. Extend acceptance test

3. Execute manual 4. Automate CI tests

EX: For User story: “As a user, I can reset my account password if I forgot my password”

Test may have questions relate to this story as follows

1. What does it mean for an email address to be ‘unknown’?

2. What does it mean to ‘require confirmation’ of the password?

And then s/he temporarily states their understanding as next picture

What are Testing Activities in Product Development?

Story Exploration

Estimation

Story Planning

Story Progression

Story Acceptance

Story Progression: uncover additional tests that reply on

exploratory testing and communication. Conducting testing while

monitor results from CI system

Story acceptance: “done” in test may be understood as all tests

pass or level of test automation is met

If acceptance criteria are not in

place, agile teams may collaborate

on writing acceptance tests during

story planning

Tester can select acceptance

tests for automation. Not all

acceptance tests will be automated

Ex: for Forgot Password story, team may

have scenario as next picture for acceptance

test

What are documents of testing in Agile Project?

As a part of process, testing in Agile has lightweight documentation with:

Use reusable checklists to

suggest tests

Focus on essential of test

rather than the incidental details

Capture test ideas for

Exploratory Testing

Agile in common, or Scrum/ Kanban in

particular, is focused on how to deliver

value to customer by involving customer

(that presented by Product Owner) in

whole process; transferring product to

live (production) environment to bring

benefits to business soon as possible

and expect changes to maximize

customer business value. While Agile

reduces documentation overhead by

simplifying process and number of

artifacts created

In case test-cases required, they should be created at high level

If any, leverage one artifact for Manual and Automation Tests

34

How to track bugs in Agile Process? Do or Do Not need Bug Tracking.

No bug Tracking is still working well when testers perform their tests in Dev environment

However, in Test environment, testers need to track bugs because they

don’t see fixes for a while

With Bug Tracking, any bugs which are identified (opened) by

developers also will be regress tested by testers. Then test scripts will

be updated to cover the case

Ultimately, bugs can be prioritized with stories

Keep in mind, in Agile, testing is not concrete phase. It is a part of

development Development

Testing

Development &

Testing 35

Agile Testing

Techniques

36

What are Agile Testing Techniques? Quadrants

Functional Testing

Story Test

Prototypes Test

Exploratory Testing

Scenario Test

Usability Test User Acceptance Test

Unit Testing

Unit tests combined with

Test coverage

Performance & Load Testing

Performance Test

Load Test

Security Test

Business Facing

Technology Facing

Supp

ort t

he te

am Critique Product

Automation &

Manual Test

Automation

Test

Manual

Test

Special Tool &

Automation

Test

37

What are Agile Testing Techniques?

As typical testing, black box testing is used.

Agile testing is focus on using

Automation testing techniques Test driven development, auto builds and continuous regression test daily all reply on automation and will not work without it. And acceptance test is carried out each Sprint that automation also has benefits

Exploratory testing technique This technique simultaneous learning, test design and test execution. While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run.

Scenario testing technique Use real-world scenarios that exercise multiple stories

Don’t think of Exploratory is Agile Testing – it is only one of Agile

testing approaches

38

What are Agile Testing Techniques? Exploratory

An example of Exploratory testing

technique.

User Story As an online shopper, I want to be able to delete items from my shopping cart Exploratory is kind of creativity techniques We start with thinking of some personas or characters and devise exploratory scenarios which role may get into. Ex: 1. Senior citizen who never shopped on web 2. Hacker looking for troubles 3. Internet-savvy teen For each role, behaviors will be different to have cases of testing generated and tested

39

References

40

Scrum and Kaban at InfoQ http://www.infoq.com/

Testing Experience http://www.testingexperience.com/

Mountain Goat Software http://www.mountaingoatsoftware.com/

Agile Estimating and Planning – Mike Cohn

User Stories Applied: For Agile Software Development

41

Recommended useful tool for Agile Team

- the ultimate screenshot capture software designed specifically for testers that: - Snap a single screenshot or record a sequence of screens - Automatically generate intelligent step-by-step narration (defect report) - Auto submit tickets to defect tracking and ALM systems

- calling for BETA user - Full featured Test Management System to give testers a simple yet productive working environment. Some of the qTest features and modules include: - Test Planning - Requirement Management - Test Case Management - Test Execution - Defect Tracking - Dashboards and Reports - Built in workflows and collaboration

Get the tools as free download at http://www.qasymphony.com/

Thank You

Agile

Project

Scrum &

Kanban

Agile

Testing

Others

1

2

3

4

Welcome your questions

42