How to Develop a Common Sense of “DONE”?

Alexander Schwartz

mobile.international, Berlin

October 13, 2009

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Introduction

About MyselfAlexander SchwartzIT industry since 1989

• Developer (Java, C++)

• Tech Lead, Project Lead

• Test Developer

• Consultant

• Build Manager

• Configuration Manager

• Scrum Master

Release Manager for mobile

(includes CM, Build Management)

About My Company Mobile.international belongs to eBay Main platform: mobile.de

•Biggest market place for automobiles in germany.

•One of the top-10 websites in germany; 49 million visitor per month (IVW 02/2009)

Other marketplaces• RO: mobile.ro• PL: pl.mobile.eu• FR: ebay-automobile.fr• IT:

ebay-automobile.it

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Focus of this talk

Main Statement:The combination of Branching + Quality Gates can be a good tool to improve the common sense of „DONE“ step by step, and to help testers to get integrated into agile teams.

This talk is related to thefollowing topics from the list

Transition to agile development Process

• How to organize testing activities?

• How to change the understanding of Testing?

Topics of interest of this Conference:

• Test Driven Development (TDD) / Behavior Driven Development (BDD)

• Acceptance Test Driven Development (ATDD)

• Automated Unit-level Tests

• Automated Integration-level Tests

• Automated System-level Regression Tests

• Automated ATDD

• Exploratory Testing on Agile projects

• Tools "for Agile Testing": for TDD/BDD; for ATDD; for Continuous Integration (CI), Build and Integration Management, Version Control Systems (VCS); etc.

• Managing testers on Agile teams

• Testing challenges for distributed team

• The roles of an Agile Tester

• How to engage testers from "traditional" development methodologies and help them transition to agile teams

• Continuous Testing (based on CI): e.g. Automated Regression Testing, Continuous Performance Testing, Continuous Load Testing, Continuous Scalability Testing etc.

• The Influence of Scrum or other Agile Methods on Conventional Testing Processes

• Evolutionary Testing Practices

• Measurable Improvements as a result of Agile Practices

• The Problems and solutions of changing to an Agile Culture seen from a Testing Perspective

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

What means „DONE“?

ScrumDONE =

releasable

Agile Release Model: • Deliver small chunks, not big fat

releases.• Deliver value to customer several

times a week w.r.t. business prios.• At any time, you can decide to

perform a release.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Definition of Done (DoD)

Source: Gupta,Definition of Done: A Reference

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

The Gap

Real World:

done means

„still a lot to do“

Ideal World:

DONE = releasable

?

Releasable

...

Feature tested

...

Unit tests green

Compiles

TestDevelopment

Mainline

Release Codeline

Releasable

...

Feature tested

...

Unit tests green

Compiles

MainlineRelease Codeline

TDevelopment

Test

►Codeline Quality ►Codeline Quality

►Process: Integrated Test Activities►Process: Waterfall

time

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

How to improve the common sense of DONE?

Many Aspects Process Tools Infrastructure People Skills Commitment Mind set

• „QA as Quality Police“

• Developer: „I am not responsible for quality.“

As member of an agile team, I am responsible to deliver value to the customer in time and quality.

Strategy - Blue or the red pill?

Strategy B:

Step by step

Strategy A:

Brute force

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

How to improve the common sence of DONE?

Long-Term Actions

Evangalize! (firm + carefully)

• Explain the need for an improved „done“ to all stakeholders.

• Convince first developers.

• Convince POs.

• Make tiny first steps for improvement + get commitment

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

The Start of Our Scrum Transition

Situation Q2 2008

• parallel development with multiple distributed teams• two outsourcing partners• 60+ team members • 10+ teams of different size• Fast rump-up: 30+ new/external team members • Scope:

• Many CRs• New marketplaces• Redesign• New apps

• One big codebase

First Steps• Evaluation projects

Decision

Use Scrum for all teams.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Phase 1 of Our Scrum Transition

TestDevelopment

►Process: SCRUM with waterfall QA

Mainline

Release Codeline

►Codeline Quality

►CM: Mainline development

Part-time assignement

Test system

Experiences

We delivered. (Almost in time.)

Integration of POs

(bridged gap between business unit and technology department)

Increased visibility

Reduced test coverage.

Reduced external and internal quality + increased technical debt

No one feels responsible for bugs.

Main impediment: low quality of trunk blocks many teams.

Release deadlocks.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

New Inspiration New Approach

Henrik Kniberg:

Actions

New Policies

• Every project teams develops in a project branch.

• No commits in the mainline.

• Every contribution for the mainline (and the next release) has to satisfy the conditions of a Quality Gate (< releasable).

Quality Gate: releasableQuality Gate:

ready for trunk

(< releasable)

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Phase 2 of Our Scrum Transition: Project Branches

►Integration of Test Activities

MainlineRelease Codeline

►Codeline quality

Test(Story, Acceptance,

Regression)

TestDev

Part-time assignement

►CM: Project branches

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Quality Gates

• Use existing DoD papers etc.

• Ask teams “Team Quality Contract”

• Discuss it, discuss why

• Get commitment by all stakeholders

• Plan iterative reviews + improvements

• Term from traditional development

• For an agile team, a quality gate can make quality expectations explicit.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Management of Integrations

Use self management of the teamBook merge slots (Wiki)

“conflict resolution”

Merge Token “Merge Frog”

Experiences

Delays

Integration queue increases

Team Reactions

- Deliver smaller chunks

- Split functionality acc. to business prio

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

How we test?

• full/partial system deployed in test system

•Module + Dependencies•Module •Class•Method

Source: Lisa Crispin & Janet GregoryAgile Testing: A Practical Guide for Testers and Agile TeamsAddison-Wesley, 2009

Selenium-Selenese

- SeleniumRC

Unit Tests Integration tests

Unit Tests

Functional Tests Exploraty Tests

UAT

Manual Functional Tests

.

UI

Tests

Acceptance Tests(API Layer)

Unit/Component Tests

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Pilot Project: Story Tests

• full/partial system deployed in test system

•Module + Dependencies•Module •Class•Method

.

ConcoridionSeleniumRC /Webdrivver

automated storytests

automated storyFRONTEND tests

Concoridion

.

UI

Tests

Acceptance Tests(API Layer)

Unit/Component Tests

Story Tests

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Expericences / Discussion

Q: Be strict?no – gates are just guidelines

Q: How many contributions missed the release stream?

> 25

Q: Pressure in the teams?

Q: Pressure on QA members?

Increased quality of TRUNK.

Increased release reliability

Increased release flexibility

Increased release „throuput“

(delivered more story points per time)

Increased responsibility

Increased understanding of DONE

Increased integration of testers

(some teams)

Small waterfall (some teams)

Merge overhead !!!

Merge risks re-test in R-test phase

Problem: completeness of test environments re-test in R-test phase

Long R-test phase

Still a QA bottleneck

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Step 2a: Perform a „Team Release“

Quality Gate: releasableQuality Gate:

ready for trunk

= releasable

Rationals:

• high business priority

• high risk

• contract with fixed starting date

Pros

• reduced release risk

• very clear responsibility of the project team

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Test

Phase 3 of Our Scrum Transition:

Mainline

Release Codeline

►CM: Project branches

►Process: SCRUM with small waterfall + big waterfall

►Codeline Quality

Experiences

More conditions of quality gates satisfied.

Decreased R-test phase length.

Increased understanding of DONE

Increased integration of testers

(most teams)

....

TestDev

Action #3

• Reduced regression test phase lenght + team size.

• Testers are assigned to their project team, rather than to the R-test team.

• Be more strict: Verify the conditions of the Quality Gate.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Integration of Supporting Disciplines

Q: How to integrate supporting disciplines / aspects?

(Site Operations, NFR, DB, Architecture, Domain Model, etc.)

Options

Option 1: Just mention in Quality Gate … distribute knowledge and expectations

Option 2: Test as part of the Quality Gate

Option 3: Use reviews

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

What about Continuous Integration?

Standard: Use...• Continous build + unit test• Package• Deployment• Regression tests• Metrics: test coverage + more

Advanced: Try to automate ...• Verification criteria for Quality Gates• Merges• CM tasks (creation of branches + CI build configurations)• Deployment• DB Setup

Rule:

Use CI for every active codeline. Tasks:

Create + maintain necessary infrastructure

Q: How is maintaining it?central team vs. project teams

Conclusion:

This is a big effort, but it‘s worth every minute.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

What‘s next?

Kanban

Simplify mergingDistributed VCS (GIT, ...)

Automatic Merging

Team ownership for...Test system

CI System

Distributed VCS (GIT, ...)

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Conclusions

The “branching approach” can help.

It requires high skills: CM, test systems

Merge costs and risks are high

…but it can help the teams to get a feeling for the quality they deliver

…and improve the common sense of DONE.

October 13, 2009 – Agile Testing Days – Alexander Schwartz – How to Develop a Common Sense of “DONE”?

Q & AQ & A