T-76.613 Software Testing and Quality Assurance

HELSINKI UNIVERSITY OF TECHNOLOGY

T-76.5613 Software Testing and Quality Assurance

3.10.2007

Testing in Agile Software Development

Juha ItkonenSoberIT

2Juha ItkonenSoberITHELSINKI UNIVERSITY OF TECHNOLOGY

Overview

Agile Testing and Testing in Agile Software DevelopmentChallenges of Testing in Agile DevelopmentHow Agile Quality Practices Work?Agile Unit Testing and TDDAgile Acceptance Testing with FIT


Process models from waterfall to agile

Time

Requirements

Design

Implementation

Testing

Waterfall Incremental,e.g. RUP

Agile - XP

Functionality

Redrawn from Beck. Embracing change with Extreme Programming, 1999.


V-model

Intuitive and easy to explainQuite adaptable to various situationsMakes a good model for training peopleBeats the code-and-fix approach on any larger project

Requirements

Unittesting

Coding

Moduledesign

Architecturedesign

Functionalspecification

Acceptancetesting

Systemtesting

Integrationtesting

Build Test


Problems in applying V-model

Document drivenRelies on the existence, accuracy, completeness, and timeliness of development documentation

Communicates change poorlyIgnores the fact that, in practice, software is developed in a series of handoffs

Does not fit into modern iterative development processes


Agile Software Development Manifesto

In agile development we value:Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

+ 12 more detailed principles

http://www.agilemanifesto.org/

How do these values affect testing?


Plan-Driven vs. Agile

Wanted result ?

Planned result


Agile Software Development – What About Testing?

Agility in software developmentWorking software is most important deliverableResponding to change – not resisting changeTight development and release cycle

Iterative and incremental processFast feedback, frequent adjustments

Efficient collaboration and communicationWithin the team and with the customer

Agility from the testing viewpointPlaces many challengesTesting must also be agile in agile developmentYou can learn a lot from the agile methods


Two views of agile testing

eXtreme TestingAutomated unit testing

Developers write tests Test first developmentDaily builds with unit tests always 100% pass

Functional testingCustomer-ownedComprehensiveRepeatableAutomaticTimelyPublic

Exploratory TestingUtilizes professional testers’skills and experienceOptimized to find bugsMinimizing time spent on documentationContinually adjusting plans, re-focusing on the most promising risk areasFollowing hunchesFreedom, flexibility and fun for testers

Focus on automated verification – enabling agile software

development

Focus on manual validation – making testing activities

agile


Some Agile Principles

Satisfy the customer through early and continuous delivery of valuable software.Working software is the primary measure of progress.Deliver working software frequently, from a couple of weeks to a couple of months.Welcome changing requirements, even late in development.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Business people and developers must work together daily throughout the project. Simplicity--the art of maximizing the amount of work not done--is essential.

http://www.agilemanifesto.org/

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.


Challenges that agile principles place on testing

Agile Principle ChallengeFrequent deliveries of valuable software

- Short time for testing in each cycle- Testing must be time-boxed, too

Responding to change even late in the development

-Testing cannot be designed beforehand based on specifications- Tests must not prevent change

Relying on face-to-face communication

- Getting developers and business people actively involved in testing

Working software is the primary measure of progress

- Quality information is required early and frequently throughout development

Simplicity is essential - Testing practices get easily dropped for simplicity’s sake


Contradictions with traditional testing principles

Testing principle Contradicting practices in agile methodsIndependency of testing and destructive attitude

- Developers write tests for their own code- Developers concentrate on constructive QA practices, i.e., building quality into the product and showing that features work

Testing requires specific skills

- Developers do the testing as part of the development- The customer has a very important and collaborative role and a lot of responsibility for the resulting quality

Oracle problem - Relying on automated tests to reveal defects

Evaluating achieved quality

- Confidence in quality through tracking conformance to a set of good practices


How Do the Agile Quality Practices Work?


Organizing testing (waterfall vs. agile)

Programmers

Testers

Waterfall model Agile models (XP)

Customer

Programmer

Programmer

Tester

Idea:Testing in collaboration


Quality Assurance in Agile Development

Plan-driven quality assurance does not work in agile context

There is no V-model and waterfallThe role of documentation and specifications is secondaryThe rhythm of development is fast and tightRoles and responsibilities are assigned differently

Cornerstones of agile quality assuranceConstant and tight rhythmCollaboration and communicationRigorous low level (quality) practicesTest-Driven Software Engineering


Rhythm

Continuous unit-level integration and test cycleShort incremental release cyclesCompleting features in short cycles and small increments

Complete, integrated, testedBuilding quality and tracking the quality level in short cycles

t

Release

Iteration Heartbeat


1 Rhythm – viewpoint of testing

Feedback is faster and more frequentbetween development and testingfrom customers and users

Defects and issues are found earlierFeel the rhythm - tackle tasks before they bunch up

Testing features in small increments is easierLess new defects to hinder testingLess open defects to distract development

Real quality level and real progress better visibleRisks of testing and deployment are smaller

Easier to estimate and plan testing phase (or tasks)

Challenges- Short time for testing in each cycle- Testing must be time-boxed, too- Testing cannot be designed beforehand - Quality information is required early and frequently - Testing practices get easily dropped


2 Communication

Communication in agile development does not work in a traditional way

Traditionally others communicate to testers with requirements and specification documents and testers respond with defect reports and test reports

“So we can let free of the illusion that documents will save us. We can view them as they are: interesting texts, partly fictional, often useful.” - Brian Marick


Collaboration

Collaboration is effective way of sharing information, knowledge, and understandingImproves common understanding and makes communication easy and efficient

Less guesswork and proceeding based on wrong assumptionsInformation moves faster and with less overhead

Especially effective in the home ground of agile development

When developing new, unfamiliar thingsWhen requirements are vague and changingWhen customer is easily available for collaboration


Communication and collaboration – viewpoint of testing

There are testers also in agile teamsTester’s role is to help the development team to create good enough quality softwareTesters bring expertise and skills as well as the testers destructive viewpoint in the teamBy working together with developers and the customer the reliance on comprehensive documentation is not needed

Challenges- Tests must not prevent change- Getting developers and business people actively involved in testing- Developers write tests for their own code as part of the development- Developers concentrate on constructive QA practices- The customer has a lot of responsibility for the resulting quality- Relying automated tests to reveal defects


3 Low Level Quality Practices

Quality practices are tightly integrated into the daily heartbeat rhythm of the development

Testing each individual feature as part of its developmentBuilding quality using constructive practicesQuality is part of development – not delayed any furtherE.g. automated unit tests, pair-programming, continuous integrations, refactoring, …

Goal is to achieve high quality level before the code leaves the developer’s desk


Some practices of an agile developer

Integrate early integrate oftenKeep your project releasable at all timesAutomate acceptance testingUse automated unit testsUse it before you build it –TDDWrite code in short edit/build/test cyclesEmphasize collective ownership of codeKeep a solutions log…

Keep it simpleDon’t fall for the quick hack Write code to be clear not cleverWarnings are really errorsProvide useful error messagesBe a mentorShare code only when it’s readyReview codeKeep others informed of the status of your work…

V. Subramaniam & A. Hunt, “Practices of an Agile Developer”,The Pragmatic Bookshelf, 2006.


Low level quality practices – viewpoint of testing

Technical quality of the software is highLess trivial defects – easier to test hard issuesBoring, routine, checking procedures are automatedTesters and developers have a common goal of creating high quality software

Testers help developers <–> developers help testers

Testers can help developers Pair tester, who helps a developer to test each functionalityTester writes tests for the current functionality under development

Tests help developer in implementation and provide fast feedback already during implementation

Challenges- Short time for testing in each cycle- Testing must be time-boxed, too- Tests must not prevent change- Testing practices get easily dropped


4 Test-Driven Software Engineering

Use it before you build it!

Agile test-driven approach can be applied also on the level of customer requirements and acceptance testing

Tests can help programmers, testers, and business people (customer) communicateTests can by used to guide development workTests act as the true measure of real progress


Test-Driven Approach

Testers, programmers, and customer design together good tests based on the customer requirements

How, exactly, the features and core business logic should workMajor problems in the implementation (risks)Documenting the important details and potential problem areas beforehand

Tests guide the programmers in implementation work

What exactly must be done; how special cases should behave; when the feature is doneWhat are the risks of this feature; which issues to pay attention to

Tracking the amount of passing tests communicates the progress of development

As agile values and principles state 0

5

1 0

1 5

2 0

2 5

3 0

1 2 3 4 5 6 7 8 9 1 0


New Roles for Tests in TDSE

In Test-Driven Software Engineering tests have a clear role in the early requirements and design phasesTests help developers, testers, and customer to communicate

Common vocabulary and understanding

Tests document and concretize the details of requirementsConcrete examples of the functions and logicSpecial cases and exceptionsEasy to extend and update during the development

Tests guide the implementation workFast feedback to developers when automated

Tests provide an honest and agile metric of true progress in terms of working software


Agile Unit Testing and

Test-Driven Development


No manual tests!

Chapter titled ”Manual Tests” in Testing ExtremeProgramming (Crispin & House 2003)

”No manual tests.”


Unit testing

”A unit is the smallest possible testable software component”

Various interpretations existsProcedure / functionClass / object / methodSmall-sized component

Goal is to ensure that software units are functioning as the developer intended

E.g. a function returns correct value when exercised with valid parametersE.g. function behaves correctly when given an invalid input

Automated unit test is a piece of code that exercises another piece of code


Document driven vs. agile unit testing

”Testing should be carried out by a group that is independent of the development group”

Course book presents a unit testing viewpoint which appreciates this principle

Unit testing should be a separate task conducted after the development is doneUnit test design is based on written specifications

Modern ideas of unit testing represent slightly different viewpoint

Unit tests are written by developerAutomation is essential part of unit testingUnit tests might even be written before the actual program code

This lecture is about agile unit testingRead the course book Ch. 6 and understand the differenceThe same tools are used by both disciplines


Unit testing - a vital testing level

Writing software is difficult and every programmer makes mistakes

There will be defects in all software you writeUnit testing lays the foundation for good quality softwareUnit testing lays foundation for effective testing on higher levels

Defects that slip to upper levels (integration, system or acceptance testing)

More difficult to spotTime consuming to debugUses organizational resources

— Writing, prioritizing and managing test and defect reportsBad unit level quality slows down integration and system testing


How do you know that your code works correctly?

Do you just trust your luck?Do you use “printf” or ”System.out.println()” outputs to see what happens in your code?Do you use debugger to find out how your code really works?Do you ask your friend/colleague to check through your code?Do you create small (temporary) widget to access the piece of code directly runtime?Do you use some ugly, cumbersome, time consuming practice that leaves trash into your code and is thrown up to the user’s face when you least expect it…

“Getting here is not possible, something is really messed up in this code!!!”“Fatal error 9981: Assertion failed : av_d_dx > 0”


Agile unit testingTest-Driven Development Writing and running tests is integral part of daily development rhythm

Tests are developers’ QA and programming tool

All unit tests are automatedAutomated tests are run as often as possible

Gather all your tests to a test suite and use it for regression testingIntegration, nightly, milestoneDuring development; after every change; new code or bug fix


Test-Driven Development

Writing test before code to be tested”a little test, a little code, a little test, a little code, ...”Tests are added gradually during implementation – not in large lump afterwards

Process of writing tests drives low-level design and programming

Tests specify what code should doTests validate that code does what it should

Actually, a design and coding practiceOne of the core practices of Extreme Programming

Developers have been applying TDD for several decades


TDD episode (1/2)

Proceeds step by stepWrite a test. Design and implement just enough to make the test pass. Repeat.

Testing and coding alternate in very small steps

Duration of one cycle should be a few minutesSmall steps – difficult to make mistake

Write test

Write code

Run tests

Fail

Pass


TDD episode (2/2)

Episode is over when you can’t write a failing test anymore

Write test for each requirement of the codeWrite test for each point that can possibly break

One cycle at a timeDon’t write a bunch of tests at once

Refactor if you ever see the chance to make the design simplerRun all tests after finishing episode

Make sure you did not brake anything else


Test-Driven Development – claimed benefits (1/2)

Close feedback loopTDD cycle is very short – know if code is working right after you programmed it

Task-orientationEncourage programmer to decompose problem into manageable programming tasksHelps to maintain focus

Low-level designProgrammer is forced to think which classes and methods to createhow they are used, how to name them, what arguments are needed, what is the return value


Test-Driven Development – claimed benefits (2/2)

Results better codeIf the test is too hard to write, the code being tested is too complicated

Results testable codeProgrammer can’t end up with code that cannot be tested

Effect on qualityTesting becomes part of the development process and gets doneSide effect of TDD is that code gets thoroughly unit tested


Is Test-Driven Development testing?

What is the goal and attitude of a developer?

Do the tests reveal defects?

What are the tests based on?

How is the quality of the unit tests assured?

What information the tests actually produce?


Unit testing frameworks

Convenient frameworks for writing unit tests as part of the programming task

In the same environment with the same language

Tools of xUnit family are the most famous onesJUnitCppUnitNUnit…

The unit testing frameworks provide a convenient way of writing and executing unit tests

Standard and convenient way to write tests, log results, group tests into suites


Basic building blocks of a unit testing framework

A way of writing test cases for each class, method, function, …Test case is a function or a class – a piece of code

A way of managing suites of test casesA data structure that is used to group test cases into logical collections

Assert methods/functions that enable easy logging and reporting test results

The framework provides standard functions that are used to checkthe test resultsAutomated result logging and reporting provided by the framework

A way of executing test cases and reporting the resultsGUI and command-line based execution tools (Integrated to IDEs)Convenient report generators (Xml, html)

Initializing and cleaning upThe framework ensures that init and clean-up functions are properly executedInitialization (or set up) function to initialize the variables, test data, etc. before executing each test caseClean up (or tear down) function to release the used resources and clean up after executing each test case

Documents

T-76.613 Software Testing and Quality Assurance