Software Testing with Python - Thinkware AB

Software Testing with Python

Magnus LyckåThinkware AB

www.thinkware.se

EuroPython Conference 2004Chalmers, Göteborg, Sweden

© 2004, Magnus Lyckå

In the next 30 minutes you should...

● Learn about different aspects of software testing such as unit tests and acceptance tests.

● See how the standard modules for unit tests are used, learn about some alternatives and have a grasp of their respective pros and cons.

● Know about some options for acceptance tests.● Know about some standard Python modules and

third party tools that are helpful in software testing and related activities.

What is software testing?

● “Testing means verifying that your code is running correctly by exercising the code under known conditions and checking that the results are as expected.” – Alex Martelli, Python in a Nutshell

● If we are serious with our requirements on a software system, we should make sure that we verify them, and testing is the most common kind of verification for software.

Software testing in a context

Software Quality Assurance

Software Verification

Software Testing

Automating tests

● Writing automated tests is usually more work than testing things manually once, but they make it much easier to...– Work in a repeatable and predicable way– Run tests more often– Run tests unattended– Find regression bugs– Test more operating systems– Manage the test load as the system grows

Example of requirements and tests on different levels

System Specification Acceptance Test

Detailed Design

API Specification

Unit Test

Integration Test

Requirements and tests on different levels in Extreme Programming (XP)

System specified through customer tests

Design by writing

API validated through

programmer tests

continuous integration

Consequences of the XP approach● Requirements are written in a verifiable way.● No discrepancies between requirements and tests.● We get unambiguous and repeatable ways of

verifying requirements.● Continuous picture of project progressBut...● Customers must be able to understand the tests.● We need test automation frameworks.● Automated tests → more software to maintain.


Unit testing

Unit tests in Python

● Unit tests, or programmer tests, are tests the programmer writes to make sure that the code does what the programmer intended it to do.

● Unit tests can also help document how a piece of code (e.g. a class) is supposed to be used.

● Python has two standard modules for unit testing, unittest and doctest.

● There are several third party modules that aim to complement or replace unittest.

The unittest module

● The Python unittest module is modeled after the unit test modules developed within the XP community by Kent Beck and Erich Gamma.

● It's intended for a test-first approach, where tests are written before the actual code that it tests.

● Always writing tests first is a bit like always eating your meat before you eat your dessert...

● When your tests pass, you're done. (XPers combine this with refactoring.)

Concepts in unittest● A test fixture consists of the actions needed to

setup for a test, and clean up afterwards.● A test case is the smallest unit of testing. It might

for instance call a function and check the results.● A test suite is a collection of test cases and/or test

suites that should be executed together.● A test runner runs test suites (or individual test

cases), collects the results and presents it as text or graphically to the user.

The TestCase class

● Test cases sharing a common fixture will be implemented as methods (whose names start with test) in a sub class of unittest.TestCase.

● The test runner will create one instance object per test case, and run SetUp, followed by the test case method and finally tearDown.

● Checks are made with the methods fail, failIf, assert_/failUnless, assertEqual/failUnlessEqual, assertNotEqual/failIfEqual and assertRaises/failUnlessRaises.

The test runner

● In many cases, running tests is as simple as executing unitest.main() in the file containing the test cases.

● Testing progress will be reported, and the test runner makes a distinction between FAILURE and ERROR.– FAILURE means that the check didn't produce the

expected result.– ERROR means that something else in the test went

wrong. (I.e. You got an exception.)

Unittest template

import unittest import MyModule class MyModuleTest(unittest.TestCase): def testOneCase(self): pass if __name__ == '__main__': unittest.main()

Unittest examples

● divtest.py– Trivial example to demonstrate features of unittest.

● container_ut.py– Real world example from SystemSpecifyer (see

http://sourceforge.net/projects/systemspecifyer)

The doctest module

● The Python doctest module was written by Tim Peters to check that coding examples in doc-strings were correct, by testing them.

● It has also found use as a more general unit testing module, and from Python 2.3, this has been made more convenient.

● Using doctest helps you combine tests and documentation in a more readable way than unittest does.

Doctest exampledef divide(a,b): """ Return a divided with b. Divide will return the result of an integer division >>> divide(6,3) 2 Division by zero is handled as expected... >>> divide(1,0) Traceback (most recent call last): ... ZeroDivisionError: integer division or modulo by zero """ result = a / b return resultif __name__ == '__main__': import doctest, sys doctest.testmod(sys.modules[__name__])

Unittest vs doctest

● Unittest mantra: Test a little, code a little– The logical choice if you use extreme programming

or test-driven development of some other kind.– Maybe better for complex tests?

● Doctest mantra: Code a little, test a little– The natural companion if you prefer to experiment

interactively with your classes and functions in the Python interactive environment.

– Probably better for adding tests to already written code.

Unittest vs doctest on the web

● Charming Python: Testing frameworks in Python– http://www-106.ibm.com/developerworks/linux/library/l-cptest.html

● Literate Testing: Automated Testing with doctest– http://www.python.org/pycon/dc2004/papers/4/

● DocTest at WikiWiki– http://c2.com/cgi/wiki?DocTest

Other contenders...

● Sancho – a unit testing framework from MEMS Exchange. Adds coverage analysis etc.– http://www.mems-exchange.org/software/sancho/

● Peckcheck by Darius Bacon.– http://www.accesscom.com/~darius/software/clickcheck.html

● The “second standard library” std.– Here, 11:00 today...

Unittest complements

● Pester – the Python version of Jester. – It finds code that is not covered by tests, makes some

change to your code, runs your tests, and if the tests pass it displays a message saying what it changed.

– http://jester.sourceforge.net/

● Test coverage support– E.g. http://www.garethrees.org/2001/12/04/python-coverage/

● Mock objects: Here, 9:30● A better test runner: Here, 10:00


Integration testing

Python Integration Test

● No static linking. We must run code to test interfaces! (On the other hand, compilers and linkers fail to see lots of integration problems...)

● Since building/linking isn't an issue in a pure Python project, we can use either unit test tools or acceptance test tools for integration tests.

● Python is good at gluing things together, and thus helpful in all sorts of integration work. See e.g. http://www.thinkware.se/cgi-bin/thinki.cgi/UsingPythonWithOtherLanguages


Acceptance testing

Test Frameworks using Python

● QMTest – Uses Python for test expressions.– http://www.codesourcery.com/qmtest/

● PyFIT – Python clone of Ward Cunningham's Framework For Integrated Testing (FIT).– http://www.xprogramming.com/software.htm

● Software Testing Automation Framework (STAF) Big framework from IBM with Python API.– http://staf.sourceforge.net/

● TextTest – See more here, today 11:30!


Not quite software testing...

The usual suspects...

● Python is excellent for analysis and manipulation of data. A great tool for test related work.

● The re library is useful but use special tools when available, such as for parsing XML files.

● For dealing with files you might use the modules gzip, zipfile, codecs, filecmp, struct, sgmllib, xml.* etc

● Python interfaces well with internet services, database, and all sorts of external programs.

difflib● More or less like the unix diff utility, but as a

Python module. (New in version 2.1.)● Useful when we want a more detailed response

than FAILED from a test – particularly if we are trying to spot small changes in big amounts of data.

● Great for spotting differences in configurations, for instance database schemas.

Difflib example

>>> import difflib; d=difflib.Differ()>>> diff =d.compare(['Hello World', "This is the same.", "Time flies like an arrow. Isn't that great?"], ['Hello World!', "This is the same.", "Fruit flies like a banana. Isn't that great?"])>>> print "\n".join(diff)- Hello World+ Hello World!? + This is the same.- Time flies like an arrow. Isn't that great?? ^ ^^ - ^^^^+ Fruit flies like a banana. Isn't that great?? ^^^ ^^ +++ ^^

AT&T Graphviz

● Tools to generate graphs from C-like text files.– Dot – for directed graphs.– Neato – for undirected graphs– Fairly clever algorithms for decent looking layout– Generates graphs in many file formats– Suitable for automatic generation of graphs– Some control over placement is possible

● Great for dependency analysis!● Not Python, but good anyway... :)

Remember this?

System Specification Acceptance Test

Detailed Design

API Specification

Unit Test

Integration Test

AT&T Graphviz example 1digraph G { label = "V-model";

sysspec [label="System Specification"]; systest [label="Acceptance Test"]; apispec [label="API Specification"]; apitest [label="Integration Test"]; unitspec [label="Detailed Design"]; unittest [label="Unit Test"];

sysspec -> apispec -> unitspec; unittest -> apitest -> systest; sysspec -> systest; apispec -> apitest; unitspec -> unittest;}

AT&T Graphviz example 2digraph G { label = "V-model"; node [shape=box, style=filled, color="#CCCCFF"]; sysspec [label= "System Specification"]; apispec [label="API Specification"]; unitspec [label="Detailed Design"]; node [shape=box, style=filled, color="#FFFF99"]; systest [label="Acceptance Test"]; apitest [label="Integration Test"]; unittest [label="Unit Test"];

{rank = same; sysspec; systest}; {rank = same; apispec; apitest}; {rank = same; unitspec; unittest};...

Useful Books

● Python in a Nutshell, by Alex Martelli● Text Processing in Python, by David Mertz● Software Test Automation, by Fewster & Graham● Just Enough Software Test Automation, by

Daniel J. Mosley & Bruce A. Posey● Testing Extreme Programming, by Lisa Crispin

and Tip House● Test-Driven Development By Example, by Kent

Beck

Documents

Software Testing with Python - Thinkware AB