Manuel Gallas

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 1

LCG-SPI: SW-TestingLCG Applications Area

GridPP 7th Collaboration Meeting [email protected]

LCG/SPILCG Software Process & Infrastructure

LCG Software Process & Infrastructure

mailto:[email protected]








Outline and Context

SPI SW-Testing

• Overview• Test FrameWorks• User support• Test policies• Test documents

LCG Infrastructure

SPI project

http://spi.cern.ch/

LHC grid software

applications (LHC experiments, projects,

etc)LCG Application Area

POOLPI

SIMU

SEAL

SW-Testing

http://spi.cern.ch/

http://spi.cern.ch/

http://spi.cern.ch/

http://spi.cern.ch/

http://spi.cern.ch/

http://spi.cern.ch/

http://spi.cern.ch/



Overview

•Software testing will be an integral part of the software development process.•All level of software testing should be run as part of an automatic process.

GOAL

Software testing

Integration test

System test

Acceptance test

Unit test

Sw-testing team

IntegrationTests

System Tests

ExamplesLHC experiments

Use in Exp.Test FrameWorks

SW-testing doc

SW- testing support

SPI SW-Testing

SW-testing polices

Code Documentation

Software developer

CODE

Work PackageTest

Automated testing

Nightly Building

System

SW-Testing



Test FrameWorks

Aim: to help developers:

• to produce code for testing • to run tests in automatic way

Our inputs: • Contacts within HEP-community.• What is available as free open source code.

Our constrains: • Avoid commercial software and licensing problems. • Avoid “do it yourself solutions”•Try to adopt commonly used open-source software.

SW-Testing

Requirements: • Work with different languages (C++, Python, …)• Allow two ways of testing: “test the output of the code” and “test inside the testing-code”.• Easy way to integrate existing tests.• Allow the creation of dependencies among tests• Organize the tests by components or packages.• Regression testing• Easy integration with the Nightly Building System.• Provide also a graphical interface.



•Scope: Unit –testing

•The same “assertion style” in different languages.

•Provides:- The name of the test case that failed.

-The name of the source file that contains the test.

-The line number where the failure occurred

• Different platforms/compilers: (Linux/Solaris/Windows)

Test FrameWorks: X-Unit family

X-Unit family

JUnit

QtUnit

CppUnit

PyUnit

A simple test:

1. Subclass the TestCase CppUnit or PyUnit class

2. Override the method runTest().

3. When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds

SW-Testing



Test FrameWorks: Oval

Oval:• validation and regression • used in CMS

•Can be use for Unit-test to Validation tests.

•Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)

• It is possible to set different run environments.

• Can run external scripts and external binaries.

Ovalfile(configuration)

ExecutableTest

Source Code

Referencefile

Log file

Oval run

Ovaldiff

Oval build

SW-Testing



Test FrameWorks: QMTest

QMTest

•Uses a graphical interface for creating and running tests

• Runs tests in parallel

• Organizes tests hierarchically

• Supports execution of a single test or many at once

• Records dependencies between tests

• Can be run in batch mode -> easy integration with the Nightly-Building systems

SW-Testing



Test FrameWorks: Global picture

TestScripts

Valgrind

Aim: to help developers:

• to produce code for testing • to run tests in automatic way

OvalFile

Oval

Logfile

Reffile

?

• Automated tests

• Regression test

CppUnit

PyUnit

X-Unit family

JUnit

QtUnit

QMTest

• Uses a graphical interface for creating and running tests• Runs tests in parallel• Organizes tests hierarchically• Supports execution of a single test or many at once• Records dependencies between tests

In LCG AppArea runs what is in the OvalFile and uses oval.

Oval

SW-Testing



SPI SW-Testing


SW-Testing



User support: Web information

http://spi.cern.ch/

Test FrameWorks

HowTo

Test doc

Policies

SW-Testing

http://spi.cern.ch/software_development.html



User support: Tools supported

CppUnit

PyUnit

Provided as free software by the SPI External Software Service

•Different platforms: (Linux/Solaris/Windows)

•Different compilers•Integrated with SCRAM and ready to be used•Installed in AFS/DFS•Instructions to install them in a local machine

How to use them ?

Oval

QMTest

SW-Testing

http://spi.cern.ch/extsoft/



1

User support: How-To

2

3 Follow sw-testing policies

*

SW-Testing



SPI SW-Testing


SW-Testing



Sw-Testing policies


For the projects

For the developersFor the testing responsibl

e

Sw-testing policies are needed to ensure:

• Software is created together with the tests.• Test directory structure and naming should be homogeneous and this will imply easy life to have automatic tools.• Common and well integrated tools

SW-Testing


http://spi.cern.ch/Components/UnitTesting/UserDocumentation/Web/SwTesting-Policy.html#1Item








SPI SW-Testing


SW-Testing



Test Documents

• For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.• We use a template•Test Case template collects information about the existing tests and how they cover the expected functionality of the software.

SW-Testing

http://spi.cern.ch/Components/UnitTesting/UserDocumentation/Web/Testing_Doc_Templates/TESTING_testplan_template.html



Thanks to:• PPARC• LHC experiments• LCG

Feedback and interaction are always welcome!!

SPI SW-Testing


SW-Testing

Conclusions:Software testing is an important task and we have tools in place to do it.