Testing & test writing SA3 All Hands Meeting CERN

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

Testing & test writingSA3 All Hands Meeting CERN

Andreas Unterkircher

CERN Grid Deployment



Content

1. Observations from EGEE II

2. Directions for EGEE III

3. Testing as part of patch certification

4. Test infrastructure and frameworks

5. Test writing

SA3 all hands meeting 2



Observations from last year

• We have a quite extensive list of tests now– https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests

• Test development centered around functionality tests• Stress tests are mainly being done manually• Focus on the test framework (SAM) was too strong and

distracted from writing tests• Individual testing through SAM is difficult (registration

of machines etc.)• Testing is not well integrated into patch certification

(lacking trace of testing in the patch)• Virtualization became indispensible at CERN




Directions for EGEE III

• Make/keep tests independent from a test framework• For the framework we follow production, i.e. SAM will

be replaced by Nagios• The framework will be used to ensure that our core

testbed is working• Enabling the framework to do testing for partner patch

certification is of lower priority• Development of regression tests• If possible move parts of testing into ETICS• We are not concerned with unit tests (they should be

written by the developers)




Testing as part of patch certification

• Testing necessary for a particular patch– Functionality tests– Regression tests– Tests for attached bugs

• A list of functionality tests and regression tests that have to be executed is available (and evolving).

• There must be a comment about the outcome of the tests in the patch. The results must be available (pasted, attached, link)

• If possible verification of a bug should result in a regression test for the bug




Functionality tests

• We have a list of mandatory functionality tests per service

– https://twiki.cern.ch/twiki/bin/view/EGEE/ServiceCertificationChecklist

• The functionality tests can be executed on the command line. We provide command line wrappers to execute the tests that were originally written for SAM (VOMS, LFC, WN/CE – more to come)

• Description of all available tests: – https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests




Regression tests

• Simple Bash “framework”• Can be executed on the command line• More information:

– https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Regression_Tests

• Currently tests for around 20 bugs are available




Tests for attached bugs

• For every attached bug there must be a comment in the patch

• If possible a regression test should be written for every bug in the patch

• Depends on the bug classification (cf. my talk at the All Hands Meeting in Dublin, Dec. 07), but many bugs are in this category (more than 50% in Dec. 07)

• The regression test should be delivered soon after patch certification




Bug regression test classification

• Test can be done on one isolated node– Case 1: installation of rpms, configuration with yaim, valid proxy

available if needed

• Test needs more nodes – Case 2a: a server (also included: BDII) is set up and needs to be

tested via a client API/CLI from another machine (more realistic)– Case 2b: an API needs to be tested and needs a server

Case 2b-1: server needs special setup (version etc.) Case 2b-2: server can be standard

– Case 2c: several nodes are involved (FTS)

• Test is difficult– Case 3: web interface tests, security bugs (no info available),

complicated workflow, problems that only occur after running the service for a long time etc.




Test infrastructure and framework

• A stable and complete testbed is kept at CERN. This includes a dedicated CA and VOMS server

• The CERN testbed is monitored and tested by NAGIOS• Partner bring up the machines necessary for certifying

and connect them to the testbed– It is up to the partner on how to provide the machines

(virtualization, permanent machines)– Connecting to the testbed should be easy (cf. Louis’ talk)

• Partner do the tests manually on the command line and provide the results (link/TWiki, attachment to patch, pasting)




Test infrastructure and framework

• Testing of partner sites/machines via Nagios will be investigated

• Goal: A set of Nagios tests will be run against partner sites/machines whenever they are connected to the testbed.

• This will not happen immediately as Nagios is currently being introduced. We have to get experience with it.

• Testing must not be slowed down because of framework issues. Thus we must ensure that tests can always be executed manually.




ETICS as a test framework

• We started to investigate the possibility to do testing within ETICS. Areas of interest are

• Deployment tests– Immediate deployment test after a build by a developer: For

any set of artifacts produced by a developer (typically in the volatile repository) this test should check if well defined versions (production, pps or cert) of the affected gLite services can be updated with these artifacts.

– Regular Deployment test for a complete build




ETICS as a test framework

• We started to investigate the possibility to do testing within ETICS. Areas of interest are:

• Simple Regression tests– Tests that can be done on one node that is configured to be part

of the certification testbed




Regression test writing

• More information: – https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Regression_Tests

• Check out from CVS• Regression test for bug 123 = one file “bug123”

implementing 3 bash functions:– test_bug123_pre ()– test_bug123 ()– test_bug123_post ()




Regression test writing


./ regTest.sh

./bugs/

bug1596 bug22165 …

./config/

commonFunctions.sh config.sh

./testlists/

mybugs.txt mybugs.sh

./regTest.sh –tl testlists/mybugs.txt

1596

22165

…

VO=dteam

SE_DPM=se.cern.ch

…

test_bug1596_pre ()

test_bug1596 ()

test_bug1596_post ()



Test writing – missing tests

• New list of missing tests (evolving list): – https://twiki.cern.ch/twiki/bin/view/EGEE/GliteTestMissing

• Test should be written to be executed on the command line. Eventual integration into other frameworks (SAM, Nagios, ETICS) will be considered later

• AMGA: API tests (when released)• APEL• BLAH: test plugins for different batch systems, test

plugins that come with CREAM• CREAM CE: direct submission to CREAM, CLI• dCache: test SRM client that comes with dCache• DPM: several tests currently being developed at CERN

but more tests are needed:





• DPM: – DPNS server via DPNS CLI, DPM server via DPM CLI– Daemons on DPM: gridftp, rfio, xrootd, httpd– DPM APIs (C, Python, PERL)– SRM tests against DPM using dCache SRM client commands

• FTS: VOMS support, SRM copy channel (dCache – DPM), Java API

• glexec• Information System: enhance GSTAT, lcg-info, lcg-

infosites• LFC: Perl API• Logging and Bookkeeping: verify and enhance existing

tests





• Medical Data Management• MyProxy• Regression tests• SCAS• VOBOX: integrate existing tests




Test writing

• First write a test plan– List which functionality should be tested– List which CLI commands should be tested– List which functions/methods in the API should be tested

• Write the tests and check with test plan if everything is covered by some test

• Provide documentation for the tests• Tests will be checked in CVS


Documents

Testing & test writing SA3 All Hands Meeting CERN