SciDAC SSS Quarterly Report

Sandia Labs

May 10, 2005

William McLendonRon OldfieldNeil Pundit

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

APItest - Overview

• Current Release : v1.0

• Test Driver tool for testing software– Ordered tests– API Testing

• Available for download:– ftp://ftp.sandia.gov/outgoing/apitest/

2

Work Since Previous Meeting

• New Look via Cascading Style Sheets– Can be turned off if needed.

• New Capabilities– PASS/FAIL status with batch files.

• Some new XML options for batch tests.– Better reporting of parse errors in XML– (very) preliminary connectivity to MySQL DB

• Saves results only, no browse capability.• Finished User Guide

– Preliminary R&A approval• Slight modifications needed and it’s out.

• Bug Fixes

3

APItest - Features

• Open Source (LGPL)• Scripted Test Driver

– Runs XML test scripts.• Written in Python

– Portable– Object Oriented

• Customizable• Built in Test Types

– Shell– Script– SciDAC SSS (ssslib)– TCP/IP

• XML Error Helper– Help find XML Errors

• Conditional execution of tests & test ordering.

• Some Test Features– SUID Capability– Test Timeouts– Pattern matched results– Batch Status Conditions

• User interfaces– Interactive via browser.– Command line mode.

• Limited MySQL DB Support– (under development)

4

APItest – Package Requirements

• Python 2.3 or greater– lower versions can work but the setup script requires

distutils, which is a python >= 2.3 package.• ElementTree (www.effbot.org)• Twisted (www.twistedmatrix.com)

– New version of 2.0.0 has changed the twisted library and added new dependencies. Discussed on next slide.

• SSSlib (ftp://ftp.mcs.anl.gov/pub/sss/)– optional, for SSSlib tests.

• MySQL– optional, for MySQL database connectivity.

5

Recent Changes in the Twisted Library

• With Twisted-2.0.0 the distribution has been split up.

• Twisted-2.0.0 Prerequisites:– Python 2.3 or greater– ZopeInterfaces-3.0.1 or greater

• TwistedWeb Prerequisites– Twisted-2.0.0 must be installed first.

• APItest still works with Twisted-1.2.0– Still available at www.twistedmatrix.org with a little

digging.

6

Batch File PASS/FAIL

• 2 New additions to Batch files• New ATTRIBUTE to a <test> element

– mustPass = { TRUE / FALSE }• Determines if a test counts towards batch file pass/fail

stuats.• If set to true (default), this test MUST PASS or the batch

from which it is called will FAIL.• If set to Fail, this test will not be considered in batch status

determination.• Must provide a <test> element to set this attribute.

• New Element– <setParameter key=“mustPass” value=“True,False”/>– Changes default for mustPass.

7

MySQL Connectivity

• Started adding MySQL connectivity• Very new and certainly in testing.• Some new packages required to make it work

– MySQL– MySQLdb python modules

• Saves results to DB only, no browse capability yet.• Command line driven

– Will show up in –help and the userguide.• More work than it seems like. Need to rewrite a couple of

things in APItests’ jobManager to make things work the “right” way.

8

New “Look and Feel”

• Added a new look to the gui browser.• Tabbed menu system• Cascading Style Sheets (CSS)

– There are some compatibility issues, not all browsers render it necessarily the same.

• Added a “compatibility mode” option incase CSS messes up.– Add –nocss to command line.

9

APItest Main Screen

• Lists available tests• Color codes tests

– Batch files– Individual tests– Invalid test files.

• Organized by directory• Details of tests can be viewed

by clicking the “View Test” button.

• Click SUBMIT to execute selected tests.

10

Running A Test

• Session screen shows the currently running test.

• Tests show their PASS/FAIL status.

• Batches show an overall PASS or FAIL and some statistics.– Total # of tests in batch– Total # that passed.– Total # that failed.

11

Viewing Test Results

• Browsing a test’s result screen shows what happened.– Execution summary– Expected and Actual

outputs– Dependency info (when

applicable)• These results are also saved

to disk in XML

Dependency Failure

• Viewing test results when a dependency failure shows what failed.

• Each “parent” test is listed with Expected and Actual status.

• Failed dependencies are highlighted to make them stand out.

13

Helping Fix Bad Tests

• APItest provides debugging assistance for test scripts.– Click “View Test” from the

Main screen for the test listing

– Lines are numbered.– The error is shown via

==ERROR===^• In this case, “&” is invalid

XML. Replace with &amp;

14

Terminal Execution

• APItest can be run at the command line.

• Useful for running a test suite unattended such as through a scheduler.

• Results are saved and can be browsed later via the browser.– A “view-only” mode is

available that only can browse saved results.

15

APItest - Ongoing & TODO

• Bug Fixing• User Guide

– Preliminary R&A given. • DB Connectivity

– Make MySQL connectivity more complete.• Validation via file matching• Configuration file• More SSS tests• Test Developer GUI• Remote test execution• Additional native test types

– HTTP, XMLRPC• Session management and security (Cookies?)

16

APItest - Execution

• There are two ways to execute APItest

• Command Line Only– $> apitest -f test_file.[apt|apb]

• Web browser GUI (Interactive)– $> apitest httpd

• -i [--iroot] : root directory to look for tests• -p [--port] : port number (default = 2112)• -h [--host] : host (default = localhost)

• Running with --help will display command line usage.

18

SSS - Service Directory Startup Test

PREP TEST SDOFF

SSS START TEST SDON

RM EMNG SDSTAT

SSS STOP TEST SDOFF

Dependency Tree for APItest

• Python >= version 2.3– ZopeInterfaces-3.0.1

• Twisted-2.0.0– TwistedWeb-0.5.0

– ElementTree– ssslib (optional)– MySQL (optional)

APItest - Test Types

• CMD– Execute a command via the command line.

• SCRIPT– Execute a script.

• SSS– Execute a ssslib communication.

APItest - Example CMD Test

<testDef>

<info> Example test. Runs ls -lr from /tmp, expects 0 exit status and any input.</info>

<test type="cmd” wdir="/tmp">

<command interpreter="/bin/ksh">ls</command>

<input name="argument" format="literal">-l</input> <input name="argument" format="literal">-r</input> <input name="stdin" format="literal" />

<output name="stdout" format="regexp">.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example Script Test<testDef>

<info>Timeout test</info>

<test name="timestamp1" type="script" timeout="4">

<command interpreter="/bin/bash" wdir="/tmp">for x in 1 2 3 4 5 6 7 8 9 10 do echo "timeout_test :${x}: `date`” sleep 1done </command>

<output name="stdout" format="regexp">.*timeout.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example ENVVAR Test<testDef> <info> This script executes but adds some stuff to the environment. NOTE: setting envvar OVERRIDES any inherited environment variables. </info> <test type="script">

<command interpreter="/bin/csh">#!/bin/csh env</command>

<input name="envvar" format="literal" key="test_envvar_1">foobar</input> <input name="envvar" format="literal" key="test_envvar_2">foo</input>

<output name="stdout" format="regexp"> .*test_envvar_1=foobar\n.*test_envvar_2=foo\n.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example SUID Test

<testDef>

<test type="script">

<command interpreter="python" uname="wcmclen" gid="500">#!/usr/bin/env python import os print os.getuid() print os.getgid() </command>

<input name="argument">/tmp/testuser/</input> <input name="argument">-l</input>

<output name="stdout" format="regexp">^27030$\n^500$</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output> </test></testDef>

APItest - Example SSS Test<testDef>

<info>Sample sss query of the service directory</info>

<test type="sss" destination="service-directory">

<input name="sendbuf">&lt;get-location>&lt;location component='service-directory' host='*' port='*' protocol='*' schema_version='*' tier='*'/>&lt;/get-location></input>

<output name="recvbuf" format="regexp">.*</output>

</test>

</testDef>

Old Graphics

APItest - GUI (Top Page)

• Top level index page for APItest

• APItest recurses from the [testroot] to find tests.

• File Extensions:– TEST : .apt– BATCH : .apb

• Batch tests shown in bold

• Organized by directory

APItest - GUI (Test Run Execution)

• Quick status view of test execution.

• This screen loads when the EXECUTE button is selected.

• Color coded for easy viewing.

• Currently running test is indicated.

• Refreshes every 15 seconds until tests complete.

APItest - GUI (Test Run Summary)

• Summary of test results

• Color coding

• Batch Features– Cumulative summaries

• Total # of tests• # that passed (green)• # that failed (red)

• Hyperlinks to see more details about a test or batch.

APItest - GUI (Test Details - PASSED)

• Example of a test that was executed and passed.

• Summary Data

• Dependency Data– Only shown if test was run

as part of a batch.

• Output– Side-by-side comparison– Dependent on type of test.

APItest - GUI (Batch Summary)

• Clicking a hyperlink into a batch test gives us a summary view.

• In this example, one test failed a dependency– FAILDEP in the status

column indicates this.

• Clicking on the scriptTest2.apt link will show us more data about this test.

APItest - GUI (Test Details - FAILDEP)

• Detailed view of scriptTest2.apt from previous slide.

• Failed a dependency on cmdTest2.apt– Expected : FAIL– Actual : PASS

APItest - GUI (List of Test Runs)

• Clicking the View All Runs button gives this screen.

• Test Runs are sorted by date and time.

• Click links to view details of a particular run.