Business Unit or Product Name © 2007 IBM Corporation Introduction of Autotest Qing Lin

Business Unit or Product Name

© 2007 IBM Corporation

Introduction of Autotest

Qing Lin <[email protected]>


© 2007 IBM Corporation2 Jun 19, 2012

Agenda

Overview of Autotest

Autotest Structure

Autotest Client Harness

Adding tests to Autotest Client

Analyze the test results

Integrate in Jenkins



Overview of Autotest – What is Autotest?

Autotest is a framework for fully automated testing.

It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.

It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.



Overview of Autotest – Goal

The aim is to produce a client which is open source, implicitly handles errors, produces consistent results, is easily installable, simple to maintain and runs either standalone or within any server harness.



Overview of Autotest – General Features

Support for functional, performance, stress tests

Simple inclusion of tests

– Trivial to incorporate existing tests

– Easy to write new tests

– Possible to write very complex tests

Complete control over environment

Consistent, machine-parseable results output

Tests can be executed standalone or as part of a larger-scale system

– Easy to run a prepackaged collection of tests



Autotest Structure

Users-->Autotest Client

Users-->Autoserver-->Autotest Client

Users-->Frontend-->Autoserver-->Autotest Client



Autotest Client Harness

Harness running on the machine under test

Written in Python– APIs to easily compile and run C code

Each test has a simple wrapper class

Standardized results reporting for all tests

Comprehensive error handling

Control flow is managed by a control file



Autotest Client Harness – Anatomy of a test



Autotest Client Harness – Test Class initialize()

- This is run before everything, every time the test is run.

setup()

- This is run when you first use the test, and normally is used to compile the source code.

warmup()

-For performance tests that need to conduct any pre-test priming to make the results valid. This is called by job.run_test before running the test itself, but after all the setup.

run_once()

- This is called by job.run_test N times, where N is controlled by the iterations parameter to run_test (defaulting to one). It also gets called an additional time with profilers turned on, if you have any profilers enabled.

postprocess_iteration()

- This processes any results generated by the test iteration, and writes them out into a keyval. It's generally not called for the profiling iteration, as that may have different performance.

postprocess()

- [DEPRECATED] This is called once to do postprocessing of test iterations, after all iterations are complete. Please use postprocess_iteration instead.

cleanup()

-Used for any post-test cleanup. It always gets called, regardless of the success or failure of the test execution.



Adding tests to Autotest Client – Contents Today

Tests / Testsuites

It only takes about 10 minutes to wrap a test



Adding tests to Autotest Client

An example control file (eg tests/mytest/control)

A test wrapper (eg (tests/mytest/mytest.py)

Some source code for the test (if it's not all done in just the python script)

The name of the subdirectory (tests/mytest), the test wrapper (mytest.py) and the name of the class inside the test wrapper (mytest), all match .



Adding tests to Autotest Client – control file

Path :tests/mytest/control



Adding tests to Autotest Client – test wrapper

Path :tests/mytest/mytest.py



Adding tests to Autotest Client – test call process



Analyze the test result

View the rough result

– ./tools/scan_results.py results/default/status




Result directory

– results/default

Detailed info for each case




Result in HTML



Integrate into Jenkins – Create a Job



Integrate into Jenkins – Configuration I

Name

Workspace

Configuration Page



Integrate into Jenkins – Configuration II Run test script



Integrate into Jenkins – Configuration III Email Notification



Integrate into Jenkins – Get the Result in Log



Thank you

Thanks to Li and Richard for reviewing it.

Documents

Business Unit or Product Name © 2007 IBM Corporation Introduction of Autotest Qing Lin