Practical Xen Testing At Intel

Practical Xen Testing at Intel

Nov. 2009

Haicheng Li <[email protected]>

Jiajun Xu <[email protected]>

Xen Summit Asia 2009

2

Software and Services Group

2

Legal Information

• INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

• Intel may make changes to specifications, product descriptions, and plans at any time, without notice.

• All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.

• Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.

• Intel is a trademark of Intel Corporation in the U.S. and other countries.

• *Other names and brands may be claimed as the property of others.

• Copyright © 2009, Intel Corporation. All rights are protected.

3


3

Agenda

• The Overview – how test helps Xen?

• Test Infrastructure – the world under automation

• Best Practices – practical test approaches

• Some Proposals – to improve Xen quality

4


4

The Overview

5


5

Test

Deliverables

Test

Coverage

Intel VT

XEN Development

Oct. 2006

Xen-3.0.3

May. 2007

Xen-3.1.0

Jan. 2008

Xen-3.2.0

Aug. 2008

Xen-3.3.0

Apr. 2009

Xen-3.4.0

Dec. 2005

Xen-3.0.0

3.0.x Release

Test Reports

3.1.x Release

Test Reports

3.2.x Release

Test Reports

3.3.x Release

Test Reports

3.4.x Release

Test Reports

Nightly Test Reports Weekly Test Reports Biweekly Test Reports

Actively Bug Tracking

SMP VM

HVM x86/IA64

Live Migration

Power Management

Device Pass-Through

NUMA RAS

SR-IOV

PV Guest Timers

TXT Real Mode

VT-x

VT- i VT-d

SR-IOV

EPT

XEN

Progressive Xen Testing

• Hundreds of Xen patches Monthly !!!

• How can TEST help?

• Expose regression timely

• Ensure release quality

6


6

Test

Deliverables

Test

Coverage

Intel VT

XEN Development

Oct. 2006

Xen-3.0.3

May. 2007

Xen-3.1.0

Jan. 2008

Xen-3.2.0

Aug. 2008

Xen-3.3.0

Apr. 2009

Xen-3.4.0

Dec. 2005

Xen-3.0.0

3.0.x Release

Test Reports

3.1.x Release

Test Reports

3.2.x Release

Test Reports

3.3.x Release

Test Reports

3.4.x Release

Test Reports

Nightly Test Reports Weekly Test Reports Biweekly Test Reports

Actively Bug Tracking

SMP VM

HVM x86/IA64

Live Migration

Power Management

Device Pass-Through

NUMA RAS

SR-IOV

PV Guest Timers

TXT Real Mode

VT-x

VT- i VT-d

SR-IOV

EPT

XEN

Progressive Xen Testing

~300 major bugs reported,

20% of Xen Bugzilla

7


7

Test Infrastructure

8


8

Solid Test Infrastructure

• Test Automation in Everywhere

– auto daily testing

– auto release resting

– auto patch testing

• Test Infrastructure is The Basis

• Extendable Test Framework

• Auto Build System

• Test Result Center

• Patch Test System

9


9

Extendable Test Framework

Harness

Management Module

Report Generator

Test Suite Pool

VT-d

SR-IOV

…

Power

Call with Scenario file

Common lib

Controller

VT-d/

| -- VT-d lib

| -- test1

| -- test2

SR-IOV/

| -- IOV lib

| -- test1

| -- test2RAS

Regular TestingAuto Build Result submitter to Result Center*XVS

Manual Testing

*XVS: eXtendable Validation System

Test Harness

10


10

Test Result Center

Find Defects Out

Timely!

11


11

Patch Test System

Web UI

Runtime DB

Recipe

R-Queue

OSD

Machines

Test Info

Users

Results

Machine Status UIRecipe Submit UI Result UI

Test Info

Test Machine Pool

Test Box 1

Test Box 2

Test Box N

…

DB Tool

Receive Recipe

Scheduler

Core ManagementTest Schedule

Results Processing

Local ControlMachine

Test Resources

XVS

12


12

Patch Test System

Ensure Patch Quality Before

Submitting Out!

13


13

Best Practices

14


14

Power Management Testing

• Challenge

– How to achieve integrated Power Management testing?

• Test Approach

– Automated functionality testing.

• Interface : xenpm, hypercall, sysfs interface

• Coverage : idle, runtime, Dom0/HVM S3

– Performance measurement.

• Scenario : idle, runtime, benchmarks

• Platform : laptop & server

• OS : Linux, Windows

• Publish : web site

15


15

Power Management Testing

• Challenge

– How to achieve integrated power testing?

• Test Approach

– Automated functionality testing.

• Interface : xenpm, hypercall, sys i/f

• Coverage : idle, runtime, Dom0/HVM S3

– Performance measurement.

• Scenario : idle, runtime, benchmarks

• Platform : laptop & server


• Publish : web site

16


16

RAS Testing

• Challenge

– How to achieve good test coverage?

• Test Approach

– Add test stub to Xen to dump page types.

– Add test stub to Xen to translate address from guest to host.

– Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages.

• To Do

– How to stress?

• Option: add more software test interface to *MCE handlers.

– CPU Migration

– MEM hot-plug*MCE: Machine Check Exception

17


17

RAS Testing

• Challenge

– How to achieve good test coverage?

• Test Approach

– Add test stub to Xen to dump page types.

– Add test stub to Xen to translate address from guest to host.

– Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages.

• To Do

– How to stress?

• Option: add more software test interface to *MCE handlers.

– CPU Migration

– MEM hot-plug*MCE: Machine Check Error

What MCE test i/f Linux provides?• MCE injector to trigger fake MCE

• madvise syscall to poison page

• HWPosion injector to poison page

• … …

18


18

VT-d/SR-IOV Testing

• Challenge

– Hardware variety & complexity : platforms + devices

• Test Approach

– Select typical test configurations.

• Platform : laptop -> desktop -> server

• Device : NIC, USB, SATA

• PCI type : PCIE MSI, PCIE INTx, PCI INTx


– Use NIC to achieve test automation.

• Methods : ping, scp (file copy), ssh

• Coverage : static assignment + VT-d hotplug

• Multiple-devices assignment.

19


19

Some Proposals

20


20

Improve Xen Quality

• Source Control

– Synchronize source control for Ioemu, Dom0 and Xen.

• E.g. VT-d is usually broken by ioemu check-in

• Bug Tracking

– Well track bug state in Xen Bugzilla

• Current Xen BZ full of obsolete bug data

• ~500 bugs , 30% Xen Bugzilla, are NEW!

• Xen Testability

– More built-in test features for Xen?

• It’s a trend for Linux: memleak checker, fault injector, ...

21


21

Questions?

22


22

Backup

23


23

Regular Testing

To keep Xen from regressions

*BAT Testing

Auto Install Regression Testing Bug Verification

Daily Build

C/S Pull Auto Build

Basic Functional TestsBooting TestsBuild Check

System Testing

Performance Testing Stress TestingFunctionality Testing

Full Scope Functional Tests

Others

Daily *BAT Test Cycle

Monthly System Test Cycle

*BAT: Build Acceptance Test

24


24

Release Testing

To make qualified Xen release

*BAT Testing

Auto Install Regression Testing Bug Verification

Release Build

C/S Pull Auto Build


System Testing

Performance Testing Stress TestingFunctionality Testing

Full Scope Functional Tests

Others

Release *BAT Test Cycle

Release System Test Cycle

*BAT: Build Acceptance Test

25


25

Patch Testing

To ensure patch quality for Xen

Patch Testing

Auto Install Regression Testing

Unit Testing

Auto Build


Patch Test Cycle

Technology

Practical Xen Testing At Intel