Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Practical Xen Testing at Intel
Nov. 2009
Haicheng Li <[email protected]>
Jiajun Xu <[email protected]>
Xen Summit Asia 2009
2
Software and Services Group
2
Legal Information
• INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
• Intel may make changes to specifications, product descriptions, and plans at any time, without notice.
• All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.
• Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.
• Intel is a trademark of Intel Corporation in the U.S. and other countries.
• *Other names and brands may be claimed as the property of others.
• Copyright © 2009, Intel Corporation. All rights are protected.
3
Software and Services Group
3
Agenda
• The Overview – how test helps Xen?
• Test Infrastructure – the world under automation
• Best Practices – practical test approaches
• Some Proposals – to improve Xen quality
4
Software and Services Group
4
The Overview
5
Software and Services Group
5
Test
Deliverables
Test
Coverage
Intel VT
XEN Development
Oct. 2006
Xen-3.0.3
May. 2007
Xen-3.1.0
Jan. 2008
Xen-3.2.0
Aug. 2008
Xen-3.3.0
Apr. 2009
Xen-3.4.0
Dec. 2005
Xen-3.0.0
3.0.x Release
Test Reports
3.1.x Release
Test Reports
3.2.x Release
Test Reports
3.3.x Release
Test Reports
3.4.x Release
Test Reports
Nightly Test Reports Weekly Test Reports Biweekly Test Reports
Actively Bug Tracking
SMP VM
HVM x86/IA64
Live Migration
Power Management
Device Pass-Through
NUMA RAS
SR-IOV
PV Guest Timers
TXT Real Mode
VT-x
VT- i VT-d
SR-IOV
EPT
XEN
Progressive Xen Testing
• Hundreds of Xen patches Monthly !!!
• How can TEST help?
• Expose regression timely
• Ensure release quality
6
Software and Services Group
6
Test
Deliverables
Test
Coverage
Intel VT
XEN Development
Oct. 2006
Xen-3.0.3
May. 2007
Xen-3.1.0
Jan. 2008
Xen-3.2.0
Aug. 2008
Xen-3.3.0
Apr. 2009
Xen-3.4.0
Dec. 2005
Xen-3.0.0
3.0.x Release
Test Reports
3.1.x Release
Test Reports
3.2.x Release
Test Reports
3.3.x Release
Test Reports
3.4.x Release
Test Reports
Nightly Test Reports Weekly Test Reports Biweekly Test Reports
Actively Bug Tracking
SMP VM
HVM x86/IA64
Live Migration
Power Management
Device Pass-Through
NUMA RAS
SR-IOV
PV Guest Timers
TXT Real Mode
VT-x
VT- i VT-d
SR-IOV
EPT
XEN
Progressive Xen Testing
~300 major bugs reported,
20% of Xen Bugzilla
7
Software and Services Group
7
Test Infrastructure
8
Software and Services Group
8
Solid Test Infrastructure
• Test Automation in Everywhere
– auto daily testing
– auto release resting
– auto patch testing
• Test Infrastructure is The Basis
• Extendable Test Framework
• Auto Build System
• Test Result Center
• Patch Test System
9
Software and Services Group
9
Extendable Test Framework
Harness
Management Module
Report Generator
Test Suite Pool
VT-d
SR-IOV
…
Power
Call with Scenario file
Common lib
Controller
VT-d/
| -- VT-d lib
| -- test1
| -- test2
SR-IOV/
| -- IOV lib
| -- test1
| -- test2RAS
Regular TestingAuto Build Result submitter to Result Center*XVS
Manual Testing
*XVS: eXtendable Validation System
Test Harness
10
Software and Services Group
10
Test Result Center
Find Defects Out
Timely!
11
Software and Services Group
11
Patch Test System
Web UI
Runtime DB
Recipe
R-Queue
OSD
Machines
Test Info
Users
Results
Machine Status UIRecipe Submit UI Result UI
Test Info
Test Machine Pool
Test Box 1
Test Box 2
Test Box N
…
DB Tool
Receive Recipe
Scheduler
Core ManagementTest Schedule
Results Processing
Local ControlMachine
Test Resources
XVS
12
Software and Services Group
12
Patch Test System
Ensure Patch Quality Before
Submitting Out!
13
Software and Services Group
13
Best Practices
14
Software and Services Group
14
Power Management Testing
• Challenge
– How to achieve integrated Power Management testing?
• Test Approach
– Automated functionality testing.
• Interface : xenpm, hypercall, sysfs interface
• Coverage : idle, runtime, Dom0/HVM S3
– Performance measurement.
• Scenario : idle, runtime, benchmarks
• Platform : laptop & server
• OS : Linux, Windows
• Publish : web site
15
Software and Services Group
15
Power Management Testing
• Challenge
– How to achieve integrated power testing?
• Test Approach
– Automated functionality testing.
• Interface : xenpm, hypercall, sys i/f
• Coverage : idle, runtime, Dom0/HVM S3
– Performance measurement.
• Scenario : idle, runtime, benchmarks
• Platform : laptop & server
• OS : Linux, Windows
• Publish : web site
16
Software and Services Group
16
RAS Testing
• Challenge
– How to achieve good test coverage?
• Test Approach
– Add test stub to Xen to dump page types.
– Add test stub to Xen to translate address from guest to host.
– Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages.
• To Do
– How to stress?
• Option: add more software test interface to *MCE handlers.
– CPU Migration
– MEM hot-plug*MCE: Machine Check Exception
17
Software and Services Group
17
RAS Testing
• Challenge
– How to achieve good test coverage?
• Test Approach
– Add test stub to Xen to dump page types.
– Add test stub to Xen to translate address from guest to host.
– Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages.
• To Do
– How to stress?
• Option: add more software test interface to *MCE handlers.
– CPU Migration
– MEM hot-plug*MCE: Machine Check Error
What MCE test i/f Linux provides?• MCE injector to trigger fake MCE
• madvise syscall to poison page
• HWPosion injector to poison page
• … …
18
Software and Services Group
18
VT-d/SR-IOV Testing
• Challenge
– Hardware variety & complexity : platforms + devices
• Test Approach
– Select typical test configurations.
• Platform : laptop -> desktop -> server
• Device : NIC, USB, SATA
• PCI type : PCIE MSI, PCIE INTx, PCI INTx
• OS : Linux, Windows
– Use NIC to achieve test automation.
• Methods : ping, scp (file copy), ssh
• Coverage : static assignment + VT-d hotplug
• Multiple-devices assignment.
19
Software and Services Group
19
Some Proposals
20
Software and Services Group
20
Improve Xen Quality
• Source Control
– Synchronize source control for Ioemu, Dom0 and Xen.
• E.g. VT-d is usually broken by ioemu check-in
• Bug Tracking
– Well track bug state in Xen Bugzilla
• Current Xen BZ full of obsolete bug data
• ~500 bugs , 30% Xen Bugzilla, are NEW!
• Xen Testability
– More built-in test features for Xen?
• It’s a trend for Linux: memleak checker, fault injector, ...
21
Software and Services Group
21
Questions?
22
Software and Services Group
22
Backup
23
Software and Services Group
23
Regular Testing
To keep Xen from regressions
*BAT Testing
Auto Install Regression Testing Bug Verification
Daily Build
C/S Pull Auto Build
Basic Functional TestsBooting TestsBuild Check
System Testing
Performance Testing Stress TestingFunctionality Testing
Full Scope Functional Tests
Others
Daily *BAT Test Cycle
Monthly System Test Cycle
*BAT: Build Acceptance Test
24
Software and Services Group
24
Release Testing
To make qualified Xen release
*BAT Testing
Auto Install Regression Testing Bug Verification
Release Build
C/S Pull Auto Build
Basic Functional TestsBooting TestsBuild Check
System Testing
Performance Testing Stress TestingFunctionality Testing
Full Scope Functional Tests
Others
Release *BAT Test Cycle
Release System Test Cycle
*BAT: Build Acceptance Test
25
Software and Services Group
25
Patch Testing
To ensure patch quality for Xen
Patch Testing
Auto Install Regression Testing
Unit Testing
Auto Build
Basic Functional TestsBooting TestsBuild Check
Patch Test Cycle