1

OS II: Dependability & Trust Testing Drivers

Dependable Embedded Systems & SW Group www.deeds.informatik.tu-darmstadt.de

Prof. Neeraj Suri

Constantin Sârbu

Dept. of Computer ScienceTU Darmstadt, Germany

2

Fault Removal: Software Testing So far: Verification & Validation

Testing Techniques Static vs. Dynamic Black-box vs. White-box

Testing of dependable systems with Modeling Fault-injection (FI / SWIFI) Some existing tools for fault injection

Last time: Testing (SWIFI) of operating systems WHERE: Error propagation in OSs [Johansson’05] WHAT: Error selection for testing [Johansson’07] WHEN: Injection trigger selection [Johansson’07]

Today: Profiling the OS extensions (drivers) State definition State changes at runtime Behavior-driven test prioritization

3

Recap: The driver problem

Device drivers Numerous: 250 installed (100

active) drivers in XP/Vista Large & complex: 70% of Linux

code base Immature: every day 25 new / 100

revised versions Vista drivers Access Rights: kernel mode

operation in monolithic OSs

Device drivers are the dominant cause of OS failures despite sustained testing efforts

Causes of WinXP outages

Causes of Win2k outages

4

Recap: The driver problem

Problem statement:Driver failures lead to OS API failures

Mitigation approaches1. Harden OS robustness2. Improve driver reliability

5

Recap: The driver problem

The problem in terms of error propagation

The effect of testing in terms of error propagation

The effect of robustness hardening in terms of error propagation

6

Main topic of today‘s lecture

What if we cannot remove defects (e.g. commercial OSs)?

Goals:

Test case prioritization for black-box components

(Limited) means to remove them: avoid fault activation

7

OS Robustness Testing Efforts at DEEDS

Our research topics presented today:

“Improving Robustness Testing of COTS OS Extensions” (ISAS’06)

• What is the state of an OS (component)?• Efforts to define the state of drivers• How a driver behaves at runtime?

Current research: Test prioritization!• Minimize testing effort based on behavior patterns

Bachelor/Master/Diplom/PhD Theses opportunities!

http://www.deeds.informatik.tu-darmstadt.de/aja/

8

On Robustness Testing of OS Device Drivers

(Preliminaries)

9

Outlook

1. Introduction

2. System model

3. Windows Driver Model (WDM) Structures

4. Robustness Testing Approach

5. Benefits

10

About drivers

• What is a driver? a collection of functions for controlling HW, provided in general by a HW supplierimplemented in general as Dynamic Link Library (DLL, SYS file)

• What does a driver? Imports other libraries Exports its own functions (like public methods) Communicates with HW

Windows drivers !

11

Different Driver Communication Models

DRIVER.DLL

Operating system

Kernelmemory space

X.DLL Y.DLL

12

Robustness testing using SWIFI

Limitations of previous techniques:

- simplified error model (bit-flips, parameter errors)

- huge number of test cases

Need a new technique able to:

• treat drivers as black-boxes;

• reduce testing time;

• increase functional coverage.

13

System & I/O Model

User spaceApplication 1 Application n

Kernel space

Physical Hardware

System Services

Other facilities

I/O Manager

Driver Driver

DriverDriver

Hardware

…

Application n

I/O Manager

System Services

Driver

Physical Hardware

“read file f”

“read(hnd, size, buf)”

“IRP_MJ_READ”

“spin HDD plates, move head h, start read…”

14

WDM (Windows Driver Model) Basics

A framework for device drivers that operate under MS Windows 98/ME/2K/XP and Server 2003

- Designed for forward compatibility across Windows versions

- Feature-set partitioning: a WDM driver must implement some standard routines, the rest is application dependent

- Provide not only protocols, but tools and documentation (Driver Developer Kit - DDK)

15

Driver lifecycle

WDM important entities:

- DRIVER_OBJECT

- IRP (MJ, MN, IOCTL)

Driver

not loaded

OS loads driver code

in virtual memory

OS creates an empty

DRIVER_OBJECT

OS runs DriverEntry function of the driver

Driver

READYStatus

Driver

WORKING

IRP

16

WDM Structures: 1. DRIVER_OBJECT

WDM.H:

typedef struct _DRIVER_OBJECT {

CSHORT Type;

CSHORT Size;

...

} DRIVER_OBJECT, *PDRIVER_OBJECT;

IRP_MJ_READ Pointer to DispatchRead function in the driver

IRP_MJ_WRITE Pointer to DispatchWrite ‘’

IRP_MJ_OPEN Pointer to DispatchOpen ‘’

IRP_MJ_CLOSE Pointer to DispatchClose ‘’

IRP_MJ_DEVICE_CONTROL Pointer to DispatchControl ‘’

not assigned NULL

not assigned NULL

not assigned NULL

not assigned NULL

not assigned NULL

17

WDM Structures: 2. IRP & I/O Stack Location

IRP Structure

I/O Stack location

Status of an operation

IRP dependent

~ 28 major function IRPs:

• 3 have minor IRPs

• 2 have HW specific control codes

18

FSM Idea

Serial.sys:

• 9 MJ

• 2 MN (POWER)

• 37 (DEVICE_CONTROL)

• 4 (INTERNAL_DEVICE_CONTROL)

------------------------------------------

52 “modes”

DEVICE_CONTROL

MN

MN

CREATE CLOSE

READ

WRITE

MN

Driver

not loaded

OS loads driver code

in virtual memory

OS creates an empty

DRIVER_OBJECT

OS runs DriverEntry function of the driver

Driver

READYStatus

Driver

WORKING

IRP

19

Current focus

1. Finding a minimal graph to describe driver functionality

2. Determining the sub-graph(s) with maximum impact on system’s robustness

3. Finding a proper fault model(s): sequences, etc.

4. Is this approach scalable/portable?

20

Improvements over traditional testing techniques

Benefits:

• error model based on actual communication level

• good functionality coverage

• can be used as an aid for fault-injection techniques

• can be used for smart placing of EDMs & ECMs

21

Serial driver example

22

Improving Robustness Testingof COTS OS Extensions

Constantin Sârbu, Andréas Johansson,Falk Fraikin and Neeraj Suri

Department of Computer ScienceTU Darmstadt, GermanyPresented at ISSRE 2006

23

Outline

System Model and MS Windows Driver Model

Driver Mode and Operational Profile

Coverage Metrics for Testing

Case Study: The Serial Driver (Windows XP)

24

Why Driver Testing?

OS extensions (drivers) COTS components enhancing OS’s adaptability collection of subroutines for controlling HW reside in kernel space

Strong OS robustness impact fast developed software high defect density ~70% of Linux kernel code, > 35 000 different drivers for Windows

XP* often unrestricted interference with OS major OS failures cause (~85% of SW related failures in Windows

XP*)

Test the drivers better!* Improving the Reliability of Commodity Operating Systems, M. M. Swift et al., SOSP 2003

25

Driver Testing

Testing of Drivers (developers and users): crucial but difficult limited access: located in kernel space driver users have limited access to source code set of loaded drivers is different across installations

Common Driver Testing Philosophies: Microsoft approach: Driver Reliability Signature (DRS) program

• DDK (Driver Developer Kit) and HCT (Hardware Compatibility Test)• based on “fault checklists”• tests available to driver developers

SWIFI (SW Implemented Fault Injection)• inject artificial faults and observe outcome• reboot system to inject into the same “state”• not considering driver’s operational state

Multiple at varied design/abstraction levels … (functional, behavioral…)

How good is a testing method?

26

State Space and Operational Profile

We need testing methods matching the operational profile

Driver’s State Space

Faults (tend to cluster*)

Operational Profile

(dependent on workload)

* An Empirical Investigation of Software Fault Distribution, K. H. Möller and D. Paulish, SMS 1993

Microsoft Tests

(fault checklists)

SWIFI

27

System Model

(Currently) MS Windows XP SP2 as case study

The set of applications is known

Application 1 Application p

System Services

Other OS

Facilities

I/O

Manager

Hardware Layer

Driver 1Driver 1 Driver 2Driver 2

Driver 3Driver 3 Driver mDriver m

…USER

SPACE

KERNEL

SPACE

HW SPACE

Drivers interact with the rest of the system via I/O Manager

Windows Driver Model (WDM) specifies the communication interface between I/O Manager and the drivers

28

Windows Driver Model (WDM) Unified interface between OS kernel and drivers

I/O Request Packet (IRP) communication media between I/O Manager and drivers I/O Manager builds IRP request and pass it to a driver driver executes associated code and returns the result using the

same IRP instance

Each driver contains a set of procedures, each one executed

when a particular request was received publishes a list with entry points to the

respective procedures

A driver can execute several IRP requestsconcurrently

DRIVER

I/ORequestHandler

I/O Request(IRP)

Result &Status

29

Driver Mode At time t, the mode of a driver is a tuple of predicates, each

assigned to one of the n IRPs the driver supports:

MD: < P(IRP1) P(IRP2) P(IRP3) … P(IRPn)

>1, if performing the functionality triggered by IRPi{P(IRPi) =0, otherwise

DRIVER

I/ORequestHandler

IRP2

Result &Status

IRP1 IRP2 IRP3 IRP4

MD: < 0 0 0 0 >

Example: a driver supporting 4 distinct IRPs:

1

30

Driver’s State Space The driver’s state space is represented by the set of all possible driver modes The operational profile is defined by the set of visited modes

Assumption:Assumption: at any instant of time, only one IRP can be received or finished by the driver

1000 0100 0010 0001

1100 1010 1001 0011 0110 0101

1110 1101 1011 0111

1111

0000

# ofActiveIRPs

0

1

2

3

4

1000 0100 0010 0001

1100 0011

0000

bidirectionaledges

1000 0100

1110 1101

1100

Total number of modes:

N = 2n

Total number of transitions:

T = n·N = n·2n

31

Testing Coverage Metrics

Ideal testing technique should test 100% of the operational profile!

1. 1. Mode CoverageMode Coverage:: every visited mode is tested

MC = |tested modes ∩ visited| / # of op. profile modes

2. 2. Transition CoverageTransition Coverage:: for every visited mode, all outgoing traversed transitions are tested

TC = |tested transitions ∩ traversed | / # op. profile transitions3. 3. Path CoveragePath Coverage:: traverse all the paths between two visited modes, over any

number of hops

1110 1101 1011 0111

1111

1010 1001 0110 0101

1000 0100 0010 0001

0000

1100 0011

32

Case Study: The Serial Driver (Windows XP)

Experimental setups: Pentium4 @2.8Ghz serial modem (external) cables (serial, loopback) various benchmark

software

Workload App.

I/O Manager

KERNEL SPACE

HW SPACE

Serial Driver

Serial Port

USER SPACE

IrpTracker

Communication Party: - 56k Modem - 2nd computer - loopback cable

Logs

Serial driver: serial.sys, provided together with Windows XP Professional

SP2 digitally signed by Microsoft passed the reliability and stress tests included in HCT (Hardware

Compatibility Tests) and DDK (Driver Development Kit)

How much of the mode graph is actually visited?

33

Experiment 1 – Driver-Usage Pattern Used a commercial modem benchmark as workload

get / set serial port settings send / receive data traffic is verified for completeness and correctness

Assumed that: the generated load is representative for normal operational mode of the driver the sequence of IRPs is repeatable

Small operational profile: only 7% of modes and 1.8% of transitions were visited -> consistent!

34

Experiment 2 – Aggregated Workload Workload: a set of 7 applications that generated a total of:

107456 requests (10 distinct) a total of 1024 modes / 10240 transitions

Operational profile: only 1.66% of modes and 0.34% of transitions were visited -> indicate where to focus testing

Observations:

some modes are visited much more frequently than others

only modes located on first levels are visited (11 levels)

existence of loops!

35

Discussion Operational profile

only a very small amount of modes are actually visited under a given workload

it indicates the modes and transitions with high likelihood to be reached in the field test those preferentially!

not many IRPs were executed concurrently short IRP sequence to bring the driver in the desired mode

IRP sequences generating those can be problematic (receipt and return occur non-

deterministically) Wave Testing: first test visited modes, then their one-hop

neighbors by trying to traverse new edges

Limitations cannot deal with parallel processing of several IRPs of the same

type assumes sequential start/finish of IRPs (no jump over one level)

36

Conclusions & Future Work Our contribution provides “means” to identify relevant

locations for focused/effective testing (& for black-box SW!)

Requires no modifications of the OS or driver source code

Assist the debugging process (we have information about which subroutine is running at a given moment)

Future work Representative set/classes of drivers, OSs Build operational graphs complementing MS testing tools (is

Microsoft testing enough?) Application profiling (build behavioral patterns for driver usage)