CHESS: A Systematic Testing Tool for Concurrent Software

CSCI6900George

Outline Introduction

Using CHESS

Capturing nondeterminism

Exploring nondeterminism

Evaluation

Related work

Introduction Concurrent programming is difficult

Testing concurrent program is difficult Famous concurrency bug: “Heisenbugs”

Most strategy used in finding concurrency bugs Stress testing

Introduction What is “Concurrency testing”

Concurrency scenario testing

Model checking techniques

Three key challengers by applying model checking

Perturbation problem Rich and complex concurrency APIs in system State-space explosion.

Introduction What is “CHESS”

Concurrency scenario testing

Model checking techniques

Scale to large concurrent program

Reproduce an erroneous execution

Been integrated into the test framework inside Microsoft

Introduction How does CHESS solve the previous key

problems?

Testing methodology of CHESS reduce the perturbation

Only one perturbation in CHESS

Wrappers to provides enough hooks

Variety of techniques to address the state-explosion

Contributions of this paper The first system for integrating model

checking into concurrent systems and test frameworks with minimal perturbation;

Techniques for systematic exploration of systems code for fine-grained concurrency with shared memory and multithreading;

Contributions of this paper Validation of the CHESS tool and its

accompany testing and wrapper methodology on three different platforms;

A substantial number of previously unknown bugs, even in well-tested systems;

The ability to consistently reproduce crashing

bugs with unknown cause.

Using CHESS Traditional testing methodology

Disadvantages of this testing methodology

Using CHESS CHESS architecture

Using CHESS Testing methodology in CHESS

Using CHESS To guarantee the previous advantages, CHESS

has to control the scheduling of the tasks

Two ways to control it

Modify the scheduler

Severely limited the deployment

Using CHESS Testing methodology in CHESS

Using CHESS Two assumption of about the testing program

Quiescence Definition of “Quiescence State”

How does CHESS handle “Quiescence State”

Capturing nondeterminism Tasks in CHESS

Threads Timers Asynchronous I/Os

Wrappers in CHESS Goal of wrappers

Capturing nondeterminism Design of Wrappers in CHESS

Capturing nondeterminism Synchronization wrappers

Capturing nondeterminism Synchronization wrappers

Capturing nondeterminism Hooking the wrappers

use various mechanisms to dynamically intercept calls to the real API functions and forward them to the wrappers.

Programs in this paper Win32 .NET Singularity

Exploring nondeterminism How CHESS systematically drives the test

Basic scheduler operation Allows only one thread to execute at a time

Repeatedly executes the same test driving each iteration of the test through a different schedule

Three phases in each iteration

Exploring nondeterminism Replay phase

Replays a sequence of scheduling choices from a trace file

Record phase Schedules a thread till the thread yields the

processor

Search phase Uses the enabled information at each schedule

point to determine the scheduler for the next iteration

Exploring nondeterminism Imperfect replay

Not rely on perfect replay capability

Common sources of nondeterminism Lazy-initialization

Interference from environment

Nondeterministic calls

Exploring nondeterminism Ensuring fair schedules

Can not enumerate all fair schedules

State-explosion Definition of State-explosion

Inserting preemptions

Capturing states

Evaluation Systems on which CHESS has been run on

Evaluation Test scenarios and findings

Evaluation Validation CHESS against stress-testing

Common objection of the CHESS

Failure and bug in CHESS

Succeeded in reproducing every stress-test failure

Evaluation Description of two bugs

PLINQ bug

Evaluation Description of two bugs

Singularity bug

Related work Repeatable deterministic testing

Systematic generation of thread schedules

Applying state exploration directly to executing concurrent programs

Replay a concurrent execution