Applying checkpoint encoding to state-base software testing

CS635 Project Report

Dinh Trong Thanh Trung

653-14-0761

Abstract

Checkpoint encoding can be applied to black box testing in order to minimize the test cases and maximize the test efficiency. Although the effectiveness of the approach depends mainly on the encoding scheme, this problem is never thoroughly addressed. Since encoding schemes should depend on the problem domains, in this paper we proposed an effectiveness scheme to encode the input of problems that specified by state machines.

1. Introduction

The purpose of testing is to surface failures and thereby detects faults. Software testing techniques are usually categorized into black-box and white-box testing. Black-box testing schemes generate test cases based on the specifications of the system under test only, without any knowledge about the implementation. In order to enhance the effectiveness of black-box testing, Malayia et al. [1, 2] proposed an automatic test generation scheme with checkpoint encoding technique. The main idea of this approach is to divide the input space of a system-under-test into different domains. The system under test is assumed to behave the same when it receives input from the same domain. Each domain is then encoded using a binary code pattern. Finally, each binary code represents a different domain in the input space and can be used to automatically generate test cases. Existing work in anti-random testing focuses on effectively generating test cases, assuming that the set of the binary code representing the input spaces is given [3, 4]. However, an automatic scheme to encode the input space is never addressed.

Checkpoints for each system under test must be defined (and encoded) based on the specific characteristic of the specification. We believe that there is no generic scheme that can be applied for every domain of problems. In this paper, we propose an effective encoding scheme that can be applied to a class of problems that are specified using finite state machines.

The rest of the paper is organized as follow: the next section discusses the background knowledge of our work including rationale of checkpoint encoding schemes and different coverage levels of state based testing. Section three describes our state based checkpoint encoding scheme. The fourth section presents the result from an experiment to illustrate the effectiveness of the encoding scheme. Section five concludes and discusses the future work.

2. Background

a. Checkpoint encoding

The checkpoint-encoding scheme, described first by Malayia [1], allows testers to enhance the test effectiveness with anti-random technique. The rationale of anti-random technique is that test inputs are chosen in a deterministic order so that any test in the sequence always has a maximum distance from all previous test cases. The distance between two test inputs is a metric value to measure the different between the natures of the tests. For example, based on the system specification, one can divide the inputs of a system into different domains, such that the behavior of the system is somewhat similar within a domain. As a consequence, the distance between two test cases chosen from the same domain will be smaller then the distance between two test cases from different domains.

In order to calculate the distances between test cases, the anti-random technique [1, 2] encodes each test case using some binary bit patterns. The distance between two tests will be the Hamming or the Cartesian distance between the two binary patterns represent the two tests. According to Malayia et al., the Cartesian distance is a better measurement than the Hamming distance.

Malayia et al. [1, 2] described an encoding scheme based on finding the boundaries and partitioning the input space of the systems [10, 11, and 12].

To illustrate this technique, let us consider an example as given in figure 2.1: a system takes two inputs X and Y, such that the range of X is between 0 and 10, and Y must be greater than 50.

Figure 2-1 An example of partitioning and finding boundary value for black box testing

Figure 2.1 shows different test cases that should be covered (based on partitioning and boundary techniques): black dots represent “partitioning” test cases, and white dots represent “boundary” test cases. The input domains can be encoded as follow:

- For the input X:

o 000: X < 0 (invalid input).

o 001: X = 0 (boundary value)

o 010: X = 10 (boundary value)

o 011: X >10 (invalid input)

o 1XX: 0 < X < 10 (valid input)

- For input Y:

o 00: Y < 50 (invalid input)

o 01: Y = 50 (boundary value)

o 1X: Y > 50 (valid input).

As the result, A(xA, yA), B(xB, yB) and C(xC, yC) are represent by ‘00011’, ‘10000’ and ‘10011’, respectively. The anti-random technique suggests that if we choose to test the point A first, then in the next test, choosing B will be better than choosing C (since C and A share the same characteristic of Y: the value of Y is chosen from the valid input domain. In contrast, A and B are really different: the values of Y is valid in A but invalid in B, also the value of X is valid invalid in A and valid in B).

Using the encoding scheme we can see that the (Cartesian) distance between A and C is 1, while the distance between A and B is 3, therefore, it also suggests that if A is tested first, then B should be the next one.

The empirical research [2] on the filed shows that anti-random testing with checkpoint encoding can result in better code coverage than random testing.

b. State based testing

Recent work in testing state-based software system (such as [6, 7, 8, 14, 15 and 16]) usually focuses on testing state-charts (for example, UML state-chart [13]) and extended finite state machines (EFSM). However, according to Kim et al. [6] and Hong et al. [7], most of the state charts can be transformed into an equivalent EFSM. Therefore, although we describe our work in the terms of EFSM only, we believe that our scheme will also be applicable to state-chart.

State based testing usually suffers from the following faults [5]:- Incorrect transition: the occurrence of a transition may change the system to an

unexpected state.

- Incorrect event: The system does not react when a valid event occurs.

- Incorrect action: The system executes an unexpected action as a result of a transition.

- Incorrect state: The behavior of the system in a state is unpredictable. This can be modeled as a combination of incorrect events, transition and action.

- Illegal event failure: The system fails because of an unexpected event.

- Trap door: The system accepts undefined events.

In our project, we focus more on the coverage aspect of state-based testing. In other word, we assume that for a given a test input, how to test the system (for example, how to develop the oracle function, how to observe the output and the state of the system after a test…) is well understood. Common coverage models for state machines can be categorized into control flow based and data flow based schemes:

- Control flow based coverage models [6, 7, 8] are usually defined based on EFSM. We have the following coverage levels:

o State coverage: every state is covered.

o Transition coverage: every transitions is covered

o Transition pair coverage: every pair of adjacent transitions is covered.

o Full predicate coverage: Every clause in each predicate on each transition is covered.

For more detail, please see Offutt et al.’s [8] work.

o Full path coverage: Every path is covered. We can easily see that if a state machine has the form of a cyclic graph, the number of distinguish paths are infinite. Therefore this coverage level is impractical.

- Data flow based coverage models [6, 7] require that the EFSM under test is transformed into a data-flow graph. The definition and usage of each variable can be determined in each transition. Based on the data-flow graph, the following coverage levels can be achieved:

o All use coverage (c-use, p-use or both).

o All-def-use coverage.

Our solution will not base on the data flow based coverage, so we will not discuss it in any more detail.

3. State based checkpoint encoding

a. The rational and the assumptions of the approach

Our checkpoint encoding approach bases on the fact that some faults in a state-based system can occur only when a certain path of transitions is taken (for example, in figure 3.1, the system may fail when a path T1-T3-T2 is executed, although a set of other test cases that executing T1, T2, T3 in some other orders may not reveal the fault). Therefore, in our test-generation scheme, each test case will test a path of transitions (such a test case will also cover every transition in the path).

We also assume that the more common a pair of paths, the higher probability that they will either both pass, or both fail under test. For example, if a system pass the path P1 =

(T1-T3-T2-T7) it is more likely for it to fail when executing P2=(T1-T4-T5-T6) than P3=(T1-T3-T2-T6). Therefore, if we need to test only those three paths, the anti-random testing scheme suggests that if we test P1 first, we should test P3 after P2.

Figure 3-0-1 A state-based specification

Also, we assume that there is no consume state in the state diagram under test. In other words, every state has at least one outgoing transition. We ague that this assumption is reasonable, because state diagrams normally have only a few consume states, which can temporary moved out and tested separately.

b. The check-point encoding scheme:

The basic encoding scheme can be described as follow:

- For each state: Assume that a state has n outgoing transitions, denoted t0, t1… tn-1, we use log2 (n+1) bits to encode the possible actions of this state: the first n bit pattern represents t0, t1…tn-1, respectively and the (n+1)th pattern represents the unexpected event (when such an event occurs, the system does not change its current state).

For example, the state S2 (in figure 3-2) has two outgoing transitions T1 and T2. We will use 2 bits to encode them: 00 for T1, 01 for T2 and 10 for any unexpected event. The state S4, on the other hand, has only 1 outgoing transition, so we will use 1 bit to encode: 0 for T5 and 1 for unexpected events.

- Each path will be represented using a bit pattern, which is formed by combining the bit patterns representing the sequence of transitions in the path.

For example, in figure 3-2, ‘00 01 0 00 10’ represents T1-T4-T5-T7-NULL (we use ‘NULL’ to denote the case when an unexpected event occur).

Figure 3-0-2 Encoding the transitions of a state diagram

- In our test generating approach, we will use the same bit-length to represent every test case. Given a non-zero integer l (the length of the bit patterns), every l-bit pattern will be considered as a potential test case, and the corresponding path will be found as follow:

o Each bit pattern will be parsed from the left-most bit to the right-most bit: if the outgoing transitions of the beginning state are encoded using m1 bits, the first m1 bit will be extracted from the pattern. If an outgoing transition ti of the beginning state match the extracted m1 bit, it will be added to the corresponding path. The next state in the path is the target of ti, and if its outgoing transitions are encoded using m2 bits, the next m2 bits will be extract from the pattern. Based on that m2 bit, the next transition in the path is selected. The process is continued until the last bit in the original pattern is extracted. For example, 00 01 0 will be transform to T1-T4-T5.

o If the last portion of a bit-pattern is too short, some minimum arbitrary bits will be added so that we can select the last transition. For example, consider a bit patter ‘00 00 0’: the first 4 bit are transformed into T1-T3, but the last portion (‘0’) is too short. In that case we can add a ‘0’ to the end of that pattern so that the selected path is T1-T3-T1.

o Some of the l-bit pattern still cannot be transformed into any path, and we call them invalid patterns. For example, in figure 3-2, ‘00 11 00’ is an invalid pattern.

c. Generating test using anti-random scheme:

With the encoding approach as described above, we can apply the anti-random technique to generate the test for a state-based system, given any non-zero integer l as the bit length. However, we need to decide what to do with the invalid patterns. In this work, we proposed three different solutions:

Solution 1: Generate the anti-random sequence that contains every l-bit pattern, and then eliminate all unused patterns.

The remaining sequence of l-bit pattern will be the test. For example, if we have l=3, and the invalid patterns are 101, 110 and 111.

- The anti-random sequence that contains all 3-bit patterns is {000, 111, 100, 011, 001, 110, 010, 101}.

- The final sequence is: {000, 100, 011, 001, 010}.

Solution 2: Eliminate all invalid patterns, then generate anti-random pattern. Using the same example as above, the final sequence is: {000, 011, 001, 010, 100}

Solution 3: Assign some random test cases for the invalid patterns.

We can see that the second solution gives us a test with the best anti-randomness. However, the computation overhead (to generate the real anti-random pattern) is greater than the first and the third solutions.

d. The minimum bit-length for transitions coverage

Until now, our test generation scheme allows one to generate an anti-random test for a given state-based specification and a given bit-length. However, it is desirable to require that the test must at least satisfy the transition coverage. We proposed here a technique that can calculate the minimum bit-length needed to generate a test that covers all transitions:

- First of all, we generate the decision tree [9] from the given state diagram.

- We define the length of a path in that tree as the bit-length to represent that tree.

- The minimum bit-length needed to generate a test that satisfies transition-coverage is the longest path in the tree.

Figure 3-2 A decision tree.

For example, the decision tree for the example in figure 3-2 is given in figure 3-3. We can easily see that the longest path is S1-S2-S4-S3, and its length is 5 (the pattern is

‘00010’). Therefore, an anti-random test generated using our scheme with 5 bit length will cover every transition.

We can also see that if we expanding the decision by one more level of transition (by adding to each leave of the tree all of the corresponding outgoing transitions), the longest path of the tree can be used to generate a test that satisfy the transition-pair coverage.

4. Empirical validation

In order to experimentally validate the efficiency of the proposed encoding scheme, we apply the method as described in the previous section to two state-based systems. For each of the two experiments, we generate one test set using anti-random checkpoint encoding and another test set using random testing technique. For each of the test set, we measure the code coverage (here we use decision coverage) per number of test cases. The tool to collect the code coverage data is XSuds. We expected that the code coverage per number of anti-random testing should be higher than that of random testing.

a. Experiment 1

Subject: The subject of this test is an implementation of a stack with a limited volume. The state-based specification of the system is given is figure 4.1. All of the events are change events. The stack is implemented as a class in C++. The side of the program is 120 lines of code.

Figure 0-3.1 The specification of a stack.

Data collected:

- Anti-random testing: The transitions of the state diagram are encoded as shown in figure 4-2 (using the technique described in section 3a and 3b). This state diagram is a special case: for each of the state, we do not have any “unexpected event”. We use the method in section 3d to find out that the minimum bit length needed to achieve transition coverage is 4. If we look at the figure 4-2, we can easily see that every 4-bit pattern is used. Therefore, we do not need to use any of the solution described in section 3c (for we do not have any unused pattern).

- Random testing: We generate the random test cases using Java library class java.util.Random. The seed is 0.

Figure 4-2 Encoded state-based specification of the stack

Result: The result of the first experiment is given in figure 4-3. Although we collect both the decision and block coverage data, the shapes of the curves for both cases are quite the same. Therefore, to save the space, we only show the decision coverage. The diagram is as expected. It shows that the anti-random testing is better in term of code coverage.

Figure 4-3 Decision coverage for anti-random and random testing

b. Experiment 2

Subject: The subject of the second experiment is the implementation of the example that is used in section 3 (figure 3.1, 3.2 and 3.3). We implement this state-based system in C++ with a very simple functionality: whenever the system receives an incoming event, it checks the current state and if the event is valid, the system will change the state. In the case the system receive an unexpected events, it will print out an error message. The side of the implementation is 136 LOC.

Data collected:

- Anti-random testing: The state-based specification of the system (figure 3.1) is encoded as shown in figure 3.2. In figure 3.2, the doted lines show the cases of unexpected events: in these cases, the system does not change the state and just print out error messages. Similarly to the first experiment, we calculate the minimum bit length to represent the tests that satisfy the transition coverage (the result is 5). We also find out that there are certain unused 5-bit patterns, such as ‘11XXX’ and ‘XX11X’. Therefore, we have to apply the first solution as described in section 3c: we generate an anti-random sequence that contains all 5-bit patterns and then remove unused patterns from the sequence. The reason we chose the first solution is that it is easy to do. Because of the time constrain, we have not tried to apply the other two solutions.

- Random testing: We generate random test cases similarly to the first experiment.

Result: The result of this experiment is shown in figure 4.4. Again, we show only the decision coverage, since the shapes of the curves for block coverage are quite the same.

Figure 4-4 Decision coverage for the second experiment

Again, the experiment result suggests that the anti-random testing with the proposed checkpoint encoding can cover the code better than the random testing.

5. Conclusion

We have described a checkpoint-encoding scheme for state based system. We also proposed three different solutions for unused anti-random bit patterns. These solutions can be applied for any checkpoint-encoding scheme. We also test the effectiveness of our technique with two simple experiments. Although we expect that this technique should improve the test effectiveness (comparing to purely random testing), we will not claim anything base on the result of the experiments. Instead, we argue that the experiments give us some confidence, and it is the motivation to conduct other empirical studies in more sophisticated state-based system.

Another future work for this topic is conducting empirical study to compare the effectiveness of three solutions for unused bit pattern. We expect that the second solution (eliminate unused bit pattern before generating anti-random sequence) provides the best code coverage.

References

[1] Malaiya Y. K. (Oct. 1995), "Antirandom Testing: Getting the most out of black-box testing". Proceedings of International Symposium On Software Reliability Engineering, pp. 86-95

[2] Wu S. H., Malaiya Y. K. and Jayasumana A. P. (1998), "Antirandom Testing: Beyond Random Testing". Computer Science Technical Report CS-98-103, Colorado State Univ., Fort Collins, Colorado.

[3] Yin H. (1996), "Antirandom test patterns generation tool," Project Report, Colorado State University, Computer Science Dept.

[4] Mayrhauser A., Chen T., Hajjar A., Bai A. and Anderson C. (1998), Fast Antirandom (FAR) Test Generation, Colorado State University.

[5] Binder R. (1999), “Testing Object-Oriented Systems, Models, Patterns and Tools”, Addition Wesley.

[6] KimY.G., Hong H.S, Cho S.M., Bae D.H and Cha S.D (1999), Test Case Generation from UML State Diagram. IEE Proceedings – Software, Vol. 146, No. 4, pp. 187-192. Offutt J. and Abdurazik A (1999), “Generating Test from UML Specification”. In France R. and Rumpe B. (Ed.), Proc. of UML'99, Lecture Notes in Computer Science 1723 (pp. 473–488), Colorado: Springer

[7] Hong H. S., Lee I. and Sokolsky O. (2001), Automatic Test Generation from Statecharts Using Model Checking. Technical Report MS-CIS-01-07, Department of Computer and Information Science, University of Pennsylvania

[8] Offutt, J. and Abdurazik, A (1999), “Generating Test from UML Specification”. In France R. and Rumpe B. (Ed.), Proc. of UML'99, Lecture Notes in Computer Science 1723 (pp. 473–488), Colorado: Springer.

[9] Clarke E. M., Emerson E. A. and Sistla A. P. (1986), Automatic verification of finite-state concurrent system using temporal logic specifications. ACM transaction on

Programming Languages and Systems (TOPLAS), Vol. 8, Is. 2 (pp. 244-263), New York: ACM Press.

[10] DeMillo, R., Lipton, R. and Sayward, F (1978), “Hints on test data selection: Help for practicing programmer”, IEEE Computer, pp. 34-41.

[11] White L. and Cohen, E. (1980), “A domain strategy for computer program testing”, IEEE trans. Software Engineer, pp. 247 – 257.

[12] Musa J. (1993), “Operational Profiles in Software Reliability Engineering”, IEEE software, pp. 14 – 32.

[13] OMG (2001), OMG Unified Modeling Language Specification, Version 1.4.

[14] Kung, D., Lu, Y., Venugopalan, N., Hsia, P., Toyoshima, Y., Chen, C. and Gao, J. (1996), Object State Testing and Fault Analysis for Reliable Software Systems. Proc. of 7th International Symposium on Software Reliability Engineering, New York: White Plains.

[15] Yannakakis, M. (1991), Testing finite state machines. Proc. of the twenty-third annual ACM symposium on Theory of computing, pp. 476 – 485, New York: ACM Press.

[16] Tsai B., Stobart S., Parrington N. and Mitchell I. (1998), An Automatic Test Case Generator Derived from State-Based Testing, Asian Pacific Software Engineer Conference, Taipei, Taiwan

.