Using Testing and JUnit Across The Curriculum

Michael Wick, Daniel Stevenson and Paul Wagner Department of Computer Science

University of Wisconsin-Eau Claire Eau Claire, WI 54701

{wickmr, stevende, wagnerpj}@uwec.edu ABSTRACT While the usage of unit-testing frameworks such as JUnit has greatly increased over the last several years, it is not immediately apparent to students and instructors how to best use tools like JUnit and how to integrate testing across a computer science curriculum. We have worked over the last four semesters to infuse testing and JUnit across our curriculum, building from having students use JUnit to having them write their own test cases to building larger integration and use case testing systems to studying JUnit as an example of good application of design patterns. We have found that, based on this increased presentation and structuring of the usage of JUnit and testing, students have an increased understanding and appreciation of the overall value of testing in software development.

Categories and Subject Descriptions K.3 [Computers & Education]: Computer & Information Science Education - Computer Science Education.

General Terms Measurement, Design, Reliability, Verification.

Keywords JUnit, Testing, Unit Testing, Unit Testing Frameworks.

1 INTRODUCTION Unit testing has achieved significant prominence in the computer science curriculum over the past several years. This has been driven by several factors: the “test first” mentality of the agile programming community, the available of good unit testing frameworks such as JUnit [6] in the Java software development world, increased interest in knowledge of and experience with testing by industry, and computer science faculty’s interest in integrating this into the curriculum.

JUnit has a number of fundamental concepts. A test case is a Java class that tests some particular usage of the class in question. A test case consists of one or more test methods, which in turn test some component of the class. Multiple test cases can be combined into a test suite. Common test data can be organized into a fixture, which then can be used with setUp and tearDown methods to isolate activity done across multiple test methods within a test case. Students quickly see the benefits of a unit testing framework, and indeed they enthusiastically run the tests we provide in our first programming class in Java. However, we originally did not see a reasonable path in our curriculum to help students progress from using unit tests to writing unit tests to developing and understanding more sophisticated testing systems for tasks such as integration testing and use case testing. As we see software development moving increasingly in a test-driven direction, we have worked to structure our presentation of testing with JUnit and infuse testing and JUnit into a variety of courses at various levels in our curriculum. We have begun using JUnit across our curriculum over the past two years. We start in our first programming course by providing JUnit test cases and suites for our students and trying to instill good testing discipline within them. In our second programming course we move onto asking students to write their own test cases and suites, once they are familiar with concepts such as inheritance and interfaces and they have had more experience with JUnit. In the first software engineering course they use design patterns to build larger testing frameworks that cover integration testing as well as unit testing. Finally, in a senior level/capstone design course, the students study how JUnit makes use of a variety of design patterns. This leads our students to better understand and appreciate the value of testing as part of software development.

2 BACKGROUND JUnit was created by Erich Gamma and Kent Beck in the mid- to late-1990s in order to provide an easy-to-use framework that encouraged unit testing. JUnit has since become very popular, and can now be considered the flagship of a much larger xUnit family of testing frameworks. There are a number of books [7, 8, 11] that discuss the use of JUnit, both the technical details of using it and to some extent good practice issues. Similarly, there are a variety of articles on the world-wide web (e.g. [3, 4, 5]) that address various aspects of JUnit. Discussion of JUnit has begun to increase in the computer science education literature as well (e.g. [9, 10]). While many of these books and articles cover technical details associated with using JUnit, relatively few focus on best practices

(though see [4,11]) especially as they relate to instruction. This has led us to focus our attention on developing a more methodical approach to the use of JUnit and the teaching of testing from our first course through our capstone courses, so that we can help our students become more skilled in this increasingly significant and important area.

3 UNIT TESTING AND JUNIT IN A FIRST PROGRAMMING COURSE

In our first programming course we try to impress upon the students the value of testing generally and unit testing specifically. We now view this as developing a testing foundation that they will carry through the curriculum, building on this by writing their own unit tests, moving on to integration and other testing, and seeing how testing frameworks can be an example of good software design. As such, we have become more careful to make sure that we provide good test cases and suites for the students. We start out by providing the students with a lab exercise on JUnit early in the first programming course, and provide JUnit tests for the programming assignments that we assign. As we don’t introduce inheritance and interfaces until later in the semester, our focus regarding testing in this course is on having the students directly experience the value of unit testing their code as they develop. At this level we introduce test suites as well as test cases. While many of the references above focus on test cases and test methods, we are finding that the use of test suites allow us to encourage modular development early, as well as providing a tool by which to evaluate various subsets of functionality or even to evaluate assignments to different levels of quality. For example, we can now give an assignment where we develop an “A” test suite, a “B” test suite, etc., so that students can see that increased levels of unit testing provide additional quality to their work. We teach the students to place their unit tests in a separate source tree within the same package. For example, under Eclipse, we can have a project JUnitExample that contains a Java folder with the package edu.university.cs.popmachine that contains the domain classes for a Pop Machine system (e.g.PopMachine.java, PopMachineException.java, etc.), and this project also includes a JUnit folder with the same package edu.university.cs.popmachine with the test classes (e.g. TestPopMachineDropPop.java, etc.). We find that this separate source tree organization allows package access for the test classes but doesn’t clutter the source tree. It also supports a more modular view of the development of a software system. We follow a simplified subset of the test case design rules in this first course that we later ask them to use in designing and writing their own test cases in the second programming course (see Section 4). Providing good and consistent examples early encourages the students to follow our model as they move on to designing their own test cases.

4 UNIT TESTING AND JUNIT IN A SECOND PROGRAMMING COURSE

Building on the testing experiences students have in their first programming course, our second programming course explicitly teaches students how to design JUnit test cases for themselves. Obviously, this isn’t the focus of the entire course (we focus on

algorithms, data structures and their applications), however it does provide a nice opportunity to help students refresh on the concepts from the previous course while learning more about the proper design of JUnit test cases. From their earlier experience, students appreciate the important role of unit testing in software development and have a firm understanding of the appropriate file organization in which to store JUnit test cases. What they lack is the ability to effectively write their own test cases. We have developed a set of JUnit design rules that help students develop a structured and relatively impressive set of testing code. The following section presents our JUnit design rules and illustrates their meaning by applying them to the testing of the following simple source code.

public class PopMachine { protected List cans; protected int costPerPop; protected int deposit; public PopMachine(int cost) { costPerPop = cost; deposit = 0; scans = new LinkedList(); } public void add(Pop p) { supply.add(p); } ... public int dropPop() throws PopMachineException { int result = 0; try { cans.removeFirst(); if (deposit >= costPerPop ){ result.change = deposit – costPerPop; deposit = 0; } else { throw new NSFException(); } } catch (NonexistentElementException e){ throw new SoldOutException(); } return result; } } This class is meant to simulate the operation of a simple pop machine. Obviously this is a simple example, but the class does illustrate several features intrinsic in more interesting code.

4.1 JUnit Design Rules For A Second Course Derive one TestClass class from junit.TestCase for each target class. To modularize the test cases, we teach our students to implement an abstract class which extends junit.TestCase for each class they wish to test. The name of each such class should be TestClass where Class is the name of the given target class (see the example combined with the next design rule). Define one nested class inside the TestClass for each collaborator of the target class. This nested class is an example of a mock class [8]. The idea is that we want to make sure that our test cases test only the target class and not any of the related collaborator classes. By using mock classes we can replace the collaborators

with special purpose objects that have hard-coded behavior consistent with our needs. The name of the nested mock class should be MockClass where Class is the name of the actual collaborator class. The following code segment illustrates the TestPopMachine class and includes the definition of a mock class to replace the Pop collaborator of our PopMachine class.

public class TestPopMachine extends TestCase { public class MockPop extends Pop { ... } public void setUp(){...} public void tearDown(){...} } Derive one TestClassMethod class from TestClass for each method of each target class. Again, to help control complexity and to create modularity within the test cases, we teach our students to define a test case (subclass of TestClass) for each method of each target class. The name of each such class should be TestClassMethod() where Class is the name of the target class and Method is the name of the specific target method within that class. For example, the following code segment illustrates the TestPopMachineDropPop class that would be responsible for testing the dropPop() method of the PopMachine class.

public class TestPopMachineDropPop extends TestPopMachine { public void testExactChange(){} public void testUnderDeposit(){} public void testOverDeposit(){} } Implement one testScenario() method in the TestClassMethod class for each possible scenario of the target method. Typically, there are several paths through a method that must be tested. For example, the method should be tested when it should produce an exception, the method should be tested when it shouldn’t produce an exception, and so on. Such tests should be modularized and defined in a common location and thus we group these tests in the TestClassMethod class. Notice how this organizational structure isolates to one class all tests that validate the behavior of a single method. One of the key advantages to this design is that students are forced to explicitly think about the various scenarios for each method. For example, the following code segment illustrates the implementation of the testExactChange scenario for the dropPop() method of the PopMachine class.

public void testExactChange(){

popMachine.deposit = COST;

assertTrue( popMachine.cans.size()

==originalPopCount);

try { changeReturned = popMachine.dropPop();}

catch (NSFException unexpected){

fail("..."); }

assertTrue(popMachine.deposit==0);

assertTrue(popMachine.cans.size()==

originalPopCount - 1);

assertTrue(changeReturned == 0);

}

The variable originalPopCount used in the above code is initialized by the setUp() method of the TestPopMachine

class to be equal to the size of the list cans prior to the actual test code (code not show due to space considerations). The next code segment illustrates the use of the MockPop mock class in our testing design. In particular, the code segment illustrates the implication of the testAddNotFull() method of the TestPopMachineAdd test case class.

public void testAddNotFull(){

assertTrue( popMachine.cans.size()

==originalPopCount);

popMachine.add( new MockPop() );

assertTrue( popMachine.cans.size()==

originalPopCount + 1);

}

Notice that by using the hard-coded MockPop class, we can avoid inter-mixing the tests of the PopMachine class and the Pop class. If instead we had actually used the Pop class and an error was encountered, we would not know whether the error was caused by the add( ) method of PopMachine or by some behavior of the actual instance of the Pop class that was given to the add( ) method. Using a hard-coded mock class greatly reduces the chance that the parameter of the add( ) method will be responsible for any errors in the testing of the add( ) method. Derive one TestClassSuite class from junit.TestSuite for each target class. This class will serve as the focal point for all tests associated with the target class. The constructor for this class should use the static addTest(...) method of the junit.TestSuite class to add an instance of each individual TestClassMethod class to the TestClassSuite.

public class TestPopMachineSuite extends TestSuite { public TestPopMachineSuite(){ suite.add(new TestPopMachineDropPop()); suite.add(new TestPopMachineDeposit()); ... } }

4.2 Advantages and Disadvantages The main advantage to our JUnit design rules is that they create a modularized library of test cases for each class under construction. Further, the organization of this library directly reflects the separation of concerns found in the testing problem – all tests associated with a given method are stored together, all collections of tests associated with a given class are defined under a common ancestor are stored together in a given test suite. The main disadvantage to our JUnit design rules is that they tend to lead to a massive number of test classes. While this is true, we believe that current development environments combined with the hierarchical nature of the test classes in our result provides a sufficiently powerful abstraction that the volume and numbers of test cases is not a significant burden for the developer.

5 TESTING AND JUNIT IN A SOFTWARE ENGINEERING COURSE

The design rules presented in the previous section not only encourage a coherent set of unit test cases, but also lay the foundation on which we can build other types of test cases including integration tests and use case tests.

5.1 Integration Testing In our software engineering course, we introduce students to the concept of integration testing. Here the focus is not on the behavior of each isolated class, but rather on the behavior of progressively larger and larger collections of classes working together. Given the design rules discussed in section 4, the students are in a wonderful position to learn an effective technique for performing integration testing. By using a factory method design pattern [2] to create the instances of all mock classes in our test cases, we can get a fairly effective means of integration testing by simply deriving new test cases from our existing test cases. In these new test cases the factory method of the original test case (which produces instances of the mock object) is replaced with a new factory method that returns the actual object with which we wish to test integration. Consider the example shown in the following code segment.

protected Pop createPop() {

return new MockPop();

}

public void testAddNotFull(){

assertTrue( popMachine.cans.size()

==originalPopCount);

popMachine.add( createPop() );

assertTrue( popMachine.cans.size()==

originalPopCount + 1);

}

public class TestPopMachineAddWithPop extends TestPopMachineAdd { protected Pop createPop(){ return new Pop(); } } For first section of code illustrates the use of a factory method within the original TestPopMachineAdd class. Notice that the construction of a concrete “Pop” instance is isolated to the createPop() method. The second section of code illustrates how the definition of createPop() can be overridden in the new test case so that it integrates with the actual Pop class (not the mock class).

5.1.1 Advantages and Disadvantages Notice how we essentially get the integration test between PopMachine’s add(...) method and the Pop class for free. All we need to do is derive a new test case that simply overrides the factory method so that all the inherited methods of the TestPopMachineAdd class are now performed using an instance of the actual Pop class. One obvious disadvantage to this approach is that it is difficult to test all possible combinations of a target class with its collaborators. For example, if a class A collaborates with both classes B and C, then to test the integration of A with B would simply derive new test cases that replace the factory of each of our test cases with a factory that creates instances of B. Now to test A, B, and C together, we just derive new classes from our

new test cases. These new test cases replace the factory method that produces mock C instances with methods that produce actual C instances. However, if you also want to test the integration of class A with class C alone (not including the actual class B), this requires that we derive new tests directly from the tests for A, replacing the factory of mock Cs but leaving the factory of mock Bs alone. This can lead to a combinational explosion on the number of possible tests cases. We are currently researching ways of removing this combinatorial problem. However, for our current purposes, we simply encourage students to develop tests for A with B, then A+B with C, and so on.

5.2 Use Case Testing We also teach students how to develop use case tests. Unlike the unit and integration tests, use case tests are not motivated by the implementation classes, but rather by the functional requirements of the system (i.e., the use cases). However, our basic approach to testing still applies. The following use case testing design rules are analogous to the design rules presented in section 4 but they apply to use cases rather than individual classes. Derive one TestUseCasePackage class from junit.TestCase for each use case package in the system. Derive one TestUseCase class from TestUseCasePackages for each use case of each target use case package. Implement one testScenario() method in the TestUseCase class for each possible scenario of the target use case. Derive one TestUseCasePackageSuite class from TestSuite for each target use case package. This set of design rules teaches our students to derive a new class from junit.TestSuite for each target use case package (collection of related use cases). Next, students derive new classes from that class, one for each specific use case in the package. Then, each specific scenario (normal flow, alternative flows, exception flows) of each use case is tested in its own method of this class. Finally, test suites are used to create a hierarchical representation of the use case tests. We have found the symmetry of this approach to use case testing to that of unit testing to be comfortable and familiar to the students

6 DESIGN PATTERNS AND JUNIT IN A SENIOR-LEVEL DESIGN COURSE

In their senior year our students take a course which focuses on design through the use of architectural and design patterns. We have found that a straight catalog approach to this course tends to bore the students. Thus, we spend a lot of time focusing on how to apply patterns to given projects. Some of this work is done through the use of refactoring an existing design to use patterns and some is based on finding patterns from the start. We have found that the JUnit framework is a great example of a system where the types of problems encountered in its design provide nice matches to particular design patterns. In addition, since they have been using JUnit throughout the curriculum they are comfortable with using the framework which provides extra motivation to learn how it is designed. Instead of tacking the entire design of JUnit at one time and trying cull out the design patterns used, we instead follow the work in

[3]. We start with a blank slate and build up the framework incrementally by applying one pattern at a time, each one solving a particular problem. A short summary of the process and patterns involved follows. Those who are interested in more details should refer to [3]. The first thing we need to do is capture the idea of a test as an object. Developers have different styles of tests in mind and we do not want our framework to be concerned with the types of tests being run – just that there are tests that need to be executed. This is an example of the Command pattern. Thus we make a class called TestCase that captures the test operation as an object. This class will have a run() (acting as the execute() method in “Gang of Four” pattern terminology [2]) to hold the actual test. Now that we have tests can be manipulated as objects, we can take a look at the development of the tests themselves (i.e. the run() methods). When a developer needs to write a specific test case they will subclass TestCase and override the run() method to implement the concrete behavior of their test. However, all tests contain the same basic steps performed in the same order. Those steps are setup a fixture, run the test using the fixture, then tear down the fixture. This is a classic example of the Template Method pattern where run() becomes the Template Method and setUp(), runTest(), and tearDown() become the Hooks. Another issue to be solved with tests is that we must collect the results somewhere. We really only care about collecting information on the failures of our tests, thus an efficient way to do this is with the Collection Parameter idiom. This basically states that we want to create a collecting object and pass it to each test where the failure results can be registered with it. In order to invoke a particular TestCase, we need our invoker to call the generic run() method because of how TestCases were setup as Commands. This in turn calls the runTest() hook method because run() was setup as a Template Method. The issue that this creates is that when the test writer is creating concrete TestCases they need to create a new subclass for each test and put all the testing code in a method called runTest(). This produces a proliferation of classes. One would much rather allow them to create several related testing methods in a single class, as we described earlier in this paper. An Adapter pattern (class version) can be used to solve this problem. It adapts the interface that the test writer wants to use (e.g. testExactChange()) to the interface that the framework needs for polymorphism to work correctly (i.e. runTest()). The best way of providing these adapters in Java is to use anonymous inner classes to subclass the concrete TestCase. Of course this still means all the required anonymous inner classes, one for each testX method the user writes, need to be coded. Here is where reflection comes to the rescue. JUnit provides a default implementation of runTest() that looks for all testX() methods. It creates an adapter specific to each of these methods by creating an instance of a dynamically tailored adapting anonymous inner class. Lastly we want to be able to abstractly run a single test or a suite of tests. That is, we do not want the invoker of the tests to care if it is running one test or many. This is an example of the Composite pattern where TestCase becomes the Leaf node and TestSuites become the Composite nodes. We have now built up the guts of the JUnit framework using Command, Template Method, Adapter and Composite patterns, along with the Collection Parameter idiom and reflection. This not only makes a powerful example of how to design with patterns, but it shows the students the inner workings of the testing framework they have been using for years. And for some

of the better students it encourages them to dive into the framework and start extending it in an attempt to create a more personalized testing environment.

7 CONCLUSION We feel that the integration of testing into the curriculum is one of the most important trends in CS today. It enables students to produce cleaner code faster than ever before by catching bugs early in the development cycle. It also facilitates group projects by giving all group members the confidence that each other’s code is correct. Early coverage of unit testing allows students be taught how to write good test cases. Just because your tests pass does not mean your code is correct. The tests are only as good as the test cases themselves. Thus we have developed our methodology for writing good test cases and when to present this information to the students. We have a gradual progression from providing test cases to the students, to students writing their own simple test cases, to the inclusion of mock object and design patterns, to the development of more complex integration testing and use case testing systems. Finally we use the JUnit framework as an example of good design and give students the necessary understand of it allow for possible extension.

8 REFERENCES [1] Bertolino, A. and Gnesi, S., “Use Case-based Testing of

Product Lines,” ,” Poster Session from the Proceedings of the 9th European software engineering conference held jointly with 10th ACM SIGSOFT international symposium on Foundations of software engineering, Helsinki, Finland, 2003, pp. 355-358.

[2] Gamma, Erich, Helm, Richard, Johnson, Ralph, Vlissides, John, “Design Patterns: Elements of Reusable Object-Oriented Software”, Addison-Wesley, 1995.

[3] JUnit: A Cook’s Tour, http://junit.sourceforge.net/doc/cookstour/cookstour.htm

[4] JUnit Best Practices, http://www.javaworld.com/javaworld/jw-12-2000/jw-1221-ju it.html

[5] JUnit Cookbook, http://junit.sourceforge.net/doc/cookbook/cookbook.htm

[6] JUnit Home Page, http://junit.sourceforge.net [7] Link, Johannes; “Unit Testing in Java: How Tests Drive the

Code”, Morgan Kaufman, 2003. [8] Massol, Vincent, with Husted, Ted; “JUnit in Action”,

Manning Press, 2004. [9] Olan, Michael, “Unit Testing: Test Early, Test Often”,

Journal of Computing in Small Colleges, v. 19, n. 2, Dec 2003, pp. 319-328.

[10] Patterson, Andrew, Kolling, Michael, Roseberg, John, “Introducing Unit Testing with BlueJ”; SIGCSE Bulletin: Proc. 8th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, v.35, n.3, September 2003, pp. 11-15.

[11] Rainsberger, J.B., Sterling, Scott; “JUnit Recipes: Practical Methods for Programmer Testing”, Manning Press, 2004.