Overview History of Testing Advantages to Automated Testing Types of Automated Testing Automated Testing Tools Disadvantages to Automated Testing
History of Testing
Four main phases in testing methodology: Debugging (1950s) Exhaustive Testing/Prove it Works (1960s) Prove it Does Not Work (1970s) Defect Prevention & Testing Process (1980s)
Debugging No distinction between debugging and testing. Consisted of making code compile and changing code so that it would not crash.
Exhaustive Testing/ Prove it Works Exhaustive testing in terms of code paths and given input. Found to be impossible Write tests to prove the program works. Testing defined as what is done to demonstrate correctness of a program.
Prove it Does Not Work Testing not for proving correctness, but for finding defects. Started testing to see if a program did what it was not suppose to do, not just test to prove that it does what it was suppose to do.
Defect Prevention & Testing Process Testing traditionally referred to what was done to the system after working code was produced. Testing moved towards an activity done during a phase, not a phase by itself. Studies show that 50% of bugs are created in requirements and design stages.
Advantages to Automated Testing Test scenarios impossible/unfeasible using manual testing. Lowers testing time and costs in an iterative environment. Remove the human error factor. Introduces stability in regression tests. Increase in testers morale.
Types of Automated Testing Split into two categories Black box tests White box tests PC applications Embedded Systems
Black Box Automated Testing GUI testing Always been prime candidates for manual testing. Need to automate has created many commercial tools. Two ways to create the tests: Scripts Recordings
Black Box Tests - Scripts Less of a testing tool/environment and more of a testing library. Flexible. Easy to run batch scripts to run multiple tests.
Black Box Tests - Recordings Full testing tool/environment. Tester performs a test once while tools is recording his actions. Tester can add ending conditions to test the success of a test. Very useful for testing GUIs of stand alone applications.
White Box Automated Testing Module testing. More APIs then environments. Embedded systems introduce more complexity to white box testing.
White Box Tests PC Apps Tests usually written in the same language as the system being tested. Tools usually come in the form of an API containing asserts and other testing utility methods. Tests can be performed on any level of abstraction required by the project.
White Box Testing Embedded Systems Test code usually not stored/run on the same hardware as the system code. Do not have direct access to system code or memory. Requires an interface with an embedded system debugger.
xUnit Compilation of popular code driven testing frameworks. Frameworks based on the design by Kent Beck. Originally implemented for SmallTalk as SUnit.
xUnit Partial List of Frameworks NUnit (.net) JUnit (java) CUnit ( C ) CPPUnit (C++) SUnit (SmallTalk)
xUnit xUnit test execution Setup Test body Teardown
SilkTest Created by Borland. Works with: Web Browsers(IE and Firefox) Java GUIs .Net GUIs Win32 GUIs Price : 4-6k
Selenium Testing tool for web applications. Comes in API and tool form. Supports: Windows: IE, Firefox, Seamonkey, Opera Mac OS X: Safari, Firefox, Camino, Seamonkey Linux: Firefox, Konqueror
Selenium Can create tests using the Selenium library in many languages (Java,.NET, Perl, Python and Ruby) Or you can create tests using their Selenium IDE which is implemented as a Firefox extension. Once created in Firefox, the tests can be run in any of the suported browsers.
Custom Write your own custom testing environment or API. Used when you have very specific testing needs. Often used for embedded systems.
Disadvantages to Automated Testing Initial cost. Who tests the testers?
References Borland. (2007). Borland SilkTest: An Automated Regression and Functional Software Testing Tool. Retrieved April 2, 2007, from http://www.borland.com/us/products/silk/silktest/index.html http://www.borland.com/us/products/silk/silktest/index.html  Dustin, E., Rashka, J., Paul, J. (1999). Automated Software Testing: Introduction, Management, and Performance. Addison-Wesley.  Lewis, E., W., (2005). Software Testing and Continuous Quality Improvement. CRC Press LLC.  NUnit. (2006). Nunit. Retrieved April 2, 2007, from http://www.nunit.org http://www.nunit.org  OpenQA. (2006). OpenQA: Selenium. Retrieved April 2, 2007, from http://www.openqa.org/selenium/index.htmlhttp://www.openqa.org/selenium/index.html