CESE
Automated Testing of Large Multi-language SW Systems using Cloud
Computing
Technical Presentation
Principal Investigator (PI): Dr. Mikael Lindvall, CESENASA POC: Markland Benson, White Sands
Team members: Dharma Ganesan, Dr. Chris Ackermann(CESE)
GMSEC, CFS, Space Network, MSL
1© 2011 Fraunhofer USA, Inc. Center for Experimental Software Engineering
CESE
Problems
• Test cases are developed manually• Some test execution is automated (e.g., JUnit)• Test cases miss valid “corner” cases• Test cases are also programs: not easy for non-
technical stakeholders to understand• Difficult to summarize what was tested
Approach: Lightweight Model-based Test Generation and Execution
2© 2011 Fraunhofer USA, Inc. Center for Experimental Software Engineering
CESE
Test Generation Workflow
CESE
The structure of the approach
ITestMonkey
TestMonkeyImpl ITestDataProvider
TestDataProviderImpl
Model
System Under Test
Interface
List of abstract actions
List of abstract input data provider methods
Model is agnostic to the test execution technology ITestMonkey interface hides the test execution framework TestMonkeyImpl uses interfaces of the test execution framework
CESE
Tools Infrastructure• Modeling: Yed Graphical Editor from yWorks• Model traversal and test generation: Jumbl- Uni. Tennessee• Test Execution:
– Junit (Java) – CuTest (C)– Selenium (Web)– UISpec (Java Swing)– Sikuli (Image-based testing of legacy systems)
• Glue scripts:– Conversion of Yed models to Jumbl models– Preparing a test suite from generated test cases– Generation of system-specific build files (e.g., makefiles)– Helper scripts to clean-up generated files
CESE
Test Generation @ GMSEC …
• State-of-the-practice: Test cases are hand-crafted
• New initiative started to evaluate the feasibility of the FAST approach
• Modeled a portion of the GMSEC Software Bus based on existing test cases and documentation
• Automatically generated test cases
• Found a few problems (already fixed now)
CESE
Hand-crafted test case (snippet) public static void main( String args[] ) { Status result = new Status(); Connection conn = new Connection(); ConnectionConfig cfg = new ConnectionConfig( args );
// Create the connection result = ConnectionFactory.Create( cfg, conn ); checkError( false, result, "Creating the connection object" );
// Disconnect result = conn.Disconnect(); checkError( true, result, "Disconnecting before connection is established" );
// Connect result = conn.Connect(); checkError( false, result, "Establishing the connection to the middleware" );
} //..main()
CESE
Manually developed test cases – source of Inspiration
• We reviewed existing Java test cases
• Found that the tester has used certain
permutations of API-usage
• Also, both good and “evil” cases are considered
• We used these test cases as a source of reference for building API usage models
CESE
Test Generation @ GMSEC …public interface IConnection{
public Status Connect();
public Status Disconnect(); …}
APIs of the module under test
CESE
Structure of cFE/CFS
CESE
Structure of OSAL
CESE
Sample APIs
/******************************************************************************** Directory API ******************************************************************************/// Makes a new directoryint32 OS_mkdir (const char *path, uint32 access);
// Opens a directory for searchingos_dirp_t OS_opendir (const char *path);
// Closes an open directoryint32 OS_closedir(os_dirp_t directory);
// Removes an empty directory from the file system.int32 OS_rmdir (const char *path);
CESE
Example of an OSAL model
CESE
API doc of open directory/*-------------------------------------------------------------------------------------- Name: OS_mkdir
Purpose: makes a directory specified by path.
Returns: OS_FS_ERR_INVALID_POINTER if path is NULL OS_FS_ERR_PATH_TOO_LONG if the path is too long to be stored locally OS_FS_ERR_PATH_INVALID if path cannot be parsed OS_FS_ERROR if the OS call fails OS_FS_SUCCESS if success
Note: The access parameter is currently unused.---------------------------------------------------------------------------------------*/
int32 OS_mkdir (const char *path, uint32 access);
CESE
Inside Open Invalid Directory
CESE
Sample IMonkey Interface
• int32 removeDirectoryValid(void);• int32 removeDirectoryPathNull(void);• int32 removeDirectoryPathTooLong(void);• int32 removeDirectoryPathUnparsable(void);• int32 removeDirectoryCurrent(void);• int32 removeDirectoryNotEmpty(void);• …
CESE
Sample generated Test in CuTest
void Testosal_Filesystem_min_2(CuTest* tc) { status = makeFilesystemValid(); CuAssertIntEquals_Msg(tc,"Filesystem could not be created", OS_FS_SUCCESS, status);
status = mountFilesystemValid(); CuAssertIntEquals_Msg(tc,"Filesystem could not be mounted", OS_FS_SUCCESS, status); pointer = openDirectoryValid(); CuAssertTrue(tc, pointer != NULL); … status = removeFilesystemValid(); CuAssertIntEquals_Msg(tc,"Filesystem could not be removed”, status);}
CESE
Issues found using this method
• File-descriptors after removing file-system:• After somewhat long tests we would run out of
file-descriptors• This would even happen with a newly created
file-system• OSAL does not remove file-descriptors for files
open when the file-system is removed• Unable to create and open files• Some wrong error codes returned
CESE
Current Results• An end-to-end approach for test generation
– Successfully used on the GMSEC and the CFS and detected bugs
Next steps are to apply to the Space Network and the MSL projects
19© 2011 Fraunhofer USA, Inc. Center for Experimental Software Engineering