Automating Software Module Automating Software Module Testing for FAA CertificationTesting for FAA Certification

Usha Santhanam

The Boeing Company

Automating Software Module Automating Software Module Testing for FAA CertificationTesting for FAA Certification

Challenges in code based software testing FAA certification testingTest Set Editor (TSE) tool set and processRegression testingBenefits of TSELimitations of TSESummary

Challenges in Code Based Challenges in Code Based Software TestingSoftware Testing

Software complexity and accelerated development schedules make avoiding defects difficult

Untested code in user environment Difficulty of accurately capturing and

analyzing the sequence of events that occur during program execution

FAA Certification TestingFAA Certification Testing

Structural coverage is often not achieved through requirements based testing alone

Testing each module in isolation provides the desired coverage

The three levels of structural coverage to deal with are statement coverage, decision coverage, and modified condition and decision coverage (MCDC)

Structural CoverageStructural Coverage

Statement coverage: execute each source level statement at least once

Decision coverage: take each execution branch at least once

MCDC: designed to demonstrate the independence of individual conditions that constitute a complex decision

TSE Tool Set and ProcessTSE Tool Set and Process

TSE is designed to achieve shorter turnaround time and cut cost during regression testing

TSE is domain-independent and removes the language-dependent test driver chores

TSE requires the test engineer to construct the test cases, but the rest of the testing process is automated

TSE Tool Set and ProcessTSE Tool Set and Process(cont.)(cont.)

Tasks of testing a module for structural coverage are– Identifying inputs and outputs of a module (or

collection of modules)– Selecting test cases and tagging them with the

requirements they address– Constructing a test driver and stubs for

subprograms called by the module under test

TSE Tool Set and Process TSE Tool Set and Process (cont.)(cont.)

– Compiling and running the tests– Analyzing coverage

TSE assists the tester with all these tasks except the selection of the test cases.

Example: Procedure Get_End Example: Procedure Get_End in Package Ioin Package Io

package Io is Debugging : Boolean := False; procedure Get_End;end Io;

with Text_Io;package body Io is K : constant := 1024; Max_Line_Length : constant :=

16*K; Line : String (1 .. Max_Line_Length);

procedure Error (Msg : String) is begin null; end Error;

procedure Get_Line is

begin null; end Get_Line;

procedure Get_End is begin if Line (1 .. 4) = ".end" then Get_Line; else Error ("Missing .END, assume

here."); end if; end Get_End;end Io;

TSE_Tools.tclTSE_Tools.tcl

Regression TestingRegression Testing

All test cases are self-checking A batch file is used to run the test files

through the TSE testing process A batch log file is used to check for

failures Failed tests are examined and justified or

updated

Benefits of TSEBenefits of TSE

Tests can be developed by engineers who are not necessarily expert programmers in Ada

TSE includes a tool to analyze the source code and identify inputs and outputs of a given module

Provides coverage information

Benefits of TSEBenefits of TSE(cont.)(cont.)

TSE tests are easy to maintain due to uniformity of test definition files

TSE tests are repeatable with no manual interaction necessary

TSE tests can be executed on host or target platform using different support packages

Productivity ImprovementsProductivity Improvements

Based on our experience we have seen a range of 1:5 to 1:10 productivity advantage.– Depends on the complexity of the module

Estimate of TSE Based Estimate of TSE Based TestingTesting

We have developed a tool that estimates the number of hours needed to test a module with TSE

Based on Statement count, number of If’s, Case’s, Loop’s, Exit’s, Logical operators, short circuit control forms, and stub calls

The Get_End procedure estimated testing time is 7 hours

Estimate of Manual TestingEstimate of Manual Testing

Estimated size of the get_end test driver is 90 SLOC– Including main driver and stubs

Programmer productivity for the test code development is 2 SLOC per hour (Hihm)

This yields the productivity improvement of 1:6.5 for get_end procedure

Limitations of TSELimitations of TSE

It is not possible to construct one generic module test to verify all instantiations– Handled on an instantiation by instantiation

basis

Tasking is difficult to test because tasks run in independent parallel threads– Each subprogram can be tested independently

regardless of the number of tasks sharing it

Limitations of TSELimitations of TSE(cont.)(cont.)

For modules using private type parameters or returning private type, construction of external test driver is awkward

– Embed the test driver in the body of the package that defines the limited private type

Nested subprogram testing not available– Requires the test driver to be nested – Must be tested in the context of enclosing subprogram

Summary of TSE BenefitsSummary of TSE Benefits

Minimizes the need for programming knowledge

Cuts testing effort several fold over manual approach

Minimizes cost of retesting Allows us to meet FAA/DO-178B

requirements in cost effective manner