Software Testing for Safety-Critical Applications

Presented by: Ciro Espinosa & Daniel Llauger

Agenda

Introduction Software and Industrial Automation Software Testing Strategies

Black-Box Testing White-Box Testing

Software Testing Processes and Steps Unit Testing Integration Testing

Top-Down Integration Testing Bottom-Up Integration Testing

System Testing Software Performance Testing

Availability, Survivability, Flexibility, Stress, Security, Usability and Maintainability Testing

Software Maintenance Conclusion

Introduction

Safety-critical applications in industrial automation have unique concerns for software quality (demanding requirements on system performance).

Software testing consists in functional and performance testing.

Software and Industrial Automation

The possibility for malfunction of complicated industrial automation software becomes much greater than the traditional one.

Embedded measurement and control systems in life-critical systems have unique concerns for system safety and reliability.

Any error in the software system is fatal to the real-world applications of industrial automation software

Basic testing principles:

Present expected testing results when designing test cases.

Separate sw testing team from sw development team.

Design invalid test cases.

Perform regression testing each time sw under-test is revised.

Tester should concentrate on error-prone program segments.

Software Testing Strategies

Black-Box Testing

White-Box Testing

In practical applications, the tester often combines black-box testing and white-box testing to conduct a more thorough test of the software.

Software Testing Processes and Steps

Testing real-time & embedded software is difficult

50 % of overall work in life cycle dedicated to testing and optimizing

Complexity – large number of potential paths

Real-time constraints exacerbate testing

Unit Testing

Integration Testing: Top-Down Integration

Bottom-Up Testing:

System testing

Find discrepancies between actual SW performance to its expected performance by comparing it w/ requirements

Test cases created based on requirement analysis description

On-site testing after SW installed with other equipment

Software Performance Testing

Real-time monitoring & control system Sensor measures monitored values Manner in which error between actual

& target measurement to be corrected determined by microprocessor-based controller

Actuator executes command to drive control variable close to target values.

Availability testing

Applies to time-critical, on-time, & real-time applications

Real-time SW needs to be designed to meet time constraints

Must meet deadline in responding to an operation

Control correctness and real-time restrictions can be considered simultaneously in design process

Availability testing (cont.)

Example: Chemical plant Alarm MGMT SW responsible for

triggering a siren (alarm) when abnormal process occurs

If it cannot perform alarm operation quickly, then disaster or production incident occurs

Reliability testing

Measures the likelihood for failure-free SW operations

Tests ability to operate w/o failure in environment designed for

Capture any abnormal status Why is this testing needed?

Low reliability SW can destroy industrial equipment or have a loss of life

To guarantee quality in manufactured products

Survivability Testing

Distributed and embedded real-time software system should be able to recover from the degraded performance.

Fault-tolerant control algorithms should be incorporated into the software design to maintain software performance in the presence of component failures.

Flexibility Testing

In the industrial automation area, the sw needs to work with heterogeneous hardware drivers and software components provided by different manufactures and vendors.

Today, open architecture-based software is gradually replacing traditional, proprietary software architecture

Stress Testing

Stress testing tests the software by pushing the system to its limits.

Stress testing may find both hardware and software defects.

Stress testing can be used to examine what types of system failures will occur when the system is heavily overloaded.

Security Testing

Security is an increasingly important issue in industrial automation software systems, especially with the proliferation of Internet-based industrial applications.

System security needs to be considered during all phases of the software life cycle.

Usability Testing

Usability tests how well the user operates the software system and likes doing so.

It is crucial to design the software with high usability

GUI design is an important component in system usability.

Maintainability Testing

High maintainability enables the released software to be revised in the presence of errors during system operations and makes the software expansion and change easy for new applications.

Maintainability should be considered from the very start of the life cycle.

For complex software, high maintainability becomes more necessary, as it is hard to identify the faulty lines of code without well-written documentation.

Software Maintenance

Establish safe & well controlled mechanism for update and modification

ID unexposed defects after installation Adapt to changing user requests & to

different operating conditions Corrective maintenance Adaptive maintenance Perfective maintenance Preventative maintenance

Conclusion

In the testing of industrial automation software systems, performance, flexibility, reliability, survivability, and usability should also be tested thoroughly.

Testing, code inspections and reviews are needed to prevent loss of life or industrial accidents

Questions?