Software QA

For Active CECs at SNS

2 Managed by UT-Battellefor the U.S. Department of Energy

“As Scott Jerome-Parks lay dying, he clung to this wish: that his fatal radiation overdose — which left him deaf, struggling to see, unable to swallow, burned, with his teeth falling out, with ulcers in his mouth and throat, nauseated, in severe pain and finally unable to breathe — be studied and talked about publicly so that others might not have to live his nightmare.”

Radiation Offers New Cures, and Ways to Do HarmNYTIMES.COM, Jan. 23, 2010

Motivation

3 Managed by UT-Battellefor the U.S. Department of Energy

• Software QA is a BHAG– Fortunately our scope is limited

Observations and Questions

4 Managed by UT-Battellefor the U.S. Department of Energy

• Software QA is a BHAG– Fortunately our scope is limited

• Are we the “before” picture or the “after” picture?

Observations and Questions

5 Managed by UT-Battellefor the U.S. Department of Energy

• Software QA is a BHAG– Fortunately our scope is limited

• Are we the “before” picture or the “after” picture?• How much software QA is enough?

– How many CEC software engineers does it take to screw in a light bulb?• Seven. One to write the specification program, two to screw it in, one to

check if they screwed it in, one to validate that it was screwed in correctly and two to explain why the project was late.

Observations and Questions

6 Managed by UT-Battellefor the U.S. Department of Energy

• Girls just want to have fun– It takes a village

• If it ain’t broke don’t fix it– Change may lead to a learning experience at 3:00 am

• KISS, maybe– Just one more feature….

Observations and Questions

7 Managed by UT-Battellefor the U.S. Department of Energy

Objective

• Review current status of QA related activities for SNS CEC software

• Establish a framework based on a consensus national standard(s)

• Come up with a comprehensive roadmap for CEC software QA at SNS

8 Managed by UT-Battellefor the U.S. Department of Energy

Things to Keep in Mind

• SNS has a large PLC based CEC for the accelerator– But most of the work now is centered around new

instruments

• These new systems are based on “safety” PLCs• Limited Variability Language

– Aimed at users to create their safety application functionality. Typical languages used are Ladder Diagram and Function Block Diagram

9 Managed by UT-Battellefor the U.S. Department of Energy

• SNS started out with the “standard” two redundant industrial PLC model with two programmers

• Transitioning from two box, two programmer model to one box, two programmer model

• One programmer writes the non-safety task and ½ the safety task while the other programmer writes the other ½ of the safety task

• AB safety PLCs have certified code modules plus diagnostics built into hardware– Use of these tools it not exactly leaping out at us

The times they are a changin’

10 Managed by UT-Battellefor the U.S. Department of Energy

Current status

• Things we do per procedure• Things we do but not proceduralized• Things we intend to start doing

11 Managed by UT-Battellefor the U.S. Department of Energy

The following activities are required by the listed procedure:

Description Doc identified

Software lifecycle V module reference

Write SSRS SNS-ASD-IC-P03 1

Two programmers required SNS-ASD-IC-P03 1

Implementation testing (module testing) SNS-ASD-IC-P03 5

Integration testing SNS-ASD-IC-P03 6

Certification testing (Validation) SNS-ASD-IC-P03, SNS-OPM-3.A-7.4

7

Modifications to software and subsequent testing requirements SNS-ASD-IC-P03

PPS logbooks SNS-ASD-IC-P03

Use of non-network laptops for PLC program work SNS-ASD-IC-P03

Storage of PLC programs on CDs SNS-ASD-IC-P03 Configuration management

Details for storing PLC programs on CDs SNS-ASD-IC-P05 Configuration management

12 Managed by UT-Battellefor the U.S. Department of Energy

The following features have been partially incorporated: • Expanded code comments

• Reference to safety function• Pulse Test• Complimentary inputs

13 Managed by UT-Battellefor the U.S. Department of Energy

Standard, standard, whose got the standard• Target standard is ISO 13849-1:2006(E), Safety of

machinery-Safety related parts of control systems- Part 1: General principles for design

• Most applicable standard to current projects (Instrument PPS equipment)

• Addresses software QA

14 Managed by UT-Battellefor the U.S. Department of Energy

Software lifecycle

15 Managed by UT-Battellefor the U.S. Department of Energy

Software related software specification: The SSRS document is unique to each PPS and provides information necessary to generate the PLC program. Included are safety functions to be accomplished, guidelines such as two programmer rule, system fault definition, particular functional systems details, etc. The SSRS addresses the safety functions from the SRD. The SSRS is identified as a requirement in document SNS-ASD-IC-P03.

16 Managed by UT-Battellefor the U.S. Department of Energy

System design: program configuration – tasks, safety tasks, routines, I/O routing, communications

Module Design: Based on the system design and SSRS. a module refers to pieces of field hardware that require software to collect inputs, process data, and provide outputs. A module could be a beam shut down station, a RAD detector, a trap key sequence, etc...

Should this be standardized?

17 Managed by UT-Battellefor the U.S. Department of Energy

Coding: Is the process of writing program modules, as a standard each module may have its own routine, of be combined with other modules that contribute to similar functions. Documentation is added to identify each module and it’s components.

Independent review?

18 Managed by UT-Battellefor the U.S. Department of Energy

Module Testing - Each module is tested individually and with the system to ensure a proper and excepted outcome.

How formal?

19 Managed by UT-Battellefor the U.S. Department of Energy

• Ideally will use an independent programmer in addition to two person programming team

• Can review code to structure test plan

• Has more latitude to change PPS equipment to facilitate testing

• Semi-formal documentation- placed in Projectwise

Integration testing

6

20 Managed by UT-Battellefor the U.S. Department of Energy

• Formal OPM procedure

• Structured such that “A cave man can do it”

• Performed per work control process annually and tracked (ASE requirement)

• Written by an independent person

• Intent is to perform “black-box” testing

• Focus on testing interlocks that do not have built in diagnostics

Validation

7Periodically testing

software or just hardware faults?

21 Managed by UT-Battellefor the U.S. Department of Energy

• Improvement initiative– Standardize format, test progression, signatures, etc.– Track safety functions per SSRS– Coordinating testing with other systems (TPPS)– Compartmentalized to facilitate testing– Including warnings and cautions in lieu of a JHA– Include brief operational description for reference

Validation

7

22 Managed by UT-Battellefor the U.S. Department of Energy

• Use resources from vendor’s PLC safety manual• Independent code review/ testing• Tracking of safety functions throughout documentation• Separate ACL for software items• Development of standard hardware/ software layouts for

instruments

Future Improvements

23 Managed by UT-Battellefor the U.S. Department of Energy

Are we there yet?

PERFORMANCEMANAGEMENT

5. WORK PROCESSES

6. DESIGN 7. PROCUREMENT 8. INSPECTION &

ACCEPTANCE TESTING

9. MANAGEMENT ASSESSMENT

10.INDEPENDENT ASSESSMENT

1. PROGRAM2. PERSONNEL

TRAINING & QUALIFICATION

3. QUALITY IMPROVEMENT

4. DOCUMENTS & RECORDS

ASSESSMENT