Replacing Software-Based Module with FPGA-Based Module

Slide 1*
*
MPR Associates
Bob Cardwell
Southern Nuclear
18th Annual Joint ISA POWID/EPRI Controls & Instrumentation Conference
*
Hello, my name is Danny Duong, and I’m an engineer at MPR Associates. I’m giving this presentation in place of David Herrell, who is unable to attend due to a family emergency. I am the Independent Verification and Validation team lead for this project so I’m familiar with the IV&V and testing aspects of this project (or at least I should be).
This presentation is on Replacing an Obsolete Software-Based Module with an FPGA-Based Module, and was created by David Herrell and Kyle Dittman of MPR Associates, with help and support from Bob Cardwell from Southern Nuclear.
There is a lot of content, and I’ll try to get through them in a timely manner, but feel free to ask questions if they come up. If I cannot answer any of your questions, I’ll be sure to make note of the question and get back to you.
*
Author Short Biography Slide
David Herrell is an Executive Engineer at MPR Associates, Alexandria, VA with 35+ years of nuclear digital I&C experience
Part of MPR’s senior I&C technical staff, he works with suppliers and nuclear power plants around the world
Worked for a system supplier, as a seconded contractor, and as a utility employee at Salem and Hope Creek prior to MPR
Bachelors and Master degrees in Electrical Engineering
Member of IEEE Nuclear Power Engineering Committee (NPEC), member and current chair of Subcommittee 6 on Safety Systems, and member of Working Group 6.4 responsible for IEEE Std. 7-4.3.2
*
*
Background
In the 1980s, Edwin I. Hatch Nuclear Power Plant replaced the electromagnetic timers on the Unit 2 EDGs with commercial software-based equipment
Three cabinets of equipment were installed, each containing 1 dc-to-dc converter, terminal blocks, 2 control modules, 2 alarm relay outputs, and 2 counters with relay interfaces to the counters
*
(2) The three cabinets are for their 2 EDGs and 1 Swing EDG.
*
Multiple failures and obsolescence concerns of Rochester Instrument Systems (RiS) modules initiated a project to generate form, fit, and function replacements for the control modules
SNC awarded a contract to MPR to re-engineer and provide replacement modules as basic components
MPR and SNC decided to base the replacement module architecture on a field programmable gate array (FPGA)
*
(1) We are mitigating our own future obsolescence issues by choosing industry standard components that have long production lifetimes. For example, we’ve learned that Actel typically extends their FPGA chips longer than other vendors. Moreover, SNC only needs 6 modules but is purchasing 14. Naturally, we are retaining additional spares.
(3) We’re choosing an FPGA-based architecture because of previous NRC acceptance of FPGA-based devices, and we perceive there to be less issues than associated with microprocessor-based systems
*
The equipment was reverse-engineered and implemented on an FPGA-based module
The Product Design (PD) Group has been developing FPGA-based designs for medical devices under FDA regulations
The Nuclear Group used the PD Group’s capabilities, by adapting the PD Group plans, procedures, and instructions into a safety-related Programmable Logic lifecycle process
*
(2)-(3) The Product Design group and the Nuclear Design group worked together to leverage both the FPGA design experience with the nuclear experience base for this project.
*
Background (cont’d)
Early in the project, MPR considered the possibility of relying solely on 100% testable and tested logic, but decided to continue with IV&V as good engineering practice
MPR had Gavial Engineering and Manufacturing procure components, assemble, solder, and preliminary test the modules under their 10CFR50 Appendix B compliant Nuclear QA program
The use of commercial components required the performance of commercial grade dedication activities on the fabricated module
Commercial grade dedication was performed per our 10CFR50 Appendix B compliant Quality Assurance Program
*
*
*
This is a picture of the OEM microprocessor based module.
*
*
This is a picture of the chassis that the OEM and replacement modules must seat into
*
Only limited design documents for the original modules were available, including cut sheets, limited functional design descriptions, and schematics from Rochester Instrument Systems
Having schematics for modules and for cabinet wiring eliminated the need for extensive wire tracing and generation of replacement prints
Verification (as-built) of the cabinet schematics was performed
The functional design descriptions were adequate to determine how the module worked in sufficient detail to avoid the need to disassemble the Fairchild F8 microprocessor software
*
*
Analysis of Existing Design (cont’d)
SNC provided a spare working cabinet for use in the design activities
Much of the software documentation (e.g., flowcharts) had little value other than showing how the module worked
FPGA implementation does not include problematic software features such as sequential instructions, jumps, multitasking, and hardware interrupts
It was determined that building a generic replacement was not appropriate
Replacement module functions were customized
Features of the OEM module were not needed
DG safety function could be simplified
*
(3) We essentially removed unnecessary I/O or features, while adding application specific enhancements
*
Re-Engineered Requirements
Unused inputs and outputs (12 inputs to 3, 12 outputs to 10), were eliminated
Reduced complexity,
Increased reliability (fewer parts),
Reduced power consumption, and
Allowed for enhanced diagnostics
Diagnostics enhanced to verify that the output relay coils have continuity, rather than just checking that the output switch turns on or off
Diagnostics use the inductive characteristics of the relay coil to check continuity
Diagnostics considered active (inject current) or passive (monitor voltage)
*
(2) Our active diagnostics are able to pulse the outputs long enough to detect continuity but short enough to not trigger the outputs
*
Design constraints for “new” EMC requirements; meets United States Nuclear Regulatory Commission Regulation Guide 1.180 requirements including:
Electrically fast transients requirements,
Electrostatic discharge requirements, and
*
*
System, Hardware, and Software Requirements
A single document was written for both module hardware and Programmable Logic requirements
Many of the detailed design decisions made during the original module’s design are now constraints on the replacement module (e.g., module size, electrical connection, and pinout; front panel size; chassis arrangement and wiring)
Requirements were created to address issues with the original design (e.g., weak ground connection from module to chassis, minimal distance from module to chassis necessitated paper insulator to protect OEM module contact with chassis)
*
(1) Particularly for the Programmable Logic requirements, this functional and structural design requirements document focused on the differences in architecture and function for VHDL.
*
System, Hardware, and Software Requirements (cont’d)
There was no attempt to recreate the original generic module requirements, as the module is being used for a single purpose in a single plant
Programmable Logic architecture created, showing how the parallel action embedded in the logic actually functions
Requirements and detailed design iterated to point where VHDL code implementation and module schematic could be started
Design iteration continued through completion of implementation, with final passes to resolve any remaining IV&V clarity issues
*
(2) Relative to what we define in the requirements. The process for creating the programmable logic also worked to improve the modularity of the programmable logic
(4) The requirements were refined and clarified throughout the requirements, implementation, and testing phases of the project
*
System, Hardware, and Software Requirements (cont’d)
MPR performed hazards analysis throughout life cycle, to inform the design, implementation, and V&V processes –hazards external to the replacement module could not be resolved
Hazards that could not be resolved involved constraints in the existing design, which were present from the initial installation; no new hazards were added
Hazards analysis activities augmented the testing program by verifying that all hazards for which testing could be performed were included in the testing program, and that those which could not be tested were reviewed independently
Routine surveillance testing covers the external hazards to the replacement module
*
*
Software Tools
Software tools are at least equally important for FPGAs as for software-based devices
Programmable logic requires evaluation of internal FPGA signal timing, which cannot be externally measured
Only way to evaluate internal timing is by use of simulation and timing evaluation software tools
Internal timing verification cannot depend on testing, since hardware may work while violating vendor internal timing constraints (e.g., setup, hold)
Most vendors provide frequent updates to their software tools, which should be considered for use, as tool errors are corrected (important) in addition to new FPGA support
*
(2)-(4) For the FPGA, we found that there were interesting challenges when considering timing constraints internal to the FPGA and the process for signal routing and timing.
(5) All software tools, including updates, were evaluated for use on this project.
*
Software Tools (cont’d)
MPR uses National Instruments LabVIEW for IV&V testing and for equipment qualification testing
Custom LabVIEW application designed, verified, and validated as a means of stimulating the module and measuring, recording, and analyzing the module’s response
MPR also uses a tool to generate requirements traceability matrices
Automatic generation of RTMs based on metadata tags embedded in documents eliminates the pain associated with manual generation and correction of generation errors
*
(2) The LabVIEW testing application was able to alert the test engineer to any unusual behavior
(4) These documents traced from the client requirements, to our requirements document, to the testing specification and procedures. The tool allows us to focus on what matters most with an RTM: the implementation and testing of the requirements
*
*
This is an image of the manufactured prototype board and faceplate. Other smaller proof of concept boards were created to test critical circuit designs.
*
Design Enhancements
As a basic component “form, fit, and function replacement,” enhancements have to fit within the module and cannot require external change; sensible enhancements are not precluded
Increase in computed MTBF based on the design changes
Original module required replacement of EPROM for each unique program and timing sequence
Replacement module has switch selectable sequence, one module with three selectable sequences
*
(2) These design changes include the specification of the generic module to a plant- and application-specific module as well as the added diagnostic functionality.
(4) Therefore, we could change the selected diesel loading sequence without have to swap EPROMs or dedicated function modules.
*
Original module had 3 LEDs – Power, Running, and Failed
Replacement keeps Power and Failed, eliminates Running as there is no equivalent in FPGA space
Adds LEDs for 3 external field contact states
Adds LEDs for 10 demanded relay output state – choice made to show demanded rather that actual state
Adds 2 numeric LEDs to provide:
Replaces external obsolete external counters (abandoned in place)
Added display of FPGA failure status code
Added display of selected sequence
*
(4) The choice was based on an I&C Technician request for surveillance testing support
*
Enhanced diagnostics for Hatch’s specific application:
Can now diagnose limited amount of relay and internal wiring failures
Did include separate watchdog timer, such that FPGA does not annunciate its own failure
Diagnostics driven by hazards analysis
*
The module does not provide extensive relay diagnostics, but could target the electrical behavior of the installed relays
(3) These include diagnostics on the state machine itself, for example.
*
Considerations
All activities were performed under our 10CFR50 Appendix B compliant Nuclear Quality Assurance Program, including Programmable Logic life cycle
Modifications were required to fit software life cycle to VHDL
Many design and review topics for software were not applicable to VHDL (e.g., interrupts, multi-tasking, constraints of sequential execution, loops, jumps, memory allocation/deallocation, paging, etc.)
Few new topics were added, including evaluation and checks of logic signal timing internal to the FPGA
*
*
Process and Implementation
Considerations (cont’d)
Since no mathematical functions, there are no typed variables; everything is either a bit or a collection of bits
There are no widespread industry consensus guidelines for good coding practices, as there are for software
MPR did not apply the equivalent of a static analyzer to the VHDL
*
*
Process and Implementation
Considerations (cont’d)
VHDL does allow for sharing signals between modules, with only explicit definition of the sharing
Program instrumentation (e.g., “printf()” in “C”) is simple in software; more complex in FPGAs and requires interesting logic to support simple scanning
Can still implement “stubs” and “drivers” for VHDL code for testing stimuli, just like for procedural languages
Software tools exist to simulate the internal logic, including delay times based on the placed and routed VHDL, which may be necessary to resolve timing issues
*
(3) Essentially, we can provide any number of stimuli signals to drive a particular response within the software testing tools. A great example is driving the FPGA with a modified frequency clock signal to test the clock diagnostics
*
With self-implemented math, precision and accuracy are still concerns
VHDL code must still initialize memory prior to use
Exception handling still must be designed in to the VHDL, with inputs and outputs checked for reasonability
Designers still make the same mistakes (e.g., bad assumptions, missing punctuation, and erroneous but compliable syntax, incomplete switch statements, etc.)
Generating good, complete unit tests is still as complex
For both software and VHDL, the quality of the product is a function of the designer’s experience and capabilities
*
(1) Obviously we’re only dealing with clocks, edges, and counter bits for this application
(2) All registers must be initialized to a known state, for instance on start-up or on reset
(3) We were able to cover this partially with diagnostics and partially by VHDL coding practice, review, and unit testing
(4) We tried to minimize coding errors not only with testing, but with 4 different code reviews: 1 design team, 2 IV&V, and 1 independent third party code review
*
QA Plans, Procedures, Processes
MPR has extensive experience with FPGAs in medical devices, under FDA rules
MPR’s Nuclear QA program requires generation of a task-specific QA plan for safety related projects, explaining how the project will work under 10CFR50 Appendix B constraints.
With the FDA-compliant processes tailored to 10CFR50 Appendix B vocabulary, work performed in accordance with our Nuclear QA program
*
(1)-(3) Our design and QA processes had to be refined for this FPGA-design project to work within the framework of our Appendix B compliant QA program
*
Intended to use the “100% testable and tested” approach
With all diagnostics and multiple state machines considered, the complexity required to apply all possible input combinations to all possible states becomes unreasonable
State Machines include: main state machine (8 states), sequencing step counter (62 states), field contact input debounce state machines (3 sets of ~22 states), active diagnostics
External and internal inputs include: 47 diagnostic failures, 3 field contact inputs, and front panel reset switch, or 251 combinations)
MPR used a traditional IV&V process, and notes that IV&V found design errors that testing would not have uncovered
*
We thought this would be feasible since we were only dealing with 3 inputs and 11 outputs
(3) Traditional IV&V process including implementation of the software life cycle on the FPGA code
*
Redesigning obsolete digital equipment is possible with no design documentation, but even a little documentation simplifies the process
Replacing an analog device is simpler than replacing a digital device
Replacing software-based devices successfully with FPGA-based devices requires thought, understanding of the original equipment, and familiarity with both software and FPGAs
Consider and implement modular reuse of VHDL code
Licensing an FPGA-based replacement is not significantly different than licensing a software-based device
*
(2) With the current technology, it’s pretty hard to hide op amps, transistors, resistors, and capacitors.
(3) The first two items are true of any design project.
(4) Within the VHDL code, it’s very efficient to make use of modular code. There is a key difference between microprocessor based implementation and FPGA based. An FPGA architecture has a single implementation of an output copied 10 times, whereas a microprocessor output subroutine would be called 10 times.
(5) The client is performing the 50-59 evaluation and has taken over all licensing aspects of this project
*
*
I’d now like to open the floor for any questions. Again, I can try and answer any questions you may have, but any questions that I can’t answer, I’ll be sure to get back to you with the answers.
REMEMBER TO REPEAT THE QUESTION!!!

Documents

Replacing Software-Based Module with FPGA-Based Module