Testable and Diagnosable Commercial off the Shelf

(COTS) Electronics

Louis Y. Ungar

A.T.E. Solutions, Inc.

Los Angeles, CA

LouisUngar@ieee.org

Abstract—Commercial off the shelf (COTS) are inexpensive

electronics modules, partly because the designs are expected to

conform to commercial needs and driven by market popularity,

rather than military and avionics logistics. Design for testability

(DFT) and for diagnosability (DFD) can serve both

manufacturing and end user concerns. COTS testability,

however, are generally focused only on improving manufacturing

test – or the vendors’ domain. End users’ testability and

diagnosability concerns are different. For COTS to serve system

level support, designers must plan for maintenance and repair

paradigms. We present a set of general and simple to implement

guidelines for COTS designers so that system designers can

create testable and diagnosable systems using COTS modules.

The guidelines will initially serve as selection criteria between

competing COTS, but we hope it will eventually become a

universal design guideline for all COTS designers.

Keywords-testability, DFT, COTS, system level test, diagnoses,

diagnostics, logistics

I. INTRODUCTION

Design for Testability (DFT) and Diagnosability (DFD) have been requirements, suggestions or part of wishful features for several decades. [1, 2, 3, 4] The motivation for testable and diagnosable products includes the following:

1. Detect and isolate workmanship defects during the manufacturing of the electronics product, including shorts, opens, missing and incorrectly oriented components.

2. Ensure the proper operation of components, modules and systems prior to delivery to the end user.

3. Provide built-in or embedded self test capabilities for end users assurance of proper operation.

4. Detect and diagnose failures that end users experience.

The first two items are intended for the circuit manufacturer and most testability guidelines are geared towards easing and ensuring test capabilities for the product vendor. The latter two items are end user oriented. They are often included as part of the procurement requirement and are usually included with military procurement.

Commercial off the shelf (COTS) is equipment that was not originally designed for a single user and not designed to meet procurement requirements or specifications. COTS, however,

is attractive, not only because of its considerably lower price and almost immediate availability, but because the large user base public has placed COTS under greater scrutiny and improved its reliability. For the past decade or so, the US military and avionics users have opted for COTS rather than custom design. This has solved many problems, but testability and diagnosability – features generally contained in product procurement specifications – were limited, if not outright ignored. Specifically, the latter two motivations affecting the end users did not get much attention. COTS can be systems, modules, boards, components and even software. Unless we specify otherwise, in this paper when we refer to COTS, we mean COTS modules.

In a previous effort [5], we determined that making COTS testable and diagnosable creates an economic value, or said in other words, a lack of these capabilities creates support costs the end user of COTS will have to endure.

P Presence

Structural C Correctness

Components O Orientation

L Live

A Alignment

S Shorts

Structural O Opens

Connections Q Quality

Functional F Feature

Components and A At-speed

Connections M Measurement

S Status

System Functional T Transmit

Inter-module and R Receive

Connections I Integrity of signal

D Diagnosability

Figure 1. PCOLA/SOQ/FAM/STRID Testability Attribute Metrics

978-1-4244-9363-0/11/$26.00 ©2011 IEEE

We suggested an extension to the popular PCOLA/SOQ testability metric [6] to encompass system level functionality as shown in Figure 1.

PCOLA and SOQ measure attributes that are important to the circuit (board) manufacturer, namely Presence, Correctness, Orientation, Live, Alignment and Shorts, Opens Quality. PCOLA offers metrics of component level testability, evaluating to what extent component presence, component correctness, proper orientation, basic sign of life or proper alignment can be determined by a test and tester. SOQ provides a similar metric for connections, evaluating the extent to which a test or tester can determine whether a short or open circuit is present, or whether the quality of the connection is adequate.

Feature, At-Speed, Measurement (FAM) are functionality measures that are important to both manufacturers and end users. FAM provides metrics of the function, by evaluating how many of the features, at what speed, and with what parameters the functionality can be determined by a test.

The inter-module system level attributes STRID – Status, Transmit, Receive, Integrity of Signal, Diagnosability – are primarily for the end user concerned with the homogeneous operation of the entire system, as well as with the proper diagnoses of the module containing the root cause of the failure. STRID provides metrics for the module’s ability to report its status (good, faulty, limited). It also determines the extent to which the module is capable of transmitting and/or receiving information to/from the outside world. It also includes metrics on its signal integrity, including degraded (but functional) characteristics. Finally, the diagnosability metric indicates the extent to which the module can distinguish its own faulty condition from one that may have originated in another module. We will detail each of the STRID attribute in Section III.

In this paper, we will focus our attention on the STRID attributes and offer guidelines to make COTS more testable. Section II provides system level testability guidelines from military and commercial standards and recommendations. Section III will focus on using STRID attributes for COTS module testability. Section IV will illustrate how these methods can be applied to an example system constructed from COTS modules that meet our testability criteria and contain STRID attributes.

This paper is intended to serve two groups: First, the guidelines are intended for COTS designers who can add the required features to new designs. Second, the guidelines can serve as selection criteria for purchasers of testable and diagnosable COTS modules.

II. SYSTEM LEVEL TESTABILITY GUIDELINES

Integrated Diagnostics introduced in the late 1980s through MIL-STD-2165 addressed many of the system level testability attributes and offered specific guidelines for system level testability and diagnosability. The Surface Mount Technology Association (SMTA) and the Testability Management Action Group (TMAG) created the SMTA/TMAG Testability Guidelines which also address system level issues.

A. The Scope of Testability and Integrated Diagnostic

MIL-HDBK-2165 (a later variation of MIL-STD-2165) states that its purpose is to

(1) recommend system test and testability requirements which best achieve availability and supportability requirements and

(2) allocate those requirements to subsystems and items diagnosable.

The handbook establishes activities that should be performed by a team consisting of test and design professionals, which will ultimately include test concerns within the design.

Specific tasks that are outlined include a uniform approach to testability program planning, establishment of testability (including built-in test or BIT) requirements, testability analysis, prediction, and evaluation, and preparation of testability documentation. The following tasks are identified:

Testability program planning

Testability requirements

Testability design

Testability prediction

Testability demonstration

Testability data collection and analysis

Documentation of testability program

Testability review

All of these activities assume that the system is being designed from scratch and that the designer has control over individual circuit configurations. When the design consists of COTS modules, the activities may add other limitations, but since the design must still be supportable, it is still important to go through these activities.

The appendices of MIL-HDBK-2165 contain some guidelines applicable to system level testability.

B. General System Level Testability Guidelines

Some of the MIL-HDBK-2165 system level testability guidelines have found their way to the SMTA/TMAG Testability Guidelines and they are presented below:

1) Organizational dissemination The goals and concepts

of Design for Testability (DFT) should be disseminated

throughout the entire organization, including suppliers and

subcontractors.

Each supplied part or module needs to be able to ascertain its own status as faulty, or alternatively allow the higher level system or module to make that conclusion. In a distributive approach the modules test themselves, while in a centralized approach, the higher level system tests its lower level components. Both of these approaches can use built-in test (BIT) or built-in self test (BIST). If the tests are not built-in,

then ATE must be capable of applying stimuli and collect responses from and external vantage point.

2) Integration with other requirements A Level of Repair

Analysis needs to be created, preferably in conjunction with

the Failure Mode Effects [Criticality] Analysis (FME[C]A), to

ensure that the repairs are feasible in terms of time and cost.

It is more cost effective to anticipate failures and the impact they will have on diagnostic and repair times and costs, and to mitigate them as much as possible while the product is in its design or (in the case of COTS) in the procurement phase.

The FME[C]A and DFT analyses should both be performed early in the design stage. Since they both analyze failures and effects, it is beneficial to exchange information between these two requirements. For example, a failure mode that has no effect is one that presents both a FME[C]A and a DFT problem. It tells us that such a failure mode would not be detectable and may result in hidden faults. Similarly, having several different failure modes from different modules result in the same effect, presents a diagnosability complication.

3) Self and neighbor test capabilities Not only should

components be designed to test themselves, they should be

designed to support the test and diagnosis of other components

in the system environment.

Self-testing components (i.e. with BIST) can perform a number of important test functions. They can detect (and diagnose to the component) a fault within their boundaries. When BIST passes the component can also be used to test its neighbors. Any fault occurring in the interaction between this (self-tested) component and its neighbors can detect and often isolate faults to the root cause. (Our use of “component” here can also mean modules, or boards, not just ICs.)

4) Conceptual design stage testability analysis The test

and diagnostic methods used to support the product should be

described in the conceptual design stage and specific

explanations provided on how the design is capable of meeting

fault coverage and fault isolation requirements during

manufacturing and support.

While it is difficult to ensure that a system that is testable at the conceptual design stage will remain so as the detailed design progresses, it is very likely that if testability is ignored at the conceptual design stage, no later stage can make up for it. Consider a block diagram that has internal blocks, which are difficult to control or observe. No matter how testable the inside of those blocks become, the blocks will still have to be tested at the system level. In many cases, opportunities to improve fault detection and fault isolation or diagnoses are only available early in the design stage.

The SMTA/TMAG Testability Guidelines [4] explains this guideline:

“Testability and diagnosability issues should be brought up during the initial design stage. Specific questions should be raised by test, testability, reliability and supportability personnel about the design. For example, during the initial Failure Mode Effects [Criticality] Analysis (FME[C]A), the reliability engineer may inquire whether all failure modes at any of the modules or

subsystems will result in a system-level effect. Test engineers may ask what percentage of the nodes in a circuit board is accessible through I/O pins and/or through boundary-scan access. Support engineers may ask whether BIST can isolate a fault to be within one replaceable module or circuit board in, say 98% of the cases. A list of many such questions was published in Appendix B of the MIL-HDBK-2165 and similar software-based questionnaires are available commercially. Organizations can create such questionnaires themselves. The most important factor is WHEN these questions are answered. In all cases they should be answered, reviewed and discussed prior to finalization of the design. “ [4]

C. How these guidelines apply to COTS

The four guidelines we discussed stand out in importance when a system is undergoing design. No distinction is made here between circuit design from scratch or one where some (or perhaps all) the system components consist of COTS modules. COTS modules, however, do alter the playing field since “design” often consists of selecting an item from a catalog. In the next section we will discuss what attributes to look for in the COTS selection process so that we can still satisfy system level testability and diagnosability requirements.

III. COTS TESTABILITY GUIDELINES

In previous efforts [5], COTS testability and diagnosability attributes were identified. The attributes include Status, Transmit, Receive, signal Integrity, and Diagnosability, which we collectively call STRID. COTS buyers should seek out and place values on COTS that contain these testability attributes. Additionally, COTS designers are urged to include at least some, preferably all these attributes in systems, modules or circuit boards they sell.

We will explain each of these attributes and advise system designers on what to look for when they select COTS modules for their systems.

A. Status (S)

This attribute is intended to provide COTS health information to the system housing the COTS and possibly outside the system.

The system designer should favor COTS that has a comprehensive built-in self test (BIST). In many cases, however, system designers may settle for less. Often embedded test or some confidence gaining mechanism provides sufficient health status. In some cases the status maps to a specific part or component within the COTS as faulty and in other cases faults can be mapped to a particular function. Depending on the logistics in repair, it may not be necessary to identify the fault beyond the COTS module itself.

The COTS designer could also benefit from status information and self test mechanism. In some instances a COTS module is equipped with status flags and words, but this information is not shared with the end user. The reason may be that false alarms may be created, or providing status information may tempt the end user to attempt repairs within the COTS module. Whatever the reason, COTS designers who have built-in test and status information should find a way to

communicate its results with end users. If status information is not available, it would help the COTS vendor’s own repair personnel when COTS are returned from the field. In short, status information should be available in all but the simplest COTS.

B. Transmit (T)/Receive (R)

Two modules communicating with each other can present a diagnostic problem. The source module, the target module or the interface between them can create a failure. It is difficult to diagnose which because all three scenarios result in a corrupted communication interface. To combat this diagnostic ambiguity it is helpful to have a separate transmit channel and a separate receive channel. This can often be achieved by serializing the transmission and receive signals, so that less interface signals are needed. Alternatively, a simple handshake protocol can be used to verify that the data transmitted by the COTS to the system is properly received.

The system designer should select COTS with separate transceiver signals. If both are not available, a separate transmit signal is usually more helpful.

For the COTS designer, a great deal of knowledge can be gained about the operation of the unit by having an independent channel through which data can be transferred. The redundancy can provide for fault tolerance and improve COTS reliability.

C. Signal Integrity (I)

Signal degradation, both in terms of signal levels and timing can be a subtle but devastating event that often results in intermittent, transient and hard to repeat failures. Measuring voltage levels and other parameters can improve fault detection and prognostics. Bit error rates and jitter can be evaluated using eye charts and similar monitoring means. Both digital and analog measurements can be communicated as well, providing signal integrity to end users. Input / Output (I/O) structures can have independent channels, so that the integrity of data transmission can be independently verified.

For the system integrator, it is important to have the COTS’s internal mechanism identify signal integrity issues. In most system environments it would be a logistic nightmare to bring in external equipment to make voltage or frequency measurements. In selecting high frequency COTS, attention should be paid to how the COTS deals with signal integrity issues.

The COTS designer can also benefit from embedded signal integrity measurements. Often the signals are highly sensitive to interfacing with external test equipment and the resulting measurements are distorted by lead effects and other interferences. The measurements available from embedded or built-in test offer a more accurate result.

D. Diagnosability (D)

Implementing testability guidelines may result in detecting the existence of more faults within a COTS unit, but many of these faults can still be ambiguous when it comes to identifying and locating which of several COTS within the system is in

fact faulty. COTS should be designed to distinguish faults that originate within their boundaries from faults of neighboring circuits. There are several ways to achieve diagnosability.

The system designer should consider the process that a diagnostic technician would undertake to find the root cause of a faulty system. He/she would likely want to disconnect modules and take them off line. COTS modules should enable one to accomplish that without the physical disconnection process. Some disable lines will tri-state the entire COTS interface and in selecting COTS, this feature should be given important consideration. Also, when COTS modules can be tested individually (off-line), they can yield perfect (unambiguous) diagnostic resolution.

The COTS designer should welcome the opportunity to separate his/her COTS from the entire system, which probably contains modules that could interfere with its own tests. This would prevent many Retest OK situations, which involve sending perfectly good units to the factory when something else in the system gave the (wrong) semblance that this particular COTS was at the root of the system failure.

One important question to answer is why COTS designers would make any effort to accommodate testability and diagnosability. Many military and avionics customers resigned themselves to poorly testable and diagnosable COTS as a fact of life. As a result few COTS purchasers have asked for testability features. With the STRID attributes, however, buyers can base their selection criteria on those COTS that provide more of these features and which ultimately reduce the support costs associated with the systems. With customer demand, COTS manufacturers will likely also find these and other such guidelines competitively advantageous.

IV. AN EXAMPLE OF COTS TESTABILITY ANALYSIS

An interesting example of COTS is a flying robot drone that is equipped with an on-board camera, as shown in Figure 2. This is an actual product and there is a demonstration of it on the web at [7]. The robot drone is equipped with a GPS and can therefore require less human commands to get it to the desired location.

Figure 2. The microdrones GmbH md4-200 Flying Drone

We will assume for the purpose of illustration that the entire drone is built from COTS modules and that Figure 3 is a close approximation of its block diagram. With that assumption in

mind, we will attempt to find COTS modules that can perform the required functions and still meet the testability and diagnosability criteria we set.

Figure 3. A Block Diagram of the Flying Drone consisting of COTS

A. System Level Testability and Diagnosability Guidelines

We will assume that the drones would be deployed in substantial quantities to serve as eyes in hard to access locations. A crash can be quite destructive and therefore it is essential that any faults be detected prior to deployment.

Prior to selecting COTS a testability analysis should be conducted by the system designer. This analysis will highlight issues specific to the design and will identify the information that needs to be obtained about individual COTS modules so that fault detection and isolation is optimized. This testability analysis includes the following considerations.

1) Organizational Dissemination

Each COTS should have some level of internal test. Built-in Self Test of each unit is essential so that system level test is limited to the interfaces. Connectors can come loose from normal or “hard” landings that can create diagnostic ambiguities. In fault isolation, therefore, connectors should be considered replaceable units and ambiguity between connectors and COTS modules should be eliminated during diagnoses.

2) Level of Repair Analysis

Repairs will include checking and reinforcing connections between COTS. When two interconnected COTS modules pass their own internal (self) tests, the only place a fault can still reside is in the interconnection. These repairs can be readily made prior to or after returning from deployment. During deployment, however, additional tests need to be run to ensure that the mission can be completed. For example, it will be essential to monitor the power source and make sure that the unit can be brought back to base before its power degrades to a level where it can cause a crash.

3) COTS Self Test Capabilities

While it may not be essential for each COTS to be able to test itself, some sort of status should be reported by each. Additionally, each pair of COTS should be able to ascertain that the connection is in fact faulty.

4) Failure Mode Effects Analysis (FMEA)

From the block diagram of Figure 3, it is possible to create a functional description of each block. That functional description can serve as a specification that COTS modules should attain or at least approximate. Additionally, specific failure modes should be assigned to each block. The transmitter could, for example, fail to transmit at all, or transmit an attenuated signal, or transmit a noisy signal. With each failure mode a system level effect can be predicted. The ultimate goal of the testability strategy will be to detect each of these failure modes, and the goal of the diagnosability strategy will be to isolate the fault to the faulty module. The repair will simply be the substitution of the COTS module found to be faulty (assuming of course that all connections have been verified).

B. Looking for COTS with STRID Attributes

In the selection of individual COTS (i.e. system design and integration) we wish to select among COTS candidates those that best meet the STRID attribute guidelines. To illustrate the process, we used a Google search to find some of the COTS modules. Then we refined the search to look for those that meet specific STRID attributes. We put special emphasis on status indicators. Here are our results:

1) Transmitter/Receiver Module (including Antenna)

Our initial search of Transmitter/Receiver (“Transceiver”) products yielded a wide range of products. Some were obviously too large and heavy to consider in the small drone and then price was also considered. We found that even after these filters we had COTS that were missing status indicators and signal integrity measurements. Our search yielded a candidate, the Eartec TD-900M - Full Duplexing Wireless Remote Transceiver. It contains “LED status indicators for signal flow monitoring” and a “low battery indicator.” This choice does provide Status, Transmit, Receive, and signal Integrity capabilities, meeting most, if not all the STRID attribute requirements.

2) Geo Positioning System (GPS)

We found the Dual XGPS 150 to contain several status indicators. In addition to the battery level indicator we expected, it also has a Bluetooth status indicator and a GPS status indicator. When paired with an iPad, iPod or iPhone the GPS status light will not illuminate until an app is actively requesting GPS information. Through these mechanisms STRID attributes are available.

3) Mass Storage Device

We found many low price hard drive devices that can serve as mass storage with 1.5T bytes or greater, but none of them had status information beyond the LEDs that indicate power on and similar conditions.

4) Camera Controller (usually including the camera itself)

While we could not find cameras with position status indicators, we did find The Marshall Electronics V-LCD-HI Camera Horizon Indicator. This is an adaptor that allows you to accurately position the camera for stable and level shooting.

5) Other COTS

The drone system contains other COTS modules, for which a similar search could take place. Main power, back up power, and the controller (PC or Mac based processor controller board) should be available with status information provided. It appears to be more difficult to ensure status information for the flight controller and the propeller as well as the engine.

In cases like this, when it is difficult to find COTS with certain STRID attributes, it may be worthwhile to have discussions with the COTS manufacturer. Often COTS contain internal status information that is not made public and used only for internal diagnostics. By voicing your concerns, the COTS manufacturer may well uncover that the features you are looking for are in fact already part of the COTS you are considering.

C. System Level Testability and Diagnosability Assessment

Having planned for testability requirements for the system, including fault detection, fault isolation, prognostics and other requirements, we carefully selected COTS that approach those requirements.

Next, we need to assess how well the system meets the requirements. To accomplish this, we would start with a FMEA created from the functional description of the modules identified in Figure 3. For each module possible failure modes are identified and an FMEA assessment is created. Wherever a failure mode is not detected or isolated, a testability or diagnosability issue exists. The system designer needs to reassess his/her choice of COTS and search for alternatives that can overcome the problem. If not, additional system design will be needed or alternatively the system’s failure detection and/or diagnosability compromised. Each of these compromises carries a cost, which should be traded off against higher cost COTS. While we do not suggest that the COTS approach should be abandoned, when all is considered, COTS may not be the most cost effective solution overall. If this is the case, it helps to know that at the earliest possible design stage.

V. SUMMARY AND CONCLUSIONS

Testability engineers have traditionally confronted fault detection and isolation issues by being part of the design effort. This has helped in creating designs that meet manufacturing testability guidelines, such as those with the PCOLA/SOQ attributes. Functional interaction between circuit boards and modules are measured by the FAM attributes. When we integrate pre-specified and pre-designed COTS modules, however, we lose the ability to influence those aspects of the COTS design that we need for achieving fault detection and fault isolation goals at the system level. With the five attributes

of Status, Transmit, Receive, signal Integrity, and Diagnosability (STRID) that are increasingly available in COTS, system designers can meet system level testability guidelines as well.

In Section II we discussed the system level testability attributes contained in military and commercial guidelines. In Section III we explained how each of the STRID attributes can lead to a more testable system. We argued that these five simple attributes can greatly enhance the fault detection and fault isolation of COTS based systems. In Section IV we applied these principles to an example COTS based system and outlined the steps that should be taken to analyze the system’s testability. In our example we used a Google search to find COTS modules that met our functional requirements as well as the testability requirements. In each instance we successfully found one or more COTS module that met both our functional and our testability requirements. We also used the search to reject other COTS that did not contain STRID attributes.

This paper provides guidance to two distinct groups. For the short term, we guided the system designer to select and procure COTS modules that already include STRID attributes in favor of those that do not. Our long term goal was also to communicate to COTS designers that certain testability features should be incorporated in emerging COTS designs. The same STRID attributes that system designers look for will qualify their products for a wider market. When system designers utilize these attributes in their selection, (or rejection) of COTS modules we will find that product manufacturers will face the need to create products that will feature STRID attributes. When that occurs, both COTS vendors and COTS users will capitalize on these five simple concepts and create testable, diagnosable and supportable systems, while maintaining the low cost advantages COTS modules offer.

REFERENCES

[1] US Navy, MIL-HDBK-2165 Testability Handbook for Systems & Equipment, July 1995.

[2] IEEE Standard 1149.1-1990 “IEEE Standard Test Access Port and Boundary-Scan Architecture,” New York, 1990.

[3] A.T.E. Solutions, Inc., The Testability Director v 3.2 software, www.besttest.com/OurProducts/TestabilityDirector/, 2005-2009.

[4] Surface Mount Technology Association (SMTA) and Testability Management Action Group (TMAG), SMTA and TMAG Testability Guidelines 2008TP-101D, SMTA Minneapolis, MN 2008.

[5] L.Y. Ungar, “Economic Evaluation of Testability and Diagnosability for Commercial off the Shelf Equipment,” Proc. AutoTestCon, Sep 2010., pp. 1-5.

[6] K. Hird, K. P. Parker, Bill Follis, “Test Coverage: What does it mean when a Board Test Passes?” Proc. IEEE International Test Conference, 2002, pp. 1066-1074.

[7] microdrones GmbH, Demonstration of md4-200, http://dvice.com/archives/2007/05/flying_robot_keeps_the_streets.php, May 2007.

[8] L. Y. Ungar, “An Economics Model of Supportability Through Design for Testability,” Proc. AutoTestCon, Sep 2006, pp. 74-79.

[9] S. Davidson and L. Y. Ungar, “A Framework for Testability Metrics Across Hierarchical Levels of Assembly,” Proc. AutoTestCon, Sep 2009.