System Models Effect on Test Integration Cost

Hugh A. Pritchett

Analysis, Integration and Design, Inc. (AIDI)

2905 Bush Dr.

Melbourne, Florida USA 32935

hpritchett@aidinc-usa.com

Abstract— Models and paradigms imposed by test development and support applications impact test program integration costs. One major cost component of test development and test rehost efforts is integration. The test system representations engaged to develop and design test applications are a primary factor that drives how much engineering effort is expended during integration. Achieving test platform independence for test applications has been a high priority and ongoing focus for the test community. Test platform independence has a distinct impact on, and a close relationship to, test integration. Much of the effort spent on platform independence thus far has been involved with how to specify test requirements and test programs so that they do not, either directly or indirectly, demand specific test platform resources or implementations. Even with strict adherence to guidelines meant to maximize platform agnostics, significant effort is still required to adjust test program software during integration to accommodate test system, test software, and test adapter characteristics that were not anticipated or compensated for. Typical integration efforts involve altering parametric values used to invoke source instrumentation, imposing test limits, defining test timing, and other test program aspects. Although many of these test program aspects are the same as those used to provide platform independence, they must be altered to accommodate specific test systems during integration. This apparent paradox where platform-independent information must be altered to accommodate specific platforms is one consequence that can be explained by evaluating the system models and paradigms used in test development and test architectures that affect integration efforts.

Keywords- migration, rehost, intrinsic, switching, path

I. INTRODUCTION

Presented are the models and paradigms that have been identified during a system evaluation effort using integration impact as the selection criteria. Discussion is provided for each model, how they affect integration, and how test requirements might be made more platform-independent if higher fidelity models or different implementations were employed.

One result of the investigation is a recommendation provided to the IEEE Standards Coordinating Committee 20 (SCC20). The recommendation was to provide better insight into how to create transitory paths and that additional system information might be needed. At the time of this writing, the SCC20 steering had voted to approve a new Project Activation Request (PAR) for instantiating a best practices guide for employing Intrinsic Path Characteristic Information (IPCI).

The committee’s Hardware Interfaces subgroup has considered the technical content of the recommendation and consolidated a proposed list of information that should be part of what is addressed. The decision concerning how the information will be implemented in the standard is currently in discussion. Identified by the IPCI recommendation is the observation that system models and paradigms related to system signal paths, which have been embraced by the test community, are oversimplified and an impediment to achieving a wide variety of more robust and better automated test-related products. Specifically, the investigation that produced the recommendation identified system models that assume perfect conductance, instant closure, no contact resistance, no insertion loss, and other perfect world suppositions. The investigation found that imposing more rigorous and higher fidelity models would require more detailed and programmatic access to specific IPCI. This finding became the impetus for the recommendation to the SCC20. The investigation found that the oversimplified models identified affect all signal path-related component representations including switches, cabling, termination, connectors, circuit board traces, and others found in Automatic Test Systems (ATSs), test adapters, Units Under Test (UUTs) and other ATS-related components. Furthermore, the investigation established a clear link between using oversimplified system models and test integration costs. Another result of the investigation has been the identification of more than a few new injection points for employing the new information. The investigation results postulate that the availability of higher fidelity models enabled through use of the IPCI will cultivate lower cost as well as better understood systems.

At the outset, the primary goal for obtaining the standardization of this information is to unleash the potential for implementing software components based on higher fidelity models that employ the more robust and standardized information set. To provide detail in the area of software components, a discussion on one such component, a path allocator, is provided. The discussion includes what path allocation is, ways path allocation is accomplished, and improvements that can be achieved with the use of intrinsic path characteristics during path allocation. Examples will show how path allocation is accomplished, how certain aspects of path allocation can encumber test application developers, and how issues related to allocation itself induce integration costs.

978-1-4244-9363-0/11/$26.00 ©2011 IEEE

Besides TPS transportability improvement, the investigation has identified that TPS integration cost reduction is an additional primary area of focus. In support of the TPS integration cost discussion is a detailed look at several aspects of test integration that can occur due to low fidelity system path models and paradigms. The examples given are real world lessons learned from test rehost integration efforts that provide insights into often unanticipated anomalies and insidious issues that arise during test migration efforts and how they can be alleviated if intrinsic path characteristics are considered. The discussion details aspects and considerations related to paths and how they are determined during test integration and test rehost integration efforts, including a discussion on Interface Test Adapters (ITAs). The ITA discussion establishes that how and when ITAs are implemented can greatly affect integration costs.

This investigation supports work done in other forums that shows that test definitions should not be used as test applications since adjustments that occur to compensate for real world circumstances during integration are not changes in test requirements. The test application should be adjusted to accommodate these changes, but the test specification should not be altered as a result of any such changes. The work supports having a mechanism for accessing the test’s definition that can be used to generate the test applications. The differences produced in parametric values and other application details between the applications and the test definitions are most often the products and issues that test integrators must deal with. The issues are also the ones that will become the automation targets that using IPCI will address. Two other aspects related to intrinsic path characteristics that are discussed are connection and closure. As defined by the Resource Adapter Information (RAI) rule set, test definitions should not specify signals in terms of closure or connection but only UUT location. So for platform independence, test requirements should not mention connection or closure but, as a practical matter, test implementations must specify them. Test integrators and rehost practitioners regularly encounter issues with new and legacy applications due to connection and closure issues during integration. During application rehost, the issues typically occur due to differing system implementations between legacy and replacement test platforms. During new development, the issues typically occur due to misunderstood implementations, incorrectly specified switch assets, or incorrect resource requirements. One commonly occurring example that is often seen during rehost integration is when legacy applications invoke closures and connections with no specification of why the action is performed. Essentially the legacy application creates a transitory path without any definition for why it is needed. In this situation, the rehost practitioner is left with the dilemma of resolving why the closure is needed, and then designing what can be equivalent functionality on the target system. Most ATLAS dialects allowed this type of closure, even though it breaks test application transportability, which was intended to be a primary benefit of using the language. All DoD ATLAS dialects allow this practice and most DoD TPSs encountered by the investigation team employ the use of this misguided facility. Presented with this discussion is a real world example

where this exact connection issue was encountered during a rehost effort and why, even when the target platform was specifically built to have equivalent assets, the resolution for providing the same switch closure in the test application implementation was not sufficient to complete the integration. Also shown is how employing the use of intrinsic path characteristics could have resolved the issue before integration had even occurred.

Critical to IPCI is the subject of return path and reference specification. Existing test application implementations and test specification methodologies do not tend to deal with return paths and references in a direct, explicit way. Because the information is not explicit and not required, it is difficult or even impossible to relate how signals are coupled or referenced. This issue becomes a critical one during test integration. The problem is a result of the system model that assumes perfect coupling and implied reference. Along with a more detailed discussion, an example of how this issue affects integration efforts is provided. Also discussed is information related to test application strategy and active ITAs used with test programs and test system self-test ITAs. This section is focused on modeling decisions that must be made before test application development and provides recommendations for how to use IPCI in their development and deployment. Since these types of ITAs are typically path intensive, often have switches, and can be viewed from an application stand point as either UUTs or system assets, it is critical to develop well-defined system models to support platform neutrality and reduce integration impacts.

To summarize, this paper provides insights into how IPCI can be employed to reduce test application integration costs which is a major cost facet in the test application lifecycle. Various application aspects, implementation situations, and frequently occurring nuances are discussed. Results and real world examples are provided as support for the statements that are made. Insights are also provided into the forthcoming IPCI best practices guide that is to be produced in the form of a new IEEE standard.

II. MODEL SIMPLIFICATION IMPACTS ON TEST

INTEGRATION

Inherent in the design philosophies of most test systems and test system software available today are models and paradigms that restrict their capabilities to deliver high fidelity, high accuracy test requirement manifestations without significant developer integration and application rework. The difficulties arise due to over simplification of intrinsic, physical, electrical aspects of path related test system components. The following subsections itemize target areas we have identified where simplification exists and where test application integrators could benefit if they started with test programs that were generated using higher fidelity models that utilized IPCI. A few of these topics are provided with more detail to highlight the considerations that might be made at test program development time if IPCI were considered.

A. Zero Resistance When Closed.

This assumption is probably the most well embraced simplification made in test system paradigms. Users assume switches have no resistance and often leave the resolution of the signal degradation that is caused by ignoring the impedances to the test integrator. When the test does not provide the desired signal to the UUT the integrator is tasked with analyzing the path and implementing compensation which can be an onerous task depending on the test and test platform complexity.

B. Infinite Resistance When Open.

Users assume that when switches are opened and/or connections are removed that there is no signal present. When signals of sufficient frequency are present on paths within close proximity transmission affects can impose unexpected signals on open paths. Integrators are again tasked with analyzing their test adapters, cabling, and test system implementations during their efforts to resolve these side effects.

C. Complete Isolation From All Other System Paths.

This topic is similar to the one described in Section II.B. The difference is that paths may not necessarily be open to encounter the transmission affects again leaving integrators with analyzing their test adapters, cabling, and test system implementations during their efforts to resolve these side effects. As an example of a higher fidelity model we might apply during test application development, Figure 1. has been provided.

Figure 1. Higher Fidelity Signal Path Isolation Model

In Figure 1. the Alternating Current (AC) source provides a signal on a path #1 with some inherent capacitance to ground C1. Another signal path, in this case signal path #2 will see noise related to the signal on signal Path #1 due to capacitive coupling C3 between the 2 signal paths. This model is slightly simplified to represent signal to load capacitance and inherent path to ground capacitance as a single entity, C2. The capacitive coupling C3, enables a current that produces a voltage Vn, across the load provided by C2 and R1. This Vn is often referred to as noise but in our analysis we prefer to view it as uncertainty that is essentially added to the signal on signal path #2.

As with most models there are assumptions. For this situation we are considering the circuit as a typical and practical small signal model. In practical consideration it can be shown that when R is much less than 1/jω(C2 +C3), which will nearly always be the case, the Vn will approximately equal jωR1C3Vs. Armed with the ability to determine a Vn an application could determine a potential path is within a measurement tolerance and proceed if it is or deny if it is not. In addition a user might decide to affect the Vn by altering their Load R1 or selecting a path that is further away thereby affecting the intrinsic characteristic C3. For this example then the intrinsic path characteristics that might be important could be a distance between paths, or a capacitance value that reflects the difference. Also a parameter for any inherent path loading might become important. Also there are rules of thumb that might be applied that could produce other intrinsic characteristic requirements. One such rule of thumb is that cross talk is reduced by half power for every conductor diameter distance that conductors are away from each other. If this rule of thumb were to be applied then the conductor diameter might be a good intrinsic characteristic to capture. Clearly the C3 is affected by the proximity of the 2 conductors and the rule of thumb mentioned is simply a quick way to accomplish the analysis without the more rigorous math. A model might use one or both of these analysis methods to make determinations needed for specific ATS applications. The goal would be to use such a model in an automated fashion at test application development time using the more detailed system knowledge that IPCI would provide to reduce the costly on station integration time.

D. Isolation from control circuitry.

Offset currents and other effects from control circuits in switching and other instrumentation are typically ignored during test application development. During integration developers can find that their expected signals are impacted by these issues and must provide compensation after investigating and isolating the causes. Availability of these values at test application production time could automate compensation for the effects of control circuitry. The compensation could be accomplished by adjusting source or sink signals for the imperfect isolation or simply by allocating alternate resources with more suitable characteristics for the situation.

E. No settling time.

Test programmers typically assume that switches settle instantaneously after they have been commanded to close. In some environments this assumption proves satisfactory as switch speeds and software overheads sufficiently dampen the ability to provide signals quickly enough in the first place. As systems are modernizing they are incorporating faster switching and software overheads are imposing lesser impacts. Integrators find that legacy code that ignored the timing constraints is no longer operational and must then investigate and provide explicit delays to satisfy instrumentation timing constraints.

F. Perfect connection.

It is most often the case that test programs assume that when connectors are engaged with their mates that they have no impact on signals that use the paths that the connectors provide. In addition, the actual bindings that the connectors use to attach to wires are also not considered although for certain scenarios the impedances that they infer can be significant. Integrators are then left with not achieving the fidelity they expected when defining their test definition. The integrators must then complete a detailed characterization of their ITAs, test systems, etc. to resolve their dilemmas.

G. Infinite lifetime.

Since in general developers ignore connection, closure and all the previously discussed issues, another result is that they do not consider the degradation that occurs over time with these items. Once developers integrate test applications they may find that overtime issues again arise as normal system wear transpires. Integrators must then again characterize their systems to determine the offending items that have impacted the application functionality as normal wear has been incurred.

H. Implied instrument limitation knowledge.

Test programs can be constructed without regard to the test assets real capabilities. Ignoring the instrumentation limitations can be destructive. Without definition of instrument parameter boundaries and insistence on imposition of the information during application development, Integrators can be faced with catastrophic failures on UUTs, test adapters and test system assets. Failures of these kinds can be elusive to identify. Even when they are obvious they cause integrators down time and repair impacts.

III. TEST DEFINITIONS VERSUS TEST APPLICATIONS

Traditional test migration is initiated based on test programs that have been adjusted, debugged, enhanced and altered from their original states for various purposes and reasons. Typically these programs have been configuration managed but often there is no mechanism for determining the changes that have been made since inception or why they occurred. Essentially the programs have been morphed to adjust to the environment in which they are employed. Although these programs might have originally been written to depict the target UUT’s test requirements, they have since been altered to accomplish some level of performance on specific rehost platforms. Because in this scenario the original test requirement is essentially lost, the migration effort is complicated with the engineering task of reformulating the actual test requirement through software, test system and circuit analysis. This developmental and maintenance model clearly imposes costs during integration when TPSs are moved, migrated or retargeted. Figure 2 provides a graphical depiction of this scenario.

Figure 2. Traditional Developmental Model

A new developmental model can help alleviate the burdens traditionally encountered during integration by maintaining the test definition as configuration managed source information. Incorporated in this model, depicted in Figure 3, would be an automated way to concoct test programs applicable to specific test systems from the test definitions. Using this approach at rehost time, practitioners would use the automation to produce new test programs for their target test systems and would be enabled with starting point test programs that are based on the true test requirements. These test programs could also be configuration managed in association with the platform for which they are targeted. In this way integration impacts due to prior test system related adjustments are eliminated and integration time is reduced.

Figure 3. Integration Impact Reduction Model

IV. REFERENCES, RETURN PATHS AND SHIELDING

One difficult and sometimes elusive aspect of test application rehost and migration is the assumptions made concerning references, return paths and shielding. These system and test application related facets are almost never explicitly defined or identifiable in test programs that employ them, even worse the system models that are available do not provide obvious ways to define the existence and presence of these system aspects. Worse than implication, the test applications and system tools almost always just assume references, return paths, and shielding.

A clear example of this issue is when ATLAS programs ask to measure analog signals. The implementations often specify “GND” or ground as the low signal connection point. What test migration practitioners encounter is that there is no specific information that defines exactly what “GND” means. For instance, it might be the station’s system ground, the ITA’s chassis ground, the UUT’s ground plane, or others. The original developer depended on station knowledge and

ATLAS implementation to know what was getting connected, referenced, isolated, etc. When migration practitioners attempt to move test applications they encounter different station implementations, test assembly configurations and UUT connection methods that are not the same as those on the original station. Since there is no explicit information available in the test applications that could be used by migration automation software, practitioners are left with the original assumptions when attempting to move the applications. During integration of those migrated programs issues with measurement accuracy, unexpected signal behavior, and other impacts are incurred, even to the point of catastrophic UUT or ITA failures induced due by the differences.

Figure 4. Legacy Test Scenario

Figure 4. shows an example legacy test station where an oscilloscope is connected to a UUT using a switch configuration that closes 2 switches with a single closure, one that is used for the signal and the other for the reference. In this example the UUT ground is used as the reference but there is no indication that the ground is connected elsewhere.

Figure 5. Target Test Station shows a scenario where the test application is migrated from the legacy station to a new target station. This station implementation is slightly different than the legacy station in that there is an instrument whose reference is the station ground. Also the switch that is employed ties all the references together when they are closed. These aspects are easily overlooked by migration practitioners who enter their integration efforts without immediate knowledge of the reference implementation of the scope reference connection to the station ground and the UUT ground to the UUT ground. Depending on the test being accomplished the integrator might find he has burned parts on his UUT, gets an out of range measurement or even damages a test station instrument or switch. Detailed analysis by the migration engineer might expose this dilemma but at the cost

of extensive analysis, review, and experience which are all costly elements. A new proposed objective for standardized intrinsic characteristics is to define the station in a way that allows us to discover these type issues using programmatic analysis, relieving the practitioner from the discovery. There are other more complex and equally difficult to troubleshoot issues that arise because of non-existent or completely assumed reference, return path and shielding.

Figure 5. Target Test Scenario

AIDI has witnessed this and other related grounding and

referencing issues as described in the example first hand in a

recent TPS migration effort. The legacy test program required

the application and measurement of several signals at the UUT

with reference to “GND”. However, the legacy test program

does not clearly specify if the references are UUT ground,

system ground, or ITA ground. This particular test program

expects the UUT to be isolated from the test system during

certain tests and tied to the same reference potential as the test

system during other tests. During a specific subtest, a voltage

measurement was performed across two UUT connection

points. In the legacy test station, an oscilloscope was used to

make the measurement across these points. The original test

requirement called for the UUT to be isolated from the test

station and floating in relation to the test system ground.

Since the reference was implied by the legacy test program,

referencing system ground instead of UUT ground during path

allocation caused the test measurement to fail at runtime.

Furthermore, because of the electrical configuration of the

components on the UUT and since the UUT was not floating

as expected during execution, damage to the UUT was very

possible. An engineering analysis of the failure during test

program integration uncovered the discrepancy and UUT

damage was avoided. Since the expected references were not

clearly defined in the legacy test program, the issue was

witnessed and resolved through an exhaustive engineering

analysis during TPS integration. The efforts during integration

unnecessarily tied up valuable system and engineering

resources and could have been mitigated if intrinsic path

characteristics were clearly defined in or available to the

legacy test program.

V. TEST PROGRAM HARDWARE DEVELOPMENT

Test application development strategies typically address test adapters in the early stages of development. This means that the resources that will be employed by the test applications being developed are essentially selected prior to software development. The resources are selected due to the wiring specified in the adapter designs that tie UUT connections to specific test platform Interface Connection Assembly (ICA) pins. The most general case is that ICA pins only provide accessibility to specific instruments and often only one specific instrument. Significant engineering effort that requires detailed test system analysis is accomplished to create these adapter designs. In addition when integration is accomplished engineers often encounter issues related to the adapters that must be analyzed and resolved. Often, the resolution requires reworking the test adapters. The rework can be due to referencing and grounding implementations, improper path capability or design, incorrect resource selection and other issues.

A new development model where users produce their test definitions initially and then use automated tools that employ IPCI to develop adapter designs could alleviate engineering spent before application development and during application integration.

VI. AN EXAMPLE IPCI AUTOMATION TARGET

To demonstrate a potential target for employing IPCI a proposed automated compensation facility for a test generation capability that could be employed in a test program migration and development environment is discussed. One of its major facets is enabling test developers with enhanced capability to define test platform agnostic test information representations. The test information includes definition of signals that are to be applied to connection points on a unit under test (UUT) and definition of signals that are to be measured at certain connection points on a UUT. Once the test information set is developed, test station resources are allocated to perform test information signal requirements against well defined and well formed ATML instances developed to emulate specific automatic test systems. Also, signal paths that provide electrical connection between test station resources and target UUT connection points are allocated. Following the allocation process, executable program test code is automatically generated by composing predefined and integrated code snippets into skeleton test program code. The result is an immediately compilable test program in a target test language that a test developer can introduce onto the target test platform and rapidly integrate. During code generation,

signal levels are assumed to be ideal, where path losses due to switching and test system wiring are treated as negligible.

The following simple example illustrates a scenario that is commonly encountered in a typical test program. Figure 6. shows the electrical configuration for a particular test scenario. A DC power supply is allocated to provide a 50Vdc signal across two connection points on a UUT. The voltage applied ideally results in a 1A current draw by the UUT. Signal paths are also allocated to provide electrical connection between the resource ports and the UUT connection points. In this example, the paths selected inject two power switches and two 3-ft segments of 12AWG system wiring into each path between the DC power supply ports and UUT connection points. As the illustration depicts, the signal losses introduced by path components are usually ignored during test information development.

Figure 6. Test Environment Assumed Allocation

In reality, the system wiring and power switching injected into the signal paths introduce signal degradation due to imposed component impedances. Figure 7. quantifies the actual impedances of the switching and wiring and its effect on the actual current draw by the UUT in this example.

Figure 7. Actual Test Environment Allocation

Figure 7. shows that the actual current draw taking into account path loss is approximately 0.943A. This equates to approximately 5.7% less current draw than is what is required by the UUT. These path losses are typically discovered during test program integration and accounted for by a test program developer in the test information itself. The test developer adjusts the test requirement in the test information accordingly, inherently injecting implied test system knowledge into the test information itself. The test requirement no longer represents what is needed at the UUT. The test requirement therefore becomes a test platform specific test requirement. Figure 8. shows how such an adjustment would be made to the applied DC signal during integration to obtain the expected voltage drop and current draw at the UUT.

Figure 8. Integration Adjusted Test Application

Utilization of a standard for the definition of intrinsic path information will allow for automation in compensating for path related phenomenon into test applications as they are developed. The compensation could occur during or after test system resources and paths are allocated. The implementation of test definitions that only address signals needed at UUT connection points facilitates this objective. The allocation mechanisms are one place where automation could account for the path characteristics. Another objective of this investigation is to develop the technical knowledge set needed to produce a Universal Switching Component (USC). The component could analyze the test information and intrinsic path characteristics provided by ATML instances that contain the information. Envisioned is a USC that could provide an optimum path that minimizes switching, assists in maximizing fidelity, and optimizes path allocation facilities. These facilities would be employed in an automated way when test applications are generated from test definitions. The product would be test applications that not only provide compensation factors in the test applications but also identify where trouble spots reside in the applications thereby injecting documented insights for Integrators as they complete the TPS development process. Once implemented, the integrators will see a stark contrast between getting insights automatically versus having insidious issues passed down to them. In this scenario an automatic test program source code generator could use the adjusted signal levels during application generation and the test developer would integrate the test program with the adjusted values already present. Figure 9. illustrates the approach.

The strategy greatly reduces the test program integration effort associated with commonly occurring manual test program adjustment associated with path related issues. It also isolates the test developer from requiring test system knowledge throughout the test definition design process and allows the developer to instead focus on what is needed at the UUT instead of how a test system resource is to be implemented on a specific platform. This paper provides insights into some of the easy that IPCI can be employed to reduce test application integration costs. These costs are a major driver to overall TPS development and migration costs. The goal is to use the IPCI to help automate a variety of test applications development tasks associated with new development and migration instead of driving the hard issues down to the integrator.

Figure 9. Automation Target

ATML instances that embody the IPCI could be required at procurement time to guarantee the information availability. Once IPCI is being employed and a USC is invoked these issues would be mitigated prior to test program integration. The additional analysis and troubleshooting that occurs during the TPS integration phase could then be greatly reduced

REFERENCES

[1] Switching Handbook, A Guide To Signal Switching In Automated Test Systems, Fourth Edition, Copyright 1987, 1989, 1995, 2001, Printed July 2001, Cleveland, Ohio U.S.A.; www.keithley.com.

[2] Wiley Electrical and Electronics Engineering Dictionary, Steven M. Kaplan, 2004; http://ieeexplore.ieee.org/xpl/bkabstractplus.jsp?bkn=5273107.

[3] RAI’s role within IEEE 1641 and ATML. Published September 2008; IEEE AUTOTESTCON 2005 Proceedings.

[4] Key Success Factors for Building an Effective Interoperability Test Program, Published September 2001; IEEE AUTOTESTCON 2001 Proceedings.

[5] IEEE Std 1671™-2006, IEEE Trial-Use Standard for Automatic Test Markup Language (ATML) for Exchanging Automatic Test Equipment and Test Information via XML. Institute of Electrical and Electronics Engineers, Inc. 3 Park Avenue, New York, NY 10016-5997 USA. Published 15 December 2006.

[6] IEEE Std 1641™-2004, IEEE Standard for Signal and Test Definition. Institute of Electrical and Electronics Engineers, Inc. 3 Park Avenue, New York, NY 10016-5997 USA. Published 25 March 2005.

Automatically

Generated Test

Program Code

(With Adjusted

Signal Values)

Automatic Test

Program Code

Generator

Test Platform

Agnostic

Test Information

ATML

Instances

Intrinsic

Characteristics

(Enhanced ATML)

+

Universal

Switching

Component

(Signal Level

Adjustment)