Integrating Developmental and Operational Testing

© 2009 The MITRE Corporation. All rights reserved.

Integrating Developmental andOperational Testing

Dr. William (Dave) Bell2 December 2009


Agenda/Overview

■What/Why Integrated Testing?■When Should We Do It?■What Can/Should Testers Do?■Summary

2


Recent DoD Integrated Testing Policy

3

Integrated testing is the collaborative planning and collaborative execution

of test phases and events to provide shared data in support of independent

analysis, evaluation and reporting by all stakeholders particularly the

development (both contractor and government) and operational test and

evaluation communities

T&E expertise must be brought to bear at the beginning

of the system life cycle to provide earlier learning about

the strengths and weaknesses of the system under development


Integrated Testing Defined

4

■ Defense Acquisition Guidebook, Chapter 9– Whenever feasible, DT&E and OT&E events should be combined, if that supports technical

and operational test objectives to gain the optimum amount of testing benefit for reasonable cost and time. The user community should be involved early in test planning to ensure the statement of desired capabilities is interpreted correctly and tested realistically. Certain events can be organized to provide information useful to developmental and operational evaluators and lend themselves to the DT and OT approach. The concept is to conduct a single combined test program that produces credible, qualitative and quantitative information that can be used to address developmental and operational issues. Examples of this approach include: combined DT/OT events or piggybacking an operational assessment onto a developmental test. Likewise, developmental testing data requirements can be accommodated by an operational test. This approach can reduce the time and expense of conducting dedicated OT events that replicate DT events, or vice versa, yet still provide adequate technical risk reduction. The developmental and operational testers can develop a test management structure to share control of the combined events. Combined DT and OT events and test data requirements must be identified early to prevent unnecessary duplication and to control costs. It is important that neither the DT&E nor OT&E objectives are compromised in designing combined events.

Combined DT & OT eventssupport technical and operational objectives

User community involved early to ensure desired capabilities

interpreted correctly and tested realistically


DT/OT to IDT/OT Evolution - Past

5

Contractors

Responsible Test Organizations

Find & fix technical problems

Operational TestAgencies

Director, OT&E

Discover ops issues& confirmation

Developmental Test

Operational Test

Certification


IntegratedTest

DT/OT to IDT/OT Evolution - Present

6

Developmental Test

Operational Test

Certification

OTAs

DOT&E

Finish discovery &

perform confirmation

Ktrs

RTOs

Find & fix technical problems

KtrsRTOsOTAs

DOT&ECertification

Discoveryof ops/tech

issues

Ktrs ContractorsRTO Responsible Test OrganizationOTA Operational Test Agency


DT/OT to IDT/OT Evolution - Future

7

Developmental Test

Operational Test

IntegratedTest

OTAs/DOT&ETitle 10 confirmation

Final certifications

KtrsRTOsOTAs

DOT&EJITC

Develop/refinerequirements

Discover/fixops/techissues

Contractors/RTOs

Early technologydemo & selection

WIN - WIN: Capability in warfighters’ hands ASAP

Reduces cost, schedule, and risks


Agenda/Overview


8


Testers Informing Early Decision Making…

■ Requirements/specification impact– Disciplined testing

mentality helps discovery process; prior to large funding commitment

9


Integrated Testing - Part of Systems Engineering


12

ComponentTesting

• Unit testing• CI testing

Stand-Alone SystemTesting

• CI-to-CI testing

System-of-SystemsTesting

• Field testing• Exercise

Integrated SystemTesting

• I&I testing

FQT I&ICert

OTCert

Contractor DT

Government DT

Government OT

Accreditation andCertification Testing

Accreditation andCertification Testing

Integrated Testing Environment

• ••

Integrated Testing Execution Period


Agenda/Overview


13


14

During Requirements Development

■ Early T&E influence during requirement development :– Ensures requirements are testable and unambiguous

■ Can you readily imagine a verification technique for this requirement?■ Your requirement may be ambiguous if interpretation is required to

design and build – Ensures that the test measures address/answer the Critical

Operational Issues– Requirements that are high risk/push the state-of-the-art are

identified/addressed

Getting the requirements right is a pothole-filled road


T&E During SOW, RFP, and Contract Development

■ T&Eers ensure that test-related scheduling, budgeting, infrastructure, other resources and personnel are included in program’s contractual documents– SOW/RFP, proposal evaluation, contract

■ T&Eers help answer the question, “How much and what type of data need to be collected?” – To inform early decisions– Assure continuing involvement

■ Test design– Test events /Operational/Mission scenarios– Distributed/Joint/LVC– Statistical sample size determination– Analysis planning

15


Integrated Test Execution

■ Early, continuous IT serves both DT and OT objectives – Provides adequate test time ensuring all test objectives are met – Allows for better/more thorough data collection– Allows PM’s an early look at and time to fix operational issues

■ Provides opportunity for assessment of reliability growth– Maintenance activities on early test units yield valuable

information■ Initial maintainability assessments■ Evolve maintenance approach or design at this relatively

inexpensive stage

16

Earlier insight = cheaper & easier fixes


17

Opportunities for Integrated Testing■ Integrated DT/OT supported in SOW and RFP■ Integrated DT/OT supported in Test Strategy and Planning documents■ Requirements unambiguous and testable■ DT&E and OT&E personnel working together from Program’s outset ■ Contractor and Government DT Data augments OT&E data■ OT&E personnel influence Operational Realism of DT&E scenarios■ Test Planning Working Groups develop common data collection and reporting formats ■ The DT&E System Under Test (SUT) reflects Operational System and is operated by typical/actual users■ OT&E personnel participate in DT Testing and/or receive test documentation■ OT&E personnel assist Program Managers to develop “Acceptable Work-arounds” mitigating serious deficiencies


Recommendations for Integrating DT/OT

1. Build a culture of Mutual Trust and cooperation ■ Demonstrate to PMs OT is not simply “Pass or Fail” ■ Assist PMs in lowering overall program risk by providing

SE and T&E involvement early in the development process2. Develop a “Partnership” not an adversarial relationship

between the Test Community and Contractors or PMs■ Identify ways Integrated DT/OT is of mutual benefit to

PM’s, Contractors and OTA’s■ Demonstrate how early detection of deficiencies reduces

overall risk and cost ■ Show how schedule test time can be reduced by

integrating DT/OT■ Assist PM’s in developing and presenting an acceptable

case for correcting deficiencies to the MDA


3. Encourage adoption of Common data Format/Standards for all Joint and Distributed■ Emphasize need for ability to exchange information in the

TEMP as part of Integrated testing■ Include requirement for common data standards/formats in all

SE and T&E Instructions and documentation■ DOT&E and OTAs encourage Test Planning Working Groups to

develop a common standard for data ■ Utilize professional organizations like ITEA and NDIA to

support the necessity of common data standards/formats

Recommendations for Integrating DT/OT

Page 19


Agenda/Overview


20


Integrated Testing Is Critical

■ Combines the engineering expertise of the developer and SE with the operational expertise of the user and operational tester

■ Ensures that requirements are written correctly■ Ensures all relevant testing is planned and budgeted■ Takes advantage of all available test data to inform

acquisition and operational decisions■ Accelerates deployment of mission capability to the

warfighter

21

Documents

Integrating Developmental and Operational Testing