A Case Study in Acquiring Robotic Capabilities Inherently a System of Systems Activity?

A Case Study in Acquiring Robotic CapabilitiesInherently a System of Systems Activity?

DeWitt T. Latimer IV, [email protected]

USC Center for Systems & Software Engineering

http://robotics.usc.edu/~dlatimer

http://csse.usc.edu

USC-CSSE Convocation 25 Oct 2006

The views expressed in this presentation are those of the author and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government.

http://robotics.usc.edu/~dlatimer

http://csse.usc.edu/

Outline

• Overview• Case Study Environment• Acquisition Program Goals• Surface Assessment Robot• Root Cause of Acquisition Failure• Summary

Overview

• Describe the acquisition environment• Examine the acquisition goals for a straight-

forward robotic system• Describe the system as it was developed• Identify a root cause of the acquisition failure at

the transition to operations stage• Discuss if there are inherent system of systems

aspects of straight-forward robotic systems and need to generate case studies for application of knowledge

Overview

Case Study Environment

• Based on completed robot built by Carnegie Mellon University graduate students and staff

• Acquiring commercial company was very familiar with subcontracting, sponsoring research projects, and a had long history of successful acquisitions with CMU


Diffuse Management

The business units involved in client organization

No senior level involvement, business units essentially autonomous

QA office provides mechanism for handling deviations

GMs manage delivery of roadways and utilize inspectors


VP R&D(Acquirer)

Quality Assurance

Office(TQM/6-Sigma

Champion)

Individual ProjectGeneral

Managers

Acquisition Program Goals

• Proposed system to:– Assess suitability of roadway subsystem by

identifying deviations from required profile– Automate measurement function performed by

aging senior field engineer– Ensure Return on Investment (ROI) achieved in

single roadway construction project– Ensure system can be delivered in time to utilize

on upcoming project– Provide system that increases the technical

prestige of the company in delivery of systems that include roadway surfaces

– Continue successful relationship between company and CMU

Acquisition Program Goals

System History

• Developed by CMU team (I was a member)

• Met all technical requirements– Autonomously detected and marked high/low

variations of a road exceeding 1/8” in 10 feet (~1 part per 1000)

• Met all business objectives– ROI: 100 to 1– Paid for R&D investment in demo project

• Currently sits on a shelf (in a bay actually)

Surface Assessment Robot

Development Activities

• Requirement to mark all deviations against profile specification handled by 2-pass autonomous detector/marking system– Driven by human operator for safety concerns


• Developer engaged significantly with the client's six-sigma organization to ensure appropriate deviation handling and recording

• Recast field engineer's role to be to review collected data to disposition deviations

Desired Profile

Actual Profile

Lower Limit of Deviation

Upper Limit of Deviation

“Low” Region of Deviation




• Developers noted system requirement for roadway was in terms of ride quality, not road roughness

• Proposed quarter-car model to characterize deviations before marking as either impacting system requirements or being insignificant– All deviations to roughness specification would

be automatically catalogued and those not impacting ride quality would be classified as no rework needed

– All deviations could be audited and reviewed



• Client's staff engineers rejected, indicating that all engineering decisions must be made by humans – field engineer to classify all deviations

• Development continued and delivered system on-time

• Robotic system demonstrated on actual program track with client and roadway subcontractor representatives


Final Roll-Out

• System marked several orders of magnitude more deviations than expected, surprising as subcontractor was the best-qualified and had a track record of low/zero defect roadways


– All deviations were verified by senior engineer, however the engineer indicated that most deviations were so minor as to have no system impacts and would have been ignored – counter to the QA culture!

End Game

• After final system demonstration, system was transitioned to customer

• No maintenance role for CMU was desired by either party (this was an upfront requirement)

• We later learned that the system was never transitioned to regular usage, indeed was not used on any future project


Candidate Root Causes(but not the actual ones)

Acquisition Requirements Development (RD)– All top level requirements were present, roughness was the right

technical requirement for contract enforcement reasons.

Developer Error– System met and exceeded all technical performance measures,

cost targets, and return on investment parameters

Human-Robot Interaction Design (HRI) factors– Operators in various interface walk-through events and field

tests were very satisfied

Acquisition Verification or Validation (VER/VAL)– How to perform VER/VAL on requirements at an acquisition level

if no engineer has experience to indicate what analysis would be needed? See next slide...

Root Cause of Acquisition Failure

Proposed Root Cause

Constraints did not conceive of the robot acting as a senior engineer agent (Acquisition TS SG1)– Acquisition constraints levelled on the project, while

valid, implied that no automated analysis tools could classify defects in this project.

– Discovered real deviation rate by roadway roughness being larger than previously believed would have required the client to entirely rethink both their roadway standard and their quality management practice – both of which were outside the scope of the project's management. This is the essence of a system of systems issue.

Root Cause of Acquisition Failure

Summary

This provides a compelling story for the study of acquisition and acquisition engineering and provides a concrete example of how an acquisition organization can fail to acquire “the right system” without appropriate attention paid to system of systems issues.

Therefore, observing an organization's acquisition and employment of robotics provides a microcosm to explore system of systems acquisition issues.

Summary

Summary

• Overview - Environment• Acquisition Program Goals• Surface Assessment Robot• Root Cause of Acquisition Failure• Summary

Summary

Backup

Acronyms

• CMU – Carnegie Mellon University• GM – General Manager (construction project)• HRI – Human-Robot Interaction• QA – Quality Assurance• RD – Requirements Development Process Area• SG – Specific Goal (of a CMMI process area)• TS – Technical Solution Process Area• USC – University of Southern California• VAL – Validation Process Area• VER – Verification Process Area

References

• “Adapting CMMI for Acquisition Organizations: A Preliminary Report”, Dodson, et. al., CMU/SEI-2006-SR-005

• "Running Surface Assessment Technology Review", Latimer, et. al., CMU/ICES-04-03-02

• Research project notes, meeting minutes, and project plan documents from CMU for Roadway Assessment System

Bio

1Lt DeWitt Latimer IV, USAF is currently assigned as a PhD Student to the Air Force Institute of Technology and working towards a PhD in Computer Science at the University of Southern California. He is advised by Prof Barry Boehm and Prof Gaurav Sukhatme. His research focuses on investigating the nature of acquiring autonomous robotic systems. He earned his MS degrees in Robotics (2001) and Civil Engineering (2002) at Carnegie Mellon University. He is a senior member of the IEEE and a member of ACM, ASCE, and AFCEA and was awarded the CSDP credentials from the Computer Society.

Documents

A Case Study in Acquiring Robotic Capabilities Inherently a System of Systems Activity?