COTS Based Systems: Benefits, Potential Risks

and Mitigation Techniques

COTS Based Systems: Benefits, Potential Risks

and Mitigation Techniques

Ronald J. Kohl Chair, GEIA IT&I TCTitan Systems Corjkohl@prodigy.net

ContentsContents

•Definitions

•Benefits of COTS

•Risks with Using COTS

•Mitigation approaches

•Some final thoughts

DefinitionsDefinitionsCommercial Off The Shelf (COTS): Commercially available product acquired in ‘as is’ condition, typically with built-in ‘tailoring’ capabilities but typically with no source code access

Other Non-Developed Item (NDI) typesMOTS (Modified Off The Shelf)GOTS (Government Off The Shelf)Reuse products, shareware, Open Source

Custom: Developed software, control of source code and development team

DefinitionsDefinitionsWrapper: Custom built software that provides an interface between a COTS product and some other software

Application Service Provider (ASP): a new business model that allows for the ‘rental’ of COTS capabilities, in lieu of purchasing the COTS product

Enterprise Application Integration (EAI): the activities needed to integrate a variety of products (COTS, etc) and services (ASP, etc) in support of creating an Enterprise solution

Kohl’s LemmaKohl’s Lemma

Using COTS products in lieu of custom built software is a trade between control of ...

Faster and cheaper vs Better

(but ‘cheaper’ is under review!)

Potential Benefits of Using COTS

Potential Benefits of Using COTS

• Reduced development cost/schedule

• Reduced maintenance cost(?)

• Product ‘improvements’ paid by the vendor

• “Proven” product

• Wide user base to identify problems

• Wide user base to build operational life

• Available skill base

• Industry investment in technology base

Potential Risk Areas

CHART 8

COTS RisksCOTS Risks

• ‘Make vs Buy’ decision has become the ‘Make

vs Buy vs Rent’ decision•Some projects have abandoned the ‘make vs buy’

decision process (assumes that COTS is the way to go)

• Introduction of ASPs further complexifies this decision

• Different lifecycle model (vs custom SW) • Requirements development (driven by COTS features)

• Testing (black box vs white box)

• Systems evolution vs product evolution (could diverge!)

• Maintenance (product volatility, vendor supportability)

CHART 9

COTS RisksCOTS Risks

• Product Volatility

• product changes when the Vendor chooses

• driven by market forces, not necessarily your needs

• not aligned with your fiscal cycles

• product lifetime usually much less than system lifetime

•No/little insight into product or process• Product flaws, documentation, source code, test reports

• Development processes, tools or skills

• can conflict with CMM Level 3 Acquisition requirements

CHART 10

COTS RisksCOTS Risks• System incompatibilities

– H/W Platforms

– Business processes

– Operational environment

•Underestimated total program costs• Integration

• Verification and Validation

• Training

• O&M

COTS Risks (con’t)COTS Risks (con’t)• Vendor viability

– as a business entity

– as a technology provider

– susceptible to acquisition, by off-shore companies?

•Risk to maintenance

• Vendor support and stability

• Vendor dependency on error reporting/fixing

• Likelihood of ‘wrappers’ •Unique operational environment

•Stringent program requirements/needs

COTS Risks (con’t)COTS Risks (con’t)• Testing is sometimes inadequate

– total systems testing still mandatory

– product testing should be across lifecycle

•Multiple COTS product integration

• Interoperability (plays well together)

• Overlap Code/Dormant Code

• Product problem determination, responsibility

• Increase in complexity of CM

• Increase in complexity of system evolution

• Increase in complexity/cost of wrappers

•Total Cost of Ownership

Mitigation Techniques

Mitigation TechniquesMitigation Techniques

• Employ good systems engineering practices– these are still large, complex systems

– they contain many risks

•Conduct a ‘make vs buy (vs rent?)’ trade

study

– do not assume that ‘buy’ is the only viable option

– let your needs define the criteria for such a study

– included full lifecycle concerns (development, O&M)

Mitigation TechniquesMitigation Techniques

• Gain Marketplace and vendor knowledge – conduct business viability analysis (early and often)

– gain confidence in vendor’s quality (early and often)

– identify alternate vendors (early and often)

– may require a new role within project and/or enterprise!

• Gain product knowledge • Learn all you can, as early as you can, however you can

• Shop early & often, allow for requirements changes

• Identify usage in similar applications (ask vendor, UG

involvement, web sources, etc)

• may require a new role within project

Mitigation Techniques(con’t)

• Build around ‘flagship’ COTS product– typical in ERP systems

– Reduces multiple vendor issues

• Early vendor involvement throughout the life

cycle– Typical in both ERP and engineering systems

• Overall robust system design that can

withstand the unexpected and ‘weak links’

• Robust verification plan and environment

Mitigation Techniques(con’t)

• Contractually require product ‘insight’

requirements

– problem reports, test plans/scripts, etc

– product development processes

– skill base

– Don’t be surprised if vendors balk

• Product and/or Vendor certification ?

– Can be the GO/NOGO decision in Safety Critical systems

• Source code escrow– Is almost always an expensive option!

System Architecting Heuristics

• Focus on those things which must change the least– Know where to put the basement, very early– Know where to put the load bearing walls, early– Design the whole building but don’t build the 22nd floor

before the basement

• Understand those things which will likely change the most– new technologies (COTS, HCIs, Processors)– Political impacts (new admin, in my district, etc)– decide/baseline/ buy as late as possible, but still support

the project schedule

Summary • Employ good systems engineering

• Shop early, shop often

• Test Drive early, often

• Know the vendor, early and often

• Know the marketplace

• Know your options

• Acquire the right skills for COTS-based systems

• Defer final decision/purchase as late as possible!

• But buy only when you are a fully informed buyer!!