172529main ken and_tim_software_assurance_research_at_west_virginia

IV&V FacilityResearch Heaven,

West Virginia

1

SA @ WV(software assurance

research at West Virginia)

Kenneth McGill NASA IV&V Facility Research [email protected]

Dr. Tim Menzies Ph.D. (WVU)Software Engineering Research Chairtim@menzies,com


West Virginia

2

Why, what is software assurance?

• Definition:– Planned and systematic set of

activities – Ensures that software

processes and products conform to requirements, standards, and procedures.

• Goals:– Confidence that SW will do what is

needed when it’s needed.

Before bad software After bad software

• Why software assurance?–bad software can kill good hardware. –E.g. ARIANE 5: (and many others)

•Software errors in inertial reference system•Floating point conversion overflow

Ariane 5


West Virginia

3

OSMA Software Assurance Research Program

• Office of Safety & Mission Assurance (Code Q- OSMA)• Five million per year• Applied software assurance research• Focus:

– Software, not hardware– SW Assurance– NASA-wide applicability

• Externally valid results; i.e. useful for MANY projects

• Organization:– Managed from IV&V Facility– Delegated Program Manager: Dr. Linda Rosenberg, GSFC


West Virginia

4

Many projects

• Mega: highest-level perspective– e.g. project planning tools like ASK-PETE

[Kurtz]• Macro:

– e.g. understanding faults [Sigal, Lutz & Mikulski]

• Micro:– e.g. source code browsing [Suder]

• Applied to basic:– Applied:

• (e.g.) MATT/RATT [Henry]: support large scale runs of MATLAB

– Basic (not many of these)• e.g. Fractal analysis of time series data

[Shereshevsky]• Many, many more

– Too numerous to list– Samples follow– See rest of SAS!

Horn of plenty


West Virginia

5

Many more projects!

0

7

11 12

6 5

1 13

1

6

2

7

27

1012

4

0 0

5

26

22

0

5

10

15

20

25

30

AR

C

GR

C

GS

FC

IV&

V

JP

L

JS

C

KS

C

La

RC

MS

FC

Ind

us

try

Un

ive

rsit

y

2002

2003Total proposals: 2.2NASA centers: 1.5Industry: 26University: 3.7

Ratio FY02/FY01

Good news!• More good proposals than we can fund

Bad news!• same as the good news


West Virginia

6

A survey of 44 FY01 CSIPs

project 1 2 3 4 5 6 7 8 9 10 11 12 13 14 to 44

AATT 2

ISS 2

Space Shuttle 2

ST5 2

Aura 1

CHIPS 1

CLCS 1

CM2 1

CMMI 1

DSMS 1

EOSDIS 1

FAMS 1

GLAST 1

HSM4 1

HST 1

Mars 07 1

Mars 08 1

PCS 1

Space Station 1

Starlight 1

Stereo 1

SWIFT 1

X-38 1

5 4 3 2 2 2 2 2 1 1 1 1 1 0

Need moretransitions!

(but don’tforget thetheory)

75% with noclaim forprojectconnections


West Virginia

7

Action plan- restructure CSIPS: more transitions!

• New (year 1)– Fund many

• Renewed (year 2)– Continue funding the promising new

projects– Recommended: letter of endorsement

from NASA project manager• Transition (year 3)

– Select a few projects– Aim: tools in the hands of project folks– Required: project manager involvement

• Reality check:– Transition needs time– Data drought


West Virginia

8

Long transition cycles

CO2 + 2H2 —> CH4 + O2

Marsatmosphere

oxidizerfuel

on-board

(no photo)

CarmenMikulski

JPL

Robyn LutzJPL, CS-Iowa

State

• Pecheur & practical formal methods

– In-Situ Propellant Production project– Taught developers:

• Livingstone model-based diagnosis

• model-checking tool tools• developed by Reid Simmons,

(CMU)– Technology to be applied to the

Intelligent Vehicle Health Maintenance (IVMS) for 2nd generation shuttles

• Lutz, Mikulski & ODC-based analysis of defects

– Deep-space NASA missions– Found 8 clusters of recurring defects– Proposed and validated 5

explanations of the clusters– Explanations changes to NASA

practices– ODC being evaluated by JPL’s defect

management tool team

Charles Pecheur

RIACS, ASE, ARC


West Virginia

9

The data drought

Gasp… need data…


West Virginia

10

End the drought:bootstrap off other systems

• Find the enterprise-wide management information system

• Insert data collection hooks– E.g. JPL adding

ODC to their defect tracking system

– WVU SIAT sanitizer


West Virginia

11

End the drought:Contractors as researchers

active data repository

• Buy N licenses of a defect tracking tool (e.g. Clearquest)

• Give away to projects– In exchange for their data

• Build and maintain a central repository for that data– With a web-based query

interface

• Data for all

take me to your data

http://www.galaxyglobal.com/home.html


West Virginia

12

End the drought:Contractors as researchers (2)

abstractionabstractionactionaction reflectionreflection

experienceexperience 1

2

3

4

Mark SuderTitan, IV&V

Hypertext power browser for source code4 SIAT-1}

For high-severity errors, recall what SIAT querieslead to finding those errors

4’

2’

Assess each such “power queries”Reject the less useful ones

3’

Procedures manual for super SIAT ornew search options in interface

SIAT2}1’ Use it.

See also:

• Titan’s new ROI project

• Any contractor proposing an NRA

• Galaxy Global’s metric project

See also:

• Titan’s new ROI project

• Any contractor proposing an NRA

• Galaxy Global’s metric project


West Virginia

13

End the drought:raid old/existing projects

• Cancelled projects with public-domain software– E.g. X-34

• Or other open source NASA projects – E.g. GSFC’s ITOS:– real-time control and

monitoring system during development, test, and on-orbit operations,

– UNIX, Solaris, FreeBSD, Linux, PC

– Free!!– NASA project connections:

• Triana, • Swift, • HESSI, • ULDB, • SMEX, • Formation Flying Testbed, • Spartan


West Virginia

14

End the drought:synergy groups

• N researchers– Same task– Different

technologies• Share found data• E.g. IV&V business

case workers• E.g. monthly fault

teleconferences– JPL:

• Lutz, Nikora

– Uni. Kentucky: • Hayes

– Uni. Maryland: • Smidts

– WV: • Chapman

(Galaxy Global) & Menzies (WVU)


West Virginia

15

End the drought:Tandem experiments

• “Technique X finds errors”– So?

• Industrial defect detection capability rates: – TR(min,mean,max)– TR(0.35, 0.50, 0.65)– Assumes manual

“Fagan inspections”• Is “X” better than a

manual 1976technique?

• Need “tandemexperiments”to check

• I.e. do it twice– Once by the researchers– Once by IV&V

contractors (baseline)

0

20

40

60

80

100

120

defects found

analysis design code test

baseline FM Fagan

fictional data

0

20

40

60

80

100

120

cost

analysis design code test


West Virginia

16

Alternatively:End your own drought

• Our duty, our goal:– Work the data problem (e.g. see above)– Goal of CI project year1: build bridges– But the more workers, the better

• Myth: there is a “data truck” parked at IV&V – full of goodies, just for you

• Reality: Access negotiation takes time– With contractors, within NASA

• We actively assist:– Each connection is a joy to behold,

an occasion to celebration– We don’t celebrate much

• Bottom line:– We chase data for dozens of projects– Researchers have more time, more focus on

their particular data needs• Ken’s law:

– $$$ chases researchers who chase projects– CI year2, year3: needs a project connection


West Virginia

17

Alternatively (2), accept the drought and sieve the dust

• The DUST project:– Assumes a few key options control the rest

• Methodology:– Simulate across range of options

– Data dust clouds– Too many options: what leads to what?

– Summarize via machine learning– Condense dust cloud– Improve mean, reduce variance

• Case studies: – JPL requirements engineering:

• Feather/JPL [Re02]

– Project planning: • DART- Raque/ IVV; Chaing/UBC; • IV&V costing: Marinaro/IVV, Smith/WVU• general: Raffo, et.al/PSU [Ase02]

– An analysis of pair programming: Smith/WVU– Better predictors for:

• testability: Cukic/WVU, Owen/WVU [Issre02, Ase02]• faults: diStefano/WVU, McGill/IVV; Chapman/GG• reuse : diStefano/WVU [ToolsWithAI02]

Figure 2. Initial (scattered black points) and Final (dense white points)

0

50

100

150

200

250

300

0 300000 600000 900000 1200000Cost

Ben

efit

Each dot = 1 random

project plan

The answer my friend, is blowin’

in the wind

But wait: the times they

are changing


West Virginia

18

Katerina Goseva Popstojanova

Other WVU SA research

Architecturaldescriptions

Fault, failure

data oncomponents,

connectors

SoftwareSpecs & design(early life cycle)

Code analysis(iv&v,operational usage)

Metrics(complexity,coupling,entropy )

Failure data from testing

Severity of failures

UML (sequence diagrams, state charts)

UML simulations

Static (SIAT, Mccabe, entrophy)

Dynamic (testing, runtime monitoring)

Testing & formal methods

Bayesian approach to reliability

Architectural metrics

Risk assessment & dynamic UML

Reliability & operational profile errors

Hany Ammar

Bojan Cukic

collaborator

Goal: accurate, stable, risk assessment early in the

lifecycle

Goal: accurate, stable, risk assessment early in the

lifecycle


West Virginia

19

More WVU research (FY02 UIs)

Architectural metrics

Risk assessment & dynamic UML

Intelligent flight controllers

Testing & formal methods

Bayesian approach to reliability

Fractal study of resource dynamics

Reliability & operational profile errors

SE research chair

interns

DUST

Ammar

Cukic

Goseva-Popstojanova

Menzies

newrenewed

c = conference w = workshopj = journal

ISS hub controller, “Dryden application”

F15

“JPL deep space mission”DART“KC-2”IVV cost models

SIAT

X34

ITOS

X38

jj

j, ccccccc, w

c

cccccc

jc

c

w

FY03 proposals = 2.2*FY02


West Virginia

20

Function Point Metrics for Safety-Critical Software

• Thesis:– Traditional function-point

cost estimation– Incorrect for safety-critical

software

• > 1 way to skin a cat– >1 way to realize a safety

critical function:– NCP=

N-copy programming– NVP=

N-Version Programming ,

– NSCP= N Self-Checking Programming,

– …– With, without redundancy,

• Method: – explore them all!

1.3000

1.4000

1.5000

1.6000

1.7000

1.8000

1.9000

2.0000

0 0.033 0.1 0.33 1

Algorithm Complexity

H2/H

1, C

2/C1

NCP

NVP, NSCP

RFCS

CRB

RB, NRB

DRB, EDRB

NCP

NVP, NSCP

RFCS

CRB

RB, NRB

DRB, EDRB

Design Diversity, add eight more

Design Diversity, add onemore

Data Diversity

H2 and C2 : effort & cost, redundant systemH1 and C1: effort & cost, non-redundant system Afzel Noore


West Virginia

21

Pre-disaster warnings [Cukic, Shereshevsky]

Can we defer a maintenance cycle and keep doing science for a while longer?

Mark Shereshevsky

CrashEarly warning

}

Time for gracefulshutdown

Bojan Cukic

ARTS II


West Virginia

22

Intelligent flight controllers [Napolitano, Cukic] (and menzies)

Marcello Napolitano(Mechanical and

Aerospace)

Bojan Cukic(CSEE)

Lifecycle opportunities for V&V of neural network based adaptive control systems.


West Virginia

23

The road ahead: applied & theoretical research

CSIPs: applied research

USIPs: applied + theoretical research

Need both

To boldly go…

Documents

172529main ken and_tim_software_assurance_research_at_west_virginia