Slide 1

Improving the Current State of Software within NASA

May 8, 2000

Presentation by CIO, CE, and Code Q

SMC and NASA Administrator

Version 5

Briefing to SMC and NASA Administrator CIO_Code Q_Chief Eng Talk_V5_March 28 V1.ppt

Slide 2

NASA Software Working Group Charge

• Identify software challenges in NASA

• What are our goals

• What are our challenges in meeting those goals

• Recommend plans to address the challenges and obtain the goals

• Provide recommendations concerning the implementation of software

best practices

• Identify methodologies to evaluate and improve the quality of software

across the Agency

Slide 3

GoalsPrimary Goal• Successfully Develop Software to support NASA Enterprises and Missions

– High Quality• Meet all project requirements• Safely• Reliably

– Within approved budget– Within approved schedule

Secondary Goal• Continual Sustained Software Process and Product Improvement

– Improve quality– Reduce cost– Increase productivity

Slide 4

Sources of Challenges Data• Survey data from individual NASA employees

– Input received from Ames, GRC, GSFC, JPL, LaRC, MSFC, & HQ

• Data provided from NASA organizations– NASA CIOs presentations (JPL and HQ)– Software Engineering Process Group (LaRC )– Software Assurance Technology Center (GSFC)– Software Engineering Laboratory (GSFC)– Flight Software Group (MSFC)

• Mishap reports– Mars Climate Orbiter Mishap Investigation Board, Phase I Report, NASA, November 10,

1999– "Report on the Loss of the Mars Climate Orbiter Mission", JPL D-18441, JPL Special

Review Board, Nov. 11, 1999– "Common Errors Afflicting Real Time Control System Software", Task 2 Interim Report,

April 30, 1999, Prepared by JSC NX Technology Division, Approved by M. Himel/Chief, NX Technology Division

Slide 5

Findings Sorted into Categories of ChallengesSoftware Project Management Skills Organizational Infrastructure*Requirements management *Software quality assurance and IV&V*Software project planning Organization process focus*Software project tracking and oversight Organizational process definition Software contract management Training and education*Software configuration management Integrated software management Software product maintenance Intergroup coordination

Institutional support / infrastructureSoftware Engineering Skills Baseline data / metrics/lessons learned*Software product engineering (includes: software

requirements, software designs, software code, softwaretesting, integration testing, system and acceptance testing)

Security

Peer reviews Reuse Management Challenges*Software reliability, safety *High Quality Systems Engineering

*Management SupportCoping with Fast Paced Evolution in Software Policies and Standards Technology *Human resources Tools General CMM related challenges Languages Faster, Better, Cheaper Methods COTS

* Indicates the top 10 challenges as prioritized by the NASA Software Group

Slide 6

Top 10 Software Challenges

0 5 10 15 20 25 30 35 40 45

Management Support

Software project planning

Software configurationmanagement

Human resources

Software reliability, safety

Software project tracking andoversight

High Quality SystemsEngineering

Software quality assurance andIV&V

Software product engineering(includes…)

Requirements management

Series1

Slide 7

Sorted List of Software Challenges

0 5 10 15 20 25 30 35 40 45

Security

Faster, Better , Cheaper

Languages

General CMM related challenges

Software product maintenance

Reuse

COTS

Organization process focus

Organizational process definition

Policies and Standards

Institutional support / infrastructure

Tools

Baseline data / metrics/lessons learned

Software contract management

Methods

Integrated software management

Intergroup coordination

Technology

Peer reviews

Training and education

Management Support

Software project planning

Software configuration management

Human resources

Software reliability, safety

Software project tracking and oversight

High Qual ity Systems Engineering

Software quality assurance and IV&V

Sof tware product engineering (includes…)

Requirements management

Slide 8

Findings

The following give examples highlighting the seriousness of the top 10 priority challenges

•Requirements Management

– The flow down of requirements from the higher-level document to the software requirements document was not done resulting in loss of mission

•Software Product Engineering

– Software requirements are not controlled to establish a baseline for software engineering and management: software requirements are poorly documented (if at all) and are not always reviewed with the customer or kept up to date to reflect and communicate changes, and are not used as the basis for software management plans, products and activities

– Software designs using effective methods are not being developed, maintained, documented, and verified to assure the requirements are fully covered and to form a framework for coding; the resulting code either does not work properly or if it works it has very high maintenance costs do to the poor design being hard to modify

Slide 9

Findings continued

• Software Product Engineering - continued

– Software code is not adequately documented and verified; the result is that the software does not meet the requirements

– Software testing is performed in an ad hoc manner, test procedures do not fully cover the requirements if they are written at all; software products rarely work as expected, lack of sufficient testing prior to delivery leads to high maintenance costs

– Integration testing is performed in an ad hoc manner, integration test procedures do not fully cover the requirements if interface requirements were written at all; partial or total system failure is experienced and hug overruns to pay for expensive error correction in the latter phases of the project

– System and acceptance testing

• Interface with navigation software not tested

• Complete end to end testing including interfaces is not done

• Acceptance criteria is either not defined or not well defined

Slide 10

Findings continued

• Software Quality Assurance and I&V– Software quality assurance staff is 0 to 1 deep at some centers…

– Testing until run out of budget or time vs. use of thorough testing methodologies

• Systems Engineering– Due to the lack of systems engineering performed, software and hardware staff are forced to fill

that role in an ad hoc manner

– System engineers often have little software expertise

– The role and responsibility of system analysis is not clearly understood or defined

– The management of complexity is one of the most important issues facing NASA missions

• Software Project Tracking and Oversight– Actual results and performance are not tracked against the plans, corrective actions are not taken

and managed to closure … changes to commitments are not agreed upon …

• Software Reliability, Safety– Making software safer and more reliable

Slide 11

Findings continued• Software Configuration Management

– Lack of proper configuration management has contributed and caused mission failure

• Human Resources

– Software professionals are the first to leave; the hardest to replace

– Obtaining and keeping Program/Project Managers sufficiently knowledgeable of software issues is a serious problem

• Software Project Planning

– The way project phases are currently structured, emphasis is not placed on software planning which leads to huge overruns in testing or high maintenance cost that all could be avoided if proper time was allotted up front

– Finding knowledgeable and experienced software project managers who can apply disciplined engineering practices to software development

• Management Support

– Engineers perceive little or no management commitment to the software process

– There are no investment, encouragement, or reward systems for improvements, best practices, or technology transfer

Slide 12

Summary

• Conclusions – Our current software state is an Ad -hoc application of engineering

– We must progress to a state of disciplined, consistent, repeatable application of software engineering principles

– We will not achieve software quality without this

• Need endorsement for the following to effect a holistic solution– Establish SEPGs at each Center

– SWG define a NASA Software Engineering Implementation Plan to manage and coordinate software improvements efforts across the Agency

– Headquarters review and approve plan and disposition SWG recommendations

• Goals– Successfully Develop Software to support NASA Enterprises and Missions

– Continual Sustained Software Process and Product Improvement

• See additional findings on the next slides

Even Successful Missions may Experience Software Problems (Excerpt From Another Briefing)

“A few days after the July 4th, 1997 landing, the Mars Pathfinder began

experiencing total system resets, each resulting in losses of data. The

problem was a logical error in the real-time scheduling system---a classic

priority-inversion problem. Fortunately, this problem was repairable from earth.”

“A malfunction in one of the on-board computers on Clementine on May 7,

1994 caused a thruster to fire until it had used up all of its fuel, leaving the

spacecraft spinning at about 80 RPM with no spin control. This made the

planned continuation of the mission, a flyby of the near-Earth asteroid

Geographos, impossible.”

“The Magellan spacecraft broke Earth lock and lost communications several

times in August 1990 (soon after entering Venus orbit). It took over six months

to identify the source of the problem, which was a timing error in the flight

software.”

Slide 13

(Excerpt From Another NASA Findings Briefing)

• Centers are almost universally weak in:– Project planning

• Estimating cost, schedule, and resource requirements for project requirements fulfilled by software

– Monitoring and control of software engineering products• i.e., tracking progress and taking effective corrective actions

• Configuration management is not universally applied throughout the software development process

• Interface between software and system engineering processes is not well defined so agreements, audits, and reviews are not well planned or performed to achieve the most benefit

• Software Quality Assurance is generally not well understood nor is its value appreciated

Findings by Raymond Kile, Authorized Lead EvaluatorCenter for Systems Management, Sept 2002

Slide 15

Findings (continued)• Software Contract Management

– Qualified software contractors are not selected

– Constantly receiving poor quality contract deliverables

• Software Product Maintenance

– The maintenance phase is rarely covered under the project initial budget estimates and plans

– Insufficient maintenance staff are available to support ongoing programs

• Peer Reviews

– Required persons were not in attendance at the walkthroughs…

– "…inadequate training of software and other personnel in the software walkthrough process"

• Reuse

– Software is re-written many times because the software is not developed and documented in such a way that it can be re-used.

• Software Coping with Evolving Technology, Tools, Methods, COTS and Environment

– Software architectures, testing, and validation for intelligent machines…

– Tools that would help economically and accurately produce software are not funded (must compensate for this by manual methods that are prone to human error and much more time consuming and therefore more expensive)

– Technology assessment and infusion

– There is a huge lack of experience and knowledge in testing, & integration on projects using COTS/GOTS/ middleware/ glueware

• Organization Process Focus

– Software process development and improvement activities are not planned, funded, or coordinated across the organization

– Strengths are not capitalized on, weaknesses go unchecked, those grass roots efforts that are trying to make progress are not coordinated resulting in duplication of effort

• Organizational Process Definition

– Software is developed using ad hoc approaches instead of sound engineering practices

Slide 16

Findings (continued)• Training and Education

– Individuals in the software arena do not receive the training necessary to perform their role and training is not always provided even if sought• Integrated Software Management

– A project’s defined software process is not a tailored version of the organization’s standard software process since one does not exist• Intergroup Coordination

– Requirements and engineering roles and responsibilities are not clearly defined and agreed to – Need improved communication channels, awareness, and understanding between project management, hardware developers, and software

developers• Institutional Support / Infrastructure

– There is no place to get mentoring and training and advice on all aspects of software development…– Need to find mechanisms to penetrate projects, to introduce software engineering, and to work in partnership with the projects.

• Baseline data / Metrics/Lessons Learned– Insufficient project tracking of size, cost, and error data to accurately do future planning, estimating, and process improvement

• Security– … ensure that security measures have been implemented on all of NASA's computing systems

• Policies and Standards– Currently there is no effective enforcement of the NASA Software Policies NPD 2820.1– There are no firm guidance or requirements on which software standards should be used in the agency

• General CMM related challenges– Flight Software Group was self-assessed at a level 1(of 5) on the CMM at two centers

• Faster, Better, Cheaper– Issues related to development/testing/maintenance of real time embedded software for Faster-cheaper-better missions