Daniel.dvorak

12/02/2009Software Architecture Review Board 1

Software Architecture Review Board for Flight Systems

Daniel DvorakJPL Systems & Software Division

SARB Team [email protected]

818.393.1986

Michael AguilarNESC Software Discipline Expert

NASA Technical Fellow in Software [email protected]

301.388.0156

Feb. 9-10, 2010

NASAProjectManagementChallenge2010

Used with Permission

2Software Architecture Review Board

Motivating Questions

• What is the purpose of a software architecture review?

• What kinds of problems are found?

• What are the benefits?

• What is software architecture, anyway?

• How do you evaluate architecture?

• Are there impediments to good software architecture within NASA?

• What’s the relationship between systems architecture and software architecture?


OverviewGoals• Help NASA missions achieve higher

reliability and cost savings• Manage flight software complexity

through better software architecture

Approach• Create a NASA-wide

software architecture review board (SARB)

• Engage with flight projects in the formative stages of software architecture

Plan• Prepare introductory document,

review process, review checklist, sample problem statement, and sample report

• Educate team on process• Practice on flown missions• Conduct real reviews


Outline

• Origin of this task• Architecture reviews: history and benefits• Architecture description• Architecture review process• Issues found in architecture reviews• How you can help


Origin

• NASA OCE study on flight software complexity involved flight software engineers from GSFC, JPL, JSC, MSFC, and APL

• The study recommended formation of a NASA software architecture review board

• Why? Because …– It saves projects time and money (AT&T and Lucent

experience)– Weak software architecture is a contributor to problems on

NASA missions (though rarely recognized as such)– Good architecture is the best defense against unnecessary

complexity. “Point of view is worth 80 IQ points” (Alan Kay)


NESC Support

• NASA Chief Engineer Michael Ryskewitsch was briefed on findings and recommendations of the Flight Software Complexity study on 4/23/2009

• One recommendation was to establish a NASA software architecture review board (SARB)

• Michael Aguilar, NESC Software Discipline Expert and NASA Technical Fellow in Software, volunteered to support the board as a technical discipline team (TDT)

• (Other recommendations are being followed up on. For example, static analysis of software code is now a NASA requirement.)


Systems vs. Software Architecture

• Architecture provides a unifying vision

• Systems architecture is comprehensive: flight and ground systems, spacecraft instruments and subsystems, hardware, software, etc.

• Software controls most of the system behavior

• Thus, the architecture of behavior is in the software architecture

• Note: Growth in size and complexity of flight software is a reflection of its role in meeting increasingly ambitious mission & system requirements

Architecture Reviews

• Description• History• Benefits


What is an Architecture Review?

• DAU lists architecture reviews as a best practice having the most supporting evidence

“Architectural Reviews are formal reviews held to evaluate how well a proposed architecture meets the needs and operational concept of the system under development. By focusing on the architecture and ops concepts, they identify mismatches early in the life cycle. An architectural review board or panel (an expert, non-advocate group) usually conducts the review.”

Best Practices ClearinghouseDefense Acquisition Universitybcph.dau.mil


History

Software Architecture Reviews AT&T Bell Labs was developing software-

intensive systems for telephony in the 1960’s

By the 1990’s AT&T had a standing Architecture Review Board that examined proposed software architectures for projects, in depth, and pointed out problem areas for rework– The board members were experts in architecture & system

analysis– They could spot common problems a mile away– The review was invited and the board provided constructive

feedback– It helped immensely to avoid big problems


History

Benefits of Architecture Reviews (1 of 2)• “Architecture reviews tend to increase quality, control

cost, and decrease budget risk.”– [Bass, Clements, and Kazman, Software Architecture in Practice, 1998]

• “In our experience, the average [architecture] review pays back at least twelve times its cost.”

– [Daniel Starr and Gus Zimmerman, STQE Magazine, July/August 2002]

• Beneficial side effects:– Cross-organizational learning is enhanced– Architectural reviews get management attention without

personal retribution– Architectural reviews assist organizational change– Greater opportunities exist to find different defects in integration

and system tests. – [Maranzano et al, IEEE Software, March/April 2005]


History

Benefits of Architecture Reviews (2 of 2)“Project teams found that the preparation for the review helped them get a clearer understanding of their projects. In several reviews the people on the project team asked more questions than the reviewers. Often, this was one of the few opportunities for the project team members to have in-depth discussions of the technical issues about the project. The review served as a catalyst to bring them together.”AT&T: “Best Current Practices: Software Architecture Validation”, 1991

“The architecture review process has helped train people to become better architects and helped establish a consistent view across our companies, both of what architecture is and what good architecture’s characteristics are.”Maranzano et al, Architecture Reviews: Practice and Experience, IEEE Software, March/April 2005.


Mission & Charter of SARB

Charter• Provide constructive feedback to flight projects in the

formative stages of software architecting• Focus on architectural improvements to reduce and/or

better manage complexity in requirements, analysis, design, implementation, verification, and operations

• Spread best architectural practices, principles, and patterns across flight software centers

• Contribute to NASA Lessons Learned

Mission:Manage flight software complexity

through better software architecture

Architecture Description

• Architecture reviews help carry out our mission• A review requires an architecture description• So …

– What is architecture?– How should an architecture be described?


What is an Architecture Description?

• Every system …– has an architecture (documented or not)– has stakeholders

• Every stakeholder has concerns• An architecture is described by an architecture description• An architecture description …

– identifies stakeholders– is organized by views that address stakeholders’ concerns– provides rationale

IEEE 1471-2000: Recommended Practice for Architecture

Description of Software-Intensive Systems


Conceptual Framework for Architecture

System

Concern Viewpoint View

Stakeholder

Rationale

Architecture Description

ArchitectureEnvironment

Mission

1..*

1..*

1..*1..*

1..*

1..*

1..*selects

1

has

identifies

organized by

conforms to

provides

described by

has an

fulfills

1..*1..*

has

inhabits

is addressed to

1

influences

ANSI / IEEE 1471-2000:Recommended Practice for Architecture Description of Software-Intensive Systems


Scope of Architectural Concerns• Architecture should address non-functional rqmts

– performance, availability, maintainability, modifiability, security, testability, operability, etc.

– Watch for unsubstantiated or ambiguous rqmts

• Architecture should address principles of design- Identify and follow architectural principles- Leverage appropriate architectural patterns

• Architecture should address verifiability- Design can simplify or complicate verification

• Architecture should address operability- Inadequate design complicates operations- Operational workarounds raises risk

• Architecture should address analyzability- “Point of view is worth 80 IQ points”

Verification & Validation

Complexity

Requirements Complexity

Operations Complexity

Flight Software Complexity

System-Level Analysis &

Design


What is Architecture?

“ Architecture is the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.”

IEEE Standard 1471: Recommended Practice for Architectural Description of Software-Intensive Systems

• where:– fundamental = essential, unifying concepts and principles– system = application, system, platform, system-of-

systems, enterprise, product line, …– environment = developmental, operational, programmatic,

… context


Two Notions of “Architecture”

• architecture — What gets built– Describing components and interfaces– The details of assembly and integration

• Architecture — Why it gets built the way it does– Identifying properties of interest beyond just the requirements,

and from all essential points of view– Defining abstractions and other patterns of design that give the

design shape and reflect fundamental concepts of the domain– Guiding design and maintaining principles throughout lifecycle– Building on a body of experience and refining concepts as necessary

Architecting is about managing complexity

Source: Bob Rasmussen, JPL

The Review Process

• AT&T Review Process• Principles• Participants• Implementation• Artifacts

• Issues found


What review process to use?

• For reviews within NASA, expert opinion favors the AT&T process over ATAM– David Garlan (CMU), Gus Zimmerman (Lucent), Katy Weiss (JPL)

• Reasons given in favor of AT&T process:– Review team composed of external experts whereas ATAM review

team is all stakeholders (potentially biased)– Better transfer: encourages cross-fertilization of

ideas/standards/approaches across organization.– Better residuals: reusable checklists, standard templates, shared

understandings about common architectural approaches– ATAM not quantitative enough to characterize quality attribute

requirements with testable figures of merit, and then map those testable requirements directly into the structures in the architecture that accomplish them

– ATAM prioritizes scenarios by voting among all stakeholders thus allowing a serious issue, identified by an expert, to be ranked low


AT&T Review Process: Principles• A clearly defined problem statement (success criteria) drives the

system architecture– The project writes it and the review team uses it– Often requires iteration before it’s good enough

• Product line and business application projects require a system architect at all phases– Architect is an explicitly defined role

• Independent experts conduct reviews– Chosen reviewers are recognized experts and are independent of the

project• Reviews are open processes

– Open approach to identifying issues and strengths– Issues written on 5x8 “snow cards” and displayed on wall

• Conduct reviews for the project’s benefit– Reactions to issues are project management responsibility– Review team does not discuss issues later without project consent


AT&T Review Process: Participants• Review client

– Pays for system development, or is review’s sponsor• Project members

– Creators, contributors, and users of the architecture– One member selected to be contact for logistics and artifacts

• Project management– All managers responsible for project’s success– Management nominates project for review

• Review team– Subject matter experts selected on basis of expertise,

independence from project, and interpersonal skills• Architecture review board

– Standing board that oversees and adjusts review process• Review angel

– ARB member with managerial experience to address political issues

• ARB chair – Strong process advocate; ensures effectiveness; secures support

project

reviewteam

board


• Phase 1: Screening– Project and ARB determine if a review would benefit project– If so, ARB selects a review angel

• Phase 2: Preparation– ARB selects a review team, including a review leader– Project prepares clear problem statement (success criteria) and docs

• Phase 3: Review Meeting– Typical review is 2 days plus ½ day of caucus plus 1-hour readout– Issues prioritized: management alert, critical, major, minor

• Phase 4: Follow-Up– Review team delivers report to project within 15 days– If management alert(s), project must respond within two weeks

AT&T Review Process: Implementation


• Architecture review checklist– Questions that architects and reviewers should consider– Evolves over time according to serious and prevalent issues– “An accumulated institutional knowledge repository”

• Input to the review– Problem statement (success criteria), system requirements,

functional requirements, architectural specification, other informational docs

• Output from the review– Set of issues (often in the form of snow cards)– A review report– Optional management alert letter

AT&T Review Process: Artifacts


Kinds of Issues Found (AT&T, Lucent)

Category Percent

Problem DefinitionProblem isn’t completely or clearly defined

10-18%

Product Architecture and DesignProposed solution doesn’t adequately solve the problem

29-49%

TechnologyInadequate languages, tools and/or components to build system

3-14%

Domain KnowledgeTeam lacks adequate knowledge or experience

2-5%

ProcessProcess isn’t systematic, complete, or designed to make development manageable

4-19%

Management ControlsInadequate management monitoring capabilities, staffing, controls, and decision-making mechanisms

14-26%


Example Issues from within NASA

• Boxes-and-lines diagrams lack clear semantics

• Flight software design details that are unnecessarily coupled to hardware details

• Lots of software “gadgets” but little in the way of abstractions tailored for the problem domain

• Excessive cross-strapping of hardware that complicates software without much reliability benefit

• Underestimation of time needed to adequately test redundancy management

• Fault protection design that doesn’t scale well

• Fault protection designed only to handle a laundry list of faults; lack of defensive mindset


Impediments to Software Architecture within NASA

• Inappropriate modeling techniques– “Software architecture is just boxes and lines”– “Software architecture is just code modules”– “A layered diagram says it all”

• Misunderstanding about role of architecture in product lines and architectural reuse– “A product line is just a reuse library”

• Impoverished culture of architecture design– No standards for arch description and analysis– Architecture reviews are not productive– Architecture is limited to one or two phases– Lack of architecture education among engineers

• Failure to take architecture seriously– “We always do it that way. It’s cheaper/easier/less risky

to do it the way we did it last time.”– “They do it a certain way ‘out there’ so we should too.”– “We need to reengineer it from scratch because the

mission is different from all others.”

As presented by Prof. David Garlan (CMU) at NASA Planetary Spacecraft Fault Management Workshop, 4/15/08


What an architecture review is NOT

An architecture review is …• not a gate, not a mandatory review (in AT&T)• not a pass/fail judgment • not an audit for a cancellation decision• not an evaluation of architect’s performance• not a tutorial• not a code review


Timeline of SARB Activities

April 2009 Mike Ryschkewitsch approves recommendation for SARBMike Aguilar funds it as a NESC Technical Discipline Team

Sep. 2009

May 2009

June 2009

July 2009

Aug. 2009

Kickoff telecon with FSW Complexity team + Mike Aguilar

Team formation and preparation phase• prepare charter• select review board members• define review and reporting process• identify focus areas for evaluation• get exposure to several flight software architectures• conduct mock reviews

Jan. 2010

Conduct first real review (goal)

Oct. 2009Nov. 2009Dec. 2009

Feb. 2010


Current Team Members (as of 12/3/2009)

Name AffiliationMichael Aguilar OCE/NESCDan Dvorak JPLMichel Ingham JPLLou Hallock (leaving) GSFCMichael Blau (coming) GSFCJohn Weir MSFCLeann Thomas (observing) MSFCHelen Neighbors JSCKevin Balon APLSteve Williams (observing) APL


How You Can Help

Projects:• Identify candidate projects for review

People:• Identify subject matter experts who may serve on reviews• Suggest potential review board members

Architectural Concerns:• Improve our architecture checklist

– Offer your strongly felt architectural issues– Negative examples are among the best teachers


“Great goulash, Stan. That reminds me, are you still in charge of our system architecture?”

Questions?

Backup

• What about ground software?• SARB telecon topics• “Early warning signs”• Mindset for reviews• Open questions


What About Ground Software?

• Initial focus is on flight software because the recommendation came from a FSW study

• Ideally, GSW should be included, but …– Architectural concerns will be somewhat different– Need different expertise on the review board– Need additional funds

• One way to “ease into” GSW is to review:– End-to-end information flow– Flight/ground operations paradigm and uplink/downlink


SARB Telecon Topics (1 of 2)Date Topic

5/18/09 Pre-planning: NESC intro, identify board members and projects

6/01/09 Kickoff: charter, questions, speakers

6/08/09 Handbook for real-time analysis (Mike Aguilar)

6/15/09 Architecture review checklist (Lou Hallock)

6/22/09 Architecture docs and review process (Katy Weiss)

6/29/09 Altair FSW architecture study (Lore Williams)

7/13/09 General discussion of problems seen in FSW

7/20/09 cFE/CFS software architecture (Charlie Wildermann)

7/27/09 Observations on weak s/w architecture (Bob Rasmussen)

8/10/09 Space Shuttle Main Engine: s/w architecture (Andy Young)

8/17/09 Examples of weak s/w architecture (all)

8/24/09 Discuss architecture review checklist (Lou Hallock)


SARB Telecon Topics (2 of 2)Date Topic

8/31/09 Overview of IEEE 1471-2000 “Recommended Practice for Architectural Description of Software-Intensive Systems” (Jeff Estefan)

9/21/09 Ares I upper stage flight computer software architecture (J. Weir)

10/05/09 Four documents we need to prepare (Dan Dvorak)

10/26/09 The architecture “problem statement” (Ken Costello)

11/02/09 Revisiting the architecture review checklist (Lou Hallock)

11/30/09 SDO Software Architecture (Manuel Maldonado, Mark Walters)

12/14/09 Architecture review overview document (H. Neighbors, J. Weir)

TBD MLAS software architecture (Mike Aguilar)

TBD Experiences in adapting cFE/CFS for LRO (Mike Blau)

TBD MSAP Software Architecture (David Hecox or Bob Denise)

TBD Architectural issues in fault protection (Dan Dvorak)


What to look forEarly Warning Signs (1 of 2)

1. The architecture is forced to match current organization. 2. Top-level architectural components number more than 25. 3. One requirement drives the rest of the design. 4. The architecture depends on alternatives in the OS. 5. Proprietary components used when standard components

would do. 6. Component definition comes from the hardware division. 7. There is redundancy not needed for reliability (e.g., two

databases, two start-up routines, two error-locking procedures).

8. The design is exception driven; that is, the emphasis is on extensibility and not core commonalities.

9. Development unit unable to identify a system architect


What to look forEarly Warning Signs (2 of 2)

10.Difficulty identifying the architecture’s stakeholders11.The architecture provides no guidance on how to code an

interaction between two components (silent or a plethora of choices)

12.Architecture documentation consists of class diagrams and nothing else

13.Architecture documentation is a large stack of documents automatically produced by some tool, but which no human has ever seen

14.Documents provided are old, not kept up to date15.A designer or developer, when asked to describe the

architecture, is either unable to or describes a much different architecture than what the architect presented

(from “Evaluating Software Architectures”, by Clements, Kazman, and Klein 2002)


Mindset for Reviews

• Architecture addresses non-functional requirements– Quality attributes such as verifiability, operability, maintainability,

interoperability, adaptability, reusability, scalability, portability, etc

• Architecture is about principles– “Software architecture involves the description of elements from

which systems are built, interactions among those elements, patterns that guide their composition, and constraints on these patterns.”[Shaw & Garlan, 1996]

– Fred Brooks writes emphatically that a system’s conceptual integrity is of overriding importance and that systems without it fail [Brooks 1975, Bass 2003]

• A major goal of a review is to elicit architectural risks• Architecture is more than high-level design


Open Questions

• Should the board’s scope include aeronautics in addition to aerospace?

• How do we measure effectiveness of reviews?

• What kind of follow-up should we do with projects?

• Should architecture reviews be made part of standard project reviews?

• How do we keep findings confidential to projects but also report upward to OCE/NESC?


About Software Architecture

Definitions• Software architecture = { elements, form, rationale}

[Perry & Wolf, 1992]

• “Software architecture involves the description of elements from which systems are built, interactions among those elements, patterns that guide their composition, and constraints on these patterns.”[Shaw & Garlan, 1996]

• “An architecture is a specification of the components of a system and the communication between them. Systems are constrained to conform to an architecture.”[Luckham et al, 1995]

• “An architecture is … a set of constraints on a design. The utility of an architecture is its ability to simplify the process of generating a design through the imposition of carefully chosen constraints.” [Gat, 1998]


Source: Anthony J. Lattanze and David Garlan

Architecture versus Design

• Architecture is design, but not all design is architectural. – Architects intentionally limit their focus and avoid

the details of how elements do what they do.• Detailed designs and implementation details are left to

downstream engineers/experts.

– Downstream engineers are expected to respect the architecture to ensure properties promised by the architect are present in the product.


Two Sources of Software Complexity

Essential complexitycomes from problem domain and mission requirements

Can reduce it only by descoping

Can move it (e.g. to ops), but can’t remove it

Sw complexity = Essential complexity + Incidental complexity

Incidental complexitycomes from choices about architecture, design, implementation, including avionics

Can reduce it by making wise choices


Miscellaneous

• Architecture is concepts, patterns, interaction, and principles of design − the things you care so much about that they are the last thing you would change. They guide the design.

• There’s a real opportunity for the SARB to document and spread good design patterns and practices

• Software and hardware must support the system architecture– Historically, software has been made to conform to the selected

hardware– Now, due to growth in software complexity, hardware decisions

should be balanced with impacts on software


Summary of IEEE 1471

• An architecture should address a system's stakeholdersconcerns

• Architecture descriptions are inherently multi-view(no single view adequately captures all stakeholder concerns)

• It separates the notion of view from viewpoint• a viewpoint identifies the set of concerns and the

representations/modeling techniques, etc. used to describe the architecture to address those concerns

• a view is the result of applying a viewpoint to a particular system

• It establishes content requirements for architecture descriptions

• It provides guidance for capturing architecture rationale