Lean SoftwareProduction and Qualification

Infrastructures

Florian Villoing – AdaCorevilloing@adacore.com

Grenoble – October 20, 2009Valence – October 22, 2009

2

What is it about?

● A software production infrastructure● A software qualification infrastructure

In a lean fashion!

3

Lean Software Production

4

About AdaCore's products

● Ada compilation toolchain➔ Compiler (based on GCC), debugger➔ Interfacing with C, C++ and Java➔ 2 IDEs: GPS + GNATbench (Eclipse plug-in)➔ Coverage analysis

● Add-ons: ASIS, XML/Ada, GtkAda, AWS, PolyORB

● Static analysis tools➔ Metrics computation➔ Stack analyzer➔ Coding standard checker➔ Automatic peer review

5

What is a compiler?

“A compiler is a computer program (or set of programs) that transforms source code written in a computer language (the source language) into another computer language (the target language, often having a binary form known as object code).” Wikipedia

6

Native and cross compilers

● A native compiler generates code for the same machine (the host)

● A cross compiler generates code for another machine (the target)

Host Target

native

cross

7

Native compilers

● Different operating systems● Windows, Linux, Solaris, Mac OS X, HP-UX, VMS,

LynxOS, Tru64, AIX, IRIX, etc.

● Different Versions● Windows XP, 2000, Vista, etc.● Linux Red Hat 3/4/5, SuSe 8/9/10, etc.

● Different processors● x86, x86 64bits, Itanium, SPARC, PowerPC, Alpha,

PA-RISC, etc.

8

Cross compilers

● Host environment (OS, OS version, processor)➔ Linux, Windows, Solaris

● Target environment (Target OS, Target OS Version, processor)➔ VxWorks, PikeOS, ElinOS, Lynx OS, Nucleus OS, etc.➔ VxWorks 5, VxWorks 6, VxWorks 653, etc.➔ PowerPC, ARM, AVR, LEON, ERC32, etc.

● Run-time system➔ ZFP, Ravenscar, Full, etc.

9

Combinatory explosion

● Native toolchains● OS x OS version x processor = 46

● Cross toolchains● Host OS x host OS version x host processor x

Target OS x target OS version x target processor x

run-time system = 49

10

And things are getting more complex...

● New platforms are added each year● Few are removed● Long-term projects need to be supported for up

10-15 years

How do I deal with that?

11

Some figures

● 95 different platforms (native and cross)● 15 independent test suites

➢ 15000 tests and 10 million lines of code ➢ 1.4 million tests run each night

● 96 scripts to control the infrastructure➢ About 2300 scripts run each day

12

How to solve the equation?

● Adopting a lean strategy● Introducing agile tactics

Let's move on!

13

TPS

The Toyota Production System

Just-in-TimeJidoka

+

● Making only what is needed, when it's needed, in the amount needed

● Limited stocks with all parts

● Replace used parts from the preceding process

● Limited stocks for the preceding process

● Automatic defects detection

● Stop the line

● Identify the problem

● Andon (problem display board)

● An operator for several machines

14

Adopting a lean strategy

● Identify the value● Identify and remove waste (muda)● Automate● Detect defects early● Fix the cause of defects

15

The value

● Provide high quality software● Fixed release schedule● Provide pre-release versions as soon as

possible● Support new platforms● Add new features● Add new tools

16

Quality & Open-Source

● All AdaCore's products are Open-Source● It's possible to achieve high quality● AdaCore is an active member of the GCC

community● We identify a reasonably stable version of GCC● We run it through our QA process● We contribute our changes back to the community

17

Lean testing strategy

● A fundamental piece toward achieving high quality

● Goals:● Detect defects as early as possible● Adopt different testing tactics to minimize risk● Keep high level of productivity● Balance the machine load

18

How to detect defects?

1. Local testing

2. Mailserver

4. Continuous Builder

SVN

3. Check-in

5. Nightly builds

19

The mailserver system

● Email-based interface● To test patches● On a remote machine● Incremental build● Full testing even on exotic platforms● Clean result mandatory before check-in

Less than 20 minutes!

20

The continuous builder

● Every check-in triggers an automatic build● Fast machines

➔ quick feedback

● Stop-the-line: should an error be reported, immediate attention/action is required

● Limit the risk of nightly builds failures➔ Delays in intermediate releases delivery➔ More waste, less value➔ Customer satisfaction decreases

21

Nightly buildsSVN

Linux

Red Hat 5 SuSe 9 XP VistaServer

Windows

Packaging

Building

Testing

22

Nightly builds

● For cross platforms➔ We test a different version of the target OS

everyday➔ Use of simulators

➔ More stability➔ Better performances

● Heavy use of virtual machines➔ Ease maintenance of obsolescent platforms

23

Lean production infrastructure

● Shippable software is produced every day● Defects are detected early● Allows to achieve high quality● Allows to deliver intermediate releases quickly

24

Official releases

● Only 2 releases a year➢ One major release (January/February)➢ One corrective release (June/July)

● Customers can plan their own release cycle in advance

● Intermediate releases are still available➢ Provide quick fixes➢ Allows to experiment with new features

25

Official releases

● Failed builds are relaunched● Manual review of each test suite report● Human sanity check of each package● Cross toolchains are tested on real boards

➢ 48h to run the ACATS on some cross platforms!

26

The release schedule

2009 2010

6.2 branch

Development branch

6.3 branch

6.2.0 Beta 6.3.0 Beta

6.2.2 Release

6.3.1 Release6.2.1 Release

6.3.2 Release

27

New features and known problems

● New features are advertised to everybody on our web site

● Known problems are listed on the customer web interface

● Improve traceability● Provide easy access to workarounds● Raises interest in evolution of the technology

28

Visual management with GAIA

● Monitor scripts● Monitor machines● Monitor test suites● Submit analysis

● Everything is stored in a database

● Human-readable weather reports

● Use of the django framework

29

GAIA: How does it work?

Test machines

DB

GAIA Server GAIA User's Interface

30

GAIA: Test suite overview

31

GAIA: Weather reports

32

Lean Software Qualification

33

Critical software certification

● Aerospace● ECSS-E-ST-40C/ECSS-Q-ST-80C

● Civil Avionics● DO-178B/ED-12B

● Air Traffic Management● DO-278B

● Security● Common Criteria

● Railway● EN-50126/EN-50128/EN-50129

● Others...

34

DO-178B/ED-12B

● “DO-178B, Software Considerations in Airborne Systems and Equipment Certification”

● Last revision: December 1, 1992● DO-178C is in the pipeline● 5 Design Assurance Levels (DAL)

● A => Catastrophic● B => Hazardous● C => Major● D => Minor● E => No effect

35

What is in DO-178B?

● Process oriented:● Planning● Development● Verification● Configuration management● Quality assurance● Certification liaison

36

DO-178B: The Waterfall Model

Source: Wikipedia

Not very agile...

37

DO-178B: The V-Model

Not agile either...

Source: Wikipedia

38

Certification and Qualification

● Embedded software is certified● A tool is qualified

● As a development tool– Output is part of the airborne software– Can introduce errors– Certification-like process

● As a verification tool– May fail to detect an error– Lightweight process– Tool Operational Requirements– Requirements based testing

39

Qualification of verification tools

● A tool may be reused in different contexts● Operational requirements may change● Different part of the tool may be used in

different context● The tool may evolve

Let's do agile qualification!

40

The qualification machine

● Based on FitNesse● Centralize all qualification artifacts● Ensure consistency between requirements, test

cases and test data● Automatic generation of qualification

documentsIt's now easy

to add new requirements and associated tests!

41

The qualification machine

Let's see it in action!

42

References

● Lean Software Development, An Agile Toolkit by Mary and Tom Poppendieck

● Implementing Lean Software Development by Mary and Tom Poppendieck

● Lean Software Strategies, Proven Techniques for Managers and developpers by and Peter Middelton and Jim Sutton

● http://www2.toyota.co.jp/en/vision/production_system/