Lean SoftwareProduction and Qualification
Infrastructures
Florian Villoing – [email protected]
Grenoble – October 20, 2009Valence – October 22, 2009
2
What is it about?
● A software production infrastructure● A software qualification infrastructure
In a lean fashion!
3
Lean Software Production
4
About AdaCore's products
● Ada compilation toolchain➔ Compiler (based on GCC), debugger➔ Interfacing with C, C++ and Java➔ 2 IDEs: GPS + GNATbench (Eclipse plug-in)➔ Coverage analysis
● Add-ons: ASIS, XML/Ada, GtkAda, AWS, PolyORB
● Static analysis tools➔ Metrics computation➔ Stack analyzer➔ Coding standard checker➔ Automatic peer review
5
What is a compiler?
“A compiler is a computer program (or set of programs) that transforms source code written in a computer language (the source language) into another computer language (the target language, often having a binary form known as object code).” Wikipedia
6
Native and cross compilers
● A native compiler generates code for the same machine (the host)
● A cross compiler generates code for another machine (the target)
Host Target
native
cross
7
Native compilers
● Different operating systems● Windows, Linux, Solaris, Mac OS X, HP-UX, VMS,
LynxOS, Tru64, AIX, IRIX, etc.
● Different Versions● Windows XP, 2000, Vista, etc.● Linux Red Hat 3/4/5, SuSe 8/9/10, etc.
● Different processors● x86, x86 64bits, Itanium, SPARC, PowerPC, Alpha,
PA-RISC, etc.
8
Cross compilers
● Host environment (OS, OS version, processor)➔ Linux, Windows, Solaris
● Target environment (Target OS, Target OS Version, processor)➔ VxWorks, PikeOS, ElinOS, Lynx OS, Nucleus OS, etc.➔ VxWorks 5, VxWorks 6, VxWorks 653, etc.➔ PowerPC, ARM, AVR, LEON, ERC32, etc.
● Run-time system➔ ZFP, Ravenscar, Full, etc.
9
Combinatory explosion
● Native toolchains● OS x OS version x processor = 46
● Cross toolchains● Host OS x host OS version x host processor x
Target OS x target OS version x target processor x
run-time system = 49
10
And things are getting more complex...
● New platforms are added each year● Few are removed● Long-term projects need to be supported for up
10-15 years
How do I deal with that?
11
Some figures
● 95 different platforms (native and cross)● 15 independent test suites
➢ 15000 tests and 10 million lines of code ➢ 1.4 million tests run each night
● 96 scripts to control the infrastructure➢ About 2300 scripts run each day
12
How to solve the equation?
● Adopting a lean strategy● Introducing agile tactics
Let's move on!
13
TPS
The Toyota Production System
Just-in-TimeJidoka
+
● Making only what is needed, when it's needed, in the amount needed
● Limited stocks with all parts
● Replace used parts from the preceding process
● Limited stocks for the preceding process
● Automatic defects detection
● Stop the line
● Identify the problem
● Andon (problem display board)
● An operator for several machines
14
Adopting a lean strategy
● Identify the value● Identify and remove waste (muda)● Automate● Detect defects early● Fix the cause of defects
15
The value
● Provide high quality software● Fixed release schedule● Provide pre-release versions as soon as
possible● Support new platforms● Add new features● Add new tools
16
Quality & Open-Source
● All AdaCore's products are Open-Source● It's possible to achieve high quality● AdaCore is an active member of the GCC
community● We identify a reasonably stable version of GCC● We run it through our QA process● We contribute our changes back to the community
17
Lean testing strategy
● A fundamental piece toward achieving high quality
● Goals:● Detect defects as early as possible● Adopt different testing tactics to minimize risk● Keep high level of productivity● Balance the machine load
18
How to detect defects?
1. Local testing
2. Mailserver
4. Continuous Builder
SVN
3. Check-in
5. Nightly builds
19
The mailserver system
● Email-based interface● To test patches● On a remote machine● Incremental build● Full testing even on exotic platforms● Clean result mandatory before check-in
Less than 20 minutes!
20
The continuous builder
● Every check-in triggers an automatic build● Fast machines
➔ quick feedback
● Stop-the-line: should an error be reported, immediate attention/action is required
● Limit the risk of nightly builds failures➔ Delays in intermediate releases delivery➔ More waste, less value➔ Customer satisfaction decreases
21
Nightly buildsSVN
Linux
Red Hat 5 SuSe 9 XP VistaServer
Windows
Packaging
Building
Testing
22
Nightly builds
● For cross platforms➔ We test a different version of the target OS
everyday➔ Use of simulators
➔ More stability➔ Better performances
● Heavy use of virtual machines➔ Ease maintenance of obsolescent platforms
23
Lean production infrastructure
● Shippable software is produced every day● Defects are detected early● Allows to achieve high quality● Allows to deliver intermediate releases quickly
24
Official releases
● Only 2 releases a year➢ One major release (January/February)➢ One corrective release (June/July)
● Customers can plan their own release cycle in advance
● Intermediate releases are still available➢ Provide quick fixes➢ Allows to experiment with new features
25
Official releases
● Failed builds are relaunched● Manual review of each test suite report● Human sanity check of each package● Cross toolchains are tested on real boards
➢ 48h to run the ACATS on some cross platforms!
26
The release schedule
2009 2010
6.2 branch
Development branch
6.3 branch
6.2.0 Beta 6.3.0 Beta
6.2.2 Release
6.3.1 Release6.2.1 Release
6.3.2 Release
27
New features and known problems
● New features are advertised to everybody on our web site
● Known problems are listed on the customer web interface
● Improve traceability● Provide easy access to workarounds● Raises interest in evolution of the technology
28
Visual management with GAIA
● Monitor scripts● Monitor machines● Monitor test suites● Submit analysis
● Everything is stored in a database
● Human-readable weather reports
● Use of the django framework
29
GAIA: How does it work?
Test machines
DB
GAIA Server GAIA User's Interface
30
GAIA: Test suite overview
31
GAIA: Weather reports
32
Lean Software Qualification
33
Critical software certification
● Aerospace● ECSS-E-ST-40C/ECSS-Q-ST-80C
● Civil Avionics● DO-178B/ED-12B
● Air Traffic Management● DO-278B
● Security● Common Criteria
● Railway● EN-50126/EN-50128/EN-50129
● Others...
34
DO-178B/ED-12B
● “DO-178B, Software Considerations in Airborne Systems and Equipment Certification”
● Last revision: December 1, 1992● DO-178C is in the pipeline● 5 Design Assurance Levels (DAL)
● A => Catastrophic● B => Hazardous● C => Major● D => Minor● E => No effect
35
What is in DO-178B?
● Process oriented:● Planning● Development● Verification● Configuration management● Quality assurance● Certification liaison
36
DO-178B: The Waterfall Model
Source: Wikipedia
Not very agile...
37
DO-178B: The V-Model
Not agile either...
Source: Wikipedia
38
Certification and Qualification
● Embedded software is certified● A tool is qualified
● As a development tool– Output is part of the airborne software– Can introduce errors– Certification-like process
● As a verification tool– May fail to detect an error– Lightweight process– Tool Operational Requirements– Requirements based testing
39
Qualification of verification tools
● A tool may be reused in different contexts● Operational requirements may change● Different part of the tool may be used in
different context● The tool may evolve
Let's do agile qualification!
40
The qualification machine
● Based on FitNesse● Centralize all qualification artifacts● Ensure consistency between requirements, test
cases and test data● Automatic generation of qualification
documentsIt's now easy
to add new requirements and associated tests!
41
The qualification machine
Let's see it in action!
42
References
● Lean Software Development, An Agile Toolkit by Mary and Tom Poppendieck
● Implementing Lean Software Development by Mary and Tom Poppendieck
● Lean Software Strategies, Proven Techniques for Managers and developpers by and Peter Middelton and Jim Sutton
● http://www2.toyota.co.jp/en/vision/production_system/