Wojciech Sliwinski [email protected] Beams Department,
Controls Group CERN
Slide 2
Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 2Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
Slide 3
Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 3Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
Slide 4
Software Quality (SWQ) We know its a good thing People think I
know it when I see it I know the value of it But what are the
actual characteristics of SWQ? Do we have a common understanding of
SWQ? The lack of it normally appears as consequences 424th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 5
1985: Therac-25 Software issues Radiation Deaths linked to
Software Errors Killed 2 people Injured dozens of others
http://www.ccnr.org/fatal_dose.html 524th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
Slide 6
1996: Ariane 5 floating point conversion error 624th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 7
2007: Airbus A340 Software issues 724th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
Slide 8
80s The Kano Model (Marketing) Customer Satisfaction Product
Features Basic Needs Met Linear Performance Delight, Surprise,
Innovate 24th October 20118Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN Satisfied Dissatisfied Needs
well fulfilled Needs not fulfilled
Slide 9
Early and Other Quality Definitions Zen and the art of
motorcycle maintenance, Robert Pirsig, 1974: If quality exists in
an object, then you must explain why scientific instruments are
unable to detect it On the other hand, if quality is subjective,
[existing only in the eye of the observer] then this Quality is
just a fancy name for whatever you'd like. Quality is not
objective. It doesn't reside in the material world [...] Quality is
not subjective. It doesn't reside merely in the mind. McCall et al,
1977 and Boehm et al, 1978 First elaborate studies on the notion of
'software quality Quality factors only indirectly measurable like
reliability Quality criteria objective aspects that support the
factors, like uptime. 924th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN
Slide 10
IEEE SWQ Standards IEEE Glossary of Software Engineering
Terminology defines it as the degree to which a system, component,
or process meets customer or user needs or expectations IEEE Std
1061-1992 IEEE Standard for a Software Quality Metrics Methodology
Does not actually prescribe any metrics! Defines method to evaluate
metrics that could be applicable to a particular case Plenty more
eg. IEEE Std 730 and friends 1024th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN
Slide 11
ISO Quality Standards ISO 9000 series of standards for quality
management systems: Process based the degree to which a set of
inherent characteristics fulfils the requirements ISO 9126 Software
Quality Characteristics and Metrics, 2001 Comprehensive but large
& complex 6 key quality attributes: Functionality, Reliability,
Usability Efficiency, Maintainability, Portability 1124th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 12
SWQ Huge Set of Characteristics 12 Only a subset applies to all
projects Choose those that suit the project goals 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 13
McCalls categories of SWQ factors Source: J.A. McCall, P.K.
Richards and G.F. Walters, Factors in Software Quality,
RADC-TR-77-369, 1977, US Dep. of Commerce. 13 Operation:
CorrectnessDoes it do what I want? ReliabilityDoes it do it
accurately all of the time? EfficiencyWill it run on my hardware as
well as it can? IntegrityIs it secure? UsabilityCan I run it?
Revision: MaintainabilityCan I fix it? TestabilityCan I test it?
FlexibilityCan I change it? Transition: PortabilityWill I be able
to use it on another machine? ReusabilityWill I be able to reuse
some of the software? InteroperabilityWill I be able to interface
it with another system? 24th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN
Slide 14
Internal & External Quality Attributes 14 Important
Primarily to UsersImportant Primarily to Developers
AvailabilityMaintainability EfficiencyPortability
FlexibilityReusability IntegrityTestability Interoperability
Reliability Robustness Usability 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
Slide 15
The SW Quality Iceberg 1524th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN
Slide 16
The Maintainability Index (MI) ParameterNameMeasures
aveVAverage Halstead complexityComputational density aveV
(g)Average extended cyclomatic complexityLogical complexity
aveLOCAverage count of lines of codeCode size PerCMAverage percent
of lines of commentsHuman insight 16 MI = 171 5.2 x ln(aveV) - 0.23
x aveV(g) - 16.2 x ln(aveLOC) + 50 x sin 2.4PerCM 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 17
MI applied to FreeBSD codebase 1724th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
Kernel CodebaseUser Programs Modules with MI < 40 are rejected
!
Slide 18
Applying Quality Computers dont care about quality! Metrics are
tests for quality characteristics Quality clusters Testing is
ensuring a quality of conformance Bugs occur in clusters too! Like
testing, if quality is low, builds should fail Quality
characteristics must be part of the whole development lifecycle
Quality products only come from quality requirements, quality
design, quality code, 24th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN18
Slide 19
Applying Quality (Assurance) Software Quality consists of: 1.
Engineering Methods (proper analysis, design, ) 2. Standards and
Procedures (common to all) 3. Technical Reviews (eg. code reviews)
4. Configuration management (repositories, versioning) 5.
Measurement (code, metrics) 6. Testing (unit, system, integration,
acceptance) 7. Improvement (of the items above) 1924th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN Quality is a way of life If you think Quality is
expensive, try an accident
Slide 20
Capability Maturity Model (CMM) Levels * 1. Initial (chaotic,
ad hoc, individual heroics) the starting point for use of a new
process. 2. Repeatable some processes are repeatable, possibly with
consistent results 3. Defined the process is defined/confirmed as a
standard business process, and decomposed to levels 1 and 2 4.
Managed (quantatively) using process metrics, management can
identify ways to adjust and adapt the process to particular
projects 5. Optimized process management includes deliberate
process optimization/improvement. * Capability Maturity Model from
SE Institute. 2024th October 2011Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN
Slide 21
Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 21Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
Slide 22
Continuous Improvement (CI) Origins Demings Plan-Do-Check-Act
(PDCA) The Toyota Production System Stop the line quality control
About eliminating waste (obstacles) A learning, non-judgmental
whole team approach Adopted by the Agile movement for SWD Caveat:
Development is an empirical process Introduced when an organization
recognizes it has arrived at some impasse 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN22
Slide 23
CI - How it works People (at all levels) freely discuss the
information, issues, ideas, evaluate them, choose, plan and execute
improvements Focuses on how things get done: Standardized (and ing)
common tasks Only fixing possible issues e.g.: repeated mistakes,
time sinks, quality holes Very suitable where tools, technology,
external outcomes and requirements change often 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN23
Slide 24
CI - What it relies on Objective information: To evaluate
current situation and processes Assess applied improvements Judge
outcome as worthwhile, or retry Demonstrate added value of CI
against overhead Improvement culture: Free speech everyones ideas
are welcome A real desire to try it and improve Accepting there
will not be total agreement Replacing skeptical inaction with real
experimentation 24th October 2011Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24
Slide 25
CI It can fail! No one owns responsibility for implementation
No charter to make changes No time allocated for improvement No
follow up by team or sponsors 24th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN25
Slide 26
Improvement Practices Change Agents AKA: Coaches, Monitors,
Consultants, Champions One or more floating people searching for
improvements Kaizen time aka renovation days Team plans some
improvements All actions executed by all at same time Reflection
workshops Regular meetings to decide on improvements Apply one or
more ideas Keep what works 24th October 2011Wojciech Sliwinski: CI
and QA for the Accelerator Controls Codebase at CERN26
Slide 27
CI - Summary Team driven incremental changes Team evolves and
improves together Drives increase in quality, efficiency and
satisfaction Not a tool or technique it is an attitude Not leader
oriented, instead peer based consensus Keeps us ahead in best
practice and professionalism 24th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN27
Slide 28
Bibliography Software Requirements K. Wiegers, Microsoft Press,
2003 Code Complete: A Practical Handbook of Software Construction
S. McConnell, Microsoft Press, 2004 Software Engineering:
Principles and Practice H. van Vliet, John Wiley & Sons, 2008
Wikipedia page on SWQ ISO standard 9126. 2824th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 29
Bibliography The Toyota Way, J. K. Liker, 2004 Agile and
Iterative Development, C. Larman, 2004 Sustainable Software
Development, Kevin Tate, 2006 Implementing Lean Software
Development, M. & T. Poppendick, 2007 Rapid Development, S.
McConnell, 1996 The Enterprise and Scrum, K. Schwaber, 2007
Collaboration Explained, J. Tabaka, 2006 Peopleware Papers: The
Human side of Software, L. Constantine, 2001 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN29
Slide 30
Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 30Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
Slide 31
Context - CERN Accelerator Complex 24th October 201131Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
Slide 32
Context - The CERN Control Centre 24th October 201132Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
Slide 33
Controls Software Infrastructure Client tier Server tier
Resource tier CTRL DB Business Layer Device Layer Presentation
Layer CMW A 3-tier architecture Presentation Layer Graphical
interactive applications Fixed Displays Business Layer General
purpose and specific Application servers Device Layer Front End
Device servers Communication to the equipment goes through Controls
MiddleWare CMW 24th October 201133Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN
Slide 34
Controls Software Codebase - Situation Complex Accelerator
domain ~6 million LOC Java/J2EE ~200 developers (CERN + other labs)
~1000 projects/modules C/C++ ~35 developers (CERN + other labs) +20
major projects Oracle database Eclipse IDE + plugins SVN + Maven +
CommonBuild( Atlassian Suite (JIRA, Fisheye, Crucible, etc.)
Nowadays encourage Scrum Focused on Software Quality 34Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN24th October 2011
Slide 35
Controls Software Codebase - Situation Lots of Code ~6 million
lines, 20000+ classes. and still growing 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN35
Slide 36
Situation Self Assessment and Outlook Unknown quality Quality
unchecked No mentoring or guidance? Limited standards High Staff
turnover Students, Project Associates, Fellows More projects coming
(PS renovation, ) More service provision (looking after running
services) Will this cause future problems like maintenance? 24th
October 2011Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN36
Slide 37
Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 37Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
Slide 38
Objectives improve Software Quality Introduce quality
improvement as an integral part of the everyday development work
Leverage tools to automate the process as much as possible
Establish guidelines and metrics to measure progress Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN3824th October 2011
Slide 39
Quality in the Development Cycle For each stage Agreed
mandatory activities and deliverables Tools identified and
integrated with IDE (Eclipse), giving immediate feedback Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN39 Design Implement, Test, Document Deploy, Maintain 24th
October 2011
Slide 40
Design Reviews Design meetings with external members To verify
the soundness of design in an early stage To identify overlapping
functionality Promote a set of common components and libraries
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase
at CERN40 At the level of sub- components, main classes and design
patterns UML: class and sequence diagrams Design Implement, Test,
Document Deploy, Maintain 24th October 2011
Slide 41
Code Reviews & Code Analysis Goal: Identify defects Ensure
maintainable code Verify conventions are followed Static code
analysis tools to identify common mistakes and bug patterns:
FindBugs Checkstyle Eclipse warnings Wojciech Sliwinski: CI and QA
for the Accelerator Controls Codebase at CERN41 Person-to-person
time consuming Only for critical code Mentoring of junior
developers Light-weight person-to- person code reviews with FishEye
+ Crucible Design Implement, Test, Document Deploy, Maintain 24th
October 2011
Slide 42
Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN42 Files being reviewed Author and reviewers
Comments inline 24th October 2011
Slide 43
Reviewing Code (Crucible) 43
Slide 44
Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN44 A list of bugs The bug line indicated Bug
explained 24th October 2011
Slide 45
Testing & Continuous Integration General agreement on
benefits of testing and CI To increase level of testing, unit tests
mandatory deliverable of each project >30% test coverage for
non-trivial classes, measured with Clover Wojciech Sliwinski: CI
and QA for the Accelerator Controls Codebase at CERN45 To discover
inter-project issues early, Continuous Builds with Bamboo: CI
server from Atlassian Compiles and runs unit tests Builds fail on:
Low test coverage Basic PMD rules Design Implement, Test, Document
Deploy, Maintain 24th October 2011
Slide 46
Continuous Integration and Test (Bamboo) 46 Test coverage for
project Code metrics Classes with highest risk
Slide 47
Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN47 Red = not covered Green= covered 24th October
2011
Slide 48
Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN48 Triggered by changes in a dependency accsoft-
commons- core accsoft- common s-util accsoft- commons-
concentration accsoft- commons- io Top/flop list generated (Work in
progress) 24th October 2011
Slide 49
Integration and System Testing with CO Testbed CO Testbed -
mini-accelerator lab Hardware and Software Core components from the
Accelerators Controls duplicated into this Test Environment System
and Integration tests are executed by Bamboo CI server Automatic
execution of tests with real system Automatic deployment of Release
Candidates 4924th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
Slide 50
TIMING FEC03 FEC01 FEC02 FEC04FEC05 SERVER06 SERVER07 5024th
October 2011 Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN
Slide 51
Integration Tests done through Client APIs 51 Bamboo CI Server
CMW Proxy CMW Directory Svc RBAC A1 Service Config DB Client APIs
(JAPC, CMW, RBAC) Timing CMW, RBAC A2, FESA PPC/LynxOSLinux/i386
TESTS Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN JMS 24th October 2011
Slide 52
CO Testbed controlled by Bamboo CI server 24th October
201152Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
Slide 53
53 1. 24th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
Slide 54
54 1. 2. 24th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
Slide 55
5524th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
Slide 56
Challenges Think of and organize us as one big team, not many
small ones The code belongs to the group, not to a project or an
individual developer The guidelines and standards established and
agreed by everyone Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN56 Challenges Agree on
standards and configurationsAgree on standards and configurations
24th October 2011
Slide 57
Encourage developers to invest time on quality Quality-related
tasks part of the yearly objectives for a project Progress measured
against the level a project adheres to guidelines top/flop lists
Focus the effort where the effect is highest, rely on tools as much
as possible CI days common days with a common goal Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN57 Challenges 24th October 2011
Slide 58
Conclusions Integrated quality improvement in the development
cycle Introduced guidelines/standards and supporting tools
Increased developer motivation through Immediate feedback when
developing Official tracking of progress and top/flop lists
Increased awareness of benefits, application in individual projects
and confidence when making changes Agile approach reflect, identify
waste and improve Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN5824th October 2011