Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department, Controls Group CERN
Preview:
Citation preview
- Slide 1
- Wojciech Sliwinski Wojciech.Sliwinski@cern.ch Beams Department,
Controls Group CERN
- Slide 2
- Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 2Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
- Slide 3
- Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 3Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
- Slide 4
- Software Quality (SWQ) We know its a good thing People think I
know it when I see it I know the value of it But what are the
actual characteristics of SWQ? Do we have a common understanding of
SWQ? The lack of it normally appears as consequences 424th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 5
- 1985: Therac-25 Software issues Radiation Deaths linked to
Software Errors Killed 2 people Injured dozens of others
http://www.ccnr.org/fatal_dose.html 524th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
- Slide 6
- 1996: Ariane 5 floating point conversion error 624th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 7
- 2007: Airbus A340 Software issues 724th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
- Slide 8
- 80s The Kano Model (Marketing) Customer Satisfaction Product
Features Basic Needs Met Linear Performance Delight, Surprise,
Innovate 24th October 20118Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN Satisfied Dissatisfied Needs
well fulfilled Needs not fulfilled
- Slide 9
- Early and Other Quality Definitions Zen and the art of
motorcycle maintenance, Robert Pirsig, 1974: If quality exists in
an object, then you must explain why scientific instruments are
unable to detect it On the other hand, if quality is subjective,
[existing only in the eye of the observer] then this Quality is
just a fancy name for whatever you'd like. Quality is not
objective. It doesn't reside in the material world [...] Quality is
not subjective. It doesn't reside merely in the mind. McCall et al,
1977 and Boehm et al, 1978 First elaborate studies on the notion of
'software quality Quality factors only indirectly measurable like
reliability Quality criteria objective aspects that support the
factors, like uptime. 924th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN
- Slide 10
- IEEE SWQ Standards IEEE Glossary of Software Engineering
Terminology defines it as the degree to which a system, component,
or process meets customer or user needs or expectations IEEE Std
1061-1992 IEEE Standard for a Software Quality Metrics Methodology
Does not actually prescribe any metrics! Defines method to evaluate
metrics that could be applicable to a particular case Plenty more
eg. IEEE Std 730 and friends 1024th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN
- Slide 11
- ISO Quality Standards ISO 9000 series of standards for quality
management systems: Process based the degree to which a set of
inherent characteristics fulfils the requirements ISO 9126 Software
Quality Characteristics and Metrics, 2001 Comprehensive but large
& complex 6 key quality attributes: Functionality, Reliability,
Usability Efficiency, Maintainability, Portability 1124th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 12
- SWQ Huge Set of Characteristics 12 Only a subset applies to all
projects Choose those that suit the project goals 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 13
- McCalls categories of SWQ factors Source: J.A. McCall, P.K.
Richards and G.F. Walters, Factors in Software Quality,
RADC-TR-77-369, 1977, US Dep. of Commerce. 13 Operation:
CorrectnessDoes it do what I want? ReliabilityDoes it do it
accurately all of the time? EfficiencyWill it run on my hardware as
well as it can? IntegrityIs it secure? UsabilityCan I run it?
Revision: MaintainabilityCan I fix it? TestabilityCan I test it?
FlexibilityCan I change it? Transition: PortabilityWill I be able
to use it on another machine? ReusabilityWill I be able to reuse
some of the software? InteroperabilityWill I be able to interface
it with another system? 24th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN
- Slide 14
- Internal & External Quality Attributes 14 Important
Primarily to UsersImportant Primarily to Developers
AvailabilityMaintainability EfficiencyPortability
FlexibilityReusability IntegrityTestability Interoperability
Reliability Robustness Usability 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
- Slide 15
- The SW Quality Iceberg 1524th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN
- Slide 16
- The Maintainability Index (MI) ParameterNameMeasures
aveVAverage Halstead complexityComputational density aveV
(g)Average extended cyclomatic complexityLogical complexity
aveLOCAverage count of lines of codeCode size PerCMAverage percent
of lines of commentsHuman insight 16 MI = 171 5.2 x ln(aveV) - 0.23
x aveV(g) - 16.2 x ln(aveLOC) + 50 x sin 2.4PerCM 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 17
- MI applied to FreeBSD codebase 1724th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
Kernel CodebaseUser Programs Modules with MI < 40 are rejected
!
- Slide 18
- Applying Quality Computers dont care about quality! Metrics are
tests for quality characteristics Quality clusters Testing is
ensuring a quality of conformance Bugs occur in clusters too! Like
testing, if quality is low, builds should fail Quality
characteristics must be part of the whole development lifecycle
Quality products only come from quality requirements, quality
design, quality code, 24th October 2011Wojciech Sliwinski: CI and
QA for the Accelerator Controls Codebase at CERN18
- Slide 19
- Applying Quality (Assurance) Software Quality consists of: 1.
Engineering Methods (proper analysis, design, ) 2. Standards and
Procedures (common to all) 3. Technical Reviews (eg. code reviews)
4. Configuration management (repositories, versioning) 5.
Measurement (code, metrics) 6. Testing (unit, system, integration,
acceptance) 7. Improvement (of the items above) 1924th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN Quality is a way of life If you think Quality is
expensive, try an accident
- Slide 20
- Capability Maturity Model (CMM) Levels * 1. Initial (chaotic,
ad hoc, individual heroics) the starting point for use of a new
process. 2. Repeatable some processes are repeatable, possibly with
consistent results 3. Defined the process is defined/confirmed as a
standard business process, and decomposed to levels 1 and 2 4.
Managed (quantatively) using process metrics, management can
identify ways to adjust and adapt the process to particular
projects 5. Optimized process management includes deliberate
process optimization/improvement. * Capability Maturity Model from
SE Institute. 2024th October 2011Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN
- Slide 21
- Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 21Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
- Slide 22
- Continuous Improvement (CI) Origins Demings Plan-Do-Check-Act
(PDCA) The Toyota Production System Stop the line quality control
About eliminating waste (obstacles) A learning, non-judgmental
whole team approach Adopted by the Agile movement for SWD Caveat:
Development is an empirical process Introduced when an organization
recognizes it has arrived at some impasse 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN22
- Slide 23
- CI - How it works People (at all levels) freely discuss the
information, issues, ideas, evaluate them, choose, plan and execute
improvements Focuses on how things get done: Standardized (and ing)
common tasks Only fixing possible issues e.g.: repeated mistakes,
time sinks, quality holes Very suitable where tools, technology,
external outcomes and requirements change often 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN23
- Slide 24
- CI - What it relies on Objective information: To evaluate
current situation and processes Assess applied improvements Judge
outcome as worthwhile, or retry Demonstrate added value of CI
against overhead Improvement culture: Free speech everyones ideas
are welcome A real desire to try it and improve Accepting there
will not be total agreement Replacing skeptical inaction with real
experimentation 24th October 2011Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24
- Slide 25
- CI It can fail! No one owns responsibility for implementation
No charter to make changes No time allocated for improvement No
follow up by team or sponsors 24th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN25
- Slide 26
- Improvement Practices Change Agents AKA: Coaches, Monitors,
Consultants, Champions One or more floating people searching for
improvements Kaizen time aka renovation days Team plans some
improvements All actions executed by all at same time Reflection
workshops Regular meetings to decide on improvements Apply one or
more ideas Keep what works 24th October 2011Wojciech Sliwinski: CI
and QA for the Accelerator Controls Codebase at CERN26
- Slide 27
- CI - Summary Team driven incremental changes Team evolves and
improves together Drives increase in quality, efficiency and
satisfaction Not a tool or technique it is an attitude Not leader
oriented, instead peer based consensus Keeps us ahead in best
practice and professionalism 24th October 2011Wojciech Sliwinski:
CI and QA for the Accelerator Controls Codebase at CERN27
- Slide 28
- Bibliography Software Requirements K. Wiegers, Microsoft Press,
2003 Code Complete: A Practical Handbook of Software Construction
S. McConnell, Microsoft Press, 2004 Software Engineering:
Principles and Practice H. van Vliet, John Wiley & Sons, 2008
Wikipedia page on SWQ ISO standard 9126. 2824th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 29
- Bibliography The Toyota Way, J. K. Liker, 2004 Agile and
Iterative Development, C. Larman, 2004 Sustainable Software
Development, Kevin Tate, 2006 Implementing Lean Software
Development, M. & T. Poppendick, 2007 Rapid Development, S.
McConnell, 1996 The Enterprise and Scrum, K. Schwaber, 2007
Collaboration Explained, J. Tabaka, 2006 Peopleware Papers: The
Human side of Software, L. Constantine, 2001 24th October
2011Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN29
- Slide 30
- Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 30Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
- Slide 31
- Context - CERN Accelerator Complex 24th October 201131Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
- Slide 32
- Context - The CERN Control Centre 24th October 201132Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN
- Slide 33
- Controls Software Infrastructure Client tier Server tier
Resource tier CTRL DB Business Layer Device Layer Presentation
Layer CMW A 3-tier architecture Presentation Layer Graphical
interactive applications Fixed Displays Business Layer General
purpose and specific Application servers Device Layer Front End
Device servers Communication to the equipment goes through Controls
MiddleWare CMW 24th October 201133Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN
- Slide 34
- Controls Software Codebase - Situation Complex Accelerator
domain ~6 million LOC Java/J2EE ~200 developers (CERN + other labs)
~1000 projects/modules C/C++ ~35 developers (CERN + other labs) +20
major projects Oracle database Eclipse IDE + plugins SVN + Maven +
CommonBuild( Atlassian Suite (JIRA, Fisheye, Crucible, etc.)
Nowadays encourage Scrum Focused on Software Quality 34Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN24th October 2011
- Slide 35
- Controls Software Codebase - Situation Lots of Code ~6 million
lines, 20000+ classes. and still growing 24th October 2011Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN35
- Slide 36
- Situation Self Assessment and Outlook Unknown quality Quality
unchecked No mentoring or guidance? Limited standards High Staff
turnover Students, Project Associates, Fellows More projects coming
(PS renovation, ) More service provision (looking after running
services) Will this cause future problems like maintenance? 24th
October 2011Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN36
- Slide 37
- Outline Defining Software Quality Continuous Improvement
Context Accelerator Controls Codebase Applying QA & CI for the
Controls Codebase Conclusions 37Wojciech Sliwinski: CI and QA for
the Accelerator Controls Codebase at CERN24th October 2011
- Slide 38
- Objectives improve Software Quality Introduce quality
improvement as an integral part of the everyday development work
Leverage tools to automate the process as much as possible
Establish guidelines and metrics to measure progress Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN3824th October 2011
- Slide 39
- Quality in the Development Cycle For each stage Agreed
mandatory activities and deliverables Tools identified and
integrated with IDE (Eclipse), giving immediate feedback Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN39 Design Implement, Test, Document Deploy, Maintain 24th
October 2011
- Slide 40
- Design Reviews Design meetings with external members To verify
the soundness of design in an early stage To identify overlapping
functionality Promote a set of common components and libraries
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase
at CERN40 At the level of sub- components, main classes and design
patterns UML: class and sequence diagrams Design Implement, Test,
Document Deploy, Maintain 24th October 2011
- Slide 41
- Code Reviews & Code Analysis Goal: Identify defects Ensure
maintainable code Verify conventions are followed Static code
analysis tools to identify common mistakes and bug patterns:
FindBugs Checkstyle Eclipse warnings Wojciech Sliwinski: CI and QA
for the Accelerator Controls Codebase at CERN41 Person-to-person
time consuming Only for critical code Mentoring of junior
developers Light-weight person-to- person code reviews with FishEye
+ Crucible Design Implement, Test, Document Deploy, Maintain 24th
October 2011
- Slide 42
- Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN42 Files being reviewed Author and reviewers
Comments inline 24th October 2011
- Slide 43
- Reviewing Code (Crucible) 43
- Slide 44
- Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN44 A list of bugs The bug line indicated Bug
explained 24th October 2011
- Slide 45
- Testing & Continuous Integration General agreement on
benefits of testing and CI To increase level of testing, unit tests
mandatory deliverable of each project >30% test coverage for
non-trivial classes, measured with Clover Wojciech Sliwinski: CI
and QA for the Accelerator Controls Codebase at CERN45 To discover
inter-project issues early, Continuous Builds with Bamboo: CI
server from Atlassian Compiles and runs unit tests Builds fail on:
Low test coverage Basic PMD rules Design Implement, Test, Document
Deploy, Maintain 24th October 2011
- Slide 46
- Continuous Integration and Test (Bamboo) 46 Test coverage for
project Code metrics Classes with highest risk
- Slide 47
- Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN47 Red = not covered Green= covered 24th October
2011
- Slide 48
- Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN48 Triggered by changes in a dependency accsoft-
commons- core accsoft- common s-util accsoft- commons-
concentration accsoft- commons- io Top/flop list generated (Work in
progress) 24th October 2011
- Slide 49
- Integration and System Testing with CO Testbed CO Testbed -
mini-accelerator lab Hardware and Software Core components from the
Accelerators Controls duplicated into this Test Environment System
and Integration tests are executed by Bamboo CI server Automatic
execution of tests with real system Automatic deployment of Release
Candidates 4924th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
- Slide 50
- TIMING FEC03 FEC01 FEC02 FEC04FEC05 SERVER06 SERVER07 5024th
October 2011 Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN
- Slide 51
- Integration Tests done through Client APIs 51 Bamboo CI Server
CMW Proxy CMW Directory Svc RBAC A1 Service Config DB Client APIs
(JAPC, CMW, RBAC) Timing CMW, RBAC A2, FESA PPC/LynxOSLinux/i386
TESTS Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN JMS 24th October 2011
- Slide 52
- CO Testbed controlled by Bamboo CI server 24th October
201152Wojciech Sliwinski: CI and QA for the Accelerator Controls
Codebase at CERN
- Slide 53
- 53 1. 24th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
- Slide 54
- 54 1. 2. 24th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
- Slide 55
- 5524th October 2011Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN
- Slide 56
- Challenges Think of and organize us as one big team, not many
small ones The code belongs to the group, not to a project or an
individual developer The guidelines and standards established and
agreed by everyone Wojciech Sliwinski: CI and QA for the
Accelerator Controls Codebase at CERN56 Challenges Agree on
standards and configurationsAgree on standards and configurations
24th October 2011
- Slide 57
- Encourage developers to invest time on quality Quality-related
tasks part of the yearly objectives for a project Progress measured
against the level a project adheres to guidelines top/flop lists
Focus the effort where the effect is highest, rely on tools as much
as possible CI days common days with a common goal Wojciech
Sliwinski: CI and QA for the Accelerator Controls Codebase at
CERN57 Challenges 24th October 2011
- Slide 58
- Conclusions Integrated quality improvement in the development
cycle Introduced guidelines/standards and supporting tools
Increased developer motivation through Immediate feedback when
developing Official tracking of progress and top/flop lists
Increased awareness of benefits, application in individual projects
and confidence when making changes Agile approach reflect, identify
waste and improve Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN5824th October 2011