IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 1 of 14
IINNFFOO RRMMAATTIIOONN SS OOCCIIEETTII EESS TTEECCHHNNOO LLOO GGYY (( IISS TT)) PP RROO GGRRAAMMMM EE
IMPRESSION
“Initiative for Software Requirements Management Best Practice in Embedded Systems and E-Services Organisations”
DELIVERABLE EXTERNAL
WORKPACKAGE: WP2 – Implement systematic performance monitoring
D3-KPI Manual
Part I - General Methodology
Authors: C. Sorovou, D.Politou, A.Kalligeris, S.Topouzidou (ATC)
Submission Date: 5/10/2001
Partners: Athens Technology Center (GR), The City University (UK), University of Lancaster (UK). Members: Automate (GR), Integrated Aerospace Sciences Corporation (GR), Oxford Company (GR), International Carpet Center (GR), KOLIOS (GR), Euroelectronics (GR), Initiative Media (GR), PRISMA (GR), DISIOS (GR).
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 2 of 14
CONTENTS 1. INTRODUCTION..............................................................................................................................................................3
1.1. DEFINITIONS, ACRONYMS AND ABBREVIATIONS....................................................................................................3 1.2. LIST OF RELATED DOCUMENTS..................................................................................................................................3
2. METHODOLOGY FOR DEFINING, DEVELOPING, USING AND MAINTENING KPIS .....................3 2.1. GENERAL PRINCIPLES..................................................................................................................................................3 2.2. DEVELOPMENT AND USE OF KPIS ..............................................................................................................................3 2.3. METHODOLOGY AT A GLANCE....................................................................................................................................3
3. APPENDIX A. INDICATIVE PROJECT TEAM S TRUCTURE........................................................................3
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 3 of 14
1. INTRODUCTION
The purpose of the Key Performance Indicators (KPIs) Manual developed in the context of the IMPRESSION project is to assist the IMPRESSION members to identify, develop and measure KPIs, in order to acquire the benefits of best practice requirements management in software development for embedded systems & systems for e-business and e-services.
The manual is designed to be used by the IMPRESSION members as a practical guide to the process of introducing and using KPIs, following a methodology for their implementation.
This document describes the general methodology to be followed for defining, developing, measuring, evaluating and reporting KPIs. The Application of the IMPRESSION KPI Manual methodology for each user organisation can be found in the respective Part II documents.
1.1. Definitions, Acronyms and Abbreviations
KPI: Key Performance Indicator
CSF: Critical Success Factor
1.2. List of Related Documents
IST-2000-28760 IMPRESSION Project Annex I – Description of Work
D2- Yardsticks for each user
D3-KPI Manual Part II - Applying the KPI Manual
Ausindustries, “Key Performance Indicators Manual”, Pitman Publishing, Melbourne, 1996.
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 4 of 14
2. METHODOLOGY FOR DEFINING, DEVELOPING, USING AND MAINTENING KPIs
This section provides an overview of the total process for developing and using KPIs. It describes the general principles regarding the KPIs as well as the eight steps to be followed in the process of development and use of them.
2.1. General Principles
Key Performance Indicators are the measures that monitor the performance of 'key result areas’ of business activities, which are absolutely critical to the success and growth of the business.
The development and use of KPIs should form the basis for the analysis of an organisation’s current performance, its future requirements and the improving strategies required for ongoing success.
Some general principles regarding the KPIs should be taken under consideration during their application:
• KPIs should not be an end in themselves, but be considered as an aid to management. They are a start to a proper informed debate that should lead to a plan for improvement.
• They should be seen within their local context and have more a meaning as a comparison over time than as a comparison between organisations.
• A set of performance indicators should be balanced. For example, measures of efficiency should be set against measures of effectiveness and measures of cost against quality and user perception.
• After being proposed and applied, KPIs should be reviewed and updated. The review determines the management utility of each indicator and the feasibility of getting source data for continuing use.
• The targeted performance description, which is described in measurable terms through the KPIs, must be deployed to the organisational level that has the authority and knowledge to take the necessary action.
Every initial proposal for KPIs is expected to be imperfect. However, it is important for the organisations to understand and apply the appropriate KPIs so that they can develop some experience in using them, from which they can derive real expertise.
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 5 of 14
2.2. Development and use of KPIs
There is a nine steps approach in the procedure of development and use of KPIs, which is divided into four phases: Initiate, Build, Implement and Review.
PHASE I - INITIATE
STEP 1: Create the project team
It is proposed that a cross-functional team will be set up to implement the KPIs measurement. The team must be consisted of high- level employees, coming from different sections that have a clear view of the organisation’s goals and priorities. The team must also be familiar with the performance measures or be given a short training course, otherwise. The number of the team members depends on the size of the organisation and may vary between 4 to 6 employees. An indicative structure of the Project team is presented in Appendix A.
STEP 2: Align KPIs development with key areas and improvement strategies in requirements engineering
A strong correspondence must be established between the key areas, the Business Goals to be achieved and the improvement strategies to be followed, as KPIs will be developed to support these strategies. Therefore, the fist step is the definition of the key areas and the respective Business Goals, which are critical for the organisation, so that they can be aligned with the KPIs we intend to develop. Such Business Goals may be:
§ Improve RE Process Maturity § Improve Managerial Satisfaction § Improve User Satisfaction § Improve Efficiency of the System Implementation Process § Improve Maintainability of System § Improve Productivity of S/W development
The KPIs should focus on simple ideas that represent key goals of high relevance to the organisation in terms of the project assessed to ensure that the organisation comprehends the link between them and the business strategy.
STEP 3: Explain the purpose of the use and development of KPIs
The aim of measuring an organisation’s performance is to assess how well it is implementing strategies and action plans. To improve an organisation’s performance you must measure it first.
It is therefore necessary to develop KPIs, which is the set of indicators that will monitor the performance of 'key result areas’ of business activities. KPIs are a significant tool to assist improvement towards organisational objectives aiming to the
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 6 of 14
overall prosperity. KPIs also create an essential feedback and learning mechanism to support management decisions.
The purpose of the use and development of KPIs must be disseminated in the organisation, from the lower to the higher- level employees in order to make them understand their own role in helping to achieve the organisation’s goals and the measurement and improvement process.
STEP 4: Establish an agreed process for KPIs development and use.
After having created the team that will be responsible for the KPIs’ development, defined the key goals, which have to be measured and therefore aligned with the KPIs, and communicated and disseminated the purpose of KPIs implementation, an agreed development and use process must be established. The process should include:
• Definition of the team members’ duties
• For each KPI that will be selected by the project team, a set of Input, Calculate, Output and Report Procedures should be reported during the implementation phase as follows:
Input involves all procedures for collecting data and the parameters used for evaluating KPIs.
Calculate involves all mechanism and calculation required for evaluating KPI, from given Input data.
Output involves all processing and presentation of results e.g. graphical forms, data etc.
Report involves all reporting Requirements for monitoring RE procedure improvement. It may consist of a composite report where progress is shown through a number of Output formats
PHASE II - BUILD
STEP 5: Identify critical success factors for the organisation.
Critical success factors are the core competencies or capabilities that must be accomplished for the organisation to achieve its vision. The set of critical success factors related to the process of requirements engineering and the Business Goals already defined, may include the following:
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 7 of 14
Business Goals Critical Success Factors
Improve RE Process Maturity § Use of RE Standards
Improve Managerial Satisfaction
§ Improve Traceability of requirements throughout the System Lifecycle
§ Reduce Uncertainty throughout the System Lifecycle
Improve User Satisfaction § Reduce Users’ (internal – external) Complaints
Improve Efficiency of the System Implementation Process
§ Get things right in the first place § Reduce time to market § Implementation within the time table § Keeping within budget
Improve Maintainability of System § Improve extentability § Reduce maintenance cost
Improve Productivity of S/W development
§ Reduce code production related to rework
§ Reduce Effort of Rework
STEP 6: Teams define yardsticks and select KPIs.
Following the definition of Business Goals and critical success factors, a set of yardsticks should be defined. The yardsticks will specify the practical result that each organisation and the project should achieve according to the specified Business Goals. They will also serve in monitoring and evaluating the process towards the achievement of these goals.
A set of Yardsticks for the Business Goals and the Critical Success Factors defined, may be:
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 8 of 14
Critical Success Factors Yardsticks
Use of RE Standards § The increased number of RE Standards
Improve Traceability of requirements throughout the System Lifecycle
§ The Existence of Requirements Document
§ The increased number of unique requirements in Requirements Document
§ A high percentage of unique requirements
Reduce Uncertainty throughout the System Lifecycle
§ A low percentage of reworked requirements
Reduce Users’ (internal – external) Complaints
§ The decreased number of modifications coming from RE errors
Get things right in the first place
§ A high percentage of requirements satisfied in the initial design
Reduce time to market § The minimization of time between system conception and delivery
Implementation within the time table
§ The minimization of extra time spent for the implementation
§ The minimization of extra time spent for the RE process
Keeping within budget § A low percentage of deviation from total budget
§ A low percentage of deviation from the RE budget
Improve extentability § The increased number of additional
requirements satisfied / number of additional requirements requested
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 9 of 14
Reduce maintenance cost § The lowering in cost of maintenance per 1000
lines of code
Reduce code production related to rework
§ The decreased number of code lines / person day spent on rework
Reduce Effort of Rework § The minimization of the number of person days
needed for rework due to requirements capturing inefficiency
KPIs should have a set of characteristics that will allow them to be effective, namely they should be:
§ Restricted to a few strategically important objectives.
§ Based on strategic goals, making general goals more specific.
§ Consistent, giving a balanced view, relating to financial performance, satisfaction, efficiency and improvement.
§ Uncomplicated and understood by those being measured.
§ Easy to collect data and calculate.
§ Dynamic (reviewed at least annually as a part of business planning, to ensure that they reflect business priorities).
§ Agreed, not imposed by management.
§ Communicated.
KPIs should not:
§ Conflict with other measures without prioritising.
§ Produce misleading information.
§ Be or viewed as trivial.
All the above characteristics should be taken under consideration for the process of the KPIs development.
Suggested KPIs related to the respective Yardsticks, Critical Success Factors and Business Goals are:
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 10 of 14
Yardsticks Key Performance Indicators
The increased number of RE Standards Number of RE Standards used
The Existence of Requirements Document Existence of Requirements Document
The increased number of unique requirements in Requirements Document
Number of Unique Requirements
A high percentage of unique requirements % Unique Requirements completed
A low percentage of reworked requirements % Requirements to be reworked
The decreased number of modifications coming from RE errors
Number of modifications coming from RE errors / month
A high percentage of requirements satisfied in the initial design
% Requirements satisfied in the initial design
The minimization of time between system conception and delivery
Time between system conception and delivery
The minimization of extra time spent for the implementation
Number of extra months spent for the implementation
The minimization of extra time spent for the RE process
Number of extra months spent for the RE process
A low percentage of deviation from total budget
% Deviation from total budget
A low percentage of deviation from the RE budget
% Deviation from the RE budget
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 11 of 14
The increased number of additional requirements satisfied / number of additional requirements requested
Number of additional requirements satisfied / number of additional requirements requested
The lowering in cost of maintenance per 1000 lines of code
Cost of maintenance per 1000 lines of code
The decreased number of code lines / person day spent on rework
Lines of code/person day spent on rework
The minimization of the number of person days needed for rework due to requirements capturing inefficiency
Number of person days needed for rework due to requirements capturing inefficiency
PHASE III - IMPLEMENT
STEP 7: Develop display, reporting and review frameworks at all levels.
The KPIs should be displayed and reported frequently. The KPIs reporting should show management the trend. Thus, graphics going back are considered a good practice. Such a daily or weekly display will not take very long once the process is embedded.
The measurements collected should be analysed and form the basis for the final report.
STEP 8: Facilitate the use of KPIs to assist performance improvement
Education and training will facilitate the use of KPIs inside the organisation. Employees should be motivated and empowered by giving them controlled authority and responsibility for improvement, in relation to the processes they are responsible for. Managers should be encouraged to delegate responsibility and workers to identify their own measures and look for solutions.
PHASE IV - REVIEW
STEP 9: Refine and modify KPIs to retain their relevance.
The KPIs should be reviewed every time the critical success factors are reviewed. They also KPIs should be checked in terms of their relevance and be adapted to the eventually changing environment, processes and conditions.
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 12 of 14
Executives must be actively involved in reviewing the measures and leading the implementation effort. Continuous communication regarding the status and results of the measurement system is important in terms of cont inually improving the system.
2.3. Methodology at a glance
The overall methodology is represented in the figure below:
STEP 9 : Refine and modify KPIs to retain
their relevance
STEP 8 : Facilitate the use of KPIs to assist
performance improvement
STEP 7 : Develop display, reporting and review frameworks at
all levels
STEP 6 : Teams define yardsticks and select
KPIs
STEP 5 : Identify critical success factors
for the organisation
STEP 1: Create the project team
STEP 2: Align KPIs development with key
areas and improvement strategies
in requirements engineering
STEP 3 : Explain the purpose of the use and development of KPIs
STEP 4 : Establish an agreed process for
KPIs development and use
Initiate
Build
Implement
Review
STEP 9 : Refine and modify KPIs to retain
their relevance
STEP 8 : Facilitate the use of KPIs to assist
performance improvement
STEP 7 : Develop display, reporting and review frameworks at
all levels
STEP 6 : Teams define yardsticks and select
KPIs
STEP 5 : Identify critical success factors
for the organisation
STEP 1: Create the project team
STEP 2: Align KPIs development with key
areas and improvement strategies
in requirements engineering
STEP 3 : Explain the purpose of the use and development of KPIs
STEP 4 : Establish an agreed process for
KPIs development and use
STEP 9 : Refine and modify KPIs to retain
their relevance
STEP 8 : Facilitate the use of KPIs to assist
performance improvement
STEP 7 : Develop display, reporting and review frameworks at
all levels
STEP 6 : Teams define yardsticks and select
KPIs
STEP 5 : Identify critical success factors
for the organisation
STEP 1: Create the project team
STEP 2: Align KPIs development with key
areas and improvement strategies
in requirements engineering
STEP 3 : Explain the purpose of the use and development of KPIs
STEP 4 : Establish an agreed process for
KPIs development and use
Initiate
Build
Implement
Review
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 13 of 14
3. Appendix A. Indicative Project Team Structure
The project team will be consisted of high- level employees coming from different departments of the organisation, namely:
The Co-ordinator: High ranking official who will undertake the coordination of the team. He has an overall view of the project and he will communicate with the team involved in the project for the definition of KPIs. He will also be responsible for the measurements.
General Manager: He has an overall view of the organisation and he will set the business goals and discuss the priorities and the critical success factors with the rest members of the team, in order to proceed with the definition of the KPIs.
The Production / Technical Manager or high-ranking representative: He coordinates the resources and activities required for the production (in the case of industrial production companies) and the project implementation process (in the case of IT companies). He will be responsible for the data obtained, related the production & the IT implementation process.
The IT Manager or high-ranking representative: He will provide all the necessary knowledge about the technical part that has to be measured. He will also communicate with the software engineers and the developers during the process of the definition, the implementation and the dissemination of KPIs.
The Marketing & Sales Manager or high-ranking representative: He is responsible for the organisation’s promotion, the competition and the improvement of the organisation’s market stake. He will ensure that the decisions made are according to the marketing strategies.
The Financial Manager or high-ranking representative : He will define and monitor the critical financial figures.
Quality Manager or high-ranking representative: He is responsible for the quality assurance of the organisation. He ensures that the procedures followed during the KPIs definition & implementation procedure are according to the organisation’s predefined quality assurance procedures.
All team members will participate in the process of definition, implementation, reporting and reviewing of the KPIs. The indicative team structure is represented in the figure below:
IMPRESSION 0103F07_KPI Manual_Part I
IST-2000-28760 Page 14 of 14
Co-ordinator
Financial Manager
GeneralManager
ITManager
Marketing & Sales
Manager
QualityManager
Customer
Implement
Report Review
Define
ProductionManager
Co-ordinator
Financial Manager
GeneralManager
ITManager
Marketing & Sales
Manager
QualityManager
Customer
Implement
Report Review
Define
ProductionManager