1

Dmitry Maslov www.am-one.ru | maslow@bk.ru

Quality

Management

System

Index

Change Management Tool

Dashboard for the CAF Users

2

The Quality Management System Index (QMS-Index) is based upon the Common Assessment Framework (CAF Model) and the EFQM Excellence Model. The QMS-Index can be implemented as a diagnostic technique and an initial total quality management (TQM) instrument. Attached questionnaire will help practitioners from public sector to conduct self-assessment and start to implement CAF/EFQM models in their organisations.

The Common Assessment Framework – a TQM Model inspired by the EFQM Excellence Model. Being a generic tool CAF includes the 9 criteria, 28 sub criteria and the scoring system. The structure of the CAF Model is illustrated on fig. 1. Further description of the CAF Model is given by CAF-2013 brochure, which is available online on the CAF website www.eipa.eu/CAF (as well as Russian language version).

Figure 1. The CAF Model Working with CAF for almost 10 years we have developed the Quality Management

System Index in order to help organisations in the beginning of their quality journey with the CAF Model. QMS-Index is a set of indicators characterizing the level of quality management system in organisation.

QMS-Index is calculated on the basis of self-assessment helps to identify good practices and areas for improvement in the public sector organisations and increase the level of quality of public services provided to citizens-customers.

One of the main QMS-Index feature is involving employees in the process of diagnostic self-assessment which covers various aspects of the organisation in the framework of desired and achieved level of quality management system.

QMS-Index is based upon criteria of the CAF Model and the EFQM Excellence Model.

Proposed system is not considering as a "punitive" tool or a mechanism of dismissal. The QMS-Index aims to establish channels of communication between management and employees for more accurate diagnosis and better adjustment of the management system.

3

QMS-Index structure consists of 9 indicators divided into two groups: Enables and Results. Each group has equal ratio - 50/50. Indicators within groups have weights, shown in Table 1:

Table 1. Wight coefficient of QMS-Index criteria

coefficient, k score Enables Criteria 0,50 50 Criterion 1: Leadership 0,10 10 Criterion 2: Strategy and Planning 0,08 8 Criterion 3: People 0,09 9 Criterion 4: Partnerships and Resources 0,09 9 Criterion 5: Processes 0,14 14 Results Criteria 0,50 50 Criterion 6: Citizen/Customer-oriented Results 0,20 20 Criterion 7: People Results 0,09 9 Criterion 8: Social Responsibility Results 0,06 6 Criterion 9: Key Performance Results 0,15 15

Self-assessment is conducted by non-personalized survey. Respondents fill in

paper questionnaire or on-line form, where the first part includes questions regarding importance of different areas of organisation, and the second part achieved level on the same areas (questionnaire is attached - Annex 1).

The e-survey can be created using an open application (e.g. Google Drive or SurveyMonkey) or developed an app upon special terms and conditions. Link to the survey has to be sent individually to the work email address of each respondent.

To ensure the validity of research results with a statistical error of less than 5% it is necessary to provide participation in the survey from 50 to 80 % of employees, depending on their total population in the organisation.

It is recommended to conduct self-assessment quarterly (or at least annually) to measure the dynamic of improvements.

QMS Dashboard based on consolidated and interrelated results of two part in the questionnaire, harmonized with each other. In the first part, respondents evaluate to what extent this or that area of organisation important for its further development – it will be desired profile of management system. In the second part, respondents form actual profile of management system evaluating the same areas in terms of achieved results.

Each line in the questionnaire is harmonized with QMS-Index criteria as shown in the table 2.

Table 2. Questionnaire and QMS-Index Criteria Relationship

QMS-Index Criteria Question No

Part 1 Part 2 Enables Criteria 1: Leadership 1; 2; 3; 4; 5; 7 50; 51; 52; 53; 54; 56 2: Strategy and Planning 2; 6; 8; 9; 10; 11; 12; 13; 14 51; 55; 57; 58; 59; 60; 61; 62; 63 3: People 11; 15; 16; 17; 18; 19; 20; 21;

22 60; 64; 65; 66; 67; 68; 69; 70; 71

4: Partnerships and Resources 13; 23; 24; 25; 26; 27; 28 62; 72; 73; 74; 75; 76; 77 5: Processes 10; 13; 24; 27; 28; 29; 30; 31;

32 59; 62; 73; 76; 77; 78; 79; 80; 81

4

QMS-Index Criteria Question No

Part 1 Part 2 Results Criteria 6: Citizen/Customer-oriented Results

31; 32; 33; 34; 35; 36; 37; 38 80; 81; 82; 83; 84; 85; 86; 87

7: People Results 30; 32; 39; 40; 41 79; 81; 88; 89; 90 8: Social Responsibility Results 35; 36; 42; 43; 44; 46 84; 85; 91; 92; 93; 95 9: Key Performance Results 45; 46; 47; 48; 49 94; 95; 96; 97; 98

Consolidated QMS-Index reflecting the current state of the quality management

system is calculated based on the evaluation result of the second part of the questionnaire by the following formulas:

where i – Criterion No n – Question No k – Weight coefficient (Table 1) qn – Score (0 – 5 points): qmax – Maximum score 5 points m – Total amount of respondents QEnables – Achieved score in “Enables” QResults – Achieved score in “Results” Q1…9 – Achieved score on criterion 1 (achieved score on criteria 2-9 is

calculated with regard to the relevant coefficients and questions indicated in Tables 1 and 2).

In order to evaluate effectiveness of organisation QMS-Effect could be calculated:

Both indicators: achieved QMS-Index and QMS-Effect are reflected at the top of the

QMS Dashboard (Figure 2).

5

Following the same algorithm desired level indicators of QMS-Index are calculated. Proposed self-assessment technique allows to define gaps between desired and achieved level of performance of organisation through the prism of 9 criteria, as well as within groups of Enables and Results. In addition, the gap between desired and achieved levels of consolidated QMS-Index can be shown (middle part of the Dashboard in Figure 2).

To define a level of maturity for each criterion it is suggested to use the scale similar to CAF methodology where the PDCA (Plan-Do-Check-Act) cycle is integrated into scoring system. If an organisation performs no data or only has plans in this or that area, that means only beginning phase of continues improvement. If organisation performs plan, do, check and act (review and improve), that means high level of maturity. If organisation has fully integrated PDCA cycle – it can be consider as a best practice. PDCA scale is applicable to Enable group. Maturity of Results criteria is characterized by trends shown by arrows.

For in-depth analysis of the achieved level (results of the second part of the questionnaire) it is suggested to divide respondents into two groups “managers” (or decision makers) and employees; and calculate all the results separately for each group. The gap in score of managers related to score of employees (overestimation/underestimation) is determined for each specific area (subject) to identify areas of critical differences in the estimates over 25 % (low part of the Dashboard in Figure 2). If necessary, separate analysis by gender and/or age can be done.

QMS-Index methodology is still developing. QMS-Dashboard could be interesting

for all CAF users (and also EFQM users) which start quality journey. It is easy to use express instrument allows to get “quick wins” after first self-assessment, to understand better organisational performance and communication gaps between decision makers (managers/executives) and decision doers (employers/workers).

6

Figure 2. QMS-Dashboard

7

ANNEX. QUESTIONNAIRE Your role in organisation: ☐ manager/decision maker

☐ employee/worker Gender: ☐ man

☐ woman Age: ☐ under 25

☐ 26 – 35

☐ 36 – 45

☐ 46 – 60

☐ over 60 Part 1. Rank the following statements in terms of its contribute to organisational excellence. Use proposed scale

0 – No answer 1 – Not important 2 – Rather not important 3 – Rather not important 4 – Important 5 – Very important

n Statement Score 1 Leaders are citizen- and society-oriented in strategic planning 0 1 2 3 4 5

2 Leaders report overall development goals to each employee in the organisation

0 1 2 3 4 5

3 Leaders demonstrate their loyalty to employees 0 1 2 3 4 5 4 Leaders support initiatives from employees 0 1 2 3 4 5

5 Leaders provide all necessary resource to carry out all set organisational functions

0 1 2 3 4 5

6 Organisation has long-term development strategy 0 1 2 3 4 5 7 Former leaders demonstrate their efficiency as informal leaders too 0 1 2 3 4 5

8 Organisation is focused on maximal satisfaction of needs of each citizen and society in whole

0 1 2 3 4 5

8 Staff are well informed about mission and strategy of organisation 0 1 2 3 4 5

10 Intra-and inter-organisational relations function well and provide optimal processes

0 1 2 3 4 5

11 Personal goals of employees correspond with the mission and strategy of organisation

0 1 2 3 4 5

12 Implementation of the mission and strategy are under regular monitoring

0 1 2 3 4 5

13 Organisation systematically gather, analyse and employ information external sources

0 1 2 3 4 5

14 Organisation analyses information about performance of other organisations

0 1 2 3 4 5

15 Organisation encourages knowledge and good practices exchange among employees

0 1 2 3 4 5

16 Organisation develops staff rewarding system 0 1 2 3 4 5

8

n Statement Score 17 Organisation improve working conditions for staff 0 1 2 3 4 5

18 Organization carries out social support, pays much attention to healthcare, safety and recreation

0 1 2 3 4 5

19 Organisation set communication channels for claims and offers from employees up to leaders

0 1 2 3 4 5

20 Organisation has an effective staff appraisal system 0 1 2 3 4 5

21 Employees have opportunities to get further education and professional development

0 1 2 3 4 5

22 Equal opportunity of getting job for men and women, representatives of different religions and nationalities

0 1 2 3 4 5

23 Organisation has sufficient financial security 0 1 2 3 4 5 24 All resources are planned and managed 0 1 2 3 4 5 25 Saving energy and material resources, waste management 0 1 2 3 4 5 26 Information about organisation are available for all stakeholders 0 1 2 3 4 5

27 Organisation is in continues search for new efficient management approaches

0 1 2 3 4 5

28 Finance resource are under control 0 1 2 3 4 5 29 Systematic analysis and improvement of all organisational processes 0 1 2 3 4 5

30 Leaders delegates power to responsible employees in their professional field

0 1 2 3 4 5

31 Regular monitoring of citizens and other stakeholders satisfaction with quality of delivered public services

0 1 2 3 4 5

32 Organisation use ideas on improvement received from different stakeholders

0 1 2 3 4 5

33 Organization analyzes how citizens, society and other stakeholders perceive its activities

0 1 2 3 4 5

34 Organisation analyses needs and expectations of citizens, society, other stakeholders related to delivered public services

0 1 2 3 4 5

35 Organisation cares about its image (media coverage, number of awards, etc.)

0 1 2 3 4 5

36 Organisation analyses what benefit does it bring to citizens, society, other stakeholders

0 1 2 3 4 5

37 Organisation develops additional services to their stakeholders 0 1 2 3 4 5

38 Organisation evaluates the level of confidence of citizens, society, other stakeholders

0 1 2 3 4 5

39 Organisation motivates employees to be more involved in organisational activities

0 1 2 3 4 5

40 Organisation measures people satisfaction 0 1 2 3 4 5

41 Organisation pays attention to achievements of employees (training courses, awards, etc.)

0 1 2 3 4 5

42 Organisation contributes to local society 0 1 2 3 4 5

43 Organisation monitors opinions and complaints from citizens, society and other stakeholders

0 1 2 3 4 5

44 Organisation cooperates with private companies and NGOs in systematic way

0 1 2 3 4 5

45 Organisation analyses results of audits and inspections 0 1 2 3 4 5 46 Organisation analyses its partnerships 0 1 2 3 4 5 47 Organisation conducts cost analysis and optimises its expenses 0 1 2 3 4 5 48 Organisation has solid financial management system 0 1 2 3 4 5 49 Organisation manages its material assets and equipment in effective 0 1 2 3 4 5

9

Part 2. Rank the same statements in terms of achieve results of your organisation. Use proposed scale (assessment panels of CAF-2013 Classical Scoring are used).

0 – We are not active in this field, we have no information or very anecdotal. No results are measured and/or no information is available. 1 – We have a plan to do this. Results are measured and show negative trends and/or results do not meet relevant targets. 2 – We are implementing / doing this. Results show flat trends and/or some relevant targets are met. 3 – We check / review if we do the right things in the right way. Results show improving trends and/or most of the relevant targets are met. 4 – On the basis of checking / reviews we adjust if necessary. Results show substantial progress and/or all the relevant targets are met. 5 – Everything we do, we plan, implement, check and adjust regularly and we learn from others. We are in a continuous improvement cycle on this issue. Excellent and sustained results are achieved. All the relevant targets are met. Positive comparisons with relevant organisations for all the key results are made.

n Утверждение Оценка

50 Leaders are citizen- and society-oriented in strategic planning 0 1 2 3 4 5 51 Leaders report overall development goals to each employee in the

organisation 0 1 2 3 4 5

52 Leaders demonstrate their loyalty to employees 0 1 2 3 4 5 53 Leaders support initiatives from employees 0 1 2 3 4 5 54 Leaders provide all necessary resource to carry out all set organisational

functions 0 1 2 3 4 5

55 Organisation has long-term development strategy 0 1 2 3 4 5 56 Former leaders demonstrate their efficiency as informal leaders too 0 1 2 3 4 5 57 Organisation is focused on maximal satisfaction of needs of each citizen and

society in whole 0 1 2 3 4 5

58 Staff are well informed about mission and strategy of organisation 0 1 2 3 4 5 59 Intra-and inter-organisational relations function well and provide optimal

processes 0 1 2 3 4 5

60 Personal goals of employees correspond with the mission and strategy of organisation

0 1 2 3 4 5

61 Implementation of the mission and strategy are under regular monitoring 0 1 2 3 4 5 62 Organisation systematically gather, analyse and employ information

external sources 0 1 2 3 4 5

63 Organisation analyses information about performance of other organisations

0 1 2 3 4 5

64 Organisation encourages knowledge and good practices exchange among employees

0 1 2 3 4 5

65 Organisation develops staff rewarding system 0 1 2 3 4 5 66 Organisation improve working conditions for staff 0 1 2 3 4 5 67 Organization carries out social support, pays much attention to healthcare,

safety and recreation 0 1 2 3 4 5

68 Organisation set communication channels for claims and offers from employees up to leaders

0 1 2 3 4 5

69 Organisation has an effective staff appraisal system 0 1 2 3 4 5 70 Employees have opportunities to get further education and professional

development 0 1 2 3 4 5

10

n Утверждение Оценка 71 Equal opportunity of getting job for men and women, representatives of

different religions and nationalities 0 1 2 3 4 5

72 Organisation has sufficient financial security 0 1 2 3 4 5 73 All resources are planned and managed 0 1 2 3 4 5 74 Saving energy and material resources, waste management 0 1 2 3 4 5 75 Information about organisation are available for all stakeholders 0 1 2 3 4 5 76 Organisation is in continues search for new efficient management

approaches 0 1 2 3 4 5

77 Finance resource are under control 0 1 2 3 4 5 78 Systematic analysis and improvement of all organisational processes 0 1 2 3 4 5 79 Leaders delegates power to responsible employees in their professional

field 0 1 2 3 4 5

80 Regular monitoring of citizens and other stakeholders satisfaction with quality of delivered public services

0 1 2 3 4 5

81 Organisation use ideas on improvement received from different stakeholders

0 1 2 3 4 5

82 Organization analyzes how citizens, society and other stakeholders perceive its activities

0 1 2 3 4 5

83 Organisation analyses needs and expectations of citizens, society, other stakeholders related to delivered public services

0 1 2 3 4 5

84 Organisation cares about its image (media coverage, number of awards, etc.)

0 1 2 3 4 5

85 Organisation analyses what benefit does it bring to citizens, society, other stakeholders

0 1 2 3 4 5

86 Organisation develops additional services to their stakeholders 0 1 2 3 4 5 87 Organisation evaluates the level of confidence of citizens, society, other

stakeholders 0 1 2 3 4 5

88 Organisation motivates employees to be more involved in organisational activities

0 1 2 3 4 5

89 Organisation measures people satisfaction 0 1 2 3 4 5 90 Organisation pays attention to achievements of employees (training

courses, awards, etc.) 0 1 2 3 4 5

91 Organisation contributes to local society 0 1 2 3 4 5 92 Organisation monitors opinions and complaints from citizens, society and

other stakeholders 0 1 2 3 4 5

93 Organisation cooperates with private companies and NGOs in systematic way

0 1 2 3 4 5

94 Organisation analyses results of audits and inspections 0 1 2 3 4 5 95 Organisation analyses its partnerships 0 1 2 3 4 5 96 Organisation conducts cost analysis and optimises its expenses 0 1 2 3 4 5 97 Organisation has solid financial management system 0 1 2 3 4 5 98 Organisation manages its material assets and equipment in effective 0 1 2 3 4 5