UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl)

UvA-DARE (Digital Academic Repository)

Quality management in health care : empirical studies in addiction treatment services alignedto the EFQM excellence model

Nabitz, U.W.

Link to publication

Citation for published version (APA):Nabitz, U. W. (2006). Quality management in health care : empirical studies in addiction treatment servicesaligned to the EFQM excellence model.

General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s),other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, statingyour reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Askthe Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam,The Netherlands. You will be contacted as soon as possible.

Download date: 04 Aug 2020

C H A P T E R 9

EFQM quality management: A single-case study of an outpatient addiction

treatment service

Udo Nabitz, Achim Schaefer, Jan Walburg

Published with minor modifications in German as: Nabitz, U., Schaefer, A., & Walburg, J. (2006).

EFQM Qualitatsmanagement: Einzelfallstudie in einer Beratungsstelle. Sucht, 52, 65-75.

193

CHAPTER 9

Abstract

A I M S To determine if the European Foundation for Quality Management

(EFQM) approach is suitable for improving the level of quality of an outpatient ad­

diction treatment service.

M E T H O D A single-case study of total quality management was conducted over a

period of five years, in 1998, 2000 and 2003. The study engaged all members of staff.

Self-assessment following the workshop approach was based on the 42 criterion

parts of the EFQM Model.

R E S U L T S The quality level of the addiction treatment service improved from level

three to level four according to the E F Q M criteria. This means that the 30 criterion

parts of the EFQM Enablers are systematically implemented and evaluated. The

quality levels of the 12 Result criterion parts show a remarkable increase in result ori­

entation of the service.

C O N C L U S I O N S The EFQM approach leads to an improvement in quality of out­

patient services. The findings of the single-case study should be replicated by fur­

ther studies.

194

EFQM single-case study

Introduction

At the end of the 1980s, fourteen captains of industry met with the General Secre­tary of the European Commission Jacques Delors in Brussels to launch a pro­gramme to strengthen the competitive position of the European industry. The aim of the programme was to establish the European Foundation for Quality Manage­ment and to introduce the European Quality Model, otherwise known as the EFQM Model (Malorny, 1999). The fourteen founding members, among which were British Telecommunication, KLM NV, Volkswagen AG, NV Philips and Re­nault, made a declaration to shape quality management in their organizations in compliance with the EFQM Model (Blount, 2006). Their approach generated an unexpected positive reaction from the public sector. Subsequently, education, gov­ernment, and health care organizations also adopted the EFQM Model.

The EFQM Excellence Model, as it is named since 1999, is an open so-called non-prescriptive framework consisting of nine criteria. Five of these relate to process management and are known as Enabler criteria and four address the results of an organization and are known as Result criteria. Each of the nine criteria has a definition and is supported by a number of criterion parts. The EFQM Excellence Model is based on the following premises: Excellent results with respect to Perform­ance, Customers, People and Society are achieved through Leadership driving Poli­cy and Strategy that is delivered through People, Partnerships and Resources, and Processes (EFQM, 2003). The EFQM approach is recognized by experts as a repre­sentative for the Total Quality Management philosophy (Garvin, 1991; Zink, 1995).

In the available literature on quality assurance in health care, the EFQM Model is regarded as a promising approach which gains importance (Shaw, 2000) along­side the certification of the International Organization of Standardization (ISO) and peer review approaches. Reviews point out that an integrated approach such as EFQM appears to be more effective in improving quality than isolated quality proj­ects (Grol, 2001). The EFQM Model is the best known integrated model in Europe and is applied in many settings such as academic and acute care hospitals, psychoso­matic clinics and homes for the elderly (Brandt, 2001; Moeller, 2001; Moeller, Breinlinger-O'Reilly, & Elser, 2000; Schubert, & Zink, 2005). Some larger addic­tion treatment services have also adopted the approach (Adamski, 2004; Kunz, Böhl, & Biirkle, 2004; Nabitz, & Klazinga, 1999) and the top management within these organizations use the EFQM Model as a general, logical, and integral frame­work for assessing and outlining their approach to quality management. An alterna­tive way of using the EFQM Model was promoted by the regional coordination of­fice for drug-related issues in Germany, which initiated a quality management project for small outpatient treatment services called the QM-Project (Pittrich,

195

CHAPTER 9

Rometsch, & Winkler, 2002). The goal of the project was to acquaint the entire staff of a service with the principles and methods of quality management. In addition to the premises of EFQM, the outpatient services were defined as psychosocial service centres, that are responsible for their performance and results, that adapt constantly to new tasks and demands of their stakeholders, and in which the management and all members of staff strive for continuous improvement. This implies that although the leaders of the services are responsible for quality, the entire staff develops an awareness to improve and strive for quality. Following these principles another re­gional coordination office in northern Germany also initiated a QM-Project to sup­port outpatient services introducing the EFQM approach (Böttger, 2000; Böttger, 2005).

The engagement of all staff of a service, as it was proposed in the QM-Project, was an interesting expansion of the classical EFQM approach, which focused on the management of large organizations. Initial experiences with implementing the EFQM approach in health care including therapists, medical doctors, social work­ers, and support staff were positive, but were limited in most cases to one single self-assessment or to one specific quality project related to the EFQM Model (Beine, 2002). There was no formal overall evaluation of the QM-Projects of the regional offices. Some services conducted self-assessments but there are only anecdotal re­ports and no systematic longitudinal studies of services published. This study was, therefore, initiated to document the experiences of one service as a single-case study. The service was selected from the QM-Project and was one of the outpatient addic­tion treatment services that had implemented the EFQM approach, had carried out several self-assessments with all members of staff, and had sufficient documentation and data available for analyses. The following questions form the basis of this arti­cle:

• Is it possible to apply the EFQM approach, systematically engaging all members of staff of an outpatient addiction treatment service?

• Is there a sustainable positive effect on all criteria and criterion parts of the EFQM Excellence Model?

• Is it possible to demonstrate a general improvement in quality and a culture of excellence in the treatment service?

Method

The method section is structured like a treatment study: design, setting, sample, in­tervention and measures. To illustrate the interventions, three examples are exhibit­ed elaborately, which give an insight into the praxis of conducting improvement projects in a treatment service.

196

EFQM single-case study

Design of the single-case study After an initial orientation phase, the management of the outpatient addiction treatment service that was chosen for this study adopted the EFQM approach. Over the subsequent five years, several quality improvement projects were carried out and the staff of the service assessed the changes on three separate occasions: in February 1998 (Ti), November 2000 (T2), and March 2003 (T3). In this study, these assess­ments are seen as three measurements and the quality projects represent the inter­ventions.

Figure 1 shows the design and process steps of the single-case study. After taking the training course in EFQM quality management (1), each member of staff re-

Figure 1. Design and process steps of the single-case study

1996 & 1997

March 1998 l . Self-assessment M - 1 7 IN — i Z

Interventions

November 2000 2. Self-assessment N =13

Interventions

March 2003 3. Self-assessment N = 14

2004 & 2005

Orientation and decision making about the EFQM approach

" 1.Training and instruction 2.Homework assignment 3.Consensus workshop 4.Planning workshop

1 ' Implementation of the improvement projects

1 •

1.Training and instruction 2.Homework assignment 3.Consensus workshop 4.Planning workshop

1 ' Implementation of the improvement projects

' ' 1.Training and instruction 2.Homework assignment 3.Consensus workshop 4.Planning workshop

1 •

Documentation and publication

197

CHAPTER 9

ceived the homework assignment (2) to evaluate the service for a period of two years using the EFQM Diagnosis Addiction Services Questionnaire, containing 42 ques­tions. Documentation such as the Annual Report was made available. Two weeks later a one-day consensus workshop (3) with all staff members was held and individ­ual evaluations of the 42 questions were summarised on a poster, discussed and con­verted into a consensus score. Any suggestions for improvements, as well as the ser­vice's strengths were recorded. During the workshop, the dynamic interaction between evaluation, consensus forming and suggestions for changes were also stim­ulated (EFQM, 2003) and this resulted in a list of improvements, a quality profile, and a radar graph. These findings were then used in a subsequent planning work­shop (4) to specify improvement projects. For the period of two years, working groups then earned out the agreed quality projects, including the actions for imple­mentation.

Setting:The outpatient addiction treatment service The treatment service used in this study was established in 1971 as an office for per­sons with alcohol and pharmaceutical problems. At that time there was only one member of staff. The service expanded in 1978 to become a consultation and treat­ment service for addiction including drug addiction, and provided treatment serv­ices to persons with substance misuse and abuse, as well as their families. At present, the service provides face-to-face consultation, care, treatment, outpatient rehabili­tation, referral, a methadone programme and aftercare (Schaefer, 1997).

In 2003, fourteen members of staff were employed. These were grouped into four sub-teams with responsibilities for alcohol, pharmaceuticals, illegal drugs, preven­tion and sheltered housing. Annually, the service is in contact with over 1 500 clients and approximately 200 children of adults with addiction problems. About 80 of these clients are in a methadone programme. In 2003, the prevention team advised 50 intermediates, and approximately 700 juveniles participated in a pedagogical prevention programme. The AIDS prevention programme reached more than 1000 juveniles and the prevention team organized approximately 100 informative meet­ings (Schaefer, 2002).

Participants in the self-assessment The three assessments were carried out by 12 (Ti), 13 (T2), and 14 members of staff (T3) respectively. Due to staff fluctuation, a total of seven persons participated in all three evaluations. In 2003, the participants of the workshop were 5 men and 9 women with the following professions: 5 pedagogues; 4 social workers; 2 adminis­trative assistants; 1 medical doctor; 1 psychologist; 1 sociologist. The average age was 45 years (M = 45.5; range 34 - 54). Average number of years employed in the service was 6 years (M = 5.8; range 1 - 26) and the average weekly employment was 25 hours (M = 25.3; range 3 - 38.5).

198

EFQM single-case study

Interventions: Suggestions for improvement and quality projects The E F Q M approach comprises self-assessment, a consensus workshop, sugges­

tions for improvements and quality projects. An E F Q M self-assessment encom­

passes more than a description and the evaluation of the level of quality of an organ­

ization. In the outpatient addiction treatment service presented here, the

homework assignment and the consensus workshop triggered awareness and a

learning process in the participants. The assessments and opinions about the per­

formance based on the Result criteria structured the communication and provided

new insight based on facts and data. The review of the service's activities alongside

the Enabler criteria broadened the view on how the service functions and created a

sense of engagement and commitment. The participants were motivated by the fo­

cus on strengths and improvements. The experience of a self-assessment had a posi­

tive influence on the participants, stimulating them to shape and contribute to the

quality management of the service. Table i shows the number of improvement sug­

gestions for each criterion, which were formulated during the three consensus

workshops.

Over the five years, the working groups carried out about 20 quality projects se­

lected out of the 78 improvement suggestions. Three of these projects are illustrated

in tables 2a, 2b and 2c.

Table 1 : Improvement suggestions during the consensus workshop

Criteria

Enabler criteria

Result criteria

Sum

Leadership (1) Policy and Strategy (2) People (3) Resources (4) Professionalism (5) Processes (6)

Customer Results (7) People Results (8) Society Results (9) Key Performance Results (10)

Num

1998

1 1 3 1 1 3

4 3 1 1

19

ber of suggestions

2000

5 4 6 3 1 3

4 5 4 4

39

2003

2 3 4 2 2 1

1 3 0 2

20

199

CHAPTER 9

Table 2a: Example 1 of a quality project

1 . Child project Strong Kids

The project Strong Kids was launched in 2001 and is mainly related to the criteria Policy and Strategy (2) and Processes (6). I t is a treatment innovation positioned between prevention and the treatment task of the service. Children from the age of 3 to 14 years are diagnosed and receive support and coaching in face-to-face contact with a therapist and in a group. Strong Kids is an example for the adaptation of the outpatient services to new demands from new target groups (criterion part 2c). The development and the initial introduction of the project was successful but after a while the project became somewhat isolated in the service. The interest and the acceptance among the staff reduced. External admissions by the community office for youth and family affairs generated a waiting list, but the internal referrals by staff stagnated. To solve this problem, a simple referral statistic (criterion part 6b) was introduced, displayed, regularly updated and systematically distributed among all staff members. Everybody in the service was better informed about the progress of the project, new developments, changes and open slots for referrals of children (criterion parts 5d and 6a). In a period of a year the project regained its position in the service, the ratio between external and internal referral of children which was 9 to 1 balanced again to 5 to 5 and the isolation of this innovative treatment was overcome.

Table 2b: Example 2 of a quality project

2. EBIS Expert

This quality project had a positive influence on the criteria Resources (4) and Professionalism (5) but above all on Key Performance Results (10). In general, documentation, statistics and analyses of data are seen by professionals as annoying external requests and not as an instrument for evaluating work, goals and results (10a, 10b, 10c). Result orientation and the discussion about adequate methods of collecting data and presenting results only slowly gains a respected place in outpatient addiction treatment services. A senior staff member and not the leader of the service took the responsibility to review the problems, the insidiousness and the weaknesses of the documentation system and started to improve it. He was called the EBIS Expert and found deviations and ambiguities in the daily documentation procedures. As part of an improvement strategy everybody was offered support to restructure their own workflow and their registration. This quality project was not conducted along a formal project plan and the strict line of accountancy, but followed the flexible, patient but determined instructions of the EBIS Expert. Slowly the understanding and attitude for performance documentation changed. Insecurity was reduced, explanations and definitions (6a) were clarified and a new computer system was introduced. The new system was a breakthrough in the attitude of the professionals concering working with data and facts. The team discussions about the achieved results of the service became interesting and challenging. Everybody realized, that a good documentation system is useful for providing clarity and confidence. Facts and figures became available for use in external presentations, which in most of the cases strengthened the service position and consolidated achievements (4a, 4b).

2 0 0

EFQM single-case study

Table 2c: Example 3 of a quality project

3. Staff Survey

This project was started in summer 1998 and contributed above all to an improvement of the criterion People (3) and People Results (8). The criterion part Perception Measures (8a) shows the biggest changes in the EFQM profile in the three self-assessments. The first personnel satisfaction survey was done in 1998 with a long questionnaire. The findings of the survey gave an initial impression about the appreciation of the staff concerning various aspects of their work, communiction, style and culture. The survey also made clear that an adjusted, specific and short questionnaire was needed (3c). The prevention team that had experience with the measuring of satisfaction composed a flexible questionnaire with specific questions and a rating system. In addition to the questionnaire the survey approach was made more flexible. Each member of staff was authorized to survey colleagues about a serious topic with a maximum of 10 questions. Next to the survey, the results and the consequences had to be presented to the staff by the person who took the initiative. The surveys were carried out shortly before the staff meeting. The findings became a voting poll. Decisions for actions and changes were taken directly (3a) and the initiator had to supervise the changes. In this way the somewhat bureaucratic annual personnel satisfaction survey was transformed into a sensible and flexible opinion poll.

The working groups used the usual methods of project management (Boy, Dudek, & Kuschel, 1995; Litke, 1995). However, a complete description of the project man­agement process is beyond the purpose of this publication.

Measurement instrument:TheWorkbook EFQM Diagnosis The Workbook EFQM Diagnosis Addiction Treatment Services was developed by an expert group as part of the QM-Project (Pursche, Nabitz, Muehl, & Winterberg, 1999). The group's objective was to construct an instrument to assess the quality lev­el of a service applying EFQM self-assessment. In order to be specific for addiction treatment services, the group used the EFQM-Model-Jellinek, in which the criteri­on Processes is expanded with the criterion Professionalism (Nabitz & Walburg, 1995). The workbook consists of an introduction, a questionnaire with 42 questions, and a scoring system. The questions are grouped into six Enabler criteria and four Result criteria corresponding with the criterion parts of the EFQM Model. Each question is composed of a general part, which was derived from the EFQM criteri­on parts and specific examples, which are directly related to outpatient addiction treatment services. Each Enabler criterion has five questions, each Result criterion has three questions. The rating ranges from o to 5 and is a six-step ordinal scale. The questionnaire is similar to the EFQM instrument Assess Excellence - A Question­naire Method (EFQM, 2005). Table 3 shows examples of the types of questions used in the questionnaire.

201

CHAPTER 9

Table 3: Sample questions from the Workbook EFQM Diagnosis Addiction Treatment Services

Sample question for the Enabler criteria 1 to 6

l a . Our leader shows visible engagement and is a role model for quality management, such as providing financial rescources, making time available for the staff, communicating values and norms.

Is not a topic 0 Is now and then a topic 1 Is systematically developed 2 Is systematically developed and implemented 3 Is systematically developed, implemented and evaluated 4 Is systematically developed, implemented, evaluated and adapted 5

Sample question for the Result criteria 7 to 10

7a. The service evaluates the satisfaction of their clients using indicators, such as ease access, short waiting time, anxiety free interaction, respectful approach, agreement to the treatment plan, satisfaction with the progress.

Is not measured 0 Is now and then measured 1 Is systematically measured 2 Is systematically measured and goals are formulated 3 Is systematically measured, goals are formulated and achieved 4 Is regular practice and has been routine for several years 5

A low rating of the six Enabler criteria means that the topics of the questions have not yet been addressed (value o) or are addressed (value i) occasionally. A higher rat­ing means that the topics are systematically developed (value z) and implemented (value 3). The highest rating (value 4 and 5) means that the aspects have been imple­mented, evaluated and where necessary adapted. The ratings in the four Result cri­teria range from Is not measured (value o) to Is systematically measured, goals are for­mulated and achieved(value 4) and Is regular practice and has been routine for several years (value 5).

For the interpretation of the consensus scores and the calculated average scores, five levels are distinguished. The first level (o to 1.4) means that the service has not focused on quality. The second level (1.5 to 2.4) means that the service has adopted a systematic quality approach. The third level is characterized by the implementation of quality systems and performance measurements (2.5 to 3.4); and level four in­cludes regular evaluations and the achievement of targets. The fifth level (4.4 to 5.0) is the highest level and is an indication of excellence as defined by EFQM.

2 0 2

EFQM single-case study

The workbook recommends a consensus workshop approach: each participant rates the questions individually; the ratings are collected and discussed; merged to a consensus score; and are represented as the quality profile of 42 questions. Further­more, the averages of the consensus scores are used to calculate the scores of the nine criteria, an Enabler score, a Result score and a Total EFQM score. This scoring sys­tem is derived from the RADAR matrix of EFQM (EFQM, 2003) but the final scores cannot be compared with the RADAR matrix scores, which are much more detailed. The questions in the workbook are similar, but not identical to the criteri­on parts of the EFQM Excellence Model. Furthermore, the criteria are not adjusted with weights. These differences between the scoring systems allow only a general comparison of the profiles of the two systems. A transformation of the scores of the workbook to the EFQM scale which runs from o to 1000 points is not possible.

Results

A familiar way to present the findings of a self-assessment is by radar graph on EFQM criteria level and as Quality profile on EFQM criterion part level. Addition­ally, the individual scores of seven participants are displayed in a line graph.

RrkAir err-anh I , U U U I t^l U 1 J I •

The quality level of the outpatient service for the criteria is calculated based on the consensus scores of the questions.

In Figure 2, the radar graph with the scores of the three self-assessments carried out in 1998, 2000 and 2003 shows a positive trend. The surface of the radar graph is larg­er in 2003 than in 1998. This suggests all Enablers are systematically developed, im­plemented, and evaluated. The Result criteria are systematically measured and al­most all the goals in the Result criteria are formulated and the goals in the Key Performance Results have been achieved. The Society Results is assessed lower than level three, which means the goal definition for this criterion and the systematic measures are absent.

During the five-year period, all ten criteria show improvements. The total EFQM score raised from level three (M = 3.17) to level four (M = 3.85). The Result score im­proved with 1.19 points from M = 2.33 to M = 3.52. The Enabler score showed an im­provement of 0.47 points from M = 3.51 to M = 3.98. The Enablers are rated overall higher than the Results.

203

CHAPTER 9

Figure 2: Radar graph 1998, 2000 and 2003

Enablers ( 1 - 6 )

Professionalism Processes (5) ( 5 )

Results ( 7 - 1 0 )

20 March 1998 A _ _ 8 November 2000 • " " 5 March 2003 N =12 N = 13 N = 1 4

Quality profile Figure 3 compares the profiles from 1998, 2000 and 2003 for the criterion parts de­fined by the 42 questions. In the 2003 profde, all 30 criteria parts of the Enablers (ia to 6e) reach a level of three or higher. This means that the quality topics are regularly discussed, systematically developed and implemented. The highest level is achieved for Budgeting (4a) and Financial information (4b) followed by Professionalism (5a, 5b and 5c).

In Figure 3, the scores of the Enabler criterion parts (ia-6e) in 2003 show a fairly

204

EFQM single-case study

Figure 3: Qual i ty profi le 1998, 2000 and 2003

Level 1 Level 2 Level 3 Level 4 Level 5

Engaged leadership ( la )

Change manager ( lb )

Rewarding (1c)

Clarifying circumstances ( Id)

Role model ( le )

Norms, values, vision (2a)

Extemat demands (2b)

Changes in client groups (2c)

Communicating the strategy (2d)

Flexibility in planning (2e)

Proactive personnel planning (3a)

Responsibility, loyalty (3b)

Continuous improvement (3c)

Agreement on goals (3d)

Constructive dialogue (3e)

Budgeting (4a)

Financial information (4b)

Technical infrastructure (4c)

Information dissemination (4d)

Equipment (4e)

Continues education (5a)

Competence (5b)

Using guidelines (5c)

Transparency (5d)

Knowledge of service chain (5e)

Defined products, services (6a)

Process schemes (6b)

Treatment plans (6c)

Creative improvements (6d)

Quality system (6e)

Satisfied clients (7a)

Satisfied referrals (7b)

Satisfied insurers (7c)

Satisfied personnel (8a)

Sick leave (8b)

Work setting (8c)

Image of the service (9a)

Presence in media (9b)

Satisfied society (9c)

Key results (10a)

Clinical results (10b)

Financial results (10c)

Enablers 1 - 6

Resul ts 7 - 1 0

1998 N = 12 •2000 N = 13 2003 N = 14

205

CHAPTER 9

regular pattern on a high level, but the profile of the Result criterion parts fluctu­ates. For example, the criterion part Satisfied personnel (8a) was measured systemat­ically and regularly and the goals for 2003 were achieved and the measuring system adapted to the changing needs of the service. Other examples of high-level scores are the performance results for Key results (10a), Clinical results (10b) and Financial re­sults (10c). For these criterion parts a system was in place, goals were formulated and achieved and measuring had been a regular practice for years. In contrast, Satisfied clients (7a) was measured in 2003 but no goals were defined. Image of the service (9a) received the lowest score because no facts or figures were available.

Several positive changes can be seen between the 2003 assessments and those car­ried out in 1998 and 2000. In 1998, five criteria parts (2c, 7c, 8a, 9a and 9c) were rat­ed low, but the first improvements were achieved in 2000, and were subsequently exceeded in 2003. There is a remarkable improvement in Changes in client groups (2c) from level 1 to level 3, which suggests that the observations and experiences of the professionals concerning the changes in the client groups are systematically used to adapt the service's Policy and Strategy. Satisfied referrals (7c) and Satisfied per­sonnel (8a) also show a positive change over several levels. Satisfied personnel (8a) was not an issue before 1998, whereas in 2003 a systematic approach, measures and goals were in use and the system was adapted to the needs. An irregularity is seen in the Sick leave (8b) and Work setting (8c). Presence in the media (9b) and Sick leave (8b) show a negative shift of more than one level.

Longitudinal comparisons Seven members of staff participated in all three assessments. The average scores over the Enabler and Result criteria for each participant are presented in Figure 4. In ac­cordance with the analysed results in the radar graph and the quality profile, there are considerable changes in the average Result criteria. In 2003 (M = 3.23), all partic­ipants were more positive than in 1998 (M = 1.45). A sharp increase can be seen be­tween 1998 and 2000 (M = 2.87). The changes in the Result criteria are statistically significant: The variation of the standard deviation of the Result criteria is larger in 2003 (SD = 0.72) than in 1998 (SD = 0.33). The changes of the Enablers are smaller and statistically not significant. The scores of the Enablers are higher in 1998 than the scores of the Result criteria. All but one of the participants assessed the Enablers more positively in 2003 than in 1998.

Evaluation of the workshop At the end of the workshops, the participants were asked for comments and a rating. The ratings of workshop 2003 are presented in Table 4. The average score was 4,8 in 2003 and 4,2 in 2000. Rating was not applicable in 1998.

206

EFQM single-case study

Figure 4: Longitudinal assessments of Enablers and Results

Level

Enablers Participants 1 ••-»•••

2 • • • . .

4 ..-#...

5 .. .*..

Results Participants

1998 2000 2003

Enablers Year 1998 2000 2003

Mean 3.48 3.58 3.75

SD 0.35 0.42 0.44

Results Year 1998 2000 2003

Mean 1.45 2.87 3.23

SD 0.33 0.78 0.72

Multivariate analyses of variance with repeated measurements N = 7; F (2,5) = 2.13; P = 0.214

Multivariate analyses of variance with repeated measurements N = 7; F (2,5) = 18.03; P = 0.005

207

CHAPTER 9

Table 4: Evaluation of the consensus workshop 2003 (staff plus observer)

Very weak 1 2 3 4 5 Very good

Meaningless and oo

poorly-invested time

Nothing learned ooooo

Very poor material ™ 0 0 0

Very poor performance

Very negative oo

Discussion

The findings of this study are summarized, interpreted and discussed along the three research questions. The methodological limitations concerning the single case study, the questionnaire and the self-assessment are described and suggestions for further research are given.

Interpretation The first research question of the study was: Is it possible to apply the EFQM ap­proach, systematically engaging all staff of an outpatient addiction treatment serv­ice? The description of the method and results of this study show that this outpa­tient addiction treatment service has successfully applied the EFQM approach. All staff took part in the project, including the manager, the professionals, and the ad­ministrative staff. Self-assessments and improvement projects were carried out and an increasing quality and result orientation grew during the five years.

There are several explanations for the successful application of the EFQM ap­proach in an outpatient service. The broad, clearly structured and plausible EFQM Excellence Model appeals to highly-educated staff and supports their motivation for improvement and innovation. The 42 criterion parts are a demanding frame­work for evaluating the structure, the process and the outcome of a service, and re­flect the topics of Leadership, Policy and Strategy, People, Resources, Professional­ism, Processes and Results. The emphasis on Processes and Results remains a challenge and the fact that the results are not limited to financial outcome, but in­clude the satisfaction of clients, staff and the society, is in line with the self-percep­tion of professionals. Similar interpretations are reported elsewhere (Schubert, & Zink, 2005).

00000 0000c

Meaningful and well-0:00c 00000 000 invested time

0000 Very good material

Very good 00D00 performance 00000 ooooo Very positive 000

208

EFQM single-case study

Furthermore, the consensus method seems to be a contributing success factor. After the individual ratings and the discussion in the workshop, the participants have to agree on the level of quality, a procedure that is familiar to interdisciplinary teams of addiction services. The workshops generate a tension that stimulates staff contribution. They are always dynamic, open and flexible, which is a challenge for professionals with different backgrounds and qualifications. The direct engagement of all members of staff, the broad approach and the stimulation to discuss and clari­fy the quality topics is also an important basis for generating suggestions for im­provement and for initiating quality projects. The visualisation of the conclusions and the consensus scores of the workshops via graphs, documents and posters are details that clearly contribute to the overall success. This effect is indicated in the favourable evaluation of the workshops.

The second question: Is there a sustainable positive effect on all criteria and crite­rion parts of the EFQM Excellence Model? can also be answered positively. The de­scriptive statistics of the consensus scores show a change from level three to level four in 1998 and 2003. The scores of the six Enabler criteria Leadership (1), Policy and Strategy (2), People (3), Resources (4), Professionalism (5) and Processes (6) are 0.5 higher in 2003 and, of the 30 criteria parts, 23 showed improvements, three re­mained stable and four deteriorated. It has to be said that several studies have shown that stable quality improvements can be seen after five years (Marlony, 1999). It has also been shown that there is not always a direct relation between a quality project and the consensus score of a criterion. Most improvement projects effect several cri­teria and in the long term contribute to the quality level as a whole.

The findings also show that over the five-year period, the Enabler criteria im­proved more consistently than the Result criteria. The Enabler profile of the service is more in balance in 2003 than the profile of the Result criteria. Positive scores were achieved for Satisfied personnel and Key Results. The quality projects Staff Survey and EBIS Expert were the main reasons for the improvements. The positive changes in the Result criteria have to be attributed to the actual improvements in the services and not to the change in the composition of the workshop participants, which is demonstrated in Figure 4.

One of the premises of the EFQM is that outpatient services are defined as psy­chosocial service centres that are responsible for their performance and results, which adapt constantly to new demands and where all members of staff strive for continuous improvement. Consequently the third question of this study was: Is it possible to demonstrate a general improvement in quality and a culture of excel­lence in the treatment service? Although the management should promote and sup­port quality management, a broad quality awareness and performance orientation should also be present in the team.

There are indications that the outpatient service has taken a step towards becom-

209

CHAPTER 9

ing an excellent organization. It can be concluded that a quality culture has devel­oped over the five years of quality management: quality has been defined in line with the EFQM criteria; staff have been adequately trained; there are continuous suggestions for improvement and quality projects are being carried out. Three self-assessment workshops were held, which stimulated a critical review of the work, the service and the outcomes, which in turn, created a broad shared quality orientation. Similar findings are reported by other quality projects in addiction (Böttger, 2000). Another illustrative example of applying the EFQM approach in a small health care organization is a dental clinic in Switzerland (Harr, 2001).

Methodological problems and further research In medical research, the findings of case studies often signal the start of the develop­ment of new interventions or treatments. Single-case studies are usually followed by experimental studies and randomized clinical trials. In organization- and manage­ment science, case studies are also conducted (Zikmund, 1988) and these are fol­lowed by experimental effect studies. The study presented here can be seen in this context. It is a single-case study, which demonstrates that the EFQM approach in an outpatient addiction treatment service is effective. However further experimen­tal effect studies are needed, including the development of standardized measures and a control group. This step is both necessary in order to prove the effectiveness of quality management and important in order to follow the paradigm of an evidence-based health care. In other industries, controlled effect studies about quality man­agement have been conducted (Hendricks, & Singhal, 1997) and results show that an integrated quality approach is superior to a control group. For addiction treat­ment services, the first step in this direction could be to evaluate the work of the 50 outpatient addiction treatment services that participated in the QM-Project in Ger­many. The findings of this study suggest the hypothesis that quality management confirm EFQM has a positive effect, which is a good starting point for further re­search.

The measures in this study are based on the questionnaire EFQM Diagnosis Ad­diction Treatment Services, which was developed by an expert group. The question­naire was designed for practical use and is applied by many services. The questions are derived form the criteria and criterion parts of the EFQM Excellence Model and have a reasonable face validity. The questions are concrete and the rating categories are an ordinal scale, enabling comparisons over time between two assessments from the same service. A test-retest bias of the three assessments is not very likely because more than two years have elapsed between assessments, and pre-assessment scores were not available. In its present form, the questionnaire is sufficient for practical use. Nevertheless, there should be a thorough test construction initiated in order to create a psychometrically sound instrument. The standard statistical indicators for

210

EFQM single-case study

internal consistancy, reliability and validity of the scales have to be available in order to carry out further empirical research.

In the context of the EFQM approach, self-assessment is commonplace, al­though there are critics who say that the results are biased. However, the individual subjective ratings are thoroughly discussed during the workshop, which result in a consensus score for each of the 42 criterion parts. The procedure is standardized and the consensus score is therefore seen by experts as a reasonable, reliable and valid score representing the actual situation of the service (EFQM, 2003). To support this opinion the use of EFQM self-assessment should be based on empirical studies. Self-assessment has many advantages for practical work, such as the shared scores of a team, the exchange of individual evaluation and underlying assumptions, the pro­motion of openness and criticism, as well as the development of a common lan­guage about quality. These aspects of the self-assessment should be a topic for fur­ther research in order to clarify the reliability and validity of the scores, and the advantages and the disadvantages of the method.

Conclusion This single-case study shows that the EFQM approach has been successfully applied in an outpatient addiction treatment service. It can be concluded that the EFQM approach is not only effective in large organizations and for top managers. It is hoped that the findings motivate outpatient services to follow the route of quality management using the EFQM Excellence Model. The study also shows that the quality of the service improved in a time frame of five years, which suggests that a long-term approach is needed in order to achieve the desired effects. There are re­strictions within this study due to methodological constraints concerning the ques­tionnaire and the method of self-assessment. These methodological issues should be resolved by further studies, which in turn, will clarify the possibilities and limita­tions of the EFQM approach for addiction treatment services.

Acknowledgements

The authors wish to thank the following: Directorate of the Caritas for the City of Bonn inc. for its financial support, the staff of the Outpatient Addiction Treatment Service of the Caritas for the City of Bonn inc., C. Pursche, W. Rometsch, Head of the Coordination Office for Drug Related Issues of the Regional Association of Westfalia-Lippe in North Rhine-Westfalia, W. Heinz, Drug Treatment Services Frankfurt, J. Lindenmeyer, Salus Clinics and A. Holmes, Milliefish Editorial Serv­ices.

211

CHAPTER 9

Reference list

Adamski, D. (2004). BewerbungLudwig-Erhard-Preis2004. 30Jabreausstiegsorientierte Drogenarbeit.

Hamburg: Therapiehilfe e. V.

Beine, N. (2002). Qualitat ist kein Zufall. In W. Pitmen, W. Rometsch, & H. Winkler (Eds.), Fo­

rum SuchtBand31 (pp. 123-129). Munster: Landschaftsverband Westfalen-Lippe.

Blount, H. L. (2006). The European Quest for Excellence. Brussels: EFQM representative office.

Böttger, A. (2000). Qualtatsmangement in der ambulanten Suchrberatung: Benchmarking-Projekt

in Niedersachsen [Quality Management in Outpatient Addiction Services. Benchmarking-Pro­

ject in Lower Saxony]. Sucht aktuell, 39-44.

Böttger, A. (2005). Qualtatsmangement in der Ambulanten Suchthilfe in Niedersachsen [Quality Man­

agement in Outpatient Addiction Services in Lower Saxony]. Hannover, Germany: Niedersachsis-

che Landesstelle fur Suchtfragen [Lower Saxon Head Office for Dependency Matters].

Boy, J., Dudek, O , & Kuschel, S. (1995). Projektmanagement. Offenbach: Gabal.

Brandt, E. (2001). Qualitatsmanagement Gesundheitsforderung im Krankenhaus. Handbuch zur

EFQM-Einführung. Neuwied: Luchterhand.

EFQM (2003). Excellencebewerten. Eine praktische Anleitungfur erfolgreiche Entwicklung, Umsetzung

und Review einer Selbstbewertungsstrategiefur Ihre Organisation. Frankfurt: DGQ.

EFQM (2005). Excellence bestimmen. Eine Fragebogen-Methode. Frankfurt: DGQ.

Garvin, D. (1991). How the Baldrige Award Really Works. Harvard Business Review, 80-93.

Grol, R. (2001). Improving the Quality of Medical Care. Building Bridges Among Professional

Pride, Payer Profit and Patient Satisfaction, journal American Medical Association, 286, 2578-2585.

Harr, R. (2001). TQM in dental pracrice. International journal of health care quality assurance incor­

porating Leadership in health services, 14, 69-81.

Hendricks, K.B., & Singhal V.R. (1997). Does Implementing an Effective TQM Programme Actual­

ly Improve Operating Performance? Empirical Evidence form Firms That Have Won Quality

Awards. Management Science, 43,1258-1274.

Kunz, D., Böhl, H., & Biirkle, S. (2004). Rahmenhandbuch - ambulante Suchthilfe. Freiburg: Ver­

band ambulanrer Behandlungssrellen für Suchtkranke/Drogenabhangige e.V. Caritas.

Litke, H. (1995). Projektmanagement. Methoden, Techniken, Verhaltensweisen. München: Hanser Ver­

lag.

Malorny, C. (1999). TQMUmsetzen. Stuttgart: Schaffer-Poeschel Verlag.

Moelier, J. (2001). The EFQM Excellence Model. German experiences with the EFQM approach in

health care. International Journal for Quality in Health Care, 13, 45-49.

Moeller, J., Breinlinger-O'Reilly, J., & Eiser, J. (2000). Quality management in German health care

rhe EFQM Excellence Model. International journal of health care quality assurance incorporating

Leadership in health services, 13, 254-258.

Nabitz, U. W & Klazinga, N. S. (1999). EFQM approach and the Dutch Quality Award. Interna­

tional Journal of Health Care Quality Assurance, 12, 65-70.

Nabitz, U. W & Walburg, J. W (1995). Das EFQM-Modell Jellinek. Ein Instrument zur strukturi-

erten und systematischen Selbstbewertung des Qualitatsmanagements von Einrichtungen des

Gesundheitswesens. In W. Rometsch (Ed.), Qualitatsmanagement in der ambulanten und sta­

tionaren Suchtkrankenhilfe(pp. 7-44). Munster: Landschaftsverband Westfalen Lippe.

212

EFQM single-case study

Pittrich, W., Rometsch, W., & Winkler, H. (2002). EFQM in der Ambulanten Sucbtkrankenbilfe.

Konzepte, Analysen, Praxisbeispiele. Munster: Landschaftsverband Westfalen Lippe.

Pursche, C , Nabitz, U., Muehl, J., & Winterberg, C. (1999). Arbeitsbuch EFQM Diagnose SB. Mun­

ster: Landschaftsverband Westfalen-Lippe.

Schaefer, A. (1997). Konzepüon der Suchtberatungsstelle. Bonn: Caritas Verband Bonn.

Schaefer, A. (2002). Nurwersich verandertbleibtsich treu. Jahresbericbt2000. Bonn: Caritas-Verband-

Bonn.

Schubert, H., & Zink, K.J. (2005). Qualitdtsmanagement im Gesundheits- und Sozialwesen.

Neuwied: Luchterhand.

Shaw, C. D. (2000). External quality mechanisms for health care: summary of the Expert project on

visitatie, accreditation, EFQM and ISO assessment in Eutopean Union countries. International

Journal for Quality in Health Care, 12,169-175.

Zikmund, W. G. (1988). Business Research Methods. New York: The Dryden Press.

Zink, K. (1995). TQMals integratives Managementkonzept. Das Europeiscbe Qualitdtsmodell und seine

Umsetzung. München: Carl Hanser Verlag.

213