UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl)

UvA-DARE (Digital Academic Repository)

Effectiveness of postgraduate education in occupational medicine

Smits, P.B.A.

Link to publication

Citation for published version (APA):Smits, P. B. A. (2002). Effectiveness of postgraduate education in occupational medicine.

General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s),other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, statingyour reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Askthe Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam,The Netherlands. You will be contacted as soon as possible.

Download date: 28 Aug 2020

CHAPTERR 4

Evaluatio nn of a postgraduat e educationa l programm ee fo r occupationa l physician s

onn wor k rehabilitatio n guideline s fo r patient s wit hh low-bac k pai n

AA controlled study1

P.B.A.. Smits1-2, J.H.A.M. Verbeek2, F.J.H, van Dijk 2, J.C.M. Metz\ Th.J. ten Cate4

11 Netherlands School of Occupational Health 22 Coronel Institute, Academic Medical Centre/University of Amsterdam

33 University Medical Centre St Radboud Nijmegen 44 University Medical Centre Utrecht

Abstrac t t Objective s s Thee quality of post-graduate education for occupational physicians is an impor-tantt issue in the field of occupational health. For that reason we have evaluated thee effectiveness of the training programme 'guidel ines for work rehabil i ta-tionn of low-back pain patients'. The first question is: to what extent does knowledgee of the guidelines increase. The second quest ion is: do the part ic i-pantss follow the guidelines better in their work after the programme.

11 This chapter is adopted from two articles: 1. 'Evaluation of a post-graduate educationaleducational programme for occupational physicians on work rehabilitation guidelinesguidelines for patients with low back pain'. Occup Environ Med 2000;57:645-6.6. (this article is inserted as an appendix to this thesis) and 2. 'Evaluatie vanvan onderwijs over de concept-richtlijn 'lage rugklachten' (Evaluation of educationeducation on work rehabilitation guidelines for patients with low back pain')pain') .TBV 2000;8:260-4.

67 7

CHAPTERCHAPTER 4

Method s s Inn a quasi -exper imental design with an experimental group (n = 25) and a ref-erencee group (n = 20) of physicians in a post-graduate occupat ional health t ra in ingg programme knowledge was assessed with a pre-test ( t l ) and a post-testt (t2) consisting of 45 true/false questions. The application of the guide-liness in pract ice was evaluated with pre- and post- t raining recordings of per-formancee indicators. The experimental group attended a one-and-a-half-day programmee about the guidelines within a six-day module about rehabi l i ta-t ion.. The reference group attended classes on other subjects. After six monthss they followed the same programme as the experimental group; after thatt programme the knowledge of the reference group was re-assessed (t3). Al ll results were examined with the paired samples t-test and the independent sampless t-test.

Result s s Knowledgee of the guidelines increased significantly in both the experimental groupp ( t l - t2: p<.001) and the reference group ( t l - t2: p<.01). The ad jus-ted gainn score is posit ive (independent samples t-test p<.05). The test score of thee reference group increased further after the training programme ( t2- t3: p<.001).. The exper imental group score on the performance indicators increasedd signif icantly after the training (pre- and post-recording: p<.001); thee reference group 's score decreased slightly (not signif icant, p>.05). The adjustedd gain score is posit ive ( independent samples t-test p < .001).

Conclusion s s Thiss educat ional programme was effective. Knowledge and performance in pract icee have improved and are more in compliance with the guidel ines. The knowledgee test showed a possible ' t ra in ing effect' at t2. The results and the possibi l i t iess of using performance indicators for evaluat ion of post-graduate educat ionn are discussed.

Keyword s s Programmee evaluat ion, post-graduate t raining, occupat ional physicians, low-backk pain, guidel ines

68 8

ControlledControlled study

Introductio n n

Improvementt of quality of care is regarded as a major challenge for the medicall profession.1 '2 The development of guidelines is seen as one wayy to improve qual i ty of care. Guidelines usually aim at changing thee performance of doctors who are already pract is ing medicine,3 yet i tt is obvious that they are also important for the training and educa-t ionn of doctors who are new to the field. They provide a clear frame-workk for how to act in pract ice, and performance indicators can be usedd to test whether medical performance is in compliance with the guidelines.. The need for quality of care is even more important because off the increasing demand for the accountabi l i ty of occupat ional healthh services. Recently an instrument for quali ty assessment of oc-cupat ionall rehabi l i tat ion for occupat ional physicians has been devel-opedd and evaluated.4 The conclusion is that it is an acceptable method whichh can be used for all kinds of medical audi t.

Inn the education and training of occupational physicians there is a great needd for evaluat ion of the effectiveness of teaching programmes.5 '6 '7

Thee ult imate aim of this educat ion is to contr ibute to desirable out-comess such as better health, fewer diseases and less work incapacity. Evaluatingg whether these programmes met this aim requires longitu-dinall studies that are difficul t to carry out. In the meantime it is worth-whil ee to find out if we can attain intermediate educat ional goals such ass an increase of knowledge and improvement of performance in medicall pract ice. To this end the same instruments can be used as in thee assessment of quali ty of care. For this reason, part of a post-graduatee training programme for occupat ional physicians was evalu-ated.. The research questions were whether part ic ipat ing in the educa-t ionall programme on occupat ional rehabi l i tat ion for low-back pain pat ientss increased knowledge of the guidelines among pract ising physicianss and whether the programme improved their pract ice per-formance.. The study was carried out in 1997.

Method s s

Thee relat ionship between assessment in theory and in practise can be

i l lustratedd using Mil ler ' s pyramid of clinical assessment.8 This pyra-

69 9

CHAPTERCHAPTER 4

<D <D o o c c CO O

E E i --O O t t cu u a. .

Figur ee 4.1. Miller's pyramid of clinical assessment

midd shows the relationship between assessment of knowledge ('knows') competencee ( 'knows how ' ), performance ( 'shows how') and action ( 'does' ).. We prefer to call the first three levels 'competence' and to calll the fourth level 'performance' (in pract ice).9 (figure 4.1)

Thee educat ional programme was evaluated by using a test of knowl-edgee and performance indicators to explore the four levels of the pyramid.. The content of the educational programme discussed in this art ic lee is based on the Dutch guidelines for occupat ional physicians onn work rehabi l i ta t ion of employees with low-back pain. This educa-t ionall p rogramme takes one-and-a-half-days within a module of six dayss on work rehabi l i ta t ion. It includes a lecture on the guidelines forr occupat ional physicians and for Dutch general pract i t ioners on low-backk pa in .1 0'1 1- 12 Another presentat ion is about referral to a backk pain cl inic. Finally the students are given feedback on the low-backk pain cases they submitted beforehand.

Thee results of this research project can be significant for the quality off the occupat ional physicians programme and, indirectly, for the qual i tyy of rehabi l i ta t ion of employees with low-back pain. There havee been earlier studies on the tasks of occupat ional physicians and onn specific aspects of consul tat ions.1 3-14 Along with the studies dis-cussedd above, the Coronel Institute investigated the effectiveness of occupat ionall heal th care and work rehabi l i tat ion for employees with low-backk pain.1 5»16 These investigations are combined in this project onn the effectiveness of educat ion.

70 0

ControlledControlled study

Ann exper imental group of 25 physicians took part in the educat ional programme,, a reference group of 20 physicians did not (they took partt in the programme six months later). The two groups included physicianss of two Corvu year groups and were similar on variables as agee and durat ion of work as a doctor. In the reference group there weree less women, but there were no systematic differences in knowl-edgee and performance scores between men and women. We assessed bothh groups with a pre-test and a post-test for knowledge. The per-formancee indicators were scored in both groups in the months before andd after the educat ional programme. This study design is known as 'quas i -exper imenta l ' .1 7'188 (table 4.1)

Thee knowledge tests were executed directly before and after the train-ing,, in the class room, with one week intervals, in the second half of 1997.. Both tests consisted of 45 true/false quest ions. The quest ions weree based on the study mater ia l. The pre-test and post-test scores weree analysed with the paired samples t-test. The reliabil i ty of the itemss in the quest ionnaire had been analysed with Cronbach 's alpha. Eightt quest ions from the pre-test were not included in the analysis so thee final pre-test included 37 quest ions. The pre-test alpha was 0.65. Inn the post-test all 45 quest ions could be included, with a Cronbach 's alphaa of 0.60. The Pearson correlat ion between the pre-test and the post-testt was not significant in either group (experimental group r = -.06;; p = .83 and reference group r=.40; p = .084).

Wee used 'performance indicators' for the assessment of the compli-

ancee with the guidelines on work rehabi l i ta t ion.4 A performance

indicatorr is an essential part of the care process and is an indicator

Tabl ee 4 .1 . Research design including an untreated reference group and pre-testss and post-tests: a quasi-experimental design.

Experimenta ll grou p t1 x t2

Referenc ee grou p t1 O t2 x t3

xx = Intervention: educational programme OO = No intervention t11 and t2: knowledge tests and performance scores t3:: knowledge test only

71 1

CHAPTERCHAPTER 4

Tabl ee 4.2. Description of twelve performance indicators.

Performanc ee indicato r (P)

PP 0 diagnosis

PP 1 activating approach

PP 2-1 psycho-social judgement

PP 2-2 psycho-social approach

PP 3-1 evaluation of medical treatment

PP 3-2 action / consultation off medical colleagues

PP 4-1 evaluation of work disabilities

PP 4-2 action for work adjustment

PP 5-1 evaluation of organisational obstructions s

Explanatio n n

medicall diagnosis; classification inn aspecific back pain, radiating backk pain, specific back pain, still unclear r

Activatingg approach if diagnosis is aspecificc back pain

Requestt for and assessment of psycho-sociall problems

Actionss in consulting hour if nec-essary y

Iss medical treatment adequate and // or impediment for rehabilitation

Actionn or consultation of medical colleaguess if necessary

Requestt for and assessment of disabilitiess for own current work andd necessary work adjustments

Actionss for (temporary) work adjustmentt if necessary

Requestt for and assessment of organisationall obstructions to workk rehabilitation

PP 5-2 action towards the organisation / Actions to remove organisational thee employer obstructions if necessary

P 66 advice on return to work

PP 7 revision policy

Advicee given or revision in appointmentt if necessary

Forr aspecific low-back pain: revi-sionn within three weeks

forr the quality of medical practice. Each performance indicator contains

cri teriaa which indicate the difference between good and bad qual i ty.19

Wee used twelve well-defined performance indicators, {table 4.2)

72 2

ControlledControlled study

Too assess the students' performance we asked them to randomly collect fromm their own practice five cases of low-back pain pat ients who had beenn absent from work in the months before the educat ional pro-grammee and five cases in the months after. The students were asked too fil l in a form for each of these cases. By compar ing these forms wit hh the performance indicators the researcher scored to what extent thee students had followed the guidel ines. One researcher assessed all thee cases. The possible score for each performance indicator was: 'good',, 'not good', 'not appl icable' and 'could not be assessed'. The scoree was expressed per case as a percentage of the performance indi-catorss that met the criteria versus all the assessed performance indi-catorss that were appl icable. For feedback purposes, this was done per case.. For every student we calculated a mean percentage of the per-formancee scores over all cases before and after the training. The scores andd differences were calculated with a paired samples t-test. To eval-uatee the possible influence of differences in performance scores at basee line we also used analysis of covariance to adjust for these dif-ferences.. The Pearson correlat ion between the performance scores beforee and after was not significant (exp. group r=.33; p = .18, refer-encee group r=.33; p=.29). To evaluate the educational programme we calculatedd a gain score and an adjusted gain score for both the knowl-edgee tests and the performance indicators. This adjusted gain score is definedd as the gain score of the experimental group minus the gain scoree of the reference group. The adjusted gain scores were analysed wit hh the independent samples t-test for groups.

Result s s

AA pre-test/post-test comparison of the knowledge test was possible for

177 physicians in the experimental group and 20 in the reference group.

Thee knowledge test scores at t l were low (70% and 67%). The score

off the experimental group after the training at t2 was 8 5 %, the score

off the reference group, wi thout t raining, 7 3% at t2. (table 4.3) With

true/falsee questions a score of about 80% is supposed to be signifi-

cantlyy different from that which can be at tained by chance alone. The

gainn scores were significantly positive for both the exper imental and

thee reference group. The adjusted gain score was 9%; this is signifi-

73 3

CHAPTERCHAPTER 4

Tabl ee 4.3. Results of pre-test and post-test on knowledge of the guidelines.

Experimenta l l grou pp (n = 17)

T=11 70% Meann % correct (S.D.) (11.67)

TT = 2 Meann % correct (S.D.

gainn score t1-t2 (95%% C.I.)

TT = 3 Meann % good (S.D.)

gainn score t2-t3 (95%% C.I.)

85% % (6.75) )

15%% + (7.88 -22.0)

Referenc e e roup( nn = 20)

67% % (10.42) )

73% % (6.68) )

6%(1.9-- 11.1)

84% % (5,67) )

11%* * (7,4-- 15,3)

Differenc ee exp . and referenc ee group s

3% %

12% %

9%## &

(0.4-- 16.4)

++ paired samples t- test, T(16)=4.45; p<.001 00 paired samples t- test, T(19) = 2.93; p<.01 ** independent samples t-test, T(35) = 2.16; p<.05 && adjusted for gain score reference group ** paired samples t-test, T{18) = 6.10; p<.001 (referencee group at T=3 was n = 19)

cantt (table 4.3). The reference group participated in the educational pro-

grammee six months later. At t3 we repeated the same knowledge test

off t 2. The knowledge score further increased to a score that was com-

parablee with that of the experimental group after the educat ional

p rogramme.. Not ice that the standard deviat ion of the scores for both

thee exper imental and the reference group decreased at t2 and even

furtherr at t3 (table 4.3).

Forr the performance indicators a pre-test/post-test comparison was pos-

siblee for 18 physicians in the exper imental group (a total of 90 low-

backk pain cases before and 87 cases after) and 12 physicians in the

referencee group (56 cases before and 52 cases after). The gain score

forr the exper imental group was significantly posit ive. For the refer-

encee group the gain score was negat ive, but this was not statistically

signif icant.. The adjusted gain score is 1 5% and this is significant

(table(table 4.4). The standard deviation of the experimental group decreased,

thee s tandard deviat ion of the reference group did not (table 4.4).

74 4

ControlledControlled study

Tabl ee 4.4. Pre- and post-training scores on performance indicators.

Experimenta l l grou p p (nn = 18)

Beforee training 80% Meann % correct (S.D.) (8.41)

Afterr training 9 1 % Meann % correct (S.D.) (5.79)

gainn score 1 1 % +

(95%% C.I.) (6.6-15.0)

Referenc e e grou p p (nn = 12)

87% % (7.24 4

83% % (7.36) )

-4%(-9.3-1.5) )

Differenc e e experimenta ll and referenc ee group s

- 7 % %

8% %

15% # & &

(8.2-21.2) )

++ paired samples t-test (before / after training), T(17) = 5.41; p<.001 00 paired samples t-test (before / after training), T(11 ) = -1.60; p = .138 ## independent samples t-test (difference exp / reference), T(28) =4.67; p< ,001 && adjusted for gain score reference group

Discussio n n

Inn this study we have shown that there was an increase in knowledge

andd an improvement of performance in practice after a training pro-

grammee on low-back pain. These results could be biased in several ways.

First,, we did not use a randomised study design. For this reason the

exper imentall and reference groups might differ in other aspects than

justt the training programme which we could not measure in this

study.. It is possible that the groups differed in learning style, which

iss known to influence the outcome of a training programme.20 Apart

fromm this factor also other factors such as the rest of the training

programme,, locat ion and time of testing differed between the two

groups.. However, these differences would be more likely to bias the

resultss in the direction of not finding a difference between the two

groups.. The same holds for the increase in score on the knowledge test

off the reference group without a training programme, which is probably

duee to taking the test twice. Therefore we used adjusted gain scores

whichh take this effect into account. Even then we found a significant

differencee between the experimental and the reference groups. More-

over,, for the performance indicators there was no such increase for

thee reference group. Therefore we think that the increases in knowl-

edgee and performance can be ascribed to the training programme.

75 5

CHAPTERCHAPTER 4

Secondly,, the validity of our measurement instruments can be questioned.

Thereforee we measured the interna! consistency as a measure of the

validityy of our knowledge test. To increase the internal consistency we

leftt out a couple of quest ions that did not seem to be related to the

testt as a whole. The final version of the test yielded a Cronbach's alpha

off 0.65 pre-test and 0.60 post-test which is considered to be sufficient

forr programme evaluat ion purposes.2 1'22 The performance indicators

weree partly based on a physician's report of what they had themselves

donee in a part icular case. The criteria we used to assess deviation from

thee guidel ines for a specific performance indicator were not known

explicitlyy by the part ic ipants. Therefore we think that the performance

scoree based on these indicators does reflect real performance in prac-

ticee and not just competence. Moreover it was shown that performing

welll as measured by the same indicators predicted a better outcome

andd greater pat ient sat isfact ion.23 However, these indicators can

onlyy reflect a part of total performance because it is impossible to

coverr all aspects by using one method only. Our method covers the

moree cognit ive aspects but not the doctor-pat ient interact ion which

cann better be studied by using video record ings.24 To overcome these

difficultiess of measurement it would be better to relate the training

programmee to outcome in practice. For vocat ional rehabi l i tat ion of

low-backk pain pat ients return to work should be measured and com-

paredd between pat ients treated by physicians from the reference and

thee intervent ion groups. For future research we advocate a ran-

domisedd design and a more extended measurement of outcome.

76 6

ControlledControlled study

Reference s s

11 Kassirer JP. The quality of care and the quality of measuring it. New Engl J Med 1993;329:: 1263-5.

22 Hulshof CTJ, Verbeek JHAM, Dijk FJH van, Weide WE van der, Braam ITJ. Evaluation researchh in Occupational Health Services: General principles and a model for OHS evaluation.. In: Hulshof CTJ. Prevention and control of adverse effects of whole-bodyy vibration: an evaluation study in occupational health services. Amsterdam: Universityy of Amsterdam, 1998. Dissertation.

33 Grol R. Implementing guidelines in general practice. Quality in Health Care 1992;1:184-91. .

44 Weide WE van der, Verbeek JHAM, Dijk FJH van, Hulshof CTJ. The development and evaluationn of a quality assessment instrument for occupational physicians. Occup Environn Med 1998;55:375-82.

55 Petersen S. Time for evidence based medical education. Tomorrow's doctors need informedd educators not amateur tutors. BMJ 1999;318:1223-4.

66 Verbeek JHAM, Kroon PJ. Education and training in occupational health and safety. Safetyy Science 1995;20:iii-iv.

77 Sackett DL, Richardson WS, Rosenberg W, Haynes B. Evidence-based medicine: how too practice and teach EBM. New York:Churchill Livingston, 1997.

88 Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;; 65: S63-7.

99 Ram P. Eindverslag Project Integrale Toetsing, Werkgroep Onderzoek Kwaliteit (Finall report of a project on integral evaluation, Working group on quality research). Maastricht:: University of Maastricht, 1997.

100 Verbeek JHAM, Weide WE van der. Begeleiding van werknemers met lage rug-klachtenn (Work rehabilitation of workers with low-back pain). Richtlijnen Arbo zorg. B111 3-2-1 t/m 34. Den Haag: Delwel, 1997.

111 Faas A, Chavannes AW, Koes BW et al. NHG standaard Lage-Rugpijn (Guidelines: low-backk pain for general practitioners). Huisarts Wetenschap 1996;39(1):18-31.

122 Smeele IJM, Hoogen JMM van den, Mens JMA et al. NHG standaard Lumbosacraal radiculairr syndroom (Guidelines: lumbosacral radiculair syndrome for general prac-titioners).. Huisarts Wetenschap 1996;39(2):78-89.

133 Smits PBA, Kroon PJ, Dijk FJH van. Taken van de bedrijfsarts (The tasks of the occupationall physician). Tijdschr Soc Gezondheidsz 1992;70:211 -4.

144 Smits PBA, Verbeek JHAM. Consultvoering door bedrijfsartsen, een actieve en prob-leemgerichtee werkwijze (Consultation by occupational physicians: an active and problem-orientedd working-method). Tijdschr Bedrijfs- en Verzekeringsgeneesk 1995;3:129-34. .

11 11

CHAPTERCHAPTER 4

155 Weide WE van der, Verbeek JHAM, Dijk FJH van. An audit of occupational health care forr employees with low-back pain. Occup Med 1997;47:294-300.

166 Weide, WE van der. Quality of occupational rehabilitation for low-back pain, mea-surementt of process and outcome. Amsterdam: University of Amsterdam, 1998. Dissertation. .

177 Baarda DB, de Goede MPM. Methoden en technieken, praktische handleiding voor het opzettenn en uitvoeren van onderzoek (Methods and techniques: practical guide-liness for setting up and carrying out research). Houten: Stenfert Kroese, 1995.

188 Cook, TD and Campbell, DT. Quasi-experimentation: design and analysis issues forr field settings. Boston:Houghton Mifflin Company, 1979.

199 Agency for Health Care Policy and Research. Using clinical practice guidelines to eval-uatee quality of care. Volume 2: Methods. AHCPR Pub no.95 - 0046. USDHHS, PHS.AHCPR,, Rockville USA, 1995.

200 Vermunt JHDM. Leerstijlen en sturen van leerprocessen in het hoger onderwijs: naarr een procesgerichte instructie in zelfstandig denken. Ti lburg: Katholieke Universiteitt Brabant, 1992. (Learning styles and regulation of learning processes in higherr educat ion: towards process-oriented training in indepedent thinking. Dissertation.) )

211 Schotten JGM, Robroek WCL. Verantwoord meten in het gezondheidszorgonderwijs, handboekk voor beoordeling van leerresultaten (Responsible measuring in health education:: a handbook for the assessment of learning outcomes). Houten/Diegem: Bohnn Stafleu Van Loghum, 1997.

222 Metz JCM, Scherpbier AJJA, Vleuten CPM van der (reds). Medisch onderwijs in de praktijkk (Medical education in practice). Assen: van Gorcum, 1995.

233 Weide WE, Verbeek JHAM, Dijk FJH van. Relationship between indicators for quali-tyy of occupational rehabilitation of employees with low-back pain. Occ Env Med 1999;56:488-93. .

244 Ram P. Comprehensive assessment of general practitioners: a study on validity, reliabilityy and feasibility. Maastricht: University of Maastricht, 1998. Dissertation.

78 8