The Emergency Physician and KnowledgeTransfer: Continuing Medical Education,Continuing Professional Development,and Self-improvementBarbara J. Kilian, MD, Louis S. Binder, MD, Julian Marsden, MD

AbstractA workshop session from the 2007 Academic Emergency Medicine Consensus Conference, KnowledgeTranslation in Emergency Medicine: Establishing a Research Agenda and Guide Map for Evidence Uptake,focused on developing a research agenda for continuing medical education (CME) in knowledge transfer.Based on quasi-Delphi methodology at the conference session, and subsequent electronic discussion andrefinement, the following recommendations are made: 1) Adaptable tools should be developed, validated,and psychometrically tested for needs assessment. 2) ‘‘Point of care’’ learning within a clinical contextshould be evaluated as a tool for practice changes and improved knowledge transfer. 3) The addition ofa CME component to technological platforms, such as search engines and databases, simulation technol-ogy, and clinical decision-support systems, may help knowledge transfer for clinicians or increase utiliza-tion of these tools and should, therefore, be evaluated. 4) Further research should focus on identifying theappropriate outcomes for physician CME. Emergency medicine researchers should transition from previ-ous media-comparison research agendas to a more rigorous qualitative focus that takes into account needsassessment, instructional design, implementation, provider change, and care change. 5) In the setting ofcontinued physician learning, barriers to the subsequent implementation of knowledge transfer and behav-ioral changes of physicians should be elicited through research.

ACADEMIC EMERGENCY MEDICINE 2007; 14:1003–1007 ª 2007 by the Society for Academic EmergencyMedicine

Keywords: continuing medical education, physician education, evidence-based medicine, knowledgetranslation, continuing professional development

In 2003, the Institute of Medicine issued Health Pro-fessions Education: A Bridge to Quality, focusing onfive key areas to assist in bridging the gap between

knowledge and practice.1 One component of usingknowledge transfer to close the gaps between current ev-

From the Department of Emergency Medicine, St. Luke’s Roose-

velt Hospital Center (BJK), New York, NY; MetroHealth Medical

Center and Case Western Reserve University (LSB), Cleveland,

OH; and Department of Emergency Medicine, St. Paul’s Hospital

(JM), Vancouver, British Columbia, Canada.

Received June 8, 2007; revisions received July 11, 2007, and July

22, 2007; accepted July 23, 2007.

This is a proceeding from a workshop session of the 2007

Academic Emergency Medicine Consensus Conference,

‘‘Knowledge Translation in Emergency Medicine: Establishing

a Research Agenda and Guide Map for Evidence Uptake,’’

Chicago, IL, May 15, 2007.

Workshop participants included Barbara J. Kilian, Julian Mars-

den, Louis S. Binder, David H. Newman, William Bond, Brian

K. Nelson, Andrew Worster, Suneel Upadhye, and Chris Tselios.

Contact for correspondence and reprints: Barbara J. Kilian, MD;

e-mail: bkilian@chpnet.org.

ª 2007 by the Society for Academic Emergency Medicine

doi: 10.1197/j.aem.2007.07.008

idence and current practice lies with understanding andimproving the effectiveness of continuing medical edu-cation (CME) programs. There is a vast array of barriershindering this understanding of CME, the first being thatthere is a lack of consensus on what defines CME.2–5

CME is defined by the Accreditation Council on Continu-ing Medical Education as any educational activity thatworks to maintain, develop, or increase knowledge,skills, performance, and relationships that a physicianuses in the care of patients.6 A coordinated effort oforganizations involved is necessary to better delineatewhat CME is as an educational entity.4

A second major barrier to better understanding CMEcenters on the vast amount of literature that has failedto answer many key questions in CME. Despite years ofresearch into different educational techniques, knowl-edge outcome measurements, practice behavior, and clin-ical practice outcomes, only broad themes can be inferredto guide the practitioner and CME developers. This is pri-marily due to a heterogeneous mix of methodologies, endpoints, and quality exhibited in these studies. Multiplesystematic reviews have therefore been able only to iden-tify trends,7–14 namely, that print media is less effective

ISSN 1069-6563

PII ISSN 1069-6563583 1003

1004 Kilian et al. � CME AND KT IN EM

than live media, multiple exposures are better than singleexposure, interactive techniques are better than noninter-active ones, and didactic lectures alone are relatively inef-fective. CME has, however, been shown in these reviewsto be effective to varying degrees in enabling practi-tioners to acquire and retain knowledge, attitudes, skills,and behaviors associated with positive clinical outcomes.

Several other significant barriers impede the under-standing of CME by practitioners. There have not beenadequate needs assessments for CME.12,13,15–17 Thislack of educational needs assessment is a significant bar-rier to closing the knowledge-to-practice gaps. For anyeducational intervention to be effective, a clearer under-standing of the need, by practitioners and CME devel-opers, must be obtained. In addition, few studies havesuccessfully quantified the effect of CME on knowledgetranslation (KT).12 Research methods utilized to datehave been unable to adequately assess the impact of ed-ucational interventions on physician knowledge, practicebehavior, and patient outcomes.8,10,12,13,18

It has been postulated that a number of factors maypositively impact improved knowledge transfer of CMEprograms. These include the need for point-of-careCME through innovative delivery systems, better match-ing of CME selection to practice setting, increases inactive learning formats, and recognition of multiple edu-cational outcomes. Further investigation to establish theindividual importance and contribution of each of thesefactors is critical.

To improve knowledge transfer in CME, we must ex-amine multiple factors impacting the overall problem.The goal of this conference proceedings article is to iden-tify the most important factors for analysis and to framerecommendations for research questions for KT investi-gators in emergency medicine that can form the basisfor productive research directions in future years. Basedon quasi-Delphi methodology at the conference session,and subsequent electronic discussion and refinement,the following recommendations are therefore made.

RECOMMENDATION 1

For CME to be effective for physicians, learning needsassessment is a crucial step.9,10,13,17,18 Adaptable toolsfor needs assessment for physicians’ continuing educa-tion (both individual and in-common) should be devel-oped and validated, as well as psychometrically evaluated.

DiscussionPhysicians learn best when they recognize that they havea need that should be addressed. Knowledge and prac-tice gaps can be identified through a variety of means,including formal, informal, individual, or group needsassessments, normative needs assessment (expert opin-ion), or comparative needs analysis (group comparisons).Because needs may also be expressed as individual,organizational, clinical, administrative, subjective, andobjective, it is important that the role of the needs assess-ment be clearly defined before its implementation.15

Barriers to changing practice are not solely based ineducational deficiencies; attitudinal issues such as lackof motivation and external barriers such as lack of time,resources, or reimbursements are also important.19 The

incorporation of needs assessment in developing CMEprograms can help delineate if the barriers to changeare educational, procedural, or both. Needs assessmentalso closes the learning cycle and aids in providing objec-tive data that facilitate decisions on content, venue, andformat for program developers.

A multilevel (local, regional, national, and interna-tional) needs assessment may better inform us wherecommon gaps exist in order to design more effectiveeducational interventions. However, a CME/continuingprofessional development needs assessment tool is bestdirected at the level of the individual practitioner,because learning needs may vary between physicians,even within a single group. A common tool that is easilyadapted to the different practice settings and clinical con-tent areas should be developed to define the key needs as-sessment questions. Individual practitioner’s answers willhelp identify his or her unique CME needs. Recognition ofpractitioners who do not participate in CME, either as aresult of unperceived needs or due to current deficienciesin CME in meeting needs, is an important component inthe analysis of CME. A well-developed tool that supportsthe clinician to self-define CME may also be important.Practitioners should have confidence that their CMEactivity can improve their practice and patient care.

Research into the validity and effectiveness of suchtools in relation to KT indicators is necessary. Further-more, the issue of how to operationalize the use of sucha tool needs to be addressed, particularly in making it areal-time tool to increase clinical relevance.

Further Study

1. What are the core KT topics in emergency medicinethat should be addressed?

2. What are the educational needs surrounding thosetopics that should be addressed?

3. How can needs assessments be conducted on local,regional, national, and international levels?

4. Is there a central needs assessment tool that can beused to address these issues?

5. Can needs assessments identify all important learningneeds?

6. Does the utilization of needs assessment tools helpdelineate educational deficiencies from proceduralbarriers?

7. What is the validity and effectiveness of such a needsassessment tool in terms of KT indicators?

8. Can learning assessment tools be effectively dissemi-nated and utilized by practitionersand CME developers?

9. Can learning content management systems, whichtrack learning needs and accomplishments, be usedto effectively target CME and KT gaps?

Evidence SummaryData and expert opinion.

RECOMMENDATION 2

‘‘Point of care’’ (PoC) CME is a rapidly developing new tech-nology that utilizes physicians’ real-time experiences tomaximize physicians’ CME activities. It potentially en-hances the practitioner’s ability to apply the educational in-formation in real time. PoC is a structured, computer-based

ACAD EMERG MED � November 2007, Vol. 14, No. 11 � www.aemj.org 1005

(Internet or personal digital assistant), self-directed activitythat encompasses a clinical question based on patient care,a review of the online resource(s), and an evaluation of theapplication to the patient. Elements of PoC CME can beasynchronous, allowing physicians to evaluate the impacton patient care at a later time. PoC CME signals a shift inCME from a static, ‘‘top-down’’ method of education to adynamic, practice-based, learner-driven, ‘‘bottom-up’’mode of education.

DiscussionConsistent with the theme of adapting the needs assess-ment into a real-time tool, PoC CME is being developedto facilitate movement of learning into the clinical setting.PoC learning is now being accredited, in recognition ofthe increasing capability for, and the potential benefit,from patient-centered, real-time clinical educationalevents. Several issues need further exploration, such asdefining the best KT strategy to ensure its incorporationinto the practice setting in a user-friendly fashion that 1)is responsive to clinician time constraints, 2) assesses thereliability of the CME accessed by the practitioner, and 3)evaluates the effectiveness of this KT strategy in terms ofimplementation success, as well as patient care andsystem performance indicators.

Further Study

1. How do you evaluate if physicians have adequatelyutilized PoC CME interventions?

2. Does PoC learning, within a clinical context, result inpractice changes and improved knowledge transfer?

3. What configuration of PoC learning is most effectivein knowledge transfer?

4. When is PoC CME best utilized?5. What configuration of PoC learning leads to actual use

in practice?6. How does PoC learning actually occur?

Evidence SummaryData and expert opinion.

RECOMMENDATION 3

Various technological platforms and configurations havethe potential to enhance physician education and catalyzeknowledge transfer, including literature search enginesand databases, clinical decision support systems, and otheractive learning pedagogies. The role of CME within thesevarious educational interventions warrants investigation.

DiscussionChanging physicians’ practice to one that utilizes thebest evidence is a multifaceted, complex issue. Severalemerging technologies are being developed to addressmany of these issues. Computer decision support (CDS)systems, centralized databases specific to emergencymedicine, and simulation exercises are all tools beingdeveloped to help close these gaps. However, to date,few studies have looked into the incorporation of CMEcomponents to these technologies. The role of CME,coupled with other technologies, has the potential to in-crease utilization of these tools and to move away froma nonclinical, disconnected system of attaining CME

for credit alone. For example, CDS systems function asreal-time, clinically-oriented guides for practice. BecauseCDS systems are being hailed as a tool to help improvepatient outcomes, potential CME overlaps should beformally evaluated. Attention needs to focus on whateducational potential can be incorporated into and har-nessed from CDS systems. Likewise, there is a need fordevelopment of literature-searching engines and emer-gency medicine–specific databases that provide accessto clinically relevant, evidence-based materials.

Learning at the PoC may not always occur during thedirect clinical care context. Medical simulation createscontextual learning environments in which safe, experi-ential, and reflective learning can occur. One type ofmedical simulation, termed ‘‘in situ’’ simulation, bringsmannequin-based simulators into the practice environ-ment so that the contextual background remains inplace.20 These types of simulation can address technicalcare issues and team behavioral issues. These activitiesare resource intense and challenging to create and haveyet to be incorporated as CME. In theory, this type oftraining could perform multiple functions: identifying un-met needs, identifying contextual barriers to change inpractice, identifying changing cultural beliefs that maybe barriers to behavior change, and raising physicianand nursing awareness simultaneously.

Further Study

1. How effective are these tools in a CME context inimproving knowledge transfer?

2. Does a CME component increase utilization of thesetools? Can CME act as an incentivizing tool to utilizesuch electronic tools?

3. Does the addition of CME credit narrow the KT gap?4. Can we observe physicians in practice and see which

tools they are choosing and then ask them why?

Evidence SummaryData and expert opinion.

RECOMMENDATION 4

Further research should define what the desirable out-come(s) for physician CME should be. Changes in clinicalpractice may be one such outcome, but other outcomesmay be measurable and useful. We should transitionfrom our current media-comparison research agendasto a more rigorous, qualitative focus21–25 that takes intoaccount needs assessment, instructional design, imple-mentation, provider change, and care change.

DiscussionMore emphasis should be placed on defining clearoutcomes to judge KT success. Change in physicianbehavior is the most obvious and arguably the most im-portant of these outcomes. However, other outcomesare also likely.26–28 A new paradigm that recognizes a va-riety of learning outcomes (e.g., self-reflection, move-ment through stages, continuous practice assessment,affective change, reinforcement/reaffirmation of currentpractice, new skills gained for new positions) is war-ranted. It must also recognize the relevance of learning,accountability, and patient outcomes. This shift has the

1006 Kilian et al. � CME AND KT IN EM

Figure 1. American Medical Association (AMA) outcomes measures. CME = continuing medical education. The Continuing

Professional Development of Physicians. ª 2003, American Medical Association.

potential to alter the focus of CME from an attendance-based measurement to one that has more relevance toboth practitioners and patients. The American MedicalAssociation has proposed a model for outcome measure-ment that includes six levels26 (Figure 1). A significantportion of CME has relied only on levels 1 and 2 as out-come measurements. The recognition and adoption ofhigher levels of outcome assessment is important ifCME is to have an effect on changing practice and main-taining the competency of physicians.26

The role of CME is not simply to change practice forthe sake of change, but it is to confirm clinical compe-tency for the individual emergency physician and tohelp the practicing physician develop as a professional,5

not only as a dispenser of medical care. Theoretically,both outcomes should improve clinical patient care.

Further Study

1. What outcomes beyond behavior change are desirablein CME activities?

2. How can these outcomes be measured and quantifiedamong practicing physicians?

3. What characteristics of CME interventions result inthe best knowledge transfers relative to all outcomesidentified?

4. Does measurement/reflection/recognition of move-ment through these stages enhance or predisposephysicians to move to ‘‘higher’’ levels of movement(behavioral change, and so on)?

Evidence SummaryData and expert opinion.

RECOMMENDATION 5

In the setting of continued physician learning, barriers tothe subsequent implementation of knowledge transfer and

behavioral changes of physicians should be elicited throughresearch. Effective strategies for enhancing knowledgetransfer in the immediate aftermath of CME should be eval-uated for their effectiveness in overcoming these barriers.

DiscussionA great mystery exists surrounding the time betweencompletion of CME and a practitioner returning to pa-tient care. Most outcome modalities focus on the edu-cational intervention only. However, a closer look atwhat is happening between the end of the CME andthe implementation of (or lack of) new information andpractice at the practitioner’s home setting is warranted.A number of potential noneducational barriers existthat may impede a motivated physician returning froma CME activity.19,29,30 Examples might include returningto the home practice setting following an ultrasoundcourse in the absence of an ultrasound machine orreturning from an advanced airway course to an emer-gency department without airway management adjunctsor without privileging for procedural sedation. Otherbarriers may include local politics, lack of staff or nursingsupport, lack of support from outside the emergency de-partment (i.e., initiating hypothermia for cardiac arrestsurvivors when the medical intensive care unit cannotmanage such technology and protocols), differencesbetween the local setting and the setting reflecting thecontent of the CME information, and factors within thepsychology and motivation for behavioral change.

Further Study

1. What are the various noneducational barriers thatimpede implementation of new skills/knowledge forpractitioners?

2. Once barriers are elucidated, how can these barriersbe surmounted?

ACAD EMERG MED � November 2007, Vol. 14, No. 11 � www.aemj.org 1007

Evidence SummaryData and expert opinion.

CONCLUSIONS

Our breakout session from the 2007 Academic EmergencyMedicine Consensus Conference on KT in emergencymedicine focused on the role of CME in knowledge trans-fer and in setting a research agenda in this area. Severalareas of research priority were identified and are reportedhere. Both quantitative and qualitative research methodsmay be usable in the delineation and evaluation of thesevarious research recommendations.

References

1. Greiner AC, Knebel E. Health Professions Education:A Bridge to Quality. Washington, DC: Institute of Med-icine, National Academy of the Sciences Press, 2001.

2. Council of Medical Specialty Societies. Competencyareas for CME professionals: Alliance for ContinuingMedical Education. Available at: http://www.cmss.org/images/compareas.pdf. Accessed Jul 23, 2007.

3. Bellande BJ. The CME professional: challenges andopportunities in reforming CME. J Contin EducHealth Prof. 2005; 25:203–9.

4. Bennett NL, Davis DA, Easterling WE Jr, et al. Continu-ing medical education: a new vision of the professionaldevelopment of physicians. Acad Med. 2000; 75:1167–72.

5. MannK.ContinuingMedicalEducation. In:NormanGR,van der Vleuten CPM, Newble DI (eds). Springer In-ternational Handbooks of Education. Norwell, MA:Kluwer Academic Publishers, 2002.

6. Accreditation Council for Continuing Medical Educa-tion. ACCME Glossary of Terms and Abbreviations.Chicago, IL: Accreditation Council for ContinuingMedical Education, 2006, p. 7.

7. Bloom BS. Effects of continuing medical education onimproving physician clinical care and patient health: areview of systematic reviews. Int J Technol AssessHealth Care. 2005; 21:380–5.

8. Davis D, O’Brien MA, Freemantle N, Wolf FM,Mazmanian P, Taylor-Vaisey A. Impact of formalcontinuing medical education: do conferences, work-shops, rounds, and other traditional continuing edu-cation activities change physician behavior or healthcare outcomes? JAMA. 1999; 282:867–4.

9. Davis DA, Thomson MA, Oxman AD, Haynes RB.Evidence for the effectiveness of CME. A review of 50randomized controlled trials. JAMA. 1992; 268:1111–7.

10. Davis DA, Thomson MA, Oxman AD, Haynes RB.Changing physician performance. A systematic re-view of the effect of continuing medical educationstrategies. JAMA. 1995; 274:700–5.

11. Grimshaw JM, Shirran L, Thomas R, et al. Changingprovider behavior: an overview of systematic reviewsof interventions. Med Care. 2001; 39(Suppl 2):II2–45.

12. Marinopoulos SS, Dorman T, Ratanawongsa N, et al.Effectiveness of Continuing Medical Education.Evidence Report/Technology Assessment No. 149.(Prepared by the Johns Hopkins Evidence-based

Practice Center, under Contract No. 290-02-0018).2007. AHRQ Publication No. 07-E006.

13. Oxman AD, Thomson MA, Davis DA, Haynes RB.No magic bullets: a systematic review of 102 trialsof interventions to improve professional practice.CMAJ. 1995; 153:1423–31.

14. Thomson O’Brien MA, Freemantle N, Oxman AD,Wolf F, Davis DA, Herrin J. Continuing educationmeetings and workshops: effects on professionalpractice and health care outcomes. Cochrane Data-base Syst Rev. 2001; (2):CD003030.

15. Grant J. Learning needs assessment: assessing theneed. BMJ. 2002; 324:156–9.

16. Igarashi M, Suveges L, Moss G. A comparison of twomethods of needs assessment: implications for con-tinuing professional education. Can J Univ ContinEd. 2002; 28:57–76.

17. Mazmanian PE, Davis DA. Continuing medical edu-cation and the physician as a learner: guide to theevidence. JAMA. 2002; 288:1057–60.

18. Reed D, Price EG, Windish DM, et al. Challenges insystematic reviews of educational intervention stud-ies. Ann Intern Med. 2005; 142:1080–9.

19. Cabana MD, Rand CS, Powe NR, et al. Why don’tphysicians follow clinical practice guidelines? Aframework for improvement. JAMA. 1999; 282:1458–65.

20. Hamman W. The complexity of team training: what wehave learned from aviation and its applications to med-icine. Qual Saf Health Care. 2004; 13(Suppl 1):i72–9.

21. Cook D. The research we still are not doing: anagenda for the study of computer-based learning.Acad Med. 2005; 80:541–8.

22. Cook D. Where are we with Web-based learning inmedical education? Med Teach. 2006; 28:594–8.

23. Friedman C. The research we should be doing. AcadMed. 1994; 69:455–7.

24. Kean DN, Norman GR, Vickers J. The Inadequacy ofRecent Research on Computer-assisted Instruction.Acad Med. 1991; 66:444–8.

25. Norman G. RCT = results confounded and trivial: theperils of grand educational experiments. Med Educ.2003; 37:582–4.

26. Davis D, Barnes BE, Fox R (eds). The ContinuingProfessional Development of Physicians: From Re-search to Practice. Chicago, IL: American MedicalAssociation Press, 2003.

27. Slotnick HB. How doctors learn: physicians’ self-di-rected learning episodes. Acad Med. 1999; 74:1106–17.

28. Slotnick HB, Mejicano G, Passin SM, Bailey A. Theepidemiology of physician learning. Med Teach.2002; 24:304–12.

29. Davis DA, Taylor-Vaisey A. Translating guidelinesinto practice. A systematic review of theoretic con-cepts, practical experience and research evidence inthe adoption of clinical practice guidelines. CMAJ.1997; 157:408–16.

30. Dopson S, FitzGerald L, Ferlie E, Gabbay J, Locock L.No magic targets! Changing clinical practice tobecome more evidenced based. Health Care ManageRev. 2002; 27:35–47.