Embed Size (px)
Citation preview
Contrasting Cases – Resources
Keep B. The Science of Learning, Episode #7
https://gohighbrow.com/contrasting-cases/
Alfieri L, et al. (2013) Learning Through Case Comparisons: A Meta-Analytic Review.
http://www.lrdc.pitt.edu/schunn/papers/ContrastingCasesMeta-AlfieriEtAl2013.pdf
Schwartz, D. L., Tsang, J. M. & Blair, K. P. (2016). C is for Contrasting Cases: Discerning Critical
Information. In The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and
When to Use Them (pp. 26-38). New York, NY: W. W. Norton & Company. [Stanford Graduate School of
Education]. (2015, Aug 30).
Daniel Schwartz on learning to perceive [Video File].
https://www.youtube.com/watch?v=UHeuV6FG8Q8
Contrasting Case Examples
Podcast: https://elearning.asco.org/product-details/contrasting-cases-tailorx--chemotherapy-or-no
Podcast: https://elearning.asco.org/product-details/contrasting-cases-systemic-therapy-for-metastatic-
triple-negative-breast-cancer
Podcast: https://elearning.asco.org/product-details/contrasting-cases-cancer-prevention
Spaced Learning – Online Resources Andriotis N. Don't Forget About It: How Spaced Repetition and Microlearning Boost Memory.
eLearning Industry. https://elearningindustry.com/how-microlearning-spaced-repetition-beat-forgetting-curve. Published February 10, 2020. Accessed May 1, 2020.
Bill-Brandon. Designs That Work: Spaced Learning. Learning Solutions Magazine.
https://learningsolutionsmag.com/articles/designs-that-work-spaced-learning. Accessed May 1, 2020. Casebourne I. Spaced Learning: An Approach to Minimize the Forgetting Curve. Main.
https://www.td.org/insights/spaced-learning-an-approach-to-minimize-the-forgetting-curve. Published December 8, 2017. Accessed May 1, 2020.
Denny J. What is the Forgetting Curve (And How Do You Combat It)? eLearning Industry.
https://elearningindustry.com/forgetting-curve-combat. Published April 17, 2018. Accessed May 1, 2020. Eakin-Love A. Learning How to Use Spaced Repetition in Medical School - Osmosis - A Better
Way To Learn. Osmosis. https://www.osmosis.org/blog/2019/04/08/learning-how-to-use-spaced-repetition-in-medical-school. Published April 8, 2019. Accessed May 1, 2020.
Finkenbinder, E. O. The Curve of Forgetting. The American Journal of Psychology, vol. 24, no. 1,
1913, pp. 8–32. JSTOR, www.jstor.org/stable/1413271 Accessed May 1, 2020. Pappas C. 5 Tips To Succeed In Instructional Design For Spaced eLearning. eLearning Industry.
https://elearningindustry.com/succeed-instructional-design-spaced-elearning. Published March 12, 2018. Accessed May 1, 2020.
Pappas C. 6 Tips To Develop Spaced eLearning Courses. eLearning Industry.
https://elearningindustry.com/6-tips-develop-spaced-elearning-courses. Published March 13, 2018. Accessed May 1, 2020.
Thalheimer W. Learning and Forgetting Curves -- In Depth [Video]. YouTube.
https://www.youtube.com/watch?v=lItgAV6Ly6M. Published May 24, 2012. Accessed May 1, 2020. Thalheimer W. Spacing Learning Events Over Time: What the Research Says . A Work-
Learning Research, Inc.; 2006. https://www.worklearning.com/wp-content/uploads/2017/10/Spacing_Learning_Over_Time__March2009v1_.pdf. Accessed May 1, 2020.
1
EDUCATION PROGRAM EVALUATION STRUCTURED FORMAT
Instructions: Below are the headings for the structured format. There is an explanation of the section under each heading and a snippet of an example. Please feel free to use this template as a foundation and change it to fit your program – delete sections that don’t apply to your organization and add in new sections to correspond with the way your program functions. You’ll notice that next to many of the headings are the ACCME and ANCC criteria numbers in parentheses. These criteria numbers can assist you to match that heading to those criteria as you describe your activities for the year. When it is time to create your self-study, you will be able to utilize your previous years’ annual program evaluations to assist with that process.
1. Executive Summary - Major Accomplishments Save this section for last. After you have completed the entire evaluation, review it and select the key items that you want to highlight to your stakeholders and bullet them in this section. This section can also provide a one-page summary for your audience if they do not have the time or opportunity to review the entire document. This section can also stand alone for reporting out at other meetings.
Example of items to highlight:
New activities: 69 new activities were created in 2019; 35 CME (8 live and 27 online), 25 CME/CNE activities (6 live and 16 online with 1 that was CNE only) and 9 non-CE (7 live and 2 online).
Approved provider of continuing nursing credit from the Louisiana State Board of Nursing: an application was submitted and approval awarded to be an approved provider of continuing nursing credit. LAMMICO/Medical Interactive is now listed on their website.
MOC Part II approval: 98% of the eligible courses have approval by at least one board.
Decreased the average time of production from first draft to go-live: the average time decreased from 3.28 months in 2018 to 2.4 months in 2019.
2. Evaluation of the 20xx Goals and Strategic Plan (C13; QO1) If you have a strategic plan or targets for the year, you can use this section to specifically address each one.
Example of one way to display the data:
Goal Objectives Evaluation
Increase the number of continuing medical, nursing, and practice solution education courses
Increase the number of CME/CNE/non-CE topics by a total of 30 added to the online library
o 69 new activities were created in 2019; 35 of them were CME (8 live and 27 online), 25 were CME/CNE activities (6 live and 16 online with 1 that was CNE only) and 9 non-CE (7 live and 2 online).
o Total online library is now at 290 courses with a net of 42 courses.
o Objective was met and greatly exceeded.
Meet contractual obligations
o Three courses that were created for a client were for renewal. They were reviewed and updated in 2019.
2
Goal Objectives Evaluation
o Submitted 4 quarterly newsletters that were published according to the established deadlines.
o Objective was met.
3. Gap Analysis/Needs Assessment (C2; SC1, EDP1-2)
This section is a description of how gaps and needs are handled through the year. It could include a discussion of your education committee’s work for the year as well as a discussion of ways that topics are prioritized and the evidence that is used to determine the gaps and needs. Example of items to include: The Education Council utilized the 2019 strategic plan to develop the new course additions to the education program with a focus on the development of topics in alignment with the accreditation and commendation standards. Requests for additional topics from the evaluation forms were compiled annually as one of the means of continual gap analysis and needs assessment. Additional ways to identify gaps and needs were through the review and analysis of reports on:
RM telephone call questions/issues
Top specialties participating in our online courses The compilation of all of this data was utilized during the strategic planning process and by the Education Council to develop education courses for 2019. The following table shows the top issues that were identified in 2018 and how they were addressed in 2019.
Top Ranking Requests/Issues in 2018
Planned and Developed in 2019
1. Education topics “in my specialty” (OB was top ranking)
o OB: updated and modified 2 hours of content on shoulder dystocia to create six 30-minute interactive courses. The library of perinatal topics has a total of 46 courses.
o ED: a new course on collaboration between physicians and APRNs was added to the library. Total library for ED is 36.
o Family Practice: A new course specifically for FP was developed, and 7 additional course titles apply to primary care.
o Ophthalmology: 1 new course was developed.
2. Communication and Documentation
Documentation and communication are addressed in most courses. A new course specifically on documentation was added to the library. The Lecture II webinar was created to address physician communication.
4. Maintenance of Certification (MOC) This section is for an overview of the MOC process and MOC activity for that year. If your organization does not participate in MOC, then this section would not be needed. Example of items to include: In 2019, 30 new courses were submitted for MOC approval through both the ABMS and the ACCME systems, and all were approved by at least one board. Our learning management system electronically
3
submits learner MOC credit to the ABIM, ABA, ABP and ABPath through the ACCME’s PARS database. In 2019, the ABO and ABOHNS were added to the electronic submission process. At the end of 2019, 180 of 183 of our eligible CME courses (98%) were MOC-approved by at least one board with a total of over 3,600 electronic submissions.
5. Accreditation Use this section to provide the status of your accreditation status with all of the accrediting bodies that you utilize. Consider both state and national entities. When in the re-accreditation cycle, this section can describe the efforts towards developing the reaccreditation documentation and all that your team did towards those efforts. Example of items to include: ACCME with commendation and ANCC accreditations with distinction were maintained in 2019. As a first step to work with the LSBN, we submitted an application to be an approved provider of continuing nursing credit with them. We were awarded approval in March and are listed on their website.
6. Staff Continuing Education/Professional Development (C34; SC2-3) Maintaining our own professional development is an important aspect of the annual program. Use this section to provide an overview of that process. Include not only a list of the conferences, seminars, and webinars attended, but also include some key things that were brought back to your organization. Example of items to include: To be able to plan and develop courses that identify practice gaps and meet the assessed needs of physicians and nurses, the Education Team must continuously review the literature and attend national and regional conferences for their own continuing professional development. In 2019, we created a professional development plan for the Education Team. As part of that plan, staff have attended the following external conferences in 2019:
Accreditation Council for Continuing Medical Education Annual Conference, April
Accreditation Council for Continuing Medical Education Workshop, August (Our plan will help us to meet ACCME Commendation Criteria C24.)
7. Activity Design (C3; EDP3, EDP5-6) In this section, discuss the different design methodologies that you use or have begun to use in that year. Example:
Films/Videos: In 2019, an activity on reducing healthcare acquired infections was created for video and a lecture on diagnostic error and the electronic medical record provided by Dr. Singh was added to the library.
Interactive HTML: all new courses were created in HTML, most with interactive questions. This design allows us to be able to more easily make changes to content, especially upon renewal or if changes to the evidence or guidelines occur. This design also creates a more interactive environment for the learner. A total of 35 new courses were created in HTML design.
4
8. Implementation (C5, C6, C7; SC3, EDP4) This section is for describing all of the processes that were implemented during the year and the data related to those processes. It is helpful to compare this year to the previous year. Graphically display as much data as possible to provide visual impact. It is also helpful to show the trends of the data over several years if that data is available. Example:
Process: In 2019, a change was made to the course creation process. The receipt and approval of an outline of the proposed content was added.
Timing: Despite the changes in the process, the time from first draft to go-live is an average of 53 working days or 2.4 months. Production time in 2018 was 3.3 months. Thus, in 2019 we achieved a decrease of 19.8 working days.
Volume of Courses: One of our 2019 goals was to increase the volume of the library by at least 30 courses. The library now has a total of 290 activities at the end of 2019 showing a net gain of 42; there are 182 CME, 91 CNE and 17 non-CE courses.
Volume of Participation: A total of 28,828 courses were taken online in 2019 (a decrease of 16,000 from 2018). The number of unique users decreased from 13,447 in 2018 to 9,901 in 2019. See Appendix B (Column heading “#”) for participation rates per course.
Examples of Graphs
6861
3341
3011
2210
2008
1290
1202
976
951
848
0 1000 2000 3000 4000 5000 6000 7000
OB/Gynecology
Nurse
Family Practice/GeneralPractice
Internal Medicine
Other Physician
Anesthesiology
APRN
Pediatrics
Cardiology
Other Non-PhysicianProvider
Courses Completed by Specialty 2019 (Top 10)
5
Examples of Graphs: Trending over Time
7,079 5,9293,049 3,694
947 815
13,392
10,56818,647
6,684
1,697
1,1380
5000
10000
15000
20000
25000
2018 2019 2018 2019 2018 2019
DOCTORS NURSES Others
Courses Completed by Designation 2018-2019
LAMMICO MI Clients
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Lecture Physician Participants 1,420 1,411 1,219 1,208 1,238 1,270 1,088 1,235 1,147 1427 1161 1432 775
Internet Physician Participants 3,841 3,913 3,969 4,357 4,202 4,495 4,812 5,427 5,623 8670 17441 13632 20574
0
5,000
10,000
15,000
20,000
25,000
Lecture Vs. Internet Participation 2006-2018 MDs + DOs
6
9. Course Evaluation Results (C11, C12; EDP7)
This section is a discussion of the course evaluation data. You could include the overall evaluation
score for the program that you get by aggregating the evaluation survey data for each course. In
addition, you could show the evaluation scores of new courses or courses using new strategies such
as case comparisons to see if the scores are different in any way.
Example:
The overall average of online evaluation scores (all evaluations use a scale of 1 to 5 with 5 being the highest score) was 4.42 for the 208 activities with evaluation scores. This is a slight decrease from 2018’s average score of 4.44. The aggregate averages of the individual activities were evaluated using the goal of an average of 4.3 and above:
85% of the online activities averaged evaluation scores of 4.3 and above; a decrease from 92% in 2018.
One of the evaluation questions that is scored by a Likert scale is “I will apply knowledge learned in this activity to my practice.” This question scored an average of 4.37 across all online activities.
Another one of the evaluation questions is open-ended, requesting learners to state what changes in practice they intend to make as a result of the activity. In 2019, 25% of the participants answered this open-ended question showing an increase of 19% from 2018 to 2019. The majority of the comments indicated some type of intent to make a change.
10. Pre/Post Test Comparison Results (C11, C12; EDP7) In the pre/post-test comparison section, further analysis using statistics can help to show impact of
the activities. Each activity is scored separately and then the data is aggregated to provide one
number for the entire program. Effect size quantifies the magnitude of the difference between two
groups and is a more scientific methodology to assess the overall success of the education.
Example:
Pre/Post Test Assessment Reports were reviewed for every CME and CNE course with data available. The aggregate averages were evaluated using statistical testing for effect size (Cohen’s d or “how big was the educational impact?”). Of the 195 courses with pre- and post-test data to allow statistical testing, 41% (79) activities had large effect sizes, 40% (78) activities had medium effect sizes, and 19% (38) had small effect sizes. Of the 38 small effect sizes, 17 had very low numbers of participants (<20) making the statistical testing less reliable for about 8.7%. The overall impact of the online library for 2019 was medium with a Cohen’s d of 0.35. This reflects a decrease from the large impact in 2017 and 2018 but is above the average of 0.28.
7
11. Follow-up Survey Results (C11-12; EDP7) This section is a discussion of the Follow-up Survey results. Key discussion points in this section
would be the rate of return and if practice changes were made as expected.
Example:
Post Activity Assessment (PAA) emails were auto-generated from the learning management system 30 days after the activity was completed. Of the 186 online courses with PAAs, responses were received on 165 of them. The response rate for 2019 was 1%. The results of all available post activity assessments were evaluated for stated practice changes (the selection of “no change” was counted as a negative result). Each course was evaluated for a score at or above 50%. Using this 50% criterion, 98.7% (163 of 165) of the online activities indicated a positive change in practice and the average score was 90%.
12. Lessons Learned The Lessons learned section is for analyzing the key areas that you can identify that may need improvement. What do you want to avoid for next year if possible? What opportunities for improvement have you identified?
Example:
We noticed several complaints about the matching, prioritization and fill in the blank questions due to the difficult user navigation based on their design. It was decided to no longer use those types in the pre- and post-tests. No new courses from mid-2019 utilized these question types in the tests, and we will continue this change as we go forward with new content.
13. Conclusions for 2019
This section is where you summarize the outcomes of the program and key accomplishments for the year to determine that year’s success.
Example:
All of the 2019 goals were met, and many were exceeded.
0.41
0.62
0.35
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2017 2018 2019
Impact of Library Using Cohen's d
Mansouri & Lockyear2
8
Participation rates continued to be high with 28,828 online activity completions. The number of unique users increased.
The CME and CNE programs were determined to be effective through analysis of the data of overall impact of the online library.
14. Recommendations for 20xx (C13; SC1)
In this section, discuss what improvements you would recommend to make in the next year. If you use a strategic planning process, all of the evaluation data and these recommendations would be utilized.
Example:
Based on the analysis of 2019 data, the following are recommendations for programmatic changes:
Increase the number of webinars.
Continue to explore new ways for interactivity of online content especially using Captivate. Staff will need additional training to produce this type of content.
RESOURCES FOR STATISTICAL TESTING OF CME OUTCOMES
The effect size of education measures the impact the education had on the participants. It is a quantitative measure of the magnitude or strength of a thing. It is interpreted as small, medium, or large impact. Depending on the type of data, the appropriate statistical tests will vary. A statistician is not needed due to the availability of online tools and resources. Although it seems complicated, the methodology can be learned. The following are several resources for learning what data to use along with what test is most appropriate.
1. assessCME is a blog posted by Jason Oliviera, MPH, on outcomes data and statistical testing for CME. /there is an email subscription that you can sign up for to get the latest blogs. He often teaches at large national CE conferences (ACEhp and SACME).https://assesscme.wordpress.com/2014/03/28/statistical-analysis-in-cme/
2. This article is the CME specific benchmarking for effect size (Cohen’s d as proposed by Cohen uses a different scale). It is getting dated but there hasn’t been a more recent study. Mansouri M, Lockyear J. A meta-analysis of continuing medical education effectiveness. JCEHP 2007;27:6-15.
3. This article is a bit complex but it has a good decision tree for thinking about what test to use for the data that you have. Maher JM, Markey JC, Ebert-May D. The other half of the story: effect size analysis in quantitative research. CBE Life Sci Educ. 2013;12(3):345-351. doi:10.1187/cbe.13-0082 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3763001/
4. Attached below is a PowerPoint presentation on calculating effect sizes in CME that was presented by Jason Oliviera, MPH. He has presented this material at several conferences and gave his permission to provide this resource.
5. VassarStats is a great FREE website to use for statistical computations. Once you know the tests that you need for your data, you can plug your numbers into the calculation worksheets and the site will calculate it for you. http://www.vassarstats.net/
6. The Meta Analysis Calculator is the second step for the process that I use for unpaired ordinal data http://www.lyonsmorris.com/ma1/index.cfm
Calculating and Interpreting Effect Size
in CME
Jason Olivieri, MPHDirector of Outcomes, Med-IQ
(312) [email protected]
Why Effect Size?
• Summarizes the educational effectiveness of a CME activity
• Permits aggregate analysis of CME effectiveness by format, topic, etc
• Provides benchmarks for CME effectiveness
What is effect size?
•Quantifies the magnitude of effect (maximum expected range: -3 to +3)
•Difference in means (e.g., pre-test and post-test) divided by the square root of the pooled-group variances (Cohen’s d)
•Enables the comparison of CME effects across activities on a common dimensionless scale
•Calculated from pre- vs. post-activity data linked directly to CME content
−Knowledge questions−Case vignettes−Self-reported frequency of use of key clinical practice strategies
Calculating effect size
• Can be done using only MS Excel® and free, online resources
• Approach dependent upon variable type:
– ordinal (e.g., clinical practice strategy)– categorical (e.g., case vignette)
Example paper survey question (clinical practice strategy)Using bevacizumab-based combo therapy for non-squamous NSCLC
32%
9%
41%
14%
5%
14%
18%
23%
32%
14%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 = Never Not very often Sometimes Often 5 = Always
Current vs. planned use of bevacizumab-based combo therapy in nonsquamous NSCLC (n = 142)
Current (mean = 2.5) Planned (mean = 3.1)
Calculating effect size for an ordinal variable
• Calculate average and standard deviation for each group (e.g., pre-and post-activity)
– Pre-activity: mean (SD) = 2.5 (1.3)– Post-activity: mean (SD) = 3.1 (1.2)
• Plug these values into online calculator (http://www.uccs.edu/~lbecker/)
Clinical practice strategy (CPS) use rating (1= never, 5 = always)
Pre-test Post-test Effect size
CPS #1 Standard deviation CPS #1 Standard
deviation Cohen’s d
2.5 ±1.3 3.1 ±1.2 .48
*d = .2 (small effect), d = .5 (medium effect), d = .8 (large effect)
How is effect size interpreted? Ordinal variable example
How big was the educational effect?
Expressed in standard deviation units: The average
score of a post-test respondent was .48 standard
deviations above the average score of a pre-test
respondent
Effect sizes are proportional (.48 is twice as much effect as .24)
How does the effect compare to other activities?
Cohen (1988): .2 = small, .5 = medium, .8 = largeWolf (1986): .25 = educationally significant, .50 = clinically significant
Example ARS question (case vignette)
Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)
51%
12%19%
12%6%
75%
6% 3%9% 6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Erlotinib alone(evidence-based,
preferred)
Erlotinib plusbevacizumab
Carboplatin-paclitaxel andbevacizumab
Carboplatin-pemetrexedand bevacizumab
Full house: erlotinib pluscarboplatin-paclitaxel and
bevacizumab
Pre (n = 65) Post (n = 65)
24% increase, P < .01
Calculating effect size for a categorical variable
• Determine % correct for each group (e.g., pre- and post-activity)
– Pre-activity: .51*65 = 33 correct / 32 incorrect– Post-activity: .75*65 = 49 correct / 26 incorrect
• Plug these values into online calculator (www.vassarstats.net)
– Click “frequency data”– Click “Chi-Square, Cramer’s V and Lambda”
Calculating effect size for a categorical variable,
continued
• Visit www.lyonsmorris.com/ma1/index.cfm
• Select Correlation coefficient (r) to Effect Size
• Enter Cramer’s V
Determining overall effect size for a CME activity
• Take average of all effect sizes (Cohen’s d) at each outcome level
– Report overall effect size if you have at least four questions– Only combine effect sizes for the same question type
• Keep a database of effect sizes to develop benchmarks
How effective was your live CME in 2016?
10 live CME activities
8 ARS questions per activity
80 slides 10 effect sizesOne summary effect size for
live CME
Overall EIS (by format): 2016
n=9
1. Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15.3. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 4. Casebeer et al. BMC Med Educ 2010;10: 42.
4Casebeer et al 2010. Knowledge effect size (eLearning) = .823Drexel et al 2011.
Competence effect size = .85
2Mansouri & Lockyer 2007. Knowledge effect size = .6
1Cohen J 1988. Small effect = .2, Medium effect = .5, Large effect = .8
Effect Size Summary
•Summarizes effectiveness of individual CME activities and allows for aggregate comparisons
•Easy to calculate
•Uses:
• Evaluate CME Mission• Make improvement plans and track progress• Establish effectiveness benchmarks• Communicate effectiveness to sponsors & ACCME
LEARNING ONLINE FOR EFFECTIVE CHANGE
(PEDAGOGY)
#ACCME2020online
Zoom Tools
PARTICIPANT ICON/LIST • Rename with more info and add two ** if you are a zoom expert
CHAT BOX • Introduce yourself• Engage by answering our prompts• ask technical questions about using Zoom.
Q&A BOX • Submit your questions for the faculty• Vote up common questions
Tag us in your social posts - #ACCME2020Online
facebook.com/AccreditedCME
@AccreditedCME
linkedin.com/company/AccreditedCME
https://www.instagram.com/accreditedcme/
#ACCME2020online
Session Goals
• Translate learning science methods to applications for online and distance learning
• Create an action plan to optimize planning and delivery of online learning
• Share with colleagues what works, what we’ve learned, and how we can stay connected as a learning community.
#ACCME2020online
Reminder: Zoom Tools
CHAT BOX
• What about today’s topic interests you?!
Q&A BOX
• Submit your questions for the faculty• Vote up common questions
Faculty
• Christina Lorenzo, MSAmerican College of Chest Physicians
• Lada Krilov, PhDAmerican Society of Clinical Oncology
• Sharon Cusanza, MSN, RN, NEA-BCLammico
Contrasting Cases Lada Krilov, Ph.D.
Senior Specialist, Clinical and Scientific Digital ContentAmerican Society of Clinical Oncology
What are contrasting cases?
Comparing similar examples helps one notice features they might miss otherwise
Comparing many examples builds expertise
Wine A Wine B
Contrasting patient cases help learners
Identify key differences that impact treatment decisions
Confront misconceptions and knowledge gaps
Apply new knowledge to real-world clinical practice
Why are contrasting cases important?
What are the techniques?
Highlight the differences that lead to different treatment choice
Resources
Keep B. The Science of Learning, Episode #7https://gohighbrow.com/contrasting-cases/
Alfieri L, et al. (2013) Learning Through Case Comparisons: A Meta-Analytic Review. http://www.lrdc.pitt.edu/schunn/papers/ContrastingCasesMeta-AlfieriEtAl2013.pdf
Schwartz, D. L., Tsang, J. M. & Blair, K. P. (2016). C is for Contrasting Cases: Discerning Critical Information. In The ABCsof How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them (pp. 26-38). New York, NY: W. W. Norton & Company. [Stanford Graduate School of Education]. (2015, Aug 30).
Daniel Schwartz on learning to perceive [Video File]. https://www.youtube.com/watch?v=UHeuV6FG8Q8
Contrasting Case ExamplesPodcast: https://elearning.asco.org/product-details/contrasting-cases-tailorx--chemotherapy-or-noPodcast: https://elearning.asco.org/product-details/contrasting-cases-systemic-therapy-for-metastatic-triple-negative-breast-cancerPodcast: https://elearning.asco.org/product-details/contrasting-cases-cancer-prevention
Spaced LearningChristina Lorenzo, MSSenior e-Learning Instructional DesignerAmerican College of Chest Physicians
Forgetting Curve
Cloke H. eLearning Industry.; 2018. https://elearningindustry.com/forgetting-curve-combat. Accessed May 2020.
Spaced Learning
Cloke H. eLearning Industry.; 2018. https://elearningindustry.com/forgetting-curve-combat. Accessed May 2020.
frequency
Plan Ahead
delivery
4. feedback
1. bite-size 2. real world
3. active thinking
Package Content
Application
frequency
delivery
feedback
bite-size
real world
active thinking
PROGRAM EVALUATION PRACTICES FOR ONLINE
EDUCATIONSharon A. Cusanza, MSN, RN, NEA-BC
Senior Risk Management Education SpecialistLAMMICO/Medical Interactive
Structured Evaluation Format
• Provides a structure that allows similarity from year to year• Systematic review• Prevents missing/skipping important aspects
• Connects with the ACCME and ANCC criteria• Allows the show of all the work of the year• Provides impact
Structured Format for Annual Evaluation - 1
• Executive Summary of Major Accomplishments• Evaluation of the Strategic Plan• Gap Analysis/Needs Assessment • MOC• Accreditation Status• Staff Continuing Education • Activity Design • Implementation
Implementation
Structured Format for Annual Evaluation - 2
• Course Evaluation Results
• Pre/Post Test Comparison
• Follow-up Survey Results
Pre/Post Test Comparison
• Compares pretest to post-test scores• Same questions on both • Need at least 4 questions• Multiple choice questions
• Various statistical methodologies based on the type of data
• Measure of Effect Size (Cohen’s d)
0.41
0.62
0.35
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2017 2018 2019
Mansouri & Lockyear2
Mansouri M, Lockyear J. A meta-analysis of continuing medical education effectiveness. JCEHP 2007;27:6-15.
Impact of Library Using Cohen’s d for Effect Size
Structured Format for Annual Evaluation - 3
• Lessons Learned
• Conclusions
• Recommendations for the next year
Conclusion session
Contrasting cases help learners • Notice features they may otherwise overlook• Distinguish differences that are important for clinical decisions
• Use them for situations where people are likely to make clinical mistakes
Spaced Learning - Call to Action
delivery frequency
bite-size real world active thinking feedback
Plan
Ah
ead
Pack
age
Con
tent
Call to Action
• Create a structured evaluation template
• Utilize statistical analysis of your data (Effect Size)
• Compare data year to year
• Graphically display your data
