Learning and Applying Concepts while Having Fun: A Perfect Training EnvironmentLearning and Applying Concepts while Having Fun: A Perfect Training Environment Karen Szauter, MD and Ann Frye, PhD

University of Texas Medical Branch, Galveston TX

Introduction

The principles of educational measurement are neither easy to understand or simple to teach

A foundational understanding of validity, reliability and measurement error is essential for all medical educators

To master the basics, both explanation and application of these principles is needed.

Over the past several years, we have worked with a MedEdPORTAL resource to guide the delivery of workshops for faculty and professional colleagues addressing these challenging concepts.

Workshop Sequence

Brief didactic addressing the concepts of validity, reliability and measurement error

Interactive focus: Identification of the “best” chocolate

Attendees brainstorm on characteristics of chocolate which can be used in grading

Group consensus: 5 characteristics are chosen

Small groups each take one characteristic and develop anchors for a seven point rating scale

Faculty Development: a spokesperson from each group describes the intended use of the scale; all five scales are presented and combined into a single rating form

Individuals apply the newly developed rating scale to multiple samples of chocolate. Categorical ratings and an “overall” score are applied

Scores are collected and the score distributions are shared with the participants

Applications of the concepts of validity, reliability and measurement error are addressed in a debriefing.

Publication

Educational Measurement Workshop: A Sweet Approach to Understanding the Basic Principles of Educational Measurement Simpson D, Meurer L. MedEdPORTAL; 2007

Application

We have adapted this workshop for our Scholars in Education Program (institution-wide 20 month faculty development program; interdisciplinary)

We have used it for our Clerkship Directors conference

The workshop was also presented at the Association of Standardized Patient Educators annual meeting (2010)

Outcomes

The debriefing sessions have resulted in rich and insightful discussions. Post session written comments have included:

“Had not thought of many of these aspects of rating”

“…inherent bias in scales…”

“…need to provide greater focus in my grading criteria…”

Participants have successfully integrated the foundational concepts and have recognized the importance of application in their day to day work. Immediate feedback has included recognizing:

• the importance of piloting rating scales

• a need for attention to language used in scale anchors

• the critical need for faculty development prior to using a rating scale

• the potential for environmental influences on ratings

Evaluations for this session have been uniformly positive.

Summary

This MedEdPORTAL resource is exceptionally well developed and easy to adapt in a variety of settings.

We are indebted to Deb Simpson and colleagues for this creative and sweet approach to introducing important measurement concepts to our colleagues

Eating chocolate as part of work is a plus.