SPECIAL ELEMENT: TASK FORCE REPORT

Assessing Resident Performance on the Psychiatry Milestones

Susan R. Swing & Deborah S. Cowley &

Adrienne Bentman

Received: 15 December 2013 /Accepted: 13 March 2014 /Published online: 18 April 2014# Academic Psychiatry 2014

The assessment of learners to determine their achievement ofeducational goals is an integral and time-honored feature ofeducation across the continuum. The expectation for evalua-tion of resident performance first appeared in the generalgraduate medical education accreditation requirements in1966 [1] and was added to and made more explicit in thepsychiatry requirements in 1975 [2]. Since then, accreditationrequirements for resident assessment have evolved, for exam-ple, to broaden the scope of what to assess from clinical skillsand knowledge to competencies in six domains and to requireuse of multiple evaluators (e.g., faculty, patients, peers, self,and other professional staff) [3].

Starting in July 2014, graduate medical education (GME)programs in all specialties will be required to assess and reporton residents’ achievement of milestones in each of the sixdomains of general competencies [4]. Milestones describe thesix domains more specifically in terms of knowledge, skills,attitudes, behaviors, and other attributes residents are expectedto learn and demonstrate. Furthermore, milestones are orga-nized into levels that form a developmental progression, fromless to more advanced [5, 6].

The ACGME’s new accreditation model calls for use oftwo separate but related processes to assess residents’ demon-stration of milestones: (1) the ongoing, formative assessmentof residents’ milestone-related learning and performance byfaculty and other appropriate assessors; (2) semi-annual

reporting of resident performance on milestones by the pro-gram’s Clinical Competency Committee (CCC) using thespecialty’s standard milestones report form [4].

Current ACGME requirements for formative assessmentstate that programs must provide objective assessments basedon specialty-specific milestones [4], but do not include otherdetailed specifications unique to milestone assessment. Howthen should programs perform formative assessments of mile-stones? Or, more generally, how do approaches to assess-ments of residents need to change (if at all) in the age ofmilestones?

In this article, we focus on development and implementa-tion of formative assessments helpful for addressing the newaccreditation requirements on milestone assessment. We iden-tify desirable features of milestones assessment and thendescribe the milestone assessment tool development workthe Psychiatry Milestone Working Group (PMWG) isperforming. We then discuss the advantages and disadvan-tages of this and other approaches.

Assessment of Residents in the Age of Milestones

Assessment of milestones is the next step in the competency-based education approach launched by the Outcome Project[4]. As such, many of the criteria for good assessment andrecommended practices remain unchanged. Among these areimplementation of the multiple components of an assessmentsystem deemed essential for good assessment, such as use ofblueprints to identify which competencies and milestones willbe assessed in which settings using what methods, participa-tion of multiple and qualified assessors, assessor training, anduse of multiple tools that are easy to use, reliable and valid,and have educational impact [7–10]. Although the importanceof assessment tools is diminished in this larger view, expertscontinue to discuss the potential utility of a variety of differentmethods, with special attention to workplace-based

S. R. Swing was at the ACGME when this work was performed.

S. R. SwingACGME, Chicago, IL, USA

D. S. Cowley (*)University of Washington, Seattle, WA, USAe-mail: dcowley@u.washington.edu

A. BentmanInstitute of Living/Hartford Hospital, University of Connecticut,Hartford, CT, USA

Acad Psychiatry (2014) 38:294–302DOI 10.1007/s40596-014-0114-y

assessments [11], those performed in the work (or education-al) setting and entailing direct observation of or interactionwith the individuals being assessed.

Features of milestone assessment will likely necessitatesome change to residency programs’ current assessment prac-tices. These include the following: (1) assessment of a broadscope of relatively specific knowledge, skills, attitudes(KSAs), and related attributes; (2) semi-annual milestonereporting derived from formative assessments; and (3) assess-ments that gauge progress on the developmental progression.The extent of change will depend on what programs arecurrently doing to assess residents.

Programs that extensively use global ratings of clinicalperformance, composed of general categories of competenciesor behaviors (e.g., end-of-rotation ratings with one to twobroad items for each of the six general competency domains),will need to supplement these tools with either assessor train-ing [12] or assessment tools designed to focus attention onmilestone-related behaviors in a given context. Such globalratings do not make explicit what has been observed and ratedand thus may compromise the validity of the CCC’s milestoneassessments. Also, the closer the content (or items) of theformative assessments are to the content of the milestones,the easier it should be for the CCC to map the assessmentresults to the semi-annual milestone report and complete thereport.

In the milestones progression, it is expected that KSAs andbehaviors at each level are demonstrated competently/proficiently, and not that performance is “very good” or “ex-cellent” for a resident at that level of training. An importantimplication is that both faculty and residents will need to thinkdifferently about assessment, accepting that performance at a“lower” level on the milestone reporting form is acceptableand expected for an early learner. Assessment tools, in partic-ular instructions, but also rating scales may need to change toemphasize criterion-based assessment and ratings that enableadvancement in performance to be shown [13–15].

The Psychiatry Milestone Working Group Approach

The PMWG developed a variety of tools for assessment ofPsychiatry Milestones. The 15 members of the PMWG wereselected to represent the ACGME’s Psychiatry Review Com-mittee, the American Psychiatric Association, the AmericanBoard of Psychiatry and Neurology, and the American Asso-ciation of Directors of Psychiatric Residency Training. Twelvemembers were current or former psychiatry or psychiatricsubspecialty residency directors. The remaining three mem-bers were the resident representative on the Psychiatry ReviewCommittee and two ACGME staff members, one with exper-tise in assessment.

The tool development process began with presentationsand discussion about milestone assessment that included ex-ample tools in current use. A large selection of additionalassessment tools was provided for reference in a shared onlineworkspace. The PMWG decided to focus primarily on toolsfor use following direct observation of resident performanceand then identified key learning settings for psychiatry resi-dents. Two to three members of the PMWG then did theprimary development work for tools customizing them to thesetting. During the development process, developers present-ed their tools to the entire PMWG for feedback. Additionalsteps in progress and planned are a field test of a subset of thetools and feedback from testers to revise the tools.

In developing assessment tools, the PMWG emphasizedworkplace-based assessment and direct observation of resi-dent performance on single occasions or throughout a clinicalrotation. The resulting tools can be completed by faculty andother staff observing and working with residents on clinicalrotations and in the context of educational activities, presen-tations, and tasks, such as case conference and journal clubpresentations, multidisciplinary team meetings, and handoffsof patient care. Many of the forms require direct observationof residents in person, on videotape or audiotape, or through aone-way mirror and allow the evaluator to assess particularmilestones while or immediately after observing the resident.Construction of all forms involved selecting a subset of allmilestones considered most relevant to the context orexperience.

The list of assessment tools developed by the PMWG isshown in Table 1. These tools are intended to provide exam-ples of assessment forms and methods for optional use. Thatis, their use is not required by the ACGME. Clinical rotationtools were developed to reflect the structure of clinical rota-tions required during psychiatry residency training. Thus, thegroup created assessment forms for inpatient, consultation-liaison, emergency, outpatient, and community psychiatry, aswell as for subspecialty clinical rotations [3]. These forms aresimilar to the current evaluation forms used in many psychi-atry residency programs, except that they specifically assessachievement of milestones, and thus could be tailored by eachprogram to replace existing forms. Some (e.g., consultation-liaison and emergency psychiatry evaluations) can be usedeither for end-of-rotation assessment or for direct observationof a limited number of clinical encounters. Within the broadareas of inpatient and outpatient psychiatry, the group devel-oped some examples of specific direct observation forms foruse by faculty members observing single patient assessmentsor medication management appointments. A separate psycho-therapy assessment form was created for psychotherapy su-pervisors, who are often not the resident’s general outpatientpsychiatry supervisor.

The group’s overall goal was to evaluate as manysubcompetencies and milestones as possible using

Acad Psychiatry (2014) 38:294–302 295

workplace-based assessments. Many of the milestones can beassessed during clinical rotations and some (e.g., interpersonalskills and communication and professionalism milestones)can be and were included in multiple forms, for assessmentby multiple raters in a variety of clinical settings. Manymedical knowledge milestones require assessment by testingand thus were less often included in the PMWG’s assessmentforms. Other subcompetencies (e.g., lifelong learning, teach-ing) are not systematically evaluated during clinical rotations,but can be assessed through other educational activities, suchas journal club, case conference, Grand Rounds, or otherpresentations. The group developed several direct observationforms for use in such specific settings.

For a given assessment tool, the two to three-memberdevelopment group identified relevant milestones to include

and constructed the first draft of the assessment form. In somecases (e.g., a multidisciplinary teammeeting evaluation form),the number of relevant milestones was small and could bearranged as a checklist on which an assessor could checkwhether a milestone had been achieved or not, based on directobservation by the faculty member of the resident’s perfor-mance during a single multidisciplinary team meeting (seeFig. 1). Of note, only a yes/no choice is necessary to indicatewhether or not the resident has achieved the milestone. How-ever, the group decided that including an intermediate rating(“partially met”) was helpful in letting the resident know thathe/she was making progress and in prompting the assessor toprovide specific feedback about aspects of performance need-ing improvement to achieve the milestone satisfactorily. In thecase of some mid- or end-of-rotation forms (e.g., the inpatientand outpatient psychiatry forms), there were hundreds ofindividual milestones at different levels that could be assessed.This presented challenges of whether to include all of themilestones that could potentially be rated during inpatient oroutpatient psychiatry, how to organize them to make theassessment tool as manageable as possible for the assessor,and whether to develop separate forms for residents at differ-ent levels of training and experience. In these cases, the groupchose to organize the forms using “threads” of milestonesrelated to each other and showing progression over time. Forexample, Fig. 2 shows one item on the outpatient psychiatryend-of-rotation assessment form. Here, the assessor is asked toselect where the resident is on a thread of related milestonesreflecting increasing abilities in history taking and examina-tion skills, based on observation of the resident over the courseof the clinical rotation.

Some assessment tools, designated in Table 1 as having a“hybrid” format, organized the milestones in threads toenhance the coherence of the form and allow the rater toevaluate related items close in content to each other. However,for each individual milestone, the assessor checks whether themilestone has been met, partially met, not met, or notobserved. An example of a hybrid rating form item is shownin Fig. 3. Such a form could be used as a mid- or end-of-rotation assessment by a faculty member supervising andobserving the resident’s performance over the course of acommunity psychiatry rotation, with ratings based on acombination of the resident’s clinical work and discussionsin supervision related to use of self-help groups and othercommunity resources in patient care.

Although in general a checklist format is more manageablewith a smaller number of items and a thread format may betterorganize a large number of items, there was not consensusamong the members of the PMWG regarding which of thesethree formats to use. Thus, the group decided not to adopt auniform layout, but rather to provide examples of alternativeformats for programs to consider. Overall, the assessment toolformats correspond to either developmental rubrics or

Table 1 Assessment tools developed by the ACGME Psychiatry Mile-stone Working Group

Assessment tool Assessment method Format

ROTATIONS

Inpatient psychiatry GAF Threads

Inpatient admission assessment DOBS Y/N/PM

Consultation-liaison psychiatry GAF, DOBS Y/N/PM

Emergency psychiatry GAF, DOBS Y/N/PM

Outpatient psychiatry GAF Threads

Outpatient intake assessment DOBS Y/N/PM

Outpatient medication management DOBS Y/N/PM

Community psychiatry GAF Hybrid

SUBSPECIALTY ROTATIONS

Addiction psychiatry GAF Hybrid

Child and adolescent psychiatry GAF Hybrid

Forensic psychiatry GAF Hybrid

Geriatric psychiatry GAF Hybrid

PSYCHOTHERAPY

Psychotherapy supervision GAF, DOBS Y/N/PM

EVENTS/PRESENTATIONS/PROJECTS

Case conference presentation DOBS Hybrid

Chief resident GAF Y/N/PM

Family meeting DOBS Y/N/PM

Handoff evaluation DOBS Y/N/PM

Journal club presentation DOBS Y/N/PM

Multidisciplinary team meeting DOBS Y/N/PM

Quality improvement project GAF, DOBS Y/N/PM

DOBS direct observation on a single or several occasions, GAF global(mid- or end-of-rotation) assessment form, Threads form asks assessor torate level of resident performance using milestone threads, Y/N/PM formasks assessor to rate whether milestones are met (Y), partially met (PM),or not met (N),Hybrid form groups related milestones in threads, but asksthe assessor to rate whether each individual milestone is met, partiallymet, or not met

296 Acad Psychiatry (2014) 38:294–302

checklists composed of milestones. Both formats can be usedeffectively for performance assessment [13, 14, 16, 17].

Once all of the assessment tools were developed, the groupconstructed a matrix to determine which milestones wereincluded in each evaluation form and to detect milestonesnot assessed in any of the forms. These latter non-assessedmilestones were then included in an evaluation form if possi-ble, or identified as milestones requiring another assessmentmethod. As a simple example of the use of a matrix, Table 2

shows a matrix where those Patient Care 1 (PC1) milestonesassessed in item 1 of the Outpatient Psychiatry end-of-rotationform (see Fig. 2) have been entered. It is immediately obviousthat PC1 milestones 2.3, 2.5, 3.3, 5.1, and 5.2 are not assessedin this item and require assessment elsewhere on this form oras part of another assessment tool.

Finally, a field test of a subset of the assessment tools is inprogress. With the test, the PMWG aims to determine whetherfaculty who assess resident performance in psychiatry training

Yes: Performs proficiently and reliably; if multiple observations conducted, the resident consistently demonstrates this attribute Partially Met : Performs some aspects proficiently, but other aspects may be unskillful, incomplete, or inaccurate; if multiple observations conducted, the resident demonstrates attributes proficiently on some occasionsNo: Performs, but significant improvement is needed or fails to perform even when called for Not Observed : The encounter did not require demonstration of this milestone

Milestone Behavior YES PARTIALLY MET

NO NOT OBSERVED

ICS1 1.3, 2.4 Actively participates during meeting and supports activities of other team members

PROF1 2.1 Seeks and is open to feedback and diversity of perspective from team members

ICS1 1.1 Demonstrates respect for other team members and interest in forming/maintaining collegial relationships

ICS2 2.1 Organizes both written and oral information to be shared with team

ICS2 4.1 Verbal communication with team is appropriate, efficient, concise and pertinent

SBP1 2.2 Structures communication with team to minimize errors and adverse events

PROF2 2.1, 3.3 Displays attitude of ownership and accountability to team for performance of assigned tasks/roles

Instructions: This form is designed as an “encounter form”, for use by an attending to assess a resident based on direct observation of a single team meeting or several team meetings within a short period of time (e.g. a day or week). This will most commonly occur on inpatient, but could also occur in certain community practice settings. It is anticipated that residents will require multiple rating cycles (with feedback and suggestions for improvement based on performance) to complete all milestones.

The attending should select one of the following response options by checking in the space for each milestone listed below

Fig. 1 Instructions and items from multidisciplinary team meeting direct observation evaluation form

1. Patient Care: History Taking and Examination Skills Not Observed

Level 0 Level 1 Level 2 Level 3 Level 4History and collateral information are inconsistently obtained or inaccurate; does not screen for patient safety; mental status examination incomplete, not well performed

Obtains history and relevant collateral information; screens for patient safety, including suicidal and homicidal ideation; performs mental status examination (PC1:1.1, 1.2, 1.3)

History taking is efficient, accurate, relevant and customized to patient complaint; shows sufficient knowledge of and assesses safety (suicide, homicide); shows sufficient knowledge of and can perform a mental status examination relevant to the patient’s complaints (PC1: 2.1, 2.2, 2.4; MK2: 2.2, 2.3)

History taking is completeaccurate, relevant and efficient, with flexibility appropriate to clinical setting and workload demands; uses-hypothesisdriven information gathering (PC1: 3.1, 3.2,3.4)

Routinely identifies subtle, unusual findings and follows clues to relevant information in complex clinical situations; uses own emotional responses as a diagnostic tool (PC1: 4.1, 4.2, 4.4)

Comments:

Instructions: The attending rater should select the level of knowledge, skills, and attitudes (KSAs) that best describe the resident’s performance. *

* The complete instructions and form are available on the ACGME’s Psychiatry webpage.

Fig. 2 Item from outpatient psychiatry global evaluation form

Acad Psychiatry (2014) 38:294–302 297

programs think that the items identify essential knowledge,skills, behaviors, and attitudes relevant to the clinical task,whether the items are specific enough to allow meaningfulassessment and feedback, whether the tool is easy to use, andwhether using the tool is helpful in completing the overallresident evaluation in the CCC. The plan is to revise the toolsusing tester feedback. Final versions of the assessment tools,together with the matrix, showing all milestones assessed oneach form, will be available on the ACGME website once theprocess of field testing and revision of tools is completed.

Practical Suggestions for Implementation

Psychiatry residencies vary in size, setting, and the timing andexact nature of clinical rotations and educational experiences.The flexibility afforded by the ACGME’s Psychiatry ReviewCommittee permits each program to implement milestoneassessment in a manner that best suits the program, faculty,and residents while meeting the fundamental expectation toevaluate all milestones. The program director, with faculty andresident representatives, may select the best rotations, settings,patient care experiences, and educational experiences (e.g.,conferences, journal clubs, presentations, didactics) forassessing residents in their own program and then select thecorresponding forms from the PMWG collection, or otherassessment methods. PMWG forms can be modified byadding or omitting milestones or by including other KSAs ofinterest to the program. Each residency program may alsowish to construct its own matrix to ensure that each milestoneis evaluated adequately across the four years of residency.Alternative approaches to milestone assessment, not using

PMWG tools, are discussed under “Alternative Approaches,”below.

Programs will need to decide which assessment tools willbe distributed automatically by their electronic evaluationsystem, which will be available electronically for assessorsto select and complete as needed, and which will be used aspaper forms. Updating forms within an electronic evaluationsystem will require time and may be costly.

Early involvement of faculty, and perhaps residents, inreviewing, selecting, and modifying forms serves as an initialstep in preparing them to implement milestones-based assess-ments. Faculty participation in selection of assessment toolsand settings also will enhance the likelihood that they will beaccepted, easy to use, and imbedded in daily patient care. Sucha collaborative process will lead the program director, faculty,and residents to more readily embrace this new system. Bothfaculty and residents will benefit from education about anddiscussion of milestones and their assessment, the CCC and itsfunction, any anticipated programmatic changes, modificationof the evaluation system, and expected changes in the experi-ence of residency evaluation and progression [13, 15, 18].

Program directors will also need to educate departmentleaders regarding this transformation in resident assessmentand its impact on the program director, residency staff, andfaculty in terms of additional time and cost for development,implementation, and system maintenance, faculty develop-ment, conducting milestones-based assessments and provid-ing feedback to residents, and faculty preparation for andparticipation in the CCC.

Assessment of residents using milestones represents a fun-damental shift in approach, away from rating how well aresident performs a particular task, with descriptors such as

Y= Yes, acceptable performance; milestone met PM= partially metNo = observed, but milestone not met NOB= not observed

Subcompetency: Thread Milestones

SBP3:Community-Based CareB. Self-help groups 1.2/B Gives examples of

self-help groups (e.g. AA, NA), other community resources (e.g. church, school), and social networks (e.g. family, friends, acquaintances)

2.2/B Recognizes role and explains importance of self-help groups and community resource groups(e.g. disorder-specific support and advocacy groups)

3.1/B Incorporates disorder-specific support and advocacy groups in clinical care

4.1/B Routinely uses self-help groups, community resources, and social networks in treatment

Comments:

Instructions: The attending should assess performance by marking each small box with a Y, PM, N, or NOB for each milestone listed below *:

* The complete instructions and form are available on the ACGME’s Psychiatry webpage.

Fig. 3 Item from community psychiatry evaluation form

298 Acad Psychiatry (2014) 38:294–302

“outstanding” or “above average,” to a system assessingwhether a resident has acquired a particular competency andassessing the resident’s performance along a developmentalprogression from novice to expert [5, 6]. Potential advantagesof this shift include clarity regarding expected benchmarks,more meaningful feedback on performance, and early identi-fication of residents in need of extra help [19–21]. However,this shift will require faculty development as well as educationof residents regarding the way in which they will be evaluated.Faculty development about assessment using milestones andabout specific tools might include group discussion of theassessment of videotaped resident–patient encounters (e.g.,having faculty all use the psychotherapy supervision

assessment tool to rate a segment of a psychotherapy session)or having faculty members familiar with a particular residentcompare their ratings on an end-of-rotation assessment form,to establish a common frame of reference for assessment [12,22]. In addition, faculty will need to focus on ways to deter-mine the resident’s independent knowledge and skills, ratherthan their performance in carrying out instructions and advicefrom their faculty supervisor. Residents will need to be ori-ented to a novel evaluation system focusing on achievingspecific milestones rather than being rated as “excellent” andwill need to accustom themselves to the idea that the facultyon the CCC discusses their progress in order to identify areasof accelerated progress and areas in need of attention.

Benefits, Limitations, and Challenges

The Psychiatry Milestone Group developed context-basedassessment tools along with an assessment matrix. The assess-ment tools have milestones as items. The matrix shows themilestones assessed by each tool and the contexts in which themilestones will be assessed. This approach facilitates mappingof each resident’s formative assessments to the milestonereport and supports selection of a set of tools for assessingmilestones in different settings and at-a-glimpse determinationof which milestones have been assessed.

There are several limitations of the PMWG’s approach toassessment tool development, however. First, the assessmenttools were developed based on the clinical rotations andeducational experiences involved in current psychiatric resi-dency training [3]. The utility of the tools would decrease ifthe current structure of rotations and experiences were tochange. They were not developed by focusing on individualmilestones and how to assess each one, which means that thetools are not comprehensive in assessing all milestones. Theuse of a single format across evaluation tools may have madethem easier to use and aided faculty development. The threadformat is limited by the fact that the Psychiatry milestones arenot uniformly organized in threads and the threads that existwithin the milestones are often discontinuous, with no item ata particular level.

Since the PMWG tools have not yet been field tested andused, it is unclear how many observations using each form, orof each milestone, are required to achieve adequate validityand reliability. Past studies of required numbers of observa-tions have yielded numbers ranging from 7 or 8 for directobservation of student or resident skills by faculty to 25 forpatient ratings [23, 24]. However, workplace-based assess-ment tools designed using constructs of increasing clinicalsophistication and independence, similar to constructs in-volved in milestones and entrustable professional activities(see below) may yield greater reliability and validity [16].

Table 2 Matrix showing PC1 milestones included in Outpatient Psychi-atry Form Item 1. Patient care: history taking and examination skills

PC1. Psychiatric evaluation

Milestone Assessment

IP OP CL ER

1.1. Obtains general medical and psychiatric historyand completes a mental status examination

x

1.2. Obtains relevant collateral information fromsecondary sources

x

1.3. Screens for patient safety, including suicidal andhomicidal ideation

x

2.1. Acquires efficient, accurate, and relevant historycustomized to the patient’s complaints

x

2.2. Performs a targeted examination, includingneurological examination, relevant to the patient’scomplaints

x

2.3. Obtains information that is sensitive and notreadily offered by the patient

2.4. Assesses patient safety, including suicidal andhomicidal ideation

x

2.5. Recognizes that the clinician’s emotionalresponses have diagnostic value

3.1. Consistently obtains complete, accurate, andrelevant history

x

3.2. Performs efficient interview and examinationwith flexibility appropriate to the clinical settingand workload demands

x

3.3. Selects laboratory and diagnostic tests appropriateto the clinical presentation

3.4. Uses hypothesis-driven information gatheringtechniques

x

4.1. Routinely identifies subtle and unusual findings x

4.2. Follows clues to identify relevant historicalfindings in complex clinical situations andunfamiliar circumstances

x

4.3. Begins to use the clinician’s emotional responsesto the patient as a diagnostic tool

x

5.1. Serves as a role model for gathering subtle andreliable information from the patient

5.2. Teaches and supervises other learners in clinicalevaluation

IP inpatient psychiatry,OP outpatient psychiatry, CL consultation-liaisonpsychiatry, ER emergency psychiatry

Acad Psychiatry (2014) 38:294–302 299

Finally, the incorporation of multiple new assessment toolsis challenging, and programs may not wish to adopt specifictools for all of the individual rotations and educational expe-riences identified by the PMWG.

Alternative Approaches

There are other approaches to assessing milestones. Each hasits own advantages and disadvantages, but few or no studiesare available yet to support or discourage their use. TheACGME’s Advisory Group for Educational Outcome Assess-ment recommended adoption of a small core set of specialtyappropriate tools [7]. In addition to direct observation ofpatient encounters by faculty, the ACGME Advisory Groupconsiders medical record audit and feedback an acceptableassessment method [7]. Use of a subset of the PMWG toolswould be consistent with this recommendation. Alternatively,a small set of tools could be constructed with milestones orrelated core KSAs for use in broader contexts such as patient-based clinical activities, team-based clinical activities, didacticactivities, etc. The National Health Service (NHS) of GreatBritain offers a small standard set of tools for use to evaluatemedical residents in their foundation program [25]. The NHSincludes, in addition to direct observation methods, multi-

source feedback and case-based discussion in its small set ofmethods. Use of all these tools has been met with mixedreviews [7, 26].

Assessments can be organized by entrustable professionalactivities (EPAs). EPAs are defined as the essential workactivities that make up a profession, or in the case of GME,a specialty. An EPA identifies the context and implies thatassessment will be of the specific KSAs related to that context(e.g., the specific knowledge and skills needed to care forpatients with psychosis). Programs in internal medicine andpediatrics are testing approaches in which assessment toolsformed using milestones or behavioral descriptors of mile-stones are used for direct observation and feedback for an EPA[19, 27] and some have recommended the use of EPAs inpsychiatry [28]. There is support from faculty and residentevaluations that this approach, when accompanied by specificfeedback to learners, is effective for promoting learning anddetermining when a resident is competent (i.e., in this case,ready for indirect supervision for the EPA) [19]. The PMWGapproach functionally overlaps with an EPA approach. Forexample, assessment and care of patients in the inpatient andoutpatient settings, care of geriatric patients, and working in amulti-disciplinary team all could be considered EPAs.

The milestone report forms themselves can be used as end-of-rotation assessment forms. This use has all the caveats

Table 3 Approaches to assessment of milestones

Selected characteristics

Assessment approach Advantages Disadvantages

PMWG approach • Comprehensive coverage of milestones• Organized to reflect curriculum organization• Provide for assessment of same milestones inmultiple context

• Aligned with semi-annual milestone report

• Numerous tools challenging to manage• Milestone-based tools may not support/ promptobservation of important “granular” behaviors

Small, milestone-based specialtyspecific set of assessmentmethods

• Multiple tools, but manageable number • May not provide comprehensive coverage of milestones• To address multiple settings, may be overly inclusive anddilute focused attention to behaviors essential to that context

Milestone report forms as end-of-rotation evaluation

• No new development needed• Commercial vendors are building these into their

computer-based residency evaluation systems• Easy compilation and direct alignment withsemi-annual report

• Unlikely that assessors can/will remember performancerelated to the large number of milestones on the forms

Current assessment tools • No or limited new development• Familiar to faculty and residents

• Global ratings likely lack content validity for milestones andrequire extensive assessor training to correct this problem

• Unlikely to provide comprehensive coverage• Likely more difficult to translate and map results for semi-annual reporting

EPA organized assessments • Could provide comprehensive coverage ofmilestones

• Intuitively meaningful to faculty•Could provide for assessment of samemilestones

in multiple contexts• Could be aligned with semi-annual milestonereport

• Extensive development required• Multiple assessment tools challenging to manage

300 Acad Psychiatry (2014) 38:294–302

associated with global ratings: leniency effect, validity issues,and less helpful for feedback [23, 29, 30]. Programs can alsouse their current assessment tools. If this approach is taken,special attention should be paid to the extent to which KSAsassessed using these tools reflect and cover milestones on thesemi-annual form. See Table 3 for additional advantages anddisadvantages of each approach to milestone assessment.

It is important to note that the PMWG tools are not pro-posed as the sole assessment method for milestones. Othermethods, such as standardized patients, simulations, OSCEs,in-training examinations, multisource feedback, chart review/audit, and portfolios are all useful options to supplementPMWG forms, or any of the alternative general approachestomilestones assessment discussed above, and can both assessmilestones not routinely observed in the workplace as well asproviding additional information to strengthen the reliabilityand validity of CCC milestones evaluation [7, 24, 26].

As with all assessment, high quality results do not auto-matically follow implementation. Much attention is needed toapply best practices [7, 12, 24, 26, 31, 32].

Conclusions

Assessing resident achievement of the Psychiatry Milestonesrequires a fundamental change in approach to assessment,from a focus on degrees of excellence to one of ascertainingsuccess in reaching specified benchmarks. For most psychia-try residency programs, this will require changes in assess-ment tools and processes.

Putting into place a new assessment system is a challengeunder any circumstances. Assessment processes that are easyto implement, require limited resources, and produce reliableand valid results are elusive and will continue to be so.Creating milestone assessments with these characteristics,likewise, is a challenge. A particular challenge pertains tothe feasibility of developing and managing a multi-assessment tool system. Equally challenging will be ensuringthat faculty have sufficient time to observe residents anddiscuss patient cases so as to obtain evidence-based assess-ment information and then to participate on the CCC. Ascer-taining that residents have competently or proficiently dem-onstrated a large number of milestones will require multipleobservations or interactions. Thus, implementing milestones-based assessment will require faculty development, orienta-tion of residents to a new form of evaluation, and systemsresources.

The assessment tools developed by the PMWG emphasizeintegration of formative assessment into routine clinical work,direct observation of resident performance, and providing anarray of tools that can be flexibly used and adapted to fit theneeds of individual residency programs. These tools are op-tional, are not mandated by the ACGME, and form only part

of a comprehensive evaluation system. It is to be hoped,though, that they can be of practical utility to programs aspossible examples of format and content for their own assess-ment forms. The PMWG also hopes that competency-basedassessment using milestones will bring with it long-term ad-vantages in clarifying expectations for trainees and supervi-sors and allowing constructive feedback and improvementbased on a developmental model of skill and knowledgeacquisition.

Disclosures On behalf of all authors, the corresponding author statesthat there is no conflict of interest.

References

1. American Medical Association. Directory of approved internshipsand residencies 1966. Chicago: American Medical Association;1966.

2. Liaison Committee on Graduate Medical Education 1975–76.Directory of accredited residencies. Chicago: American MedicalAssociation; 1975.

3. Accreditation Council for Graduate Medical Education. ACGMEprogram requirements for graduate medical education in psychiatry,2014. Available at: http://www.acgme.org/acgmeweb/tabid/147/ProgramandInstitutionalAccreditation/ MedicalSpecialties/Psychiatry.aspx. Accessed 9 March 2014.

4. Accreditation Council for Graduate Medical Education. ACGMEcommon program requirements, 2013. Available at: http://www.acgme.org/acgmeweb/ tabid/429/ProgramandInstitutionalAccreditation/CommonProgramRequirements.aspx2013. Accessed 8 Dec 2013.

5. Dreyfus H, Dreyfus S. Mind over machine: the power of humanintuitive expertise in the era of the computer. New York: Free Press;1986.

6. Carraccio CL, Benson BJ, Nixon J, Derstine PL. From the educa-tional bench to the clinical bedside: translating the Dreyfus develop-mental model to the learning of clinical skills. Acad Med. 2008;83:761–7.

7. Swing S, Clyman S, Holmboe E, Williams R. Advancing assessmentin graduate medical education. J Grad Med Educ. 2009;1(2):279–86.

8. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model forprogrammatic assessment fit for purpose. Med Teach. 2012;34:205–14.

9. Norcini JJ, Anderson B, Bollela V, et al. Criteria for good assessment:consensus statement and recommendations from the Ottawa 2010Conference. Med Teach. 2011;33:206–14.

10. Ling L, Derstine P, Cohen N. Implementing milestones and clinicalcompetency committees, 2013. Available at: http://www.acgme.org/acgmeweb/Por ta ls /0 / PDFs/ACGMEMiles tones-CCC-AssesmentWebinar.pdf. Accessed 8 Dec 2013.

11. Norcini J, Burch V. Workplace-based assessment as an educationaltool: AMEE Guide No. 31. Med Teach. 2007;29(9):855–71.

12. Holmboe ES, Ward DS, Reznick RK, et al. Faculty development inassessment: the missing link in competency-basedmedical education.Acad Med. 2011;86:460–7.

13. Varney A, Todd C, Hingle S, Clark M. Description of a developmen-tal criterion referenced assessment for promoting competence ininternal medicine residents. J Grad Med Educ. 2009;1(1):73–81.

14. Meade LB, Borden SH, McArdle P, et al. From theory to actualpractice: creation and application of milestones in an internal medi-cine program, 2004–2010. Med Teach. 2012;34:717–23.

Acad Psychiatry (2014) 38:294–302 301

15. Baglia J, Foster E, Dostal J, et al. Generating developmentallyappropriate competency assessment at a family medicine residency.Fam Med. 2011;43:90–6.

16. Crossley J, Johnson G, Booth J, Wade W. Good questions, goodanswers: construct alignment improves the performance ofworkplace-based assessment scales. Med Educ. 2011;45:560–9.

17. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation andassessment of clinical skills of medical trainees: a systematic review.JAMA. 2009;302:1316–26.

18. Meade LB, Caverzagie KJ, Swing SR, et al. Playing with curricularmilestones in the educational sandbox: Q-sort results from an internalmedicine educational collaborative. Acad Med. 2013;88:1142–8.

19. Meade LB, O’Malley CW, Swing SR et al. Applying curricularmilestones to an early training entrustable professional activity: acollaborative study among fourteen internal medicine programs.Under review.

20. Swing SR, Beeson MS, Carraccio C, et al. Educational milestonedevelopment in the first 7 specialties to enter the next accreditationsystem. J Grad Med Educ. 2013;5:98–106.

21. Psychiatry Milestone Group. Psychiatry milestone alpha test phase 1and 2 questionnaires. 2013. Unpublished data.

22. Kogan JR, Conforti L, Bernabeo E, et al. Opening the black box ofclinical skills assessment via observation: a conceptual model. MedEduc. 2011;45:1048–60.

23. Gray JD. Global rating scales in residency education. Acad Med.1996;71(1):S55–63.

24. Donnon T, Al Ansari A, Al Alawi S, Violato C. The reliability,validity, and feasibility of multisource feedback physician assess-ment: a systematic review. Acad Med. 2014;89:511–6.

25. Carr S. The Foundation Programme assessment tools: an opportunityto enhance feedback to trainees? Postgrad Med J. 2006;82:576–9.

26. Miller A, Archer J. Impact of workplace based assessment on doc-tors’ education and performance: a systematic review. BMJ.2010;341:c5064.

27. Schwartz A. Assessment of pediatrics milestones and development ofa collaborative research network for GME. Chicago: Invited presen-tation, DeWitt Baldwin Seminar Series, ACGME; 2012.

28. Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable profes-sional activities to guide curriculum development in psychiatry train-ing. BMC Med Educ. 2011;11:96.

29. Murphy KR, Cleveland JN. Understanding performance appraisal:social, organizational, and goal-based perspectives. Thousand Oaks:Sage; 1995.

30. Williams RG, Klamen DA, McGaghie WC. Cognitive, social, andenvironmental sources of bias in clinical performance ratings. TeachLearn Med. 2003;15:270–92.

31. Tochel C, Haig A, Hesketh A, et al. A systematic review of theefficacy of portfolios for post-graduate assessment and education:BEME Guide 12. Med Teach. 2009;31:299–318.

32. Brannick MT, Erol-Korkmaz HT, Prewett M. A systematic review ofthe reliability of objective structured clinical examination scores.Med Educ. 2011;45:1181–9.

302 Acad Psychiatry (2014) 38:294–302