Surveys: One More Outcomes Measure

Jay Shapiro, MD

Program Director

Anesthesiology

Program Evaluation and Improvement 1. The program must document formal, systematic evaluation

of the curriculum at least annually. The program must monitor and track each of the following areas:

a) resident performance; b) faculty development;c) graduate performance, including performance of program

graduates on the certification examination; and, d) program quality. Specifically:

(1) Residents and faculty must have the opportunity to evaluate the program confidentially and in writing at least annually, and

(2) The program must use the results of residents’ assessments of the program together with other program evaluation results to improve the program.

2. If deficiencies are found, the program should prepare a written plan of action to document initiatives to improve performance in the areas listed in section V.C.1. The action plan should be reviewed and approved by the teaching faculty and documented in meeting minutes.

Graduate performance: Results of performance on board certification examinations is one measure of graduate performance. Data can also be collected by annual surveys of graduates. Typically, such surveys target physicians one year and five years after graduation. Forms used may be provided by the institution, developed locally or adapted from the published literature (or unpublished but available online). Survey questions may inquire about such items as current professional activities of graduates and perceptions on how well prepared they are as a result of the program.

REMEMBER:

Can, Should, Must…For ACGME all these MEAN:

Thou Shalt Do These Things or Be Cited

Surveys in Common Use

• ACGME Annual Resident Surveys– Common Survey; Specialty Specific Survey

• Exit Surveys-currently done by GME at Graduation

• Post-NRMP applicant surveys

• Alumni Surveys

Alumni Surveys

• When– 1 year, 5 year, after fellowships (1-3 yrs)

• Who– Graduate or Employer

• How– Traditional Mail– Email– Web-based Survey tools (eg. Survey Monkey)

Other uses of Alumni Surveys

• Assess CME Needs

• Tie Alumni closer to program– Maintain accurate contact info– Provide database to solicit funds for education– Provide job contact network for current

Residents

Structure of a Survey

• General Demographics– Name, Office/Home addresses, email

• Type of Practice• General Questions

– Open ended-comments– Likert Scale (strongly agree/disagree)– True/False (probably least information)

• Competency Based Questions?

Competency Based Survey

Open Ended Survey

Objective Specific SurveyPathology Trainee Post-Graduation Survey

Anatomic and Clinical Pathology Resident Training Program - VCU Department of Pathology

As part of our effort to continuously improve the quality of training we provide, we ask you to please respond to the followingquestions about your residency experience.1. How do you rate the training you received as a resident/fellow in each of the following areas?

1=poor 2=fair 3=good 4=excellentAutopsy 1 2 3 4 N/A Hematopathology 1 2 3 4 N/AImmunohistochemistry 1 2 3 4 N/A Transfusion Medicine 1 2 3 4 N/ASurgical Pathology 1 2 3 4 N/A Microbiology 1 2 3 4 N/ANeuropathology 1 2 3 4 N/A Molecular Pathology 1 2 3 4 N/ACytopathology 1 2 3 4 N/A Cytogenetics 1 2 3 4 N/AClinical Chemistry 1 2 3 4 N/A Forensic Pathology 1 2 3 4 N/AToxicology 1 2 3 4 N/A Informatics 1 2 3 4 N/ALaboratory Management 1 2 3 4 N/A

2. How well prepared were you for the practice of pathology upon completion of your residency?Not well-prepared_____ Somewhat prepared_____ Ver y well-prepared_____

3. In what area(s) did you feel your preparation was weakest?

4. In what area(s) did you feel your preparation was strongest?

5. What specific recommendations do you have to improve the quality of training in this program?

6. Year completed residency at VCU_____ 7. Year completed fellowship at VCU_____

Send Me Money

Survey Points

• Ask open ended questions or incorporate comment boxes

• Don’t make your survey so long that responders give up

• Consider a short “preamble” to explain the importance of a response

• Thank the responder for taking the time to complete the survey

• Follow up, when appropriate with direct communications

Input from the Coordinators

Name_____________________ Grad Yr. ______Address_________________________ Home___Office__City/State/Zip____________________Email __________________________

Current Practice (Type, Location, Yrs in practice)Business Contact Info“Did your career path follow your original plan orwhat you had discussed with your mentor?”

Do you work full time or part time (FTE)

Are you in an academic practice?What are your primary responsibilities?W/In your institution to you teach: med students,residents, post-grads

Board Certification: did you take the certifying exam y/nDid you pass on 1st attempt ___ 2nd ___Date of initial Certification ________

Did the Program adequately prepare you for your exams?

Did the Program adequately prepare you for yourpractice?

How useful were these activities to your success inthe program:

Journal ClubsLecturesGrand RoundsM & M conferencesWorkshops

If you work in the region, are you willing to be apreceptor/mentor to future Residents?

Is your group looking to hire? Would you consider hiringcurrent or future residents from this Program?

WHAT ELSE???

What to do with data?• Obtain and Assess Data• Formulate an action plan• Compare any new data against previous action

plans• Develop new programs to address deficiencies

delineated in the data…• And DOCUMENT that you did this

– Who was present in meeting– When did you meet– Outcome products of the meeting

• Minutes, Report to Faculty, New Action Plans