Residency Competency Instrument - APTACompetencies versus DSP The purpose of the residency...
18
Residency Competency Instrument - APTACompetencies versus DSP The purpose of the residency competency instrument is to define a broad set of competencies that a resident training in
• February 2011: Generative discussion topic during CSM
• August 2013: Approval of work group creation
• October 2013: Call for work group applicants launched
• February 2014: 10 members appointed from an applicant pool of over 70
Presenter
Presentation Notes
During the Residency/Fellowship Program Director Breakfast at CSM 2011, APTA staff led a generative discussion regarding the need to establish broad competencies for residency training that would cross-specialties. Currently, there is duplicative work among residency programs. This duplicative work exists not only program to program, but specialty to specialty. The result of this duplicative work is a sharing of outcome measures and tools between programs. At present, most tools being utilized in residency education have not undergone psychometric testing and therefore are neither valid nor reliable. When programs utilize tools that are not valid and reliable, it places them at an increased risk of liability should a resident file a grievance against the program. In August 2013, APTA, through its American Board of Physical Therapy Residency & Fellowship Education, approved the establishment of a work group to develop the competencies for residency training, create an instrument to evaluate resident achievement of these competencies during training, and oversee psychometric testing of the instrument to determine validity. In October 2013, APTA launched a call for applicants to this workgroup. After receiving over 70 applications, a 10-member Residency Competency Instrument Work Group was established.
The following individuals have been appointed to the Residency Competency Instrument Work Group. The represent a variety of residency specialty areas as well as different levels of involvement in residency education from program directors, faculty, mentors, and residency graduates. �APTA has also secured the services of Statistician and Primary Investigator, Masaru Teramoto, with the University of Utah to lead the psychometric testing of the instrument.
Reason for Development
• Profession
• Regulatory stakeholders
• Program participants (residents)
Presenter
Presentation Notes
The reasons for the development of this instrument are threefold. The largest interest rests with the physical therapist profession as it seeks to determine what defines a specialist (master of clinical practice) irrespective of area of specialty. Competency assessments embrace the concept that capabilities of residents have to be measured, rather than assuming the attainment of competence is an intrinsic part of the training process. Second, as changes to regulations continue within health care, stakeholders are looking for a more defined set of competencies for postprofessional education and growth of providers. Following the move in public education to evaluate outcomes of graduates, educational processes for the profession of medicine and many other health professions came under scrutiny. In addition, as the cost of education has escalated, the public wants to know if the health care they are paying for equates to the service they are receiving. Competency statements can be used as milestones or benchmarks to assess the progress of residents through training, with the ultimate goal of assuring the public that graduates have the knowledge, skills, and abilities required to perform their roles in society. Last, program participants (residents) are looking for confidence as they apply to residency programs that regardless of what program they choose to enroll in, there is a consistent core set of competencies that each residency meets as they prepare for board certification. A well thought out approach to defining and measuring competency provides clear goals for residency curricula. It also defines for residents what they are expected to achieve during training. Period assessments of progress towards competence lets residents know what skills and behaviors they are mastering and where they need to focus their learning activities.
Benefits to Programs
• Economies of Work• Identify resident performance deficits• Justify remediation and termination, if
required.• Comparison of resident performance
Presenter
Presentation Notes
First and foremost, the competencies are designed to help all residency programs produce highly competent physical therapists to meet the health and health care needs of the public. For those programs that choose to utilize the Residency Competency Instrument, this this tool would create economies of work and effort among residency programs by supporting better assessment practices. Second, it would enhance opportunities for early identification of struggling residents. The performance expectations would be clear for residents upon entry into the program and throughout the residency. The tool would identify when remediation is needed. Third, this tool will undergo psychometric testing which will result in a sound tool allowing programs to identify early and support remediation where necessary. In addition, the tool will justify termination from the residency program, if required. Fourth, this tool would allow for program outcome assessment in that the program can make comparisons of individual residents performance within their program as well as comparison of residents of different programs in the same specialty area, assuming a similar curriculum.
Benefits to Residents
• Provide transparent expectations• Self-directed assessment and learning• Facilitate better feedback
Presenter
Presentation Notes
The development of a common set of resident competencies benefits the residents by providing more explicit and transparent expectations of performance, supports better self-directed assessment and learning, as well as facilitates better feedback for professional development.
Development Summary
• Benchmarking other health organizations
• Review of PT entry-level competencies
• Review of Description of Specialty Practice (DSP) documents
Presenter
Presentation Notes
The first meeting of the Residency Competency Instrument Work Group occurred at CSM 2015. Prior to this meeting, the group was provided numerous resources from other healthcare professional organizations including the Accreditation Council for Graduate Medical Education (ACGME). The competencies, behaviors, and benchmarks established by these organizations for residency training were reviewed and discussed. In addition, a review of all current published Description of Specialty Practice (DSP) documents from the American Board of Physical Therapy Specialties was conducted to determine commonality among defined specialty knowledge, skills, and attributes. �During their CSM 2015 meeting, the work group began generative discussions about physical therapist education, defined entry-level competencies, and expected outcomes in residency training. Following this brainstorming, the work group established an initial 10 resident competencies with associated critical behaviors. �Throughout 2015, the work group continued to discuss and refine the competencies and today, the list has been narrowed to 7 resident competencies. A full list of these competencies and critical behaviors will be released for public comment in the coming weeks.
Competencies versus DSP
Presenter
Presentation Notes
The purpose of the residency competency instrument is to define a broad set of competencies that a resident training in any area of specialty is expected to achieve by the end of the program. While many of the DSP documents do provide broad areas of knowledge and skills (eg, professionalism, clinical reasoning), these documents are published at different times and updated only once every 10 years. ��As education and practice expectations change dramatically within a 10 year period, some DSPs do not reflect current practice. �The residency competency instrument however is intended to be forward thinking as the physical therapist profession transforms.
Sample Competency
• Professionalism
Sample Competency
• Knowledge for Specialty Practice
Rating Scale
• Categorical versus Benchmarking
• Levels (0 to 5) versus Terms (Entry-level; Specialist)
• Milestones– Provides public accountability– Provides program better assessment practices and
early warning of struggling residents– Provides residents more explicit and transparent
expectations of performance
Presenter
Presentation Notes
The work group had to decide whether it wanted to utilize a categorical rating scale, like the one used in the entry-level CPI instrument, or a benchmarking rating scale. To help make this determination, the work group took each of the competencies and associated behaviors and drafted an instrument using both a categorical rating scale and a benchmarking rating scale. Upon review of the 2 instruments, the work group unanimously agreed that the use of a benchmarking scale was most appropriate for this residency competency. Following the decision to proceed with a benchmarking scale, the group needed to define the scale. A comparison was made between using common terminology seen in current residency programs’ evaluation tools (terms like Entry-level, Satisfactory, Specialist, Expert) versus broad categories. The group agreed that the use of terms like Entry-level, Satisfactory, Specialist, Expert can be controversial as individuals define these terms differently. Therefore, the use of a broad scale was determined. ACGME in recent years moved to the use of outcomes-based milestones as a framework for determining resident performance within their core competencies. The milestones are competency-based developmental outcomes (eg, knowledge, skills, attitudes, performance) that can be demonstrated progressively by residents from the beginning of their education through graduation. In summary, the use of milestones in the residency competency instrument benefit numerous stakeholders. First, the milestones provide public accountability by reporting at a national level on aggregate competency outcomes by all specialties. Second, the milestones provides residency programs better assessment practices and enhanced opportunities for early detection of struggling residents. Last, the milestones provide residents more explicit and transparent expectations of performance. �
Milestones
• Levels (0 to 5)
• Bloom’s Taxonomy
Presenter
Presentation Notes
This work group began creating the instrument using Levels 1 through 5 with Level 1 representing entry-level graduate competency. However, a discussion occurred regarding those residents that are noted to fall below entry-level and how these occurrences would be identified in this competency instrument. Therefore, the decision was made to create a Level 0 on the rating scale. While there are no benchmark descriptors provided for Level 0, its purpose is to represent residents that are not meeting the benchmarks outlined within Level 1 thereby highlighting to the program and resident that immediate remediation is required. Bloom’s taxonomy was utilize in the creation of the benchmark descriptors for each level of the competency measure.
Feedback – Administration
• Proposed administration schedule:1. Upon entry into the program (eg, first 90-days):
– Matches Evaluative Criterion 4.2.1 (“The program faculty determines that the participant is competent and safe to function upon entry into the program.”)
2. Midterm
3. Near end of residency (Final)
Presenter
Presentation Notes
The members of the Residency Competency Instrument Work Group are proposing the following administration of the Instrument. While the tool can be administered more frequently, the work group believe at a minimum, the resident should be evaluated 3 times over the course of the program. First, residents would be evaluated using the instrument upon entry into the program. This would allow programs to meet ABPTRFE Evaluative Criterion 4.2.1 which requires residency programs to have an evaluation procedure to determine resident competence and safety at the start of the residency program. During this initial assessment, the tool would identify those residents who are not at entry-level competence and facilitate immediate remediation efforts. The resident would then be evaluated mid-residency and the final time just before graduation.
Feedback – Rating Scale
• Level 0• Level 1 – upon entry into program (unless
prior work experience)• Level 2 – at mid-residency• Level 4 – upon graduation
Presenter
Presentation Notes
The benchmarks were written with the expectation that a resident is at a Level 1 upon starting the residency program if that resident is a new graduate from their professional degree. However, for those individuals that had prior working experience prior to starting the residency program, scores of greater than Level 1 may be noted. At the residency midterm, it is expected that the resident is at least at a Level 2 with Level 4 being defined as the residency graduation target. In reviewing other competency assessment instruments, it was determined that the instrument had to clearly indicate when a resident was not performing at Level 1 within a competency/behavior. After deliberation, the work group added in a Level 0 column. When scoring a resident, the resident must be meeting the benchmark descriptor in full for that Level. If not, the resident would automatically be scored at the Level below.
Feedback – Comments
• Competency versus Behavior• Required:
– When resident is not meeting targets (eg, Level 1 upon entry into program; Level 2 at midterm; Level 4 at graduation)
– For areas needing development– Positive feedback outlining areas of strength
Presenter
Presentation Notes
The work group is proposing that summative comments are provided for each competency rather than for each behavior under a specific criteria. In addition, comments would be required for competencies that the resident is not meeting the thresholds (eg, below a Level 1 upon starting the program; below a Level 2 at mid-term; below a Level 4 at the end of the program). Comments should be provided for areas needing development, while at the same time, positive feedback should be provided to the resident by outlining areas of strength within the comments box.
Next Steps• After CSM 2016:
– Tool refinement• March – May 2016:
– Online Public Comment Period• June 2016:
– Open Forum at NEXT• September – October 2016:
– Recruit program participation in pilot• January – December 2017:
– Pilot instrument• Spring 2018:
– Publish instrument and validity testing results
Presenter
Presentation Notes
We would like to thank you for your attendance and participation in today’s open forum. The Residency Competency Instrument Work Group is seeking to create a tool that works for all residency programs and residents. Therefore, feedback that has been received will be used to refine the instrument following today’s session. In mid-March, a formal electronic public comment session will begin. At that time, the full list of proposed competencies, critical behaviors, and associated benchmarks will be publicized. Information received during this time will be used to make additional revisions to the instrument. �Another feedback session will be conducted in June during APTA’s NEXT Conference and Exposition. Following this session, the Residency Competency Instrument will be finalized and preparation for pilot testing in 2017 will begin. We hope that you will continue to offer feedback and assist in the process as we continue our work. Thank you.
