Upload
dasia-whitcraft
View
215
Download
0
Embed Size (px)
Citation preview
Component 2The Certification of Graduates Using
a National Competency-based Examination
Tri Hanggono Achmad 2nd HPEQ Conference
Denpasar, 4 December 2011
Outline of presentation • Overview of Component 2 • Report and key performance indicator • Lesson learned • Challenges and Future plan
OVERVIEW OF COMPONENT 2
Health Professions Education Quality (HPEQ) Project
• Rationale: Indonesia’s current health indicators, the current status of Indonesia’s health professional education programs, and on the increasing demand for larger numbers of and more highly qualified health professionals.
• Goals: strengthen quality assurance policies governing the education of health professionals in Indonesia through 1. rationalizing and assuring competency focused accreditation
of public and private health professional training institutions,2. developing national competency standards and testing
procedures for certification and licensing of health professionals, and
3. building institutional capacity to employ results based grants for encouraging use of accreditation and certification standards in the development of medical school quality.
Objectives
Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals
Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination
Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination
Milestones for Component 2Main Program 2010 2011 2012 2013 2014
2.1 National Agency kick offmid-term
evaluationfinal
evaluationtake off sustainable
2.2.a Computer-based Testing (CBT)
Paper-based, Preparation and
Try out
Implementation
Implementation
Implementation
Implementation
Paper-based, CBT Preparation and
Try Out
Paper-based, CBT
Preparation and Try Out
Implementation
Implementation
Implementation
PreparationBenchmarking
TestPreparation and Try Out
Implementation
Implementation
2.2.b Objective Structured Clinical
Examination (OSCE)
Preparation Try-out Implementati
onImplementati
onImplementati
on
PreparationTry out
Implementation
Implementation
Implementation
Preparation Preparation Preparation Try outImplementati
on
2.3 National Item Bank Networking for
Assessment (NIBNA)
Preparation Establishment
Enhancement for Pre-clinical
Formative Test
Enhancement for Clinical Formative
Test
Enhancement for Progress
Test
Preparation Establishment
Enhancement for Pre-clinical
Formative Test
Enhancement for Pre-clinical
Formative Test
Enhancement for Clinical Formative
Test
Preparation Preparation Preparation PreparationEstablishmen
t
Medicine Dentistry Nursing/Midwifery
HPEQ +
• Pharmacy • Nutrition • Public Health • Other health professionals related to MTKI
KPI PROJECT per YEAR
Key Performance Indicator Successfull rates on the first take ofNational Competence Examination
Health Professional
Baseline (2009)
Target Value
2010 2011 2012 2013 2014
Medicine 71,67 % 71,67 % 73,25 % 75,50 % 79,60 % 84 %Dental 79 % 80 % 80 % 81 % 82 % 83 %Nursing 60 % 65 %Midwifery 60 % 65 %
ACHIEVEMENT OF COMPONENT 2
Objectives
Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals
Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination
Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination
Subcomponent 2.1
Lembaga Pengembangan Uji Kompetensi (LPUK)• An independent national agency that responsible in
developing assessment strategy, methodology, and tools to evaluate competency of graduation from health professional education institution (medicine, dentist, nurse, midwife)
Function: 1. Assessment service: as resource person and
facilitator to help the management of examination for organization in each health professions.
2. Academic: to improve assessment system in health professional education with evidence based practice
Organization
• Tim Task Force 1300a/E3-HPEQ/SK/08.11• Director: Iwan Dwi Prahasto• Secretary: YulherinaMember: • Moh. Ghozali (Medicine) Gandes Rahayu (Medicine)• Tatong Haryanto (Medicine) Mei Syafriadi (Dentist)• Iwan Dewanto (Dentist) Mia Damayanti (Dentist)• I Made Kariasa (Nurse) Pramita Iriana (Nurse)• Yetty L Irawan (Midwife) Ani Kusumastuti (Midwife)• Rahayu Endah Astuti (HPEQ)
Advisory board
National
Regional
Examination Center
Chariman
Examination Division
Assess Management
Division ICT Division Item Bank
Division R & D Division
Executive secretary
The Milestone LPUK
Before 2010: First phase of development
2010 – 2014: Program Acceleration and Expantion
After 2014: Sustainability Phase
Establishment of KBUKDI & KDGI
Development of KNUKP
Various workshop in item development & review, standard setting
Implementation of UKDI & UKDGI, and competence examination in some MTKPs
Academic paper & various guidelines for workshop & implementation of examination
Resource Sharing
International Benchmarking
Human resource
development
Regional & International collaboratio
n
Recertification & assessment of professional behaviourResearch
grant & scientific publicati
onResource & ICT support for assessment Establishme
nt of LPUK & quality standard
Collaboration w/ other HP
Subcomponent 2.1
• Benchmarking to National Board Examination for medicine, nurse & dentist
• Technical assistance from Prof. Gordon Page from Univ. British Columbia – Canada
• Agreement with Ministry of Health MoH Regulation Number 1796/Menkes/Per/VIII/2011 about Registration of health professional
• Guidelines to conduct various workshop on assessment• Item development for CBT • IT and CBT management • Item development for OSCE• OSCE Examiner • Trainer for Standardized Patient• Item bank administration• OSCE Coordinator
• Research grant on assessment
Objectives
Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals
Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination
Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination
Subcomponent 2.2
• Medicine & Dentist: • Standard setting with Modified Angoff • Try out & CBT Implementation• Try out OSCE
• Nurse & Midwife: • Try out CBT in November 2011
Try out & CBT Implementation
Activity Number of CBT Center Medicine Dentist Nurse Midwife
Try Out 1 6 1 12 (809) 13 (610)Try Out 2 10 6Try Out 3 12 11Try Out 4 14 13Try Out 21CBT 1 10CBT 2 14CBT 3 27
Try Out OSCE
Medicine
OSCE Center
Candidates
Try Out 1 1 14Try Out 2 12 160
Dentist OSCE Center
Candidates
Try Out 1 1 9Try Out 2 13 274
Standard setting
Standard Setting
Medicine Dentist
Number of judges Cut Score Number of judges Cut Score
I 52 53 21 49
II 52 58 21 49
III 46 58
IV 48 58
V 50 58
VI 66 61
• Judges: from health professions institutions • Implementation of standard setting in some institutions
Total= 32 CBT CenterTotal workstations= 2892 PC/laptop(842 : 2140)
Number of CBT Center & Workstations
Average result of UKDI & UKDGI in 2010 – 2011
2010 201170%
72%
74%
76%
KPI HPEQ UKDI
2010 201169%71%73%75%77%79%81%83%85%
UKDGI KPI
Comparison of UKDI result based on Accreditation
ANOVA p=0,000
Objectives
Sub-component 2.1: Establishing an Independent National Agency for Competence Examination of Health Professionals
Sub-component 2.2: Improving the Methodology and Management of the National Competency-based Examination
Sub-component 2.3: Developing an Item Bank Networking System to Support the National Competence Examination
Subcomponent 2.3 Establishment of NIBNA • Item development for knowledge based assessment –
CBT bank masing2 profesi• Item development for skill based assessment - OSCE• Development of IT support system for student assessment • Technical assistance for Item Bank Administration: Prof. Clarke Hazlet from Univ. Alberta – Canada
Number of workshops & participants in 2011
No. Workshop Frequency Participant Institution
1 Item Review and Development (CBT)
6 (4 M, 2 D)8 (4 N, 4 Mid)2 (other HP)
389 (M)45 (D)168 (N)132 (Mid)
65 (M)26 (D)168 (N)129(Mid)
2 CBT Coordinator 2 26 (M)8 (D)
26 (M)8 (D)
3 Standard Setting 7 65 (M)26 (D)
65 (M)26 (D)
4 OSCE Examiner 3 (2 M, 1 D) 131 (M)28 (D)
65 (M)26 (D)
5 Item Review and Development (OSCE)
3 (2 M, 1 D) 130 (M)53 (D)
65 (M)26 (D)
6 OSCE Coordinator 2 12 (M) 12 (M)
7 Standardized Patient 2 65 (M)26 (D)
65 (M)26 (D)
M: Medicine; D: Dentist; N: Nurse; Mid: Midwife; HP: Health Professions
Item Bank
Medicine Dentist Nurse Midwife 0
500
1000
1500
2000
2500
3000
3500
4000
Number of MCQ Item in Banks
OSCE: Medicine: 12 stations Dentist: 9 stations
Jumping Hurdles SuccessfullyDeveloping adequate quantity of high quality items: achievable target
If development is collaboratively produced & organized, realistic timelines for achieving a “large-enough” bank are
- 5 yrs for Medicine (given 72 medical schools & 40,000 items req’d)
- 10 yrs in Dentistry (if 26 dental schools & 30,000 (?) items req’d)
- 4 yrs in Nursing (if 57 nursing schools & 25,000 (?) items req’d)
- 3 yrs in Midwifery (if 60 midwifery schools & 20,000 (?) items req’d)
SIPENA Development Road Map
© PadmaNet 2010 – ss100722
IT Modules - interconnectivity
Examinee OL-Reg
Examinee On Line Registration
Information Web
Exam Location Adm
Local Exam Supervisor
Registration
National Exam Supervisor
Registration
Exam Process
PBT
CBT
OSCE
BAU
Summative - Scoring
Absolut
Normatif
Formative - Institution
Selected Item Bank
Tested Item Bank
New Item Bank
Summative - Assesment
Bank 1 : New Item
Bank 2 : Reviewed Item
Bank 3 : Tested Item
Bank 4 : Selected Item
Call for Item
Item Writer / Reviewer Registration
Peer Review
Panel Expert
Blinding
Review
Blinding – Review
Lesson learned • Building awareness of the role of assessment in improving the
quality of life (health) and driving in prioritizing quality• Support and Commitment from, and collaboration among various
stakeholders• More potencies being identified• Resources sharing (human, infrastructure, items, system)
efficiency & close the gap• NCE support the improvement of assessment method & learning
process in institutions assessment drives learning (standard of competence)
• Assessment as a learning / education materials and management process
• Progress in adopting evidence based protocol security means strong committment
Challenges & Future plan
• The dynamic of the project continuity of the program based on the road map with possible modification
• Maintaing quality in bigger scope & quantity of project prioritizing & synchronize & coordination among Project Components & stakeholdersresources sharing among health professionals
• Closing the gap of development & implementation paralel implementation of assessment development & learning response (case : OSCE & Clinical teaching)
• Customers friendly item banking & evidence based security system• Credit for item citation index academic merit system• Future research in professional assessment (incl. Publications)• Collaboration with other international examination agency
This Project Can be & Will be
Collaboration is Project’s Mode of Operation
Since collaboration among health care educators also involves endorsement & participation of government & professional bodies in health care
Successful !