View
36
Download
0
Category
Tags:
Preview:
DESCRIPTION
You’ve Been Shown: Now It’s Your Turn to Ask!. Responding to Your Questions About Testing. CLEAR 2004. Kansas City, Missouri. Credentialing and Exam Issues: Sessions at CLEAR 2004. Ten Year Review of CBT When Certification Replaces Licensure: Show me Accreditation - PowerPoint PPT Presentation
Citation preview
You’ve Been Shown: Now It’s Your Turn to Ask!
CLEAR 2004 Kansas City, Missouri
Responding to Your Questions About Testing
Credentialing and Exam Issues: Sessions at CLEAR 2004
• Ten Year Review of CBT
• When Certification Replaces Licensure: Show me Accreditation
• ADA: Let Us Show You What Works
• Show Me What Globalization Means for Credentialing Programs
• Multiple Tracks to Licensure: Show me the Way
• Test Piracy and Security in a Wired World
CLEAR 2004 Kansas City, Missouri
• Show Me the Validity
• Show Me How to Get Past MCQs: New Opportunities in Measurement
• Show me an Evidential Approach to Assessment Design
• The Case of the Repeating Licensure Examination Candidate: Psychometric, Legal, and Policy Issues
CLEAR 2004 Kansas City, Missouri
Credentialing and Exam Issues: Sessions at CLEAR 2004
Panelists:Carol Morrison Assistant Vice President National Board of Medical Examiners
Elizabeth Witt Senior Statistician American Board of Emergency Medicine
Rose McCallin Director of Office of Examination Services
State of Colorado
CLEAR 2004 Kansas City, Missouri
Moderator: Chuck Friedman Assistant Vice President, ACT
Speakers
You . . . the audience!!!
CLEAR 2004 Kansas City, Missouri
ValidityReliability
Item Analysis
Item Review
Item Writing
Passing ScoreScoring
Job Analysis
Test Blueprint
Standard Administration
Valid
ity
Reliability
Item
Anal
ysis
Item
Writin
g
Pass
ing
Scor
e
Scorin
g
Test Blueprint
Standard
Administ
ratio
n
Job Analysis Item
Rev
iew
Should state licensing boards require continuing education?
CLEAR 2004 Kansas City, Missouri
How do I find out if my test has been compromised? If so, what should I do?
CLEAR 2004 Kansas City, Missouri
CLEAR 2004 Kansas City, Missouri
What should I do if I do or not do if I think a candidate or candidates have cheated?
Test Compromise?
• Missing test book– When, who?– Statistical analyses
• Memorization of items– Memorize and reconstruct items– Study guides– Internet postings– Statistical analyses
Is a self assessment exam good enough for continuing competency?
CLEAR 2004 Kansas City, Missouri
Four Components ofContinuous Certification
• Professional StandingProfessional Standing
• Lifelong Learning and Self AssessmentLifelong Learning and Self Assessment
• Assessment of Cognitive ExpertiseAssessment of Cognitive Expertise
• Assessment of Practice PerformanceAssessment of Practice Performance
Considerations in Determining the Value of a Self-Assessment
• How is the exam developed and scored?
• Can you be certain who took the exam?
• How high are the stakes for maintaining the credential?
• Does the self-assessment process foster learning?
• Is it easy to participate as part of one’s normal professional activities?
Is performance testing useful in state licensure?
CLEAR 2004 Kansas City, Missouri
Performance Assessment
• Does performance assessment measure something different?
• Reliability, validity• How do you score it?• Cost, complexity
• Answer: It depends!
CLEAR 2004 Kansas City, Missouri
What is your opinion about open book exams?
What should I do if a candidate receives an incorrect score report?
CLEAR 2004 Kansas City, Missouri
How do you determine if a request for accommodations is appropriate? If a test is administered under special conditions, should the score report be “flagged”?
CLEAR 2004 Kansas City, Missouri
Is Request Appropriate?
• Use expert consultant• Level the playing field, don’t tilt it too far the
other way
To Flag or Not to Flag?
• Many testing companies have stopped flagging• Some claim that flagging may lead to unintended
consequences• Standards for Educational and Psychological
Testing (APA/AERA/NCME)– Flag if do not have evidence of score
comparability
Do we have to equate our tests?
Is there one best way?
ONE BEST WAY?
CLEAR 2004 Kansas City, Missouri
Reasons to Consider Reasons to Consider
NOT EquatingNOT Equating• Small sample size
• Multidimensional exam
• Changing content outline
What If We Can’t Equate?
• Create a detailed blueprint
• Write the best possible items
• Build forms that are closely parallel in both content and statistics
• Conduct frequent standard setting studies
• Include common items across forms; study differences qualitatively
CLEAR 2004 Kansas City, Missouri
An examinee complains that he failed the test by one point/question. What do you tell him?
Other Sources of Information• CLEAR Exam Review• Other CLEAR Publications
– ADA Handbook– Development, Administration, Scoring and
Reporting of Credentialing Exams• CLEAR web site: Frequently Asked
Questions• Joint Standards: AERA, NCME, APA• Professional psychometricians
CLEAR 2004 Kansas City, Missouri
Recommended