Embed Size (px)
Citation preview
Wendy Turell, DrPH, CHCP
Senior Director, Outcomes and Analytics
PlatformQ Health Education
• Measurement about how people think or act gathered via Q&A process
• Can be used in experimental, quasi-experimental, and non experimental research
• Initially developed in the 1920s, now have theories and principles that guide survey
science
• Survey Methods = commonly use in CEHP
• Surveys:
• Can be administered via self administered questionnaires or interviews
• Paper/pencil, verbally, online
• Are most often quantitative (but can be qualitative)
• Are a standardized measurement instrument
• Replicable
• Can measure:
• Attitudes and opinions
• Behaviors
• Motivations
• Emotions
• Satisfaction
Measurement Validity
How accurately the instrument measures the construct you are trying to measure
Examples:
A Ruler = High measurement validity
Measurement reliability
How consistently your instrument measures what you are trying to measure.
Does your testing instrument generate the same result in similar circumstances?
Many errors “bias” research results
Examples: sampling error, nonresponse error, self-presentation/social desirability bias, acquiescence bias, etc.
Today: Focus on measurement error
The difference between the estimate produced and the true value because respondents gave inaccurate answers to survey questions.
Occurs when respondents are unable / unwilling to provide accurate answers
May be due several factors……..including poor question design
Does the question adequately measure the idea or concept of interest?
If not = specification error (low construct validity)
Pilot your questions
Psychometric review
validate your questions against an external construct (e.g., correlate with a measure that is known to measure the construct of interest very well)
examine internal consistency of constructs
To begin with………..write solid questions by using question structures that have been shown to elicit more valid & reliable responses!
TIP 1:
Question:
Measurement studies have shown that 5 option MCQ formats are more reliable and valid than dichotomous answer “true/false” style questions:
a. True
b. False
True/false; Yes/no; Agree/disagree
Probability of guessing correctly = .5 (dichotomous) versus .2 (5 choice MCQ)
“Acquiescence” = tendency to agree as a response style
When guessing, examinees tend to lean towards “true” > “false”
Grosse, Martin E.; Wright, Benjamin D. (1985) Validity and Reliability of True-False Tests. Educational and Psychological
Measurement, 45(1): 1-13
Norcini JJ, Swanson DB, Grosso LJ, Webster GD. (1985). Reliability, validity and efficiency of multiple choice question and patient
management problem item formats in assessment of clinical competence. Med Educ. 19(3):238-47.
TIP 2:
Examples:
All of the Above, None of the Above, A and E, All except D
More likely to lead to measurement error
More challenging to think of good distractor answers
All of the Above: When studied, responses skew more towards “all of the above” response (typically artificially
elevating scores)
Loss of discriminatory power in reviewing other response options
Cuing Effect of "All of the above" on the Reliability and Validity of Multiple-Choice Test Items P. H. Harasym,E. J. Leong, C. Violato, R. Brant,and F. L. Lorscheider. Evaluation & the Health Professions, March 1998; vol. 21, 1: pp. 120-133.
TIP 3:
Novel targets in clear cell RCC include all of the following EXCEPT:
a. PD-1
b. PD-L1
c. MET
d. WNT pathway
Which of the following prognostic features is associated with WORST overall survival?
a. Low Ki-67 proliferative index
b. Morphology of small cells resembling follicular leukemia
c. Presence of 3 or more complex karyotype abnormalities
d. Translocation involving chromosomal 11
“not”, “except”, “least likely” etc.
Flip cognitive reasoning
> Taxing = > measurement error
TIP 4:
Questions that contain multiple concepts
Do you recommend changes in diet and exercise for all of your overweight patients?
In the past month, how frequently have you self-monitored your blood sugar and blood pressure?
How satisfied are you with the accuracy and delivery of the content?
Do you desire more live and online CME activities in oncology?
Rewrite:
Do you recommend changes in diet for all of your overweight patients?
Do you recommend changes in exercise for all of your overweight patients?
TIP 5:
I can rarely differentiate a diagnosis of patients with schizophrenia from those with schizoaffective disorder.
how to define “rarely”
My health has improved since participating in the educational session.
how to define “health”
I have visited my doctor in the past 6 months.
how to define “doctor” – do other healthcare providers count?
how to define “visit” – do phone/email consults count? What if you just saw the nurse?
Which question format do you prefer?
Question 1: Basal insulin should provide about ______ of daily insulin requirement.
a. 10%
b. 25%
c. 50%
d. 75%
Question 2: In the evaluation and diagnosis of a person with mild cognitive impairment, which of the following biomarkers are considered most likely to indicate that MCI is due to AD:
a. A positive CSF Aβ biomarker only
b. A positive Aβ biomarker and a positive tau, FDG, or sMRI biomarker
c. Positive tau and FDG biomarkers
d. A positive FDG biomarker only
Which question format do you prefer?
Question 1: Some ER/LA opiates have eliminated risk of abuse.
a. True
b. False
Question 2: Patients with an unprovoked VTE are at risk for a recurrence for what amount of time?
a. 3 months
b. 6 months
c. 12 months
d. Indefinitely
Based on what you learned in this session, rewrite this question:
Which is NOT your favorite color crayon?
Yellow
Blue
Green
Brick Red
Based on what you learned in this session, rewrite this question:
The following outcomes level is the most difficult for me to evaluate and report:
Level 3: Knowledge
Level 4: Competence
Level 5: Performance
Level 6: Patient Health
Level 7: Community Health
Wednesday April 8th, 2015
1:30 PM ET
5 Tips to Make You a Survey Measurement Rock Star
Did you know that there are measurement experts who spend their lives researching the validity of different question styles and formats? How ’bout we spend 30 minutes reviewing some of the most relevant findings in survey methods research!? During this presentation, you will learn about fascinating topics such as the validity of true/false questions, when to use “none of the above” style questions, and which question styles will help you assess the information you are actually trying to asses (read= minimize bias and error)! Participants will be asked to vote on sample question strength, and recraft “before’s” into highly valid “after’s.”
