Guidelines for cognitive and pilot testing of questions ...· Guidelines for cognitive and pilot testing

  • View
    212

  • Download
    0

Embed Size (px)

Text of Guidelines for cognitive and pilot testing of questions ...· Guidelines for cognitive and pilot...

  • Guidelinesforcognitiveandpilottestingofquestionsforuseinsurveys

    ESCAP PROJECT ON IMPROVING DISABILITY MEASUREMENT AND STATISTICS IN THE ASIA PACIFIC REGION

    December2010

    StatisticsDivision EconomicandSocialCommissionforAsiaPacific Region

  • 2

    Tableofcontents

    1. Backgroundtotheguidelines ....................................................................... 4

    1.1 TheESCAPProjectonimprovingdisabilitymeasuresandstatisticsinAsiaPacific ..............4

    1.2 Aimsoftheguidelines ...........................................................................................................5

    2 Currenttrendsindisabilitymeasurementandstatistics .............................. 5

    2.1 Shiftofthinkingondisabilityanditsinfluenceinmeasuringit ............................................5

    2.2 WashingtonGrouponDisabilityStatistics(WG) ...................................................................7

    3 Reasonsfor,andoverviewofcognitiveandpilottesting............................. 9

    3.1 Questionevaluation ..............................................................................................................9

    3.2 Selfreportinginsurveysandcensuses..................................................................................9

    3.3 Cognitivetesting..................................................................................................................10

    3.4 Pilottesting..........................................................................................................................12

    4 Thecognitivetesting................................................................................... 13

    4.1 Aimandobjectives ..............................................................................................................13

    4.2 Developingthequestionnaire .............................................................................................14

    4.3 Trainingofinterviewers.......................................................................................................15

    4.4 Numberandselectionofrespondents................................................................................16

    4.5 Practiceinterviews ..............................................................................................................16

    4.6 Summarisingnotes ..............................................................................................................17

    4.7 Analysis ................................................................................................................................18

    4.8 Revisionofquestionsforpilottesting.................................................................................22

    4.9 Analysisexamples................................................................................................................22

    4.9.1 Adomainwherequestionsperformasintended:Hearing.............................................23

    4.9.2 Adomainwherequestionsrequirefurthertesting:Pain ...............................................26

    4.9.3 Adomainwherequestionsarerejected:Learning .........................................................29

    5 Thepilottesting.......................................................................................... 32

    5.1 Aimsandobjectivesofthepilottest ...................................................................................32

    5.2 Developingthequestionnaire .............................................................................................33

    5.3 Samplesizeandspread .......................................................................................................33

    5.4 Selectionandtrainingofinterviewers ................................................................................36

    5.5 Datacapturing .....................................................................................................................38

    5.6 Analysis ................................................................................................................................39

  • 3

    6 Informationdissemination ......................................................................... 39

    6.1 Developingthequestionsfortesting ..................................................................................39

    6.2 Reportingoncognitivetestresultsandpreparingforpilottesting ....................................40

    6.3 Analysisandreportingofpilottestresults..........................................................................40

    6.4 ESCAPprojectexperiences ..................................................................................................40

    7 Translationofcognitiveandpilotquestionnaires ...................................... 40

    7.1 Forwardandbackwardtranslations....................................................................................41

    7.2 Translationbycommittee ...................................................................................................41

    7.3 ThetranslationprocessintheESCAPProject .....................................................................41

    8 Concludingremarksandlessonslearnt ...................................................... 43

    8.1 Lessonslearnt ......................................................................................................................43

    8.2 Beneficialoutcomesofcombiningcognitiveandpilottesting ...........................................43

    8.3 Usingthisapproachinareasotherthandisability ..............................................................44

    8.4 Conclusion ...........................................................................................................................44

    9 Resources.................................................................................................... 46

    9.1 Websitesandprogrammes .................................................................................................46

    9.2 Resourcepersons ................................................................................................................46

    9.3 Articles .................................................................................................................................47

    10 Appendices ................................................................................................. 49

    10.1 AppendixA:ESCAPprojectcognitiveandpilottestquestionnaires ...................................49

    10.2 AppendixB:Examplesofnarrativesfromthreedomainsandcommentsontheinterpretations ..................................................................................................................................49

    10.2.1 Pointsaboutthenarratives ........................................................................................49

    10.2.2 HearingDoyouhavedifficultyhearing,evenifwearingahearingaid?(nodifficulty,somedifficulty,alotofdifficulty,cannotdoatall) ......................................................49

    10.2.3 LearningDoyouhavedifficultyunderstandingandusinginformationlikefollowingdirectionstogettoanewplace?(no,somedifficulty,alotofdifficulty,cannotdoatall) .........52

    10.2.4 PainDoyouhavefrequentpain(yes/no) ................................................................53

  • 4

    1. BackgroundtotheguidelinesThe growing interest in disability statistics globally has led to improved conceptualizations ofdisabilityand relatedmeasures toprovide statistics.These improvedmeasureswere required fortwomajorreasons.Thefirstrelatestothelackofcomparabilityacrosscountries.Thewiderangeofvariationinreportednationalprevalenceestimatesfordisability1suggestedthatthemeasuresuseddidnotnecessarilyreflectthesamephenomenonacrossdifferentcountriesandthatthemeasuresthemselveswerenotthesame.Asecondreasonwasthegrowingrecognitionofdisabilityasbeingabout thewholepopulation andnot only aminority groupof peoplewith disabilities commonlyreferred toas thedeaf,blind,crippledormentally retarded.This recognition isembodied in theWorldHealthOrganizations InternationalClassificationof Functioning,Disability andHealth (ICF)(WHO,2001).Given these reasons theWashingtonGrouponDisability Statisticswas formed as a resultof theUnitedNations InternationalSeminaronMeasurementofDisabilitythattookplace inNewYork inJune2001,todevelopmeasuresofdisabilityforuseincensusesandsurveys.ThiswasinparallelandwithadifferentpurposetodevelopmentsontheuseoftheICFforclinicalpurposes.2Thedevelopmentofquestions foruse in censuses and surveys requires aprocessof testing andrevisiontoensurethatthequestionsareinfactmeasuringtheintentofthequestion.Theoutcomeoftestingprovidesevidenceonwhetherrespondentsunderstandthequestionsandareabletousetheresponseoptionseffectively.Theconclusionsfromthetestingprocessallowthequestionstobeaccepted as is (because theyworkwell), revised (if there are somemisinterpretations or otherproblemsthatcanbefixed)orrejected(ifthequestionsdonotperformwellatall).Disabilitystatistics(oranyotherstatisticsbasedonselfreportmeasures)arereallyonlyusefuliftheperformance of the questions is understood. This understanding provides a transparency in thestatisticsthatexplainwhattheyreflectandmeasure.

    1.1 TheESCAPProjectonimprovingdisabilitymeasuresandstatisticsinAsiaPacific

    TheUNDevelopment Account project Improvement ofDisabilityMeasurement and Statistics inSupportofBiwakoMillenniumFrameworkandRegionalCensusProgramme (hereafter theESCAPproject) aimed to train people from statistical offices in the region on question performanceevaluationanduseofdisabilitymeasuresandstatistics.Thisprojectwassuccessfulinprovidingstafffrom the six partici