Disability questions in censuses and surveys Methodological issues: proxy response, non-response and mode of administration 21-23 September 2005 Montserrat

Disability questions in censuses and surveys

Methodological issues: proxy response, non-response and

mode of administration

21-23 September 2005

Montserrat López-Cobo

Fifth Meeting of the Washington Group

Fifth Meeting of the WASHINGTON GROUP 2005

Are proxy-responses different from self-responses? Why?

Do all proxies provide the same quality responses? Who is the best proxy?

Is there an effect on the quality of data? Can we measure it? Can we correct it?

Proxy response


It is generally accepted that proxy- and self- responses differ

Many studies found that proxy-respondents underreport disability-related information: Disability Impairments of emotion and Pain Limitation of activities, long-term

disability, need for assistance with ADL Chronic condition, days of activity

restriction, bed disability days General health events and conditions

Proxy responseLevel differences 1


... While some other studies found that proxies overreport: Cognitive status, Mild Cognitive

Impairment ADL disability Functional health, when the self-

respondent is demented

Proxy responseLevel differences 2


Differences in cognitive procedures (1/2) Type and level of available information

o Proxies report impairments or conditions that are more observable or mentioned to them

o Proxy reports are more accurate for conditions that are serious, painful, persistent or potentially life threatening

If person under 65: proxies underreportIf person 65 and over: proxies overreport

Proxy information is more stable. Self information is more dynamic Self-responses are less consistent across time

Proxy responseUnderstanding self-proxy differences 1


Differences in cognitive procedures (2/2) Proxies engage in less extensive cognitive

process. Provide heuristic-based responses Proxies rely more on inferences and estimations.

Proxies overreport disabilities aparently related to a previously reported disability and underreport disabilities seemingly unrelated to the previously reported.

Proxies are less affected by social desirability concerns Proxies are more willing to report disability

Proxy responseUnderstanding self-proxy differences 2


Social relationship with the selected person The closest the relationship, the most

accurate the proxy-response Spouse’s responses are better than other

proxies’ Proffesionals and caregivers provide more

accurate information than lay proxies

Length of relationship Positive effect in convergence self-proxy

Memory capacity of respondent Affect to data quality

Proxy responseThe best proxy-respondent


Types of methods: 1) Traditional

First step: Measure the bias by comparing proxy- estimates with self- estimates

Second step: Introduce statistical control adjusting for demographic or health-related variables

2) New strategy First: Set assumptions about the nature of

differences between self- and proxy-responses

Second: Model and estimate these differences based on the assumptions

Proxy responseMeasuring and correcting bias


Mode of administration


Differences in the sample Population coverage Response rates

Differences due to social context within which data are collected Contact with interviewer

Non-verbal cuesSocial desirability bias: Reduced with

self-administered questionnaires

Mode of administrationSources of differences among modes 1


Differences intrinsic to the mode Visual materials cannot be used by phone Complexity of the questionnaire is limited

by the mode Stimuli in a visual mode Primacy effects

Stimuli in a hearing mode Recency effectsDifferences in responses involving long

lists Order effects Effects of the mode on the interviewer

Mode of administrationSources of differences among modes 2


Telephone interviews Do not underrepresent people with disabilities Sometimes is not as well accepted by population

as face-to-face interviews Proved useful to assess mental health using

recommended instruments (GHQ-12, CIS-R, CIDIS), the Expanded Disability Status Sacle

Telephone vs Mail Mail responses report poorer health and more

chronic conditions than Telephone Differential non-response rates by age:

Elder: NR (Telephone) > NR (Mail) Young: NR (Telephone) < NR (Mail) General: Item NR (Telephone) < Item NR

(Mail)

Mode of administrationResults from comparative studies 1


Face-to-Face vs Mail Health differences between

respondents and non-respondents by mode: non-response bias Non-respondents and late

respondents to Mail are more cognitively impaired and more disabled than respondents (among elderly).

Non-respondents to F-t-F are similar to respondents.



Telephone/CATI vs Face to Face Differential non-response rates:

NR (CATI) > NR (F-t-F) Measurement bias:

MBias (CATI) < MBias (F-t-F) No differences between CATI and F-t-F

for reports on chronic conditions, activity limitations and disability rates

CATI vs CAPI Differential non-response rates by age:

Elder: NR (CATI) > NR (CAPI)Young: NR (CATI) < NR (CAPI)



CASI Reduces social desirability bias Enhances the feeling of privacy Respondents generally like CASI Technological possibilities have a

positive influence on data quality (minimizing errors)

Respondent’s self-disclosure is higher in CASI



Disabilities: Physical, sensory, mental illness and mental retardation

Challenges addressed: communication, fatigue and cognitive issues

Questionnaire design Eliminate soft consonant sounds (s, z, t, f and g)

to overcome high-frequency hearing loss Build in "breaks" for respondents to let them rest Incorporate neutral encouragement to avoid

drop-outs Design checks for unexpected responses Use structured probes for questions that might

be difficult to understand

Mode of administrationAn experience interviewing disabled people by telephone


Interviewer training and supervision Usual background and purpose of study Training on challenges likely to face Sensitive exercise regarding the treatment

to the disabled person Guidance to overcome each of the

challenges Support to interviewers and reduction of

stress

Other recommendations Interviews take longer Multiple sessions

may be required

Mode of administrationAn experience interviewing disabled people by telephone


Non-response


Components of non-response Non-contact Non-cooperation

Non-contact: the household pattern of disabled Are older and likely to live in non-

metropolitan areas higher probability of being contacted

Less likely to live with children lower prob. of contact

May be more fearful of opening doors to strangers lower prob. of contact

Non-response and Disability


Non-cooperation: exchange theory vs social isolation theory Exchange theory: Persons who feel that the

survey sponsor has provided (or could provide) benefits to them are more likely to cooperate

Persons with disabilities might be expected to be more cooperative with a government-sponsored survey

Social isolation theory: People who are isolated from the mainstream society feel less responsibility toward government and are less likely to cooperate

Persons with disabilities might be expected to be less cooperative

Non-response and Disability


Persons with severe disability are more likely than persons with less severe disability to be contacted and to cooperate, but when they are interviewed they are less likely to answer for themselves; instead, proxy and assistant respondents tend to answer for them.

Elder people with disabilities respond sooner than young and non-disabled to mail surveys. These findings support Exchange theory

Self-perceived memory problems increase item non-response and “I don’t know” answers.Not preceived memory deficits implies inaccurate information on items requiring recall.

Non-response and DisabilityResults from studies 1


Health of non-respondents is worse than respondents’ in terms of: stroke, Basic ADL, mobility disabilities, self-rated health and mortality rates.

Respondents with poorer physical functioning and/or limiting long-term illness have higher non-response rates. These findings support Social isolation

theory

Non-response and DisabilityResults from studies 2


Recommendations for including disabled people in

interview surveys


Bias due to exclusion is amplified in surveys where disability is a key measure of interest

Recommendations: Include institutionalised population

in samples Carry out the interview in a private

environment Provide adaptative technologies and

procedures (of primary importance in visual, hearing and speech impairment).

Use simple and clear questions


Recommendations: Special training for interviewers Provide alternative modes of

administration If proxy is used:

Respondent’s assessments of proxy’s answer is valuable

The proxy should be nominated by the selected respondent


Proxy response validity depends on factors such as: proxy choice and distinction, topic investigated, health condition of the selected person...

Mode of administration (unique or a mix-mode) should be decided taking into account: topic investigated, population objective of the survey, expected acceptability of the mode by the population, technological possibilities...

Final conclusion


Different patterns of response might be indicative of bias which can affect estimates.

Non-response is one of the sources of data error. But not the unique... Emphasis should be given not only to minimazing non-response rates but also to estimation and control of measurement error.

Final conclusion

21-23 September 2005

Fifth Meeting of the Washington Group

Documents

Disability questions in censuses and surveys Methodological issues: proxy response, non-response and mode of administration 21-23 September 2005 Montserrat