How to assess questionnaires

QUESTIONNAIRE

EVALUATION Dr. Ahmed-Refat AG Refat December 12,

2014

1

Contents

December 12, 2014 Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat

2

Main Concepts & terms ( Pre-test)

Sequence of questionnaire development &

testing

Questionnaire Testing Methods

Errors Detection

Questionnaire Evaluation Criteria

Criteria for a Good Questionnaire

Using Questionnaire Appraisal System (QAS)-99


3

Main Concepts

& Terms ( Pre-test :10 terms)

Define the underlined terms in the following slide…….

It has to be established that the Qes. is valid and reproducible in the

context in which it is going to be employed.

The validity of the questionnaire should have been investigated;

The questionnaire are a correct and comprehensive reflection of the

concept the questionnaire is intended to measure (content validity).

the dimensionality (factor structure) and internal consistency of the

questionnaire should have been investigated;

the test-retest reliability of the questionnaire should have been

investigated;

Responsiveness will need to be determined for questionnaires being

to be used to measure changes within individuals .

Information regarding the interpretation of questionnaire scores should

be available

Floor effects or ceiling effects less than 15%

Accepted score 7+ December 12, 2014

4

Dr. AhmedRefat *** WWW.Slideshare.net/AhmedRefat

Recommended Source-1


5



6



7

Questionnaire Appraisal System

(QAS-99)

QAS-99 is based on a system that developed for

Behavioral Surveillance Branch of the Centers for

Disease Control and Prevention for use in evaluating

questions for the Behavioral Risk Factor Surveillance

System (BRFSS).

http://www.cdc.gov/nccdphp/brfss

Questionnaire development approach


8

I. Determine Analytic Objectives

• What types of data will answer the research question?

II. Develop general concepts to be covered

• List areas to be covered by questions

III. Translate concepts into questions

IV. “Appraise” questions for common pitfalls

V. Evaluate questions empirically

December 12, 2014

9


Sequence of questionnaire development & testing


10

1- Conceptualization


11

Conceptual frame of the Questionnaire is more

important for new surveys, whereas in existing

surveys concepts might be already well established.

An integral part in every change in a questionnaire.

The main output are :

An entities/relationships scheme,

An area tree about the structure of the targeted

questionnaire,

List of target variables

2- Questionnaire design.


12

First draft of a questionnaire

Appropriate wording,

Order of questions and

Definition of answering categories

3- Testing.


13

The questionnaire needs to be tested regarding:

Wording of questions/answers, order and

structure of the questionnaire;

Problems related to translation, and cultural

background ;and

Data collection mode and the involvement of an

interviewer ……….V&R

Testing the Questionnaire-Cont’


14

Pre-field and field methods.

It is recommended to involve respondents.

A combination of different methods is

advisable

4- Revision.


15

After testing, making revisions to the

questionnaire

Afterwards, a new round of testing is often

essential.

This process may be repeated through two,

three or even more phases of testing.

The aim is to check if the changes are really

resulting in a higher validity and reliability of

data in relation to the specific objectives of

the survey.

5- Data collection


16

With the implementation of the survey (either by

a pilot study or as the real survey) the process of

development and revision is terminated,

but the process of observation should be

continued via

monitoring the interviewers

5-Data collection-cont’


17

Monitoring can be regarded as :

Continuous tool of evaluation .

The monitoring of fieldwork can be essential

for the further phase of post survey

evaluation when conducting ongoing surveys

or having implemented a new survey on full

scale.


18 ?

Questionnaire Testing

Methods



19

Aims Phase of testing Methods

1- Focus

groups

(FGs)

(respondent

group

discussion)



20


Early stage of

questionnaire

design

1- Focus

groups

(FGs)

(respondent

group

discussion)



21


To gain a

reflection of

the target

population

perspective

• To check

terms

Early stage of

questionnaire

design

1- Focus

groups

(FGs)

(respondent

group

discussion)

Questionnaire Testing Methods -Cont


22


2- Informal

test

(evaluation by

colleagues)



23


Possible in

each

phase,

preferably at

the beginning

2- Informal

test

(evaluation by

colleagues)



24


• To detect all

kinds of

mistakes:

wording,

layout, skips

etc.

Possible in

each

phase,

preferably at

the beginning

2- Informal

test

(evaluation by

colleagues)


25


3- Expert group

Group

discussion among

design and matter

experts,

sometimes

with users

********************

********************



26


Initial phase of

questionnaire

development

3- Expert group

Group

discussion among

design and matter

experts,

sometimes

with users

********************

********************



27


• To check

concepts,

definitions,

vocabulary

against

survey’s

objectives

• To discuss data

processing

requirements

Initial phase of

questionnaire

development

3- Expert group

Group

discussion among

design and matter

experts,

sometimes

with users

********************

********************



28 Focus Specifically on Methods

• Terms and wording of questions;

• Structure of questions;

• The response alternatives;

• Order of questions;

• Navigational rules

• Instructions to interviewers .

• Confusing layout;

• Typographical errors

3- Expert

group

Group

discussion among

design and matter

experts,

sometimes

with users

********************

********************



29


4- In-depth or

qualitative

interviews

(interviews with

respondents)



30


Early stage of

development and

testing the

questionnaire

4- In-depth or

qualitative

interviews

(interviews with

respondents)



31


• To evaluate

respondents’

viewpoint and

understanding of

the

questionnaire

• Explorative

nature

Early stage of

development and

testing the

questionnaire

4- In-depth or

qualitative

interviews

(interviews with

respondents)



32


5- Cognitive

interviews :(one-

to-one

in-depth,

structured

interviews

with specially

trained

interviewers and

researchers)



33


Middle of the

development

process, once a

draft

questionnaire

has

been developed

5- Cognitive

interviews :(one-

to-one

in-depth,

structured

interviews

with specially

trained

interviewers and

researchers)



34


• To gain

qualitative

information on

how a

questionnaire is

understood and

answered

Middle of the

development

process, once a

draft

questionnaire

has

been developed

5- Cognitive

interviews :(one-

to-one

in-depth,

structured

interviews

with specially

trained

interviewers and

researchers)



35


6- Observational

Interviews

(observation of

respondents while

completing a

questionnaire)



36


Middle of

development,

when a

tested

questionnaire

exists

6- Observational

interviews (observation of

respondents while

completing a

questionnaire)



37


• To check self

completed

questionnaire by

observing

potential

respondents in

the lab

Middle of

development,

when a

tested

questionnaire

exists

6- Observational

Interviews

(observation of

respondents while

completing a

questionnaire)


Questionnaire Testing Methods -Cont’


38


7- Behaviour

coding

(coding behaviour

and interaction of

interviewers and

respondents )



39


After a set of

pre-field

methods have

been

conducted

7- Behaviour

coding

(coding behaviour

and interaction of

interviewers and

respondents )



40


To evaluate the

question-

answering

process by

standardized

methods

and coding

scheme

After a set of pre-

field

methods have been

conducted

7- Behaviour

coding

(coding behaviour

and interaction of

interviewers and

respondents )

Error Detection


41

Completely identify the errors

is virtually impossible;

therefore, the goal should be to

design a testing procedure

capable to catch as many

errors as possible

Errors Detection


42

Q-by –Q Analysis of single questions .

Testing by task Assignment of specific tasks to different testers, so that each

of them focuses on given issues.

Scenario testing Hypothesizing some real situations, entering them in the

questionnaire and checking the performance and the results.


43

Questionnaire

Evaluation

Criteria


The best-known and most comprehensive

criteria are those from the Scientific

Advisory Committee (SAC) of the Medical

Outcomes Trust*….. The SAC defined

eight attributes

that deserve considerations in evaluation.

December 12, 2014

44



(SAC)

Eight Attributes

Of the good Questionnaire

?

December 12, 2014

45



(1) Conceptual and measurement model,

(2) Validity, (……,……,……)

(3) Reliability,

(4) Responsiveness,

(5) Interpretability,

(6) Respondent and administrative burden,

(7) Alternative forms, and

(8) Cultural and language adaptations (translations).

December 12, 2014

46


Specific Criteria

Within each attributes, specific criteria were defined :

(1) content validity,

(2) construct validity,

(3) criterion validity,

(4) internal consistency

(5) reproducibility,

(6) responsiveness,

(7) floor and ceiling effects, and

(8) interpretability. December 12, 2014

47


?


48

Validity

&

Reliability

Validity


49

Validity addresses the

amount of systematic or

"built-in" error contained

in the measure.

Reliability

Reliability refers to random error in measurement.

Reliability indicates the accuracy or precision of the measuring instrument .

The pilot test seeks to answer the question,

Does the questionnaire consistently measure whatever it measures?

December 12, 2014

50


Reliability

Several methods ……………all involve administering

the instrument to a small sample during a pilot test.

A common procedure………is the test/re-test

The instrument is given to the same group of individuals

twice (about one week apart) and the two sets of

scores are correlated, resulting in a coefficient of

stability.

A correlation above 0.7 would indicate acceptable

reliability.

Other techniques …….Cronbach's Alpha.

December 12, 2014

51


Specific Criteria

Within each attributes, specific criteria were defined :

(1) content validity,

(2) construct validity,

(3) criterion validity,

(4) internal consistency

(5) reproducibility,

(6) responsiveness,

(7) floor and ceiling effects, and

(8) interpretability.

December 12, 2014

52


1-Content Validity


53

Content validity examines

the extent to which the concepts

of interest are comprehensively

represented by the items in the

questionnaire

1- Content Validity-cont;


54

To evaluate content validity the following

aspects should be considered :

1. Measurement aim of the questionnaire

2. Target population

3. Concepts that intended to measure.

4. Item selection and item reduction

5. Interpretability of the items.

Completing the questionnaire should not require reading skills beyond that of

a 12-year-old to avoid missing values and unreliable answers

2- Internal Consistency


55

Internal consistency is a measure of the extent to

which items in a questionnaire (sub)scale

are correlated (homogeneous), thus measuring

the same concept.

Internal consistency is an important measurement

property for questionnaires that intend to measure

a single underlying concept (construct) by using

multiple items.

3- Construct Validity


56

Construct Validity is used to ensure that the

measure is actually measure what it is intended to

measure (i.e. the construct), and not other variables.

Using a panel of “experts” familiar with the

construct is a way in which this type of validity can

be assessed.

The experts can examine the items and decide what

that specific item is intended to measure.

4- Reproducibility


57

Reproducibility concerns the degree to which repeated measurements in stable

persons (test-retest) provide similar answers.

5- Criterion Validity


58

Criterion validity refers to

the extent to which scores on

a particular instrument

relate to a gold standard

6- Responsiveness


59

The ability of a questionnaire to

detect important changes over time,

even if these changes are small

Responsiveness is measure of

longitudinal validity.

6- Responsiveness-cont,

A longitudinal study with pre- and post-

testing is required for determining

responsiveness.

It is important in this process that the

design and potential intervention being

used represent the situation in which the

questionnaire will be used in the future.

December 12, 2014

60


7- Floor or Ceiling Effects


61

The number of respondents who achieved the

lowest (floor/ground/basement) or highest

(ceiling) possible score.

Calculation : The % of subjects who achieved the

maximum score (ceiling) or the minimum score (floor).

These effects are considered when 15% of

respondents reach the ceiling or floor scores.

Their effects :implications on the questionnaire’s

reproducibility and responsiveness

7- Floor or Ceiling effects


62

The number of respondents who achieved the lowest or highest possible score.

If floor or ceiling effects are present, it is likely that extreme items are missing in the lower or upper end of the scale, indicating limited content validity. As a consequence, subjects with the lowest or highest possible score cannot be distinguished from each other, thus reliability is reduced.

Furthermore, the responsiveness is limited because changes cannot be measured in these patients.

A floor effect


63

A Floor Effect is when most of your

subjects score near the bottom.

There is very little variance because

the floor of your test is too high.

The question is too hard for the

group you are testing.

A ceiling effect


64

A Ceiling Effect :Most of subjects

score near the top. There is very

little variance because the ceiling of

your test is too low.

The question is too easy for the

group you are testing.

8- Interpretability


65

Interpretability is defined as

the degree to which one can

assign qualitative meaning to

quantitative scores.

Systematic Evaluation of Quest.


66

How do we find questionnaire

problems????????

By conducting a s tructured

expert review

( Technical Review )

Example: QAS-99



67


(QAS-99)

QAS-99 is based on a system that developed for

Behavioral Surveillance Branch of the Centers for

Disease Control and Prevention for use in evaluating

questions for the Behavioral Risk Factor Surveillance

System (BRFSS).




68

For structured expert reviews a set of criteria

(coding schemes ) by which each question is

to be examined are implemented.

The coding scheme needs to be filled in for

each question and consequently the draft

questionnaire needs to be completed by the

standardized coding categories .

QAS-99 ( Items )


69

1. PROBLEMS WITH READING:.

2. PROBLEMS WITH INSTRUCTIONS

3. PROBLEMS WITH ITEM CLARITY:

4. PROBLEMS WITH ASSUMPTIONS

5. PROBLEMS WITH KNOWLEDGE/MEMORY:

6. PROBLEMS WITH SENSITIVITY/BIAS:

7. PROBLEMS WITH RESPONSE CATEGORIES

1. PROBLEMS WITH READING:


70

Determine if it is difficult for the interviewers to read the question uniformly to all

respondents.

1a – What to read: Interviewers may have difficulty determining what parts of the question are to be read.

1b – Missing information: Information the interviewer needs to administer the question is not contained in the question.

1c – How to read: Question is not fully scripted and therefore difficult to read.

2. PROBLEMS WITH INSTRUCTIONS:


71

Look for problems with any introductions,

instructions, or explanations from the

respondent’s point of view.

2a – Conflicting or inaccurate instructions,

introductions, or explanations.

2b – Complicated instructions, introductions, or

explanations

3. PROBLEMS WITH ITEM CLARITY:


72

Identify problems related to communicating the intent or meaning of the question to the

respondent 3a – Wording: The question is lengthy, awkward,

ungrammatical, or contains complicated syntax.

3b – Technical terms are undefined, unclear or complex.

3c – Vague: The question is vague because there are multiple ways in which to interpret it or to determine what is to be included and excluded.

3d – Reference periods are missing, not well specified, or are in conflict.

4. PROBLEMS WITH ASSUMPTIONS:


73

Determine there are problems with assumption if s made or the underlying logic.

4a – Inappropriate assumptions are made about the

respondent or his/her living situation.

4b – Assumes constant behaviour: The question

inappropriately assumes a constant pattern of

behaviour or experience for situations that in fact vary.

4c – Double-barrelled question that contains multiple

implicit questions.

5. PROBLEMS WITH

KNOWLEDGE/MEMORY:


74

Check whether respondents are likely to not know or have trouble remembering

information.

5a – Knowledge: The respondent is unlikely to

know the answer.

5b – An attitude that is asked about may not

exist.

5c – Recall failure.

5d – Computation or calculation problem

6. PROBLEMS WITH SENSITIVITY/BIAS


75

: Assess questions for sensitive nature or wording, and for bias.

6a – Sensitive content: The question is on a topic

that people will generally be uncomfortable

talking about.

6b – A socially acceptable response is implied.

7. PROBLEMS WITH RESPONSE

CATEGORIES:


76

Asses the adequacy of the range of responses to be recorded.

7a – Open-ended question that is inappropriate or difficult.

7b – Mismatch between question and answer categories.

7c – Technical terms are undefined, unclear, or complex.

7d – Vague response categories.

7e – Overlapping response categories.

7f – Missing response categories.

7g –Illogical order of response categories

QAS-99 ( Reference)


77

Question Appraisal System QAS-99 . By:

Gordon B.Willis and Judith T. Lessler . Research

Triangle Institute Suite 420 6110 Executive

Blvd. Rockville, MD 20852 August, 1999


Biemer, P.P. and Lyberg, L.E. (2003).

Introduction to Survey Quality. Hoboken, New

Jersey: John Wiley&Sons

Cited References


78 http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf

http://www.joe.org/joe/2007february/tt2.php

https://readability-score.com/

http://www.joe.org/joe/1990summer/tt2.php

http://www.emgo.nl/kc/preparation/research%20design/8%20Questionnaires%20selecting,%20translating%20and%20validating.htm

Scientific Advisory Committee of the Medical Outcomes Trust. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002;11:193e205.

http://ec.europa.eu/eurostat/ramon/statmanuals/files/Handbook_of_Practices_for_Quest.pdf

http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf

















http://www.emgo.nl/kc/preparation/research design/8 Questionnaires selecting, translating and validating.htm



















79

Thank You

Education

How to assess questionnaires