U.S. DEPARTMENT OF STATE THIRD ANNUAL CONFERENCE ON PROGRAM EVALUATION:

NEW PARADIGMS FOR EVALUATING DIPLOMACY IN THE 21ST CENTURY

June 8-9, 2010

Health Track

PowerPoint presentation from workshop on Validation of a Measure of Household Hunger for Cross-

Cultural Use

Session transcript: http://www.state.gov/s/d/rm/rls/rm/2010/148053.htm

Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: fanta2@aed.org Website: www.fanta-2.org

Validation of a Measure of

Household Hunger for Cross-

Cultural Use

Findings of work by:

Megan Deitchler, Terri Ballard, Anne Swindale and

Jennifer Coates

What is Food Security?

“When all people at all times have physical and economic

access to sufficient food to meet their dietary needs for a

productive and healthy life.” (USAID 1992)

This definition is founded on 3 elements:

1. Food availability

2. Food access

3. Food utilization/consumption

3

Food

Access

Food

Consumption

& Utilization

Food

Availability

Food Security

National level

Individual level

Household level

Food balance

sheets (FAO)

Various, no

standard

method

Anthropometric

indicators

Le

ve

l of

Me

asu

rem

en

tS

tan

da

rd M

eth

od

of

Mea

sure

me

nt

Defining and Measuring Food Security

Prevalence of “undernourishment” by country,

FAOSTAT 2009

Why do we need information on household

food access?

• To understand who is affected by food insecurity

• To understand to what extent households are affected by

food insecurity

• To understand how households are affected by food

insecurity

Useful measurement criteria for an indicator of

household food access

• Simple, internally, externally valid measure

• Provides timely information

• Possibility to obtain disaggregated data by region,

urban/rural communities

• Comparable, cross-culturally valid measure

Motivation for a comparable measure of

household food access for cross-cultural use

• Decision making: Facilitates prioritization of geographic

regions for intervention

• Monitoring and evaluation: Enables multi-regional and

cross-country evaluation of programs

• Policy advocacy: Increased attention to food

deprivation with availability of comparable data for

resource-poor areas

7

Process to identify a comparable measure of

household food access for cross-cultural use

Phase 1. Development of Initial Scale

• Background concept paper

• Multi-year field validation studies

• Literature review

Phase 2. Validation and Revision of Scale

• Identify partners and datasets for validation study

• Assess internal, external and cross-cultural validity

• Revision of scale according to validation results

8

Phase 1: Development of initial scale

9 scale items (I) defined:

1. Worry that the household would not have enough food

2. Not able to eat the kinds of food preferred

3. Eat a limited variety of foods

4. Eat some foods that you really did not want to eat

5. Eat a smaller meal than you felt you needed

6. Eat fewer meals in a day

7. No food to eat of any kind in your household

8. Go to sleep at night hungry

9. Go a whole day and night without eating

4 frequencies (F) defined:

Never, Rarely, Sometimes, Often

9

Phase 2: Validation and revision of scale

Data sets for validation study:

Mozambique Round 1 (Moz R1) 2006 n=591 rural; 4 districts

Mozambique Round 2 (Moz R2) 2007 n=299 rural; 2 districts

Malawi (Mal) 2007 n=1161; 3 districts

Kenya 2006 n=152 urban; HIV/AIDS affected hhs; (purposive sampling)

Zimbabwe (Zim) 2007 n=176; HIV/AIDS affected hhs in 3 districts

South Africa (S Africa) 2006 n=491; 1 district municipality

West Bank Gaza Strip (WBGS) 2006 n=1973; nationally representative

Primary validation analysis method:

Rasch modeling using CML for polytomous and dichotomous data

10

Example of Rasch analysis: Scale items and

households on a logit continuum

Less severe

Item 1

-5.1

Item 3

-3.4

Item 4

-2.5

Item 2

-4.8

Item 6

0.7

Item 5

-0.9

Item 7

2.2

Item 8

3.9

Item 9

5.8

HH 1

-1.4

HH2

-2.9

HH 3

2.5

More severe

Scale measurement criteria and method for

evaluation

• Internal validity: Infit and outfit statistics

• External validity: Direction and strength of association

with related indicators

• Cross-cultural validity: Cross-cultural comparison

plots of standardized item and household calibrations

Example of Rasch analysis: Perfect cross-

cultural equivalence

item1step1

item2step1item3step1

item4step1item5step1

item6step1

item7step1item8step1

item9step1item1step2item2step2

item3step2item4step2

item5step2item6step2

item7step2

item8step2item9step2

item1step3item2step3

item3step3item4step3

item5step3

item6step3item7step3

item8step3

item9step3

pop

ula

tio

n2

population1

Two of the alternative [shorter] scales tested

5 Item 3 Frequency (5I 3F) Scale:

5. Eat a smaller meal than you felt you needed

6. Eat fewer meals in a day

7. No food to eat of any kind in your household

8. Go to sleep at night hungry

9. Go a whole day and night without eating

3 Item 3 Frequency (3I 3F) Scale:

7. No food to eat of any kind in your household

8. Go to sleep at night hungry

9. Go a whole day and night without eating

14

Internal validation results: Assessing equal

discrimination of items and item steps

15

Infit

Range % of Items and Items Steps Outside of Infit Range

5I 3F Moz R1 Moz R2 Mal WBGS Kenya Zim S Africa

0.7-1.3 33.3% 26.7% 0.0% 0.0% 0.0% 40.0% 6.7%

0.6-1.4 6.7% 20.0% 0.0% 0.0% 0.0% 6.7% 6.7%

0.5-1.5 0.0% 6.7% 0.0% 0.0% 0.0% 0.0% 0.0%

3I 3F Moz R1 Moz R2 Mal WBGS Kenya Zim S Africa

0.7-1.3 11.1% 22.2% 0.0% 33.3% 0.0% 33.3% 55.6%

0.6-1.4 0.0% 0.0% 0.0% 0.0% 0.0% 22.2% 22.2%

0.5-1.5 0.0% 0.0% 0.0% 0.0% 0.0% 22.2% 0.0%

Cross-cultural validation results for item steps:

5I 3F, Moz R2 and Moz R1

5step1

5step2

6step1

6step2

7step1

7step2

8step1

8step2

9step1

9step2

-6-5

-4-3

-2-1

01

23

45

6

Mo

zam

biq

ue

R2

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1

16

17

Cross-cultural validation results for item steps:

5I 3F, Zim and Moz R1

5step1

5step2

6step1

6step27step1

7step2

8step1

8step2

9step1

9step2

-6-5

-4-3

-2-1

01

23

45

6

Zim

bab

we

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1

Cross-cultural validation results for item steps:

3I 3F, Moz R2 and Moz R1

q7step1

q7step2

q8step1

q8step2

q9step1

q9step2

-6-5

-4-3

-2-1

01

23

45

6

Mo

zam

biq

ue

R2

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1

Cross cultural validation results for item steps:

3I 3F, Zim and Moz R1

q7step1

q7step2

q8step1

q8step2

q9step1

q9step2

-6-5

-4-3

-2-1

01

23

45

6

Zim

bab

we

-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6Mozambique R1

Cross cultural validation results for HH

measure: 3I 3F, Moz R2 and Moz R1

1

2

3

4

5

-4-3

-2-1

01

23

4

Mo

zam

biq

ue

R2

-4 -3 -2 -1 0 1 2 3 4Mozambique R1

Cross-cultural validation results for HH

measure: 3I 3F, Zim and Moz R1

1

2

3

4

5

-4-3

-2-1

01

23

4

Zim

bab

we

-4 -3 -2 -1 0 1 2 3 4Mozambique R1

22

Results for Household Hunger Scale (HHS) by

data set

Little to no

Household

Hunger

Moderate

Household

Hunger

Severe

Household

Hunger

Mozambique R1 42.8 46.4 10.8

Mozambique R2 43.1 48.8 8.0

Malawi 51.9 37.2 10.9

West Bank

Gaza Strip74.9 18.7 6.5

Zimbabwe 51.4 33.7 14.9

South Africa 31.2 46.4 22.4

External validation results: Proportion of households

classified in each HHS category by household wealth

score, Moz R1

0

10

20

30

40

50

60

70

80

90

100

0 1 2 3 4 5 6 7 8 9 10Pro

po

rtio

n C

lass

ifie

d i

n H

HS

C

ate

go

ry

HH Wealth Score

Little to no hunger

Moderate hunger

Severe hunger

23

External validation results: Median monthly HH income

by consumption unit for each HHS category, WBGS

and S Africa

0

50

100

150

200

250

300

350

400

450

500

Little to no hunger

Moderate hunger

Severe hunger

Me

dia

nH

H I

nc b

y C

U

WBGS

SA

24

0

10

20

30

40

50

60

70

80

90

100

Pre-harvest, Dec '06 Post-harvest, July '07

Little to no hunger Moderate to severe hunger

0

10

20

30

40

50

60

70

80

90

100

Pre-harvest, Dec '06 Post-harvest, July '07

Little to no hunger Moderate to severe hunger

External validation results: Is the HHS sensitive to detecting

change? Trends from 2 districts in Mozambique

Household hunger remains

relatively constant over time.

Series of climactic shocks:

Droughts, Floods in early 2007, and

Cyclone Favio just before harvest.

Household hunger improves

significantly over time.

Gondala harvest less affected by

climactic shocks.

25

External validation results: Is the HHS sensitive to

detecting change? Trends from 2 districts in Mozambique

Household hunger remains

relatively constant over time.

Series of climactic shocks:

Droughts, Floods in early 2007,

and Cyclone Favio just before

harvest.

Household hunger improves

significantly over time.

Gondala harvest less affected by

climactic shocks.

Chibabava, Sofala Province,

Mozambique

Gondala, Manica

Province, Mozambique

Limitations of Study

• Data sets used in validation study were mostly limited to

Southern and Eastern Africa

• Sample size limitations for the Kenya and Zim data sets

• Adaptation work not undertaken for the WBGS and SA

data sets

• Rasch assumption of conditional item independence was

not assessed

• External validation was limited by the variables available

in data sets

• The data used for validation of the 3I 3F scale (and other

scales tested) were collected as part of a longer, 9I 4F

scale26

Next steps

• Publish a HHS operational guide, to provide

instruction for data collection and tabulation

• Continue internal and cross-cultural validation

of HHS, as data are collected in more contexts

• More external validation analyses, with more

robust measures of food deprivation

• Potential study to evaluate if the HHS collected

with a 3I3F questionnaire performs differently

than a HHS collected with a 9I4F questionnaire

Conclusions

• There are trade-offs between identifying a measure appropriate for use within one culture and identifying a measure appropriate for use across cultures

• 3I 3F scale shows reasonable equal item discrimination for most data sets and expected associations with „external‟ variables. Appears to provide a measure for cross-cultural comparison.

• Use of the 3I 3F scale should not preclude the concurrent use of a culturally specific measure of food insecurity, when available (e.g. ELCSA in LAC region).

• Other longer scales should be explored if a measure of food insecurity (rather than food deprivation) is desired for any one context.

• In the absence of cross-cultural validation, the resultant measure should not be assumed to be appropriate for cross-cultural comparative purposes.

28

Acknowledgments

The contribution of data by the following organizations is gratefully acknowledged:

Mozambique: FAO Representation

Malawi: Dept. of HIV/AIDS and Nutrition, UNICEF

West Bank Gaza Strip: FAO Jerusalem office

Kenya: Samwel Mbugua and Human Nutrition, Egerton University

Zimbabwe: Center for Applied Social Science, University of Zimbabwe

South Africa: South African Human Sciences Research Council

29

This presentation is made possible by the generous support of the American people through the support of the Office of Health, Infectious Disease, and Nutrition, Bureau for Global Health, United States Agency for International Development (USAID), under terms of Cooperative Agreement No. GHN-A-00-08-00001-00, through the Food and Nutrition Technical Assistance II Project (FANTA-2), managed by AED. The contents are the responsibility of AED and do not necessarily reflect the views of USAID or the United States Government.

Food and Nutrition Technical Assistance II Project (FANTA-2)Academy for Educational Development 1825 Connecticut Ave., NW Washington, DC 20009Tel: 202-884-8000 Fax: 202-884-8432 E-mail: fanta2@aed.org Website: www.fanta-2.org