Unit 3

Siobhan Carey

Department for International Development

Making cross-national comparisons using micro data

Content

• Why it’s important to get it right

• Preparation

• Assessing the data

• Organising your analysis

• Presentation of results

Does it matter if it’s wrong?

• Don’t be famous for the wrong reasons– Importance of reproducibility– Be able to defend your analysis if challenged

• Two examples of why it’s important

International Adult Literacy Survey

• A survey run by very reputable organisations on literacy skills of adults

• How do you explain the differences?

• Various theories

….cont’d

• Methodological review found– differences in sampling methods– differences in fieldwork practices – deviation from the survey design

• A second review – translation– scoring of tests– …………

Programme for International Student Achievement (PISA)

• How do you explain the differences?– didn’t lead to any investigation– results accepted but lead to huge debate

• Was IALS worse that other surveys?

• Why did PISA not lead to the same questions?

Preparation

• Know the data• Read the documentation before you do the

analysis• Have clearly defined research questions • Which countries are you going to include

– Small number and expand?– Large number and reduce?– Combine countries?

What do you need to consider

• Was the survey intended to support cross-national research from the outset?

• Is the survey design the same across all countries?

• In what areas do the countries differ in design or execution and are these important? e.g. – all school types included?

Understanding the data

• Comparability of overall design• What questions were asked – who was asked• Sample design and coverage• Mode• Survey response• Data collection methods• Translation• Data processing and imputation• Missing values• Topic measurement – any special features

(anthropometry, biochemical…)

Understanding the variables

• Which variables are derived

• Which are imputed– How have they been imputed– What other modifications have been made

e.g. hours spent watching TV – a categorical variable modified into a continuous variable

Preparation

• Creation or transformation of variables– Recording, derived variables, dummy

variables, reference group

• Reducing dataset to improve speed• Document your analysis - always• Develop good habits

– in-flight v programme based - Naming conventions

Weights

• Design weights

• Population weights

• Compound weights

• Replicate weights

Which you use will depend on what comparisons you are making

Analysis

• Rarely unique

• Use existing analyses to learn

• Check and double check – – Reproduce, check bases, check population,

check text to tables, check tables to source….

• Check significance

Analysis - output

• Does it pass the common sense test?• Are the bases right?• Are the right weights on?• Is it plausible?• What can you triangulate against? • Is your analysis simply a reflection of

distribution? • What could be behind it? – e.g. poverty

reduction and population growth

Analysis

• Horse race?– Gets media attention but …

• Correlates are more interesting – e.g. relationship between sexual behaviour and HIV

prevalence

• Analysis needs to be set in context• Take into account system factors – e.g. school

characteristics, hierarchical features, standardisation

Presenting your analysis

• Borrow from others – OECD, World Bank, UN, Lancet …

• Try to keep it simple – – what’s the story – and why is it interesting

Rewards?

• Help understand social condition

• Help unpick complex relationships

• Contribute to evidence base so as to improve policy and outcomes

• Interesting – only if you’re curious!

Activity • Using ESS – on a topic of interest

– formulate a research question - which countries are of interest?– investigate some variables that are relevant– look at difference when weights applied– are there differences in distribution between

countries? Why might that be?– what is the unit of analysis and which weights should

you use?– Develop an analysis plan

Suggestions • Crime and fear of crime

• Religiosity

• Attitudes to organs of the state

• Values – attitudes to cheating on tax– attitudes to sexual behaviour