Lecture 1What is a good question?

RESEARCH METHODSNATIONAL RESEARCH UNIVERSITYHSE ECONOMICS, PH. D PROGRAM DR C S LEONARD JUNE 2011

RESEARCH METHODS

2OUTLINE OF

LECTURE

Knowledge Claims Classical Approaches Post-positivism

Strategies of inquiry Data mining Design-based research

Good questions emerge from good research designs

8/06/2011

RESEARCH METHODS 3

KNOWLEDGE CLAIMS

8/06/2011

RESEARCH METHODS

4 KNOWLEDGE CLAIMS

What warrants knowledge?

How is scientific method applied?

8/06/2011

LEONARD GSOM PH.D. RESEARCH METHODS 2010

5 TWO POSITIONS

4/02/20010

Two scientific positions, inductivism and deductivism

From which method emerges

Deductive (logic) argumentation If premises are true and no fallacies in the argument,

then conclusion will be true

Not concerned with truth or falsity

Inductive arguments may have true premises, but we cannot be certain that conclusions will also be true (ampliative reasoning)

LEONARD GSOM PH.D. RESEARCH METHODS 2010

6INDUCTIVE, DEDUCTIVE

4/02/20010

Deductive, from general to particular, and inductive, from particulars to general

Inductive: Frances Bacon vs the medieval Church: purging ourselves of idols

Problem (Hume) Can we predict the future? No

Positivism is descended from Bacon

Research becomes historical, truth confined to a systematic empirical study, that might obtain general laws

Empirical findings worthless to some deductivists

LEONARD GSOM PH.D. RESEARCH METHODS 2010

7FALSIFICATION:

KARL POPPER (1902-1994)

4/02/20010

Karl Popper’s critical rationalism has generated much debate since the 1930s

Intellectual autobiography, Unended Quest

Background, early Marxist, training with Adler and Freudian theories convinced them that the theories were too broad

Later rejected psychologism

Favored theory of relativity, Einstein, could be tested, verified, falsified

RESEARCH METHODS

8LOGIC OF

FALSIFICATIONISM

Scientific theories are abstract

can be tested only indirectly, by reference to their implications.

Scientific theory, and human knowledge generally, is irreducibly conjectural or hypothetical

to solve problems that have arisen in specific historio-cultural settings

Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory

A single counterexample is logically decisive: it shows the theory, from which the implication is derived, to be false.

8/06/2011

LEONARD GSOM PH.D. RESEARCH METHODS 2010

9 VS INDUCTIVISM

4/02/20010

Direct antithesis of inductivism

Growth of knowledge requires overturning previous beliefs

Have a theory, test it, falsify it and move on

Many economists seek to prove theories correct; the job of science is to disprove them

Don’t strive for certainty (verificationism)

Bans ad hoc adjustments to a theory to prevent it from being falsified

A new theory will possess greater empirical content than its predecessors

LEONARD GSOM PH.D. RESEARCH METHODS 2010

10 CRITICS

4/02/20010

This hinders not promotes science

Can’t reject theories so easily, some theories are better at some things than others

Marxism, accommodates business cycles and disequilibrium better then orthodox Keynsianism, but the Monetarists better understand inflationary processes than the Keynsians, who argue that their theories and policies are more effective against unemployment

Should not critique a new theory too rigorously, because it may have something good in it

RESEARCH METHODS

11DEFENDER:

BLAUG

Blaug (be taken seriously)

Have a prediction about the future

Require a formal model

Falsification is essential

Be scientific, or not

Falsificationism: much tougher

Lay down restrictions on what Popper calls immunizing strategems

8/06/2011

LEONARD GSOM PH.D. RESEARCH METHODS 2010

12ENORMOUS INFLUENCE

4/02/20010

Mark Blaug

Even econometricians, however, econometric results difficult to falsify

Plain fact Most economists tend to verify..

LEONARD GSOM PH.D. RESEARCH METHODS 2010

13 THOMAS KUHN

4/02/20010

Structure of Scientific Revolutions

Science is not good science unless it is working under the framework of a theory, makes no progress

It adopts a particular view of the world, and all subsequent research adds to that

Pre science—lots of theories hoping to explain the same thing

The paradigm: an achievement so important that it attracts an enduring group of adherents Commitment and consensus are prerequisites for normal

science

LEONARD GSOM PH.D. RESEARCH METHODS 2010

14 PARADIGMS

4/02/20010

Older generations stay with their paradigms, new ones acquire new paradigms

Releases scientists from the necessity of debating fundamentals

They can then concentrate on subtle, esoteric aspects of their subject

LEONARD GSOM PH.D. RESEARCH METHODS 2010

15KUHN: NORMAL

SCIENCE

4/02/20010

Mopping up

Determination of facts

Setting the facts within theory

Articulating the theory

Then, anomalies, followed by crisis, followed by fundamental change

LEONARD GSOM PH.D. RESEARCH METHODS 2010

16APPLIED

ECONOMETRICS

4/02/20010

Middle ground

Use theory, provide initial specification

Data exploration techniques to extend or refine it

Bridge theory and empirical data analysis

How do we know a theory is correct?

Different users have different tastes and beliefs

Complications with computation: large data sets numerous models possible

LEONARD GSOM PH.D. RESEARCH METHODS 2010

17 PARADIGMS

4/02/20010

How researchers will learn/what they will learn, assumptions

Philosophical assumptions, epistemologies (how we know something), ontologies (what is knowledge), axiology (what values go into knowledge), methodology (process for studying)

LEONARD GSOM PH.D. RESEARCH METHODS 2010

18 EXTENSIONS

4/02/20010

These debates shaped much social science theory about

Innovations

Science

Path dependence

Historical legacies

Nature of change

Nature of reform and timing

RESEARCH METHODS

19 POSITIVISM

Can we be positive about our claims of knowledge when studying behavior and actions (Comte, Mill, Durkheim, Newton and Locke)?

Causes probably determine effects? Reductionism: reduce ideas into

small discrete sets to be tested

8/06/2011

RESEARCH METHODS

20POSITIVISM VS

INTERPRETIVISM

Interpretivism: Weber (Verstehen)

8/06/2011

RESEARCH METHODS

21POSITIVISM AND ITS

OPPONENTS

Quantitative, positivist, post-positivist research, empirical science

Challenge to positivism: against the traditional notion of the absolute truth of knowledge; playing tennis with the net down

8/06/2011

RESEARCH METHODS

22METHODOLOGICAL

PLURALISM?

Bruce Caldwell (let 100 flowers bloom)

Little economics will survive if we take this seriously

Confirmationism Verification

Falsificationism is never practiced because it is unpracticeable

8/06/2011

RESEARCH METHODS

23BAYSIAN

METHODS

Test, verify and refine the laws and theories governing behavior

Baysian methods: from theory, to test, to revision

8/06/2011

RESEARCH METHODS

24(1) POST

POSITIVISM

Knowledge is conjectural

Research is to make claims and refine or abandon them

Data, evidence and rational considerations shape knowledge

Being objective is key

8/06/2011

RESEARCH METHODS

25(2) SOCIAL

CONSTRUCTION

Social construction

Mannheim, Burger, Luckmann, Neuman

Look at participants views

Judgments are subjective, meanings are varied and multiple

Interviews: open ended questioning, the more the interviewee talks spontaneously, the better

Participants allowed to construct meaning (rather than responding to concrete situations)

Process of interaction, context of work

8/06/2011

RESEARCH METHODS

26 (3) PRAGMATISM

Pierce, James and Dewey

Knowledge claims arise out of actions, situations, consequences, rather than ex ante conditions

Concern with what works

Pluralistic approach

Mixed methods, qualitative, quantitative

Research always occurs in social contexts

Stop asking questions about the laws of nature

8/06/2011

RESEARCH METHODS 27

LOGIC OF RESEARCH DESIGN

Influence

Research

Design

8/06/2011

RESEARCH METHODS

28“ACROSS MOST FIELDS… APPLIED

ECONOMISTS ARE NOW LESS LIKELY TO PIN A CAUSAL INTERPRETATION OF

THE RESULTS ON ECONOMETRIC METHODOLOGY ALONE.

DESIGN-BASED STUDIES ARE DISTINGUISHED BY THEIR PRIMA FACIE

CREDIBILITY AND BY THE ATTENTION INVESTIGATORS DEVOTE TO MAKING

BOTH AN INSTITUTIONAL AND A DATA-DRIVEN CASE FOR CAUSALITY.”

JOSHUA D. ANGRIST AND JÖRN-STEFFEN PISCHKE

8/06/2011

RESEARCH METHODS

29GENERAL TO SPECIFIC

METHODOLOGY

“LSE” tradition of time-series econometrics that began in the 1960s at the London School of Economics

Mizon (1995) a brief history The practitioners of LSE econometrics are now widely dispersed among academic institutions throughout Britain and the world.

The LSE approach is described sympathetically in Gilbert (1986), Hendry (1987,1995, esp. chs. 9-15), Pagan (1987), Phillips (1988), Ericsson, Campos and Tran(1990), and Mizon (1995). For more sceptical accounts, see Hansen (1996) and Faust and Whiteman (1995, 1997)

8/06/2011

RESEARCH METHODS

30GENERAL TO

SPECIFIC

Context: Linear (cross-country growth) Use: Time-SeriesStep 1. General regression will include every possible variable -- all the information about the true determinants. Step 2.The information content is then sharpened by a more parsimonious regression – the specific regression

(a) it is statistically well specified (for example, it has white noise errors);

(b) that it is a valid restriction of the general regression, and (c) that it encompasses every other parsimonious regression

that is a valid restriction of the general regression

Criticism: data-mining,

8/06/2011

RESEARCH METHODS

31 EXTREME BOUNDS ANALYSIS: THE NEW CRITIQUE Edward Leamer’s “extreme-bounds analysis” (1983, 1985).

A coefficient of interest is robust only to the degree that it displays a small variation to the presence or absence of other regressors. Leamer and Leonard (1983) define the extreme-bounds for the coefficient of a particular variable within a search universe as ranging between the lowest estimate of its value minus two times its standard error to the highest estimate of its value plus two times its standard error, where the extreme values are drawn from the set of every possible subset of regressors that include the variable of interest. A variable is said to be robust if its extreme bounds lie strictly to one side or the other of zero.

8/06/2011

RESEARCH METHODS

32 JUDGMENT CALLS

The main difference between structural and experimental (or ``atheoretic'') approaches is not in the number of assumptions but the extent to which they are made explicit. (Michael Keane)

8/06/2011

RESEARCH METHODS

33 SUMMARY

Quantitative (numbers)

Experimental design--controls

Non experimental design, surveys

Qualitative (words)

Narratives, phenomenologies, ethnographies, grounded theory, case studies

Mixed methods

Sequential, concurrent, transformative

8/06/2011

RESEARCH METHODS

34 QUANTITATIVE

Random assignment of subjects to treatment

Quasi experiments: non random designs

Surveys cross sectional and longitudinal, generalize from sample to population

8/06/2011

RESEARCH METHODS

35 QUALITATIVE

Ethnographies: researcher studies an intact cultural group in its setting over time (responses)

Grounded theory

Derive an abstract theory of a process, action or interaction, grounded in views of participants

Case Studies

Exploring in depth a program, event, activity, process, or individuals, bound in time, variety of procedures

Phenomenological: lived experiences

Narrative research: lives, stories, retellings

8/06/2011

RESEARCH METHODS

36 MIXED

Gets around biases in any one method used exclusively

Progress from one method to another

Illustrate

Determine what the concept is

Test assumptions on one case

8/06/2011

RESEARCH METHODS

37 LOGIC Qualitative:

Wider range of methods, non-numerical by definition

Small n, intensive interviews, depth analysis, discursive, account of event or unit

Focus on event, decision, institution, location, issue or legislation

Incident important in its own right (war, election, change in leadership, marketing strategy, community decision, etc)

Against bifurcation? mixed methods Systematic, scientific research of all kinds

Most research does not neatly fit one or other category

8/06/2011

RESEARCH METHODS

38GOAL OF SOCIAL

SCIENCE RESEARCH

Inference (descriptive, explanatory)

Attempting to infer beyond immediate facts to something broader

Learning about causal effects from data

Public procedures (explicit, codified)

Replication

Conclusions are uncertain

Observes rules of inference

8/06/2011

RESEARCH METHODS

39DESCRIPTIVE

INFERENCE

Distinguish systematic from non-systematic features

Systematic from stochastic

Counter-factuals (what would have happened, had meters not struck the earth 65 million yrs ago)

8/06/2011

RESEARCH METHODS

40RULES FOR RESEARCH

DESIGN

Intuition: Choice of better topics is idiosyncratic

Two ways to test if it is a good topic

Is it about something important in the real world

A research topic should make a specific contribution to an identifiable scholarly literature by increasing our ability to construct verified scientific explanations

ie: locating it within the framework of existing social science literature

This is the subject of the second lecture today—what makes a theory or theoretical contributions valuable to the community of editors of journals

8/06/2011

RESEARCH METHODS

41 CAUTION

There may be reasons a theory is practicable, even though its long term scientific value has been questioned

Theoretically incoherent models used to forecast the US economy—diversion of macroeconomic theory and applied macroeconomics (see Mankiw 1990)

New theories, however, remain speculative

8/06/2011

RESEARCH METHODS

42DESIGN BASED

RESEARCH

Leamer 1983 highlighted the benefits of sensitivity analysis, a procedure in which researchers show how their results change with changes in specification or functional form. Sensitivity analysis has had a salutary but not a revolutionary effect on econometric practice.

As we see it, the credibility revolution in empirical work can be traced to the rise of a design-based approach that emphasizes the identification of causal effects.

8/06/2011

RESEARCH METHODS

43 WOW-FACTOR

Design-based studies typically feature either real or natural experiments and are distinguished by their prima facie credibility and by the attention investigators devote to making the case for a causal interpretation of the findings their designs generate.

Design-based studies are most often found in the microeconomic fields of Development, Education, Environment, Labor, Health, and Public Finance, but are still rare in Industrial Organization and Macroeconomics.

8/06/2011

RESEARCH METHODS 44

LITERATURE REVIEW

8/06/2011

RESEARCH METHODS

45LITERATURE

REVIEW

Classical

Systematic review

Meta-analysis

Narrative review

Search issues

Presentation

8/06/2011

RESEARCH METHODS

46CLASSICAL

LITERATURE REVIEW

Large Disciplinary Differences Sociology, Psychology, Business Economics

8/06/2011

RESEARCH METHODS

47SYSTEMATIC

REVIEW

A systematic review aims to provide an exhaustive summary of literature relevant to a research questions

The first step of a systematic review is a thorough search of the literature for relevant papers.

The Methodology section of the review will list the databases and citation indexes searched as well as any individual journals. Next, the titles and the abstracts of the identified articles are checked against pre-determined criteria for eligibility and relevance.

8/06/2011

RESEARCH METHODS

48 MORE

Ability to control for between-study variation

Including moderators to explain variation

Deal with information overload: the high number of articles published each year.

It combines several studies and will therefore be less influenced by local findings than single studies will be.

Makes it possible to show if a bias for published works exists.

8/06/2011

RESEARCH METHODS

49SYSTEMATIC

ANALYSIS

Meta-Analysis

Meta-analysis leads to a shift of emphasis from single studies to multiple studies. It emphasizes the practical importance of the effect size instead of the statistical significance of individual studies. This shift in thinking has been termed "meta-analytic thinking"

8/06/2011

RESEARCH METHODS

50 META-ANALYSIS(AVERAGE EFFECT SIZE, FOREST PLOT)

8/06/2011

RESEARCH METHODS

51META-ANALYSIS:

CRIT

About science, but not science

Statistical examination of scientific studies

Cannot propose ways to falsify a theory

8/06/2011

RESEARCH METHODS

52META-NARRATIVE

REVIEW

Example: Connected Communities’,

a research programme of the Institute of Health and Human Development (IHHD) to investigate the meanings of community within and across research disciplines by adopting an innovative methodology based on a meta-narrative systematic review approach. Policy and academic interest in the concept of ‘community’ is longstanding and such interest has become central to policy making in the last two decades.

Meta-narrative review shows the diversity in the meaning of ‘communities’ --various conceptualisations and meanings of community across disciplines, over time, and within different cultures and contexts

(Greenhalgh et al, 2005) is a type of ‘systematic’ review rather than a traditional expert driven literature review

A focus on identifying the ‘storylines of research’ within and across disciplinary boundaries. Identifies the meta-narratives of each discipline and analyse the different ‘discourses’ and languages of ‘community’.

8/06/2011

Until Next HourTHE END