Model-Based and Design-BasedMethods in “Empirical Micro”

David Card, UC Berkeley

“Empirical micro” has emerged in the past 3 decades out of labor, public economics, and related fields

- undergraduate level: econometrics curriculum

- graduate level: “applied econometrics” courses, JMP’s

- individual fields:- big changes in development and health economics

- public perception: econ is not just about interest rates

Today’s talk:

- a little background

- some lessons from the design-based perspective

- sources of the tension between design-based and model-based approaches

- thoughts on going forward

I. Background Issuesa. Models and Causality- a primary goal in economic research is to draw causal inferences. What role do models play in this process?

- one view (e.g., Heckman): causality is model-based: causality only exists within the framework of a theory that says 'x causes y’

- an opposing view (e.g., Holland): causality is design-based: a claim of causality requires that you can design a manipulation in which ‘x causes y’

b. Working Models in Economics- economic studies always have a “working model”

-the working model specifies (at least implicitly)-whose behavior are we modeling? -what are the constraints?-what are the endogenous choice variables?-what variables are known/observed?-what time horizon, level of aggregation...?

- successful working model constrains the focus of the study (and holds back the audience)

BUT: wide debate about what is legitimate working model

- trend in fields outside “empirical micro” is toward more complex working models with dynamics, imperfect information, and GE

- come back to this later...

Three main classes of working models in emp. micro:

1. explicit/approximating Archetype: Cowles Commission; consumer demand models (Deaton and Muellbauer, 1980)

-theory implies yi=fi(Xi), with restrictions on fi-researcher estimates approximating model:

yi = Xiβ + εi (White, 1980)

-error “tacked on”: meas. error, specification error, ...

-focus on testing ‘general predictions’ of theory

2. explicit/exactArchetype: McFadden’s (1974) multinomial logit

- model gives complete d.g.p. (“all causes” model)e.g. utility function + unobserved heterogeneity in tastes

complete demand system

-focus on estimating model parameters and distribution of unobservables

-requires specific functional forms, but once estimated can be used for prediction, welfare analysis, etc.

3. implicit working modelArchetype: program evaluation (training, etc.)

-no precisely specified behavioral model. Focus instead on a “structural” outcome equation with a signed coefficient

-can measure causal effect of existing policy, but cannot extrapolate very far

-widely extended to studies of non-market behavior

c. Emergence of Design-based research

The “crisis” in econometric practice in the late 1970s/early 1980s

- Lucas (1976) critique - Hendry (1980) “Alchemy or Science” - Leamer (1983) “specification searches”- failure of replication efforts (JMCB study)-Lalonde (1985) critique of program eval. methods

1980s/1990s Responses

in labor economics:- randomized instruments (draft lottery)- arguably exogenous instruments - quasi-experiments - Mariel boatlift, NJ min. wage - RD (Angrist-Lavy, Lee's vote share idea)

outside labor:- adoption of design-based approaches in public

finance, development, health economics- explosion in lab-based and field experiments

III. Insights from Design-Based Approacha) counterfactuals and causality-widespread adoption of Rubin causal model-identification equated with research design (LATE)

b) substantive specification testing-randomization/balance tests-pre-treatment comparisons, falsification exercises,

multi-way differencing for observational designs-hierarchy: randomized design > RD > IV with

balanced design > event study > DD > OLS

c) replicability and pre-specified designs

- growing emphasis on “replicability” as a condition for credibility (program/data achives) -- replication failures even in lab experiments (Psychology)- do we reward or punish replicators in Economics?- movement for pre-specified research plans, particularly for field experiments (multi-testing)

d) how much “prior knowledge” do we have?

- Deaton and Cartwright (2018): randomized designs are less appealing when we have strong prior knowledge (Fisher vs. Student)

- design based studies often turn up small or even wrong-signed results (income experiments,....)

- especially in labor economics: standard models often fall short. Other fields may be better...

IV. Tensions between model-based and design-based approaches

Background factors

1. DSGE model becomes the working model in macro. (Lucas, Prescott, ...). Models that ignore dynamics, imperfect information, and GE are seen as incomplete and potentially misleading

2. growth of empirical research in IO, heavily influenced by McFadden’s logit framework

- limited data and sources of exogenous variation (e.g., only market shares directly observable)

- relatively complete model needed for anti-trust policy analysis, analysis of dynamic games (entry)

- structural model needed to infer much from observed bids in auctions

3. Developments in econometric methodology

- mixed logit and extensions (BLP)

- Rust, Hotz-Miller, … adaptation of nested logit to intertemporal context

- advances in simulation/integration techniques

- partial identification

Key Areas of Disagreement

a. Completeness/complexity of the working model

- design-based papers use simplified working models (often single equation model) with no formal derivation of the DGP. Emphasis is on “sharp” predictions and falsification.

- model-based papers work from the assumed model (though estimation may only use moment conditions). Emphasis on sophisticated agents, equilibrium behavior.

•Sims (2010) comment on Angrist&Pischke:

... by promoting single-equation, linear, single-instrument modeling focused on single parameters ... they are helping create an environment in which applied economists emerge from PhD programs not knowing how to undertake deeper and more useful analyses of the data.

b. What are the “parameters of interest”?

- design-based approach estimates E[y|X] = Xβ,with focus on “clean identification” of β

- Is β endogenous? (e.g., welfare takeup; retirement; monetary policy rules)

- model-based approach posits g(y,x,e; θ)=0with focus on estimating θ, sometimes using the

implied linear projection L(y|X) = Xβ(θ)

c. Identification

- design-based practitioners emphasize "credible" identification, with testing/evaluation of assumptions

-model-based studies are often identified by combination of stochastic assumptions, functional form and exclusion restrictions, and “outside information” (calibration)

- differences reflect confidence in the model

d. Inherent confidence in the working model

- some design-based studies attempt to test basic predictions. E.g., min wage studies, lab experiments

- in program evaluations: null hypothesis of a zeroeffect "proof of concept"

- model-based studies usually stipulate a relatively complete model. Goal of research is to estimate parameters and conduct welfare analysis. (Some attempts at out-of sample prediction)

e. Choice of Topic

- design based approach has crossed over into many areas on the fringes of economics (including non-neoclassical topics/questions) - e.g. peer effects, happiness

- model-based approach tends to emphasize neoclassical topics, in part because of need for model development

- nevertheless: some model-based behavioral economics (DLM charitable giving study)

V. Going Forward

a. Generalizability /Extrapolation- classic criticism of experiments: generalizability- simplest design-based studies recover some average treatment effect for the subjects in the study, relative to a counterfactual

Some new directions:1. MTE’s (Heckman and Vytlacil) or other generalized IV approaches - “mine” the available data in a given design

2. richer model of the counterfactuals/treatmentsE.g., models with two treatments

- Kline-Walters - Head Start evaluation- Mountjoy - 2 year and 4 year colleges

3. combining evidence from multiple studies- works when there is a single treatment effect of interest that is estimated in different settings

e.g., treatment effects for ALMP’s- can potentially recover design biases (e.g. randomized v. non-randomized designs) and publication biases

Medium Term Effect Sizes/Confidence Intervals

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

Effect Size on P[Employment] 1-2 years out

b. Ideas from Machine Learning in Stats/CS

- ML models provide ‘best possible’ predictions- can be useful for classifying/stratifying

- cross validation and related ideas helping address ‘overfitting’

- LASSO useful in model selection and many-instruments problems

- new work on multiple testing (Efron): Kline-Walters randomized audit model

c. Really Big Data Sets- design-based approaches are highly

complementary to really big (population) data- can search for “rare” designs- or power up IV/RD/RKD designs

- but: many “big” data sets are sparse (e.g. tax data have limited behavioral measures) need for creative modeling

- budget and access limitations for big data are changing empirical micro into a lab-based model

Final Thoughts- Unlike physics, there is no ‘standard model’ of microeconomic behavior- Many basic premises in conventional working

models – such as temporally separable preferences, risk neutrality, etc., – are probably wrong for many/most agents-

- There is a need for many kinds of applied work:“descriptive work” that can point to new facts“program evaluation”“simple hypothesis testing” (1st order predictions)“fit the best available model”