View
4
Download
0
Category
Preview:
Citation preview
NBER WORKING PAPER SERIES
WHAT WORKS? A META ANALYSIS OF RECENT ACTIVE LABOR MARKET PROGRAM EVALUATIONS
David CardJochen KluveAndrea Weber
Working Paper 21431http://www.nber.org/papers/w21431
NATIONAL BUREAU OF ECONOMIC RESEARCH1050 Massachusetts Avenue
Cambridge, MA 02138July 2015, Revised April 2017
We are extremely grateful to the editor and five referees for helpful comments on an earlier draft, and to seminar participants at IRVAPP Trento, ILO Geneva, OECD Paris, European Commission Brussels, The World Bank Washington DC, University of Oslo, ISF Oslo, MAER-Net 2015 Prague Colloquium, IFAU Uppsala. We also thank Diana Beyer, Hannah Frings and Jonas Jessen for excellent research assistance. Financial support from the Fritz Thyssen Foundation and the Leibniz Association is gratefully acknowledged. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.
NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications.
© 2015 by David Card, Jochen Kluve, and Andrea Weber. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source.
What Works? A Meta Analysis of Recent Active Labor Market Program Evaluations David Card, Jochen Kluve, and Andrea WeberNBER Working Paper No. 21431July 2015, Revised April 2017JEL No. J08,J24
ABSTRACT
We summarize the estimates from over 200 recent studies of active labor market programs. We classify the estimates by type of program and participant group, and distinguish between three different post-program time horizons. Using regression models for the estimated program effect (for studies that model the probability of employment) and for the sign and significance of the estimated effect (for all the studies in our sample) we conclude that: (1) average impacts are close to zero in the short run, but become more positive 2-3 years after completion of the program; (2) the time profile of impacts varies by type of program, with larger average gains for programs that emphasize human capital accumulation; (3) there is systematic heterogeneity across participant groups, with larger impacts for females and participants who enter from long term unemployment; (4) active labor market programs are more likely to show positive impacts in a recession.
David CardDepartment of Economics549 Evans Hall, #3880University of California, BerkeleyBerkeley, CA 94720-3880and NBERcard@econ.berkeley.edu
Jochen KluveHumboldt-University and RWI Spandauer Str. 1 10178 Berlin Germany jochen.kluve@hu-berlin.de
Andrea WeberVienna University of Economics and BusinessEconomics DepartmentWelthandelsplatz 11020 ViennaAustriaandrea.weber@wu.ac.at
1
InthelongperiodofrecoveryaftertheGreatRecessionthereisrenewed
interestinthepotentialuseofactivelabormarketpolicies(ALMPs)tohelpeaseawide
rangeoflabormarketproblems,includingyouthunemploymentandpersistent
joblessnessamongdisplacedadults(e.g.,Martin,2014).Althoughtrainingprograms,
employmentsubsidies,andsimilarpolicieshavebeeninuseforwellover50years,
credibleevidenceontheircausalimpactshasonlybecomeavailableinrecentdecades
(seeLalonde2003forabriefhistory).Withinarelativelyshortperiodoftimethe
numberofscientificevaluationshasexploded,offeringthepotentialtolearnwhattypes
ofprogramsworkbest,inwhatcircumstances,andforwhom.
InthispaperwesynthesizetherecentALMPevaluationliterature,lookingfor
systematicevidenceontheseissues.1Weextendthesampleusedinourearlieranalysis
(Card,Kluve,Weber,2010;hereafterCKW),doublingthenumberofstudies(from97to
207)andincreasingthenumberofseparateprogramestimatesfrom343to857.Many
ofthelatestALMPstudiesmeasureimpactsontheemploymentrateofparticipants,
yieldingover350estimatesforthisoutcomethatcanbereadilycomparedacross
studies.
Thisnewsampleofestimatesallowsusextendourearlierworkin4mainways.
First,wecanmorepreciselycharacterizeaverageprogramimpactsbytypeofALMPand
post-programtimehorizon.Second,weareabletocomparetherelativeefficacyof
differenttypesofALMP’s(e.g.trainingversusjobsearchassistance)fordifferent
participantgroups(e.g.,youthsversusolderworkers).Third,weprovidenewevidence
onthevariationinprogrameffectsatdifferentpointsinthebusinesscycle.Finally,we
1PreviousreviewsincludeHeckman,LalondeandSmith(1999),whosummarize75microeconometricevaluationsfromtheU.S.andothercountries,Kluve(2010),whoreviewscloseto100studiesfromEurope,andFilgesetal.(2015),whoanalyzeanarrowersetof39studies.Greenberg,MichalopoulosandRobins(2003)reviewU.S.programstargetedtodisadvantagedworkers.BergemannandvandenBerg(2008)surveyprogrameffectsbygender.IbarraránandRosas(2009)reviewprogramsinLatinAmericasupportedbytheInter-AmericanDevelopmentBank.RelatedmetaanalysesfocusingonlabormarketinterventionsinlowandmiddleincomecountriesincludeChoandHonorati(2014)andGrimmandPaffhausen(2015).
2
conductasystematicanalysisofpotentialpublicationbiasesintherecentALMP
literature.
Wesummarizetheestimatesfromdifferentstudiesintwocomplementaryways.
Ourmainapproachistoexaminetheestimatedprogrameffectsonemployment,
ignoringthefindingsfromstudiesthatmodelotheroutcomes(suchasthedurationof
timetoanunsubsidizedjob).Oursecondapproach--whichcanbeappliedtoallthe
estimatesinoursample,regardlessoftheoutcomevariable--istoclassify"signand
significance"basedonwhethertheestimatedimpactissignificantlypositive,statistically
insignificant,orsignificantlynegative.Thenarrowerfocusofthefirstapproachis
preferredinthemetaanalysisliterature(e.g.,HedgesandOlkin,1985;Robertsand
Stanley,2005;StanleyandDoucouliagos,2012),becausethemagnitudeoftheeffectis
notmechanicallyrelatedtothenumberofobservationsusedinthestudy,whereas
statisticalsignificanceis(inprinciple)sample-sizedependent.Fortunately,thetwo
approachesyieldsimilarconclusionswhenappliedtothesubsetofstudiesforwhich
employmenteffectsareavailable,givingusconfidencethatourmainfindingsare
invarianttohowwesummarizetheliterature.
Wereachfourmainsubstantiveconclusions.First,consistentwiththepattern
documentedinCKW,wefindthatALMPshaverelativelysmallaverageeffectsinthe
shortrun(lessthanayearaftertheendoftheprogram),butlargeraverageeffectsin
themediumrun(1-2yearspostprogram)andlongerrun(2+years).Acrossstudiesthat
modelimpactsonemployment,theshortrunimpactsarecenteredbetween1and3
percentagepoints(ppt.)Thedistributionofmediumruneffectsisshiftedtotheright,
centeredaround3to5ppt.,whilethelongerruneffectsarecenteredbetween5and12
ppt.Asabenchmark,thegapinemploymentratesbetweenU.S.menwithonlyahigh
schooleducationandthosewitha2or3yearcommunitycollegedegreeis10ppts.,
suggestingthata5-10ppt.longer-runimpactiseconomicallymeaningful.2
2In2015,theaveragemonthlyemploymentrateofmenoverage25withahighschooleducationintheU.S.was63.5%;theaverageformenwithanAssociatedegreewas73.8%.(U.S.DOL,2016).
3
Second,thetimeprofileofaverageimpactsinthepost-programperiodvaries
withthetypeofALMP.Jobsearchassistanceprogramsthatemphasize"workfirst"
tendtohavesimilarimpactsintheshortandlongrun,whereastrainingandprivate
sectoremploymentprogramshavelargeraverageeffectsinthemediumandlonger
runs.Publicsectoremploymentsubsidiestendtohavesmallorevennegativeaverage
impactsatallhorizons.
Third,wefindthattheaverageimpactsofALMPsvaryacrossgroups,withlarger
averageeffectsforfemalesandparticipantsdrawnfromthepooloflongterm
unemployed,andsmalleraverageeffectsforolderworkersandyouths.Wealsofind
suggestiveevidencethatcertainprogramsworkbetterforspecificsubgroupsof
participants.Jobsearchassistanceprogramsappeartoberelativelymoresuccessfulfor
disadvantagedparticipants,whereastrainingandprivatesectoremploymentsubsidies
tendtohavelargeraverageeffectsforthelongtermunemployed.Finally,comparing
therelativeefficacyofALMPsofferedatdifferentpointsinthebusinesscycle,wefind
thatprogramsinrecessionaryperiodstendtohavelargeraverageimpacts,particularly
ifthedownturnisrelativelyshort-lived.
Onthemethodologicalside,wefindthattheaverageprogrameffectsfrom
randomizedexperimentsarenotverydifferentfromtheaverageeffectsfromnon-
experimentaldesigns.Thisisreassuringgivenlongstandingconcernsoverthereliability
ofnon-experimentalmethodsforevaluatingjobtrainingandrelatedprograms(e.g.,
Ashenfelter,1987).Wealsofindthatthereissubstantialunobservedheterogeneityin
theestimatedprogramimpactsintheliterature.Thisheterogeneityislargerelativeto
thevariationattributabletosamplingerror,leadingtorelativelywidedispersioninthe
estimatedimpactsfromdesignswithsimilarprecision.Incontrasttothepatterns
uncoveredinmetaanalysesofminimumwageeffects(DoucouliagosandStanley,2009)
andtheintertemporalsubstitutionelasticity(Havranek,2015)thisdispersionisalso
nearlysymmetric.Asaresult,standardtestsforpublicationbias,whichlookfor
asymmetryinthedistributionofprogramestimates,areinsignificant.
4
II.SampleConstruction
a.SamplingImpactEvaluationStudies
WeextendthesampleinCKW,usingthesamecriteriatoselectin-scopestudies
andthesameprotocolstoextractinformationaboutprogramfeaturesandimpacts.The
CKWsamplewasderivedfromresponsestoa2007surveyofresearchersaffiliatedwith
theInstitutefortheStudyofLabor(IZA)andtheNationalBureauofEconomicResearch
(NBER)askingaboutevaluationstudieswrittenafter1995.3Toextendthissamplewe
beganbyreviewingtheresearchprofilesandhomepagesofIZAresearchfellowswitha
declaredinterestin“programevaluation”,lookingforstudieswrittensince2007.We
alsosearchedtheNBERworkingpaperdatabaseusingthesearchstrings“training”,
“active”,“publicsectoremployment”,and“searchassistance.”
InasecondstepweusedaGoogleScholarsearchtoidentifyallpapersciting
CKWortheearlierreviewbyKluve(2010).WealsosearchedthroughtheInternational
InitiativeforImpactEvaluation's"RepositoryofImpactEvaluationPublishedStudies,"
theonlineprojectlistoftheAbdulLatifJameelPovertyActionLab(J-PAL),andthelistof
LatinAmericanprogramevaluationsreviewedbyIbarraránandRosas(2009).
Afteridentifyinganinitialsampleofstudies,wereviewedthecitationsinallthe
paperstofindanyadditionalALMPstudies.Wealsoidentifiedfouradditionalpapers
presentedataconferenceinearlyfall2014.ThesearchprocesslastedfromAprilto
October2014andyielded154newstudiesthatwereconsideredforinclusioninour
ALMPimpactevaluationdatabase.
b.InclusionCriteria
3The1995startingpointwasdeterminedinpartbytheexistenceofseveralwell-knownsummariesoftheliteratureuptothemid-1990s,includingFriedlander,GreenbergandRobins(1997),Heckman,LalondeandSmith(1999),andGreenberg,MichalopoulosandRobins(2003).
5
Inordertogenerateaconsistentdatabaseacrossthetwowavesofdata
collection(2007and2014)weimposedthesamerestrictionsadoptedinCKW.First,the
program(s)analyzedintheevaluationhadtobeoneoffollowingfivetypes:
• classroomoron-the-jobtraining
• jobsearchassistance,monitoring,orsanctionsforfailingtosearch
• subsidizedprivatesectoremployment
• subsidizedpublicsectoremployment
• otherprogramscombiningtwoormoreoftheabovetypes.4
Sinceourfocusison"active"labormarketpolicies,weexcludestudiesoffinancial
incentives,suchasre-employmentbonuses(summarizedinMeyer,1995)orearnings
subsidyprograms(discussedinBlank,CardandRobins2000).Wealsoexcludeopen-
endedentitlementprogramslikechildcaresubsidies,andincludeonlyindividually
targetedemployersubsidyprograms,excludingtaxincentivesorothersubsidiesthat
areavailableforallnewlyhiredorexistingworkers.Finally,weexcludestudiesthat
reviseorupdateanolderstudyintheCKWsample,orhavesubstantialoverlapwithan
olderstudy.Methodologically,weincludeonlywell-documentedstudiesthatuse
individualmicrodataandincorporateacounterfactual/controlgroupdesignorsome
formofselectioncorrection.
Imposingthesecriteriaweretain110ofthe154studiesidentifiedinthesearch
process.5Weaddedthesetothe97studiesfromCKW,yieldingafinalsampleof207
impactevaluations.AcompletelistofthesestudiesiscontainedintheonlineData
Appendix,alongwithourentiredatabaseofprogramestimates.
Weemphasizethattheevaluationsinoursamplehavemanylimitations.At
best,thesestudiesmeasurethepartialequilibriumeffectsofALMPs,comparingthe
meanoutcomesinatreatmentgrouptothoseofanuntreatedcontrolorcomparison4Mostoftheseprogramscombineanelementofjobsearchwithtrainingorsubsidizedemployment.Wealsoinclude7estimatesofthe“threatofassignment“toaprograminthiscategory.5Themainreasonsforexclusionwere:overlapwithotherpapers(i.e.estimatingimpactsforthesameprogram);programoutofscope;andnoexplicitcounterfactualdesign.
6
group.6Evenfromthisnarrowperspectivefewstudiespresentinformationonthecosts
oftheprogram,anddetailedcost-benefitcalculationsareveryrare.Moreover,although
werestrictattentiontostudieswithacomparisongrouporselectioncorrectiondesign,
wesuspectthattheremaybesomebiasintheestimatesfromanyparticularstudy.We
donotbelieve,however,thatauthorshaveastrongincentivetochoosespecifications
thatleadtopositiveprogramestimates,sincemanywell-knownstudiesintheliterature
reportinsignificantorevennegativeimpactsforsomeprogramsorsubgroups(e.g.,
Bloometal.,2007).Thus,wedonothaveastrongpresumptionthatthebiasesinthe
literaturetendtobeone-sided.
c.ExtractingImpactEstimatesandInformationonProgramsandParticipants
Thenextstepwastoextractinformationabouttheprogramsandparticipants
analyzedineachstudy,andthecorrespondingprogramimpactestimates.7Usingthe
classificationsystemdevelopedinCKW,wegatheredinformationonthetypeofALMP,
onthetypesofparticipantsthatareadmittedtotheprogram(longtermunemployed,
regularunemploymentinsurancerecipients,ordisadvantagedindividuals8),thetypeof
dependentvariableusedtomeasuretheimpactoftheprogram,andtheeconometric
methodology.Wealsogatheredinformationonthe(approximate)datesofoperationof
theprogram,theageandgenderofparticipantsintheprogram,thesourceofthedata
usedintheevaluation(administrativerecordsoraspecializedsurvey),andthe
approximatedurationoftheprogram.
Ifastudyreportedseparateimpactestimateseitherbyprogramtypeorby
participantgroup,weidentifiedtheprogram/participantsubgroup(PPS)andcodedthe
impactestimatesseparately.Overall,wehaveinformationon526separatePPSsfrom6TheliteratureontheequilibriumeffectsofALMPisscarce.Foranotableexception,seeCreponetal.(2013).7AsinCKW,weextractedtheinformationfromthestudiesourselves,sincewefoundthatsubstantialknowledgeofevaluationmethodologyandtheALMPliteratureisoftenneededtointerpretthestudies.8Weclassifytheintakegroupas"disadvantaged"ifparticipantsareselectedfromlow-incomeorlow-labormarketattachmentindividuals.
7
the207studies,withaminimumof1andamaximumof10PPSsineachstudy.Wealso
identifieduptothreeimpactestimatesforeachPPS,correspondingtothreedifferent
post-programtimehorizons:short-term(approximatelyoneyearaftercompletionof
theprogram);mediumterm(approximatelytwoyearsafter);andlonger-term
(approximately3yearsafter).Intotal,wehave857separateprogramestimatesforthe
526program/participantsubgroups,withbetweenoneandthreeestimatesoftheeffect
oftheprogramatdifferenttimehorizons.9
Weusetwocomplementaryapproachestoquantifytheestimatedprogram
impacts.First,weclassifytheestimatesassignificantlypositive,insignificantlydifferent
fromzero,orsignificantlynegative(atthe5%level).Thismeasureofeffectivenessis
availableforeveryestimateinourdatabase.Forthesubsetofstudiesthatmeasure
effectsontheprobabilityofemployment,wealsoextractanestimateoftheprogram
effectontheemploymentrateofparticipants.10
Thefinalstepinourdataassemblyprocedurewastoaddinformationonlabor
marketconditionsatthetimeofoperationoftheprogram.Specifically,wegathered
informationonGDPgrowthratesandunemploymentratesfromtheOECD,theWorld
Bank,andtheILO.Forourmainanalysiswefocusonhowprogrameffectivenessis
relatedtotheaveragegrowthrateandtheaverageunemploymentrateduringthe
periodtheprogramgroupparticipatedintheALMP,thoughwealsolookattheeffectof
conditionsinthepost-programperiod.
9ForaspecificPPSandtimehorizonwetrytoidentifyandcodethemainestimateinthestudy.WedonotincludemultipleestimatesforthesamePPSandtimehorizon.10Wealsoextracttheaverageemploymentrateofthecomparisongroup,andforsomeanalysiswemodeltheprogrameffectdividedbyeitherthecomparisongroupemploymentrateorthestandarddeviationofthecomparisongroupemploymentrate.
8
III.DescriptiveOverview
a.ProgramTypes,ParticipantCharacteristics,EvaluationDesign
Table1presentsanoverviewoftheprogramestimatesinourfinalsample.As
noted,wehaveatotalof857differentimpactestimatesfor526differentPPSs
(program-type/participantsubgroupcombinations)extractedfrom207separate
studies.Todealwithpotentialcorrelationsbetweentheprogramestimatesfroma
givenstudy--arisingforexamplefromidiosyncraticfeaturesoftheevaluation
methodology--wecalculatestandarderrorsclusteringbystudy.
Column1presentsthecharacteristicsofouroverallsample,whilecolumns2-6
summarizetheestimatesfromfivecountrygroups:theGermaniccountries(Austria,
GermanyandSwitzerland),whichaccountforaboutonequarterofallstudies;the
Nordiccountries(Denmark,Finland,NorwayandSweden),whichaccountforanother
quarterofstudies;theAnglocountries(Australia,Canada,NewZealand,U.K.andU.S.),
whichaccountforjustover10%ofstudies;andtwonon-mutuallyexclusivegroupsof
lower/middleincomecountries--"non-OECD"countries(10%ofstudies),andLatin
AmericanandCaribbean(LAC)countries(10%ofstudies).AppendixFigure1showsthe
numbersofestimatesbycountry.ThelargestsourcecountriesareGermany(253
estimates),Denmark(115estimates),Sweden(66estimates),theU.S.(57estimates)
andFrance(42estimates).
ThesecondpanelofTable1showsthedistributionofprogramtypesinour
sample.Trainingprograms(includingclassroomandon-the-jobtraining)accountfor
aboutonehalfoftheprogramestimates,withbiggersharesinthenon-OECDandLAC
countries.Publicsectoremploymentprograms,bycomparison,arerelativelyrare
amongrecentevaluations,whilejobsearchassistance(JSA)programs,private
employmentsubsidiesandother/combinedprogramseachrepresentabout15%ofthe
estimates.11
11TheJSAcategoryincludesasmallnumberofevaluations(withatotalof8programestimates)forprogramsthatmonitorsearchactivityandthreatensanctionsforlowsearcheffort.Wecombinethese
9
Thenextthreepanelsofthetableshowthecharacteristicsoftheprogram
participants,classifiedbyagegroup,gender,and"type"ofparticipant.Aboutone-half
oftheestimatesareformixedageandmixedgendergroups,butwealsohaverelatively
largesubsetsofestimatesthatarespecifictoeitheryoungerorolderworkers,or
femalesormales.Sixty-fivepercentoftheprogramestimates(andnearlyallthe
estimatesfromtheGermaniccountries)areforparticipantswhoenterfromthe
unemploymentinsurance(UI)system.Typicallytheseparticipantsareassignedtoa
programandrequiredtoattendasaconditionforcontinuingbenefiteligibility.12The
remaining35%ofestimatesaresplitbetweenprogramsthatservethelongterm
unemployed(LTU)andthosethatservedisadvantagedparticipantgroups.Inmany
cases,thesegroupsarerecruitedbyprogramoperatorsandenrollvoluntarily.Such
voluntaryprogramsaremorecommonintheAngloSaxoncountriesandinless
developedcountriesthatlackaformalUIsystem.13
Nextweshowtheoutcomevariablesusedtomeasuretheprogramimpactand
thetimehorizonsoftheestimate.Themostcommonoutcome–particularlyinthe
Germanicandnon-OECDcountries–istheprobabilityofemployment,whilethelevelof
earningsisthemostcommonmetricintheAngloSaxoncountries.Aboutonesixthof
theprogramestimates–but40%ofthosefromNordiccountries–measuretheexitrate
fromthebenefitsystem,typicallyfocusingontherateofexittoanew(unsubsidized)
job.Finally,asmallsubsetofestimates–mostlyfromAngloSaxoncountries–focuson
theprobabilityofunemployment.Aboutonehalfoftheestimatesareforashortterm
horizon(<1year)afterprogramcompletion,35%foramediumterm(1-2years),and
16%foralongerterm(morethan2yearafter).
withJSAprogramsbecausebothtypesofprogramshavesimilarincentiveeffectsonparticipants'searchactivity.12ThistypeofprogramrequirementiswidespreadinEurope--seeSianesi(2004)foradiscussion.13TheU.S.jobtrainingprogramsanalyzedintheseminalpapersofAshenfelter(1978),AshenfelterandCard(1985),Lalonde(1986),Heckman,Ichimura,Smith,andTodd(1998)areallofthistype.
10
ThelastrowoftheTableshowsthefractionofprogramestimatesthatarebased
onanexperimentaldesign.Inmostofourcountrygroupsabout30%ofestimatescome
fromrandomizedcontrolledtrials(RCTs)thathavebeenexplicitlydesignedtomeasure
theeffectivenessoftheALMPofinterest.AnimportantexceptionistheGermanic
countries,wherenoexperimentallybasedestimatesareyetavailable.
Thedistributionofprogramestimatesovertime(definingtimebytheearliest
intakeyearoftheprogram)isshowninFigure1,withseparatecountsforexperimental
andnon-experimentalestimates.Oursampleincludesprogramsfromasfarbackas
1980,thoughthemajorityofestimatesarefromthe1990sandearly2000s,reflecting
ourfocusonstudieswrittensince1995.Thereisclearevidenceofatrendtoward
increasinguseofexperimentaldesigns:amongthe210estimatesfrom2004andlater,
61%arefromrandomizeddesigns.
b.MeasuresofProgramImpact-Overview
Table2givesanoverviewofourtwomainmeasuresofprogramimpact,
contrastingresultsfortheshortterm,mediumterm,andlongterm.Columnone
summarizesthesignandsignificanceofalltheavailableprogramestimates.Amongthe
415shorttermestimates,40%aresignificantlypositive,42%areinsignificant,and18%
aresignificantlynegative.Thepatternofresultsismorepositiveinthemediumand
longerterms,withamajorityofestimates(52%)beingsignificantlypositiveinthe
mediumterm,and61%beingsignificantlypositiveinthelongerterm.
Column2showsthedistributionofsignandsignificanceforthesubsetofstudies
thatusepost-programemploymentratestoevaluatetheALMPprogram.These111
studiesaccountfor490programestimates(57%ofourtotalsample).Theshortterm
programestimatesfromthissubsetofstudiesaresomewhatlesspositivethaninthe
overallsample.Inthemediumandlongerterms,however,thediscrepancydisappears.
Asdiscussedbelow,thesepatternsarenotexplainedbydifferencesinthetypesof
ALMPprogramsanalyzedindifferentstudies,orbydifferencesinparticipant
11
characteristics.Instead,theyreflectatendencyforstudiesbasedonmodelsofthetime
tounemploymentexit(whichareincludedincolumn1butexcludedincolumn2)to
exhibitmorepositiveshorttermimpactsthanstudiesbasedonemployment.
Column3ofTable2showsthedistributionsofsignandsignificanceassociated
withtheestimatedemploymenteffectswherewecanextractbothanactualprogram
effect(typicallythecoefficientfromalinearprobabilitymodel)andtheemployment
rateofthecomparisongroup.14Thedistributionsareverysimilartothoseincolumn2,
suggestingthatthereisnosystematicbiasassociatedwiththeavailabilityofanimpact
effectandthecomparisongroupemploymentrate.
Finally,columns4and5reportthemeanandmedianofthedistributionsof
estimatedprogrameffectsforthesubsampleincolumn3.Theshortrunprogram
effectsarecenteredjustabovezero,withameanandmedianof1.6ppt.and1.0ppt.,
respectively.Inthemediumtermthedistributionshiftsrightbutalsobecomesslightly
moreasymmetric,withameanandmedianof5.4and3.0ppt.,respectively.Inthelong
termthereisafurthershiftright,particularlyintheupperhalfofthedistribution,witha
meanandmedianof8.7ppt.and4.9ppt.,respectively.
Positiveskewinthedistributionofestimatedeffectsisofteninterpretedinthe
metaanalysisliteratureasevidenceof"publicationbias",particularlyifthepositive
effectsareimpreciselyestimated(seee.g.,StanleyandDoucouliagos,2012).Some
insightintothisissueisofferedbythe"forestplots"inFigures2a,2b,and2c,which
showthecumulativedistributionsofprogramestimatesateachtimehorizon,along
withbandsrepresentingthestandarderrorsoftheestimates.15
Inspectionofthesegraphsconfirmsthattheoveralldistributionofprogram
effectsshiftstotherightasthetimehorizonisextended.Atallthreehorizonsthereis14Inmanycasesastudyreportstheimpactontheemploymentrateoftheprogramgroupbutdoesnotreporttheemploymentrateofthecomparisongroup.Asdiscussedbelow,weneedthelatternumbertoconstructeffectsizesorproportionalimpactsontheemploymentrate.15Thedistributionsarelimitedtoestimatesforwhichwealsohaveanestimateoftheassociatedstandarderror.InformationonthestandarderrorsofprogramestimateswasnotextractedinCKW.Thus,theestimatesarefromthelateststudiescollectedinoursecondround.
12
alsosomepositiveskewinthedistributionofestimatedeffects.Interestingly,however,
theconfidenceintervalsdonotappeartobesystematicallywiderforestimatesinthe
upperorlowertailsofthedistribution.Instead,thereareahandfulofpositiveoutliers
intheshortandmediumtermdistributionsthatpushtheunweightedmeanabovethe
medianandtheprecision-weightedmean.
ReturningtoTable2,columns4and5alsoshowthemeanandmedianprogram
effectsforestimatesthatareclassifiedassignificantlypositive,insignificant,or
significantlynegative.Aswouldbeexpectedifdifferencesinsignandsignificanceare
mainlydrivenbydifferencesinthemagnitudeoftheprogramestimates–ratherthanby
differencesinthestandarderrorsoftheestimates–themeanandmedianarelargeand
positiveforsignificantpositiveestimates,largeandnegativeforsignificantnegative
estimates,andclosetozeroforinsignificantestimates.Thispatternisillustratedin
AppendixFigures2a,2b,and2c,whereweplotthehistogramsofestimatedeffectsat
eachtimehorizon,separatingtheestimatesbycategoryofsignandsignificance.Atall
threetimehorizons,thesubgroupsofestimatesappeartobedrawnfromdistributions
thatarecenteredondifferentmidpoints.Thisseparationsuggeststhatthesignand
significanceofanestimatecanserveasnoisyindicatoroftheunderlyingeffect.
c.VariationinAverageProgramImpacts
Tables3aand3bprovideafirstlookatthequestionofhowaverageALMP
impactsvaryacrossdifferenttypesofprogramsanddifferentparticipantgroups.For
eachsubsetofestimatesweshowthemeanprogrameffectsateachtimehorizonand
thecorrespondingfractionofprogramestimatesthatissignificantlypositive.
Focusingfirstoncomparisonsacrossprogramtypes(Table3a),noticethat
trainingandprivatesectoremploymentprogramstendtohavesmallaverageeffectsin
theshortrun,coupledwithmorepositiveaverageimpactsinthemediumandlonger
runs.Incontrast,JSAprogramsandALMPsinthe"other"categoryhavemorestable
impacts.Theseprofilesareconsistentwiththenatureofthetwobroadgroupsof
13
programs.Participantsintrainingandprivatesubsidyprogramsoftensuspendtheir
normaljobsearcheffortsanddevotetheirtimetoprogramactivities--aso-called"lock-
in"effectthattypicallyleadstoworseoutcomesintheimmediatepost-programperiod
(seee.g.,HamandLalonde,1996).16Assumingthatinvestmentsmadeduringthe
programperiodarevaluable,however,theoutcomesofparticipantswillgraduallycatch
upwiththoseofthecomparisongroup.17Bycomparison,JSAprogramsandother
programsthatincludemonitoringofsearcharedesignedtopushparticipantsintothe
labormarketquickly,withlittleornoinvestmentcomponent.Intheabsenceoflarge
returnstorecentjobexperience,itisunlikelythattheseprogramscanhavelargelong
runeffects.18
AnotherclearfindinginTable3aistherelativelypoorperformanceofpublic
sectoremploymentprograms–aresultthathasbeenfoundinotherpreviousanalyses
(e.g.,Heckmanetal.,1999,andCKW).
AppendixFigure3ashowshowtherelativeshareofdifferenttypesofALMPs
havechangedoverthe30yearperiodcoveredbyoursample.Thesharesoftrainingand
JSAprogramsisrelativelystable,whiletheshareofpublicsectoremploymentprograms
hasfallensharply,perhapsreflectingthemorenegativeevaluationresultsthatthese
programshaveoftenreceived.
AppendixFigure3bshowsthevariationovertimeinourtwomeasuresof
programimpact.Overall,thesignandsignificanceclassificationsofshortterm,medium
termandlongtermestimatesarequitestableovertime,withlittleindicationthatmore
recentprogramsaremoreorlesslikelytoshowsignificantpositiveresults.Thereis
16IncaseswheretheprogramgroupisdrawnfromtheregularUIsystem,participantsintrainingandsubsidizedemploymentopportunitiesareoftenexemptfromsearchrequirementsthatareimposedonnon-participants--seee.g.Biewenetal.(2014)foradiscussionintheGermancontext.17AsnotedbyMincer(1974)asimilarcross-overpatternisobservedinthecomparisonofearningsprofilesofhighschoolgraduatesandcollegegraduates.18Evidenceonthevalueoflabormarketexperienceforlowerskilledworkers(GladdenandTaber,2000;CardandHyslop,2005)suggeststhatthereturnsaremodestandunlikelytoexceed2or3percentperyearofwork.
14
morevariabilityinthemeanimpactsontheprobabilityofemployment,withsome
evidenceofanupwardtrend,particularlyfortheshortandmediumtermimpacts.
ThemiddlerowsofTable3bcomparethedistributionsofprogrameffectsby
participantagegroupandgender.TheresultsforPPSswhichincludeallagegroupsare
quitesimilartotheresultsfortheoverallsample,whiletheresultsforyouth
participantsshowamixedpattern,withrelativelysmallaverageprogrameffectson
employmentatalltimehorizons(columns4-6),butmoreevidenceofpositivelong-run
impactsbasedonsignandsignificance(columns7-9).Thedifferencesacrossgender
groupsaremoresystematicandindicatethataverageestimatedprogrameffectsare
slightlylargeratalltimehorizonsforfemales(columns4-6)andhaveahigher
probabilityofbeingsignificantlypositive(columns7-9).
Finally,thebottomrowsofTable3bcontractsresultsfromevaluationsbasedon
randomizeddesignsandnon-experimentaldesigns.Thecomparisonsofmeaneffects
suggestthatexperimentallybasedestimatestendtobelargerintheshortrunand
declineovertime,whereasnon-experimentallybasedestimatestendtobecomelarger
(morepositive)overtime.Wecautionthatthesesimple"oneway"contrastsmustbe
interpretedcarefully,however,becausetherearemultiplesourcesofpotential
heterogeneityintheprogramimpacts.Forexample,manyoftheexperimental
evaluationsfocusonJSAprograms,whereasmanyofthenon-experimentalevaluations
focusontrainingprograms.ThemetaanalysismodelsinSectionIVdirectlyaddressthis
issueusingamultivariateregressionapproach.
d.ProfileofPost-ProgramImpacts
Simplecomparisonsacrosstheimpactestimatesinoursamplesuggestthat
ALMPshavemorepositiveaverageeffectsinthemediumandlongerterms.Toverify
thatthisisactuallytrueforagivenprogramandparticipantsubgroup–andisnot
simplyanartifactofheterogeneityacrossstudies–weexaminethewithin-PPS
evolutionofimpactestimatesinTable4.
15
Columns1-3showthechangesinestimatedprogrameffectsontheprobability
ofemploymentforthesubsetofstudiesforwhichweobservebothshortandmedium
termestimates,mediumandlongtermestimates,andshortandlongtermestimates,
respectively.Consistentwiththesimplecross-sectionalcomparisons,thewithin-PPS
effectstendtoincreaseasthetimehorizonisextendedfromtheshortruntothe
mediumrun,orfromtheshortruntothelongrun.Theaveragechangebetweenthe
mediumandlongerrunsisessentiallyzero.
Comparingacrossprogramtypesitisclearthatthepatternofrisingimpactsis
drivenbytrainingprograms,whichshowarelativelylargegaininestimatedprogram
effectsfromtheshorttermtothemediumterm.Thepatternsfortheothertypesof
programssuggestrelativelyconstantordecliningaverageprogrameffectsoverthe
post-programtimehorizon.Inparticular,incontrasttothepatternsinTable3a,thereis
noindicationofariseinimpactsforprivateemploymentsubsidyprogramsovertime,
suggestingthatthegainsinTable3amaybedrivenbyheterogeneitybetweenstudies.
Wereturntothispointbelow.
Incolumns4-6weexaminethewithin-studychangesinsignandsignificancefor
abroadersetofstudies.Here,weassignavalueof+1toPPSestimatesthatchange
frominsignificanttosignificantlypositiveorfromsignificantlynegativetoinsignificant;
-1toestimatesthatchangefromsignificantlypositivetoinsignificantorfrom
insignificanttosignificantlynegative;and0toestimatesthathavethesame
classificationovertime.Thissimplesummarypointstosimilarconclusionsasthe
changesinestimatedprogrameffects,thoughJSAprogramsshowmoreevidenceofa
riseinimpactsfromtheshort-runtothemediumrunincolumn4thanthecomparison
ofestimatedeffectsontheprobabilityofemploymentincolumn1.
AppendixTables1aand1bpresentfullcross-tabulationsofsign/significanceat
thevariouspost-programtimehorizons.Assuggestedbythesimpleclassification
systemusedinTable4,mostprogramestimateseitherremaininthesamecategory,or
becomemorepositiveovertime.
16
IV.MetaAnalyticModelsofProgramImpacts
a.ConceptualFramework
ConsideranALMPevaluationthatmodelsanoutcomeyobservedformembers
ofbothaparticipantgroupandacomparisongroup.Letbrepresenttheestimated
impactoftheprogramontheoutcomesoftheparticipantsfromagivenevaluation
design,andletβrepresenttheprobabilitylimitofb(i.e.,theestimatethatwouldbe
obtainedifthesamplesizefortheevaluationwereinfinite).Understandardconditions
theestimatebwillbeapproximatelynormallydistributedwithmeanβandsomelevel
ofprecisionPthatdependsonboththesamplesizefortheevaluationandthedesign
featuresofthestudy.19Thereforewecanwrite:
b=β+P─1/2z, (1)
wherezisarealizationfromadistributionthatwillbeclosetoN(0,1)ifthesamplesize
islargeenough.ThetermP─1/2zhastheinterpretationoftherealizedsamplingerror
thatisincorporatedinb.
Assumethatthelimitingprogrameffectassociatedwithagivenstudy(β)canbe
decomposedas:
β=Xα+ε. (2)
whereαisavectorofcoefficientsandXcapturestheobservedsourcesof
heterogeneityinβ,arisingforexamplefromdifferencesinthetypeofprogramorthe
genderorageoftheprogramparticipants.Thetermεrepresentsfundamental
heterogeneityinthelimitingprogrameffectarisingfromtheparticularwayaprogram
wasimplemented,orspecificfeaturesoftheprogramoritsparticipants,orthenature
ofthelabormarketenvironment.
19Forexample,inanexperimentwith50%ofthesampleinthetreatmentgroupandnoaddedcovariates,P=N/[2σ2(1+δ2)],whereNisthesamplesize,σisthestandarddeviationoftheoutcomeyinthecontrolgroup,andδσisthestandarddeviationoftheoutcomefortheprogramgroup.Inmorecomplexdesignssuchasdifferenceindifferencesorinstrumentalvariablestheprecisionwillbesmaller.
17
Equations(1)and(2)leadtoamodelfortheobservedprogramestimatesofthe
form:
b=Xα+u, (3)
wheretheerroru=ε+P─1/2zincludesboththesamplingerrorintheestimateband
theunobserveddeterminantsofthelimitingprogrameffectforagivenstudy.Weuse
simpleregressionmodelsbasedonequation(3)toanalyzetheprogrameffectsonthe
probabilityofemploymentthatareavailableinoursample.Weinterpretthesemodels
asprovidingdescriptivesummariesofthevariationinaverageprogrameffectswith
differencesintheobservedcharacteristicsofagivenprogramandparticipantgroupin
oursample.Recognizingthestructureoftheerrorcomponentin(3)wepreferOLS
estimation,whichweightseachestimatedprogrameffectequally,ratherthanprecision-
weighedestimation,whichwouldbeefficientundertheassumptionthatε=0.20Aswe
showbelow,incontrastto“classical”metaanalysissettingswhereeachestimateis
basedonaclinicaltrialofthesamedrug,thevariationinεappearstobeparticularly
largeforALMP’s,reflectingthewiderangeoffactorsthatcanpotentiallycausea
programtobemoreorlesssuccessful.
Forourfullsampleofprogramestimatesweuse(unweighted)orderedprobit
(OP)modelsforthe3-wayclassificationofsignandsignificanceofeachestimate.Note
thatthet-statisticassociatedwiththeestimatedimpactbisjusttheratioofthe
estimatetothesquarerootofitsestimatedsamplingvariance(whichistheinverseof
itsestimatedprecision).Usingequation(3),wecanthereforewrite:
t=P1/2b
=P1/2Xα+z+P1/2ε.
IftheprecisionPoftheestimatedprogrameffectsisconstantacrossstudiesandthere
arenounobserveddeterminantsofthelimitingprogrameffect(i.e.,ε=0)thet-statistic
willbenormallydistributedwithmeanXα'whereα'=P1/2α.Inthiscasethe
20SeeSolon,HaiderandWooldridge(2015)foradiscussionofweighting.
18
coefficientsfromanOPmodelforwhetherthetstatisticislessthan-2,between-2and
2,orgreaterthan2(i.e.,thesignandsignificanceoftheestimatedprogrameffects)will
bestrictlyproportionaltothecoefficientsobtainedfromaregressionmodelofthe
correspondingestimatedprogrameffects.
Inoursampletheestimatedprecisionoftheprogramestimatesvarieswidely
acrossstudies,andthereisclearlyunobservedheterogeneityintheimpacts.21
Surprisingly,however,forstudiesthatexaminetheprobabilityofemploymentasan
outcometheestimatedcoefficientsfromOLSmodelsbasedonequation(3)andOP
modelsforsign/significanceareverynearlyproportional,suggestingthatthesame
observablefactorsthattendtoraisetheestimatedprogrameffectsalsotendtoleadto
morepositivetstatistics.Ourinterpretationofthispatternisthatthesamplingerror
componentoftheprogramestimatesissmallrelativetothevariationduetoobserved
andunobservedheterogeneity,sothet-statisticvariesacrossstudiesinproportionto
therelativemagnitudeoftheestimatedprogrameffect.WethereforeusetheOP
modelstosummarizethebroadersetofprogramestimates.
b.BasicModelsforProgramEffectandforSignandSignificance
Table5presentstheestimatesfromaseriesofregressionmodelsforoursample
ofestimatedprogrameffectsontheprobabilityofemployment.Wepooltheeffectsfor
differentpost-programhorizonsandincludedummiesindicatingwhethertheestimate
isforthemediumorlongterm(withshorttermestimatesintheomittedgroup).The
basicmodelincolumn1includesonlythesecontrolsandasetofdummiesforthetype
ofprogram(withtrainingprogramsintheomittedcategory).Consistentwiththesimple
comparisonsinTable3a,wefindthattheprogramestimatesarelargerinthemedium
21AscanbeseenfromthevaryingwidthsoftheconfidenceintervalsinFigures2a,2band2c,theprecisionoftheestimatedprogrameffectsvarieswidelyacrossstudiesinoursample.Theprecisionisessentiallyuncorrelatedwiththesamplesizeoftheevaluation(correlation=─0.02),suggestingthatstudieswithlargersamplesizeshavemorecomplexeconometricdesignsthatoffsetanypotentialgainsinprecision.
19
andlongrun,andthatpublicsectoremploymentprogramsareassociatedwithsmaller
programeffects.
Themodelincolumn2introducesadditionalcontrolsforthetypeofparticipant
andstudycharacteristics,whicharereportedinTable7anddiscussedbelow.These
controlsslightlyattenuatethegrowthinprogrameffectsoverlongerpost-program
horizonsandalsoreducethemagnitudeoftheJSAprogrameffectfrom-3.2ppts.(and
significant)to-0.1ppts(andinsignificant).
Columns3-5introduceaparallelsetofmodelsthatallowthetimeprofilesof
post-programimpactstovarywiththetypeofprogram.Inthesespecificationsthe
"maineffects"foreachprogramtypeshowtheshorttermimpactsrelativetotraining
programs(theomittedtype),whiletheinteractionsofprogramtypewithmediumterm
andlongtermdummiesshowhowtheimpactsevolverelativetotheprofilefortraining
programs(whicharesummarizedbythemaineffectsinthefirsttworows).Wepresent
modelswithandwithoutadditionalcontrolsincolumns3and4,andamodelwith
dummyvariablesforeachparticipant/programsubgroupincolumn5.Inthelatter
specificationthe"maineffects"forthetypeofprogramareabsorbedbythePPSfixed
effects,butwecanstillestimatethecoefficientsformediumandlongtermeffects--
whichnowmeasuretheevolutionoftheprogrameffectsforthesamePPSover
differenttimehorizons--aswellasinteractionsofthetimehorizondummieswiththe
typeofprogram.
Threekeyconclusionsemergefromthesemoreflexiblespecifications.First,as
suggestedbythepatternsinTable4,theprogramimpactsfortrainingprogramstendto
riseovertime,whiletheeffectsforjobsearchassistanceprogramsandotherprograms
(whichareobtainedbyaddingtheprogram-type/timehorizoninteractionstothetime
horizoneffectsinrows1and2)areroughlyconstant.22Second,inthemodelswithout
22Toaidintheinterpretationoftheinteractedestimates,AppendixTable4presentstheimpliedmeanprogrameffectsbyprogramtypeandtimehorizonforthemodelsincolumns3and4,andtheassociatedstandarderrors.Notethatbyconstructionthemodelincolumn3reproducesthemeansreportedin
20
PPSfixedeffects(columns3and4)theimpliedprofileofimpactsforprivatesector
employmentprogramsisrelativelysimilartotheprofilefortrainingprograms.When
thePPSeffectsareadded,however,theinteractionsbetweenprivatesectorprograms
andbothmediumtermandlongtermhorizonbecomerelativelylargeandnegative--
similartotheinteractioneffectsforJSAandotherprograms.Athirdconclusionisthat
publicsectoremploymentprogramsappeartoberelativelyineffectiveatalltime
horizons.
WehavealsoestimatedmodelssimilartothespecificationsinTable5,butusing
twoalternativemeasuresofprogramimpacts:theestimated"effectsize"(the
estimatedeffectontheemploymentrateofparticipantsdividedbythestandard
deviationofemploymentratesinthecomparisongroup),andtheproportionalprogram
effect(theestimatedeffectforparticipantsdividedbythemeanemploymentrateof
thecomparisongroup).ThesespecificationsarereportedinAppendixTables2aand2b,
respectively,andyieldverysimilarconclusionstothemodelsinTable5.Essentially,
thesealternativechoicesleadtorescalingofthecoefficientsofthemetaanalysis
modelswithverysmallchangesintherelativemagnitudesofdifferentcoefficients.
AlimitationoftheanalysisinTable5isthatestimatedprogrameffectsareonly
availablefor40%ofouroverallsample.Tosupplementthesemodelsweturnto
orderedprobitmodelsforsignandsignificance.Thefirst4columnsofTable6presenta
seriesofOPmodelsthatareparalleltothoseinTable5,butfittoouroverallsampleof
programestimates.Thespecificationsincolumns1and3havenocontrolsotherthan
dummiesformediumandlongtermhorizonsandthetypeofALMP--inthelattercase
interactingthetypeofprogramwiththetimehorizondummies.Columns2and4
reportexpandedspecificationsthataddthecontrolvariablesreportedinTable7.
Column5ofTable6repeatsthespecificationfromcolumn4,butfittothesubsampleof
columns4-6ofTable3a.Forthespecificationincolumn4ofTable5wenormalizethecovariatestohavemean0andfitthemodelwithoutanintercept:thusthemeanprogrameffectsareinterpretedasmeansforaprogramandparticipantgroupwiththemeancharacteristicsofoursample.
21
352programestimatesforwhichwehaveanestimateoftheprogrameffectonthe
probabilityofemployment.Themodelincolumn6reproducesthespecificationin
column3,butaddingPPSfixedeffects.AsinthemodelinthelastcolumnofTable5,
thecoefficientsinthisspecificationmeasuretheevolutionofthefactorsdetermining
signandsignificancewithinagivenPPS.Finally,column7presentsestimatesfroma
linearregressionreplacingthecategoricaloutcomevariablewithvaluesof-1,0,and+1,
alsoincludingPPSfixedeffects.
TheOPmodelsinTable6yieldcoefficientsthatareveryhighlycorrelatedwith
thecorrespondingcoefficientsfromtheprogrameffectmodelsinTable5,butroughly
10timesbiggerinmagnitude.Forexample,thecorrelationofthe14coefficientsfrom
thespecificationincolumn4ofTable6withcorrespondingcoefficientsfromthe
specificationincolumn4ofTable5is0.84.23InparticulartheOPmodelsconfirmthat
theimpactsofjobsearchassistanceandotherprogramstendtofaderelativetothe
impactsoftrainingprograms,andthatpublicsectoremploymentprogramsare
relativelyineffectiveatalltimehorizons,regardlessofhowtheoutcomesaremeasured
intheevaluation.24
ThemodelsincludingPPSfixedeffectsincolumns6(orderedprobit)and7
(linearregression)ofTable6alsoimplythesamequalitativefindings,indicatingthat
withinagivenPPStheestimatedprogrameffectbecomesmorepositiveinthelonger
run.Thecoefficientsofthelinearregressionmodelarescaledbyafactorof
approximatelyfiverelativetotheorderedprobitspecification.
c.ParticipantandStudyCharacteristicsintheBasicModels
23Theregressionmodelis:OP-coefficient=-0.02+10.57×Effect-coefficient,R-squared=0.70.24Wealsofittwosimplerprobitmodelsfortheeventsofreportingapositiveandsignificantornegativeandsignificantestimate,reportedinAppendixTable3.Aswouldbeexpectediftheorderedprobitspecificationiscorrect,thecoefficientsfromthemodelforasignificantlypositiveeffectarequiteclosetotheOPcoefficients,whilethecoefficientsfromthemodelforasignificantlynegativeeffectarecloseinmagnitudebutoppositeinsign.
22
Theestimatedcoefficientsfortheextracontrolvariablesincludedinthemodels
incolumns2,4ofTable5andcolumns2,4,and5ofTable6arereportedinTable7.The
coefficientestimatesfromthetwomodelsfortheeffectsontheprobabilityof
employment(columns1,2)arequitesimilarandsuggestthattheimpactofALMPsvaries
systematicallywiththetypeofparticipant(withlargereffectsforthelongterm
unemployed),theiragegroup(morenegativeimpactsforolderandyounger
participants),andtheirgender(largereffectsforfemales).Theestimatedprogram
effectsarealsosomewhatlargerforstudiesestimatedonGerman,AustrianorSwiss
data,buttherearenolargeorsignificantdifferencesacrosstheothercountrygroups.
ThecoefficientsfromtheOPmodels(columns3-4)confirmmostofthese
conclusionsaboutthedifferentialimpactsofALMPsacrossdifferentparticipantgroups
anddifferentcountries.25Inparticular,theOLSmodelsfortheprogrameffectsonthe
probabilityofemploymentandtheOPmodelsforsignandsignificanceshowsmaller
impactsforyoungparticipantsandolderparticipants,relativetotheimpactsonmixed
agegroups,andlargerimpactsforlong-termunemployedparticipants.TheOPmodels
fittotheoverallsample(columns3,4)alsopointtoalargerpositiveimpactfor
disadvantagedparticipantsrelativetoUIrecipients,whereastheprogrameffectmodels
andtheOPmodelsfittotheprogrameffectsubsample(column5)yieldaninsignificant
coefficient,arguablyduetothesmallnumberofstudiesthatfocusonthisgroup.26
25Thecorrelationbetweenthecoefficientsincolumns2and4ofTable7is0.69.26ApotentiallyrelevantdimensionofprogrameffectivenessconcernsthetimeALMPparticipantshavespentinunemploymentbeforeenteringtheprogram.Biewenetal.(2014)investigatethisissueusingthreestratatoestimatetreatmenteffects:1-3months,4-6months,and7-12monthsofunemployment,respectively.Forlongertrainingprograms(meandurationof226days)theestimatedshort-termimpactsandstandarderrorsforthethreestrataare0.00(0.04),0.00(0.04),0.08(0.40)formales,and0.06(0.03),-0.01(0.045),-0.04(0.02)forfemales;medium-termimpactsare0.05(0.03),0.07(0.04),0.09(0.045)formales,and0.06(0.03),0.11(0.05),0.09(045)forfemales.Theseresultsshowsomeindicationthatprogramparticipantsfromstratawithlongerelapsedunemploymentdurationsbenefitmorethanothergroups.Thisresultisinlinewiththefindingsfromourstratificationofprogramintakegroupintoshort-termunemployed(“UIrecipients”),long-termunemployed,andparticipantswithoutbenefitentitlement(“disadvantaged”).Amoredetailedinvestigationofthisissuewithinthemeta-analysisframeworkisnotpossible,sincetheprimarystudiesrarelyreportinformationonelapsedtimeinunemploymentbeforeprogramstart.
23
OnenotabledifferencebetweentheprogrameffectmodelsandtheOPmodels
concernstherelativeimpactofALMPsonfemaleparticipants.IntheOLSprogrameffect
modelstheestimatedcoefficientsforfemaleparticipantsarearound0.04to0.05in
magnitude,andstatisticallysignificantatconventionallevels(withtstatisticsaround2).
IntheOPmodels,bycomparison,thecorrespondingcoefficientsarerelativelysmallin
magnitudeandfarfromsignificant.Furtherinvestigationrevealsthatthisdivergenceis
drivenbytheuppertailofprogrameffectestimatesforfemaleparticipants(see
AppendixFigure4),andinparticularbytherelativelylargeestimatedeffectsforfemale
PPSsthatshowasignificantpositiveeffect.27Thisuppertaildoesnotappeartobe
drivenbyafewoutliers,butinsteadreflectsasystematicallyhigherprobabilityof
estimatingalargepositiveeffectwhentheparticipantgroupislimitedtofemales.28
AninterestingaspectoftheOPmodelsisthepatternofcoefficientsassociated
withthechoiceofdependentvariable,reportedinthetoprowsofTable7.These
coefficientsshowthatstudiesmodelingthehazardrateofexitingthebenefitsystemor
theprobabilityofunemploymentaresignificantlymorelikelytoreportpositivefindings
thanstudiesmodelingemployment(theomittedcategory)orearnings.29
ThemodelssummarizedinTable7alsocontrolforthedurationoftheprogram,
usingasimpledummyvariableforwhethertheprogramlastedlongerthan9months.
Programdurationcanbeseenasaroughproxyforthecostoftheprogram,as
informationoncosteffectivenessisverysparseinmostofthesurveyedstudies.Our
estimatesdonotindicateanysystematicadvantageforlongerprograms--indeed
acrossallthespecificationsthecoefficientisnegative,thoughstatisticallyinsignificant.
27Themedianand75thpercentilesoftheeffectsizedistributionforfemaleparticipantgroups,conditionalonapositiveimpact,are0.25,and0.46respectively.Bycomparison,thecorrespondingstatisticsforthecombinedmaleandmixedgenderparticipantgroupsare0.15,and0.27.28Wealsoestimatedseparateprogrameffectmodelsfordifferenttypesofparticipants--thosefromtheregularUIsystemversuslongtermunemployedordisadvantagedgroups.WefoundasignificantpositivecoefficientforfemaleparticipantsinthemodelsforbothUIrecipientsandthelongtermunemployed.29Estimatesfrominteractedmodelsthatallowdifferenteffectsofthedependentvariableatdifferenttimehorizons(notreportedinthetable)showthatthepositivecoefficientassociatedwiththeuseofexithazardsislargelyconfinedtoshorttermimpacts.
24
d.RandomizedversusNon-ExperimentalDesigns
AlongstandingconcernintheALMPliterature(e.g.,Ashenfelter,1987)isthe
unreliabilityofnon-experimentalestimators.Thisconcernledtoaseriesoflargescale
experimentsintheU.S.designedtotestjobtrainingprograms(Bloometal.,1997;
Schochet,Burghardt,andGlazerman,2001)andalargeliteraturecomparing
experimentalandnon-experimentalestimatesforthesameprogram(e.g.,Lalonde,
1986;SmithandTodd,2005).
Onesimplewaytoassesstheseconcernsistocomparethemagnitudesofthe
estimatedeffectsfrompapersbasedonexperimentaldesignstothosefromnon-
experimentaldesigns.Todothis,weincludeasimpleindicatorforanexperimental
designinthemodelsinTable7.Intheprogrameffectmodels(columns1and2)the
coefficientofthedummyisverysmallinmagnitude(lessthan1.0ppt)and
insignificantlydifferentfromzero.Likewise,inthemainOPmodels(columns3-4)the
coefficientissmallinmagnitude.ItisalittlelargerinmagnitudewhentheOPmodels
arerestrictedtothesetofstudieswithestimatedprogrameffects(column5)but
relativelyimprecise,perhapsreflectingtherelativelylargenumberofparameters
includedinthemodelrelativetothesamplesizeortheunevendistributionof
experimentalevaluationsacrossprogramtypes.Weconcludefromthisanalysisthat
thereislittleornoevidencethatresultsfromexperimentally-baseddesignsinthe
recentALMPliteratureare"lesspositive"or"lesssignificant"thanresultsfromnon-
experimentaldesigns.
e.PublicationBiasandp-hacking
Arelatedconcern,widelydiscussedinthemetaanalysisliterature(e.g.,
Rothstein,Sutton,andBorenstein,2005)isthatthesetofestimatedprogramimpactsin
theavailableliteraturecontainasystematicpositivebias,eitherbecauseanalystsonly
writeupandcirculatestudiesthatshowapositiveeffect(so-calledfiledrawerbias)or
25
becausetheychoosespecificationsthattendtoyieldpositiveandsignificanteffects(so-
calledp-hacking).
Astandardwaytolookforevidenceofpublicationbiasistoexaminefunnel
plotsoftherelationshipbetweentheestimatedprogrameffectsandtheirprecision
(Suttonetal.,2000).Figures3a,3b,and3cpresenttheseplotsfortheprogrameffects
foremploymentinoursample,restrictingattention(asinFigures2a-2c)toestimates
thathaveacorrespondingsamplingerroravailable.Forreference,wealsoshowthe
boundariesofthe“t=2”relationshipineachgraph.30Contrarytotheinvertedfunnel
patterntypicallyuncoveredinthemetaanalysisliterature(e.g.,Doucouliagosand
Stanley,2009;Havranek,2015;WolfsonandBelman,2015),thereisalotofdispersion
intheestimatedprogrameffectsatallthreetimehorizons,evenamongstudieswith
highlevelsofprecision,suggestingthatthevariationintheestimatesisnotjustaresult
ofsamplingerror.
Amoreformaltestforpublicationbiasistoregresstheestimatedprogram
effectfromagivenstudyandspecificationontheassociatedsamplingerrorofthe
estimateandotherpotentialcontrolvariables.UsingthenotationofSectionIVa,the
regressionmodelis:
b=Xα+θP─1/2+ν (4)
whereνrepresentsaresidual.Theestimateofθisinterpretedasatestforasymmetry
inthefunnelplotrelationshipbetweentheestimatedprogrameffectsandtheir
precision:ifthesamplecontainsmoreimpreciselyestimatedlargepositiveeffectsthan
largenegativeeffects,θwillbepositive.Stanley(2008)suggeststhatthemodelbe
estimatedbyweightedleastsquares,usingtheprecisionofeachestimate(i.e.,its
inversesamplingvariance)asaweight.Ifsamplingerroristheonlysourceofresidual
variationin(4)thiswillleadtoefficientestimates.
30Sincet=P1/2b,the“t=2”relationshipisP=4/b2whichisapairofhyperbolascenteredaroundtheyaxisinafunnelplot.
26
EstimationresultsforthismodelarepresentedinTable8.Wepresent
estimatesfromfourspecificationsestimatedbyunweightedOLSandprecision-weighted
leastsquares.31Theresultsgivenoindicationofpublicationbias.Theunweighted
estimateincolumn3,forexample,whichisbasedonamodelthatincludesourrichest
setofcontrols,impliesthatastudywitha1percentagepointlargerstandarderrorfor
theestimatedprogrameffectontheprobabilityofemploymentwillhaveabouta0.01
percentpointlargerestimatedprogrameffect.Thecorrespondingweightedestimate
suggestsaslightlylarger0.03percentpointlargerestimate,butislessprecise.
Wesuspectthatthereareatleasttwoexplanationsfortheapparentlackof
publicationbiasintheALMPliterature.ThefirstisthatALMPevaluationsareoften
conductedincloseco-operationwiththeagenciesthatoperatetheprograms.Insuch
settingsresearcherscannotsimplyshelvepapersthatshowsmallor"wrongsigned"
impacts.Theymayalsofindithardtochooseamongspecificationstoobtainamore
positiveprogrameffect.Asecondfactoristhatrefereesandotherresearchershaveno
strongpresumptionthatALMP'snecessarily"work",orthatafindingofanegativeor
insignificanteffectisuninteresting,sincemanyimportantpapersinthefield(e.g.,
Lalonde,1986;HeckmanandHotz,1989)reportsmallornegativeimpactsofALMPs.
Ourmodelsalsocontrolfortwootherfeaturesofastudythatmaybe
informativeaboutthepresenceofpublicationbias:whetheritwaspublished,andthe
numberofcitationsitreceived(measuredfromaGoogleScholarsearchinSpring
2015).32Thecoefficientsassociatedwithbothvariables(showninTable7)aresmall
andinsignificantacrossallspecifications,confirmingthatthereisnotendencyformore
positivestudiestobepublishedortobemorehighlycited.
f.AreSomeProgramsBetter(orWorse)forDifferentParticipantGroups?
31Theestimatedprecisionoftheestimatesinoursamplerangesfrom0.026toover50,000.TostabilizetheestimatesweWinsorizetheprecisionweightsatthe10thand90thpercentiles.32Toaccountforlagsinthecitationprocesswemodelcitationsastherankwithinthedistributionofcitationsforpaperswritteninthesameyear.
27
AlongstandingquestionintheALMPliteratureiswhethercertainparticipant
groupsare"bettermatched"tospecifictypesofprograms(forananalysisinthe
GermancontextseeBiewenetal.2007).WeaddressthisinTable9,whichpresents
separatemodelsfortheprogrameffectsfromdifferenttypesofALMPs.
Asabenchmarkcolumn1presentsabaselinespecificationfittoall5program
types,withdummiesfortheprogramtypes(notreported)andcontrolsfortheintake
group,thegendergroup,andtheagegroup.33The(omitted)basegroupiscomprisedof
mixedgenderandagegroupsfromtheregularUIrolls.Inthispooledspecificationthe
estimatedeffectsforfemalesandlongtermunemployedparticipantsaresignificantly
positive,whilethecoefficientforolderparticipantsissignificantlynegative,andthe
coefficientforyoungparticipantsisnegativeandmarginallysignificant.
Columns2-6reportestimatesforthesamespecification(minusthecontrolsfor
thetypeofprogram)fitseparatelytotheestimatedeffectsforeachofthe5program
types.Comparisonsacrossthesemodelssuggestthatlong-termunemployed
participantsbenefitrelativelymorefrom"humancapital"programs(i.e.,trainingand
privatesectoremployment),andrelativelylessfrom"workfirst"programs(i.e.,job
searchandotherprograms).Incontrast,disadvantagedparticipantsappeartobenefit
morefromworkfirstprogramsandlessfromhumancapitalprograms.Female
participantsalsoappeartobenefitrelativelymorefromtrainingandprivatesector
subsidyprograms,whiletherelativeeffectsforyouthsandolderparticipantsarenot
muchdifferentacrosstheprogramtypes.
Overalltheseresultssuggestthattheremaybepotentialgainstomatching
specificparticipantgroupstospecifictypesofprograms,thoughthesmallsamplesizes
formostoftheprogramtypesmustbenoted.Attemptstoexpandthepowerofthe
analysisbyusingOPmodelsforthesignandsignificanceoftheprogramestimateslead
33Thisisasimplifiedversionofthespecificationreportedincolumn2ofTable5andcolumn1ofTable7.
28
togenerallysimilarconclusionsastheprogrameffectmodelsreportedinTable9with
onlymodestgainsinprecision.
g.EffectsofCyclicalConditions
AnotherlongstandingquestionintheALMPliteratureiswhetherprogramsare
more(orless)effectiveindifferentcyclicalenvironments.34Oneviewisthatactive
programsarelesseffectiveinadepressedlabormarketbecauseparticipantshaveto
competewithother,moreadvantagedworkersforalimitedsetofjobs.Analternative
viewisthatALMPsaremoreeffectiveinweaklabormarketsbecauseemployers
becomemoreselectiveinaslackmarket,increasingthevalueofaninterventionthat
makesworkersmorejob-ready.
ThreepreviousstudieshaveinvestigatedALMPeffectivenessoverthebusiness
cycle.Kluve(2010)usesbetween-countryvariationinasmallEuropeanmetadataset,
whileLechnerandWunsch(2009)andForslundetal.(2011)analyzeprogramsin
GermanyandSweden,respectively.Allthreestudiessuggestapositivecorrelation
betweenALMPeffectivenessandtheunemploymentrate.
Toprovidesomenewevidenceweaddedtwoalternativecontextualvariablesto
ouranalysis,representingtheaveragegrowthrateofGDPandtheaverage
unemploymentrateduringtheyearsthetreatmentgroupparticipatedintheprogram.
Sincegrowthratesandunemploymentratesvarywidelyacrosscountries,wealso
introducedasetofcountrydummiesthatabsorbanypermanentdifferencesinlabor
marketconditionsacrosscountries.Theeffectofthesedummiesisinterestinginitsown
rightbecausethesharesofdifferentprogramtypesandparticipantgroupsalsovary
widelyacrosscountries,leadingtothepossibilityofbiasinthemeasuredeffectsof
programtypesandparticipantgroupsifthereareunobservedcountryspecificfactors
thataffecttheaveragesuccessofALMPsindifferentcountries.
34Arelatedquestioniswhetherprogramexternalitiesarebiggerorsmallerinweakorstronglabormarkets.ThisisaddressedintheexperimentconductedbyCreponetal.(2013).
29
TheresultsofouranalysisaresummarizedinTable10.Forreferencecolumn1
presentsabenchmarkspecificationidenticaltothesimplifiedmodelincolumn1of
Table9.Column2presentsthesamespecificationwiththeadditionof37country
dummies.Theadditionofthesedummiesleadstosomemodestbutinteresting
changesintheestimatedcoefficientsinthemetaanalysismodel.Mostnotably,the
coefficientsassociatedwithjobsearchassistance(JSA)and"other"programsboth
becomemorenegative,indicatingthattheseprogramstendtobemorewidelyusedin
countrieswhereallformsofALMPsarerelativelysuccessful.
Column3presentsamodelthatincludesthecontrolforaverageGPDgrowth
rateduringtheprogramperiod.Thecoefficientisnegativeandmarginallysignificant
(t=1.7)providingsuggestiveevidencethatALMPsworkbetterinrecessionarymarkets.A
modelthatcontrolsfortheaverageunemploymentrateshowsthesametendency
(coefficient=0.006,standarderror=0.007)thoughtheeffectislessprecise.
Aconcernwiththespecificationincolumn3isthattheaveragenumberof
programestimatespercountryissmall(manycountrieshaveonly2or3estimates)
leadingtopotentialover-fitting.Toaddressthis,weestimatedthemodelsincolumns4-
6,usingonlydatafromthefourcountriesthataccountforthelargestnumbersof
programestimates-Denmark(17estimates),France(20estimates),Germany(147
estimates)andtheU.S.(16estimates).Asshownincolumn4,ourbaselinespecification
yieldscoefficientestimatesthatarequitesimilartotheestimatesfromtheentire
sample,thoughtherelativeimpactsofJSAandotherprogramsaremorenegativein
these4countries.
Columns5and6presentmodelsthataddtheaverageGDPgrowthrateandthe
averageunemploymentrate,respectively,tothisbaselinemodel.Thesespecifications
suggestrelativelyimportantcyclicaleffectsonALMPeffectiveness.Forexample,
comparingtwosimilarprogramsoperatinginlabormarketswitha3percentagepoint
gapingrowthrates,theprogramintheslowergrowthenvironmentwouldbeexpected
tohavea0.1largerprogrameffect.
30
Toillustratethevariationdrivingtheresultsincolumns5and6,Figure4plots
theannualGDPgrowthrateinGermanyalongwiththemeanprogrameffectsin
differentyears,groupingprogramestimatesbytheyearthattheprogramwasfirst
operated.Wealsoshowthenumberofprogramestimatesforeachyear,whichvaries
substantiallyovertime.Thefigureshowsthecounter-cyclicalpatternofprogrameffects
ismainlydrivenbylargepositiveeffectsofprogramsimplementedintheearly2000s
duringaperiodofslowgrowth.
Althoughpolicymakershavetodecidewhethertoincreasespendingforactive
programsandenrollmoreparticipantsknowingonlycurrentbusinesscycleconditions,it
isalsoofinteresthowlabormarketconditionsatthetimeofprogramcompletionare
relatedtotheprogrameffects.InAppendixTable5weestimatealternative
specificationsthatcontrolfortheGDPgrowthrateandunemploymentrateatthe
beginningandaftertheendoftheprogramperiod.TheseestimatessuggestthatALMP
programstendtobeparticularlysuccessfulifparticipantsareenrolledinaprogram
duringadownturnandexittheprogramduringaperiodoffavorableeconomic
conditions.
WhiletheevidenceinTable10suggestsacountercyclicalpatternofprogram
effectiveness,itisworthemphasizingthattheexplanationforthispatternislessclear.It
ispossiblethatthevalueofagivenprogramishigherinarecessionaryenvironment.It
isalsopossible,however,thatthecharacteristicsofALMPparticipants,orofthe
programsthemselves,changeinawaythatcontributestoamorepositiveimpactina
slow-growth/high-unemploymentenvironment.
V.SummaryandConclusions
Wehaveassembledandanalyzedanewsampleofimpactestimatesfrom207
studiesofactivelabormarketpolicies.Buildingonourearlierstudy(CKW),weargue
thatitisimportanttodistinguishbetweenimpactsatvarioustimehorizonssince
completionoftheprogram,andtoconsiderhowthetimeprofileofimpactsvariesby
31
thetypeofALMP.Wealsostudytheimportanceofparticipantheterogeneity,andlook
forevidencethatspecificsubgroupsmaybenefitmoreorlessfromparticulartypesof
programs.Finally,westudyhowthestateofthelabormarketaffectsthemeasured
effectivenessofALMPs.
WithregardtotheimpactsofdifferenttypesofALMPs,wefindthatthetime
profilesof"workfirst"styleprogramsthatofferjobsearchassistanceorincentivesto
enterworkquicklydifferfromtheprofilesof"humancapital"styletrainingprograms
andpublicsectoremploymentprograms.Humancapitalprogramshavesmall(orin
somecasesevennegative)shorttermimpacts,coupledwithlargerimpactsinthe
mediumorlongerrun(2-3yearsaftercompletionoftheprogram),whereastheimpacts
fromworkfirstprogramsaremorestable.Wealsoconfirmthatpublicsector
employmentprogramshavenegligible,orevennegativeprogramimpactsatalltime
horizons.
Withregardtodifferentparticipantgroups,wefindthatfemaleparticipantsand
thosedrawnfromthepooloflongtermunemployedtendtohavelargerprogram
effectsthanothergroups.Incontrast,theprogramestimatesforyouthsandolder
workersaretypicallylesspositivethanforothergroups.Wealsofindindicationsof
potentialgainstomatchingdifferentparticipantgroupstospecificprograms,with
evidencethatworkfirstprogramsarerelativelymoresuccessfulfordisadvantaged
participants,whereashumancapitalprogramsaremoresuccessfulforthelongterm
unemployed.
Withregardtothestateofthelabormarket,wefindthatALMPstendtohave
largerimpactsinperiodsofslowgrowthandhigherunemployment.Inparticular,we
findarelativelylargecyclicalcomponentintheprogramestimatesfromfourcountries
thataccountforone-halfofoursample.Wealsofindsuggestiveevidencethathuman
capitalprogramsaremorecyclicallysensitivethanworkfirstprograms.
Ourfindingsontherelativeefficacyofhumancapitalprogramsforlongterm
unemployed,andonthelargerimpactsoftheseprogramsinrecessionary
32
environments,pointtoapotentiallyimportantpolicylesson.AsnotedbyKrueger,Judd
andCho(2014)andKroftetal.(2016),thenumberoflongtermunemployedrises
rapidlyasarecessionpersists.Thisgrouphasahighprobabilityofleavingthelabor
force,riskingpermanentlossesintheproductivecapacityoftheeconomy.Onepolicy
responseiscountercyclicaljobtrainingprogramsandprivateemploymentsubsidies,
whichareparticularlyeffectiveforthelonger-termunemployedinarecessionary
climate.
Methodologically,wefindanumberofinterestingpatternsintherecentALMP
literature.Wefindthattheestimatedimpactsderivedfromrandomizedcontrolled
trials,whichaccountforone-fifthofoursample,arenotmuchdifferentonaverage
fromthenon-experimentalestimates.Wealsofindnoevidenceof"publicationbias"in
therelationshipbetweenthemagnitudeofthepointestimatesfromdifferentstudies
andtheircorrespondingprecision.Wedofindthatthechoiceofoutcomevariableused
intheevaluationmatters,withatendencytowardmorepositiveshorttermimpact
estimatesfromstudiesthatmodelthetimetofirstjobthanfromstudiesthatmodelthe
probabilityofemploymentorthelevelofearnings.
Finally,weconcludethatmetaanalyticmodelsbasedonthesignand
significanceoftheprogramimpactsleadtoverysimilarconclusionsasmodelsbasedon
programeffects.Wearguethatthisarisesbecausemuchofthevariationinthesignand
significanceofestimatedimpactsacrossstudiesintheALMPliteratureisdrivenby
variationinestimatedprogrameffects,ratherthanbyvariationinthecorresponding
samplingerrorsoftheestimates.
33
References
Angrist,J.D.,&Pischke,J.S.(2010)."TheCredibilityRevolutioninEmpiricalEconomics:HowBetterResearchDesignisTakingtheConOutofEconometrics.JournalofEconomicPerspectives24(2):3-30.Ashenfelter,Orley(1978).“EstimatingtheEffectofTrainingProgramsonEarnings”ReviewofEconomicsandStatistics60(1):47-57.Ashenfelter,OrleyandDavidCard(1985).“UsingtheLongitudinalStructureofEarningstoEstimatetheEffectofTrainingPrograms.”ReviewofEconomicsandStatistics67(4):648-660.Ashenfelter,Orley(1987)."TheCaseforEvaluatingTrainingProgramswithRandomizedTrials."EconomicsofEducationReview6(4):333-338.Bergemann,A.H.andG.J.vandenBerg(2008)."ActiveLaborMarketPolicyEffectsforWomeninEurope-ASurvey".Annalesd'EconomieetdeStatistique91/92:385-408.Biewen,M.,B.Fitzenberger,A.Osikominu,andM.Waller(2014)" TheEffectivenessofPublicSponsoredTrainingRevisited:TheImportanceofDataandMethodologicalChoices",JournalofLaborEconomics,32:837-897.Biewen,M.,B.Fitzenberger,A.Osikominu,andM.Waller(2007)"WhichProgramforWhom?EvidenceontheComparativeEffectivenessofPublicSponsoredTrainingProgramsinGermany",IZADiscussionPaperNo.2885.Blank,CardandRobins(2000)."FinancialIncentivesforIncreasingWorkandIncomeAmongLow-IncomeFamilies."InRebeccaM.BlankandDavidCard,editors,FindingJobs:WorkandWelfareReform.NewYork:RussellSageFoundation.Bloom,HowardS.,LarryL.Orr,StephenH.Bell,GeorgeCave,FredDoolittle,WinstonLin,andJohannesM.Bos(1997).“TheBenefitsandCostsofJTPATitleII-APrograms:KeyFindingsfromtheNationalJobTrainingPartnershipActStudy.”JournalofHumanResources32(3):549–576.Card,DavidandDeanR.Hyslop(2005)."EstimatingtheEffectofaTime-LimitedEarningsSubsidyforWelfareRecipients."Econometrica73(6):1723-1770.Card,D.,J.KluveandA.Weber(2010)."ActiveLabourMarketPolicyEvaluations:AMeta-analysis."EconomicJournal120(548):F452-F477.
34
Cho,Y.andM.Honorati(2014).“EntrepreneurshipProgramsinDevelopingCountries:AMetaRegressionAnalysis.”LabourEconomics28(C):110-130.Crépon,Bruno,EstherDuflo,MarcGurgand,RolandRathelot,andPhilippeZamora(2013)."DoLaborMarketPolicieshaveDisplacementEffects?EvidencefromaClusteredRandomizedExperiment."QuarterlyJournalofEconomics128(2):531-580.Doucoulias,HristosandT.D.Stanley(2009).“PublicationSelectionBiasinMinimumWageResearch?AMeta-RegressionAnalysis.”BritishJournalofIndustrialRelations47(2):406-428.Filges,Trine,GeirSmedslund,Anne-SofieDueKnudsen,andAnne-MarieKlintJørgensen(2015)."ActiveLabourMarketProgrammeParticipationforUnemploymentRecipients:ASystematicReview."CampbellSystematicReviews2015(2).Forslund,A.,P.FredrikssonandJ.Vikström(2011).“WhatActiveLaborMarketPolicyWorksInaRecession?”.NordicEconomicPolicyReview1:171-207.Friedlander,Daniel,DavidH.GreenbergandPhilipK.Robins(1997)."EvaluatingGovernmentTrainingProgramsfortheEconomicallyDisadvantaged".JournalofEconomicLiterature,25(4):1809-1855Gladden,TriciaandChristopherTaber(2000)."WageProgressionAmongLessSkilledWorkers."InRebeccaM.BlankandDavidCard,editors,FindingWork:JobsandWelfareReform.NewYork:RussellSageFoundation.Greenberg,DavidH.,CharlesMichalopoulosandPhilipK.Robins(2003).“AMeta-AnalysisofGovernment-SponsoredTrainingPrograms.”IndustrialandLaborRelationsReview57():31-53.Grimm,M.andA.L.Paffhausen(2015).“DoInterventionsTargetedatMicro-EntrepreneursandSmallandMedium-sizedFirmsCreateJobs?ASystematicReviewoftheEvidenceforLowandMiddleIncomeCountries”.LabourEconomics32:67-85.Ham,JohnC.andRobertJ.Lalonde(1996)."TheEffectofSampleSelectionandInitialConditionsinDurationModels:EvidencefromExperimentalDataonTraining."Econometrica64(1):175-205.Havránek,Tomáš(2015)MeasuringIntertemporalSubstitution:TheImportanceofMethodChoicesandSelectiveReporting."JournaloftheEuropeanEconomicAssociation,13(6).
35
Heckman,JamesJ.,andV.JosephHotz(1989)."ChoosingAmongAlternativeNonexperimentalMethodsforEstimatingtheImpactofSocialPrograms:TheCaseofManpowerTraining."JournaloftheAmericanstatisticalAssociation84(408):862-874.Heckman,JamesJ.,HidehikoIchimura,JeffreyA.Smith,andPetraTodd(1998).“CharacterizingSelectionBiasUsingExperimentalData.”Econometrica66(5):1017-1098.Heckman,JamesJ.RobertJ.LaLondeandJeffreyA.Smith(1999)."TheEconomicsandEconometricsofActiveLaborMarketPrograms".InOrleyAshenfelterandDavidCard,editors,HandbookofLaborEconomicsVolume3a.Amsterdam:Elsevier.Hedges,LarryV.andIngramOlkin(1985).StatisticalMethodsforMeta-Analysis.Orlando:AcademicPress.Ibarrarán,P.andD.Rosas(2009),EvaluatingtheImpactofJobTrainingProgramsinLatinAmerica:EvidencefromIDBfundedoperations,JournalofDevelopmentEffectiveness1:195-216.Kluve,J.(2010)."TheEffectivenessofEuropeanActiveLaborMarketPrograms."LabourEconomics17:904-918.Kroft,Kory,FabianLange,MatthewJ.Notowidigdo,andLawrenceF.Katz(2016)."Long-termUnemploymentandtheGreatRecession:TheRoleofComposition,DurationDependence,andNon-Participation".JournalofLaborEconomics34(S1):S7-S54.Krueger,AlanB.,JuddCramer,andDavidCho(2014)."AretheLong-TermUnemployedontheMarginsoftheLaborMarket?"BrookingsPapersonEconomicsActivitySpring:229-280.Lalonde,RobertJ(1986).“EvaluatingtheEconometricEvaluationsofTrainingProgramswithExperimentalData.”AmericanEconomicReview76(4):604-620.Lalonde,RobertJ(2003)."EmploymentandTrainingPrograms."InRobertA.Moffit,editor,MeansTestedTransferProgramsintheUnitedStates.Chicago:UniversityofChicagoPress.Lechner,MichaelandConnieWunsch(2009).“AreTrainingProgramsMoreEffectivewhenUnemploymentisHigh?”JournalofLaborEconomics27(4):653-692.
36
Martin,JohnP.(2014)"ActivationandActiveLabourMarketPoliciesinOECDCountries:StylizedFactsandEvidenceonTheirEffectiveness."IZAPolicyPaperNo.84.Meyer,BruceD.(1995)"LessonsfromtheU.S.UnemploymentInsuranceExperiments."JournalofEconomicLiterature33(1):91-131.Mincer,Jacob.(1974).Schooling,ExperienceandEarnings.NewYork:NationalBureauofEconomicResearch.Roberts,ColinD.andT.D.Stanley,editors(2005).Meta-RegressionAnalysis:IssuesofPublicationBiasinEconomics.NewYork:Wiley.Rothstein,HannahR.,AlexanderJ.SuttonandMichaelBorenstein(2005).PublicationBiasinMetaAnalaysis-Prevention,AssessmentandAdjustments.NewYork:Wiley.Schochet,PeterZ.,JohnBurghardt,andStevenGlazerman(2001).NationalJobCorpsStudy:TheImpactsofJobCorpsonParticipants’EmploymentandRelatedOutcomes.Princeton,NJ:MathematicaPolicyResearch.Sianesi,B.(2004).AnEvaluationoftheSwedishSystemofActiveLaborMarketProgramsinthe1990s."ReviewofEconomicsandStatistics86(1):133-155.Smith,JeffreyandPetraTodd(2005).“DoesMatchingOvercomeLaLonde’sCritiqueofNonexperimentalMethods?”JournalofEconometrics125(1-2):305-353. Stanley,T.D.(2008)."Meta-RegressionMethodsforDetectingandEstimatingEmpiricalEffectsinthePresenceofPublicationSelection."OxfordBulletinofEconomicsandStatistics70(1):102-127.StanleyT.D.andHristosDoucouliagos(2012).Meta-RegressionAnalysisinEconomicsandBusiness.MiltonParkUK:Routledge.Solon,Gary,StevenJ.HaiderandJeffreyM.Wooldridge(2015)."WhatAreWeWeightingFor?"JournalofHumanResources50(2):301-316.Sutton,A.J.,S.J.Duval,R.L.Tweedie,K.R.AbramsandD.R.Jones(2000).“EmpiricalAssessmentofEffectofPublicationBiasonMeta-analyses”.BritishMedicalJournal320:1574–77.UnitedStatesDepartmentofLabor(2016).LaborForceStatisticsfromtheCurrentPopulationSurvey2015,http://www.bls.gov/cps.
37
Wolfson,PaulJ.andBelman,Dale(2015.)"15YearsofResearchonU.S.EmploymentandtheMinimumWage".TuckSchoolofBusinessWorkingPaperNo.2705499.
Figure 1: Number of Program Estimates, By Year of Program Start
0
10
20
30
40
50
60
70
80
90
100
1980 1989 1994 1999 2004 2009
Year of Program Start
Num
ber o
f Program
Estim
ates
Experimental Design
Non‐experimental Design
Figure 2a: Short Term Effects and Confidence Intervals
-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Short Term Effect
Note: Diamonds represent estimated short term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 56 estimates -- 2 large positive estimates are not shown for clarity.
precision-weighted mean = 0.028unweighted mean = 0.053(median = 0.029)
Figure 2b: Medium Term Effects and Confidence Intervals
-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Medium Term Effect
Note: Diamonds represent estimated medium term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 69 estimates -- 3 large positive estimates are not shown for clarity.
precision-weighted mean = 0.053unweighted mean = 0.086 (median = 0.051)
Figure 2c: Long Term Effects and Confidence Intervals
-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Estimated Long Term Effect
Note: Diamonds represent estimated long term treatment effects on probability of employment for a program/participant subgroup (PPS). Horizontal lines represent 95% confidence intervals. Graph shows 39 estimates.
precision-weighted mean = 0.109unweighted mean = 0.123(median = 0.090)
0
2000
4000
6000
8000
10000
12000
14000
-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25
Precision
ofE
stim
ate
EstimatedShortTermEffect
Figure3a:FunnelPlotofShortTermEstimates
Note:Diamondsrepresentprecisionofestimatedshorttermtreatmenteffectforaprogram/participant(PPS)subgroup,graphed againsttheestimatedtreatmenteffect.Graphshows56estimates-- 2largepositiveestimatesarenotshownforclarity.
unweightedmeantreatmenteffect=0.053
t=2boundaries
precision-weightedmean=0.028
0
2000
4000
6000
8000
10000
12000
14000
-0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Precision
ofE
stim
ate
EstimatedMediumTermEffect
Figure3b:FunnelPlotofMediumTermEstimates
Note:Diamondsrepresentprecisionofestimatedmediumtermtreatmenteffectforaprogram/participantsubgroup(PPS),graphedagainsttheestimatedtreatmenteffect.Graphshows69estimates-- 3largepositiveestimatesarenotshownforclarity.
meantreatmenteffect=0.086
t=2boundaries
precision-weightedmean=0.053
0
2000
4000
6000
8000
10000
12000
14000
-0.1 -0.05 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
Precision
ofE
stim
ate
EstimatedLongTermEffect
Figure3c:FunnelPlotofLongTermEstimates
Note:Diamondsrepresentprecisionofestimatedlongtermtreatmenteffectforaprogram/participant(PPS)subgroup,graphedagainsttheestimatedtreatmenteffect.Graphshows39estimates.
meantreatmenteffect=0.123
t=2boundaries
precision-weightedmean=0.109
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
-5
0
5
10
15
20
25
30
35
40
1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007
GDP
Gro
wth
Rate
Dur
ing
Year
of P
rogr
am
Mea
n Pr
ogra
m E
ffect
for P
rogr
ams
in a
Yea
r
Figure 4: Mean Program Effects and GDP Growth Rate - German ALMP Evaluations
Mean Program Effect, Left Scale GDP Growth Rate, Right Scale
Note: mean program effect on the employment rate of ALMP participants (in percentage points) is plotted for programs offered in different years, with number of program estimates in that year. GDP growth rate is for the first year the program was operated.
721 1
4
27
5
17
6
3
6
Table 1: Description of Sample of Program Estimates
Austria, U.S., U.K, Latin Amer.
Germany, Nordic Aust., N.Z., and
Full sample Switzerland Countries Canada Non‐OECD Carribean
(1) (2) (3) (4) (5) (6)
Number of estimates 857 290 212 87 132 72Number of PPS's 526 163 127 45 86 54Number of studies 207 52 48 24 33 19
Type of program (%):
Training 49 62 17 45 79 97
Job Search Assistance 15 8 26 22 2 0
Private Subsidy 14 17 15 5 11 3
Public Employment 9 9 10 3 6 0
Other 14 5 32 25 2 0
Age of program group (%):
Mixed 59 54 61 72 40 25
Youth (<25 years) 21 12 20 15 53 69
Older (≥25 years) 20 33 19 13 8 6
Gender of Program group (%):
Mixed 54 53 67 43 43 11
Males only 22 24 18 25 23 44
Females only 23 23 16 32 31 44
Type of Program Participants (%):
Registered unemployed 65 86 67 33 24 0
Long‐term unemployed 12 8 10 25 7 0
Disadvantaged 23 6 23 41 69 100
Outcome of Interest (%):
Employment status 57 83 31 26 63 54
Earnings 21 8 25 47 36 43
Hazard to new job 12 7 25 3 0 0
Other hazard 6 0 16 2 0 3
Unemployment status 4 2 4 21 1 0
Effect Measured at (%):
Short Term 48 42 54 37 47 57
Medium Term 35 34 31 40 45 42
Long Term 16 23 16 23 8 1
Experimental Design (%) 19 0 39 31 28 26
Note: see text for description of sample. Study refers to an article or unpublished paper. PPS refers to a
program/participant subgroup (e.g., a job search assistance program for mixed gender youths). Estimate refers
to an estimate of the effect of the program on the participant subgroup at either a short‐term (<1 year after
completion of the program), medium term (1‐2 years post completion) or long term (2+ years post completion)
time horizon. A9Job search assistance programs include sanction programs. "Other" programs include those
that combine elements of the four distinct types.
Country Group:
Table2:SummaryofProgramEstimatesbyAvailabilityofEstimateofProgramEffectonProbabilityofEmployment
Subsamplewith SubsamplewithOutcome= EstimateofEffect MeanEffect MedianEffect
Fullsample Prob.ofEmp. onProb.ofEmp. (std.error)a (std.error)a
(1) (2) (3) (4) (5)
Numberofestimates 857 490 352
NumberofPPS's 526 274 200
Numberofstudies 207 111 83
ShortTermEstimates
AllSTestimates--number[pct.oftotalestimates] 415[48] 205[42] 141[40] 1.6(0.8) 1.0(1.0)
SignificantpositiveSTestimate--pct.ofSTsample 40 31 33 8.8(1.3) 6.0(1.3)
InsignificantSTestimate--pct.ofSTsample 42 47 44 0.5(0.4) 0.0(0.6)
SignificantnegativeSTestimate--pct.ofSTsample 18 22 23 -6.4(0.8) -5.0(0.7)
MediumTerm(MT)Estimates
AllMTestimates--number[pct.oftotalestimates] 301[35] 194[40] 143[41] 5.4(1.2) 3.0(0.7)
SignificantpositiveMTestimate--pct.ofMTsample 52 50 47 11.3(1.9) 8.5(1.1)
InsignificantMTestimate--pct.ofMTsample 40 41 43 1.3(0.3) 1.0(0.4)
SignificantnegativeMTestimate--pct.ofMTsample 8 9 10 -5.0(1.2) -4.9(2.2)
LongTerm(LT)Estimates
AllLTestimates--number[pct.oftotalestimates] 141[16] 91[19] 68[19] 8.7(2.2) 4.9(1.4)
SignificantpositiveLTestimate--pct.ofsample 61 65 65 13.0(2.7) 9.0(2.2)
InsignificantLTestimate--pct.ofsample 35 32 32 1.3(0.6) 1.1(0.7)
SignificantnegativeLTestimate--pct.ofsample 4 3 3 -4.2(0.5) --
aStandarderrorsareclusteredbystudy.
SignandSignificanceofProgramEffects: EstimatedProgramEffectonProb.OfEmp(col.3subsample)
352
200
83
Notes:seenotetoTable1.Shorttermprogramestimatesarefortheperiodupto1yearafterthecompletionoftheprogram.Mediumtermestimatesarefortheperiodfrom1to2yearsaftercompletionoftheprogram.Longtermestimatesarefortheperiod2ormoreyearsaftercompletionoftheprogram.Effectsizesareonlyavailableforstudiesthatmodeltheprobabilityofemploymentastheoutcomeofinterest,andprovideinformationonmeanemploymentrateofcomparisongroup.
Table 3a: Comparison of Impact Estimates by Program Type and Participant Group
MedianNumber Sample Percent Short Medium Longer Short Medium Longer
Est's. Size RCT's Term Term Term Term Term Term(1) (2) (3) (4) (5) (6) (7) (8) (9)
All 857 10,709 19.4 1.6 5.4 8.7 40 52 61(141) (143) (68) (415) (301) (141)
By Program Type:Training 418 7,700 12.9 2.0 6.6 6.7 35 54 67
(90) (92) (35) (201) (163) (54)
Job Search Assist. 129 4,648 51.2 1.2 2.0 1.1 53 63 43(16) (13) (7) (68) (40) (21)
Private Subsidy 118 10,000 8.5 1.1 6.2 21.1 37 65 88(13) (17) (16) (49) (37) (32)
Public Sector Emp. 76 17,084 0.0 3.6 -1.1 0.8 32 25 27(14) (12) (6) (41) (24) (11)
Other 116 17,391 31.0 7.2 5.8 2.0 52 38 43(8) (9) (4) (56) (37) (23)
By Intake Group:UI Recipients 554 11,000 17.1 -0.1 4.3 8.5 34 47 59
(93) (101) (50) (258) (193) (103)
Long Term Unem. 106 8,900 16.0 5.8 13.0 12.7 50 65 63(17) (16) (10) (50) (40) (16)
Disadvantaged 197 7,027 27.4 4.2 5.3 5.0 50 59 68(31) (26) (8) (107) (68) (22)
Mean Program Effect on Prob. Emp. (×100) Pct. Of Estimates with Sig. Positive Impact
Notes: see Tables 1 and 2. Number of program estimates associated with each table entry is reported in parentheses. Program effects (columns 4-6) are only available for studies that model the probability of employment as the outcome of interest.
Table 3b: Additional Comparisons of Impact Estimates by Participant Groups and Design
MedianNumber Sample Percent Short Medium Longer Short Medium Longer
Est's. Size RCT's Term Term Term Term Term Term(1) (2) (3) (4) (5) (6) (7) (8) (9)
All 857 10,709 19.4 1.6 5.4 8.7 40 52 61(141) (143) (68) (415) (301) (141)
By Age:Mixed Age 505 10,000 16.6 1.7 6.7 10.5 47 57 65
(71) (84) (51) (238) (178) (89)
Youth (<25) 180 3,000 33.3 2.9 2.7 0.2 32 41 67(34) (29) (5) (92) (64) (24)
Non-Youth 172 25,850 12.8 0.1 4.5 4.6 31 51 43(36) (30) (12) (85) (59) (28)
By Gender:Mixed Gender 466 11,000 19.7 1.6 4.4 5.6 39 52 59
(89) (85) (45) (224) (155) (87)
Males Only 191 10,000 15.2 1.4 6.1 13.1 41 50 58(24) (28) (9) (95) (72) (24)
Females Only 200 8,345 22.5 4.1 7.8 15.9 41 55 70(28) (30) (14) (96) (74) (30)
By Evaluation Design:Experimental 166 1,471 100.0 4.4 2.5 0.5 40 41 37
(28) (25) (15) (78) (58) (30)
Non-experimental 691 16,000 0.0 0.9 6.0 11.0 40 55 68(113) (118) (53) (337) (243) (111)
Mean Program Effect on Prob. Emp. (×100) Pct. Of Estimates with Sig. Positive Impact
Notes: see Tables 1 and 2. Number of program estimates associated with each table entry is reported in parentheses. Program effects (columns 4-6) are only available for studies that model the probability of employment as the outcome of interest.
Table 4: Transitions in Program Impacts for a Given Program and Participant Subgroup
(1) (2) (3) (4) (5) (6)All 0.021 0.024 -0.006 0.231 0.250 0.020
(0.008) (0.015) (0.004) (0.055) (0.103) (0.052) Number Studies 105 43 47 225 100 102
By Program TypeTraining 0.032 0.044 -0.004 0.314 0.439 0.048
(0.008) (0.018) (0.005) (0.072) (0.085) (0.049) Number Studies 70 28 28 121 41 42
Job Search Assist. 0.003 -0.001 0.000 0.265 0.143 -0.111(0.008) (0.001) (0.002) (0.095) (0.167) (0.144)
Number Studies 10 7 7 34 21 18
Private Subsidy -0.020 -0.004 -0.012 0.083 0.167 -0.062(0.049) (0.078) (0.013) (0.150) (0.267) (0.068)
Number Studies 9 2 6 24 12 16
Public Sector Emp. 0.016 -0.049 -0.019 0.158 -0.143 -0.143(0.014) (0.049) (0.019) (0.170) (0.494) (0.285)
Number Studies 10 2 2 19 7 7
Sanction/Threat 0.004 -0.021 -0.014 0.000 0.158 0.211(0.012) (0.008) (0.006) (0.108) (0.182) (0.092)
Number Studies 6 4 4 27 19 19Notes: Change in estimated effect size in column 1 represents the difference between the estimated medium term and short term effects on the probability of employment for a given program and participant subgroup (PPS). Changes in columns 2 and 3 are defined analogously. Change in sign/significance in column 4 is defined as +1 if the short term estimate is significantly negative and the medium term estimate is insignificant, or if the short term estimate is insignificant and the medium term estimate is significantly positive; 0 if the sign and significance of the short term and medium term estimates is the same; and -1 if the short term estimate is significantly positive and the medium term estimate is insignificant, or if the short term estimate is insignificant and the medium term estimate is significantly negative. Changes in columns 5 and 6 are defined analogously. Standard deviations (clustered by study number) in parenthesis.
Change in Program Effect on Prob. Of Emp. Change in Sign/Significanceshort term to medium term
short term to long term
medium term to long term
short term to medium term
short term to long term
medium term to long term
Table 5: Estimated Program Effects on Probability of Employment
(1) (2) (3) (4) (5)Effect Term (Omitted = Short Term)Medium Term 0.035 0.029 0.045 0.040 0.032
(0.012) (0.009) (0.016) (0.011) (0.008)Long Term 0.064 0.045 0.046 0.045 0.035
(0.019) (0.015) (0.018) (0.017) (0.014)Program Type (Omitted = Training)Job search Assist. -0.032 -0.009 -0.008 0.007 --
(0.012) (0.020) (0.011) (0.021)Private Subsidy 0.042 0.042 -0.009 0.016 --
(0.030) (0.026) (0.037) (0.041)Public Sector Emp. -0.065 -0.08 -0.056 -0.058 --
(0.013) (0.020) (0.014) (0.023)Other 0.007 0.003 0.052 0.047 --
(0.027) (0.036) (0.026) (0.038)
Interaction with Medium Term:Job search Assist. -- -- -0.037 -0.037 -0.028
(0.019) (0.018) (0.011)Private Subsidy -- -- 0.005 -0.012 -0.048
(0.045) (0.044) (0.046)Public Sector Emp. -- -- -0.02 -0.024 -0.015
(0.027) (0.027) (0.015)Other -- -- -0.059 -0.048 -0.028
(0.022) (0.021) (0.014)
Interaction with Long Term :Job search Assist. -- -- -0.048 -0.03 -0.034
(0.019) (0.022) (0.014)Private Subsidy -- -- 0.153 0.088 -0.061
(0.060) (0.056) (0.044)Public Sector Emp. -- -- -0.002 -0.053 -0.061
(0.028) (0.039) (0.031)Other -- -- -0.098 -0.109 -0.051
(0.030) (0.037) (0.015)
Additional Controls No Yes No Yes
Notes: Sample size is 352 estimates. Standard errors (clustered by study) in parentheses. Models are linear regressions with the effect size as dependent variable. Coefficients of additional control variables are reported in Table 7. Model in column 5 is estimated with fixed effects controlling for 200 program participant subgroups.
PPS fixed effects
Table 6: Models for Sign/Significance of Estimated Program Effects
(1) (2) (3) (4) (5) (6) (7)
Effect Term (Omitted = Short Term)
Medium Term 0.372 0.483 0.563 0.639 0.491 2.008 0.387
(0.088) (0.099) (0.130) (0.138) (0.145) (0.452) (0.158)
Long Term 0.597 0.742 0.901 1.053 1.03 2.536 0.457
(0.157) (0.167) (0.175) (0.171) (0.206) (0.449) (0.135)
Program Type (Omitted = Training)
Job search Assist. 0.274 0.286 0.531 0.532 0.569 ‐‐ ‐‐
(0.156) (0.168) (0.180) (0.197) (0.459)
Private Subsidy 0.139 0.076 ‐0.04 ‐0.132 ‐0.166 ‐‐ ‐‐
(0.189) (0.210) (0.224) (0.263) (0.438)
Public Sector Emp. ‐0.677 ‐0.758 ‐0.383 ‐0.489 ‐1.399 ‐‐ ‐‐
(0.219) (0.228) (0.276) (0.279) (0.496)
Other ‐0.11 ‐0.205 0.318 0.202 1.148 ‐‐ ‐‐
(0.172) (0.184) (0.206) (0.236) (0.653)
Interaction with Medium Term:
Job search Assist. ‐‐ ‐‐ ‐0.289 ‐0.283 ‐0.004 ‐0.743 ‐0.157
(0.235) (0.249) (0.343) (0.618) (0.218)
Private Subsidy ‐‐ ‐‐ 0.138 0.226 0.353 ‐1.085 ‐0.266
(0.289) (0.311) (0.486) (1.025) (0.289)
Public Sector Emp. ‐‐ ‐‐ ‐0.645 ‐0.573 0.051 ‐1.394 ‐0.229
(0.285) (0.288) (0.477) (0.861) (0.305)
Other ‐‐ ‐‐ ‐0.764 ‐0.705 ‐0.662 ‐2.04 ‐0.387
(0.226) (0.245) (0.278) (0.796) (0.234)
Interaction with Long Term :
Job search Assist. ‐‐ ‐‐ ‐1.017 ‐1.022 ‐0.832 ‐1.842 ‐0.331
(0.313) (0.294) (0.313) (0.775) (0.258)
Private Subsidy ‐‐ ‐‐ 0.611 0.58 1.274 ‐2.295 ‐0.385
(0.375) (0.387) (0.798) (1.428) (0.350)
Public Sector Emp. ‐‐ ‐‐ ‐0.643 ‐0.675 0.131 ‐2.366 ‐0.45
(0.490) (0.497) (0.832) (1.568) (0.822)
Other ‐‐ ‐‐ ‐0.999 ‐1.021 ‐1.638 ‐0.735 ‐0.194
(0.353) (0.375) (0.430) (1.282) (0.344)
Additional Controls No Yes No Yes Yes PPS fixed
effects
PPS fixed
effects
Notes: Sample size is 857 program estimates, except column 5, which is based on 352 estimates for which
program effect on probability of employment is also available. Standard errors (clustered by study) in
parentheses. Models in columns 1‐6 are ordered probits, fit to ordinal data with value of +1 for significantly
positive, 0 for insignificant, ‐1 for significantly negative estimate. Estimated cutpoints (2 per model) are not
reported in the Table. Coefficients of additional control variables are reported in Table 7. Model in column 6 is
estimated with fixed effects controlling for program participant subgroups. Model in column 7 is a linear
regression with fixed effects for program participant subgroups.
Table 7: Estimated Coefficients of Control Variables Included in Models in Tables 5 and 6
(1) (2) (3) (4) (5)Outcome of Interest (Omitted = Probability of Employment)Earnings -- -- -0.003 -0.01 --
(0.130) (0.132)Hazard to New Job -- -- 0.275 0.264 --
(0.211) (0.212)Other Hazard -- -- 0.613 0.547 --
(0.275) (0.263)Unemployment Status -- -- 0.598 0.591 --
(0.293) (0.285)Age of Program Group (Omitted = Mixed)Youths (<25) -0.031 -0.025 -0.368 -0.348 -0.518
(0.018) (0.018) (0.151) (0.153) (0.287)Older (>=25) -0.07 -0.063 -0.423 -0.425 -0.671
(0.020) (0.020) (0.157) (0.160) (0.297)Gender of Program Group (Omitted = Mixed)Males only 0.011 0.007 -0.007 -0.006 -0.328
(0.022) (0.022) (0.149) (0.149) (0.266)Females only 0.047 0.041 0.064 0.053 0.000
(0.023) (0.023) (0.144) (0.146) (0.250)Country Group (Omitted = Nordic)Germanic 0.056 0.045 0.250 0.176 0.910
(0.027) (0.026) (0.192) (0.196) (0.488)Anglo -0.026 -0.028 0.177 0.14 1.231
(0.030) (0.028) (0.241) (0.236) (0.579)East Europe 0.022 0.028 0.131 0.096 0.618
(0.031) (0.028) (0.201) (0.202) (0.378)Rest of Europe 0.016 0.012 0.125 0.088 0.738
(0.024) (0.023) (0.187) (0.189) (0.483)Latin America 0.009 0.012 0.108 0.1 1.012
(0.053) (0.053) (0.338) (0.338) (0.826)Remaining Countries 0.035 0.038 -0.063 -0.064 1.124
(0.035) (0.035) (0.281) (0.286) (0.529)Type of Program Participant (Omitted = Registered Unemployed)Disadvantaged 0.018 0.013 0.542 0.527 0.356
(0.036) (0.036) (0.228) (0.228) (0.623)Long-term Unemployed 0.083 0.08 0.388 0.404 0.392
(0.032) (0.031) (0.181) (0.179) (0.332)
-0.029 -0.023 -0.135 -0.122 -0.55(0.016) (0.016) (0.179) (0.177) (0.232)
Randomized Experimental -0.009 -0.008 -0.065 -0.095 -0.314 Design (0.020) (0.019) (0.170) (0.170) (0.332)
Square Root of Sample Size -0.003 0.001 0.159 0.098 0.484(0.042) (0.037) (0.184) (0.191) (0.706)
Published Article -0.024 -0.026 -0.203 -0.213 -0.41(0.017) (0.017) (0.133) (0.132) (0.254)
Citations Rank Index -0.001 -0.001 0.007 0.005 -0.005(0.002) (0.001) (0.012) (0.012) (0.024)
R-squared/ Log Likelihood 0.36 0.40 -765 -752 -283Notes: Standard errors (clustered by study) in parentheses. Coefficients in columns 1 and 2 are from models reported in columns 2 and 4 of Table 5. Coefficients in columns 3-5 are from models reported in columns 2, 4, and 5 of Table 6. See notes to Tables 5 and 6 for more information.
Program Effect - OLS Models Specifications in Table 5 (cols 2,4)
Sign/Significance - Ordered Probit Models Specifications in Table 6 (cols 2,4,5)
Program Duration - Dummy if > 9 months
Table 8: Tests for Publication Bias (Funnel Asymmetry Tests)
Control for Time
Horizon Only
Basic Controls (col.
2 of Table 5)
Interacted Controls
(col. 4 of Table 5)
PPS Fixed Effects (col.
5 of Table 5)
(1) (2) (3) (4)
Estimated coefficient of sampling error of estimated program effect:
Unweighted OLS 0.007 0.009 0.009 0.001
(0.005) (0.008) (0.008) (0.014)
Precision‐Weighted 0.022 0.035 0.032 ‐0.005
(0.031) (0.024) (0.020) (0.019)
Notes: Entry in each row and column corresponds to estimated coefficient of sampling error of estimated
program effect from a different specification. Models in column 1 include only a constant and dummies for
medium term and long term time horizon as additional controls; models in columns 2, 3 and 4 include same
controls included in specifications in columns 2, 4 and 5 of Table 5, respectively. Unweighted models are fit by
OLS. Precision‐weighted models are fit by weighted least squares using Winsorized inverse sampling variance
of estimated program effect as weight. Weight is Winsorized at 10th and 90th percentiles, respectively,
corresponding to values of 19.2 and 10000, respectively. Standard errors, clustered by study, in parentheses.
Table 9: Comparison of Relative Impacts of Different Program Types on Different Participant Groups
All Job Search Private Sector Public SectorProgram Types Training Assistance Job/Subsidy Employment Other
(1) (2) (3) (4) (5) (6)
Number Estimates 352 217 36 46 32 21Number of Studies 83 51 15 19 14 8Mean Effect Size (×100) 4.52 4.71 1.48 9.91 -1.83 5.65
Constant 0.024 0.020 -0.003 -0.053 -0.004 0.106(0.015) (0.014) (0.017) (0.043) (0.027) (0.022)
Medium Term 0.032 0.041 0.017 0.028 0.020 0.004(0.009) (0.010) (0.014) (0.045) (0.018) (0.019)
Long Term 0.054 0.055 0.005 0.133 0.009 -0.011(0.016) (0.016) (0.012) (0.068) (0.023) (0.014)
Intake Group (Base=Regular UI Recipients)Disadvantaged -0.002 -0.020 0.037 0.053 (omitted) 0.030
(0.019) (0.020) (0.017) (0.026) (0.030)Long Term Unemployment 0.072 0.122 0.024 0.088 0.029 -0.109
(0.032) (0.059) (0.018) (0.038) (0.030) (0.019)Gender Group (Base=Mixed)Male 0.019 0.028 (omitted) 0.110 -0.048 -0.022
(0.020) (0.025) (0.069) (0.031) (0.023)Female 0.054 0.058 (omitted) 0.163 -0.024 -0.111
(0.022) (0.028) (0.050) (0.030) (0.025)Age Group (Base=Mixed)Youth -0.037 -0.030 0.009 0.034 -0.059 (omitted)
0.018 0.021 0.011 0.037 0.030Older Participants -0.048 -0.059 0.016 -0.094 -0.047 0.046
(0.019) (0.024) (0.021) (0.053) (0.038) (0.002)
Controls for Program Typea Yes No No No No No
aFour dummies for different types of programs included.
Notes: standard errors, clustered by study, in parenthesis. See note to Table 5.
Table 10: Impacts of Macro Conditions on the Effectiveness of ALMP's
Baseline +Country Effects +GDP Growth Baseline +GDP Growth +Unemp. Rate
(1) (2) (3) (4) (5) (6)
Medium Term 0.032 0.030 0.028 0.043 0.034 0.040
(0.009) (0.009) (0.009) (0.009) (0.008) (0.009)
Long Term 0.054 0.046 0.040 0.056 0.031 0.048
(0.016) (0.017) (0.015) (0.020) (0.014) (0.020)
GDP Growth Rate (%) ‐‐ ‐‐ ‐0.010 ‐‐ ‐0.032 0.034
(Unemp. Rate in col. 6) (0.006) (0.008) (0.011)
Program Type (Base=Training)
Job Search Assistance ‐0.033 ‐0.053 ‐0.056 ‐0.076 ‐0.103 0.010
(0.017) (0.028) (0.027) (0.034) (0.023) (0.069)
Private Sector Job/Subsidy 0.031 0.027 0.019 0.020 ‐0.002 0.012
(0.026) (0.028) (0.028) (0.029) (0.024) (0.031)
Public Sector Employment ‐0.079 ‐0.074 ‐0.069 ‐0.096 ‐0.073 ‐0.102
(0.019) (0.027) (0.025) (0.029) (0.024) (0.025)
Other Programs ‐0.015 ‐0.045 ‐0.066 ‐0.096 ‐0.188 ‐0.104
(0.033) (0.027) (0.033) (0.034) (0.046) (0.050)
Intake Group (Base=Regular UI Recipients)
Disadvantaged ‐0.002 0.000 0.010 0.046 0.106 0.050
(0.019) (0.034) (0.033) (0.028) (0.035) (0.027)
Long Term Unemployed 0.072 0.098 0.095 0.112 0.109 0.092
(0.032) (0.033) (0.031) (0.034) (0.027) (0.033)
Gender Group (Base=Mixed)
Female 0.019 0.048 0.053 0.066 0.081 0.052
(0.020) (0.025) (0.024) (0.026) (0.022) (0.022)
Male 0.054 0.086 0.092 0.094 0.111 0.081
(0.022) (0.030) (0.030) (0.033) (0.030) (0.029)
Age Group (Base=Mixed)
Youth ‐0.037 ‐0.026 ‐0.026 ‐0.024 ‐0.057 ‐0.040
(0.018) (0.018) (0.020) (0.023) (0.020) (0.051)
Older Participants ‐0.048 ‐0.055 ‐0.062 ‐0.073 ‐0.097 ‐0.066
(0.019) (0.024) (0.025) (0.028) (0.023) (0.023)
Country Dummies No Yes Yes Yes Yes Yes
All Available Program Effect Estimates Denmark, France, Germany, and US Only
Notes: standard errors, clustered by study, in parenthesis. Models in columns 1‐3 are fit to 352 program estimates from 83 studies, with mean of dependent
variable = 0.0452. Models in columns 4‐5 are fit to 200 program estimates from Denmark, France, Germany, and the U.S. from 38 studies, with mean of
dependent variable = 0.0423. Model in columns 6 is fit to 181 program estimates from the same four countries from 34 studies, with mean of dependent variable
= 0.0441.
Appendix Figure 1: Distribution of Program Estimates by Country
224
258
313241
1158
26
42253
11212443
192659
125
424
12388
213
84
166
6612
813
57
0 25 50 75 100 125 150 175 200 225 250 275
ArgentinaAustraliaAustriaBelgium
BrazilBulgariaCanadaChina
ColombiaCzech
DenmarkDominican
EstoniaFinlandFrance
GermanyHungary
IndiaIrelandIsraelItaly
JordanKoreaLatvia
MalawiMexico
NetherlandsNZ
NicaraguaNorwayPanama
PeruPoland
PortugalRomania
RussiaSerbia
SlovakiaSlovenia
South AfricaSpain
Sri LankaSweden
SwitzerlandTurkey
UKUS
Number of Program Estimates
0
5
10
15
20
25
30
-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4
Num
bero
fEstim
ates
RangeofEstimatedProgramEffect:MidpointofInterval
AppendixFigure2a:HistogramofShortTermEstimatedEffects
SignificantlyNegative
Insignificant
SignificantlyPositive
0
5
10
15
20
25
30
-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4
Num
bero
fEstim
ates
RangeofEstimatedProgramEffect:MidpointofInterval
AppendixFigure2b:HistogramofMediumTermEstimatedEffects
SignificantlyNegative
Insignificant
SignificantlyPositive
0
2
4
6
8
10
12
-0.12 -0.08 -0.04 0 0.04 0.08 0.12 0.16 0.2 0.24 0.28 0.32 0.36 0.4
Num
bero
fEstim
ates
RangeofEstimatedProgramEffect:MidpointofInterval
AppendixFigure2c:HistogramofLongTermEstimatedEffects
SignificantlyNegative
Insignificant
SignificantlyPositive
050
100
150
num
ber o
f est
imat
es
1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation
Training JSA Private Public
Appendix Figure 3a: Number of estimates by program type over time
0.2
.4.6
perc
enta
ge
1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation
positive insignificant negative
Sign/Significance Short Term
0.2
.4.6
perc
enta
ge
1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation
positive insignificant negative
Sign/Significance Medium Term0
.2.4
.6.8
perc
enta
ge
1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation
positive insignificant negative
Sign/Significance Long Term
0.0
5.1
.15
effe
ct s
ize
1985-1989 1990-1994 1995-1999 2000-2004 2005-2008Time of Program Operation
short run medium run long run
Program Effect
Appendix Figure 3b: Program Effects over Time
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Under-0.075
-0.075to-0.05
-0.05to-0.025
-0.025to0
0to0.025
0.025to0.05
0.05to0.075
0.075to0.10
0.10to0.125
0.125to0.15
0.15to0.175
0.175to0.20
0.20to0.225
0.225to0.25
0.25to0.275
0.275to0.30
0.30to0.325
0.325to0.35
0.35to0.375
Over0.375
Fractio
n
RangeofEstimatedProgramEffect
AppendixFigure4:EffectDistributionsConditionalonSign/SignificanceFemalesvs.OtherParticipantGroups
Males/MixedGender- Sign.Negative Females- Sign.NegativeMale/MixedGender- Insignficant Females- InsignificantMales/MixedGender- Sign.Positive Females- Sign.Positive
Note:figureplotshistogramsforestimatedprogrameffectsonprobabilityofemployment,classifyingparticipantgroupsaseitherfemaleormale/mixedgender,andclassifyingestimatedprogrameffectsasnegativeandsignificant,insignificant,orpositiveandsignificant.Thereareatotalof352estimatedprogrameffects:72forfemalesand280formales/mixedgender.
Appendix Table 1a: Transitions between Sign‐ and Significance Categories, All Estimates
Short‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=80) 84 16 0
Insignificant (N=95) 37 60 3
Significantly Negative (N=50) 20 46 34
Short‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=39) 72 23 5
Insignificant (N=39) 46 51 3
Significantly Negative (N=22) 32 55 14
Medium‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=58) 88 10 2
Insignificant (N=35) 14 83 3
Significantly Negative (N=9) 22 33 44
Note: see notes to Table 2.
Short‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=31) 87 13 0
Insignificant (N=48) 31 67 2
Significantly Negative (N=26) 12 42 46
Short‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=15) 80 13 7
Insignificant (N=19) 37 63 0
Significantly Negative (N=9) 22 67 11
Medium‐term Estimates Significantly Positive Insignificant Significantly Negative
Significantly Positive (N=27) 89 11 0
Insignificant (N=17) 6 88 6
Significantly Negative (N=27) 0 67 33
Note: see notes to Table 2.
Percent of Long‐term Estimates
Percent of Long‐term Estimates
Appendix Table 1b: Transitions between Sign‐ and Significance Categories, Subsample with
Program Effects
Percent of Medium‐term Estimates
Percent of Medium‐term Estimates
Percent of Long‐term Estimates
Percent of Long‐term Estimates
Appendix Table 2a: Models for Standardized Effect Size
(1) (2) (3) (4)
Effect Term (Omitted = Short Term)
Medium Term 0.071 0.056 0.101 0.088
(0.027) (0.021) (0.037) (0.025)
Long Term 0.131 0.091 0.097 0.099
(0.044) (0.038) (0.040) (0.040)
Program Type (Omitted = Training)
Job search Assist. ‐0.059 ‐0.012 0.002 0.029
(0.027) (0.043) (0.026) (0.044)
Private Subsidy 0.094 0.086 ‐0.007 0.044
(0.068) (0.057) (0.091) (0.099)
Public Sector Emp. ‐0.120 ‐0.152 ‐0.081 ‐0.084
(0.034) (0.044) (0.055) (0.062)
Other 0.036 0.007 0.139 0.108
(0.071) (0.094) (0.068) (0.098)
Interaction with Medium Term:
Job search Assist. ‐0.098 ‐0.092
(0.043) (0.041)
Private Subsidy ‐0.016 ‐0.055
(0.102) (0.104)
Public Sector Emp. ‐0.081 ‐0.09
(0.070) (0.073)
Other ‐0.133 ‐0.105
(0.048) (0.045)
Interaction with Long Term :
Job search Assist. ‐0.115 ‐0.083
(0.041) (0.052)
Private Subsidy 0.329 0.182
(0.142) (0.127)
Public Sector Emp. ‐0.030 ‐0.156
(0.081) (0.108)
Other ‐0.239 ‐0.273
(0.073) (0.092)
Additional Controls No Yes No Yes
Number of Observations 352 352 352 352
R Squared 0.13 0.33 0.21 0.37
Notes: Standard errors (clustered by study) in parentheses. Models are linear
regressions with the standardized effect size (program effect on probability of
employment, divided by standard deviation of employment rate of comparison
group) as dependent variable. Additional regressors see Table 7.
Appendix Table 2b: Models for Proportional Program Effects
(1) (2) (3) (4)
Effect Term (Omitted = Short Term)
Medium Term 0.062 0.037 0.134 0.111
(0.057) (0.054) (0.059) (0.037)
Long Term 0.120 0.053 0.103 0.117
(0.083) (0.079) (0.051) (0.044)
Program Type (Omitted = Training)
Job search Assist. ‐0.102 ‐0.086 ‐0.023 ‐0.049
(0.043) (0.090) (0.053) (0.096)
Private Subsidy 0.157 0.09 0.019 0.075
(0.119) (0.100) (0.185) (0.202)
Public Sector Emp. ‐0.021 ‐0.113 0.204 0.17
(0.159) (0.116) (0.379) (0.350)
Other 0.100 ‐0.073 0.265 0.069
(0.172) (0.213) (0.171) (0.218)
Interaction with Medium Term:
Job search Assist. ‐0.136 ‐0.101
(0.073) (0.066)
Private Subsidy ‐0.031 ‐0.133
(0.194) (0.198)
Public Sector Emp. ‐0.437 ‐0.439
(0.381) (0.399)
Other ‐0.19 ‐0.123
(0.080) (0.077)
Interaction with Long Term :
Job search Assist. ‐0.13 ‐0.075
(0.063) (0.082)
Private Subsidy 0.45 0.16
(0.291) (0.240)
Public Sector Emp. ‐0.307 ‐0.619
(0.410) (0.513)
Other ‐0.437 ‐0.497
(0.171) (0.188)
Additional Controls No Yes No Yes
Number of Observations 352 352 352 352
R Squared 0.04 0.23 0.09 0.28
Notes: Standard errors (clustered by study) in parentheses. Models are linear
regressions with the proportional effect size (program effect on probability of
employment, divided by mean employment rate of comparison group) as dependent
variable. Additional regressors see Table 7.
Appendix Table 3: Alternative Models for Subsample with Estimated Program Effect
(1) (2) (3) (4)
Medium Term 0.029 0.489 0.429 ‐0.719
(0.009) (0.114) (0.137) (0.173)
Long Term 0.045 0.969 0.851 ‐1.365
(0.015) (0.221) (0.218) (0.437)
Program Type (Omitted = Training)
Job search Assist. ‐0.009 0.443 0.507 ‐0.531
(0.020) (0.436) (0.434) (0.528)
Private Subsidy 0.042 0.252 0.733 0.475
(0.026) (0.300) (0.314) (0.315)
Public Sector Emp. ‐0.08 ‐1.356 ‐1.239 1.349
(0.020) (0.287) (0.330) (0.335)
Other 0.003 0.55 0.987 0.595
(0.036) (0.542) (0.492) (0.628)
Age of Program Group (Omitted = Mixed)
Youths (<25) ‐0.031 ‐0.614 ‐0.643 1.052
(0.018) (0.282) (0.322) (0.499)
Older (>=25) ‐0.07 ‐0.735 ‐0.657 0.845
(0.020) (0.280) (0.272) (0.428)
Gender of Program Group (Omitted = Mixed)
Males only 0.011 ‐0.289 ‐0.591 ‐0.101
(0.022) (0.273) (0.310) (0.279)
Females only 0.047 0.043 ‐0.196 ‐0.476
(0.023) (0.251) (0.272) (0.315)
Country Group (Omitted = Nordic)
Germanic 0.056 1.033 1.043 ‐1.351
(0.027) (0.460) (0.498) (0.454)
Anglo ‐0.026 1.265 1.312 ‐‐
(0.030) (0.577) (0.625)
East Europe 0.022 0.615 0.644 ‐0.579
(0.031) (0.358) (0.447) (0.437)
Rest of Europe 0.016 0.825 0.909 ‐0.984
(0.024) (0.469) (0.560) (0.456)
Latin America 0.009 1.017 1.334 ‐1.075
(0.053) (0.826) (0.879) (1.170)
Remaining Countries 0.035 1.161 1.138 ‐‐
(0.035) (0.521) (0.618)
Type of Program Participant (Omitted = Registered Unemployed)
Disadvantaged 0.018 0.428 0.294 ‐0.99
(0.036) (0.618) (0.588) (1.022)
Long‐term Unemployed 0.083 0.436 0.481 ‐0.512
(0.032) (0.311) (0.325) (0.332)
Program Duration > 9 Months ‐0.029 ‐0.599 ‐0.526 0.563
(0.016) (0.234) (0.265) (0.326)
Experiment ‐0.009 ‐0.312 ‐0.677 ‐0.95
(0.020) (0.330) (0.395) (0.461)
Square Root of Samplesize ‐0.003 0.471 0.796 0.817
(0.042) (0.689) (0.851) (0.719)
Published Article ‐0.024 ‐0.374 ‐0.41 0.328
(0.017) (0.252) (0.277) (0.298)
Citations Rank Index ‐0.001 ‐0.009 0.003 0.057
(0.002) (0.023) (0.024) (0.034)
Number of Observations 352 352 352 315
R Squared/ Log Likelihood 0.356 ‐288 ‐190 ‐92
Notes: Standard errors (clustered by study) in parentheses. Dependent variables are: estimated program effect in column
1; ‐1, 0 or 1 indicating sign/significance in column 2; indicator for significant positive effect in column 3; indicator for
significant negative effect in column 4. Sample in column 4 excludes observations that are perfectly predicted.
Probit for Sig.
Positive
Probit for Sig.
Negative
Ordered Probit for
Sign/Significance
OLS Model for
Program Effect
Appendix Table 4: Re‐normalized Program Effects on Probability of Employment
(1) (2)
Program Type = Training
Short Term 0.020 0.010
(0.009) (0.010)
Medium Term 0.066 0.053
(0.017) (0.014)
Long Term 0.067 0.058
(0.018) (0.018)
Program Type=Job Search Assistance
Short Term 0.012 0.019
(0.010) (0.019)
Medium Term 0.020 0.029
(0.008) (0.017)
Long Term 0.011 0.037
(0.016) (0.021)
Program Type=Private Subsidy
Short Term 0.011 0.021
(0.037) (0.038)
Medium Term 0.062 0.052
(0.027) (0.032)
Long Term 0.211 0.192
(0.043) (0.038)
Program Type=Public Sector Employment
Short Term ‐0.036 ‐0.045
(0.013) (0.012)
Medium Term ‐0.011 ‐0.020
(0.011) (0.021)
Long Term 0.008 ‐0.028
(0.016) (0.029)
Program Type=Other
Short Term 0.072 0.073
(0.024) (0.039)
Medium Term 0.058 0.064
(0.022) (0.038)
Long Term 0.020 0.015
(0.028) (0.050)
Additional Controls No Yes
Notes: Sample size is 352 estimates. Standard errors (clustered by study) in
parentheses. Models correspond to specifications estimated in columns 3 and 4 of
Table 5, but are re‐normalized to give estimated impacts by program type and time
horizon (at mean values of all covariates for model in column 2).
AppendixTable5:ImpactsofMacroConditionsontheEffectivenessofALMP's
(1) (2) (3) (4) (5) (6) (7) (8)
IndicatorforMediumTermImpact 0.028 0.032 0.032 0.036 0.034 0.041 0.040 0.039(0.009) (0.009) (0.011) (0.011) (0.008) (0.008) (0.009) (0.009)
IndicatorforLongTermImpact 0.040 0.035 0.046 0.049 0.031 0.035 0.048 0.050(0.015) (0.013) (0.018) (0.018) (0.014) (0.013) (0.020) (0.019)
MeasuresofMacroconditions:
Conditionsoverprogramperiod -0.010 0.006 -0.032 0.034(0.006) (0.007) (0.008) (0.011)
Conditionsinfirstyearofprogram -0.006 0.012 -0.007 0.022(0.003) (0.007) (0.004) (0.013)
Conditionsinyearafterprogramend 0.013 -0.003 0.020 0.011(0.006) (0.006) (0.006) (0.011)
NumberofObservations 352 351 333 314 200 200 181 175
CountryDummies Yes Yes Yes Yes Yes Yes Yes YesNotes:standarderrors,clusteredbystudy,inparenthesis.Macroconditionsaremeasuredasaveragesovertheyearsofprogramimplementationincolumns(1),(3),(5),(7).Conditionsaremeasuredinthefirstyearofprogramimplementationandtheyearaftertheendofprogramimplementationincolumns(2),(4),(6),(8).AllmodelsincludecountryfixedeffectsandthesamesetofcovariatesasinTable10.
AllAvailableProgramEffectEstimates: EffectsforDenmark,France,Germany,andUSOnly:
GDPGrowth UnemploymentRate GDPGrowth UnemploymentRate
Recommended