Tracy Dyke Redmond Senior Associate June 23, 2011

INDUSTRIAL ECONOMICS, INCORPORATED

IEc


Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi-Experimental Evaluation Design

Tracy Dyke RedmondSenior Associate

June 23, 2011


Typology of Evaluation Designs*

Experimental Design

Quasi-Experimental Design

Non-Experimental Design

2

IncreasingStatistical Strength*

*Notnecessarilyoverall evaluation strength

• Pre-test + post-test comparison with statistical matching

• Regression discontinuity• Pre-test + post-test

comparison with judgmental matching

• Pipeline control group• Pre-test + post-test

comparison with delayed baseline

• Pre-test + post-test treatment group with post-test only comparison group

• Post-test only for treatment and comparison group

* Not all possible evaluation designs shownAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006.


“Statistically Valid” Pilot Project for Auto Body Shops








Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas

Pro:•Flexible•Reasonably good estimate of project impact when there are good matching criteriaCon:•Assumes comparison group similar to treatment group and equally willing to participate•Does not assess project implementationAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.










Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1

Pro:•Does not require external control group; design relatively inexpensive and easy to useCon:•Assumes phase 1 and 2 groups are similar (which may not be true)•Requires that phase 2 group does not have access to phase 1 treatment (spillover problem)

Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.


• Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations

• Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule)

• Requirements: • Conduct representative measurement of all regulated

entities, not just voluntary participants

• Do not prevent shops from receiving compliance assistance (or not for very long)

• Also test phone survey validity (that methodology not covered in this presentation)



Control and Comparison Groups

• Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule

• Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign

• Treatment and control group in Massachusetts: randomly assigned, considered equivalent

• Includes auto body shops in eastern MA with elevated risk

• Excludes communities with pre-existing aggressive assistance/enforcement campaigns

• Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of:

• No expected compliance assistance from EPA or state

• State regulations related to RCRA and air emissions

• Number of shops located in areas of elevated-risk


“Statistically Valid” Pilot Project Design

MA - A

MA - B

VA

October 2009 –January 2010

March – Early July 2010

ComplianceAssistance (CA):• Mailings• Workshops/ Webinars

On-site surveysFollowed by on-site CA

On-site surveys

Summer 2010-January 2011

CA:• Mailings• Webinars









MA - A

MA - B

VA





On-site surveys









Compare randomly assigned treatment vs. control group in MA



MA - A

MA - B

VA





On-site surveys









Compare “difference-in-differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA


Methodological Notes and Next Steps

• Among valid shops, response rates between 80 – 85% for both states and both years

• However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project

• Evidence of shops not on the list, operating “under the radar” in VA

• Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars)

• Currently working to analyze 2011 data and develop comparisons; report expected in late 2011


Thanks to:

• EPA HQ for developing the vision and supporting the project

• EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits

• EPA Region 3, Virginia, and Massachusetts for participating

• ERG and Ski Fabyanic for conducting hundreds of site visits

• Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data

• The auto body shops, for letting us in the door!


IEcINDUSTRIAL ECONOMICS, INCORPORATED

617.354.0074

Documents

Tracy Dyke Redmond Senior Associate June 23, 2011