vvEPA United States Environmental Protection Agency Office of Policy (1807T) November 2012 EPA-100-F-12-004 Measuring the Effects of EPA Compliance Assistance in the Auto Body Sector: A Statistically Valid Pilot Project Fact Sheet http://www.epa.gov/evaluate For more information on this and other completed evaluations at EPA or the Evaluation Support Division, visit the above link. Introduction • This report describes the results of a study designed to assess the impact of compliance assistance efforts offered by EPA Region 1 to the auto body sector, prior to the compliance date for a new EPA air regulation. • EPA provided the compliance assistance in 2009-2010 through mailings, workshops, webinars, and site visits. The compliance assistance focused primarily on spray coating operations and hazardous waste storage by auto body shops. Evaluation Questions • Did EPA Region 1 's compliance assistance activities contribute to behavior change in the auto body sector? • Is the telephone survey a valid and reliable technique for performance measurement and program evaluation? • Are the measurement methods employed in the pilot transferable to other assistance activities? • What specific characteristics of the auto body sector influence the transferability of the measurement approach in this evaluation? Evaluation Methods • The study assessed the impacts of EPA compliance assistance in this sector using probability sampling, random assignment (i.e., to treatment and control groups), and on-site observations. The study also assessed the validity of gathering information about facility performance through phone surveys. The study was designed, to the extent possible, to address the challenges of self-selection bias, non-response bias, and self-reporting bias in assessing the effects of the Agency's compliance assistance. Key Findings Effectiveness of EPA Compliance Assistance • The random assignment experiment did not find evidence that EPA assistance to auto body shops affected sector-wide performance in the short term. A simple comparison of the groups' performance levels shows statistically significant differences for two performance measures, but the differences were too small to be of practical significance. • In the short term study, shops that chose to participate in workshops/webinars (15 percent of the treatment group in the sample) performed significantly better on five measures than the remaining shops in the treatment group that did not avail themselves of those opportunities. However, it is not possible to separate out the impact of the workshops/webinars relative to the potential effect of self-selection bias. ------- Effectiveness of EPA Compliance Assistance (continued) • The quasi-experiment suggests that overall impact of EPA assistance was minimal for the performance measures evaluated over the longer term (approximately one year). After controlling for shop characteristics that could influence performance, three of the seventeen performance measures showed statistically significant evidence that shops that were offered compliance assistance performed better than shops that did not. Validity of Telephone Survey Data for Assessing Performance • This study finds that, while the phone survey results were similar to the site visit results for the majority of the performance measures examined, very large differences were observed for several performance measures. The study finds that self-reporting bias was more of a concern than non- response bias. Transferability of Measurement Methods • Several measurement methods used in this study may be broadly transferable, including (1) obtaining representative data on baseline performance, (2) using phone surveys to assess baseline performance (though further study would be required to better understand the circumstances under which telephone survey results may be relatively reliable); and (3) delaying treatment for a randomly assigned group of entities in order to establish a control group, and then providing treatment to these entities as needed after measurement is complete. • Sector characteristics will influence the transferability of these measurement approaches. For example, it is more difficult to draw statistically-based samples in sectors with high turnover rate of businesses. Conclusions Overall, this evaluation found that EPA compliance assistance had only minimal impact on the auto body sector as a whole. Potential explanations include the following: • The direct assistance provided by EPA may not have been effective in influencing the targeted sector. • The performance of auto body shops appears to have been positively influenced by vendors and suppliers, potentially dampening measurable impacts of EPA assistance provided directly to auto body shops. It is possible that the indirect approach of influencing auto body shops by disseminating information through vendors and suppliers is more effective than direct assistance from EPA. • Despite considerable outreach efforts by EPA Region 1, fewer than 20 percent of the shops in the treatment group received interactive assistance during the study (i.e., workshops, webinars, or site visits). Thus, even if the interactive assistance was extremely effective for the shops that received it, the impact may be difficult to detect when this small group of shops is pooled with the remainder of the auto body population. • For many of the performance measures evaluated, baseline performance was high, leaving little room for performance improvement. The auto body sector in the treatment group had been exposed to considerable government assistance efforts over the last few decades, which may have limited the impact of additional assistance. Contact(s) • Terell Lasane, Office of Policy, Evaluation Support Division, Lasane.Terell@.epamail.epa.gov • Mary Dever-Putnam, US EPA Region 1, Dever.Marv@epamail.epa.gov Report Link: http://www.epa.qov/evaluate/reports/ Date Completed: May 2012 ------- |