vvEPA
United States
Environmental Protection
Agency
Office of Policy
(1807T)
November 2012
EPA-100-F-12-004
Measuring the Effects of EPA Compliance
Assistance in the Auto Body Sector:
A Statistically Valid Pilot Project
Fact Sheet
http://www.epa.gov/evaluate
For more information on this and
other completed evaluations at
EPA or the Evaluation Support
Division, visit the above link.
Introduction
• This report describes the results of a study designed to assess the impact of
compliance assistance efforts offered by EPA Region 1 to the auto body sector,
prior to the compliance date for a new EPA air regulation.
• EPA provided the compliance assistance in 2009-2010 through mailings,
workshops, webinars, and site visits. The compliance assistance focused primarily
on spray coating operations and hazardous waste storage by auto body shops.
Evaluation Questions
• Did EPA Region 1 's compliance assistance activities contribute to behavior change
in the auto body sector?
• Is the telephone survey a valid and reliable technique for performance
measurement and program evaluation?
• Are the measurement methods employed in the pilot transferable to other
assistance activities?
• What specific characteristics of the auto body sector influence the transferability of
the measurement approach in this evaluation?
Evaluation Methods
• The study assessed the impacts of EPA compliance assistance in this sector using
probability sampling, random assignment (i.e., to treatment and control groups),
and on-site observations. The study also assessed the validity of gathering
information about facility performance through phone surveys. The study was
designed, to the extent possible, to address the challenges of self-selection bias,
non-response bias, and self-reporting bias in assessing the effects of the Agency's
compliance assistance.
Key Findings
Effectiveness of EPA Compliance Assistance
• The random assignment experiment did not find evidence that EPA assistance to
auto body shops affected sector-wide performance in the short term. A simple
comparison of the groups' performance levels shows statistically significant
differences for two performance measures, but the differences were too small to be
of practical significance.
• In the short term study, shops that chose to participate in workshops/webinars (15
percent of the treatment group in the sample) performed significantly better on five
measures than the remaining shops in the treatment group that did not avail
themselves of those opportunities. However, it is not possible to separate out the
impact of the workshops/webinars relative to the potential effect of self-selection
bias.
-------
Effectiveness of EPA Compliance Assistance (continued)
• The quasi-experiment suggests that overall impact of EPA assistance was minimal for the
performance measures evaluated over the longer term (approximately one year). After controlling
for shop characteristics that could influence performance, three of the seventeen performance
measures showed statistically significant evidence that shops that were offered compliance
assistance performed better than shops that did not.
Validity of Telephone Survey Data for Assessing Performance
• This study finds that, while the phone survey results were similar to the site visit results for the
majority of the performance measures examined, very large differences were observed for several
performance measures. The study finds that self-reporting bias was more of a concern than non-
response bias.
Transferability of Measurement Methods
• Several measurement methods used in this study may be broadly transferable, including (1) obtaining
representative data on baseline performance, (2) using phone surveys to assess baseline
performance (though further study would be required to better understand the circumstances under
which telephone survey results may be relatively reliable); and (3) delaying treatment for a randomly
assigned group of entities in order to establish a control group, and then providing treatment to these
entities as needed after measurement is complete.
• Sector characteristics will influence the transferability of these measurement approaches. For
example, it is more difficult to draw statistically-based samples in sectors with high turnover rate of
businesses.
Conclusions
Overall, this evaluation found that EPA compliance assistance had only minimal impact on the auto body
sector as a whole. Potential explanations include the following:
• The direct assistance provided by EPA may not have been effective in influencing the targeted sector.
• The performance of auto body shops appears to have been positively influenced by vendors and
suppliers, potentially dampening measurable impacts of EPA assistance provided directly to auto
body shops. It is possible that the indirect approach of influencing auto body shops by disseminating
information through vendors and suppliers is more effective than direct assistance from EPA.
• Despite considerable outreach efforts by EPA Region 1, fewer than 20 percent of the shops in the
treatment group received interactive assistance during the study (i.e., workshops, webinars, or site
visits). Thus, even if the interactive assistance was extremely effective for the shops that received it,
the impact may be difficult to detect when this small group of shops is pooled with the remainder of
the auto body population.
• For many of the performance measures evaluated, baseline performance was high, leaving little room
for performance improvement. The auto body sector in the treatment group had been exposed to
considerable government assistance efforts over the last few decades, which may have limited the
impact of additional assistance.
Contact(s)
• Terell Lasane, Office of Policy, Evaluation Support Division, Lasane.Terell@.epamail.epa.gov
• Mary Dever-Putnam, US EPA Region 1, Dever.Marv@epamail.epa.gov
Report Link: http://www.epa.qov/evaluate/reports/
Date Completed: May 2012
------- |