vvEPA
 United States
 Environmental Protection
 Agency
                                                         Office of Policy
                                                         (1807T)
November 2012
EPA-100-F-12-004
    Measuring the Effects of EPA Compliance
    Assistance in the Auto Body Sector:
    A Statistically Valid Pilot Project
  Fact  Sheet
http://www.epa.gov/evaluate
For more information on this and
other completed evaluations at
EPA or the Evaluation Support
Division, visit the above link.
Introduction
•  This report describes the results of a study designed to assess the impact of
   compliance assistance efforts offered by EPA Region 1 to the auto body sector,
   prior to the compliance date for a new EPA air regulation.
•  EPA provided the compliance assistance in 2009-2010 through mailings,
   workshops, webinars, and site visits. The compliance assistance focused primarily
   on spray coating operations and hazardous waste storage by auto body shops.

Evaluation Questions
•  Did EPA Region 1 's compliance assistance activities contribute to behavior change
   in the auto body sector?
•  Is the telephone survey a valid and reliable technique for performance
   measurement and program evaluation?
•  Are the measurement methods employed in the pilot transferable to other
   assistance activities?
•  What specific characteristics of the auto body sector influence the transferability of
   the measurement approach in this evaluation?

Evaluation Methods
•  The study assessed the impacts of EPA compliance assistance in this sector using
   probability sampling, random assignment (i.e., to treatment and control groups),
   and on-site  observations.  The study also assessed the validity of gathering
   information  about facility performance through phone surveys. The study was
   designed, to the extent possible, to address the challenges of self-selection bias,
   non-response bias, and self-reporting bias in assessing the effects of the Agency's
   compliance assistance.

Key Findings

  Effectiveness of EPA Compliance Assistance
•  The random assignment experiment did not find evidence that EPA assistance to
   auto body shops affected sector-wide performance in the short term. A simple
   comparison of the groups' performance levels shows statistically significant
   differences  for two performance measures, but the differences were too small to be
   of practical significance.
•  In the short term study, shops that chose to participate in workshops/webinars (15
   percent of the treatment group in the sample) performed significantly better on five
   measures than the remaining shops  in the treatment group that did not avail
   themselves of those opportunities. However,  it is not possible to separate out the
   impact of the workshops/webinars relative to  the potential effect of self-selection
   bias.

-------
  Effectiveness of EPA Compliance Assistance (continued)
•   The quasi-experiment suggests that overall impact of EPA assistance was minimal for the
    performance measures evaluated over the longer term (approximately one year).  After controlling
    for shop characteristics that could influence performance, three of the seventeen performance
    measures showed statistically significant evidence that shops that were offered compliance
    assistance performed better than shops that did not.
  Validity of Telephone Survey Data for Assessing Performance
•   This study finds that, while the phone survey results were similar to the site visit results for the
    majority of the performance measures examined, very large differences were observed for several
    performance measures. The study finds that self-reporting bias was more of a concern than  non-
    response bias.

  Transferability of Measurement Methods
•   Several measurement methods used in this study may be broadly transferable, including (1) obtaining
    representative data on baseline performance, (2) using phone surveys to assess baseline
    performance (though further study would be required to better understand the circumstances  under
    which telephone survey results may be relatively reliable); and (3) delaying treatment for a randomly
    assigned group of entities in order to establish a control group, and then providing treatment to these
    entities as needed after measurement is complete.
•   Sector characteristics will influence the transferability of these measurement approaches.  For
    example, it is more difficult to draw statistically-based samples in sectors with high turnover rate of
    businesses.

Conclusions
Overall, this evaluation found that EPA compliance assistance had only minimal impact on the auto body
sector as a whole. Potential explanations include the following:
•   The direct assistance provided by EPA may not have been effective in influencing the targeted sector.
•   The performance of auto body shops appears to have been positively influenced by vendors and
    suppliers, potentially dampening measurable impacts of EPA assistance provided  directly to auto
    body shops. It is possible that the  indirect  approach of influencing auto body shops by disseminating
    information through vendors and suppliers  is more effective than direct assistance from EPA.
•   Despite considerable outreach efforts by EPA Region 1, fewer than 20 percent of the shops in the
    treatment group  received interactive assistance during the study (i.e.,  workshops, webinars, or site
    visits). Thus, even if the interactive assistance was  extremely effective for the shops that received it,
    the impact may be difficult to detect when this small group of shops is pooled with  the remainder of
    the auto body population.
•   For many of the  performance measures evaluated,  baseline performance was high, leaving little room
    for performance  improvement. The auto body sector in the treatment  group had been exposed to
    considerable government assistance efforts over the last few decades, which may have limited the
    impact of additional assistance.

Contact(s)
•   Terell Lasane, Office of Policy, Evaluation  Support Division, Lasane.Terell@.epamail.epa.gov
•   Mary Dever-Putnam, US EPA Region 1,  Dever.Marv@epamail.epa.gov
Report Link: http://www.epa.qov/evaluate/reports/

Date  Completed: May 2012

-------