Evaluation  of Three
                 Environmental  Results
                 Programs  (ERPs)
                                                                                    September 2009
 Fact Sheet

For more information on
this and other completed
evaluations at EPA or the
Evaluation Support Division,
visit the above link.
    There are an estimated 35,000 to 80,000 facilities across the country that are in
     the business of repairing and refinishing vehicles, particularly cars. Auto body
     shops present a wide array of environmental concerns, from use and emissions of
     hazardous materials such as methylene chloride, to discharges of polluted water
     into water systems, and worker exposure to toxic solvents and particulate matter.
    The National Emission Standards for Hazardous Air Pollutants: Paint Stripping and
     Miscellaneous Surface Coating Operations at Area Sources, is a regulation that
     existing shops must comply with by January, 2011 and that new shops must
     comply with by January, 2008 or upon startup of operations. The rule requires
     that, for example: all spray painting must be done in a spray booth, painters
     must use spray guns and techniques that reduce overspray, all painters must
     receive training, and paint spray gun cleaning cannot release any mist of cleaning
     solvent to the air.
    States are currently considering how to implement the new regulation; ERP is
     one potential policy tool for implementation. ERP is an innovative approach to
     improving facilities' management practices within small business sectors. ERP is
     an integrated system that incorporates plain language compliance assistance that
     promotes pollution prevention, facility self-assessment and self-certification,
     agency inspections, statistically-based performance measurement, and where
     necessary, comprehensive facility investigations and targeted enforcement
    The purpose of this evaluation is to inform states and EPA regions who are
     currently considering developing programs to encourage auto body shops to
     adopt best practices and improve compliance with environmental regulations.

Evaluation Questions

    To what extent have the ERPs in DE, RI, and ME led to actual and/or expected
     adoption of selected best practices that reduce the environmental footprint of
     auto body shops?
    What environmental and health outcomes are estimated to result from auto body
     shops implementing these best practices?
    What are the cost implications of each program for regulators and auto body
     shops initially and over time?
    What is the cost-effectiveness of each program?
    Overall, what are the advantages and disadvantages of each of the three ERPs in
     terms of reaching auto body shops, generating environmental and worker health
     outcomes, and achieving cost-effective results?
    What factors influenced the outcomes of each program (e.g., existing or
     impending regulations, regulatory/assistance offices involved in conducting the
     program, and extent of coordination with industry representatives)?

Evaluation Questions Con't.

     What do our findings suggest regarding the circumstances under which ERPs are
      likely to produce cost-effective results?
     What is the current status of each program?

     What are the primary implementation challenges states faced in developing and
      implementing their programs?

     What factors influenced  the states' decisions to continue, not continue, or modify
      their programs after initial pilots?

     How does the state and/or EPA Region involved in implementing each  program
      view the program's results, and why?

Evaluation Methods

     The evaluation used a non-experimental design known as "One Group
      Pretest/Posttest," which involves measurement or observation of a group of
      subjects (auto body shops) prior to and after the application of an intervention
      (the ERP). There were no control groups used, i.e., the states did not  measure
      performance in a control group of auto body shops not subject to the ERP.
     To assess the  behavioral and  environmental outcomes of the ERPs, the
      evaluation primarily relied  on  existing data reported by each program,  which
      captured  what percentage of facilities use certain best practices during
      independent, random inspections before and after ERP implementation. The
      analysis compares the percentage of shops using certain best practices before
      and after the ERP  program and translates those changes in behaviors to
      environmental outcomes, where possible. Where this translation is not  possible,
      the evaluation qualitatively discuss the benefits one would expect facilities would
      observe as a result of adopting these behaviors.

     ERP states did not collect quantitative data on long-term outcomes (e.g.,
      emissions reductions), and therefore the evaluation was limited to estimating this
      information where possible.

     To assess the  remaining evaluation questions, the  evaluation relies on  interviews
      with state program staff, EPA staff involved in supporting ERP, representatives of
      the States' ERP Consortium, and selected auto body shops.  For example, these
      interviews provided insights on program status, costs, factors that influenced
      outcomes, and implementation experiences.

Key Findings

     The evaluation findings suggest that ERP is associated with improved business
      practices in the auto body sector and is regarded as successful by both state and
      industry representatives. Quantifying environmental outcomes associated with
      ERP is difficult, and those outcomes that were quantified were relatively small. In
      addition, sustaining the program has proven to be difficult given resource
      constraints and overall regulatory priorities.

     Each state selected between 19 and 24 indicators of environmental performance.
      States observed improved  performance between the samples of facilities
      measured for the vast majority of indicators (observed performance

Key Findings Con't.

      improved for 54 out of 65 indicators (83 percent) between baseline and post-
      certification). Of these 54 indicators, 29 (54 percent of the indicators where
      observed performance improved, and 45 percent of all indicators) were found
      to have statistically significant changes in performance. In addition, no
      statistically significant declines in performance were found.

     For the measures where there were statistically significant improvements, one
      can infer that there was an increase in the proportion of the entire population
      of auto body shops in the state following best practices. In other words, for
      nearly half of the measured indicators, the evaluation findings suggest that the
      auto body sector as a whole is shifting to increased use of best practices,  not
      just those shops that were included in the samples measured.

     The greatest percentage of the total number of indicators with statistically
      significant improvements was observed in the air emissions and worker health
      and safety categories. In both of these categories, half of the total set of
      indicators measured (not solely those improving, but the total number of
      indicators) showed statistically significant improvements.

     With regard to air  emissions, the evaluation estimates that improvements in
      usage of low-VOC/waterbased solvents may have reduced VOC emissions by as
      much as 1.7 tons per year for all auto body shops in each of two states:
      Delaware and Maine (actual amounts could be less). In addition, estimated
      material usage associated with auto body shop spray gun cleaning operations
      declined by between 0.6 and 2.0 tons per year for all shops in Delaware.
      Finally, estimates are that improvements in hazardous waste  management in
      Delaware may have increased the amount of hazardous waste being properly
      identified by as much  as 22,440 pounds per month (actual amounts are likely
      to be less). All three of these estimates rely on numerous assumptions, and
      have considerable uncertainties, which are described in the full evaluation
      report. The evaluation was unable to quantify reductions in water discharges or
      improvements to worker health and safety.

     Note that while the overall trend  in improving performance measured by states
      is consistent with the hypothesis that ERP leads to adoption of selected best
      practices, one cannot be certain of the extent to which the states' ERP alone
      caused or contributed to the observed changes in performance, in comparison
      to other factors. Other factors happening concurrently with ERP may have also
      contributed to the observed changes  in performance.  For example, in Rhode
      Island ERP was implemented  in tandem with the  auto body shop license
      renewal process. In Delaware ERP was developed in tandem with a source
      category permit for this sector.

     The three states included in the evaluation spent a range of resources
      developing and implementing their ERPs. Estimated government costs
      (including state resources and an EPA State Innovation Grant) are $800 to
      $2,000 per auto body shop in the population for states conducting a single
      cycle of ERP. These costs are not inclusive of all of the state resources spent on
      staff throughout the program. States also dedicated staff time that is not fully
      captured in these cost estimates. For Rhode Island, total costs range from $400
      to $700 per shop for two cycles of ERP (in other words, the costs per shop per
      cycle would be half as much).

Key Findings Con't.

    Compared to the likely outcomes of alternative approaches (e.g., following up
     on complaints or conducting infrequent inspections) that these three states
     considered, states perceive that ERP resulted in greater improvements in
     performance. These benefits of ERP typically come at a higher initial cost
     compared to these less resource-intensive alternatives. However, if states
     conducted successive rounds of ERP, it's likely that per facility costs would be
     substantially reduced, as Rhode Island has found.
    A key factor related to cost effectiveness is the number of facilities targeted by
     ERP. Specifically, ERP can be more cost effective for larger populations of
     facilities, so long as the population is relatively homogenous, with a common
     set of regulatory requirements and best practices.
    Another factor related to the likelihood of a state achieving cost-effective
     results through ERP is the degree to which state staff can build partnerships
     within (e.g., between different environmental media offices) and outside
     agency walls (e.g. Rhode Island's partnership with the University of Rhode
     Island). State staff within all of the ERPs described in this evaluation forged
     partnerships within and outside their agencies to implement the ERP.
     Successful design and implementation of ERPs that deal with multiple
     environmental  media require the cooperation among and  buy-in from the
     various offices that are affected. Outside partners may provide support in the
     form of technical expertise and/or funding.
    Baseline  performance and the  likely extent of improvement (i.e., to what extent
     are facilities likely to be able or willing to change their behavior) both have a
     direct effect on the extent to which statistically significant changes may be
    Key factors leading to success of an ERP include sufficient funding, the
     regulatory context in which the program was implemented, effective
     coordination and communication among involved offices, upper management
     buy-in, and continuing program support.
     Combine forces. States could work together to realize the economies of scale
      possible with ERP.
     Decide on a set of common indicators. It would be helpful to be able to
      compare and aggregate ERP data for the same sector across programs.
     Collect quantitative data on facility practices, not just information on
      the proportions of shops utilizing specific practices, for a small
      number of indicators. This could help quantify environmental outcomes.
     Develop a tool to help states estimate environmental outcomes.
     Un-package ERP. States could consider different ways to implement different
      components of ERP.
     Consider implementing ERP primarily where larger populations of
      facilities are present. This approach  has the potential to reduce per facility
      expenditures and increase the cost-effectiveness of the program.
     Develop a clearer agreement between EPA and states as to whether
      or not ERP can be used to address traditional regulatory programs.