X-/ER&
Evaluation of Three
Environmental Results
Programs (ERPs)
September 2009
EPA-100-F-09-056
Fact Sheet
Introduction
http://www.epa.gov/evaluate
For more information on
this and other completed
evaluations at EPA or the
Evaluation Support Division,
visit the above link.
• There are an estimated 35,000 to 80,000 facilities across the country that are in
the business of repairing and refinishing vehicles, particularly cars. Auto body
shops present a wide array of environmental concerns, from use and emissions of
hazardous materials such as methylene chloride, to discharges of polluted water
into water systems, and worker exposure to toxic solvents and particulate matter.
• The National Emission Standards for Hazardous Air Pollutants: Paint Stripping and
Miscellaneous Surface Coating Operations at Area Sources, is a regulation that
existing shops must comply with by January, 2011 and that new shops must
comply with by January, 2008 or upon startup of operations. The rule requires
that, for example: all spray painting must be done in a spray booth, painters
must use spray guns and techniques that reduce overspray, all painters must
receive training, and paint spray gun cleaning cannot release any mist of cleaning
solvent to the air.
• States are currently considering how to implement the new regulation; ERP is
one potential policy tool for implementation. ERP is an innovative approach to
improving facilities' management practices within small business sectors. ERP is
an integrated system that incorporates plain language compliance assistance that
promotes pollution prevention, facility self-assessment and self-certification,
agency inspections, statistically-based performance measurement, and where
necessary, comprehensive facility investigations and targeted enforcement
actions.
• The purpose of this evaluation is to inform states and EPA regions who are
currently considering developing programs to encourage auto body shops to
adopt best practices and improve compliance with environmental regulations.
Evaluation Questions
• To what extent have the ERPs in DE, RI, and ME led to actual and/or expected
adoption of selected best practices that reduce the environmental footprint of
auto body shops?
• What environmental and health outcomes are estimated to result from auto body
shops implementing these best practices?
• What are the cost implications of each program for regulators and auto body
shops initially and over time?
• What is the cost-effectiveness of each program?
• Overall, what are the advantages and disadvantages of each of the three ERPs in
terms of reaching auto body shops, generating environmental and worker health
outcomes, and achieving cost-effective results?
• What factors influenced the outcomes of each program (e.g., existing or
impending regulations, regulatory/assistance offices involved in conducting the
program, and extent of coordination with industry representatives)?
-------
Evaluation Questions Con't.
• What do our findings suggest regarding the circumstances under which ERPs are
likely to produce cost-effective results?
• What is the current status of each program?
• What are the primary implementation challenges states faced in developing and
implementing their programs?
• What factors influenced the states' decisions to continue, not continue, or modify
their programs after initial pilots?
• How does the state and/or EPA Region involved in implementing each program
view the program's results, and why?
Evaluation Methods
• The evaluation used a non-experimental design known as "One Group
Pretest/Posttest," which involves measurement or observation of a group of
subjects (auto body shops) prior to and after the application of an intervention
(the ERP). There were no control groups used, i.e., the states did not measure
performance in a control group of auto body shops not subject to the ERP.
• To assess the behavioral and environmental outcomes of the ERPs, the
evaluation primarily relied on existing data reported by each program, which
captured what percentage of facilities use certain best practices during
independent, random inspections before and after ERP implementation. The
analysis compares the percentage of shops using certain best practices before
and after the ERP program and translates those changes in behaviors to
environmental outcomes, where possible. Where this translation is not possible,
the evaluation qualitatively discuss the benefits one would expect facilities would
observe as a result of adopting these behaviors.
• ERP states did not collect quantitative data on long-term outcomes (e.g.,
emissions reductions), and therefore the evaluation was limited to estimating this
information where possible.
• To assess the remaining evaluation questions, the evaluation relies on interviews
with state program staff, EPA staff involved in supporting ERP, representatives of
the States' ERP Consortium, and selected auto body shops. For example, these
interviews provided insights on program status, costs, factors that influenced
outcomes, and implementation experiences.
Key Findings
• The evaluation findings suggest that ERP is associated with improved business
practices in the auto body sector and is regarded as successful by both state and
industry representatives. Quantifying environmental outcomes associated with
ERP is difficult, and those outcomes that were quantified were relatively small. In
addition, sustaining the program has proven to be difficult given resource
constraints and overall regulatory priorities.
• Each state selected between 19 and 24 indicators of environmental performance.
States observed improved performance between the samples of facilities
measured for the vast majority of indicators (observed performance
-------
Key Findings Con't.
improved for 54 out of 65 indicators (83 percent) between baseline and post-
certification). Of these 54 indicators, 29 (54 percent of the indicators where
observed performance improved, and 45 percent of all indicators) were found
to have statistically significant changes in performance. In addition, no
statistically significant declines in performance were found.
• For the measures where there were statistically significant improvements, one
can infer that there was an increase in the proportion of the entire population
of auto body shops in the state following best practices. In other words, for
nearly half of the measured indicators, the evaluation findings suggest that the
auto body sector as a whole is shifting to increased use of best practices, not
just those shops that were included in the samples measured.
• The greatest percentage of the total number of indicators with statistically
significant improvements was observed in the air emissions and worker health
and safety categories. In both of these categories, half of the total set of
indicators measured (not solely those improving, but the total number of
indicators) showed statistically significant improvements.
• With regard to air emissions, the evaluation estimates that improvements in
usage of low-VOC/waterbased solvents may have reduced VOC emissions by as
much as 1.7 tons per year for all auto body shops in each of two states:
Delaware and Maine (actual amounts could be less). In addition, estimated
material usage associated with auto body shop spray gun cleaning operations
declined by between 0.6 and 2.0 tons per year for all shops in Delaware.
Finally, estimates are that improvements in hazardous waste management in
Delaware may have increased the amount of hazardous waste being properly
identified by as much as 22,440 pounds per month (actual amounts are likely
to be less). All three of these estimates rely on numerous assumptions, and
have considerable uncertainties, which are described in the full evaluation
report. The evaluation was unable to quantify reductions in water discharges or
improvements to worker health and safety.
• Note that while the overall trend in improving performance measured by states
is consistent with the hypothesis that ERP leads to adoption of selected best
practices, one cannot be certain of the extent to which the states' ERP alone
caused or contributed to the observed changes in performance, in comparison
to other factors. Other factors happening concurrently with ERP may have also
contributed to the observed changes in performance. For example, in Rhode
Island ERP was implemented in tandem with the auto body shop license
renewal process. In Delaware ERP was developed in tandem with a source
category permit for this sector.
• The three states included in the evaluation spent a range of resources
developing and implementing their ERPs. Estimated government costs
(including state resources and an EPA State Innovation Grant) are $800 to
$2,000 per auto body shop in the population for states conducting a single
cycle of ERP. These costs are not inclusive of all of the state resources spent on
staff throughout the program. States also dedicated staff time that is not fully
captured in these cost estimates. For Rhode Island, total costs range from $400
to $700 per shop for two cycles of ERP (in other words, the costs per shop per
cycle would be half as much).
-------
Key Findings Con't.
• Compared to the likely outcomes of alternative approaches (e.g., following up
on complaints or conducting infrequent inspections) that these three states
considered, states perceive that ERP resulted in greater improvements in
performance. These benefits of ERP typically come at a higher initial cost
compared to these less resource-intensive alternatives. However, if states
conducted successive rounds of ERP, it's likely that per facility costs would be
substantially reduced, as Rhode Island has found.
• A key factor related to cost effectiveness is the number of facilities targeted by
ERP. Specifically, ERP can be more cost effective for larger populations of
facilities, so long as the population is relatively homogenous, with a common
set of regulatory requirements and best practices.
• Another factor related to the likelihood of a state achieving cost-effective
results through ERP is the degree to which state staff can build partnerships
within (e.g., between different environmental media offices) and outside
agency walls (e.g. Rhode Island's partnership with the University of Rhode
Island). State staff within all of the ERPs described in this evaluation forged
partnerships within and outside their agencies to implement the ERP.
Successful design and implementation of ERPs that deal with multiple
environmental media require the cooperation among and buy-in from the
various offices that are affected. Outside partners may provide support in the
form of technical expertise and/or funding.
• Baseline performance and the likely extent of improvement (i.e., to what extent
are facilities likely to be able or willing to change their behavior) both have a
direct effect on the extent to which statistically significant changes may be
observed.
• Key factors leading to success of an ERP include sufficient funding, the
regulatory context in which the program was implemented, effective
coordination and communication among involved offices, upper management
buy-in, and continuing program support.
Recommendations
• Combine forces. States could work together to realize the economies of scale
possible with ERP.
• Decide on a set of common indicators. It would be helpful to be able to
compare and aggregate ERP data for the same sector across programs.
• Collect quantitative data on facility practices, not just information on
the proportions of shops utilizing specific practices, for a small
number of indicators. This could help quantify environmental outcomes.
• Develop a tool to help states estimate environmental outcomes.
• Un-package ERP. States could consider different ways to implement different
components of ERP.
• Consider implementing ERP primarily where larger populations of
facilities are present. This approach has the potential to reduce per facility
expenditures and increase the cost-effectiveness of the program.
• Develop a clearer agreement between EPA and states as to whether
or not ERP can be used to address traditional regulatory programs.
------- |