United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
September 2009
EPA-100-F-09-055
Evaluation of Three Environmental
Results Programs (ERPs)
September 2009
Fact Sheet
Introduction
http://www.epa.gov/evaluate
For more information on
this and other completed
evaluations at EPA or the
Evaluation Support Division,
visit the above link.
• There are an estimated 35,000 to 80,000 facilities across the country that are in
the business of repairing and refinishing vehicles, particularly cars. Auto body
shops present a wide array of environmental concerns, from use and emissions of
hazardous materials such as methylene chloride, to discharges of polluted water
into water systems, and worker exposure to toxic solvents and particulate matter.
• The National Emission Standards for Hazardous Air Pollutants: Paint Stripping and
Miscellaneous Surface Coating Operations at Area Sources, is a regulation that
existing shops must comply with by January, 2011 and that new shops must
comply with by January, 2008 or upon startup of operations. The rule requires
that, for example: all spray painting must be done in a spray booth, painters
must use spray guns and techniques that reduce overspray, all painters must
receive training, and paint spray gun cleaning cannot release any mist of cleaning
solvent to the air.
• States are currently considering how to implement the new regulation; ERP is
one potential policy tool for implementation. ERP is an innovative approach to
improving facilities' management practices within small business sectors. ERP is
an integrated system that incorporates plain language compliance assistance that
promotes pollution prevention, facility self-assessment and self-certification,
agency inspections, statistically-based performance measurement, and where
necessary, comprehensive facility investigations and targeted enforcement
actions.
• The purpose of this evaluation is to inform states and EPA regions who are
currently considering developing programs to encourage auto body shops to
adopt best practices and improve compliance with environmental regulations.
Evaluation Questions
• To what extent have the ERPs in DE, RI, and ME led to actual and/or expected
adoption of selected best practices that reduce the environmental footprint of
auto body shops?
• What environmental and health outcomes are estimated to result from auto body
shops implementing these best practices?
• What are the cost implications of each program for regulators and auto body
shops initially and over time?
• What is the cost-effectiveness of each program?
• Overall, what are the advantages and disadvantages of each of the three ERPs in
terms of reaching auto body shops, generating environmental and worker health
outcomes, and achieving cost-effective results?
• What factors influenced the outcomes of each program (e.g., existing or
impending regulations, regulatory/assistance offices involved in conducting the
program, and extent of coordination with industry representatives)?
-------
Evaluation Questions Con't.
• What do our findings suggest regarding the circumstances under which ERPs are likely to produce
cost-effective results?
• What is the current status of each program?
• What are the primary implementation challenges states faced in developing and implementing their
programs?
• What factors influenced the states' decisions to continue, not continue, or modify their programs after
initial pilots?
• How does the state and/or EPA Region involved in implementing each program view the program's
results, and why?
Evaluation Methods
• The evaluation used a non-experimental design known as "One Group Pretest/Posttest," which
involves measurement or observation of a group of subjects (auto body shops) prior to and after the
application of an intervention (the ERP). There were no control groups used, i.e., the states did not
measure performance in a control group of auto body shops not subject to the ERP.
• To assess the behavioral and environmental outcomes of the ERPs, the evaluation primarily relied on
existing data reported by each program, which captured what percentage of facilities use certain best
practices during independent, random inspections before and after ERP implementation. The analysis
compares the percentage of shops using certain best practices before and after the ERP program and
translates those changes in behaviors to environmental outcomes, where possible. Where this
translation is not possible, the evaluation qualitatively discuss the benefits one would expect facilities
would observe as a result of adopting these behaviors.
• ERP states did not collect quantitative data on long-term outcomes (e.g., emissions reductions), and
therefore the evaluation was limited to estimating this information where possible.
• To assess the remaining evaluation questions, the evaluation relies on interviews with state program
staff, EPA staff involved in supporting ERP, representatives of the States' ERP Consortium, and
selected auto body shops. For example, these interviews provided insights on program status, costs,
factors that influenced outcomes, and implementation experiences.
Key Findings
• The evaluation findings suggest that ERP is associated with improved business practices in the auto
body sector and is regarded as successful by both state and industry representatives. Quantifying
environmental outcomes associated with ERP is difficult, and those outcomes that were quantified
were relatively small. In addition, sustaining the program has proven to be difficult given resource
constraints and overall regulatory priorities.
• Each state selected between 19 and 24 indicators of environmental performance. States observed
improved performance between the samples of facilities measured for the vast majority of indicators
(observed performance improved for 54 out of 65 indicators (83 percent) between baseline and post-
certification). Of these 54 indicators, 29 (54 percent of the indicators where observed performance
improved, and 45 percent of all indicators) were found to have statistically significant changes in
performance. In addition, no statistically significant declines in performance were found.
• For the measures where there were statistically significant improvements, one can infer that there
was an increase in the proportion of the entire population of auto body shops in the state following
best practices. In other words, for nearly half of the measured indicators, the evaluation findings
suggest that the auto body sector as a whole is shifting to increased use of best practices, not just
those shops that were included in the samples measured.
• The greatest percentage of the total number of indicators with statistically significant improvements
was observed in the air emissions and worker health and safety categories. In both of these
Page 2
-------
categories, half of the total set of indicators measured (not solely those improving, but the total
number of indicators) showed statistically significant improvements.
• With regard to air emissions, the evaluation estimates that improvements in usage of low-
VOC/waterbased solvents may have reduced VOC emissions by as much as 1.7 tons per year for all
auto body shops in each of two states: Delaware and Maine (actual amounts could be less). In
addition, estimated material usage associated with auto body shop spray gun cleaning operations
declined by between 0.6 and 2.0 tons per year for all shops in Delaware. Finally, estimates are that
improvements in hazardous waste management in Delaware may have increased the amount of
hazardous waste being properly identified by as much as 22,440 pounds per month (actual amounts
are likely to be less). All three of these estimates rely on numerous assumptions, and have
considerable uncertainties, which are described in the full evaluation report. The evaluation was
unable to quantify reductions in water discharges or improvements to worker health and safety.
• Note that while the overall trend in improving performance measured by states is consistent with the
hypothesis that ERP leads to adoption of selected best practices, one cannot be certain of the extent
to which the states' ERP alone caused or contributed to the observed changes in performance, in
comparison to other factors. Other factors happening concurrently with ERP may have also
contributed to the observed changes in performance. For example, in Rhode Island ERP was
implemented in tandem with the auto body shop license renewal process. In Delaware ERP was
developed in tandem with a source category permit for this sector.
• The three states included in the evaluation spent a range of resources developing and implementing
their ERPs. Estimated government costs (including state resources and an EPA State Innovation
Grant) are $800 to $2,000 per auto body shop in the population for states conducting a single cycle of
ERP. These costs are not inclusive of all of the state resources spent on staff throughout the program.
States also dedicated staff time that is not fully captured in these cost estimates. For Rhode Island,
total costs range from $400 to $700 per shop for two cycles of ERP (in other words, the costs per
shop per cycle would be half as much).
• Compared to the likely outcomes of alternative approaches (e.g., following up on complaints or
conducting infrequent inspections) that these three states considered, states perceive that ERP
resulted in greater improvements in performance. These benefits of ERP typically come at a higher
initial cost compared to these less resource-intensive alternatives. However, if states conducted
successive rounds of ERP, it's likely that per facility costs would be substantially reduced, as Rhode
Island has found.
• A key factor related to cost effectiveness is the number of facilities targeted by ERP. Specifically, ERP
can be more cost effective for larger populations of facilities, so long as the population is relatively
homogenous, with a common set of regulatory requirements and best practices.
• Another factor related to the likelihood of a state achieving cost-effective results through ERP is the
degree to which state staff can build partnerships within (e.g., between different environmental media
offices) and outside agency walls (e.g. Rhode Island's partnership with the University of Rhode
Island). State staff within all of the ERPs described in this evaluation forged partnerships within and
outside their agencies to implement the ERP. Successful design and implementation of ERPs that deal
with multiple environmental media require the cooperation among and buy-in from the various offices
that are affected. Outside partners may provide support in the form of technical expertise and/or
funding.
• Baseline performance and the likely extent of improvement (i.e., to what extent are facilities likely to
be able or willing to change their behavior) both have a direct effect on the extent to which
statistically significant changes may be observed.
• Key factors leading to success of an ERP include sufficient funding, the regulatory context in which the
program was implemented, effective coordination and communication among involved offices, upper
management buy-in, and continuing program support.
Recommendations
• Combine forces. States could work together to realize the economies of scale possible with ERP.
• Decide on a set of common indicators. It would be helpful to be able to compare and aggregate
Page 3
-------
ERP data for the same sector across programs.
• Collect quantitative data on facility practices, not just information on the proportions of
shops utilizing specific practices, for a small number of indicators. This could help quantify
environmental outcomes.
• Develop a tool to help states estimate environmental outcomes.
• Un-package ERP. States could consider different ways to implement different components of ERP.
• Consider implementing ERP primarily where larger populations of facilities are present.
This approach has the potential to reduce per facility expenditures and increase the cost-effectiveness
of the program.
• Develop a clearer agreement between EPA and states as to whether or not ERP can be
used to address traditional regulatory programs.
Contact(s)
• John Heffelfinger, Office of Policy, Economics and Innovation, Evaluation Support Division,
heffelfinger.john@epa.gov
Report Link: http://www.epa.gov/evaluate/aboutjnnovations4.htm
Date Completed: September 2009
Page 4
------- |