?/ER/\
United States
Environmental Protection
Agency
Office of Policy,
Economics and Innovation
(1807T)
August 2009
EPA-100-K-09-021
Evaluation of Three
Environmental
Results Programs
(ERPs)
Final Report
Promoting Environmental Results
^ 1
Through Evaluation
-------
ERP Evaluation: Final Report August 2009
TABLE OF CONTENTS
EXEUCTIVE SUMMARY ES-i
CHAPTER 1 | INTRODUCTION
I'I III I
II t ' , ii ' ;) i i ' I J .< n> 3
III : - i '' In l'' 111 ill- i'h I i 1111 3
I 'i v 10
e i h I.' "ii 12
CHAPTER 2 | EVALUATION METHODOLOGY
13
13
l , iors. Outcomes. Costs, and Program Hxpenenees 14
. , > 19
CHAPTER 3 | FINDINGS
hi .-I. 'i» ' 20
f ' .' I. .I li ' ||' I ' I 'r ]i l| I i Ii I I' I 21
r..' '. .. 22
I , '23
h" I Ln 24
II > , , ' ' I I I '26
p)l Ill I .i'i i 26
'-i H l . I1 in 29
,!" 31
II. i'li ,, -te 42
i . ii ' 51
I I .. ii ' ' 56
I 11 11 I ii 64
iM . >I I I'i l 'I l I I' I 70
' ,1 ' ' 79
I i'79
k, 8?
I i, I ': S3
-------
ERP Evaluation: Final Report August 2009
! !--1 I ; n Kxpenence 85
' ' 85
I 86
I 87
I I ! ,89
I . . -II 89
- : 90
I I I 92
CHAPTER 4 | CONCLUSIONS AND RECOMMENDATIONS
I. Conclusions 93
Question 1: To \\ hat extent ,. -'. I !,-r -I ,, .. .. ;, r!t <,
Island led to adoption ofsel - . >~; ' i: :,
footprint of auto body shops'.' 93
Question 2: \\l\i\i cm ironmcntal and health outcomes are estimated to result
from the implementation of these best practices'.' 95
99
'. , 101
Questions 1 and X: What is the current status of each program'* and What factors
influenced the states decisions to continue, not continue, or modify their
programs after initial pilots'* 102
i ' * t __ i
: . . ' 103
. i " ' i . ' 104
11 \ I ' ' ' * '
" : ' ^ , '104
II. Recommendations 104
-------
ERP Evaluation: Final Report
August 2009
APPENDIX A:
APPENDIX B:
APPENDIX C
APPENDIX D:
APPENDIX E:
APPENDIX F:
APPENDIX G:
DATA AVAILABLE FOR DFE CALCULATOR ANALYSIS A-1
DFE CALCULATOR INPUTS B-1
EMISSIONS REDUCTIONS ESTIMATE c-1
DELAWARE COST SUMMARY D-1
STATE PROGRAM STAFF INTERVIEW GUIDE E-1
AUTO BODY REPRESENTATIVES INTERVIEW GUIDE F-1
EPA STAFF INTERVIEW GUIDE 0-1
in
-------
ERP Evaluation: Final Report August 2009
ACKNOWLEDGEMENTS
The Evaluation of Three Environmental Results Programs: Draft Report was developed
for the U.S. Environmental Protection Agency's Office of Policy, Economics and
Innovation under Contract EP-W-04-023 between EPA and Industrial Economics, Inc.
(lEc) of Cambridge, MA. The evaluation team consisted of lEc and consultant Michael
Crow. Tracy Dyke Redmond, Nora Scherer, Colin Mahoney, and Eric Ruder represented
lEc.
Beth Termini and Scott Bowles of EPA's Office of Policy, Economics, and Innovation
(OPEI), provided important information and insights on ERP, as well as input and
feedback throughout the evaluation process. John Heffelfinger of OPEI's Evaluation
Support Division served as the technical advisor and reviewer for this evaluation. Special
thanks go to several representatives from ERP states, including Kim Chesser, Julie
Churchill, Rich Enander, Ron Gagnon, Renee Bashell, and Alister Innes, who shared
their reflections, insights, and recommendations about ERP, and in some cases data they
collected during ERP. Their input was invaluable in constructing our findings, and we
are grateful for their time and willingness to share information. We also thank the
owners/operators of several auto body shops who participated in interviews designed to
help us understand business perspectives on participating in ERP.
IV
-------
ERP Evaluation: Final Report August 2009
EXECUTIVE SUMMARY
BACKGROUND
The auto body sector includes between 35,000 and 80,000 facilities across the country
that are in the business of repairing and refinishing vehicles, primarily cars. Auto body
shops present a wide array of environmental concerns, from use and emissions of
hazardous materials such as methylene chloride, to discharges of polluted water into
water systems, and worker exposure to toxic solvents and particulate matter. The auto
body sector has recently received regulatory attention under the National Emission
Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface
Coating Operations at Area Sources. The rule requires, among other things, that all spray
painting must be done in a spray booth, painters must use spray guns and techniques that
reduce overspray (such as high volume, low pressure, or HVLP, spray guns), all painters
must receive training, and paint spray gun cleaning cannot release any mist of cleaning
solvent to the air. Although new auto body shops (those that started operations after
September, 2007) are required to comply with these requirements by January, 2008 or
upon startup of operations, existing auto body shops must comply with these
requirements by January, 2011 (with the exception of an initial notification date in 2010).
States are considering how they may implement these regulations, and some states are
considering adopting Environmental Results Programs (ERP) as a means to implement
them. As the compliance date approaches, this evaluation of ERPs in the auto body sector
is intended to inform states and EPA regions considering approaches to improve the
environmental performance of the auto body sector.
ENVIRONMENTAL RESULTS PROGRAMS (ERP)
ERP is an innovative approach to improving facilities' management practices within
small business sectors. ERP is an integrated system of:
Plain language compliance assistance that promotes pollution prevention;
Facility self-assessment and self-certification;
Agency inspections;
Statistically-based performance measurement; and
Where necessary, comprehensive facility investigations and targeted enforcement
actions.
These elements are intended to work together to achieve the goals of ERP, which are to
improve compliance and reduce environmental impacts of the target sector, while
deploying government resources strategically and efficiently.
ES-i
-------
ERP Evaluation: Final Report August 2009
ERP components are combined into a cyclical process. Generally speaking, after
identifying facilities in the target sector, states conduct inspections at a random sample of
facilities at the outset of the program (i.e., at baseline). States then offer compliance
assistance to all facilities, and then encourage (or in some cases, require) them to conduct
a self-assessment and submit self-certification forms. This self-certification period is
followed in turn by another set of inspections at a random sample of facilities to measure
group performance compared to the baseline. Compliance assistance specifies how
facilities should assess their operations and certify compliance, while agency inspectors
document progress against specific performance indicators, which are also tracked in self-
certification checklists. Performance data, in turn, can inform and improve the next round
of compliance assistance.
EVALUATION DESIGN
This evaluation reviews the experience of three states, Delaware, Maine, and Rhode
Island, which established ERPs for the auto body sector. Each of these ERPs incorporates
voluntary self-certification and encompasses cross-media environmental and/or health
concerns. While these states are similar in the types of programs they have implemented,
they differ in terms of several circumstances in the state that could affect ERP
implementation, participation of auto body shops, and outcomes. Therefore, our primary
intent is to describe each state's experience, rather than to directly compare their
outcomes, costs, or cost-effectiveness. The primary purpose of this evaluation is to
inform states and EPA regions who are currently considering developing programs to
encourage auto body shops to adopt best practices and improve compliance with
environmental regulations.
The evaluation is designed to answer the following questions related to three categories:
environmental outcomes, program costs/cost-effectiveness, and implementation
experiences:
To what extent have the ERPs in Delaware, Rhode Island, and Maine, led to
actual and/or expected adoption of selected best practices that reduce the
environmental footprint of auto body shops?
What environmental and health outcomes are estimated to result from auto body
shops implementing these best practices?
What are the cost implications of each program for regulators and auto body
shops initially and over time?
What is the cost-effectiveness of each program?
Overall, what are the advantages and disadvantages of each of the three ERPs in
terms of reaching auto body shops, generating environmental and worker health
outcomes, and achieving cost-effective results?
What factors influenced the outcomes of each program (e.g., existing or
impending regulations, regulatory/assistance offices involved in conducting the
program, and extent of coordination with industry representatives)?
ES-ii
-------
ERP Evaluation: Final Report August 2009
What do our findings suggest regarding the circumstances under which ERPs are
likely to produce cost-effective results?
What is the current status of each program?
What are the primary implementation challenges states faced in developing and
implementing their programs?
What factors influenced the states' decisions to continue, not continue, or modify
their program after initial pilots?
How does the state and/or EPA Region involved in implementing each program
view the program's results, and why?
EVALUATION METHODOLOGY
We use existing data reported by each program included in this study to summarize the
changes in auto body shop behaviors that affect environmental and health concerns. As
noted earlier, ERP involves selecting a random sample of shops prior to ERP to assess
baseline performance, and then selecting a second, independent random sample of shops
to assess performance after ERP implementation. We compare the percentage of shops
using certain best practices during baseline inspections to the percentage of shops using
these same practices after ERP implementation (i.e., during post-certification
inspections). The indicators are organized in five categories of environmental concern: air
emissions, water discharges, waste management, pollution prevented, and worker health
& safety. We calculate the observed percentage point change over the ERP cycle (i.e., the
difference between the baseline percentage and the post-certification inspection
percentage). We then use the data from these independent random samples to make
inferences about changes in the proportion of shops in the entire population of facilities
subject to ERP using specific practices. Since the samples are drawn from the entire
population of eligible facilities (not just those that self-certify), changes in
implementation of best practices reflect the sector as a whole, not just those shops that
elected to participate in the voluntary ERP self-certification.
Once we analyze changes in practices following each program intervention, we translate
these behaviors into environmental outcomes, where possible. For other categories of
environmental concern, we qualitatively discuss the benefits we would expect facilities
would observe as a result of adopting these behaviors.
Overall, we cannot be certain of the extent to which the states' ERP caused or contributed
to observed changes in performance, in comparison to other factors (e.g., regulations or
permit requirements) happening concurrently with ERP. In addition, while we have
extensive, statistically-based data on behavior changes associated with ERP, states did
not collect quantitative data on long-term outcomes (e.g., emissions reductions), and
therefore, we were limited to estimating this information where we could.
To assess the remaining evaluation questions, we rely on interviews with state program
staff, EPA staff involved in supporting ERP, representatives of the States' ERP
Consortium, and operators of selected auto body shops. For example, these interviews
ES-iii
-------
ERP Evaluation: Final Report August 2009
provided insights on program status, costs, factors that influenced outcomes, as well as
implementation experiences.
KEY FINDINGS
The experiences of the three ERPs reviewed in the course of this evaluation suggest that
ERP is associated with improved business practices in the auto body sector and is
regarded as successful by both state and industry representatives. Quantifying
environmental outcomes associated with ERP is difficult, and those outcomes that we
were able to quantify were relatively small. In addition, sustaining the program has
proven to be difficult given resource constraints and overall regulatory priorities.
Each state included in this analysis selected between 19 and 24 indicators of
environmental performance, and measured the percentage of shops meeting these criteria
at baseline and post-certification. States observed improved performance between the
samples of facilities measured for the vast majority of indicators (observed performance
improved for 54 out of 65 indicators (83 percent) between baseline and post-
certification). Of these 54 indicators, 29 (54 percent of the indicators where observed
performance improved, and 45 percent of all indicators) were found to have statistically
significant changes in performance. For the measures where there were statistically
significant improvements, we can infer1 that there was an increase in the proportion of
the entire population of auto body shops eligible to participate in ERP following best
practices. In other words, for nearly half of the measured indicators, we believe that the
auto body sector as a whole is shifting to increased use of best practices, not just those
shops that were included in the samples measured. Note that there were no statistically
significant declines in performance.
For the 55 percent of indicators (36 of 65 indicators) where we did not detect a
statistically significant change in performance, we cannot be certain at the specified
confidence level (90 percent in Delaware and Maine, 95 percent in Rhode Island) that
there was a change in the proportion of the entire population of auto body shops
following the best practices.
The greatest percentage of the total number of indicators with statistically significant
improvements were observed in the air emissions and worker health and safety
categories. In both of these categories, half of the total set of indicators measured (not
solely those improving, but of the total number of indicators) showed statistically
significant improvements. A small number of indicators showed worsening performance,
but none of these observations were statistically significant.
With regard to air emissions, we were able to estimate reductions in emissions of VOCs
from auto body shops in Maine and Delaware associated with improving spray gun
cleaning methods. In both states, we estimate that VOC emissions reductions associated
with the behavior changes related to low-VOC/waterbased solvent usage measured by
1 We make this inference with 90 percent confidence for indicators measured by Maine and Delaware, and 95
percent for indicators measured by Rhode Island.
ES-iv
-------
ERP Evaluation: Final Report August 2009
ERP could be as much as 1.7 tons per year for all auto body shops in each state (although
the actual amount may be less). Based on calculations presented in the final Surface
Coating Rule, we estimate that this is equivalent to approximately a 1 percent reduction
in reduction in total VOC emissions per shop associated with the use of low -
VOC/waterbased solvent.
Regarding hazardous waste management, in one state we were able to develop a first
order approximation of the change in the amount of hazardous wastes subject to proper
hazardous waste determination, which can serve as proxy for proper waste management.
We estimate that there was a potential increase of as much as 22,440 pounds per month of
hazardous waste being properly identified at all auto body shops in Delaware (although
the actual amount is likely to be less). This amount is equivalent to the minimum waste
generated by 10 large quantity generators per month.
We were not able to quantify reductions in water discharges associated with ERP, but two
of the three states did find statistically significant improvements in behaviors that are
expected to reduce water discharges. For example, both Delaware and Rhode Island
observed significant improvement in the percentage of shops that post signs prohibiting
the discharge of industrial chemicals to non-industrial drainage outlets.
We were also not able to quantify the improvements associated with worker health and
safety, but in Rhode Island, three of the six indicators measured showed statistically
significant improvement, specifically, the indicators related to whether the shop has
established a Personal Protective Equipment Program, whether the shop has a
Lockout/Tagout Program, and whether the employer has posted the Job Safety & Health
Protection poster. All of these improvements are expected to reduce worker risks.
We were able to estimate materials reductions associated with increased usage of
automatic spray gun cleaning methods in Delaware. We estimate that the material usage
associated with auto body shop spray gun cleaning operations declined by between 0.6
and 2.0 tons per year for all shops in Delaware.
Note that while the overall trend in improving performance measured by states is
consistent with the hypothesis that ERP leads to adoption of selected best practices, we
cannot be sure that the observed changes can be fully attributed to the presence of ERP.
For example, in Delaware, the fact that ERP was developed in tandem with a source
category permit may have made the program more successful. In Rhode Island, the fact
that the state had passed regulations specifically addressing the auto body sector, and had
conducted outreach related to this rule, meant that for some indicators, baseline
performance was relatively high. Also in Rhode Island, ERP was implemented in tandem
with the auto body license renewal process, which may have given the program more
visibility and clout with auto body shops. Note that the presence of the federal Surface
Coating Rule may now be affecting shops' behavior regarding air emissions, but the vast
majority of data analyzed in this report (with the exception of the Rhode Island's second
ERP cycle) was gathered prior to the promulgation of the federal rule, and therefore we
do not expect that the rule had a substantial impact on shop behavior reported in these
ERPs.
ES-v
-------
ERP Evaluation: Final Report August 2009
The three states included in the evaluation spent a range of resources developing and
implementing their ERPs. We estimate government costs (including state resources and
an EPA State Innovation Grant) of $800 to $2,000 per auto body shop in the population
for states conducting a single cycle of ERP (Delaware and Maine). We note that these
costs are not inclusive of all of the resources spent by the state for staff time throughout
the program. Therefore, our per shop estimates are underestimating the total costs of the
program to the states. For Rhode Island, costs range from $400 to $700 per shop for two
cycles of ERP (in other words, cost per shop for each cycle of ERP would be half as
much). States also spent a range of staff resources, which are partially, but not completely
captured in these cost estimates. Delaware dedicated an average of roughly half of a Full
Time Equivalent (FTE) for 2.5 years (330 days of staff time over this period). A total of
23 staff members were involved to various extents over this period. For Maine, the design
and development phase required two FTEs for the period of six months, the initial
implementation phase also required two FTEs for six months, and program follow-
up/ongoing support required one and a half FTEs for two years. In Rhode Island's first
round, the design and development phase required one to two FTEs, the initial
implementation phase also required one to two FTEs, and program follow-up/ongoing
support required less than one FTE.
The best assessment of the cost effectiveness of ERP would take into account alternative
approaches to regulating (or not-regulating) the target sector. For the states included in
this evaluation, the alternatives to ERP considered by the states were to: 1) follow-up on
complaints or other information about violations, but otherwise not focus on the auto
body sector, 2) continue inspecting less than five percent of auto body shops for air, water
and hazardous waste compliance each year, or 3) implement a source category permit, but
without the focused compliance assistance efforts associated with ERP.
Compared to the likely outcomes of these less re source-intensive alternatives, states
perceive that ERP resulted in greater improvements in performance. For the first two
alternatives (following up on complaints or conducting infrequent inspections) states
predicted that low levels of compliance would persist, and that ERP provided an avenue
for the states to improve performance. It is more difficult to tease out the relative benefits
of ERP and a source category permit, but the state's feedback suggests that ERP led to
more shops getting a source category permit than would have done so otherwise, which
presumably led to improved performance/compliance. Moreover, in addition to resulting
in better performance than the alternatives considered by these three states, ERP provides
far more data about the results achieved than any of the alternatives.
These benefits of ERP typically come at a higher initial cost, however, compared to these
three less resource-intensive alternatives. Two of the three states in this evaluation have
not found sustainable funding models, and have had to discontinue ERP for the auto body
sector. If states conducted successive rounds of ERP, it's likely that per facility costs
would be reduced, since initial program design and development costs would be reduced
or eliminated in successive rounds. Rhode Island has found that subsequent rounds of its
ERP have required substantially less resources than the initial phases.
ES-vi
-------
ERP Evaluation: Final Report August 2009
A key factor related to cost effectiveness is the number of facilities targeted by ERP.
Specifically, ERP can be more cost effective for larger populations of facilities, so long as
the population is relatively homogenous, with a common set of regulatory requirements
and best practices.
Another factor related to the likelihood of a state achieving cost-effective results through
ERP is the degree to which state staff can build partnerships within and outside agency
walls. ERP staff within all of the ERPs described in this evaluation forged partnerships
within and outside of their agencies to implement the ERP. Outside partners may provide
support in the form of technical expertise and/or funding. For example, Rhode Island
received both technical and financial support for its ERP from the University of Rhode
Island.
Successful design and implementation of ERPs that deal with multiple environmental
media require the cooperation among and buy-in from the various offices that are
affected. For example, in both Delaware and Rhode Island, regulators found that the high
level of input and collaboration with the various media offices made the process much
more successful, primarily because the workbook they developed accurately covered all
of the various regulations involved. While this cooperation is integral to the success of an
ERP that deals with multiple environmental media, it can also be a challenge to
coordinate and facilitate the involvement of so many offices.
Baseline performance and the likely extent of improvement (i.e., to what extent are
facilities likely to be able or willing to change their behavior) both have a direct effect on
the extent to which statistically significant changes may be observed. It may be advisable
for states to undertake a limited assessment before committing resources to conducting a
full ERP. For example, states may find it useful to conduct a relatively small, targeted
survey to assess baseline performance on key indicators and to ask facilities about what
would enable or motivate them to change their behavior.
As noted above, of the three ERP programs, only one of them is currently ongoing: the
ERP in Rhode Island. Both ERPs in Delaware and Maine were operational for a specific
period of time, with a specific amount of funding with which to implement the program,
and have not continued. Delaware and Maine each received EPA State Innovation Grants
(SIG) to design and conduct their ERPs. These grants were intended to support the
development and implementation of an ERP in each state, and were not intended to
support an ongoing ERP effort. Rhode Island is the only state to continue its ERP. Its
ERP was developed without the direct assistance of EPA SIG funding. Instead, DEM
conducted its own studies prior to the development of its ERP that revealed significant
pollution and health concerns arising from auto body operations. In addition, its
traditional inspections reached less than five percent of shops in the state. Therefore the
department saw a benefit of an ERP as a way to address the significant occupational
health, safety, and environmental issues in the auto body sector, because of the program's
potential to reach more shops. Rhode Island recently completed its second cycle of ERP,
and intends to continue implementing the program. Moving forward, however, the state
plans on spending minimal additional resources in outreach efforts.
ES-vii
-------
ERP Evaluation: Final Report August 2009
If states see ERP as a one-time trial program, and not as an alternative or complement to
their traditional permitting and inspection process, they may not see the benefit of
continuing to support the program once grant funding has expired. This was the case in
both Delaware and Maine: program managers used SIG funding to conduct an ERP as a
pilot, with the possibility of continuing the program. However, once the funding ended,
the states were unable to commit the necessary resources to continue their programs. If,
on the other hand, regulators see ERP as a new way to approach to permitting and
compliance, the state may choose to commit the resources necessary to continue the
program. This was the case in Rhode Island: the state saw the long-term benefit of ERP
as an effective tool to reach more shops and achieve a higher rate of compliance than with
its traditional permitting process, so the state provided the support necessary to maintain
the program. This "buy-in" and support, either from EPA or state agencies, is integral to
the program moving beyond a one-time effort.
In addition to sustainable funding, we found that key factors leading to success of an ERP
include the regulatory context in which the program was implemented, effective
coordination and communication among involved offices, upper management buy-in, and
continuing program support.
RECOMMENDATIONS
Based on our findings from this evaluation, we offer several recommendations for
consideration:
1. Combine forces. ERP offers economies of scale, but until recently states have
been working independently. The Region 5 effort to conduct a multi-state ERP
offers great potential to streamline the investment in ERP (e.g., developing
materials and conducting inspections); this should allow for more cost effective
results.
2. Decide on a set of common indicators. Much work on this has already been
done as part of the Common Measures project. It would be helpful to be able to
compare and aggregate ERP data for the same sector across programs.
3. Collect quantitative data on facility practices, not just information on the
proportions of shops utilizing specific practices, for a small number of
indicators. Measuring a few key indicators (e.g., quantity of methylene chloride
used, quantity and type of paint used, amount of hazardous waste generated)
could help quantify environmental outcomes.
4. Develop a tool to help states estimate environmental outcomes. Transforming
the DfE calculator into something states could easily use themselves could
encourage more states to make the effort to collect key inputs to the tool so that
they can generate outcome data.
5. Un-package ERP. While ERP as a package offers value, it requires more
resources and effort than some states can provide on an ongoing basis. We
suggest that states consider different ways to apply the measurement component
ES-viii
-------
ERP Evaluation: Final Report August 2009
of ERP (e.g., for initial assessment and longer-term monitoring) alone as well as
in combination with compliance assistance and self-certification.
6. Consider implementing ERP primarily where larger populations of facilities
are present. This approach has the potential to reduce per facility expenditures
and increase the cost-effectiveness of the program on a per facility basis.
7. Develop a clearer agreement between EPA and states whether ERP can be
used to address traditional regulatory programs. In circumstances where this
is suitable, develop appropriate guidance and a sustainable funding mechanism.
ES-ix
-------
ERP Evaluation: Final Report August 2009
CHAPTER 1 | INTRODUCTION
I. INTRODUCTION AND BACKGROUND
This report summarizes an evaluation of three state Environmental Results Programs
(ERPs) designed to encourage auto body shops to comply with environmental
regulations, adopt best practices, reduce worker health and safety risks, and prevent
pollution. The auto body sector includes between 35,000 and 80,000 facilities across the
country that are in the business of repairing and refmishing vehicles, particularly cars.2
Auto body shops present a wide array of environmental concerns, from use and emissions
of hazardous materials such as methylene chloride, to discharges of polluted water into
water systems, and worker exposure to toxic solvents and particulate matter.
The auto body sector has recently received regulatory attention under the National
Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous
Surface Coating Operations at Area Sources, hereafter, the Surface Coating Rule.3 States
are considering how they may implement these regulations, and some states are
considering adopting ERP as a means to implement the federal rule. Exhibit 1-1
summarizes the rule requirements that will affect auto body shops. Although new auto
body shops (those that started operations after September, 2007) are required to comply
with these requirements by January, 2008 or upon startup of operations, existing auto
body shops must comply with these requirements by January, 2011 (with the exception of
an initial notification date in 2010). As this compliance date approaches, this evaluation
of ERPs in the auto body sector is intended to inform states and EPA regions considering
approaches to improve the environmental performance of the auto body sector.
The next section in this chapter describes the purpose and scope of the evaluation in more
detail. The chapter then provides an introduction to ERP, including a logic model that
illustrates the basic structure and design of this type of program. The following section
presents the specific questions the evaluation is designed to answer and the relationship
2 Estimates of the number of auto body shops in the nation vary widely, from roughly 35,000 to 80,000. A
study conducted for EPA in support of rule development found that the estimated number of auto body shops
nationwide ranged from approximately 35,200 reported by the U.S. Census in 2002, to over 70,000 shops
found in an online "yellow pages" listing. (Source: Eastern Research Group, Memorandum to EPA,
"Estimating the number of automotive refinishing shops and the projected number of new shops," dated
February 22, 2006, included as part of the Technical Support Document for Proposed Rule: National Emission
Standards for Hazardous Air Pollutants Paint Stripping and Miscellaneous Surface Coating Operations at Area
Sources, Docket ID EPA-HQ-OAR-2006-0306, Document ID EPA-HQ-OAR-2006-0306-0041.3. Available online at
http://www. regulations.gov/fdmspublic/component/main?main=DocumentDetail£to=09000064804808ea.
(Last viewed September 24, 2008.) EPA staff familiar with the auto body Surface Coating Rule, estimated
that there are approximately 80,000 auto body shops nation-wide. (Source: Discussion at States ERP
Consortium All Members Meeting August 26th - 28th, 2007)
3 Environmental Protection Agency, FR Vol. 73, No. 6, Wednesday, January 9, 2008. 40 CFR Part 63: National
Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface Coating Operations
at Area Sources. Final Rule, http://www.epa.gov/ttn/atw/area/fr09ia08.pdf. (Last viewed: July 22, 2009).
1
-------
ERP Evaluation: Final Report August 2009
between the evaluation questions and key components of the logic model. The chapter
concludes by orienting readers to the structure of the evaluation report.
EXHIBIT 1-1 SUMMARY OF REQUIREMENT FOR AUTO BODY SHOPS
1) All spray painting must be done in a spray booth
Full cars must be painted in a spray booth with four walls, a roof
and a ventilation system. (Filters in the booth have to remove at
least 98% of the particulates.)
Parts of cars must be painted in a booth with at least three walls or
flaps, a roof and a ventilation system that pulls air into the spray
booth.
Spot repairs must be done in an enclosure which prevents any mist
from getting out of the enclosure.
2) Painters must use spray guns and techniques which reduce overspray
(such as high volume, low pressure, or HVLP, spray guns).
3) All painters must receive training. Owners must keep records of the
training of each painter. (Specific training requirements are specified in
the rule.)
4) Paint spray gun cleaning cannot create any mist of cleaning solvent to the
air. Workers may spray solvent through the gun for cleaning purposes
using an enclosed gun cleaner, or they may clean the gun manually.
5) All shops must also send a notification to EPA with some general
information by January 2010:
Location of facility
Description of spray painting equipment
Confirmation that shop has necessary equipment and training.
6) Exemptions to the rule are facility maintenance activities, which include
the application of coatings to stationary structures or their appurtenances
at the site of installation, to portable buildings at the site of installation,
and to pavements and curbs.
Source: Brief Summary New EPA regulations for Autobody Refinishing Shops, 40 CFR Part 63 Subpart HHHHHH,
August 2008. online at http://www.epa.gov/ttn/atw/area/autobodvbs.doc. (Last viewed: September 24,
2008)
-------
ERP Evaluation: Final Report August 2009
II. PURPOSE AND SCOPE OF THE EVALUATION
This evaluation is intended to inform states and EPA regions who are currently
considering developing programs to encourage auto body shops to adopt best practices4
and improve compliance with environmental regulations. This evaluation considers the
experience of three states in implementing ERP in this sector. Given the significant
differences in the contexts and circumstances in which the programs were implemented,
the intention of this evaluation is to describe each state's experience, rather than to
directly compare their outcomes, costs, or cost-effectiveness.
One purpose of the evaluation is to characterize environmental outcomes associated with
changes in facilities' management practices and compliance at auto body shops resulting
from the ERP program. To assess these outcomes, we analyze and report data collected
by the ERP states on the proportion of auto body shops in each state following specific
best practices before and after the ERP. We then discuss how changes in auto body shops
behavior (e.g., extent of adoption pollution prevention techniques) may translate into
long-term environmental outcomes (e.g., reductions in air emissions) using available
emissions factors.
In addition, this evaluation presents information on the costs required to develop,
implement, and participate in the program, including the staff time and resources invested
by the states, EPA, and participating auto body shops.
We also explore the implementation experience of each state to understand some of the
challenges and opportunities states experienced in implementing ERP programs. We
discuss how these programs unfolded in each state, their current status, and any lessons or
insights that may suggest how states can bolster program sustainability over time.
III. OVERVIEW OF ENVIRONMENTAL RESULTS PROGRAMS
ERP is an innovative approach to improving facilities' management practices within
small business sectors (e.g., dry cleaners, gas stations, and auto body shops).5 ERP is an
integrated system of:
Plain language compliance assistance that promotes pollution prevention;
Facility self-assessment and self-certification;
Agency inspections;
Statistically based performance measurement; and
Where necessary, comprehensive facility investigations and targeted
enforcement actions.
These elements are intended to work together to achieve the goals of ERP, which are to
improve compliance and reduce environmental impacts of the target sector, while
4 The term "best practices" is used to refer to certain environmentally-preferred business practices, such as
use of specific types of efficient spray-painting equipment. We do not seek to distinguish between practices
that are voluntary and those that may be required, now or in the future, by state or federal regulations.
5 The term "sector" is used to refer to groups of facilities that ERP targets. In many cases these are business
sectors, but in other cases ERP targets other groups of regulated entities that are found in multiple business
sectors (e.g., small quantity generators of hazardous waste).
-------
ERP Evaluation: Final Report
August 2009
deploying government resources strategically and efficiently. States believe that ERP's
mix of tools drives facilities to hold themselves more accountable and gives them the
capability and incentive to improve performance.
Exhibit 1-2 shows how the components of ERP are combined in a cyclical process.
Generally speaking, after identifying facilities in the target sector, states conduct
inspections at a random sample of facilities at the outset of the program (i.e., at baseline).
States then offer compliance assistance to all facilities, and then encourage (or in some
cases, require) them to conduct a self-assessment and submit self-certification forms.
This self-certification period is followed in turn by another set of inspections at a random
sample of facilities to measure group performance compared to the baseline. Compliance
assistance specifies how facilities should assess their operations and certify compliance,
while agency inspectors document progress against specific performance indicators,
which are also tracked in self-certification checklists. Performance data, in turn, can
inform and improve the next round of compliance assistance. Note that no two ERPs are
exactly alike, because states have adapted this approach for a wide variety of
circumstances.
EXHIBIT 1-2 A TYPICAL ERP CYCLE
1: Identify tti9 myriad small facilitios that are
sources of pollution, many of which tra ofte n unknown to
regulators,
: . ' : ' i ,-p iH|i -t Conduct random
inspections to a e eurataly measure existing environ m ental
parformancg and focus outreach on the biggost problems.
iVjj \ i lyl - K r, £ «! i,, mr, Wort with trade
associations In create and provid9 plain-language, usar-
friandly assistance that improves compliance and promotes
pollution prevention.
i. FaclMas conduct«K-
assessmsnts using a detailed checklist closely linked to
assistance materials. Responsible officials certify to their
facilities' environmental performance cm each item. If
necessary, they submit plans to return to compliance.
51 t < r«n"-*" ,>l low-Up. Identify potential prabJwn
facilities via certification ana lysis, and ta rget than fa*
inspections, correspondence or phono calls. Provide
assistan ce and/or initiate enforcement, as needed.
& Post-CcnnV - n Conduct
random inspections to accurately estimate performance
changes and verify facility c ertifications.
i* 5 jsiffBT f iii- ut .t v 1.1< i Asssssperfornttncsd«li
and consider whether to adjust compiianc e assistanc e or
other strategies dire cted at the sector or, i sufficient progress
has been made overtime, target resources elsewhere.
-------
ERP Evaluation: Final Report
August 2009
In this document, the term "cycle" refers to this process of random inspections, followed
by facility certification and then follow-up random inspections. The term "round" is used
to designate each new self-certification period. Exhibit 1-3 illustrates the first two cycles
in an ERP over time. In the first ERP cycle, Baseline Random Inspections are followed
by Round 1 Facility Certification, which is in turn followed by another set of random
inspections. These "Post-Round 1 Random Inspections" also serve as the beginning of the
second ERP cycle, when followed by Round 2 Facility Certification and a set of Post-
Round 2 Random Inspections.6
EXHIBIT 1-3 TERMINOLOGY FOR ERP MILESTONES OVER TIME
Baseline Round 1
Random Facility
Inspections Certification
n n
u u
First ERP Cycle
Post-Round 1
Random
Inspections
r\
U
Round 2 Post-Round 2
Facility Random
Certification Inspections
n PI
Tme U U
X
Second ERP Cycle
Round 3
Facility
Certification
0 >
U '
ERPs are typically cross-media focused, meaning that they address multiple types of
pollution (i.e., air emissions, water discharges, or waste management). While individual
facilities within these sectors may release small amounts of pollution, their aggregate
impact can be significant.
The Massachusetts Department of Environmental Protection first developed ERP over 10
years ago as an alternative to traditional site-specific state permits for sectors
characterized by large numbers of small, relatively similar facilities. The ERP approach
has been picked up and adapted by other states, and now 18 states have developed or are
implementing at least one ERP to address environmental issues in one of 11 sectors. To
date, six of those states have applied ERP in the auto body sector.
This evaluation reviews the experience of three states, Delaware, Maine, and Rhode
Island, which established ERPs for the auto body sector. Each of these ERPs
incorporates voluntary self-certification7 and encompasses cross-media environmental
and/or health concerns. While these states are similar in the types of programs they have
implemented, they differ in terms of several circumstances in the state that could affect
ERP implementation, participation of auto body shops, and outcomes. For example,
Rhode Island had been focusing on improving facilities' management practices at auto
body shops for several years before implementing an ERP, and in fact the state had
6 This text is drawn from The States ERP Consortium Guide to Reporting ERP Results, May 2009. The Guide is
intended to help ensure transparency, consistency, and credibility in ERP results reporting.
7 Whether or not facilities opt to participate in the self-certification component of ERP, all facilities are
expected to comply with applicable regulatory requirements.
-------
ERP Evaluation: Final Report August 2009
promulgated regulations specifically requiring auto body shops to adopt practices to
reduce their air emissions nearly 10 years before the start of the ERP. Rhode Island is
also unique in that it has completed two full cycles of ERP, while Delaware and Maine
have completed only one full cycle, although Delaware also conducted a Round 2 Facility
Certification. Delaware developed a source category permit for the auto body sector to
make the permitting process easier for auto body shops, and the Delaware ERP featured
incentives to encourage facilities to participate in ERP by voluntarily self-certifying. The
Maine ERP was geographically focused on, and limited to, three counties in the southern
part of the state that had historically been classified as non-attainment areas (i.e., the
counties had not met ground-level ozone federal national ambient air quality standards).
To illustrate the different components of ERP as applied in the three states under review,
the evaluation team developed a logic model, i.e., a graphical representation of the
relationships between program inputs, outputs, and intended outcomes (see Exhibit 1-4).
Key components of ERP are listed below. Note that the logic model and components of
ERP described below focus on common aspects of ERP shared across all three programs
included in this evaluation. Important distinctions between the programs are noted in the
text.
Resources are the basic inputs of funds, staffing, and knowledge dedicated to
the program. For example, the ERPs in question have been led by staff from
pollution prevention/compliance assistance offices, who have worked in
coordination with regulatory and enforcement staff. Their efforts have typically
been complemented by outside resources such as EPA and contractor support.
We believe that some of the resources listed in Exhibit 1-4 may not have been
utilized in all the ERPs in question. Specifically, not all states included in this
evaluation had access to existing data systems for housing ERP data; delegated
resources redirected to the project; EPA training/feedback on project work plan
and measurement/quality approach; EPA ERP tools and guidance documents
for data management and analysis; or partner agencies.
Activities/Outputs are the specific actions taken to achieve program goals and
the immediate products that result. For example, in ERPs, we typically find 10
categories of activities/outputs: external stakeholder involvement; internal
stakeholder involvement; facility inventory; statistically-based baseline
inspections; compliance assistance; self-certification; targeted follow-up;
statistically-based post-certification inspections; data management and analysis;
and formal reporting. Note that the word "inspection" is used for convenience
in this logic model to refer to all manner of site visits. ERPs typically involve
some combination of formal regulatory inspections and non-regulatory visits
(the latter for assistance and/or measurement). External stakeholder
involvement is generally limited to activities/outputs related to identifying
facilities, and participating in compliance assistance and self-certification.
However, some ERP states may involve external stakeholders in other
activities. For example, Rhode Island Department of Environmental
Management (DEM) worked closely with the industry in developing the self-
certification checklist, with the understanding that the checklist would also
-------
ERP Evaluation: Final Report August 2009
serve as an inspection checklist. Some states also sought feedback from a small
number of auto body shops regarding the design of their programs.
Target Audience represents the groups that will use or be influenced by the
project activities and outputs provided. The principal target audience for ERP
is the population of regulated facilities, in this case auto body shops, whose
changes in attitude and behavior are expected to directly lead to changes in
environmental conditions. Other audiences include internal state agency
stakeholders, EPA, industry leaders, and the public.
Short-Term Outcomes are intended changes in awareness, attitudes,
understanding, knowledge, and skills resulting from program outputs. For
example, states implementing ERP expect that the program will increase
facility owner/operators/workers' awareness of environmental issues associated
with facility operations and business practices that can reduce environmental
and health risks. ERP is also designed to change facility
owner/operators/workers' attitudes, so that they are willing and motivated to
improve their performance. Note that most ERPs focus on cross-media
environmental issues (i.e., air emissions, water discharges, and waste
management), and incorporate both compliance and voluntary best management
practices. Rhode Island's ERP also devotes considerable attention to
addressing worker health and safety issues.
Intermediate Outcomes involve changes in behaviors resulting from short-
term outcomes such as changes in awareness and understanding. For example,
the targeted population is expected to adopt specific improved best practices as
a result of short-term increases in knowledge, awareness, and willingness to act.
Long-Term Outcomes parallel the overarching goals of the program and
include minimizing environmental/health impacts of facility operations (such as
by reducing pollution and waste, minimizing health risk, and improving
water/air quality).
Contextual/External Factors are factors, not directly controlled by the
program or its entities, which may affect program performance. For example,
past ERPs have been forced to adapt substantially when confronted with new
budgetary constraints and/or unexpected changes in federal requirements for the
targeted population.
Note: In the logic model for ERP shown in Exhibit 1-4, the activities and outputs are
generally organized in a rough chronological order (e.g., the stakeholder involvement and
facility inventory precede baseline inspections, which in turn precede compliance
assistance and self-certification.) In contrast, the target audience and outcomes are
arranged roughly in order of importance to achieving the goals of the program. For
example, while all target audiences are important to establishing a successful ERP,
influencing the population of auto body facilities is crucial to achieving improved
compliance and reduced environmental impacts, and internal state agency stakeholders
are central in deploying agency resources strategically and effectively. To make it easier
to track connections between related activities, outputs, target audiences, and outcomes
7
-------
ERP Evaluation: Final Report August 2009
across the logic model, we have used color coding. Activities and outputs designed for,
and outcomes produced by, the target population of facilities are highlighted in light
orange. Activities and outputs that gather input from and inform internal state agency
stakeholders, along with their outcomes, are highlighted in green. Activities and outputs
designed to communicate with external audiences (including EPA, industry leaders, and
the public), along with their outcomes, are highlighted in purple.
Also note that the letter and number codes shown in the logic model in Exhibit 1-4 are
used to show the connection between components of the logic model and the evaluation
questions. These relationships are shown later, in Exhibit 1-6.
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 1-4 LOGIC MODEL FOR ERP
Stale Agency Resources
P2'assistance staff
« Regulatory
enforcement staff
Information
technology staff
State conlraclor
support
E\i\Iiiig tttilif M-.S/IVM.S
Existing
materials experience
vvith ERP and or sector
A-l
EPA Resources
Project-specific grant
funding
to the
project
EPA ftinlniflor
support
EPA
training feedback on
project work plan
Kl'A ERP tools unit
guidance documents
for tluui miuiiijtv
Outside Resources
Other states' ERP
and/or sector
materials, tools and
expertise
Data sets identifying
potential facilities in
the population
Partner agencies
Input from
community public
interest groups
Input from industry
stakeholders
Participation from
target facililie.v
A-.1
External Stakeholder Involvement
Identity and contact industry representatives, community public
interest group1-, paniici agencies, and 01 other external stakeholder*
Internal Stale Agency Stakeholder Involvement
Identify and contact internal stakeholders
Facility Inventory
Create data set identifying potential facilities in ERP population
Statistical Baseline Inspections
ID Environmental Business Practice Indicators (EBPlsl'key measures
Develop inspection checklist
Develop ERP enforcement policy
ID sampling straleuy
Train inspectors on proper data collection and tmiliimedia issues
Conduct baseline random inspections
Compliance Assistance
Provide one-on-one compliance assistance
Develop and deliver workbook, workshops, website and other
a-isistmiee materials
SclC-( erlilkation
ID/develop authority to collect data | for mandatory sell-certification)
!D self-certification incentives, including self-disclosure policy
Develop and distribute self-certification package (i.e.. certification
form, rclurn-to-compliaiicc tbmi. and non-applicability form)
Receive relumed forms
Targeted Follow-lip
Conduct targeted follow-up based upon data received and upon non-
siibmittals (compliance assistance or enforcement-oriented letters,
phone calls, inspections, etc.)
Statistical Post-Certification Inspections
Conduct post-certification random inspections
Data Management and Analysis
Develop project database(s)
Develop and implement approach s\ stem dialling and managing all
ERP data
At each step, analyze data from baseline inspections, certification
process, and post-certification mspei.ln'iis
Target
i Audience I
i....... . ;
i
Short-term
Outcomes
*
1 |
i Intermediate
i i
1 !
1
Long-term
i
External Stakeholder Involvement
Meetings wiih stakeholders and comments on draft
outputs
Internal State Agency Stakeholder Involvement
Meetings with stakeholders and comments on draft
outputs
Facility Inventory
Data set listing polenlial ERP lacililies
Statistical Baseline Inspections
List of HBI'ls key measures
Inspection checklist
Inspector training materials
Documented enforcement policy
Documentation of sampling strategy
Baseline inspection records
Compliance Assistance
One-on-one compliance assistance interactions
Delivered workshops, website and assistance materials
Self-C ertificnlion
Documented self-certification incentives
Delivered self-certification packages
Targeted Follow-up
Records of follow-up activity
Statistical I'ost-Certincation Inspections
Records of post-certification inspections
Data Management ami Analysis
Project dalabaseist
Approach for digitizing and managing all ERP data
Documentation of analysis of data from baseline
inspections, certification process, and post-
eerlification inspections
Formal Reporting
Dewlap d'AM/fa /V/WMV.I far.publiefiftdustiy
Develop final project report
Reporting
Reports to public industry on results
Formal final project lepott (and PowerPoint)
Targeted
Population of
Facilities
(owners, operators
and workers)/
I)-1
Internal State
Agency
Stakeholders
< regulatory/
enforcement slaff
and upper
management)
EPA
Industry Leaders
The Public
Awareness of
environmental issue
associated with
facility operations
« Understanding of
improve business
practices
Willingness
mom ;IIKIII to
improve facilities'
management
practices
Install equipment
needed to improve
management practices
(e.g.. HVLP spray
guns)
Train workers to use
Ivst practices (e.g..
proper handling of
solvents and wastes)
Continue using best
practices over ti
Prevent pollution and
reduce emissions risk from
air water waste discharges
Reduce worker
exposure i isk
Reduce "take home toxics"
affecting workers' families
Improve community's
environment (e.g. by
reducing dust from auto
body shops)
Understand
advantages and
disad\ aiilnges of ERP
Understand
Jiaraetenslie^ and
performance nf the
laiget sector
Support ERP
implementation
Improve efficient
allocation of agency
resources across met! i LI
and sectors, as informed
by ERP
I'-l
Unrteastand ERR
implementation
approach
Understand ERP
performance in terms of
environmental outcomes
and cosl effectiveness
Provide continuing or
increased support for hkt'
at state, regional, and
national level*, as
appropriate ( F-5
Understand benefits of
t;KP to regulated
eommunitv
Promote participation in
ERP among tiirgeled
facilities
Support continued I-RI'
implementation in the
sector
Perceive improvements
in the environmental
health impact of the
sector
Perceive greater
efficiency in use of slale
resources
External Factors Influencing Program and Outcomes:
- Unexpected changes in budget for the stale agency or partners «-
- Changes in federal policy regulations impacting agency's managcmeV l bdit'v t'U ui Ix1 iipplieabk1 for one yr more of the auu> body l:KIJs h*.3inLj
-------
ERP Evaluation: Final Report
August 2009
IV. EVALUATION QUESTIONS
Based on the logic model shown in Exhibit 1-4, the evaluation team developed and
refined the specific questions to be addressed in this evaluation. The evaluation is
designed to answer the following questions related to three categories, as described in
Exhibit 1-5.
EXHIBIT 1-5 EVALUATION QUESTIONS
Environmental Outcomes
1. To what extent have the ERPs in Delaware, Rhode Island, and Maine, led to
actual and/or expected adoption of selected best practices that reduce the
environmental footprint of auto body shops?
2. What environmental and health outcomes are estimated to result from auto body
shops implementing these best practices?
Program Costs/Cost-Effectiveness
3. What are the cost implications of each program for regulators and auto body
shops initially and over time? For example:
a. What resources did states use to develop and implement each program?
b. What resources did auto body shops invest initially, and on an ongoing
basis, to participate in each program?
4. What is the cost-effectiveness of each program?
5. Overall, what are the advantages and disadvantages of each of the three ERPs in
terms of reaching auto body shops, generating environmental and worker health
outcomes, and achieving cost-effective results? (a)
Implementation Experiences
6. What factors influenced the outcomes of each program (e.g., existing or
impending regulations, regulatory/assistance offices involved in conducting the
program, and extent of coordination with industry representatives)?(a)
7. What do our findings suggest regarding the circumstances under which ERPs are
likely to produce cost-effective results?
8. What is the current status of each program?
9. What are the primary implementation challenges states faced in developing and
implementing their programs?
10. What factors influenced the states' decisions to continue, not continue, or modify
their programs after initial pilots?
11. How does the state and/or EPA Region involved in implementing each program
view the program's results, and why?
Notes:
(a) This question relates to environmental outcomes as well as cost effectiveness, but for simplicity of
presentation it is included in the category shown.
10
-------
ERP Evaluation: Final Report
August 2009
Exhibit 1-6 below shows the relationship between the components of the program's logic
model and the evaluation questions. While not every specific component of the logic
model is addressed in the evaluation questions, the questions address the most essential
elements needed to understand the programs' costs and outcomes. Note that evaluation
question 8 (current status of each program) is difficult to show in a logic model format,
since it relates to the evolution of the programs overtime; therefore we have not
attempted to link question 8 to a specific component of the logic model.
EXHIBIT 1-6 RELATIONSHIP BETWEEN EVALUATION QUESTIONS AND ERP LOGIC MODEL
EVALUATION QUESTION
1.) To what extent have the ERPs in Delaware, Rhode Island, and
Maine, led to actual and/or expected adoption of selected
business practices that reduce the environmental footprint of
auto body shops?
2.) What environmental and health outcomes are estimated to
result from auto body shops implementing these best practices?
3.) What are the cost implications of each program for regulators
and auto body shops initially and over time?
a. What resources did states use to develop and
implement each program?
b. What resources did auto body shops invest initially,
and on an ongoing basis, to participate in each
program? (exclusive of resources spent to comply with
regulatory requirements)?
4.) What is the cost-effectiveness of each program?
5.) Overall, what are the advantages and disadvantages of each
of the programs in terms of:
a. Reaching auto body shops
b. Generating environmental and worker health
outcomes
c. Achieving cost-effective results
6.) What factors influenced the outcomes of each program?
7.) What do our findings suggest regarding the circumstances
under which ERPs are likely to produce cost-effective results?
8.) What is the current status of each program?
9.) What are the primary implementation challenges states faced
in developing and implementing their program?
10.) What factors influenced the states' decision to continue, not
continue, or modify their programs after initial pilots?
11.) How does the state and/or EPA Region involved in
implementing each program view the program's results, and why?
KEY COMPONENTS OF ERP
LOGIC MODEL
F-2
F-3
A-1, A-2, A-3
A-3
A, F-4 , F-5
D-1, F-1
F-2, F-3
A, F-3
A, B, C, E
A, B, C, E, F-5
Not shown on logic model
A, B, C, E
A, F, E
11
-------
ERP Evaluation: Final Report August 2009
V. STRUCTURE OF THE REPORT
The remainder of this report describes the approach and findings of the evaluation:
Chapter 2 presents the methodology used in conducting the evaluation,
including the evaluation design, analytic approach, indicators for changes in
behaviors, outcomes, costs and program experiences, and key analytical
limitations.
Chapter 3 present findings for the intermediate and long-term outcomes of the
three programs, including changes in behavior and resulting changes in
environmental conditions.
Chapter 4 present conclusions and recommendations, including an overarching
summary of findings and suggestions for going forward.
12
-------
ERP Evaluation: Final Report August 2009
CHAPTER 2: EVALUATION METHODOLOGY
This chapter summarizes the key aspects of the methodology used to evaluate the three
ERP programs. It begins with a description of the evaluation design, followed by the
analytical approach and the indicators of changes in behaviors, outcomes, costs, and
program experiences. It concludes by discussing the key limitations of the analysis.
I. EVALUATION DESIGN
As described in Chapter 1, the evaluation includes an analysis of three aspects of the ERP
programs:
Environmental outcomes;
Program costs/cost-effectiveness; and
Program implementation experience.
Approaches for addressing each component of this analysis are described below.
II. ANALYTICAL APPROACH
We use existing data reported by each program included in this study to summarize the
change in auto body shop behaviors that affect environmental and health concerns. The
evaluation uses a non-experimental design known as "one group pretest/posttest," which
involves measurement or observation of a group of subjects (auto body shops) prior to
and after the application of an intervention (the ERP). As noted earlier, ERP involves
selecting a random sample of shops prior to ERP to assess baseline performance, and then
selecting a second, independent random sample of shops after ERP implementation. We
compare the percentage of shops using certain best practices during baseline inspections
to the percentage of shops using these same practices after ERP implementation (i.e.,
during post-certification inspections). The basis for these comparisons is data collected
on-site by state inspectors or data collectors at a randomly selected sample of facilities.8
We calculate the observed percentage point change over the ERP cycle (i.e., the
difference between the baseline percentage and the post-certification inspection
percentage). We then use the data from these independent random samples to make
inferences about changes in the proportion of shops in the entire population of facilities
subject to ERP using specific practices. Changes in implementation of best practices
reflect the sector as a whole, not just those shops that elected to participate in the
voluntary ERP self-certification.
8 We do not rely on data reported by facilities on self-certification forms for our summary of intermediate
outcomes (behavior changes).
13
-------
ERP Evaluation: Final Report August 2009
We are not able to quantitatively assess what portion of observed changes in behavior are
a result of the ERP, however, since no control or comparison groups were used that were
not subject to the states' ERP. We note substantial external factors (e.g., other concurrent
outreach and education initiatives) that may have contributed to observed outcomes.
Note that we present the percentage of the sample following each indicator before and
after program deployment, as well as the change in the percentage points. We feel it is
important to show baseline levels of performance in order to provide the complete
picture. For example, if 100 percent of facilities were conducting a certain behavior (e.g.,
using FfVLP spray guns) before the ERP, and all facilities continued this behavior after
the ERP self-certification period, the percentage point change in performance for that
behavior would be zero. Reported on its own, it would appear that the program was not
effective for this indicator, when in reality there was no room for improvement to occur.
We also note that we include all data in our analysis, whether or not it is statistically
significant. We clearly label which results are statistically significant, and which are not.
For results that are statistically significant, we can infer the proportion of facilities in the
entire population (statewide in DE and RI; three counties in ME) using certain practices
changed with a specified degree of confidence. For results that are not statistically
significant, we cannot make this inference; we can only report the observed changes in
the proportion of shops using certain practices for the samples of facilities visited.
III. INDICATORS FOR CHANGES IN BEHAVIORS, OUTCOMES, COSTS, AND
PROGRAM EXPERIENCES
A summary list of the types indicators related to changed behavior tracked by states is
provided in Exhibit 2-1 (Note, the specific wording of each indicator varies by state, and
not all states track each indicator. Exact wording of each indicator tracked by each state
is included in our findings in Chapter 3). This list includes only Environmental Business
Practice Indicators (EBPIs) selected by the state programs, which they prioritized as the
best indicators of compliance and adoption of best management practices for the auto
body sector.
The indicators are organized in five categories of environmental concern: air emissions,
water discharges, waste management, pollution prevented, and worker health & safety.
These categories of concern were selected based on their inclusion in Part IV of EPA's
Comparison Matrix for Element 13 Proposals.9
For each indicator, we summarize the change in percentage of shops that conducted
specific practices (e.g., use low VOC/HAP paints and coatings) for each state that
included that practice as an EBPI. We also summarize the outcomes for each of the five
categories to provide an overarching picture of the states and environmental areas of
concern (air, water, waste, etc.) where the greatest changes have been observed.
9 Included as part of a draft EPA document, "Instructions for Using Comparison Matrix for Element 13
Proposals," May 9, 2007. Personal communication with John Heffelfinger on May 2, 2008. Note that the
Comparison Matrix also identifies additional categories of environmental outcomes for which we do not have
existing data: changes in ecological conditions, population impacts (e.g., impacts on local communities), and
energy/water consumption. Given resource constraints, we do not propose to address these indicators in this
evaluation.
14
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 2-1 INDICATORS OF CHANGES IN BEHAVIOR
CATEGORIES OF
ENVIRONMENTAL
CONCERN
SUMMARY DESCRIPTION OF INDICATORS OF CHANGES IN BEHAVIOR (INTERMEDIATE
OUTCOMES)
Air Emissions
Waste
Management
Related to Types of Paint Usage, change in the percentage of shops that:
Use low VOC/HAP paints and coatings (lower than the federal standard), and
percentage of the shop's painting/coating is done with low VOC/HAP materials
Use compliant surface coatings
Related to Spray Gun Usage, change in the percentage of shops that:
Use HVLP spray equipment and/or other preferred painting application
techniques
Related to Training, change in the percentage of shops that:
Employ a training program in the proper use and handling of coatings, solvents
and/or waste products to minimize air emissions
Related to Spray Booth Usage, change in the percentage of shops that:
Carry out all painting and coating in a spray booth to contain paint emissions and
over-spray
Related to Storage, change in the percentage of shops that:
Store absorbent paint applicators (e.g., shop rags/towels) in closed containers.
Store solvents, thinners, or other VOC and HAP containing materials in closed
containers when not in use.
Related to Practices to Control Dust, change in the percentage of shops that:
Use vacuum sanders or other equipment to control fugitive dust emissions
Have no dust exhausted to the outside, or do not release any airborne emissions
from painting and coating off site
Related to Cleaning/Stripping, change in the percentage of shops that:
Minimize/do not use methylene chloride
Use low VOC/HAP solvents
Utilize an enclosed spray gun cleaner, solvent recycler, or other spray gun
cleaning methods to reduce VOC emissions
Use detergents, high-pressure water, or other non-VOC cleaning options to clean
coating lines and containers when practical
Related to Hazardous Waste, change in the percentage of shops that:
Label containers properly
Properly containerize and dispose of rags and other absorbent materials
contaminated with a listed hazardous waste or flammable waste
Have an EPA ID number
Have a contingency plan
Conduct container inspections
Have a personnel training program
Track waste through a manifest
Track hazardous waste totals
15
-------
ERP Evaluation: Final Report
August 2009
CATEGORIES OF
ENVIRONMENTAL
CONCERN
SUMMARY DESCRIPTION OF INDICATORS OF CHANGES IN BEHAVIOR (INTERMEDIATE
OUTCOMES)
Waste
Management
cont'd.
Water Discharges
Pollution
Prevention1
Worker Health
and Safety11
Conduct complete/accurate hazardous waste determination
Use permitted hazardous waste facilities
Related to Universal Waste, change in the percentage of shops that:
Are aware of universal waste handling requirements
Properly dispose of fluorescent light bulbs
Related to Used Oil, change in the percentage of shops that:
Burn waste oil in a waste oil furnace
Burn waste oil contaminated with hazardous wastes
Undertake any reclamation activities, such as with refrigerants
Change in the percentage of shops that:
Have proper closure/permitting/registering of floor drains
Do not conduct vehicle maintenance and repair in areas with unsealed floor
drains
Do not store oil or hazardous materials in areas with unsealed floor drains
Manage wash water properly
Post signs prohibiting industrial discharge
Have secondary containment
Store chemicals/materials securely away from stormwater
Dry cleaning/sweeping methods employed
Utilize an enclosed spray gun cleaner, solvent recycler, or other spray gun
cleaning methods to reduce or eliminate VOC emissions
Use HVLP spray equipment and/or other preferred painting application
techniques
Painting carried out in a spray booth
Change in the percentage of shops that:
Have a personal protective equipment (PPE) program
Have a respiratory protection program
Conduct hazard communication training
Post safety/health poster
Have a lockout/tag out program
Provide employee medical exams specific to respirator use
Avoid use of methylene chloride-based paint stripper
Control dust emissions
Properly store solvents in closed containers
Carry out painting in a spray booth
Use low VOC/HAP solvents and coatings
Use an enclosed spray gun cleaner
10 These indicators are a subset of indicators in the air, water, and waste categories.
11 Six of these indicators are unique to this category; the remainder a subset of indicators in the air, water,
and waste categories.
16
-------
ERP Evaluation: Final Report
August 2009
Once we analyze changes in practices following each program intervention, we translate
these behaviors into environmental outcomes, where possible. Specifically, for air
emissions, we use an emissions calculator developed by EPA's Design for the
Environment (DfE) program to estimate VOC and PM emissions reductions associated
with adoption of best practices for shops influenced by ERP.12 To use the calculator, we
need to make a number of assumptions and use a number of default values embedded in
the calculator (these values are based on industry research conducted by the DfE
program). We present the emissions reductions associated with two behavior changes:
switching to automatic spray gun cleaning in Delaware, and use of low VOC/water-based
solvents in Delaware and Maine. The DfE calculator also allows us to estimate materials
use reductions associated with automatic spray gun cleaning operations in Delaware. In
addition, we present the potential increase in hazardous waste that is properly being
managed in Delaware following its ERP. (See Exhibit 2-2)
EXHIBIT 2-2 INDICATORS OF CHANGES IN OUTCOMES
CATEGORIES OF
ENVIRONMENTAL CONCERN
Air Emissions
Hazardous Waste
Pollution Prevented
INDICATORS OF ENVIRONMENTAL OUTCOMES (LONG-TERM
OUTCOMES)
Volatile Organic Compounds (VOC)/ Hazardous Air
Pollutants (HAPs) reduced through spray gun cleaning
methods and types of solvents used.
Amounts of hazardous waste that are properly controlled
or managed
Reductions in material usage of solvents
For other categories of environmental concern, we qualitatively discuss the benefits we
would expect facilities would observe as a result of adopting these behaviors. For
example, in the area of worker health and safety, we would expect that having a
respiratory protection program would limit worker exposure to heavy metals, solvents,
and other contaminants in auto repair facilities. We qualitatively discuss the types of
health and safety benefits that would be associated with reducing worker exposure to
these contaminants. Likewise, in the area of behaviors related to hazardous waste
management, we expect that increased adoption of practices such as labeling containers
properly, conducting container inspections, and having a contingency plan would result in
fewer hazardous waste leaks and associated emissions to the atmosphere, fewer facility
emergencies, and more appropriate responses in the event an accident does occur. In our
analysis, we qualitatively describe these expected outcomes associated with the changes
in behavior observed following implementation of ERP.
12 The DfE emissions calculator focuses on emissions reductions associated with best practices for painting,
with a specific focus on VOC and PM emissions. The tool is intended to estimate emissions reductions
associated with certain best practices for a single shop; however data collected in the three states represent
changes for multiple facilities. Therefore, we needed to make a series of assumptions in using the calculator,
which we document in our analysis.
17
-------
ERP Evaluation: Final Report August 2009
Evaluating Program Effectiveness and Cost Implications
In addition to evaluating environmental outcomes, we conducted an assessment of
program costs spent in developing and implementing the three ERPs. To develop these
cost estimates, we reviewed project summary reports submitted to the State Innovation
Grants program for Delaware and Maine, and for all three states we interviewed program
staff representing to assess the amount of staff time (e.g., full-time equivalents, FTEs)
invested over a given period, the dollar value of contractor services, and other direct costs
such as the costs for printing and mailing outreach materials.
We also explored the level of resources expended by auto body shops to participate in the
program. We interviewed five auto body shop owners/operators representing seven auto
body shops that participated in the ERPs. These auto body shops were suggested by the
ERP states, as shops that were representative of typical participants in ERP. A few of the
shops had provided input to the state on the development of ERP, and thus had a broader
perspective on the program (as well as a greater investment in it). In assessing costs of
participation, we asked shops to describe, for example, time and resources facilities
invested in participating in compliance assistance workshops and completing self-
certification forms and return to compliance plans. We do not include the costs to the
facility of purchasing new equipment or changing business practices in order to come into
compliance, since we assume these costs would be constant regardless of the policy tool
used to motivate facilities to comply with the law.
Evaluating Program Implementation Experiences
An important subset of our evaluation questions relates to the implementation experience
of the programs, specifically:
What factors influenced the outcomes of each program?
What do our findings suggest regarding the circumstances under which ERPs
are likely to produce cost-effective results?
What is the current status of each program?
What are the primary implementation challenges states faced in developing and
implementing their program?
What factors influenced the states' decision to continue, not continue, or
modify their programs after initial pilots?
How does the state and/or EPA Region involved in implementing each program
view the program's results, and why?
This evaluation does not have sufficient data to answer these questions quantitatively,
through methods such as correlation analysis. Instead, we answer these questions
qualitatively, by exploring program implementation experiences in interviews with state
regulators, EPA staff involved in supporting ERP, representatives of the States' ERP
Consortium, and a small number of facility representatives.
18
-------
ERP Evaluation: Final Report August 2009
IV. LIMITATIONS OF THE ANALYSIS
A key challenge we faced in conducting the evaluation was that we cannot be certain of
the extent to which the states' ERP caused or contributed to observed changes in
performance, in comparison to other factors (e.g., regulations or permit requirements)
happening concurrently with ERP. We did seek to qualitatively assess other external
factors that may have influenced observed outcomes; however, we did not gather
extensive data on this point.
While we have extensive, statistically-based data on behavior changes associated with
ERP, states did not collect quantitative data on long-term outcomes (e.g., emissions
reductions), and therefore we were limited to estimating this information where we could,
which required a number of assumptions which are described in detail in Appendix A.
Regarding evaluation questions 3-11, our primary data source was interviews with state
representatives directly involved in designing and implementing ERP programs, as well
as EPA staff involved in supporting ERP. While these individuals were best positioned
to have information to respond to the questions, by the nature of their involvement in
ERP, they may have had a biased perspective, and may in some cases have an interest in
advocating ERP. On the other hand, several of these individuals are no longer involved
in implementing ERP, and we heard critiques of ERP as well as praise. Overall, we feel
that our interviews provided a candid assessment of the program from the point of view
of states that have implemented it.
Another challenge in our analysis is that the ERP states included in this evaluation have
not all used the same indicators of changes in behavior (intermediate outcomes).
Therefore, results are not entirely comparable across states. However, this limitation
does not represent a significant concern, since it is not our aim to directly compare the
results across the state ERPs.
In addition, we are limited by the fact that the ERP states have not collected extensive
data on environmental outcomes (e.g., emissions reductions) or costs involved in
implementing the program. We were limited by our ability to extrapolate from available
data, as well as the ability of state representatives to accurately recall or estimate the time
and effort spent in developing their programs.
19
-------
ERP Evaluation: Final Report August 2009
CHAPTER 3: FINDINGS
INTRODUCTION
This chapter documents the findings from our review of three state Environmental
Results Programs (ERPs) designed to improve compliance and environmental
performance of the auto body sector. These findings are based on a review of ERP
performance measures collected and analyzed by three states (Delaware, Maine, and
Rhode Island), as well as limited additional research on environmental impacts of the
sector and projected long-term environmental outcomes (i.e., emissions reductions)
associated with changes in the kinds of behaviors tracked by ERP. These findings also
reflect the perspectives of 12 interviewees:
Four individuals representing three states involved in developing and
implementing the ERPs under review;
Five auto body shop owners/operators representing seven auto body shops that
participated in the ERPs;
Two EPA staff involved in supporting ERPs at the state level; and
Two representatives of the States' ERP Consortium, a collection of 18 states and
three supporting organizations that collaborate in promoting the use of ERP
approaches for effective and efficient environmental protection and improvement.
This chapter is organized as follows:
Section I provides a description of each state's program and its current status.
Section II includes an overview of the behavioral changes and related
environmental outcomes from the three ERPs reviewed.
Section III provides a qualitative review of each program's effectiveness, and a
summary of its costs.
Section IV discusses the implementation experience of each program, including
program advantages/disadvantages, factors influencing program outcomes,
implementation challenges, and program sustainabiliry/transferabiliry.
Section V includes broader reflections on challenges in ERP implementation and
sustainability (not limited to the three state programs that are the focus on this
evaluation) based on interviews with EPA staff and representatives of the States'
ERP Consortium.
Section VI describes characteristics of a successful ERP, and the future direction of
the program overall, drawn from interviews with EPA staff and representatives of
the States' ERP Consortium.
20
-------
ERP Evaluation: Final Report August 2009
I. DESCRIPTION AND STATUS FOR EACH ERP
This section describes the state ERPs developed for the auto body sector in Delaware,
Maine, and Rhode Island. Each of these three ERPs integrates the standard components
of ERP:
Plain language compliance assistance that promotes pollution prevention;
Facility self-assessment and self-certification;
Agency inspections; and
Statistically-based performance measurement. The states measured
performance through baseline and post-certification on-site surveys at a random
sample of facilities, and analyzed this data to make statistical inferences about
changes at facilities across the entire population of shops subject to ERP.
Where necessary, regulators also conducted a comprehensive facility inventory and
targeted enforcement actions. These elements are intended to work together to achieve
the goals of ERP, which are to improve compliance and reduce environmental impacts of
the target sector, while simultaneously deploying government resources strategically and
efficiently.
The three ERP programs reviewed here all commenced in 2003 and 2004.13 Each of the
reviewed states structured their ERPs to encourage, but not require, participation in the
outreach offered and in self-certification.14 Each program encompasses several
environmental issues and/or health concerns (i.e., each ERP addresses multiple
environmental media).
While these states are similar in the types of programs they have implemented, they differ
in terms of several circumstances that could affect ERP implementation, participation of
auto body shops, and outcomes. One difference was the population of auto body shops
that each state was seeking to address: Maine and Delaware were targeting relatively
small populations (104 and 152 shops at baseline, respectively), while Rhode Island was
targeting a larger population of 367 shops.15 Rhode Island was also unique in that the
state had been focusing on improving facilities' management practices at auto body shops
for several years before implementing an ERP, and it had promulgated regulations
specifically requiring auto body shops to adopt practices to reduce their air emissions
nearly 10 years before the start of its ERP. A notable feature of the Delaware ERP was
that the state had developed an air source category permit for the auto body sector to
make the permitting process easier for auto body shops. The Maine ERP was unique in
that it addressed only part of the state: the program was limited to three counties that had
13 For Delaware and Maine, these dates reflect the start of the State Innovation Grants that provided a
significant share of the funding for the ERP. For Rhode Island, this date reflects the launch of the program,
which was initiated after more than two years of industry research and stakeholder meetings.
14 Whether or not facilities opt to participate in the self-certification component of ERP, all facilities are
expected to comply with applicable regulatory requirements.
15 While the population of shops in Rhode Island is about 3.5 times larger than the population in Maine, and
more than twice as large as the population in Delaware, all three states have relatively small populations of
auto body shops, compared to a large state such as Texas, which is estimated to have 5,000 auto body shops.
(Information on the number of auto body shops is based on an interview with representatives of the Texas
Commission on Environmental Quality in June 2008.)
21
-------
ERP Evaluation: Final Report August 2009
historically been classified as non-attainment areas (i.e., the counties had not met ground-
level ozone federal national ambient air quality standards). The Rhode Island ERP is
ongoing, while the programs in Delaware and Maine have concluded.
The following sections describe each state's program development, structure, and status
in more detail.
DELAWARE16
The Delaware Department of Natural Resources and Environmental Control (DNREC)
received an EPA State Innovation Pilot Grant (SIG) to design and conduct its Auto Body
Self-Certification Program. The ERP commenced in March 2003, and was designed to
complement an air source category permit, which DNREC developed for the auto body
sector at the same time as it was developing the ERP. The source category permit was
intended to make it easier for the 152 regulated shops in the state to file for permit
coverage; moreover, DNREC permit staff conducted the emissions calculations required
for the permit, so that auto body shops would not have to determine their own emissions.
Delaware's ERP was not intended to replace permitting, but work in conjunction with the
permitting process. Delaware17 thought that a self-certification program with a strong
outreach and education component would help enhance the regulatory programs within
DNREC and would help move the auto body sector closer to their goal of 100 percent
compliance.
Without the ERP the Air Quality Management Program at DNREC would simply have
implemented the source category permit on its own. Beyond supporting the new permit
and improving compliance, DNREC also sought to use ERP to promote beyond-
compliance best management practices and pollution prevention.
DNREC addressed all environmental regulations pertaining to the auto body sector,
across several environmental media. DNREC offered typical components of ERP
outreach (i.e., workshops, workbook, and self-certification package) for auto body shops
in the state.18 The four ERP workshops explained the program and the broad range of
environmental regulations that applied to this sector; the workbook included a concise,
easy to understand summary of information about environmental requirements and
beyond-compliance practices; and the self-certification package allowed shops to conduct
a self-audit of their environmental performance through a series of yes/no questions. If, in
this audit, shops discovered they were out of compliance in a given area, they were
required to submit a return-to-compliance plan for that area.
16 These findings, as well as those discussed in Section III and IV, are primarily based on an interview with the
Delaware Department of Natural Resources and Environmental Control (DNREC) on February 12, 2009. In
addition to the interview, these findings incorporate supplemental information provided DNREC, information
provided in the Final State Innovation Grant Report for the Delaware Auto Body Self-Certification Program,
which is posted on EPA's website (see
http://www.epa.gov/innovation/stategrants/PDFs/DEautobodvfinalreport.pdf. last accessed May 6, 2009),
and interviews with representatives from two auto body shops that participated in the ERP.
17 Here, and throughout the report, we use the name of the state as shorthand to refer to statements made
by the representatives of the state that we interviewed.
18 Eligible shops include those that have either a) operations or student training in at least one of the
following areas: collision repair, vehicle painting, paint stripping or sanding, body work, antique restoration;
or b) painting operations, as part of a dealership or general auto repair shop.
22
-------
ERP Evaluation: Final Report August 2009
The ERP provided incentives to encourage shops to participate, including:
Waived permit fee: DNREC waived the $165 advertising fee associated with
permit applications.
Amnesty period: DNREC gave shops time to come into compliance without
being penalized, provided they met the provisions of the Penalty Mitigation
Policy.
Technical assistance and audits: DNREC offered free technical assistance,
and conducted pollution prevention audits when requested.
Electronic reporting: DNREC established web-based reporting so that shops
could submit their self-certification forms electronically.
After Delaware's SIG funding ended in September 2005, the state did not continue full
implementation its ERP. The state encouraged program participants to renew their self-
certifications in July 2007, but no further inspections to measure sector performance
occurred after the grant period. Delaware notes that the ERP was a pilot program, and
DNREC does not currently have the sufficient funding or manpower to continue it.
Delaware is incorporating requirements of the federal Surface Coating Rule for auto body
shops19 into its auto body permitting program, which will include an outreach component.
It is also conducting a self-certification component for the dry cleaning sector, but the
state says it cannot afford to conduct a complete ERP for that sector.
MAINE20
The Maine Department of Environmental Protection's (DEP) Office of Innovation
received an EPA State Innovation Pilot Grant in 2004 to conduct its Auto Body ERP pilot
project. The auto body industry in Maine is subject to federal and state air, water, and
solid and hazardous waste regulations; however, there was limited awareness of and
compliance with these regulations in the sector. The DEP sought to address this situation
by implementing an ERP that covered regulations across all environmental media. The
DEP believed that auto body shops were posing a particular concern for air compliance
issues. Beyond increasing compliance and awareness, DEP sought to use ERP to promote
beyond-compliance best management practices and pollution prevention. The program's
manager also wanted to present ERP as a model for the Department that could potentially
be applied to other sectors where small businesses were not receiving adequate regulatory
attention.
19 Environmental Protection Agency, FR Vol. 73, No. 6, Wednesday, January 9, 2008. 40 CFR Part 63: National
Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface Coating Operations
at Area Sources. Final Rule, http://www.epa.gov/ttn/atw/area/fr09ja08.pdf
20 These findings, as well as those discussed in Section III and IV, are primarily based on an interview with the
Maine Department of Environmental Protection on April 8, 2009. In addition to this interview, these findings
incorporate information provided in the Final State Innovation Grant Report for the Maine Auto Body
Environmental Results Program, which is posted on EPA's website (see
http://www.epa.gov/innovation/stategrants/PDFs/maineerpfinalreport.pdf, last accessed May 10, 2009) and
an interview on April 29, 2009 with an owner of three auto body shops that participated in the program.
23
-------
ERP Evaluation: Final Report August 2009
Maine's ERP applied to the 100 auto body shops21 in the state's three southernmost
counties, which were classified as a non-attainment area for ozone. DEP offered
compliance assistance workshops and provided shops with a plain language workbook
that covered regulations, best management practices, and pollution prevention. DEP also
provided shops with a voluntary self-certification checklist that corresponded to the
workbook and allowed shops to self-identify where they were out of compliance through
simple yes/no questions. The ERP provided incentives to encourage shops to participate,
including:
Environmental Leadership recognition: shops who participated in the ERP
were recognized as Environmental Leaders, received a decal to display on-site,
and were listed on the DEP's ERP webpage. Additionally, LaserPaint devices
were awarded to those participants that implemented the most pollution
prevention practices.
Use of incentives policy: The DEP's Small Business Compliance Incentives
Policy (SBCIP) allowed small businesses the opportunity to work with
technical assistance staff from the Department to solve environmental
violations while avoiding the threat of enforcement action, provided certain
requirements were met.
Free technical assistance.
Enforcement action avoidance: self-certification allowed shops to find and fix
violations that could lead to enforcement actions if they were inspected.
Maine's SIG funding ended in March 2007. The state's auto body ERP is no longer
active due to lack of sufficient funding and a hiring freeze that prevents bringing on a
new staff person to coordinate the program. Currently, the DEP is engaged in another
ERP, a pilot program that seeks to reduce the stormwater impacts of drive-through
commercial establishments (e.g., quick service/fast food restaurants). This stormwater
ERP is funded by a separate SIG from EPA.
RHODE ISLAND22
Prior to developing its ERP for the auto body sector, the Rhode Island Department of
Environmental Management (DEM) had been involved in efforts to prevent pollution
from the auto body sector dating back to the early 1990s. With support from EPA Region
1 grants, and in partnership with the University of Rhode Island, local vocational schools,
the state Department of Health, and the auto body industry, Rhode Island DEM
conducted detailed surveys on pollution prevention, environmental controls, and
occupational health and safety practices at auto body shops in the state. These efforts
confirmed significant pollution and health concerns arising from auto body operations.
21 The population of shops was 104 at the outset of the program; it had dropped to 100 at the time of post-
certification inspections.
22 These findings, as well as those discussed in Section III and IV, are primarily based on two interviews with
the Rhode Island Department of Environmental Management (DEM), one on May 14, 2008 and the other on
February 12, 2009 In addition to the interviews, these findings incorporate supplemental information
provided by Rhode Island DEM and interviews with representatives from two auto body shops that
participated in the ERP.
24
-------
ERP Evaluation: Final Report August 2009
In 1994, the DEM implemented specific regulations pertaining to air emissions from auto
body shops. However, due to a lack of resources, less than five percent of shops were
being inspected each year prior to the start of the ERP. When the DEM learned about
ERP and its statistical sampling method from ERP proponents in Massachusetts, it
seemed like the ideal approach to increase compliance and address pollution prevention,
and in the process provide data on the sector's performance over time.
The DEM considered implementing a traditional permitting program instead of ERP.
However, without ERP, the state likely would have simply continued with the existing
low inspection levels, due to the high level of resources a permitting program would have
required.
The state's earlier research had indicated that there were significant occupational health
issues such as lead and methylene chloride exposure occurring in the auto body sector.
Consequently, the ERP was designed to address occupational health issues, as well as
more traditional ERP topics of compliance with air, water, and waste requirements, and
pollution prevention.
DEM started outreach and self-certification in 2003. Unlike the other states discussed in
this report, Rhode Island funded the auto body ERP on its own, without a SIG.23 The
program applied to all auto body shops in the state24, and included standard components
of ERP (technical/compliance assistance, self-certification, and statistically-based
performance measurement). If in the process of self-certification shops discovered they
were out of compliance in a given area, they were required to submit a return to
compliance plan for that area.
DEM developed incentives to encourage shops to participate in the ERP by taking
advantage of compliance assistance and filling out a self-certification form. The
incentives for those firms that self-certified included reduced inspection priority, the
ability to correct violations without gravity-based penalties, and technical and compliance
assistance.
To date, Rhode Island has conducted two cycles of ERP and intends to continue
implementing the program, albeit with minimal additional resources invested in outreach.
(For a description of what constitutes an ERP cycle, see Exhibit 1-3 in the Introduction to
this report.) DEM is adapting the program to be consistent with the federal Surface
Coating Rule for auto body shops,25 and plans to extend it to a third and maybe even a
fourth cycle. Additionally, DEM is currently using ERP in additional sectors, including
23 Rhode Island has received State Innovation Grant funding for its other ERP initiatives, including a 2004
grant for an auto salvage ERP, a 2006 grant for an underground storage tank (UST) ERP, a 2007 grant for a
small municipal separate storm sewer system (AAS4) construction site runoff control ERP, and a 2008 grant
focused on sustainable energy management practices for wastewater treatment, which includes an ERP
component for food processing businesses to encourage use of collected grease as a renewable energy
source. For more information, see the State Innovation Grant website at
http://www.epa.gov/NCEI/stategrants/rhodeisland.htm.
24 All facilities that were licensed by the Rl Department of Business Regulation as an auto body or collision
repair facility were eligible to participate.
25 Environmental Protection Agency, FR Vol. 73, No. 6, Wednesday, January 9, 2008. 40 CFR Part 63: National
Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous Surface Coating Operations
at Area Sources. Final Rule, http://www.epa.gov/ttn/atw/area/fr09ja08.pdf
25
-------
ERP Evaluation: Final Report August 2009
auto salvage, underground storage tanks, MS4 Construction Site Runoff Control, and
reduction of fats/oils/grease discharges from food processing facilities.
II. OVERVIEW OF ERP ENVIRONMENTAL OUTCOMES
This section addresses ERP outcomes related to five categories of environmental
concerns associated with auto body shops: three environmental media (air emissions,
hazardous waste generation and handling, and water discharges), as well as two cross-
cutting issues (worker health and safety, and pollution prevention).
The section begins with an explanation of how ERP measurement works, and then
provides an overview of results across the categories of indicators. For each category of
concern, this section includes a summary of the problem, i.e., how auto body shops
contribute to an environmental or health concern and an overview of relevant behavior
changes that could reduce auto body shops' impacts. Each category summary also
provides a description of ERP outcomes, including observed intermediate outcomes, i.e.,
estimates of the extent to which auto body shops subject to the three state ERPs changed
their behaviors over the course of the three ERPs and anticipated long-term outcomes that
can be expected based on the observed behavior changes. For the most part, these long-
term outcomes are presented in qualitative terms, with the exception of those related to
air emissions, for which we were able to model emissions and materials reductions using
EPA's Design for Environment (DfE) emissions calculator for the auto body sector and
hazardous waste properly managed in Delaware.
This section primarily focuses on outcomes from a single cycle of ERP, since in two
states only a single cycle of the program was completed. However, Rhode Island has
completed two rounds of facility certification and associated measurement. Therefore, at
the end of this section there is a separate discussion of Rhode Island's data from post-
Round 2 random inspections, and how these data compare to baseline and post-Round 1
random inspection data.
EXPLANATION OF ERP MEASUREMENT
As noted earlier, a key component of the standard ERP design is statistically-based
performance measurement. In this design, states measure facility performance at baseline
(i.e., before conducting any compliance assistance or outreach that will be part of the
ERP). To do so, a state selects a representative (random) sample of the facilities eligible
for the ERP, and conducts site visits at each of the facilities in the sample. During the site
visits, state staff (e.g., inspectors, program staff) assess performance by filling out a
detailed checklist that indicates whether or not the facility is following certain compliance
and pollution prevention practices.26 For example, in the auto body sector, a state might
check whether or not each facility in the sample conducts its painting operations within a
spray booth. The state then calculates the percentage of shops in the sample following
this practice. We call this percentage of shops in the sample following a certain practice
the observed proportion of shops.
26 In some cases, inspectors may also collect other types of performance information, e.g., quantities of
specific materials used or hazardous wastes generated, however this type of measurement is less frequently
part of ERP, and was not analyzed by any of the states included in this evaluation.
26
-------
ERP Evaluation: Final Report August 2009
Next, the state conducts the outreach component of ERP, e.g., by holding compliance
assistance workshops, providing compliance assistance materials such as plain language
workbooks, and encouraging27 facilities to fill out self-certification forms. States often
also provide on-site outreach and assistance as part of the ERP. After ERP outreach,
facilities are given time following the certification deadline to implement their return to
compliance plans. Then the state measures performance again by conducting inspections
at a second, independently selected representative (random) sample of the facilities
eligible for ERP (i.e., from the entire population of facilities, not just facilities that
participated in the ERP). This is called the "post-certification" measurement. As in the
baseline, post-certification performance is measured by the percentage of facilities in the
sample (i.e., the observed proportion of shops) that are following key compliance
assistance and pollution prevention practices. The state then looks to see whether the
percentage of shops following each practice changed relative to baseline, for example,
whether the percentage of shops conducting painting operations within a spray booth
increased.
Importantly, the random sampling approach allows states to use information from the
sampled facilities to draw inferences about all facilities being eligible for ERP (i.e., the
population) - not just those facilities visited by inspectors. Thus, by using statistics,
states can use the observed proportion of shops in the sample following each practice to
estimate, within a certain range, the proportion of all shops in the population (all facilities
eligible to participate in ERP) following that practice (this range is often called a
confidence interval28}. Moreover, the state can compare estimates of the population's
baseline performance to estimates of its post-certification performance to assess whether
the overall percentage of the facilities in the entire population following specific practices
changed, and if so, by how much.
When reporting results from their ERPs, states distinguish between changes in
performance that are statistically significant vs. those that are not. A statistically
significant change in performance is one where states can be confident that a change in
performance occurred in the population of all shops eligible for ERP, not just in the
samples of facilities visited. In other words, if a statistically significant change occurred,
a state can be confident that the percentage of all shops in the eligible population
following a certain practice is different at baseline and post-certification. The degree of
confidence in this conclusion is expressed as the confidence level. In our analysis, we test
for statistical significance using a 90 percent confidence level (95 percent in Rhode
Island).29 At a 90 percent confidence level, there is at most a 10 percent chance of being
27 Some ERPs require participating facilities to fill out self-certification forms, however the ERPs included in
this evaluation allowed voluntary self-certification.
28 A confidence interval represents the entire range of possible proportions for the population, whereas the
margin of error is the distance from the observed estimate to each end of the confidence interval range. For
example, if the observed proportion is 10 percent, and the margin of error is 5 percent, then the confidence
interval is 5 percent to 15 percent.
29 For Delaware and Maine, we tested for statistical significance using a 90 percent confidence level,
calculated using a two-sided hypothesis test, which accounts for the possibility that the true change in the
population could have been either positive or negative. Rhode Island tested for statistical significance using a
95 percent confidence level, calculated using a one-sided hypothesis test, which only tests for the possibility
that the true change in the population was greater than zero. The confidence associated with a two sided
27
-------
ERP Evaluation: Final Report August 2009
mistaken in saying that there was a change in performance for the group as a whole (i.e.,
the entire population of auto body shops eligible for ERP).
Saying that a change is statistically significant does not, however, indicate the degree of
change that occurred. Rather, to understand how much performance changed, we need to
consider the difference between the baseline and post-certification measurements, as well
as the confidence interval for the difference. This confidence interval expresses the range
within which we estimate the true difference in population proportions between baseline
and post-certification is expected to fall. Our confidence in this interval is 90 percent for
Delaware and Maine, 95 percent for Rhode Island. Exhibit 3-1 provides an example to
illustrate how this works in practice.
EXHIBIT 3-1 EXAMPLE OF MEASURING CHANGE IN PERFORMANCE
As part of its ERP, Delaware measured the percentage of auto body
shops that had conducted a complete and accurate hazardous waste
determination at baseline and at post-certification. At baseline,
Delaware observed that 17 out of 47 randomly selected shops met this
criterion (36 percent). At post-certification, 43 out of 47 randomly
selected shops met this criterion (91 percent). (Note, these
measurements reflect two independent samples, therefore the set of
shops included in the first sample is not the same set of shops
included in the second sample, although some shops may be included
in both random samples).
The difference in the observed proportion of shops conducting a
complete and accurate hazardous waste determination between
baseline and post-certification is 55 percentage points (91 percent
minus 36 percent). Using statistics, we can calculate that this change
is statistically significant at the 90 percent confidence level. In other
words, we can be 90 percent confident that the percentage of all auto
body shops in the state that were conducting a complete and accurate
hazardous waste determination increased from baseline to post-
certification. Again, using statistics, we can also calculate the
confidence interval for the difference in proportions, to estimate the
degree of change in performance from baseline to post-certification.
In this case, we can be 90 percent confident that the increase in the
percentage of all shops in the state meeting this criterion was
between 44 and 67 percentage points.
Note, if a change in performance is not statistically significant at a particular confidence
level, it could still be true that the entire eligible population proportion changed between
baseline and post-certification, however we cannot be as certain that this occurred.
hypothesis test at the 90 percent confidence level is equivalent to a one-sided hypothesis test at the 95
percent confidence level.
28
-------
ERP Evaluation: Final Report August 2009
Therefore, while real changes in performance can be observed in the facilities included in
the random sample, the lack of statistical significance prevents us from drawing
conclusions about the likely change in performance of the statewide population of auto
body shops. Lack of statistically significant results may occur, even when a change in the
population has actually occurred, for any of several reasons, including samples sizes
being too small to detect a difference, performance changes being too small to detect, and
simply by chance. Note that small increases in performance may occur when the vast
majority of shops are meeting a given criteria and at baseline. For example, if 98 percent
of shops were meeting a certain indicator at baseline, only a 1 or 2 percentage point
increase in performance would be possible. It would be very difficult to detect this small
amount of change unless sample sizes were quite large.
SUMMARY OF OUTCOMES
Each state included in this analysis selected between 19 and 24 indicators of
environmental performance, and measured the percentage of shops meeting these criteria
at baseline and post-certification (see Exhibit 3-2 below). States observed improved
performance between the samples of facilities measured for the vast majority of
indicators (observed performance improved for 54 out of 65 indicators (83 percent)
between baseline and post-certification). Of these 54 indicators, 29 (54 percent of the
indicators where observed performance improved, and 45 percent of all indicators) were
found to have statistically significant changes in performance.
Delaware measured statistically significant improvements in a substantial majority of
indicators (15 out of a total of 19 indicators, or 79 percent, show statistically significant
improvements between baseline and post-certification); in Maine and Rhode Island, less
than half of the improvements observed were statistically significant. Each state
observed a small number of indicators with declining performance, but none of these
declines was statistically significant. For a few indicators, all of the shops in the sample
were already following the best practice being measured, and therefore no improvements
were possible (the count of these indicators is shown in the "# No Change (100%)"
column in Exhibit 3-2 below).
Note, in this section, and throughout the report, to the extent possible the evaluation team
used the raw data30 provided by the states to calculate observed proportions, percent
changes, and which changes are statistically significant. We took this approach to allow
for consistency of analysis. However, in Rhode Island, we did not take this approach,
since that state relied on a more complex, stratified sample, which is also statistically
valid and based on random sampling. Rather than trying to replicate analysis of this data,
we simply report the interpretation of the data provided by the state.
30 Raw data provided by the state included the number of facilities meeting each indicator at both baseline
and post-Round 1 inspections. This did not include facility-specific data; we relied on the states' summary for
each indicator.
29
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-2 SUMMARY OF INDICATORS BY STATE
STATE
Delaware
Maine
Rhode
Island
Overall
#OF
INDICATORS
19
22
24
65
# IMPROVING
(# SIGNIFICANT3)
17(15)
18(7)
19(7)
54 (29)
# WORSENING
(# SIGNIFICANT)
1 (0)
4(0)
3(0)
8(0)
# NO CHANGE
(100%)
1
0
2
3
Note:
a) Significance measured at a 90 percent confidence level in Delaware and Maine, and
95 percent in Rhode Island.
Exhibit 3-3 below shows a summary of indicators broken out by environmental medium.
The greatest number of indicators measured was in the air emissions category, followed
by waste management and water discharges. Relatively few indicators were measured
related to worker health and safety. Note that indicators for pollution prevention are not
included in this chart, since these indicators are a subset of the indicators in the other
environmental media categories. The greatest percentage of the total number of
indicators that made statistically significant improvements were observed in the air
emissions and worker health and safety categories; in both of these categories, half of the
total set of indicators measured (not of those improving, but of the total number of
indicators) showed statistically significant improvements. A small number of indicators
showed worsening performance, but none of these observations were statistically
significant.
EXHIBIT 3-3 SUMMARY OF INDICATORS BY ENVIRONMENTAL MEDIUM
ENVIRONMENTAL MEDIUM
Air Emissions
Waste Management
Water Discharge
Worker Health and Safety(b)
Overall
#OF
INDICATORS
26
21
12
6
65
# IMPROVING (#
SIGNIFICANT3)
19(13)
18(9)
11 (4)
6(3)
54 (29)
# WORSENING
(# SIGNIFICANT)
5(0)
3(0)
0(0)
0(0)
8(0)
# NO CHANGE
(100%)
2
0
1
0
3
Notes:
(a) Significance measured at a 90 percent confidence level in Delaware and Maine, and 95 percent
in Rhode Island.
(b) This row only counts indicators that are exclusively related to worker health and safety. There
are several additional indicators discussed later in this section which relate to both worker health
and safety and air emissions or waste management. These cross-cutting indicators are only
counted once in the table above.
30
-------
ERP Evaluation: Final Report August 2009
The following sections describe each environmental medium in more detail, including
observed changes in behavior and estimated environmental outcomes.
AIR EMISSIONS
Summary of the Problem
The paints, coatings, and solvents used in auto body shops contain several materials that
contribute to air pollution when emitted. In particular, auto body shops contribute to the
following types of air pollutants:
Volatile Organic Compounds (VOCs) - Auto body paints used for coating
operations and solvents used during sanding and cleaning operations contain
volatile organic compounds (VOCs). When paints and solvents are exposed to the
air, the VOCs evaporate and are released into the atmosphere, reacting with
nitrogen oxides in sunlight to form ground-level ozone.31 According to the U.S.
EPA, paints used in auto body shops contain higher concentrations of more
reactive VOCs than do other types of paints. In 2008, the EPA estimated that
120,400 tons of VOCs per year were released from surface coating operations,
which are primarily made up of auto body shops.32
Hazardous Air Pollutants (HAPs) - Many components of paints used in auto
body shops produce significant quantities of HAPs.33 HAPs are toxic air
pollutants that are known or suspected to cause serious health effects, such as
cancer and reproductive effects, and adverse environmental effects such soil or
surface waster deposition.34 Methylene chloride is the primary HAP emitted by
auto body shops, where it evaporates from paint stripping solvents.
Particulate Matter (PM) - Dust from paint pigments and atomized paint from
spray applications are released during shop operations as PM. PM also includes
sanding dust, which can contain toxic metals such as lead and chromium. PM is
made up of acids, organic chemicals, metals, soil, and/or dust particles. The
smaller the particles, the higher the potential for causing health problems, as they
can get deep into the lungs and potentially enter the bloodstream. Potential health
problems from PM inhalation include irritation of the airways, decreased lung
function, aggravated asthma, irregular heartbeat, development of chronic
bronchitis, and nonfatal heart attacks. Environmental effects include: reduced
visibility, lake and stream acidity (caused by PM settling), altered nutrient
balance in coastal water and large river basins, depletion of nutrients in soil,
31 Kansas Small Business Assistance Program - Pollution Prevention Institute at Kansas State University.
"Autobody Shops: A Primer on Environmental Regulation and Pollution Prevention."
http://www.sbeap.org/publications/autobodv.pdf
32 40 CFR part 63: National Emission Standards for Hazardous Air Pollutants: Paint Stripping and Miscellaneous
Surface Coating Operations at Area Sources; Final Rule. Wednesday, January 9, 2008.
33 Kansas Small Business Assistance Program - Pollution Prevention Institute at Kansas State University.
"Autobody Shops: A Primer on Environmental Regulation and Pollution Prevention."
http://www.sbeap.org/publications/autobodv.pdf
34 U.S. EPA Air Toxics Web Site: About Air Toxics, http://www.epa.gov/ttn/atw/allabout.html
31
-------
ERP Evaluation: Final Report August 2009
damage to forests and farm crops, and damage to stone and other building
materials.35
Overview of Relevant Behavior Changes
There are several steps that shops can take to reduce their air emissions. Specifically,
shops can:
Use spray paint booths to reduce paint overspray and fugitive paint emissions;36
Increase paint transfer efficiency by using high transfer efficiency spray
equipment, such as high volume low pressure (HVLP) spray guns;37
Store solvent rags in closed containers;
Manage sanding dust by using a ventilated or wet sander, or by installing room
ventilators and filtration equipment that remove airborne dust;38
Properly train employees in the use of equipment and materials;
Use low-VOC paints, such as waterborne paints. Standard solvent-based paints
contain 4.5 to 5.5 pounds of VOC per gallon, while waterborne paints only
contain 1.9 pounds per gallon;39
Reduce or eliminate the use of methylene chloride-based paint strippers;
Clean spray guns with an enclosed spray gun cleaner, one that recirculates
cleaning solvent during the cleaning process and is vapor tight;40
Use less toxic solvents, install a gun washer, or adopt distillation/recycling
practices;41
Minimize the use of paint additives such as chemical hardeners, flex additives,
and retarders.42
35 U.S. EPA Air Et Radiation Website: Particulate Matter - Health and Environment.
http://www.epa.gov/air/particlepoUution/health.html
36 Missouri Department of Natural Resources, Division of Environmental Quality. "Preventing Pollution in
Collision Repair". May, 2007.
37 Illinois Sustainable Technology Center at the University of Illinois at Urbana-Champaign. "Metal Painting
and Coating Operations: Overview of Pollution Prevention in Coating Application Processes."
http://www.istc.illinois.edu/info/library_docs/manuals/coatings/overvp2.htm
38 Pollution Prevention in Auto Body Shops and Paint Shops, "Sanding Waste Management", CA Department of
Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD FS SandingWaste.pdf
39 Edwards, Joseph D., Local Hazardous Waste Management Program in King County, Technical Assistance and
Pollution Prevention Team. "Waterborne Coatings and the Autobody Shop: A Status Report. Publication
Number SQG-ABODY-3(10/94) rev 9/00, October, 1994.
40 Rhode Island Department of Environmental Management - Office of Technical £t Customer Assistance, Small
Business Assistance Program. "Air Pollution Control in Autobody Shops.
http://www.dem.ri.gov/programs/benviron/assist/pdf/airconab.pdf
41 Illinois Sustainable Technology Center at the University of Illinois at Urbana-Champaign. "Metal Painting
and Coating Operations: Overview of Pollution Prevention in Coating Application Processes."
http://www.istc.illinois.edu/info/librarv docs/manuals/coatings/overvp2.htm
42 Kansas Small Business Assistance Program - Pollution Prevention Institute at Kansas State University.
"Autobody Shops: A Primer on Environmental Regulation and Pollution Prevention."
http://www.sbeap.org/publications/autobody.pdf
32
-------
ERP Evaluation: Final Report August 2009
Use waterborne cleaners, and use mechanical cleaning when possible.
43
Summary of ERP Outcomes
In this section of the evaluation we are interested in the extent to which the ERPs led to
adoption of selected best practices that reduce air emissions from auto body shops, as
well as the environmental and health outcomes estimated to result from the
implementation of these best practices. We classify adoption of best practices as
intermediate outcomes (as noted in the ERP logic model). We consider environmental
and health outcomes (e.g., emissions reductions associated with adopting these best
practices) as long-term outcomes. We consider intermediate and long-term outcomes of
ERP related to air emissions in turn in the sections that follow.
Intermediate Outcomes
The ERP states included in this evaluation tracked several indicators of performance
relevant to air emissions, including behaviors related to:
Painting techniques, equipment, and materials;
Cleaning techniques, equipment, and materials;
Methylene chloride-based paint stripper usage;
Sanding equipment;
Emissions control equipment;
Materials storage; and
Employee training.
Exhibit 4 shows the specific indicators tracked, along with the observed percentage of
facilities in the sample following each practice sampled at baseline and post-certification,
and the difference in percentages between these two measurements (i.e., the observed
percentage point change). The exhibit then shows whether each observed change in
behavior is statistically significant. Findings are listed by state, and ordered from the
greatest observed change to the smallest observed change. Those changes that are
statistically significant are shown in bold. Note, in cases where indicators measured a
change in a negative condition (e.g., when airborne emissions from painting and coating
leave the business premises), we transformed the data so that increases in observed
percentages represent an improvement.
43 Kansas Small Business Assistance Program - Pollution Prevention Institute at Kansas State University.
"Autobody Shops: A Primer on Environmental Regulation and Pollution Prevention."
http://www.sbeap.org/publications/autobody.pdf
33
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-4 SUMMARY OF AIR EMISSIONS INDICATORS44
INDICATOR
BASELINE
o/0(a)
POST-
CERTIFICATION
INSPECTIONS
o/0(b)
OBSERVED
PERCENTAGE
POINT
CHANGE'0'
SIGNIFICANT?
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
DELAWARE
Does the shop employ a training program in the proper use and handling of
coatings, solvents and waste products to minimize air emissions?
Does the shop store absorbent paint applicators (e.g., shop rags/towels) in
closed containers?
Does the shop use detergents, high-pressure water, or other non-VOC
cleaning options to clean coating lines and containers when practical?
Does the shop clean the spray guns using only methods that comply with
Delaware Regulations?
Does any airborne sanding or painting dust (i.e. fugitive dust) leave the
business premises and create a condition of air pollution? (Note, since this
indicator measures a condition that the state wants shops to discontinue, the
percentages shown here reflect the number of shops not performing this
practice. )
47%
53%
11%
89%
89%
91%
96%
35%
100%
98%
45
43
24
11
9
yes
yes
yes
yes
yes
(33 - 56)
(32 - 53)
(13-36)
(5-17)
(2-15)
44 For Delaware and Maine, these calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and post-certification, the number of facilities
sampled at each time period for which the answer to the question was "yes," and the total number of facilities in the population). Raw data was drawn from the State Innovation Grant final reports
for these two states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdf and http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We
then used this information to calculate the observed proportion of shops at baseline and post-certification, the observed percentage point change, whether or not this change is significant, and the
confidence interval for the difference. We used the ERP Results Analyzer tool to conduct these calculations The Results Analyzer calculates confidence intervals for the difference between
proportions observed in two different rounds of inspections. A version of the Results Analyzer can be accessed at: http://www.epa.gov/erp (however, the authors used an updated version of the
results analyzer that had not been posted on line as of the time of writing this report; for the most up-to-date version of the Results Analyzer, contact Scott Bowles, bowles.scott@epa.gov).
Calculations for the confidence intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York, NY. p.41. We use a 90 percent, two-sided
significance test and a 90 percent confidence interval in our calculations. In both Maine and Delaware, the number of facilities in both samples was relatively small (59 and 47 facilities, respectively).
Smaller sample sizes typically only show significant results when observed changes are fairly large. We believe that 90 percent significance and confidence levels are acceptable in this case. We use
the raw data, rather than observed percentages, statistical significance, or confidence intervals reported by the states to ensure consistency in the analytic approach. In some cases, rounding leads
to an observed percentage point change that is slightly different than the difference between the reported baseline and post-certification inspection percentages.
For Rhode Island, we could not use the ERP Results Analyzer because the state had a stratified random sampling approach (this approach is also statistically valid, but requires more complex
analytical techniques). Therefore, for Rhode Island we simply report the values given by the state in the article summarizing their results (Enander et al., "Environmental Health Practice: Statistically
Based Performance Measurement," American Journal of Public Health.) Note that Rl reports a 95 percent confidence interval and a one-sided significance test.
34
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Do the painters and technicians use only painting techniques that comply
with Delaware Regulations?
Is all painting carried out in a spray booth to contain paint emissions and over-
spray?
Does the facility avoid any use of methylene chloride-based paint strippers?
MAINE
Does the shop utilize low VOC/HAP solvents?
Does the shop utilize low VOC/HAP paints and coatings? (lower than the
federal standard)
Does the shop utilize a dust control system to control dust generated from
the sanding process?
Do any airborne emissions from painting and coating leave the business
premises? (Note, since this indicator measures a condition that the state wants
shops to discontinue, the percentages shown here reflect the number of shops
not performing this practice.)
Does the shop exhaust air from process areas to the outside? (Note, since this
indicator measures a condition that the state wants shops to discontinue, the
percentages shown here reflect the number of shops not performing this
practice. )
Does painting and coating take place in areas outside of a spray booth? (Note,
since this indicator measures a condition that the state wants shops to
discontinue, the percentages shown here reflect the number of shops answering
"no" to the question.)
Does the shop carry out painting and coating in a spray booth to contain paint
emissions and over spray?
Does the shop utilize an enclosed spray gun cleaner, solvent recycler, or other
spray gun cleaning methods to reduce or eliminate VOC emissions?
Are solvents, thinners, or other VOC and HAP containing materials stored in
closed containers when not in use?
BASELINE
o/0(a)
89%
81%
43%
49%
59%
34%
71%
27%
75%
80%
54%
93%
POST-
CERTIFICATION
INSPECTIONS
o/0(b)
98%
87%
35%
97%
93%
47%
85%
37%
85%
86%
61%
90%
OBSERVED
PERCENTAGE
POINT
CHANGED
9
6
-9%
47
34
14
14
10
10
7
7
-3
SIGNIFICANT?
yes
no
no
yes
yes
yes
yes
yes
yes
no
no
no
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
(2-15)
N/A
N/A
(40 - 55)
(26 - 42)
(4 - 23)
(5 - 22)
(1 -19)
(2-18)
N/A
N/A
N/A
35
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Does the shop train all employees in the proper use and handling of paints and
coatings according to the manufacturers' recommendations to minimize air
pollution?
Does the shop employ a training program in the proper use and handling of
solvents and waste products to minimize air emissions?
Does your facility use a methylene chloride-based paint stripper? (Note, since
this indicator measures a condition that the state wants shops to discontinue,
the percentages shown here reflect the number of shops not performing this
practice. )
Do you control dust emissions from your facility using a specific device?
Do you store solvents, waste paint, sludge, and shop rags/towels saturated with
solvent in closed containers?
Is your cleaning device totally enclosed during cleaning, rinsing, and draining
operations?
Do you use coatings that comply with the emission limitations listed in Rhode
Island Air Pollution Control Regulation No. 30, Control of Volatile Organic
Compounds from Automobile Refinishing Operations?
Do your painters and technicians use spray guns that have a transfer efficiency of
at least 65% such as High Volume Low Pressure (HVLP) spray equipment?
Does your shop use a ventilated sander (dustless vacuum) system?
BASELINE
o/0(a)
100%
100%
67%
33%
81%
83%
100%
100%
31%
POST-
CERTIFICATION
INSPECTIONS
o/0(b)
97%
97%
95%
48%
88%
88%
100%
100%
30%
OBSERVED
PERCENTAGE
POINT
CHANGED
-3
-3
28
15
7
5
0
0
-1
SIGNIFICANT?
no
no
yes
no
no
no
not tested
not tested
not tested
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
N/A
N/A
(12-44)
N/A
N/A
N/A
N/A
N/A
N/A
36
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
BASELINE
o/0(a)
POST-
CERTIFICATION
INSPECTIONS
o/0(b)
OBSERVED
PERCENTAGE
POINT
CHANGED
SIGNIFICANT?
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
Notes:
(a) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the baseline sample for which the answer to the indicator question is
"Yes," which indicates positive environmental performance.
(b) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the post-certification sample for which the answer to the indicator
question is "Yes," which indicates positive environmental performance.
(c) All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of shops in the sample following the behavior at
baseline prior to ERP from the observed percentage of shops in the sample following the behavior at post-certification. We use percentage point changes, rather than percent
changes, to more clearly show the magnitude of change. For example, suppose the observed proportion of shops following a behavior increased from 6 percent at baseline to 12
percent at post certification: this change could be expressed as a 100 percent improvement, or a 6 percentage point improvement. We believe the latter approach is a clearer,
more accurate description of changes in performance.
37
-------
ERP Evaluation: Final Report August 2009
The ERPs addressed in this evaluation included more indicators related to air pollution
than any other category. States observed improvements in the majority of the air
emissions indicators tracked (i.e., 19 out of 26 indicators, or 73 percent shown in Exhibit
3-4). Thirteen of these improvements were statistically significant, meaning that we can
infer that the total universe of shops eligible for the ERPs showed improvements for these
indicators. Among the statistically significant improvements, the average observed
change was 23 percentage points. Performance on five of the indicators declined;
however, none of those declines were statistically significant.
Since each state selected a different set of indicators, the types of observed improvements
varied among states. However it is notable that two states measured statistically
significant improvements related to sanding dust. Specifically, Maine detected a 14
percentage point improvement in shops using a dust control system, while Delaware
found a nine percentage point improvement in shops preventing fugitive dust from
leaving the building premises. Rhode Island also observed a 15 percentage point
improvement in shops controlling dust emissions; however this change was not
statistically significant.
Another area where multiple states measured statistically significant improvements
related to the types of solvents used at the shops. Specifically, Maine observed a 47
percentage point increase in shops using low VOC/HAP solvents. Rhode Island observed
a 28 percentage point reduction in the percentage of shops using methylene chloride-
based paint strippers.45 On the other hand, Delaware also measured the percentage of
shops using methylene chloride-based paint strippers, and found that more shops were
using this solvent at post-certification than at baseline, which represents a worsening of
environmental performance (although the decline in performance was not statistically
significant).
Notably, indicators for which the greatest improvements were observed in some states
showed minimal improvements or declines in performance in other states. For example,
one of the largest observed improvements in this category was in Delaware, where the
state observed a 45 percentage point improvement related to employee training for use
and handling of coatings, solvents and wastes. However, for a very similar indicator
(employee training in the proper use and handing of paints and coatings according to
manufacturers recommendations), Maine found a three percentage point decline in
performance. (This decline was not statistically significant.)
Aside from changes in performance, the ERP data reveal interesting patterns in the
overall level of performance at post-certification. The indicators where one or more
states observed at least 95 percent of shops sampled following preferred behaviors
include:
Use of spray guns cleaning methods that comply with state regulations
Use of coatings that comply with emissions limits
45 Note, this finding applies to Rhode Island's first cycle of ERP. For further results on Rhode Island's findings
for this indicator in its second cycle of ERP, see the section "Discussion of Results from a Second Cycle of
ERP" later in this report.
38
-------
ERP Evaluation: Final Report August 2009
Use of HVLP spray guns
Prevention of fugitive dust from leaving the business premises
Use of painting techniques that comply with state regulations
Use of low VOC/HAP solvents
Employee training related to proper use and handling of paints and coatings
Employee training related to proper use and handling of solvents and waste
products
Proper storage of absorbent paint applicators
Avoiding use of methylene chloride-based paint stripper
Conversely, the behaviors where at least one state observed relatively poor performance
(less than 40 percent of shops following preferred behavior) include:
Allowing air from process areas to exhaust to the outside
Use of methylene chloride-based paint strippers
Use of non-VOC options to clean coating lines and containers when practical
Use of a ventilated sander system
Notably, there are some indicators (e.g., fugitive dust/air from process areas exhausted to
the outside and use of methylene chloride-based paint strippers) that were observed at a
high level of performance in one state, but a low level of performance in another state.
These variations in performance may well be due external factors, e.g., the history of each
state in terms of prior outreach to auto body shops, rather than the influence of the ERP
itself in affecting shop behavior. These variations could also be the result of differences
in ERP implementation between the states.
Long-term Outcomes
The Emissions Reduction Calculator from the U.S. EPA's Design for the Environment
(DfE) Program46 was developed to provide rough estimates for reductions of VOCs, PM,
and materials used through implementation of certain best practices in auto body shops.
The calculator estimates that there are five primary best practices that small shops can
adopt to reduce their emissions and materials used: (1) waterborne paint usage, (2) HVLP
spray gun usage, (3) training in spray gun usage, (4) spray booth usage, and (5)
equipment cleaning improvements. The tool is intended to estimate emissions reductions
associated with these practices for a single shop; however, the data available to the
evaluators represent changes in the percentage of a group of facilities following these key
practices. Therefore, a number of assumptions are required to use the emissions calculator
to estimate emissions reductions associated with ERP, as described below and in
Appendix A.
This analysis considers observed behavior changes that are statistically significant and are
modeled in the DfE tool to extrapolate emissions reductions from facilities in the sample
46 A copy of the Emissions Reduction Calculator is available at:
http://www.epa.gov/dfe/pubs/projects/auto/
39
-------
ERP Evaluation: Final Report August 2009
to facilities across the state. There are two behavior changes that meet these criteria:
switching to automatic spray gun cleaning in Delaware, and use of low-VOC/water-based
solvents in Delaware and Maine. Specifically, for each of these best practices we use the
emissions factors built into the tool to estimate a range of annual emissions reductions
that could be expected. The results of this analysis are summarized below, and details on
the assumptions and methodology for conducting the analysis are included in Appendix
A.
Emissions reductions associated with increased use of automatic cleaning
methods at shops in Delaware.47 We are 90 percent confident that the
percentage point increase in the percent of shops cleaning spray guns only with
methods that comply with Delaware regulations is between 5 and 17 percentage
points. We expect that the VOCs reductions from this increase in the percent of
auto body shop improving their spray gun cleaning operations in the entire
population of auto body shops in Delaware would lead to a decline of between
296 and 962 Ibs/year, relative to baseline.48
Emissions reductions associated with increase in shops in Delaware that use
non-VOC cleaning methods when possible.49 We are 90 percent confident that
the percentage point increase in the percent of shops that use non-VOC cleaning
methods when possible is between 13 and 36 percentage points. We expect that
the VOCs reductions from this increase in auto body shops improving their
cleaning operations in the entire population of auto body shops in Delaware
would lead to a decline of up to 3,355 Ibs/year, relative to baseline. We include
only the maximum of this range because it is difficult to estimate the minimum
emissions reductions, since some shops likely began using non-VOC cleaning
methods when possible, but did not completely switch over to low-VOC/water-
based solvents. If many of the shops that Delaware counted as having switched
to using non-VOC cleaning methods when possible were only using these non-
VOC materials for a small percentage of their cleaning operations, the emissions
reductions would be much lower.
Emissions reductions associated with increase in shops in Maine that use low
VOC/HAP solvents.50 We are 90 percent confident that the percentage point
increase in the percent of shops that use low VOC/HAP solvents is between 40
and 55 percentage points. We expect that the VOCs reductions from this increase
in auto body shops improving their cleaning operations at shops across the state
47 To calculate this emissions decrease we assume adoption of automatic cleaning methods (the variable
tracked in the DfE calculator) is equivalent to the indicator tracked by Delaware: use of cleaning methods
that comply with Delaware regulations (specifically, enclosed spray gun cleaning systems that are kept closed
when not in use, unatomized discharge of solvent into a paint waste container that is kept closed when not in
use, disassembly of the spray gun and cleaning in a vat that is kept closed when not in use, or atomized spray
into a paint waste container that is fitted with a device designed to capture atomized solvent emissions).
48 The percentage point increases reported in this section represent the increase in the percentage of shops
adopting a certain best practice over the course of the ERP. The emissions reductions reported reflect the
annual emissions reductions that are estimated from the increase in shops adopting that behavior.
49 To calculate this emissions decrease we assume adoption of low VOC-water-based cleaners is equivalent to
shops that use non-VOC cleaning methods when possible.
50 To calculate this emissions decrease we assume adoption of low VOC-water-based cleaners is equivalent to
shops that use non-VOC cleaning methods when possible.
40
-------
ERP Evaluation: Final Report August 2009
would lead to a decline of up to 3,416 Ibs/year, relative to baseline. As in
Delaware, we include only the maximum of this range because it is difficult to
estimate the minimum emissions reductions, since some shops likely began using
low VOC /HAP solvents, but did not completely switch over to these alternative
solvents. If many of the shops that Maine counted as having switched to low
VOC/HAP solvents were only using low VOC or water-based solvents for a
small percentage of their cleaning operations, the emissions reductions would be
much lower.
Note that each practice is considered individually, but in reality the practices influence on
another (e.g., type of spray gun used and training for spray gun use are interrelated, and
there is overlap in the potential emissions reductions between these two indicators).
Therefore, the results are not additive, i.e., the total potential emissions reductions
associated with adopting all practices is less than the sum of potential emissions
reductions for each practice. In other words, the DfE calculator combines emissions
reductions associated with using automatic cleaning methods and low VOC/water-based
solvents into one estimate. However, to use the ERP data, we have broken out these
behaviors into separate estimates. If we used the DfE calculator to estimate the emissions
associated with a single shop that used automatic cleaning methods and low VOC/water-
based solvents, the emissions reductions would be less than if we used the DfE calculator
to estimate the emissions reductions associated with each of these behaviors separately,
and then added them together.
Appendix B presents default values embedded in the DfE calculator which are used in our
analysis. Appendix C presents potential emissions estimates that could be expected for
each typical small auto body shop that adopts certain best practices tracked in the DfE
tool.
Based on estimates from the Surface Coating Rule of the VOC emissions from small auto
body shops before improvements51, the declines in Delaware and Maine in VOC
emissions represent the following percentage VOC emissions reductions per shop:
Automatic cleaning methods: 0.6 percent reduction in total VOC emissions per
shop.
Low-VOC or water-based cleaning methods: 0.9 percent reduction in total VOC
emissions per shop.
While these percent reductions are small, they only represent the percent reductions from
the change associated with a single behavior change. For comparison, the final Surface
51 The final Surface Coating Rule (40 CFR part 63: National Emission Standards for Hazardous Air Pollutants:
Paint Stripping and Miscellaneous Surface Coating Operations at Area Sources; Final Rule. Wednesday,
January 9, 2008) estimates that there are 36,000 surface coating operations in the United States, and 35,000
of these are auto body shops involved in motor vehicle and mobile equipment. Further, the rule estimates
that VOC emissions for surface coating operations is 120,400 tons per year. The rule does not directly
estimate emissions associated with auto body shops, excluding other surface coating operations. However,
if, we assume that per shop emissions of auto body shops are equivalent to emissions from other surface
coating operations, then 1736 of the 120,400 tons per year total estimate (or 3,344 tons per year) can be
attributed to other surface coating operations, and the remainder (117,056 tons per year) can be attributed
to auto body shops. Further, to calculate average VOC emissions per shop, we divide 117,056 tons by 35,000
shops to estimate total average emissions per shop of 3.34 tons per year.
41
-------
ERP Evaluation: Final Report August 2009
Coating Rule estimates achieving full compliance will the rule will results in a 17.4
percent reduction in total VOC emissions per shop.
HAZARDOUS WASTE
Summary of the Problem
Auto body shops generate many materials that are regulated as hazardous wastes. For
example, waste solvent and coatings, contaminated rags, wipes and absorbents, empty
containers, used oil, waste antifreeze, sanding or grinding dusts, and contaminated wash
waters.52 These wastes can generally be classified as hazardous because they are
ignitable, corrosive, reactive, and/or toxic.53 They can be in liquid, solid, contained gas,
or sludge form and are considered dangerous or potentially harmful to human health or
the environment.54 Used solvent combined with paint waste is often the largest waste
stream in auto body shops.55 If not properly managed, the wastes generated by shops have
the potential to be accidentally released into the environment.
State and federal regulations dictate procedures for the proper handling, management, and
storage of hazardous wastes. Hazardous waste generators are divided into categories
based on the amount of waste they produce each month, and different regulations apply to
each generator category. In general, there are three categories of hazardous waste
generators, as described in Exhibit 3-5.
In addition, shops must handle universal wastes, which are not considered "hazardous"
but are collected and managed with hazardous wastes, because they have at least one of
the four characteristics of hazardous waste.56'57 For auto body shops, this includes wastes
such as fluorescent light bulbs and car batteries. In the shop, these wastes must be
handled as hazardous.
52 Pollution Prevention in Auto Body Shops and Paint Shops, "Hazardous Waste Management", CA Department
of Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD FS Hazwaste.pdf
53 U.S. EPA, Hazardous Waste Information Pages. Available at:
http://www.epa.gov/ebtpages/wasthazardouswaste.html. or
http://www.epa.gov/osw/hazard/wastetypes/characteristic.htm
54 U.S. EPA Hazardous Waste Home Page. Available at: http://www.epa.gov/waste/hazard/index.htm
55 Pollution Prevention in Auto Body Shops and Paint Shops, "Solvent Recycling", CA Department of Toxic
Substances Control, September 2006. Available at:
http: / /www. dtsc. ca.gov/PollutionPrevention /ABP /upload /TD_FS_SolventRecvcling. pdf
56 Rhode Island Department of Environmental Management, Office of Technical and Customer Assistance.
"Rhode Island Universal Waste Rule Fact Sheet." January 2003. Available at:
http://www.dem.ri.gov/programs/benviron/assist/pdf/univrule.pdf
57 U.S. EPA "Universal Wastes". Available at:
http://www.epa.gov/osw/hazard/wastetypes/universal/index.htm
42
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-5 HAZARDOUS WASTE GENERATOR CATEGORIES
Large Quantity
Generators
Small Quantity
Generators
(called "Small
Quantity Generator
Plus" in Maine)
Conditionally
Exempt Small
Quantity Generators
(called "Small
Quantity
Generators" in
Maine)
DELAWARE1
Generates more than
2,200 pounds or 300
gallons (1,000
kilograms) of hazardous
waste in any calendar
month.
Generates more than
220 and less than 2,200
pounds or about 25 to
under 300 gallons
(between 100 kilograms
and less than 1,000
kilograms) of hazardous
waste in any calendar
month.
Generates no more
than 220 pounds or 25
gallons (100 kilograms)
of hazardous waste in
any calendar month
MAINE2
Generates more than
220 pounds (about 27
gallons) of hazardous
waste per month or
accumulates more than
1,320 pounds of
hazardous waste on site
at any one time.
Generates less than 220
pounds (about 27
gallons) of hazardous
waste per month and
accumulates one to
three drums, but no
more than 1,320
pounds of hazardous
waste on site at any
one time.
Generates less than 220
pounds (about 27
gallons) of hazardous
waste per month and
accumulates a total of
no more than 55 gallons
(1 drum) of hazardous
waste on site at any
one time.
RHODE ISLAND3
Generates more than
1 ,000 kilograms per
month of hazardous
waste, more than 1
kilogram per month of
acutely hazardous
waste, or more than 100
kilograms per month of
acute spill residue or
soil.
Generates more than
100 kilograms, but less
than 1 ,000 kilograms, of
hazardous waste per
month.
Generates 100 kilograms
or less per month of
hazardous waste, or 1
kilogram or less per
month of acutely
hazardous waste, or less
than 100 kilograms per
month of acute spill
residue or soil.
Sources:
1 . Delaware Department of Natural Resources and Environmental Control. "Delaware's
Hazardous Waste Regulations and You." January 1995.
2. Maine Department of Environmental Protection. "Auto Body Environmental Results Program:
Workbook". February 2006.
3. Rhode Island Department of Environmental Management. "Rules and Regulations for
Hazardous Waste Management." February 9, 2007, Section 5.02 D. and U.S. Code of Federal
Regulations §261.5 (a) and (e), §262.34 (d) and §262.
Overview of Relevant Behavior Changes
There are several steps that an auto body shop can take to reduce the amount of hazardous
waste that it generates and to properly manage the waste that it does generate. State
compliance regulations for the auto body sector differ to some degree; however, in
general, to properly manage their waste, shops must:
Properly determine what waste qualifies as hazardous;
Obtain a hazardous waste identification number;
Properly label hazardous waste containers;
Maintain accurate recordkeeping;
Develop emergency procedures; and
43
-------
ERP Evaluation: Final Report August 2009
Ship waste to a permitted facility and/or store waste on-site in a storage area that
meets criteria for secondary containment.
In addition, shops can reduce the amount of hazardous waste they generate for example,
by:
Using enclosed spray gun cleaners - These systems reduce waste solvent by
recirculating the cleaning solvent until it can no longer be reused.58
Mixing paint efficiently - There are several steps shop workers can take to
reduce wasted paint that must be managed as hazardous, including:
o Managing inventory: Limit the amount of paint that must be thrown
away as a result of overstocking or old paint becoming unusable (i.e.,
from separation).
o Matching colors: Better color matching and/or using small test panels
reduce the need to re-spray if the color is mixed incorrectly.
o Measuring paint: Mix only the amount of paint needed.
o Minimizing paint transfers: When paint is transferred from one container
to another, some paint sticks to the original container and is wasted.
Shops can use disposable spray gun cup liners to reduce this transfer
waste, because paint can be mixed and sprayed in the same cup.59
Improving painting efficiency - Improving the efficiency of the painting
process reduces the amount of paint that is wasted and must be managed as
hazardous waste. Shops can:
o Plan primer and clear coat work on multiple cars back-to-back;
o Use tintable primer systems to improve color matches and get complete
coverage with fewer coats;
o Remove parts and perform like jobs together;
o Improve training of employees on proper painting techniques;
o Use high volume low pressure (HVLP) spray guns;
o Use laser-based spray paint technology;
o Maintain equipment; and
o Use waterborne paints.60
Recycling solvent - Recycling solvent reduces the amount of used solvent
hazardous waste shipped off-site.61
58 Enander, Richard T., Gute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in Rhode Island Automotive Refinishing Industry." American Industrial Hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
59 Pollution Prevention in Auto Body Shops and Paint Shops, "Minimizing Paint Waste", CA Department of
Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD FS PaintWasteMin.pdf
60 Ibid
44
-------
ERP Evaluation: Final Report August 2009
Summary of ERP Outcomes
Taking steps to reduce or recycle potentially hazardous wastes should result in less waste
that needs be managed as hazardous in the environment. Proper management of the
hazardous wastes that are generated reduces the likelihood of accidental releases or other
associated hazards to the environment or human health.
Intermediate Outcomes
Exhibit 3-6 shows the specific indicators tracked, along with the observed percentage of
facilities in the sample following each practice at baseline and post-certification, and the
difference in percentages between these two measurements (i.e., the observed percentage
point change). The exhibit then shows whether each observed change in behavior is
statistically significant at a 90 percent confidence level. For each change that is
statistically significant, it presents the confidence interval for the difference (i.e., range
within which we can be 90 percent confident the true percentage point change for the
entire population of facilities will fall). Findings are listed by state, and ordered from the
greatest observed change to the smallest observed change. Those changes that are
statistically significant are shown in bold. Note, in cases where indicators measured a
change in a negative condition (e.g., when shops add hazardous wastes such as waste
gasoline, solvents, or paint thinner into the waste oil), we transformed the data so that
increases in observed percentages represent an improvement.
In total, across all three ERP states, this category of indicators improved substantially. Of
the 21 indicators in this category, 18 (86 percent) improved, and nine of those
improvements were statistically significant, meaning that we can infer with 90 percent
confidence that the total universe of shops eligible for the ERPs showed improvements
for these indicators. Among the statistically significant improvements, the average
observed change was 36 percentage points. The performance for three indicators
declined, but none of those changes were statistically significant.
61 Pollution Prevention in Auto Body Shops and Paint Shops, "Solvent Recycling", CA Department of Toxic
Substances Control, September 2006. Available at:
http: / /www. dtsc. ca.gov/PollutionPrevention /ABP /upload /TD_FS_SolventRecvcling. pdf
45
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-6: SUMMARY OF HAZARDOUS WASTE INDICATORS62
INDICATOR
DELAWARE
Has the shop conducted a complete, accurate hazardous
waste determination for each waste it generates?
Does the shop have a program that trains employees who
handle hazardous waste in proper waste management
procedures?
Are shop employees aware that batteries, mercury
thermostats, and mercury containing fluorescent light bulbs
need to be handled according to Delaware requirements for
universal waste?
Are all hazardous waste containers properly labeled?
Does the shop track hazardous waste accumulation totals?
Does the shop send all hazardous wastes to a permitted
hazardous waste treatment storage, or disposal facility or a
state authorized facility?
BASELINE
o/0(a)
36%
34%
44%
45%
53%
66%
POST-CERTIFICATION
INSPECTIONS %(b)
91%
89%
91%
91%
85%
91%
OBSERVED
PERCENTAGE
POINT CHANGED
55
55
47
47
32
26
SIGNIFICANT?
yes
yes
yes
yes
yes
yes
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
(44 - 67)
(44 - 67)
(35 - 59)
(35 - 58)
(20 - 44)
(14-37)
62 For Delaware and Maine, these calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and post-certification, the number of facilities
sampled at each time period for which the answer to the question was "yes," and the total number of facilities in the population). Raw data was drawn from the State Innovation Grant final reports
for these two states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdf and http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We
then used this information to calculate the observed proportion of shops at baseline and post-certification, the observed percentage point change, whether or not this change is significant, and the
confidence interval for the difference. We used the ERP Results Analyzer tool to conduct these calculations. The Results Analyzer calculates confidence intervals for the difference between
proportions observed in two different rounds of inspections. A version of the Results Analyzer can be accessed at: http://www.epa.gov/erp (however, the authors used an updated version of the
results analyzer that had not been posted on line as of the time of writing this report; for the most up-to-date version of the Results Analyzer, contact Scott Bowles, bowles.scott@epa.gov).
Calculations for the confidence intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York, NY. p.41 We use a 90 percent, two-sided
significance test and a 90 percent confidence interval in our calculations. In both Maine and Delaware, the number of facilities in both samples was relatively small (59 and 47 facilities, respectively).
Smaller sample sizes typically only show significant results when observed changes are fairly large. We believe that 90 percent significance and confidence levels are acceptable in this case. We use
the raw data, rather than observed percentages, statistical significance, or confidence intervals reported by the states to ensure consistency in the analytic approach. In some cases, rounding leads
to an observed percentage point change that is slightly different than the difference between the reported baseline and post-certification inspection percentages.
For Rhode Island, we could not use the ERP Results Analyzer because the state had a stratified random sampling approach (this approach is also statistically valid, but requires more complex
analytical techniques). Therefore, for Rhode Island we simply report the values given by the state in the article summarizing their results (Enander et al., "Environmental Health Practice: Statistically
Based Performance Measurement," American Journal of Public Health.) Note that Rl reports a 95 percent confidence interval and a one-sided significance test.
46
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Is the shop undertaking any reclamation activities?
MAINE
Does the shop properly dispose of (recycle) fluorescent light
bulbs?
Has anyone ever filled out or signed a hazardous waste manifest?
Does the shop burn oil in a waste oil furnace?
Does the shop properly label containers of hazardous waste?
Does the shop containerize rags and other absorbent materials
contaminated with a listed hazardous waste or flammable waste
and dispose of it as hazardous waste?
Does the shop ever add hazardous wastes such as waste gasoline,
solvents, or paint thinner into the waste oil? (Note, since this
indicator measures a condition that the state wants shops to
discontinue, the percentages shown here reflect the number of
shops not performing this practice.)
ISLAND
Do you have appropriate documentation which shows where
hazardous waste is being shipped?
Is the area (satellite accumulation area) clearly marked and the
containers labeled with: (1 ) the words "Hazardous Waste", (2)
Name of the waste and its waste code?, (3) the hazard
classification?, and (4) the date that they were placed in the
storage area?
Has your shop submitted to DEM a list of authorized agents that
are allowed to sign the manifest (hazardous waste manifest)?
Does your shop have a written contingency plan designed to help
your shop reduce hazards associated with the possibility of an
explosion, fire, or unplanned/accidental release of hazardous
materials?
Is the area (hazardous waste storage area) inspected weekly for
signs of spills or container deterioration?
BASELINE
o/0(a)
47%
29%
71%
28%
68%
27%
100%
56%
21%
28%
6%
6%
POST-CERTIFICATION
INSPECTIONS %(b)
70%
85%
78%
32%
71%
29%
96%
89%
39%
44%
22%
22%
OBSERVED
PERCENTAGE
POINT CHANGED
23
56
7
4
3
2
-4
33
18
16
16
16
SIGNIFICANT?
yes
yes
no
no
no
no
no
yes
no
no
no
no
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
(10-37)
(48 - 64)
N/A
N/A
N/A
N/A
N/A
(15-51)
N/A
N/A
N/A
N/A
47
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
BASELINE
o/0(a)
POST-CERTIFICATION
INSPECTIONS %(b)
OBSERVED
PERCENTAGE
POINT CHANGED
SIGNIFICANT?
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
Does your shop have an employee training program that teaches
them proper hazardous waste management procedures, including
how to implement the contingency plan?
Do you use coatings that comply with the emission limitations
listed in Rhode Island Air Pollution Control Regulation No. 30,
Control of Volatile Organic Compounds from Automobile
Refinishing Operations?
What is your facility's hazardous waste identification number?
Does your hazardous waste storage area meet the criteria for
secondary containment (i.e. spill/leak containment capability)?
6%
22%
16
no
N/A
100%
88%
63%
100%
86%
-2
-7
not tested
not tested
not tested
N/A
N/A
N/A
Notes:
(a) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the baseline sample for which the answer to the indicator question is
"Yes," which indicates positive environmental performance.
(b) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the post-certification sample for which the answer to the indicator
question is "Yes," which indicates positive environmental performance.
(c) All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of shops in the sample following the behavior at
baseline prior to ERP from the observed percentage of shops in the sample following the behavior at post-certification. We use percentage point changes, rather than percent
changes, to more clearly show the magnitude of change. For example, suppose the observed proportion of shops following a behavior increased from 6 percent at baseline to 12
percent at post certification: this change could be expressed as a 100 percent improvement, or a 6 percentage point improvement. We believe the latter approach is a clearer,
more accurate description of changes in performance.
48
-------
ERP Evaluation: Final Report August 2009
Delaware by far saw the greatest improvements in this category, as all seven of their
indicators significantly improved. In addition, the average performance increase for these
indicators was 41 percentage points. Delaware's improvements were primarily related to
the proper management of hazardous waste. For example, the observed proportion of
shops properly identifying their hazardous waste and training their employees in proper
management procedures increased by 55 percentage points over the course of the ERP,
while the observed proportion of shops tracking hazardous waste accumulation totals
increased by 32 percentage points and the proportion of shops shipping their waste to
permitted or state authorized facilities increased by 26 percentage points.
Maine and Rhode Island observed more modest improvements in the management of
hazardous waste. Both of these states detected one statistically significant improvement in
performance related to hazardous waste management. Specifically, Rhode Island found a
33 percentage point increase in the percent of shops with appropriate documentation
showing where hazardous waste is being shipped, while Maine found a 56 percentage
point increase in shops properly disposing of fluorescent light bulbs.
In addition to changes in performance, the ERP data also track overall levels of
performance at post-certification. In contrast to the air emissions category, where for 10
indicators at least 95 percent of shops sampled at post-certification were following
preferred behaviors; in the waste management category, only one indicator showed this
level of performance. Specifically, Maine observed that 96 percent of shops at post
certification did not ever add hazardous wastes such as waste gasoline, solvents, or paint
thinner into their waste oil. (Note that this performance represented a decline from
baseline, when 100 percent of shops met this criterion.) Other waste management
indicators with relatively high levels of performance (at least 90 percent compliance with
the indicator at post-certification) in at least one state include:
Hazardous waste determination
Awareness of requirements for universal waste
Labeling of hazardous waste containers
Sending hazardous wastes to a permitted/state authorized facility
Conversely, the behaviors where at least one state observed relatively poor performance
(less than 40 percent of shops following preferred behavior) include:
Labeling of hazardous waste containers
Burning oil in a waste oil furnace
Proper handling of rags and other absorbent materials contaminated with
hazardous waste
Having a written contingency plan
Inspection of hazardous waste storage areas
Employee training for proper hazardous waste management procedures
Notably, one of these indicators (labeling of hazardous waste containers), is an area of
relatively high performance in one state and poor performance in another. As in the air
49
-------
ERP Evaluation: Final Report August 2009
emissions category, these variations in performance across states may be due to external
factors, e.g., the history of each state in terms of prior outreach to auto body shops, rather
than differences in the influence of the ERP itself in affecting shop behavior.
Long-term Outcomes
In general, we expect that changes in the kinds of behaviors observed by the three states
should result in better management of hazardous waste and fewer potential accidental
releases to the environment. While these states did not track the amounts of waste
generated by each shop, which would allow us to quantify the amount of hazardous waste
managed, we can develop a first order approximation of the potential increase in the
amount of hazardous waste controlled in one state due to improved waste management
procedures.
Specifically, Delaware found that nearly all shops inspected were Conditionally Exempt
Small Quantity Generators (CESQGs) (45 out of 47 shops in the baseline inspections met
this criterion). According to regulatory requirements, CESQGs are not allowed to
generate more than 220 pounds or 25 gallons of hazardous waste in any calendar month.
(While these shops may generate less than 220 pounds of waste a month, that quantity
represents the potential amount of waste to be controlled.) In developing this estimate,
we assume that all shops in the state are CESQGs. In addition, since we cannot estimate
the actual hazardous waste generated by each shop every month, we assume that each
shop is generating the maximum allowed by their regulatory status (as CESQGs). In
reality, these shops could (and most likely are) generating less than 220 pounds of waste
per month. To estimate the total potential amount of waste generated by the shops in the
state, we multiply the 152 shops in the state by 220 pounds of waste for each shop, which
equals 33,440 pounds of hazardous waste per month). This means that the 152 shops in
the state have a combined potential to generate 33,440 pounds (16.7 tons) of hazardous
waste a month, according to the statutory limit.
Moreover, based on input from EPA, we assume that complete and accurate hazardous
waste determination for each waste generated is a useful indicator of proper management
of hazardous waste. Using the data from Delaware, we know that Delaware observed a
maximum 67 percentage point increase in the number of shops that complete hazardous
waste determination from baseline to post-Round 1 random inspections. This translates to
an increase of 102 shops completing hazardous waste determinations (67 percent of the
152 shops in the state). Again, according to the regulations, each shop can generate a
maximum of 220 pounds of waste per month. If the 102 additional shops properly
manage their 220 pounds of waste, this means that there is potentially an increase of as
much as 22,440 pounds (11.2 tons) per month of hazardous waste properly being
identified as hazardous waste in the state (although the actual amount may be far less, and
we are not able to estimate bottom of the range). If the increase in waste being properly
50
-------
ERP Evaluation: Final Report August 2009
identified is equal to 22,440 pounds per month, this would be equivalent to the minimum
waste generated by 10 large quantity generators per month.63
WATER DISCHARGES
Summary of the Problem
If not controlled properly, wastewater produced by auto body shops has the potential to
contribute harmful pollutants into groundwater, storm drains, wastewater systems and
soil. Several activities in shops have the potential to release pollutants to surface waters,
including: surface preparation, wet and dry sanding, painting, vehicle washing, floor
cleaning, and product and waste storage. Water wastes generated during these processes
include:
Sanding operations - heavy metals like cadmium, chromium, lead, nickel, and
zinc;
Vehicle preparation - oil, grease, and coolant removed from vehicles;
Painting process - toxic chemicals from cleaners, strippers, solvents, and paints
in the form of scrubber water, paint sludge, spent solvents, aqueous cleaners, and
wastewater.
Vehicle washing - soaps.
64,65,66
Unused floor drains should be properly sealed, but if they are currently active, shops
generally must comply with their state's Underground Injection Control Program's
regulations to ensure that wastewater discharges do not have the potential of polluting the
soil and groundwater.67
Overview of Relevant Behavior Changes
There are two primary categories of behaviors that shops can employ to reduce the
likelihood of releasing harmful pollutants into the water systems: spill prevention and
drainage control. In other words, shops should manage the wastewater that they create so
that it does not pollute. Shops can take the following steps to prevent spills in their shops:
63 This is based on the high end of the confidence interval for the difference between baseline and post-
Round 1 random inspections, measured in percentage points (67), multiplied by the number of shops in
Delaware (152) and the potential amount of waste generated (220 pounds per month). We use only the high
end of the confidence interval because we are attempting to quantify the maximum potential amount of
waste that is properly characterized as hazardous; in fact CESQGS may generate much less than 220 pounds
of hazardous waste per month, and therefore we cannot accurately estimate the minimum amount of waste
that shops in Delaware may be properly characterizing. In addition, if hazardous waste determination is not
the best indicator of good management practices, then we may not be capturing the true improvement in
performance for this indicator.
64 Illinois Sustainable Technology Center at the University of Illinois at Urbana-Champaign. "Metal Painting
and Coating Operations." http://www.istc.illinois.edu/info/librarv docs/manuals/coatings/toc.htm
65 Enander, Richard T., et. al. "The Concordance of Pollution Prevention and Occupational Health and Safety:
A Perspective on U.S. Policy. American Journal of Industrial Medicine. Vole. 44, 2003, pp. 312-320.
66 Pollution Prevention in Auto Body Shops and Paint Shops, "Wastewater Management", CA Department of
Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD FS WastewaterMgmt.pdf
67 Rhode Island Department of Environmental Management - Office of Technical £t Customer Assistance, Small
Business Assistance Program. Factsheet: Water Pollution in the Autbody Shop.
http://www.dem.ri.gov/programs/benviron/assist/pdf/h2opcab.pdf
51
-------
ERP Evaluation: Final Report August 2009
Keeping containers closed;
Using secondary containment for hazardous materials storage;
Inspecting containers for leaks;
Storing hazardous materials away from sanitary sewer and storm drains;
Keeping trash containers and dumpsters closed and inspect for leaks; and
Inspecting vehicles for leaks and use drip pans as needed.68
Shops should also control any active drains in their shops by:
Clearly marking all indoor drains;
Posting signs prohibiting the discharge of industrial chemicals to non-industrial
outlets;
Properly closing all inactive floor drains;
Registering all active (and inactive, if required) floor drains as required by state
regulations; and
Having a system in place for recycling or proper disposal of wastewater.
Shops can also use dry cleaning methods to reduce the amount of wastewater they
generate during the cleaning process.
Summary of ERP Outcomes
By managing the wastewater they produce, auto body shops can reduce the potential for
the harmful pollutants in that wastewater to leak into groundwater. Properly closing
inactive floor drains ensures that no wastewater is leaking into sewer systems. Increasing
awareness about the importance of not discharging wastewater into non-industrial
drainage outlets and not performing maintenance in areas with unsealed floor drains
decreases the likelihood of accidental leakages.
Exhibit 3-7 shows the specific indicators tracked, along with the observed percentage of
facilities in the sample following each practice at baseline and post-certification, and the
difference in percentages between these two measurements (i.e., the observed percentage
point change). The exhibit then shows whether each observed change in behavior is
statistically significant at a 90 percent confidence level. For each change that is
statistically significant, it presents the confidence interval for the difference (i.e., range
within which we can be 90 percent confident the true percentage point change for the
entire population of facilities will fall). Findings are listed by state, and ordered from the
greatest observed change to the smallest observed change. Those changes that are
statistically significant are shown in bold. Note, in cases where indicators measured a
change in a negative condition (e.g., when shops conduct vehicle maintenance and repair
in areas with unsealed floor drains), we transformed the data so that increases in observed
percentages represent an improvement.
68 Pollution Prevention in Auto Body Shops and Paint Shops, "Wastewater Management", CA Department of
Toxic Substances Control, September 2006. Available at:
http:/ /www. dtsc.ca.gov/PollutionPrevention/ABP /upload /TD_FS_WastewaterMgmt.pdf
52
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-7 SUMMARY OF WASTEWATER INDICATORS69
INDICATOR
DELAWARE
Does the shop post signs prohibiting the discharge of
industrial chemicals and/or wastewater to
bathroom/kitchen sinks, toilets, showers, shop wash
basins, emergency showers, eyewash stations, or
other non-industrial drainage outlets?
Does the shop have secondary containment for all
chemicals, including paints, thinners, strippers,
cleaners and automotive fluids, so as to prevent
potential spills from entering open floor drains or
other access ways to water sources?
Does the shop use dry cleaning methods such as
sweeping and vacuuming, when cleaning the shop?
Does the shop keep paints, cleaners, and any chemicals
or materials that can cause runoff (indoors or otherwise)
protected from rainwater?
BASELINE %(a)
4%
49%
98%
100%
POST-
CERTIFICATION
INSPECTIONS %(b)
81%
79%
100%
100%
OBSERVED
PERCENTAGE
POINT
CHANGE'0'
77
30
2
0
SIGNIFICANT?
yes
yes
no
no
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
(68 - 86 )
(17-43)
N/A
N/A
69 For Delaware and Maine, these calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and post-certification, the
number of facilities sampled at each time period for which the answer to the question was "yes," and the total number of facilities in the population). Raw data was drawn from
the State Innovation Grant final reports for these two states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdfand
http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We then used this information to calculate the observed proportion of shops at baseline and post-
certification, the observed percentage point change, whether or not this change is significant, and the confidence interval for the difference. We used the ERP Results Analyzer
tool to conduct these calculations. The Results Analyzer calculates confidence intervals for the difference between proportions observed in two different rounds of inspections. A
version of the Results Analyzer can be accessed at: http://www.epa.gov/erp (however, the authors used an updated version of the results analyzer that had not been posted on
line as of the time of writing this report; for the most up-to-date version of the Results Analyzer, contact Scott Bowles, bowles.scott@epa.gov). Calculations for the confidence
intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York, NY. p.41 We use a 90 percent, two-sided significance test and a
90 percent confidence interval in our calculations. In both Maine and Delaware, the number of facilities in both samples was relatively small (59 and 47 facilities, respectively).
Smaller sample sizes typically only show significant results when observed changes are fairly large. We believe that 90 percent significance and confidence levels are acceptable in
this case. We use the raw data, rather than observed percentages, statistical significance, or confidence intervals reported by the states to ensure consistency in the analytic
approach. In some cases, rounding leads to an observed percentage point change that is slightly different than the difference between the reported baseline and post-
certification inspection percentages.
For Rhode Island, we could not use the ERP Results Analyzer because the state had a stratified random sampling approach (this approach is also statistically valid, but requires
more complex analytical techniques). Therefore, for Rhode Island we simply report the values given by the state in the article summarizing their results (Enander et al.,
"Environmental Health Practice: Statistically Based Performance Measurement," American Journal of Public Health.) Note that Rl reports a 95 percent confidence interval and a
one-sided significance test.
53
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
MAINE
Have inactive floor drains been properly sealed /closed?
Does the shop conduct vehicle maintenance and repair in
areas (bays) with unsealed floor drains? (Note, since this
indicator measures a condition that the state wants
shops to discontinue, the percentages shown here reflect
the number of shops not performing this practice.)
Does the shop store oil or hazardous materials in areas
that have unsealed floor drains?
(Note, since this indicator measures a condition that the
state wants shops to discontinue, the percentages shown
here reflect the number of shops not performing this
practice. )
Does the shop have any active floor drains? (Note, since
this indicator measures a condition that the state wants
shops to discontinue, the percentages shown here reflect
the number of shops not performing this practice.)
Are active and inactive floor drains registered with the
DEP?
Does your shop post signs prohibiting the discharge of
industrial chemicals and/or wastewater to
bathroom/kitchen sinks, toilets, showers, shop wash
basins, emergency showers, eyewash stations, or
other non-industrial drainage outlets?
Does your shop allow process wastewater (i.e., from
wet sanding, car washing, work area washing) to run
off your site to storm drains or other areas (i.e., water
runs down the street, water runs off to soil or sand
area, water just puddles up and evaporates)?
(Note, since this indicator measures a condition that the
state wants shops to discontinue, the percentages shown
here reflect the number of shops not performing this
practice. )
BASELINE %(a)
75%
77%
96%
31%
39%
0%
37%
POST-
CERTIFICATION
INSPECTIONS %(b)
87%
86%
100%
33%
41%
48%
74%
OBSERVED
PERCENTAGE
POINT
CHANGED
12
9
4
3
2
48
37
SIGNIFICANT?
no
no
no
no
no
yes
yes
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
N/A
N/A
N/A
N/A
N/A
(33 - 63 )
(17-57)
54
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Does your shop contain open floor drains that are not
currently in use?
(Note, since this indicator measures a condition that the
state wants shops to discontinue, the percentages shown
here reflect the number of shops not performing this
practice. )
BASELINE %(a)
67%
POST-
CERTIFICATION
INSPECTIONS %(b)
69%
OBSERVED
PERCENTAGE
POINT
CHANGED
2
SIGNIFICANT?
no
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
N/A
Notes:
(a) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the baseline sample for which the answer to the
indicator question is "Yes," which indicates positive environmental performance.
(b) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the post-certification sample for which the answer to
the indicator question is "Yes," which indicates positive environmental performance.
(c) All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of shops in the sample following
the behavior at baseline prior to ERP from the observed percentage of shops in the sample following the behavior at post-certification. We use percentage
point changes, rather than percent changes, to more clearly show the magnitude of change. For example, suppose the observed proportion of shops following
a behavior increased from 6 percent at baseline to 12 percent at post certification: this change could be expressed as a 100 percent improvement, or a 6
percentage point improvement. We believe the latter approach is a clearer, more accurate description of changes in performance.
55
-------
ERP Evaluation: Final Report August 2009
States observed improvements at the sampled facilities in almost all indicators for this
category. Of the 12 wastewater indicators, 11 (92 percent) improved, and the one that did
not improve had a baseline compliance level of 100%, so had no room for improvement.
Four of the improvements were statistically significant, meaning that we can infer with 90
percent confidence that the total universe of shops eligible for the ERPs showed
improvements for these indicators. Among the statistically significant improvements, the
average observed change was 48 percentage points.
Both Delaware and Rhode Island observed significant improvement in the percentage of
shops that post signs prohibiting the discharge of industrial chemicals to non-industrial
drainage outlets, with 77 and 48 percentage point improvements respectively.
Delaware found a statistically significant improvement in the percentage of shops that
have secondary containment for all chemicals. In addition, all of the shops included in the
samples in Delaware were keeping paints, cleaners, and any chemicals that can cause
runoff protected from rainwater at both the baseline and post-certification inspections.
Maine saw modest improvements in all of the indicators in this category for the facilities
in the sample. None of them were statistically significant, however, meaning that we
cannot make inferences with a 90 percent degree of confidence about whether there were
changes related to these indicators among the entire population of facilities.
Rhode Island found a 37 percentage point statistically significant improvement in the
percent of shops that have a proper system in place for recycling or properly disposing of
wastewater, which should substantially reduce the likelihood of that wastewater
accidentally leaking into water sources.
In addition to changes in performance, the ERP data also track overall levels of
performance at post-certification. For three indicators, at least 95 percent of shops
sampled in at least one state showed preferred behaviors:
Use of dry cleaning methods such as sweeping and vacuuming;
Keeping chemicals or materials that can cause runoff protected from rainwater;
and
Avoiding storing oil or hazardous materials in areas that have unsealed floor
drains.
Conversely, poor performance (less than 40 percent of shops following the preferred
behavior) was observed for one indicator at post-certification:
Avoiding having any active floor drains.
WORKER HEALTH ft SAFETY
Summary of the Problem
Operations in auto body shops potentially expose workers to a variety of harmful
contaminants. Paint formulations, surface and equipment cleaners, adhesives, and paint
strippers contain organic solvents, isocyanates, metal particulates and other airborne
contaminants. In addition, metal-bearing fumes, paint pigments, and other fine solid
particulates are generated during welding, spray painting, and sanding/ grinding
56
-------
ERP Evaluation: Final Report August 2009
operations. In general, during these operations, workers are exposed through: (1)
inhalation of volatized material, (2) inhalation of particulates, fibers, and gases, (3)
dermal absorption of solvents and isocyanate monomers, and (4) incidental ingestion of
contaminants resulting from unsanitary work practices such as not washing hands.70
Specifically:
Sanding, grinding, and welding - These activities release aerosols which may
contain lead, cadmium, or chromium.71 In addition to inhalation, these metals can
adhere to workers' hands and can be ingested from food handling and hand-to-
mouth contact. Health impacts from exposures to these chemicals include asthma,
heart attacks, bronchitis, neurotoxicity, lung cancer, and premature mortality.72
Sanding dust can also be tracked home on workers' clothes and shoes.73
Paint stripping - The chemicals in common paint strippers are intended to loosen
paint from the surfaces of cars; however, if not used properly, these chemicals
can pose a serious health risk to auto body shop workers. Short-term exposure
adversely affects the nervous system and the heart, and can cause skin and eye
irritation. Long-term exposure is expected to lead to cancer.74
Painting - Painting operations can expose workers to: solvents in the paint,
metal-bearing pigments in the paint like lead and zinc chromates, and
isocyanates. Chromates and lead can cause skin, eye, and respiratory irritation,
nervous system damage, liver and kidney disease, and chromates have been
linked to cancer. Isocyanates can cause skin problems as well as allergic and
asthmatic reactions.75 The U.S. Department of Labor, Occupational Safety &
Health Administration has found evidence that exposure to auto body paint can
lead to cancer, and they have found that painters suffer from allergic and
nonallergic contact dermatitis, chronic bronchitis, asthma, and adverse central
nervous system effects.76 While isocyanate inhalation exposure is the most
70 Enander, Richard T., Hute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
71 U.S. Department of Labor, Occupational Safety Et Health Administration. In-Depth Survey Report: Control
Technology for Autobody Repair and Painting Shops at Jeff Wyler Autobody Shop, Betavia, Ohio.
http://www.osha.gov/SLTC/autobody/docs/ectb179-15a/ectb179-15a.html
72 U.S. EPA Region 10 Collision Repair Campaign - Auto Body Shops. Health Impacts from Collision Repair
Activities. http://yosemite.epa.gov/R10/AIRPAGE.NSF/Collision+Repair+Campaign/crc-auto
73 Prevention in Auto Body Shops and Paint Shops, "Sanding Waste Management", CA Department of Toxic
Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD_FS_SandingWaste.pdf
74 U.S. EPA, Chemicals in the Environment: Office of Pollution Prevention and Toxics Chemical Fact Sheets -
Methylene Chloride, August 1994. http://epa.gOV/chemfact/f dcm.txt
75 Rhode Island Department of Environmental Management - Office of Technical £t Customer Assistance, Small
Business Assistance Program. Factsheet: Safety Concerns in the Autobody Shop.
http://www.dem.ri.gov/programs/benviron/assist/pdf/safetvab.pdf
76 U.S. Department of Labor, Occupational Safety £t Health Administration. In-Depth Survey Report: Control
Technology for Autobody Repair and Painting Shops at Jeff Wyler Autobody Shop, Betavia, Ohio.
http://www.osha.gov/SLTC/autobody/docs/ectb179-15a/ectb179-15a.html
57
-------
ERP Evaluation: Final Report August 2009
common cause of health problems in workers, there has also been a link to skin
exposure causing allergic contact dermatitis and skin irritation.77
Solvent-based cleaning - Exposure to solvents include irritation, headache,
nausea, and liver, kidney and nervous system damage.78 In addition, VOCs in
solvents and paints contribute to ground-level ozone, exposure to which can
cause a range of adverse respiratory system effects, including decreased lung
function, increased susceptibility to respiratory infection, airway inflammation,
and increased lung reactivity.79
Other shop operations - Other contaminants generated in the shops include
asbestos, glass fibers, styrene, crystalline silica, and carbon monoxide, which
may pose additional health hazards.80
Overview of Relevant Behavior Changes
There are several things that auto body shops can do to protect their workers from the
potential chemical hazards present in the shop. Shops can invest in equipment and
materials that help minimize worker exposure to harmful chemicals. Specifically, shops
can use:
Spray booths, which help contain the metal-bearing fumes, paint pigments, and
organic solvents present in paint.
Enclosed spray gun cleaners, which limit workers' dermal and respiratory
exposure to solvents.
Dust control equipment, such as ventilated sanding equipment, which limits
worker exposure to the harmful metal particulates and toxins in sanding
atmospheric dust concentrations.81 Controlling sanding dust also limits the
amount of dust that is tracked home by workers.
Low VOC/HAP paints and solvents, which reduce the amount of VOCs and
HAPs that workers are exposed to during painting and cleaning.
Personal protective equipment, such as gloves, masks, respirators, and paint
suits, which reduce worker exposure during all shop operations.
Shops can also implement management standards that reduce the risk of worker exposure,
such as hazard communication programs, lockout/tagout programs (outlining specific
77 Dhimiter, Bello, et al. "Skin Exposure to Aliphatic Polyisocyanates in the Auto Body Repair and Refinishing
Industry: II. A Quantitative Assessment." American Occupational Health. Vol. 52, No. 2, January 2008, pp.
117-124.
78 U.S. EPA Region 10 Collision Repair Campaign - Auto Body Shops. Health Impacts from Collision Repair
Activities. http://yosemite.epa.gov/R107AIRPAGE.NSF/Collision+Repair+Campaign/crc-auto
79 Enander, Richard T., Hute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
80 Enander, Richard T., Hute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
81 Enander, Richard T., Hute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
58
-------
ERP Evaluation: Final Report August 2009
practices and procedures to safeguard employees from unexpected startup of machinery
or release of stored energy during operation or maintenance),82 performing regular spray
booth maintenance, and respiratory protection programs.
Summary of ERP Outcomes
While we cannot quantitatively measure the improvements in worker health and safety
that would result in shops adopting the practices and equipment described above, it is
clear that adopting these practices will reduce the risk of harmful health effects to auto
body shop workers. By limiting worker exposure to the harmful chemicals in the
materials used in shop operations, shops can also limit the likelihood that workers will
experience the negative health effects associated with shop materials and operations.
Note that many indicators related to worker health and safety are also related to air
emissions or waste management; indeed, Rhode Island was the only state to specifically
track worker health and safety indicators, therefore only six of 20 indicators reviewed
here are not also tracked in these other categories (these six indicators are in italics in
Exhibit 3-8). The full set of indicators related to worker health and safety are included
here to suggest the extent to which behavior changes at shops may influence the working
environment. In fact, the people most affected by air emissions and wastes on site at an
auto body shop are the workers. For example, changes in a behavior like using
methylene chloride-based paint stripper can directly affect worker health, even though
this indicator is also tracked as it relates to air emissions.
Exhibit 3-8 shows the specific indicators tracked, along with the observed percentage of
facilities in the sample following each practice at baseline and post-certification, and the
difference in percentages between these two measurements (i.e., the observed percentage
point change). The exhibit then shows whether each observed change in behavior is
statistically significant at a 90 percent confidence level. For each change that is
statistically significant, it presents the confidence interval for the difference (i.e., range
within which we can be 90 percent confident the true percentage point change for the
entire population of facilities will fall). Findings are listed by state, and ordered from the
greatest observed change to the smallest observed change. Those changes that are
statistically significant are shown in bold. Indicators that are unique to this category (i.e.,
are not also tracked under air emissions or waste management) are shown in italics.
82 U.S. Department of Labor, Occupational Safety Et Health Administration. "Lockout/Tagout Tutorial".
Available at http://www.osha.gov/dts/osta/lototraining/tutorial/tu-overvw.html
59
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-8 SUMMARY OF WORKER HEALTH AND SAFETY INDICATORS83
INDICATOR
Has the employer established a Personal Protective Equipment Program
(PPE)?
Has your company developed a Lockout 'Tagout Program?
Has the employer posted the Job Safety & Health Protection poster?
Does your facility use a methylene chloride-based paint stripper? (Note,
since this indicator measures a condition that the state wants shops to
discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
Has the employer established a Respiratory Protection Program?
Has a Hazard Communication (Right-To-Know) Program been established?
Do you control dust emissions from your facility using a specific device?
Hove all employees who wear respirators had a medical examination
specific for respirator use?
BASELINE %(a)
9%
6%
42%
67%
33%
28%
33%
33%
POST-
CERTIFICATION
INSPECTIONS %(b)
63%
56%
83%
95%
61%
46%
48%
46%
OBSERVED
PERCENTAGE
POINT
CHANGE'0'
54
50
41
28
28
18
15
13
SIGNIFICANT?
yes
yes
yes
yes
no
no
no
no
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
(36 - 72 )
(31-69)
(21-61 )
(12-44)
N/A
N/A
N/A
N/A
83 For Delaware and Maine, these calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and post-certification, the number of facilities
sampled at each time period for which the answer to the question was "yes," and the total number of facilities in the population). Raw data was drawn from the State Innovation Grant final reports
for these two states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdf and http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We
then used this information to calculate the observed proportion of shops at baseline and post-certification, the observed percentage point change, whether or not this change is significant, and the
confidence interval for the difference. We used the ERP Results Analyzer tool to conduct these calculations. The Results Analyzer calculates confidence intervals for the difference between
proportions observed in two different rounds of inspections. A version of the Results Analyzer can be accessed at: http://www.epa.gov/erp (however, the authors used an updated version of the
results analyzer that had not been posted on line as of the time of writing this report; for the most up-to-date version of the Results Analyzer, contact Scott Bowles, bowles.scott@epa.gov).
Calculations for the confidence intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York, NY. p.41. We use a 90 percent, two-sided
significance test and a 90 percent confidence interval in our calculations. In both Maine and Delaware, the number of facilities in both samples was relatively small (59 and 47 facilities, respectively).
Smaller sample sizes typically only show significant results when observed changes are fairly large. We believe that 90 percent significance and confidence levels are acceptable in this case. We use
the raw data, rather than observed percentages, statistical significance, or confidence intervals reported by the states ensures consistency in the analytic approach. In some cases, rounding leads to
an observed percentage point change that is slightly different than the difference between the reported baseline and post-certification inspection percentages.
For Rhode Island, we could not use the ERP Results Analyzer because the state had a stratified random sampling approach (this approach is also statistically valid, but requires more complex
analytical techniques). Therefore, for Rhode Island we simply report the values given by the state in the article summarizing their results (Enander et al., "Environmental Health Practice: Statistically
Based Performance Measurement," American Journal of Public Health.) Note that Rl reports a 95 percent confidence interval and a one-sided significance test.
60
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Do you store solvents, waste paint, sludge, and shop rags/towels saturated
with solvent in closed containers?
Is your cleaning device totally enclosed during cleaning, rinsing, and
draining operations?
Do you use coatings that comply with the emission limitations listed in
Rhode Island Air Pollution Control Regulation No. 30, Control of Volatile
Organic Compounds from Automobile Refinishing Operations?
Does your shop use a ventilated sander (dustless vacuum) system?
DELAWARE
Does the shop store absorbent paint applicators (e.g., shop rags/towels)
in closed containers?
Is all painting carried out in a spray booth to contain paint emissions and
over-spray?
Does the facility avoid any use of methylene chloride-based paint strippers?
MAINE
Does the shop utilize low VOC/HAP solvents?
Does the shop utilize low VOC/HAP paints and coatings? (lower than the
federal standard)
Does the shop utilize a dust control system to control dust generated
from the sanding process?
Does painting and coating take place in areas outside of a spray booth?
(Note, since this indicator measures a condition that the state wants shops
to discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
Does the shop utilize an enclosed spray gun cleaner, solvent recycler, or
other spray gun cleaning methods to reduce or eliminate VOC emissions?
Does the shop carry out painting and coating in a spray booth to contain
paint emissions and over spray?
Are solvents, thinners, or other VOC and HAP containing materials stored in
closed containers when not in use?
BASELINE %(a)
81%
83%
100%
31%
53%
81%
_
49%
59%
34%
75%
54%
80%
93%
POST-
CERTIFICATION
INSPECTIONS %(b)
88%
88%
100%
30%
96%
87%
'T^-
97%
93%
47%
85%
61%
86%
90%
OBSERVED
PERCENTAGE
POINT
CHANGED
7
5%
0
-1
43
6
_
47
34
14
10
7
7
-3%
SIGNIFICANT?
no
no
not tested
not tested
yes
no
no
yes
yes
yes
yes
no
no
no
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
N/A
N/A
N/A
N/A
(32 - 53 )
N/A
~~~^^
(40 - 55 )
(26 - 42 )
(4 - 23 )
(2-18)
N/A
N/A
N/A
61
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
BASELINE %(a)
POST-
CERTIFICATION
INSPECTIONS %(b)
OBSERVED
PERCENTAGE
POINT
CHANGED
SIGNIFICANT?
CONFIDENCE INTERVAL FOR
THE DIFFERENCE IN
PERCENTAGE POINTS
Notes:
(a) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the baseline sample for which the answer to the indicator question is
"Yes," which indicates positive environmental performance.
(b) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the post-certification sample for which the answer to the indicator
question is "Yes," which indicates positive environmental performance.
(c) All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of shops in the sample following the behavior at
baseline prior to ERP from the observed percentage of shops in the sample following the behavior at post-certification. We use percentage point changes, rather than percent
changes, to more clearly show the magnitude of change. For example, suppose the observed proportion of shops following a behavior increased from 6 percent at baseline to 12
percent at post certification: this change could be expressed as a 100 percent improvement, or a 6 percentage point improvement. We believe the latter approach is a clearer,
more accurate description of changes in performance.
62
-------
ERP Evaluation: Final Report August 2009
This category of indicators improved substantially overall in the sampled facilities, with
the majority of the improvements (and indicators) occurring in Rhode Island and Maine.
Of the 22 total indicators in the category, 18 (82 percent) improved; of the six unique
indicators (i.e. those that are not also tracked in other categories), all improved. Nine of
the total improvements and three of the unique improvements were statistically
significant, meaning that we can infer with 90 percent confidence that the entire universe
of shops eligible for the ERPs showed improvements for these indicators. Among the
statistically significant improvements, the average observed change was 36 percentage
points and 48 percentage points for the total indicators and unique indicators respectively.
The performance for two indicators declined; however, neither of these declines was
statistically significant.
In three areas, multiple states observed improvements to related indicators. Rhode Island
and Maine observed improvements related to dust control (statistically significant for
Maine and not for Rhode Island); Rhode Island and Delaware observed improvements
related to proper storage of solvents, waste paint, sludge, rags/towels, and paint
applicators; Delaware and Maine observed improvements related to painting and coating
in spray booths. In addition, while Rhode Island observed a 28 percentage point
statistically significant improvement in avoiding the use of methylene chloride paint
stripper, Delaware observed a nine percentage point increase in paint stripper usage (not
statistically significant).
Rhode Island measured changes in shops' management practices and equipment, and
observed improvements in both areas. All four of the state's indicators relating to
equipment and material changes improved. Of the six indicators relating to management
practices that improved, three (50 percent) showed significant improvements. The
progress that shops in Rhode Island made in limiting worker exposure in shops may be
due, in part, to the fact that Rhode Island DEM partnered with the Rhode Island
Department of Health in designing and implementing the program. From OEM's
perspective, the fact that the Department of Health performed inspections, led workshops,
and was an active participant in the program made a substantial contribution to improved
facility performance.
Maine and Delaware did not measure changes in management practices; however, they
did measure changes in the percentages of shops using equipment that reduces worker
exposure. Delaware observed a 43 percentage point improvement in the percent of shops
that store absorbent paint applicators in closed containers, a change that was statistically
significant. Maine observed a three percentage point decline in the percent of shops that
properly store solvents, thinners, or other VOC containing materials, although this change
was not statistically significant. Maine observed improvement in the remaining six
indicators regarding safer equipment, and four of these were statistically significant.
Aside from changes in performance, the ERP data demonstrates the overall level of
performance at post-certification. The indicators where a state observed at least 95
percent of the shops sampled following preferred behavior include:
Use of coatings that comply with specific regulatory emission limitations;
Use of low VOC/HAP solvents;
63
-------
ERP Evaluation: Final Report August 2009
Proper storage of absorbent paint applicators; and
Avoiding use of methylene chloride-based paint strippers.
Conversely, the behaviors where at least one state observed relatively poor performance
(less than 40 percent of shops following preferred behavior) include:
Use of a ventilated sander (dustless vacuum) system
Avoiding use of methylene chloride-based paint strippers
Notably, one of these indicators (avoiding use of methylene chloride-based paint
strippers), is an area of high performance in one state and poor performance in another.
As in the air emissions and waste management categories, these variations in
performance may be due to external factors, e.g., the history of each state in terms of
prior outreach to auto body shops, rather than differences in the influence of the ERP
itself in affecting shop behavior.
POLLUTION PREVENTION
Summary of the Problem
This category generally encompasses all of the ways in which auto body shops can reduce
the amount of pollution they produce at their shops. Pollution prevention here refers to
reducing or eliminating waste at the source of its production by modifying production
processes, promoting the use of non-toxic or less-toxic substances, implementing
conservation techniques, and re-using materials rather than putting them into a waste
stream.84 Specifically, pollution prevention includes raw material substitution, improved
operating practices, process and equipment modifications, and energy and water
conservation. Note that the indicators tracked in this section are also presented in the
other media categories (e.g., air emissions, water discharges, etc.), however we repeat
them here to highlight the extent to which ERPs have measured changes in behavior with
prevent pollution at the source, which results in less extraction and use of resources, as
well as less waste.
Overview of Relevant Behavior Changes
In auto body shops, pollution prevention practices generally involve product changes,
good operating practices, or technology changes.85
Product Changes - Material substitution is a way for shops to replace a more harmful
product for a safer one. This ensures that the waste generated by those materials is
potentially less harmful and that the negative effects of the products are lessened. For
example:
o Replace solvent-based cleaners with water-based or low VOC materials;
o Use high-solids, low VOC coatings;
84 U.S. EPA's Pollution Prevention Website - General Information, http://www.epa.gov/p2/
85 Rhode Island Department of Environmental Management - Office of Technical Et Customer Assistance, Small
Business Assistance Program. Factsheet: Pollution Prevention in the Autobody Shop.
http://www.dem.ri.gov/programs/benviron/assist/pdf/general.pdf
64
-------
ERP Evaluation: Final Report August 2009
o Eliminate the use of solvents to clean hands and other skin surfaces;
o Use cadmium-free solder and resistant spot welding and/or metal adhesives
in place of conventional welding practices;
o Use abrasive blasting techniques in place of methylene-chloride strippers;
o Use disposable masking as an alternative to solvent cleaning; and
o Use paintless dent removal as a replacement for conventional
refmishing.86'87
Good Operating Practices - Good operating practices focus on containing or
controlling the source(s) that produce the pollution. These practices often create cost-
saving opportunities for shops as well. For example:
o Keep solvent containers closed when not in use;
o Protect raw materials from damage, contamination or exposure to the
elements;
o Supervising and controlling the dispensing of raw materials;
o Recycle solvents;88 and
o Efficiently mix paint to reduce the amount of paint needed for each j ob and
the labor, materials, and disposal costs required.89
Technology Changes - While technology changes can require significant investment
from auto body shops, they can substantially reduce the amount of pollution
generated in the shop. For example:
o High Volume Low Pressure (HVLP) spray guns result in a 30% or more
reduction in coating usage, a reduction in paint overspray resulting in
cleanup cost savings and decreased frequency of spray booth filter changes,
and improved ability to apply thick or high-solid coatings as compared to
conventional high-pressure siphon guns.
o Enclosed spray gun cleaners reduce solvent losses to the environment by
containing solvent vapors. They also recirculate solvent so that less fresh
solvent is needed.90
86 Rhode Island Department of Environmental Management - Office of Technical Et Customer Assistance, Small
Business Assistance Program. Factsheet: Pollution Prevention in the Autobody Shop.
http://www.dem.ri.gov/programs/benviron/assist/pdf/general.pdf
87 Enander, Richard T., Gute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial Hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
88 Rhode Island Department of Environmental Management - Office of Technical £t Customer Assistance, Small
Business Assistance Program. Factsheet: Pollution Prevention in the Autobody Shop.
http://www.dem.ri.gov/programs/benviron/assist/pdf/general.pdf
89 Pollution Prevention in Auto Body Shops and Paint Shops, "Minimizing Paint Waste", CA Department of
Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD FS PaintWasteMin.pdf
90 Enander, Richard T., Gute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial Hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
65
-------
ERP Evaluation: Final Report August 2009
o Spray booths reduce the amount of overspray, resulting in less paint
needed, which also cuts down on VOC emissions and the amount of
solvent needed for cleanup.91
Summary of ERP Outcomes
The primary expected long-term outcome of pollution prevention activities in auto body
shops is the reduction in material usage of paints and solvents. If shops use alternative
materials, practice good housekeeping techniques, and make the shift to better
technology, they should use fewer materials and produce less waste.
Exhibit 3-9 shows the specific indicators tracked, along with the observed percentage of
facilities in the sample following each practice at baseline and post-certification, and the
difference in percentages between these two measurements (i.e., the observed percentage
point change). The exhibit then shows whether each observed change in behavior is
statistically significant at a 90 percent confidence level. For each change that is
statistically significant, it presents the confidence interval for the difference (i.e., range
within which we can be 90 percent confident the true percentage point change for the
entire population of facilities will fall). Findings are listed by state, and ordered from the
greatest observed change to the smallest observed change. Those changes that are
statistically significant are shown in bold.
States observed improvements in the sampled facilities in the majority of indicators
tracked for this category. Of the 11 pollution prevention indicators, 8 (73 percent)
improved, and all three of the indicators that did not improve had a baseline performance
level of 100 percent, so had no room for improvement. Four of the improvements were
statistically significant, meaning that we can infer with 90 percent confidence that the
total universe of shops eligible for the ERPs showed improvements for these indicators.
Among the statistically significant improvements, the average observed change was 19
percentage points. Of the three indicators that had a baseline performance level of 100
percent, two declined slightly, but neither of these declines was statistically significant.
91 Pollution Prevention in Auto Body Shops and Paint Shops, "Minimizing Paint Waste", CA Department of
Toxic Substances Control, September 2006. Available at:
http://www.dtsc.ca.gov/PollutionPrevention/ABP/upload/TD_FS_PaintWasteMin.pdf
66
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-9 SUMMARY OF POLLUTION PREVENTION INDICATORS92
INDICATOR
DELAWARE
Does the shop employ a training program in the proper use and handling
of coatings, solvents and waste products to minimize air emissions?
Does the shop clean the spray guns using only methods that comply with
Delaware Regulations?
Do the painters and technicians use only painting techniques that comply
with Delaware Regulations?
Is all painting carried out in a spray booth to contain paint emissions and
over-spray?
MAINE
Does painting and coating take place in areas outside of a spray booth?
(Note, since this indicator measures a condition that the state wants shops
to discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
Does the shop carry out painting and coating in a spray booth to contain
paint emissions and over spray?
BASELINE %(a)
47%
89%
89%
81%
75%
80%
POST-
CERTIFICATION
INSPECTIONS %(b)
91%
100%
98%
87%
85%
86%
OBSERVED
PERCENTAGE
POINT
CHANGE'0'
45
11
9
6
10
7
SIGNIFICANT?
yes
yes
yes
no
yes
no
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
(33 - 56 )
(5-17)
(2-15)
N/A
(2-18)
N/A
92 For Delaware and Maine, these calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and post-certification, the number of facilities
sampled at each time period for which the answer to the question was "yes," and the total number of facilities in the population). Raw data was drawn from the State Innovation Grant final reports
for these two states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdf and http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We
then used this information to calculate the observed proportion of shops at baseline and post-certification, the observed percentage point change, whether or not this change is significant, and the
confidence interval for the difference. We used the ERP Results Analyzer tool to conduct these calculations. The Results Analyzer calculates confidence intervals for the difference between
proportions observed in two different rounds of inspections. A version of the Results Analyzer can be accessed at: http://www.epa.gov/erp (however, the authors used an updated version of the
results analyzer that had not been posted on line as of the time of writing this report; for the most up-to-date version of the Results Analyzer, contact Scott Bowles, bowles.scott@epa.gov).
Calculations for the confidence intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York, NY. p.41 We use a 90 percent, two-sided
significance test and a 90 percent confidence interval in our calculations. In both Maine and Delaware, the number of facilities in both samples was relatively small (59 and 47 facilities, respectively).
Smaller sample sizes typically only show significant results when observed changes are fairly large. We believe that 90 percent significance and confidence levels are acceptable in this case. We use
the raw data, rather than observed percentages, statistical significance, or confidence intervals reported by the states to ensure consistency in the analytic approach. In some cases, rounding leads
to an observed percentage point change that is slightly different than the difference between the reported baseline and post-certification inspection percentages.
For Rhode Island, we could not use the ERP Results Analyzer because the state had a stratified random sampling approach (this approach is also statistically valid, but requires more complex
analytical techniques). Therefore, for Rhode Island we simply report the values given by the state in the article summarizing their results (Enander et al., "Environmental Health Practice: Statistically
Based Performance Measurement," American Journal of Public Health.) Note that Rl reports a 95 percent confidence interval and a one-sided significance test.
67
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Does the shop utilize an enclosed spray gun cleaner, solvent recycler, or
other spray gun cleaning methods to reduce or eliminate VOC emissions?
Does the shop train all employees in the proper use and handling of paints
and coatings according to the manufacturers' recommendations to minimize
air pollution?
Does the shop employ a training program in the proper use and handling of
solvents and waste products to minimize air emissions?
BASELINE %(a)
54%
100%
100%
POST-
CERTIFICATION
INSPECTIONS %(b)
61%
97%
97%
OBSERVED
PERCENTAGE
POINT
CHANGED
7
-3
-3
SIGNIFICANT?
no
no
no
CONFIDENCE INTERVAL
FOR THE DIFFERENCE IN
PERCENTAGE POINTS
N/A
N/A
N/A
Is your cleaning device totally enclosed during cleaning, rinsing, and
draining operations?
Do your painters and technicians use spray guns that have a transfer
efficiency of at least 65% such as High Volume Low Pressure (HVLP) spray
equipment?
83%
100%
88%
100%
5
0
no
not tested
N/A
N/A
Notes:
(a) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the baseline sample for which the answer to the indicator question is
"Yes," which indicates positive environmental performance.
(b) Unless otherwise indicated in the table, this column represents the observed proportion of shops in the post-certification sample for which the answer to the indicator
question is "Yes," which indicates positive environmental performance.
(c) All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of shops in the sample following the behavior at
baseline prior to ERP from the observed percentage of shops in the sample following the behavior at post-certification. We use percentage point changes, rather than percent
changes, to more clearly show the magnitude of change. For example, suppose the observed proportion of shops following a behavior increased from 6 percent at baseline to 12
percent at post certification: this change could be expressed as a 100 percent improvement, or a 6 percentage point improvement. We believe the latter approach is a clearer,
more accurate description of changes in performance.
68
-------
ERP Evaluation: Final Report August 2009
Most of the related behavior changes in this category relate to technological changes.
Specifically,
HVLP guns - Delaware saw a statistically significant 9 percentage point
improvement in the proportion of shops that use high transfer efficiency spray
guns, such as HVLP guns. Rhode Island's shops all used high transfer efficiency
spray guns at both the baseline and post-certification inspections.
Enclosed spray gun cleaners - All three states saw an improvement in the
percentage of shops that use enclosed spray gun cleaners. Delaware observed a
statistically significant 11 percentage point increase, while Maine and Rhode
Island only had 7 and 5 percentage point improvements, respectively, neither of
which was statistically significant.
Spray booths - Maine observed a 10 percentage point improvement in the
proportion of shops that carry out all painting in spray booths (this was
statistically significant), while Delaware observed a 6 percentage point
improvement that was not statistically significant.
Using the DfE tool, we are able to consider observed behavior changes that are
statistically significant and model them in the DfE tool to extrapolate materials reductions
from facilities in the sample to facilities across the state. There is one indicator for which
we can use the DfE tool to estimate the reductions in materials use: switching to
automatic spray gun cleaning in Delaware. We are 90 percent confident that the
percentage point increase in the percent of shops cleaning spray guns only with methods
that comply with Delaware regulations is between 5 and 17 percentage points. We expect
that material usage associated with auto body shop spray gun cleaning operations
declined by between l,2241bs/year and 3,978 Ibs/year for all shops in Delaware. Please
see Appendix A for a full description of the calculations used for this estimate.
In addition, Delaware observed a statistically significant 45 percentage point
improvement related to employee training in the proper use and handling of coatings,
solvents, and waste products. Maine observed 3 percentage point declines for two
indicators related to employee training, although both declines were not significant.
Aside from changes in performance, the ERP data shows the overall level of performance
at post-certification. The indicators where a state observed at least 95 percent of the shops
sampled following preferred behavior include:
Use of HVLP spray guns
Use of spray gun cleaning methods that comply with state regulations
Use of painting techniques that comply with state regulations
Employee training related to proper use and handling of paints and coatings
Employee training related to proper use and handling of solvents and waste
products
No behaviors existed for pollution prevention where states observed relatively poor
performance (less than 40 percent of shops following preferred behavior).
69
-------
ERP Evaluation: Final Report August 2009
DISCUSSION OF RESULTS FROM A SECOND CYCLE OF ERP
All of the results discussed to this point reflect a single cycle of ERP (i.e., changes in
behavior between baseline and post-Round 1 certification inspections, and associated
environmental/health benefits). For two of the ERPs addressed in this evaluation
(Delaware and Maine), no further data on the results of ERP are available. However, in
Rhode Island, the state has continued to implement ERP for a second cycle, and has
provided results from post-Round 2 certification inspections. These results provide
potential insights into the effects of ERP overtime, if the program is continued. Note that
the second cycle of ERP in Rhode Island involved much more limited outreach than the
first cycle, as discussed in the next section of this chapter (ERP Effectiveness and Costs).
Therefore, the second-cycle results of Rhode Island's ERP may be best understood as
representing the impacts of an ERP in "maintenance" mode, where the state is not
continuing to provide extensive outreach, but is rather using additional rounds of facility
certification to remind facilities of their obligations and encourage incremental
improvements.
Exhibit 3-10 shows the observed proportion of facilities in the sample meeting each
indicator at baseline, post-Round 1 certification inspections, and post-Round 2
certification inspections, as well as the incremental percentage point changes across each
cycle. Changes that are statistically significant between baseline and post-Round 1
certification are shown in bold. Indicators are organized within each category by
descending order of observed percentage point change (Baseline to Post-Round 1).
Exhibit 3-11 shows the cumulative changes in performance between baseline and post-
Round 2 certification inspections. This type of cumulative performance is consistent with
how ERP states are expected to report their results over time.93 Changes that are
statistically significant between baseline and post-Round 2 certification are shown in
bold. Indicators are organized within each category by descending order of cumulative
observed percentage point change (Baseline to Post Round 2).
93 These expectations are formalized in a document prepared by the States ERP Consortium, with support
from EPA, entitled The States ERP Consortium Guide to Reporting ERP Results. For more information on the
States ERP Consortium, see www.erpstates.org. Last accessed August 3, 2009.
70
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-10 INCREMENTAL RESULTS FROM FIRST AND SECOND CYCLES OF RHODE ISLAND ERP
INDICATOR
WASTE
Do you have appropriate documentation which shows where
hazardous waste is being shipped?
Is the area (satellite accumulation area) clearly marked and the
containers labeled with: (1 ) the words "Hazardous Waste", (2)
Name of the waste and its waste code?, (3) the hazard
classification?, and (4) the date that they were placed in the
storage area?
Is the area (hazardous waste storage area) inspected weekly for
signs of spills or container deterioration?
Does your shop have a written contingency plan designed to help
your shop reduce hazards associated with the possibility of an
explosion, fire, or unplanned/accidental release of hazardous
materials?
Does your shop have an employee training program that teaches
them proper hazardous waste management procedures,
including how to implement the contingency plan?
Has your shop submitted to DEM a list of authorized agents that
are allowed to sign the manifest (hazardous waste manifest)?
What is your facility's hazardous waste identification number?
Does your hazardous waste storage area meet the criteria for
secondary containment (i.e. spill/leak containment capability)?
BASELINE %(a)
56%
21%
6%
6%
6%
28%
88%
63%
POST- ROUND 1
CERTIFICATION
INSPECTIONS %(b)
89%
39%
22%
22%
22%
44%
86%
56%
OBSERVED
PERCENTAGE
POINT CHANGE
(Baseline to
Post-Round 1)
33
18
16
16
16
16
-2
-7
POST-ROUND 2
CERTIFICATION
INSPECTIONS %(b)
92%
87%
DNC
DNC
DNC
DNC
100%
DNC
INCREMENTAL
OBSERVED
PERCENTAGE
POINT CHANGE
(Post-Round 1
to Post Round
2)
3
48
DNC
DNC
DNC
DNC
14
DNC
71
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
AIR
Does your facility use a methylene chloride-based paint
stripper? (Note, since this indicator measures a condition that
the state wants shops to discontinue, the percentages shown
here reflect the number of shops not performing this practice.)
Do you control dust emissions from your facility using a specific
device?
Do you store solvents, waste paint, sludge, and shop rags/towels
saturated with solvent in closed containers?
Is your cleaning device totally enclosed during cleaning, rinsing,
and draining operations?
Do you use coatings that comply with the emission limitations
listed in Rhode Island Air Pollution Control Regulation No. 30,
Control of Volatile Organic Compounds from Automobile
Refinishing Operations?
Do your painters and technicians use spray guns that have a
transfer efficiency of at least 65% such as High Volume Low
Pressure (HVLP) spray equipment?
Does your shop use a ventilated sander (dustless vacuum)
system?
WATER
Does your shop post signs prohibiting the discharge of
industrial chemicals and/or wastewater to bathroom/kitchen
sinks, toilets, showers, shop wash basins, emergency
showers, eyewash stations, or other non-industrial drainage
outlets?
BASELINE %(a)
67%
33%
81%
83%
100%
100%
31%
0%
POST- ROUND 1
CERTIFICATION
INSPECTIONS %(b)
95%
48%
88%
88%
100%
100%
30%
48%
OBSERVED
PERCENTAGE
POINT CHANGE
(Baseline to
Post-Round 1)
28
15
7
5
0
0
-1
48
POST-ROUND 2
CERTIFICATION
INSPECTIONS %(b)
89%
84%
92%
89%
100%
100%
53%
55%
INCREMENTAL
OBSERVED
PERCENTAGE
POINT CHANGE
(Post-Round 1
to Post Round
2)
-6
36
4
1
0
0
23
7
72
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Does you shop allow process wastewater (i.e., from wet
sanding, car washing, work area washing) to run off your site
to storm drains or other areas (i.e., water runs down the
street, water runs off to soil or sand area, water just puddles
up and evaporates)?
(Note, since this indicator measures a condition that the state
wants shops to discontinue, the percentages shown here reflect
the number of shops not performing this practice.)
Does your shop contain open floor drains that are not currently
in use?
(Note, since this indicator measures a condition that the state
wants shops to discontinue, the percentages shown here reflect
the number of shops not performing this practice.)
SAFETY
Has the employer established a Personal Protective
Equipment Program (PPE)?
Has your company developed a Lockout/! agout Program?
Has the employer posted the Job Safety 6t Health Protection
poster?
Has the employer established a Respiratory Protection Program?
Has a Hazard Communication (Right-To-Know) Program been
established?
Have all employees who wear respirators had a medical
examination specific for respirator use?
BASELINE %(a)
37%
67%
9%
6%
42%
33%
28%
33%
POST- ROUND 1
CERTIFICATION
INSPECTIONS %(b)
74%
69%
63%
56%
83%
61%
46%
46%
OBSERVED
PERCENTAGE
POINT CHANGE
(Baseline to
Post-Round 1)
37
2
54
50
41
28
18
13
POST-ROUND 2
CERTIFICATION
INSPECTIONS %(b)
84%
87%
77%
56%
83%
71%
77%
44%
INCREMENTAL
OBSERVED
PERCENTAGE
POINT CHANGE
(Post-Round 1
to Post Round
2)
10
18
14
o
0
10
31
-2
73
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
BASELINE %(a)
POST- ROUND 1
CERTIFICATION
INSPECTIONS %(b)
OBSERVED
PERCENTAGE
POINT CHANGE
(Baseline to
Post-Round 1)
POST-ROUND 2
CERTIFICATION
INSPECTIONS %(b)
INCREMENTAL
OBSERVED
PERCENTAGE
POINT CHANGE
(Post-Round 1
to Post Round
2)
Notes:
Baseline Total Sample Size = 40
Post-Round 1 Sample Size = 44. Rhode Island randomly inspected 42 facilities in at this time, but included 2 additional facilities in their calculations based on
corrective action statements acknowledging regulatory deficiencies that were assumed to be accurate indicators of nonperformance for statistical analysis purposes.
Post-Round 2 = 38
(a) Proportion of sample facilities following indicators at baseline.
(b) Proportion of sample facilities following indicators at post-certification.
"DNC" (Did Not Collect) means post-certification data was not collected for the indicator at the Post-Round 2 sample.
Changes that are statistically significant between baseline and post-Round 1 certification are shown in bold across the row.
74
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT 3-11 CUMULATIVE RESULTS FROM FIRST AND SECOND CYCLES OF RHODE ISLAND ERP
INDICATOR
WASTE
Is the area (satellite accumulation area) clearly marked and the
containers labeled with: (1) the words "Hazardous Waste", (2) Name of
the waste and its waste code?, (3) the hazard classification?, and (4) the
date that they were placed in the storage area?
Do you have appropriate documentation which shows where hazardous
waste is being shipped?
Has your shop submitted to DEM a list of authorized agents that are allowed
to sign the manifest (hazardous waste manifest)?
Is the area (hazardous waste storage area) inspected weekly for signs of
spills or container deterioration?
Does your shop have a written contingency plan designed to help your shop
reduce hazards associated with the possibility of an explosion, fire, or
unplanned/accidental release of hazardous materials?
Does your shop have an employee training program that teaches them proper
hazardous waste management procedures, including how to implement the
contingency plan?
Does your hazardous waste storage area meet the criteria for secondary
containment (i.e. spill/leak containment capability)?
What is your facility's hazardous waste identification number?
BASELINE %(a)
21%
56%
28%
6%
6%
6%
63%
88%
POST- ROUND 2
CERTIFICATION
INSPECTIONS %(b)
87%
92%
DNC
DNC
DNC
DNC
DNC
100%
CUMULATIVE
OBSERVED
PERCENTAGE
POINT
CHANGE
(Baseline to
Post Round 2)
66
36
DNC
DNC
DNC
DNC
DNC
12
SIGNIFICANT?
yes
yes
no
no
no
no
no
no
75
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Ail?
Do you control dust emissions from your facility using a specific device?
Does your shop use a ventilated sander (dustless vacuum) system?
Does your facility use a methylene chloride-based paint stripper? (Note,
since this indicator measures a condition that the state wants shops to
discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
Do you store solvents, waste paint, sludge, and shop rags/towels saturated
with solvent in closed containers?
Is your cleaning device totally enclosed during cleaning, rinsing, and draining
operations?
Do you use coatings that comply with the emission limitations listed in Rhode
Island Air Pollution Control Regulation No. 30, Control of Volatile Organic
Compounds from Automobile Refinishing Operations?
Do your painters and technicians use spray guns that have a transfer
efficiency of at least 65% such as High Volume Low Pressure (HVLP) spray
equipment?
WATER
Does your shop post signs prohibiting the discharge of industrial
chemicals and/or wastewater to bathroom/kitchen sinks, toilets, showers,
shop wash basins, emergency showers, eyewash stations, or other non-
industrial drainage outlets?
BASELINE %(a)
33%
31%
67%
81%
83%
100%
100%
0%
POST- ROUND 2
CERTIFICATION
INSPECTIONS %(b)
84%
53%
89%
92%
89%
100%
100%
55%
CUMULATIVE
OBSERVED
PERCENTAGE
POINT
CHANGE
(Baseline to
Post Round 2)
51
22
22
11
6
0
0
55
SIGNIFICANT?
yes
no
no
no
no
not tested
not tested
yes
76
-------
ERP Evaluation: Final Report
August 2009
INDICATOR
Does your shop allow process wastewater (i.e., from wet sanding, car
washing, work area washing) to run off your site to storm drains or other
areas (i.e., water runs down the street, water runs off to soil or sand
area, water just puddles up and evaporates)?
(Note, since this indicator measures a condition that the state wants shops
to discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
Does your shop contain open floor drains that are not currently in use?
(Note, since this indicator measures a condition that the state wants shops
to discontinue, the percentages shown here reflect the number of shops not
performing this practice.)
BASELINE %(a)
37%
67%
POST- ROUND 2
CERTIFICATION
INSPECTIONS %(b)
84%
87%
CUMULATIVE
OBSERVED
PERCENTAGE
POINT
CHANGE
(Baseline to
Post Round 2)
47
20
SIGNIFICANT?
yes
no
AND
Has the employer established a Personal Protective Equipment Program?
Has your company developed a Lockout/Tagout Program?
Has a Hazard Communication Program been established?
Has the employer posted the Job Safety 6t Health Protection poster?
Has the employer established a Respiratory Protection Program?
Have all employees who wear respirators had a medical examination specific
for respirator use?
9%
6%
28%
42%
33%
33%
77%
156%
77%
83%
71%
44%
68
50
49
41
38
11
yes
yes
yes
yes
yes
no
Notes:
(a) Proportion of sample facilities following indicators at baseline
(b) Proportion of sample facilities following indicators at post-certification.
"DNC" (Did Not Collect) means post-certification data was not collected for the indicator in Round Two.
Changes that are statistically significant between baseline and post-Round 2 certification are shown in bold across the row.
77
-------
ERP Evaluation: Final Report August 2009
As shown in Exhibit 3-10, between the baseline and post-Round 1 certification
inspections, Rhode Island observed seven statistically significant improvements out of the
24 indicators measured (29 percent): one related to waste, one related to air emissions,
two related to water discharges, and three related to worker health and safety. Of these,
the greatest percentage point improvements were related to worker health and safety,
where for two indicators (establishing a Personal Protective Equipment Program and a
Lockout/Tagout program), the observed proportion of shops with these programs in place
increased by at least 50 percentage points.
In the second cycle of ERP (comparing post-Round 1 certification inspections to post-
Round 2 certification inspections), one large change was observed (the proportion of
facilities following container labeling guidelines increased by 48 percentage points), and
several smaller but still substantial changes were also observed (for example, the
proportion of facilities controlling dust with a specific device increased by 36 percentage
points, and the proportion of facilities with a Hazard Communication Program increased
by 31 percentage points). Note that in two instances, observed performance declined
after the first cycle. For example, in the air emissions category, the observed proportion
of shops not using methylene chloride paint stripper increased from 67 to 95 percent over
the first cycle of ERP, but then declined to 89 percent after the second cycle of ERP.
Overall, including all results (those that were and were not statistically significant), the
initial average observed percentage point change between the baseline and post-Round 1
certification inspections was 19 percentage points. In comparison, the incremental
average observed percentage point change between post-Round 1 certification inspections
and post-Round 2 certification inspections was 11 percentage points. This would suggest
that in Rhode Island, the state observed substantial gains in each cycle of ERP, but that
rate of improvements was smaller in the second cycle of ERP compared to the first.
Exhibit 3-11 shows that when the cumulative effect of the two cycles of ERP is measured
(by comparing baseline to post-Round 2 certification inspections), the state found 10 of
24 indicators (42 percent) with statistically significant improvements. Five of these were
in the worker health and safety category, two each in the waste and water categories, and
one in the air category. These include all of the same indicators that showed statistically
significant improvements after the first cycle of ERP except one air indicator (methylene
chloride usage); plus four additional indicators that are only statistically significant when
comparing across the two cycles of ERP.
Note that Rhode Island did not collect data for all indicators across both cycles of ERP;
five indicators in the waste category were measured at baseline and after the first cycle of
ERP, but not after the second cycle.
78
-------
ERP Evaluation: Final Report August 2009
III. ERP EFFECTIVENESS AND COSTS
As described in the previous section, each of the three states reviewed achieved
quantifiable, statistically significant improvements for some indicators related to
intermediate outcomes/behavior changes, and in some cases we can assess the types of
likely long-term outcomes associated with these improvements. This section takes a
broader view of program effectiveness, as defined by the states themselves and program
participants. Key aspects of program effectiveness typically addressed by states include
participation in the voluntary self-certification component of ERP, increased awareness
of compliance obligations among auto body shops, observed changes in behavior at the
shops, improved relationships, and sharing of information between regulators and the
auto body sectors.
While each state views its ERP as successful, these results must be balanced with the
program's costs. This section provides a description of resources each state invested in
developing and implementing the ERP, as well as an indication of the extent of resources
that auto body shops spent participating in the ERP.
It is important to note that each state interviewed weighed the costs and benefits of
conducting an ERP differently, since each state had its own alternative scenario about
what it would have done in the absence of ERP.
DELAWARE
Program Effectiveness
DNREC was able to achieve a 68 percent participation rate in its ERP (i.e., 103 out of
152 eligible auto body shops volunteered to participate in self-certification).94 The
program helped shops gain a better understanding of their regulations, which in turn
increased compliance. According to Delaware, the ERP generated more results because
shops now took the time to make sure that they understood what they needed to do to be
in compliance. Before ERP, shops got their permit applications in the mail and they may
not have even read them because they were not familiar with the regulations.
As a result of increased awareness, one of the shops interviewed as part of this evaluation
started a recycling program for fluorescent tubes, addressed air permits for spray booths,
plugged floor drains, and labeled containers for hazardous rags. It is also currently
attempting to transition to waterborne paints. These sorts of improvements can lead to
positive environmental outcomes, especially when they occur throughout a sector.
The self-certification program improved the relationship between regulators and shops.
The shops interviewed for this evaluation were very complimentary of how the DNREC
sought to help them come into compliance and make improvements, as opposed to
penalizing them for not being in compliance; the shops saw the regulators as being on
their side. Additionally, the shops gained contacts they could go to with questions during
and after the ERP. From the regulators' perspective, DNREC achieved improved
94 Throughout this section, "participation rate" refers to the percentage of facilities that volunteered to
participate in the self-certification component of the ERP. It is calculated by taking the number of facilities
that completed self-certification forms divided by the total number of eligible auto body shops in the state.
It does not include shops that, for example, attended a workshop but did not complete a self-certification
form.
79
-------
ERP Evaluation: Final Report August 2009
relationships and a better understanding of the universe of auto body shops through ERP.
Less directly, the permitting staff who wrote the source category permit became more
sensitive to the needs of small businesses as a result of collaborating with ERP, and the
Department's air office - which did not participate in ERP - gained improved
relationships with auto body shops as well.
There were some suggestions for improvement. Specifically, one of the shops
interviewed for this evaluation felt the record-keeping aspect of ERP was time-
consuming and inefficient. Furthermore, this respondent felt ERP could be improved by
offering more workshops and providing even more information, in order to share best
practices and cost saving ideas and keep shops abreast of changes and developments in
the sector.
Program Costs
The EPA State Innovation Grant for Delaware's ERP totaled $116,500, with DNREC
contributing an additional $10,376 in state funding.95'96 These costs include $95,000 in
consultant costs, which covered a range of tasks related to ERP development (e.g.,
identification of an initial facility universe; and developing the workbook, self-
certification forms, return to compliance forms, inspector checklists, and a database). On
top of consultant costs and internal staff hours, there was an additional $10,000 spent on
things like supplies, printing, postage, workshop materials, and workshop locations.
In addition to the direct costs, DNREC dedicated staff resources. A total of 23 staff
members from various offices participated in the program at different times, investing
approximately 330 days overall over 2.5 years, which is approximately equivalent to one
half of an FTE working over this period.97'98 A total of 75 staff days were spent on the
Design and Development phase; 220 staff days were spent on the Initial Implementation
phase; 32 staff days were spent on Program Follow-Up/Ongoing Support (see Appendix
D for a detailed breakdown of DNREC's investment of staff time). The $10,376
contributed by the state covered a portion of these staff costs, not the entire amount.99
Delaware was surprised by the amount of time required to support auto body shops
participating in ERP once the materials had been sent out. Despite the state's best efforts
to provide everything the shops needed to understand how to improve their practices,
DNREC's outreach still prompted many phone calls, requests for in-person assistance,
and interest in pollution prevention assistance. When returning their self-certification
forms, if shops did not know an answer to one of the questions, they would often leave it
blank, which required follow up from DNREC.
Overall, Delaware found that implementing the self-certification program in tandem with
the source category permit required significantly more resources than the permit would
95 "Final Report: EPA 2002 State Innovation Pilot Grant Program", Delaware Department of Natural Resources
and Environmental Control.
96 Note that the remaining information in this section is drawn from our interview with Delaware state
program staff and additional information sent by the state.
97 This length of time of the project is based on the duration of the State Innovation Grant, which provided
significant funding for the project.
98 The FTE calculation is based on 52 weeks per year, five days per week, which is equal to 650 days. The
330 days divided by 650 days is 51 percent.
99 The state was unable to give an estimated dollar value these additional staff costs. Therefore, our cost
estimates are underestimating the total costs of the program to the state.
80
-------
ERP Evaluation: Final Report August 2009
have required on its own. However, the permitting process was more efficient and
effective because it was implemented in conjunction with ERP.
Auto body shops also had to invest staff resources to participate in the program. The two
shops interviewed as part of this evaluation spent time attending compliance assistance
workshops (one shop spent two days and the other spent two hours), reading compliance
assistance materials (one shop spent one day and the other spent four hours), completing
self-certification forms (one shop spent five days and the other spent four hours), and
completing return to compliance documents (one shop spent four hours and the other
spent two hours). One of the interviewees, a training shop at a vocational high-school,
also implemented yearly ERP-related training for ninth graders in their program as a
result of their participation.
That said, both shops realized cost savings as a result of changes made during their
involvement in ERP. One obtained savings by decreasing waste and becoming generally
more efficient, the other through changes in materials and equipment. Both shops felt that
the program was designed to be sensitive to their operations and cost structures.
MAINE
Program Effectiveness
DEP was able to achieve a 42 percent participation rate in its ERP (i.e., 42 out of a
universe of 100 eligible auto body shops participated in self-certification100). Among the
shops that self-certified, 81 percent self-identified as being out of compliance with one or
more requirements. This sort of increased awareness is an important short-term program
outcome that can lead to changed behaviors and ultimately reduced environmental
impacts. From the state's perspective, these participation and self-identification rates
made the program a success. Maine had a very positive experience with the auto body
ERP, and believes that ERP is one of the best programs which has been administered and
supported by EPA.
Maine notes that program participants report that the ERP led to compliance
improvements in areas such as air quality and hazardous and universal waste. Specific
improvements included increased shop-licenses, raised awareness about compact
fluorescent light bulbs, increased conscientiousness about covering solvent containers,
and many updated air filtration systems. The auto body shop representative interviewed
for this evaluation owns three participating shops. This shop owner reports that the major
change in his shops occurred in terms of awareness:
[Participating in ERP] heightened our awareness of what we do. We already had
things like state of the art spray booth technology in place; the ERP brought
issues like hygiene to the forefront of our thoughts. We now make sure best-
100 According to the Maine DEP's final State Innovation Grant Report, "The selected universe for this project
included 104 auto body facilities within the 3 targeted counties during the baseline surveys. [Four] auto body
facilities went out of business prior to the post-certification surveys. [Therefore,] the universe for the post-
certification surveys included 100 facilities."
81
-------
ERP Evaluation: Final Report August 2009
practices are in place. A lot of these practices were in place to begin with - the
ERP made us think about and confirm these things.
In addition, as a result of ERP participation, his shops became involved in an additional
program called the Governor's Carbon Challenge.
This shop owner also notes that ERP's influence goes beyond participating shops:
.. .whenever you focus on an industry - like the auto body industry - that word
spreads. Even the folks who don't participate or attend are making adjustments
as well. Nothing but good comes out of an ERP like this, both directly and
indirectly.
In this case, a message from the DEP rang out. It basically gave shops a heads
up to get in compliance, saying that down the road if you're not in compliance
you will be penalized.
The ERP provided an opportunity for regulators as well as auto body shops to learn.
Increased communication between the industry and DEP led to a better understanding on
the part of regulators as to which requirements were feasible in the field. The shop owner
interviewed believes that ERP was sensitive to his shops' operations and cost-structures.
In one instance, DEP representatives made a site-visit to one of his facilities and became
aware that a program requirement was impractical in the field; in turn, program staff
modified what they were asking in this regard.
Lastly, ERP improved the relationship between regulators and the auto body industry.
The DEP connected with both trade associations and individual shops. From one shop
owner's perspective, the DEP appeared intent on helping and improving compliance, and
was not heavy-handed.
Program Costs
Total program costs for the Maine ERP included an expenditure of approximately
$200,000. This included an EPA contribution of a $152,000 State Innovation Grant used
for a range of program expenses, including $20,100 in contractor support. The DEP
contributed an additional $47,000 in matching funds.101
The DEP invested substantial internal staff resources in the ERP.102 The design and
development phase required two Full Time Equivalents (FTEs) for the period of six
months; the initial implementation phase also required two FTEs for six months; and
program follow-up/ongoing support required one and a half FTEs for two years. Most of
these staff time resources were supported by the State Innovation Grant mentioned above.
101 "Auto Body Environmental Results Program: Final Report". Maine Department of Environmental Protection,
Office of Innovation, May 7, 2007.
102 The majority of the state staff costs were covered by the State Innovation Grant and the DEP
contributions. However, some additional staff costs were incurred during the program that are not included
in this value. Therefore, our cost estimates are underestimating the total costs of the program to the state.
82
-------
ERP Evaluation: Final Report August 2009
The auto body ERP was the first such program developed by Maine, and therefore it
required more resources to implement than the projected costs for subsequent ERPs.
Initial costs for the auto body ERP were high, but they tapered off over time as the staff
member working on the program required less assistance and became more independent.
EPA financial support - in the form of an SIG - allowed Maine to implement the
program.
Maine believes that the ERP required fewer resources than outreach alone would have,
while reaching a greater percentage of the population of auto body shops. At the same
time, Maine asserts that the ERP generated more environmental/health results than the
alternative assistance approach would have. According to the state, a major reason for
these improvements is self-accountability; self-certification gives auto body shops more
ownership, because they actually have to sign the form and send it off.
The auto body shop representative interviewed for this evaluation indicated that
participation requires resources as well. He estimated that attending two compliance
assistance workshops required four hours, reading compliance assistance materials
required two hours, and completing self-certification forms an additional hour. That said,
due to involvement in ERP, he notes that his shops are doing a better job minimizing
waste, which has led to savings on purchasing and disposal of materials. Furthermore,
these shops will also be switching over to waterborne base coat in the near future as a
result of the program, which will bring further savings along with improved
environmental outcomes.
RHODE ISLAND
Program Effectiveness
DEM was able to achieve a 47 percent participation rate in its ERP (i.e., 171 out of 367
eligible auto body shops volunteered to participate in self-certification).103 From the
state's perspective, the program was a success, with its most important achievement being
statistically significant improvements across multiple regulatory areas. For the shops that
were interviewed for this evaluation, ERP was an educational experience for the shops,
increasing their awareness of what was required of them. For one of the shops, it showed
them what it needed to do and where it could go for more information. As a result of
ERP, this shop went above and beyond what was needed in regards to Right-to-Know
requirements. This, in turn, had a positive effect on employee morale.
Program participants also report that the ERP affected the awareness and attitude of auto
body shops. One auto body shop owner reports:
[As a result of participating in ERP] the shop has evolved into a cleaner,
healthier environment. It has changed from a dirty body shop to something you
can walk through that is nice and clean. There's been an attitude change, though
it's hard to pinpoint anything specifically.
Source: "Rhode Island ERP auto body results - extracted from region 5 meeting slides"
83
-------
ERP Evaluation: Final Report August 2009
Both auto body representatives interviewed for this evaluation engaged in substantial
dialogue with the DEM and had a very positive experience. From the shops' perspective,
the regulators wanted to work with them, were on the same side, and were trying to help.
With this in mind, both shops found the cutback that occurred after the first cycle of ERP
to be an impediment to continued progress. According to one shop, the seminars and
outreach that were lost were valuable, due to the educational component and, especially,
the dialogue they provided:
.. .the problem is the communication is no longer there.. .it's frustrating. This
lack of communication is a prescription for making the program go downhill.
[ERP] becomes just a form to fill out. Relationships keep participants involved
and excited about the program, which is vital.
In its first cycle of ERP, DEM found statistically significant improvements had occurred
in seven indicators. The state continued implementing ERP for a second cycle, but due to
resource limitations, the state reduced its efforts on outreach in the second cycle. Despite
this cutback, the post-Round 2 random inspections conducted in 2008 displayed
continued incremental progress: at this time the state observed statistically significant
improvements in ten categories (comparing post-Round 2 random inspections to baseline
levels of performance).
Additionally, according to Rhode Island, preliminary analysis of data comparing shops
that self-certified and those that did not indicated that there was improvement overall,
even for facilities that did not self-certify. In other words, Rhode Island believes that the
ERP seems to be having the affect of influencing behavior across the auto body sector as
a whole in Rhode Island, not just those that formally participated in the program by
submitting a self-certification form.
Program Costs
The DEM contracted with staff from the University of Rhode Island's Center for
Pollution Prevention to assist with ERP design and implementation. The total amount
spent on the program up to this point is estimated at between $150,000 and $250,000
(including state staff time). This includes less than $10,000 spent on the direct costs of
printing, postage, and workshops. In addition, the University staff used some of their
own EPA grant money (received for pollution prevention research) for their ERP support.
DEM has invested substantial internal staff resources over the course of the first round of
the ERP. The design and development phase required one to two FTEs; the initial
implementation phase also required one to two FTEs; and program follow-up/ongoing
support required less than one FTE. The total amount invested in the program decreased
dramatically overtime due to factors like reductions in available staffing. OEM's
investment in ERP has declined such that resources expended in the second cycle of ERP
have been less than half of what was expended in the first cycle. As described earlier,
despite this cutback, additional positive outcomes occurred in the second cycle.
Auto body shops have also invested time and resources in participating in ERP. The two
shops interviewed as part of this evaluation spent time attending compliance assistance
workshops, reading compliance assistance materials, and completing self-certification
84
-------
ERP Evaluation: Final Report August 2009
forms. One of the shops also spent time assisting the DEM with the creation of the
program. Furthermore, as a result of participating, it spent $25,000 on a new vacuum
system that was above and beyond ERP requirements.
One of the shops interviewed saw cost savings as a result of changes made during its
involvement in the program, while the other shop did not make any direct changes while
in the program, and saw no financial benefit. Due to its participation, this first shop
implemented a new anti-dust system and generally cleaned up shop operations. This, in
turn, increased efficiency and improved product quality and employee morale, all factors
that affect the bottom line, even if the cost savings are not easily quantifiable or show up
in the short term. As the shop representative notes:
Overall I have no doubt that the changes we've made have been very good
investments. I've spent money that other shop owners haven't, but will receive
long run benefits for years and years to come.
Both shops felt that the program was designed to be sensitive to their operations and cost
structures.
ERP required more resources than maintaining the status quo of annually inspecting less
than five percent of facilities statewide. However, it led to substantial improvements in
performance and allowed the DEP to make inferences about the universe of facilities at
the same time.
IV. ERP IMPLEMENTATION EXPERIENCE
This section describes the insights of ERP states about what factors influenced their
success with ERP, and how the program could be sustained over time, and applied to
other sectors and states. Importantly, two of the three programs reviewed have not
continued, and one is operating on severely limited resources. ERP proponents suggest
that the program can help states use their resources more effectively to monitor
compliance and/or improve performance in sectors that states cannot address through
more traditional permitting approaches. Thus the question arises as how states view the
value of ERP, what its role should be in the future, and how the program can be applied
within available resources. This section offers state perspectives on these questions.
DELAWARE
Key Factors Influencing the Program
The ERP was primarily developed to complement the source category permit. The
DNREC believed that the ERP would enhance regulatory programs and improve
compliance. With this in mind, a portion of the results may be attributable to the permit,
rather than the ERP itself. The program's development and results should be understood
within this context of supporting the source category permit.
Cooperation among and buy-in by the various DNREC offices was integral to the
program. Without it, the comprehensive package addressing multiple environmental
media that emerged would have been impossible to develop. The program's success has
encouraged DNREC to look for further opportunities for this sort of collaboration.
85
-------
ERP Evaluation: Final Report August 2009
Sustainability and Transferability
The EPA State Innovation Grant was at the heart of the program's success. Without it,
DNREC would not have been able to hire consultants who helped with the program's
design and implementation. Similarly, the most significant current limiting factor is cost.
With additional funding and resources, the state would be interested in doing the program
again. Currently, the DNREC is doing a self-certification program for the dry cleaning
sector but they can not afford to engage in measurement (e.g., site inspections).
Involvement in the States' ERP Consortium and obtaining information such as what has
been done, and what works and what does not in implementing ERP, is essential for a
successful program. So too is EPA technical support and inter-office collaboration within
DNREC.
The small number of eligible facilities in Delaware meant that the ERP was not as
efficient as it could have been in other contexts. Delaware notes that small states still
have to conduct a relatively large number of baseline and post-certification inspections; in
other words, the benefit of only having to inspect a random selection of facilities is less
when you have a smaller number of facilities in the first place.
MAINE
Key Factors Influencing the Program
Maine cites several internal and external resources that were important to the program's
success, as follows:
Strong relationships with trade associations: DEP had an established and
positive relationship with the Maine Auto Dealers Association prior to
beginning ERP. DEP had a champion at this organization, and leveraged this
relationship to help get the ERP started and organize the workshops. This
relationship with the trade association also boosted participation in the
program.
Support from state regulators in Maine: Internal support at DEP was integral
to the program's development. Having the assistance of other regulatory offices
was key to working through technical compliance issues.
Information and resources from other states: The States' ERP Consortium
was a valuable resource to new states undertaking ERP. Maine notes that there
are ERP templates available, resources and tools that can be adapted by states
starting a new ERP.
In addition to resources, Maine DEP used several implementation strategies that
contributed to the program's success. For example, the DEP found that it was very
helpful to conduct a test-run of the survey checklist before full implementation. This type
of field testing suggests that arranging the checklist in the order shops carry out their
work can make the checklist more efficient. In addition, DEP found that postcard
reminders of the upcoming deadline for self-certification were very effective at
stimulating participation in this aspect of the program.
86
-------
ERP Evaluation: Final Report August 2009
One factor that limited Maine's success with the ERP was the short-term nature of the
program. As the state describes, it takes time for a sector to understand ERP and for large
improvements to take place, particularly in states where there has been limited
compliance assistance or outreach in the past. Maine believes substantial improvements
can occur over time, with consistent education and workshops. Many shops are reluctant
to change shop practices and how they do business; over time, these shops can be
encouraged to participate, often by peers who are champions of the program.
Sustainability and Transferability
Funding was the key challenge for Maine's ERP. The program relied on the EPA State
Innovation Grant for support, and there was no funding infrastructure available for
ongoing support after the grant ended. For ERP to be most effective, Maine believes there
should longer-term, consistent funding. One way that Maine suggests for ERP to get
dedicated funding is through Performance Partnership Agreements (PPAs) with the EPA.
From the state's perspective, pollution prevention and ERPs should be funded through
PPAs or another dedicated EPA pool.
If Maine had access to more funding, the program manager would propose to use ERP to
address approximately six sectors made up of small sources, rotating focus and budget
among them each year. State-dedicated funding to target a different sector each year
would be critical to maintaining an ERP program.
The state believes that the resources required to apply for EPA State Innovation Grants
are too great, especially given the recent drop in grant funding, which makes winning less
likely. From the state's perspective, it would be good if the application process for the
grants could be made more efficient, so it is not a deterrent.
RHODE ISLAND
Key Factors Influencing the Program
A key factor that allowed the DEM to develop and implement the ERP as they did was
the partnerships that they had established. The Department worked with the University of
Rhode Island, a vocational and technical school, the Rhode Island Department of Health,
and the Rhode Island Autobody Association to do the research that formed the basis of
the program. Part of the funding for the University's ERP work came from their own
grants. There were also six shops on the ERP steering committee that looked at materials
and workbooks before they were sent out. The ERP developers also had good
relationships with other internal offices at DEM. Due to the nature of the program
addressing multiple environmental media, support from offices such as RCRA was
integral to developing a workbook that accurately covered the various regulations
involved.
Results were influenced by regulations addressing air emissions from the auto body
sector (specifically HVLP spray guns, VOC compliant coatings, and enclosed spray gun
cleaners) that were passed in 1994, well before the ERP was implemented. These Rhode
Island regulations, along with the state's previous outreach and education, meant that
there were already high levels of compliance in this area, and thus opportunities for
improvement were limited. Outcomes for other indicators were influenced by specific
87
-------
ERP Evaluation: Final Report August 2009
factors like state health inspector on-site assistance (which affected the worker health and
safety indicators) and targeted outreach for methylene chloride. In some areas
improvements simply resulted from awareness of a previously unknown practice or
requirement, or whether or not change was easy to implement, such as by posting a sign.
The ERP's impact was also increased by being connected to the license renewal process.
As one shop owner describes:
Being tied to auto body license renewal gives the ERP extra clout. Everyone has
to look at the ERP and fill it out, so at the very least everyone sees where they're
[messing] up. Everyone sees their little non-compliances, and may eliminate
some serious ones as well.
Sustainability and Transferability
After the first cycle, it was clear to the DEM that ERP was working. DEM judges that it
does not have the resources to inspect every facility in the state, and it views ERP as a
program that successfully improved compliance without substantial resource expenditure.
In fact, growing budget concerns have caused DEM to consider ERP for other sectors
where resources are low.
According to Rhode Island, the key hurdle in getting states to see the cost-efficiency of
ERP is getting them to understand the statistical aspects of the program. With random
sampling, you can inspect a very small number of facilities and in doing so you can
potentially characterize compliance for the entire universe. In the state's view, this
method is applicable whether or not other states want to run a full ERP and should be
applied in many situations where states are conducting inspections. If states want to do
specific targeting, a combination of approaches can be used, or the ERP can be modified.
For example, Rhode Island is applying ERP to the Underground Storage Tank sector by
taking a risk-based approach. In other words, Rhode Island is inspecting all higher risk
facilities on a relatively frequent basis. For low and medium risk facilities, Rhode Island
is taking a random sample to monitor sector-wide performance (so any individual low or
medium risk facility may not be inspected as often).
While the DEM has continued to view ERP positively, the shops we interviewed have
been frustrated by the cutback that occurred after the first cycle. One shop, in particular,
feels that the benefits the program has brought could be lost. According to the shop
representative:
Rhode Island did a tremendous job at the beginning. It reached out and helped
us. However, they're running the risk of letting it all slip through their fingers.
They could send us a yearly reminder letter; it wouldn't have to be threatening,
just a simple "hello." We totally forget about little things like checklists, and
simple reminders - which don't require much time or cost - are very helpful.
Upper management buy-in and early stakeholder involvement - which occurred in Rhode
Island through, among other things, a research program and an ERP workgroup - are
essential for a successful program.
Lastly, Rhode Island perceives that ERP was a good fit for the auto body shops due to the
variety of environmental concerns associated with the sector. However, there are some
-------
ERP Evaluation: Final Report August 2009
circumstances where ERP may not be the best approach. For example, in a sector with a
relatively small number of facilities, where there are resources to inspect them all, ERP
may not make sense. That said, ERP may still be useful if it allows resources to be
reallocated elsewhere.
V. BROADER REFLECTIONS ON ERP CHALLENGES
Two key questions underlying this evaluation include what factors influenced the states'
decisions to continue, not continue, or modify their programs after initial pilots
(evaluation question 8), and what implementation challenges states face in developing
and carrying out their programs (evaluation question 9). Of the three states included in
this evaluation, only one state (Rhode Island) has continued its auto body ERP through a
full second cycle. In addition, a second state, Delaware, completed a partial second cycle
(facility certification only). Both Rhode Island and Maine have used ERP in other
sectors, and Delaware has conducted a partial ERP in another sector. However, some
observers have wondered why more states have not continued their ERPs overtime, and
what this means about the effectiveness of ERP.
While we considered these questions as they pertain to Delaware, Maine, and Rhode
Island in Section IV of the report, we also interviewed staff at EPA and representatives of
the ERP States' Consortium to gain a broader perspective on the sustainability of ERP, as
well as challenges in implementing the program.104 These interviews did not focus
primarily on the three states included in this evaluation, but rather drew on the EPA and
Consortium perspectives on a broad range of ERPs in different states and sectors.
The interviews suggest that a key challenge related to the sustainability of ERP is that in
many cases ERP is perceived as an "extra" program; i.e., one that is innovative and
suitable for addressing sectors that have not historically been well covered by regulatory
agencies. ERP has often been pioneered by a "lone staffer" or a small group of staff who
believe in the potential for ERP, and in some cases have secured a State Innovation Grant
(SIG) from EPA to fund the program. In these situations, when budgets are cut or SIG
funding ends, the program does not have the support to continue. Without a sustained
source of funding, and an overall institutional context in which ERP is viewed as integral
to how regulators will meet their goals, the program is likely to continue only as an
experimental effort that may demonstrate success but not fully take root. States and EPA
staff have different perspectives on this issue, which are discussed in turn below.
PERSPECTIVES FROM EPA
From the viewpoint of EPA staff charged with supporting ERP, one of the biggest
challenges in implementing ERP is developing in-state support for the program. The state
needs to be willing to commit resources to an ERP. To value an ERP, state management
has to perceive the program as directly addressing or otherwise enabling progress on a
high priority concern. Meanwhile, staff have to see the program as making their work
easier. If ERPs provide efficiencies that free up staff time to take care of other high
104 These findings are based on four interviews, two with EPA staff members involved in supporting ERP and
two with ERP consortium members. We interviewed Scott Bowles (EPA) on February 19, 2009; Beth Termini
(EPA) on February 24, 2009; Renee Bashel (ERP Consortium) on April 13, 2009; and Al Innes (ERP Consortium)
on April 16, 2009.
-------
ERP Evaluation: Final Report August 2009
priority concerns, then the program adds value and is more likely to be sustained. In
some states (e.g., Massachusetts and Rhode Island), ERP is part of a broader state agency
shift toward changing the way the state allocates resources to "do more with less." In the
current economic climate with severe budget cuts at the state level, some state managers
are relying on ERP as a way to get results more efficiently. However many other state
managers are inclined to rely on traditional approaches for achieving compliance and
focus exclusively on traditional regulatory programs mandated by EPA, rather than
investing state resources in ERP.
From the perspective of EPA staff, it is critical to make sure that state staff working on
ERP engage other offices and programs, and draw on their knowledge and resources. It
is a challenge for a "lone staffer" or a few staffers to convince others to buy-in to ERP.
Moreover, it can be difficult for small business assistance programs to work with their
peers in environmental regulatory offices, since these groups have different goals and
audiences. In most cases, staff implementing ERP need senior management support. In
addition to engaging peers within the state regulatory structure, staff implementing ERP
do best when they can engage industry and academics. Rhode Island exemplifies the
potential for a state to create synergistic relationships with academics and industry; these
partnerships have helped the state focus its ERPs effectively and achieve its goals. Local
governments (municipalities and counties) also represent a potential resource for ERP.
Specific local programs that ERPs could partner with include offices of weights and
measures, local departments of health, and business licensing departments.
In addition to committing resources and forging partnerships, the success or failure of an
ERP is influenced by the ability of people running ERPs to communicate program
outcomes. States need to be able to demonstrate how ERP outcomes can help regulators
do their jobs better and more efficiently. From the perspective of EPA staff, states should
spend time upfront thinking about how they are going to measure the outcomes of the
program, and what matters to potential partners (e.g., program offices within the state or
EPA). States need to allocate sufficient resources to analyzing the ERP results, so that
they have the data in hand to convince others of the value of the program. States also
need to do more to estimate the long-term environmental benefits of ERPs. In the past,
states have sometimes hesitated to try to quantify the environmental results of their
programs because they cannot demonstrate a causal link between ERP and the results
(i.e., they cannot prove that ERP caused the outcomes). However, in EPA's view, while
states should be careful not to make unsupported claims, they should still talk about the
emissions reductions and other environmental improvements associated with the behavior
changes observed as part of ERP.
PERSPECTIVES FROM STATES
The representatives of the States' ERP Consortium interviewed for this evaluation agree
with EPA that access to resources is essential to the success of ERP. However, the state
views differ somewhat from each other and from EPA's. One interviewee made the case
that EPA should be supporting ERP by allowing this approach as a compliance option for
traditional regulatory programs. This interviewee suggested that allowing ERP as an
approach to implement the recent federal Surface Coating Rule for auto body shops
represents a potential starting point. This interviewee also recommends that EPA
90
-------
ERP Evaluation: Final Report August 2009
frameworks such as Performance Partnership Agreements, and the National Program
Managers (NPM) Guidance should include ERP as an acceptable approach for working
with sectors that EPA already requires states to address. The ERP Consortium has tried
to influence the NPM Guidance in the past and, according to this interviewee, a few years
ago the Guidance did provide flexibility that would allow for ERP, but in recent years it
has not. Overall, this interviewee suggests that EPA should demonstrate to states that the
Agency views ERP as a way to address traditional regulatory programs, and the Agency
should also provide sustained funding for ERP.
Another interviewee sees the challenge of transitioning from traditional inspection
programs to statistically-based programs like ERP as an internal challenge for the states,
rather than an issue for which EPA needs to provide the primary leadership. In this
interviewee's opinion, very few states are prepared to devote the resources needed to
sustain ERP and maintain a presence in sectors over time, such as Massachusetts has
done. Other states are much more likely to use ERP as an overlay on existing programs,
which is likely to be discarded when fiscal constraints are severe. One approach in light
of this situation would be to "un-package" ERP. The statistically-based inspection
component of ERP may be best used as a "one time" tool to assess problems in a sector,
rather than as an ongoing program to maintain compliance. From this interviewee's
perspective, statistically-based inspection is too resource intensive to do on a routine
basis, and the data it produces are not compelling to existing programs that may not
demand or value such data. At the same time, the self-certification and plain language
compliance assistance components of ERP are powerful tools and can motivate behavior
changes among the regulated community in a cost-effective manner.
In addition to these issues of resources, and state demand for statistically-based data on
sector performance, the Consortium representatives identified several implementation
challenges. These include:
Finding an accurate universe for the sector, and selecting the types of
facilities to include. It is easier to capture samples in small states, as sites are
closer together and require less travel time and cost to identify.
Developing outreach materials. It takes significant resources to address
issues related to all the different environmental media, and states need to find
the right balance of providing sufficient detail but not overwhelming the target
audience. This relates back to the challenge of identifying the right universe of
facilities: a bigger, more heterogeneous universe (with a wide range of
applicable requirements) requires more information and materials.
Structure of the state regulatory agency, and whether it is conducive to
multimedia efforts. In some states, coordination across media offices is
difficult, which makes it more challenging to implement a program like ERP
that addresses multiple environmental media.
Obtaining buy-in from existing program staff, who may not see the need for
or value of ERP, and may believe that traditional approaches should be
continued, even if they cannot measure their results.
91
-------
ERP Evaluation: Final Report August 2009
Partnering with others to implement ERP, if they do not have the same
motivations in pursuing the program. This challenge, which could also be
described as trying to implement ERP by "remote control," reflects the fact that
states often need to cobble together a collection of staff from different programs
and backgrounds to carry out ERP inspections, and these staff may find it
difficult to implement the ERP as designed. For example, compliance
assistance staff may find it difficult to conduct unbiased assessment, without
first trying to help facilities improve their performance.
VI. A VISION OF SUCCESS FOR ERP
Interviews with EPA staff and the States' ERP Consortium representatives paint a picture
of the characteristics of a successful ERP, and what such a program can achieve.
Specifically, these interviewees suggest that a successful ERP has many attributes,
including:
Regular schedules of self-certification and random inspections. These schedules
are facilitated by systems such as reminder postcards to regulated facilities.
A regular system of reporting results in an easy to understand format. ERP
results should explain how the program helps regulatory programs meet their
goals, such as fulfilling the requirements of the federal Surface Coating Rules.
An adaptable program format that can incorporate new requirements over time
(e.g. through workbook updates).
A concise set of indicators that are regularly used to track performance.
A sustained source of funding and ongoing management support.
Successful ERPs have the potential to generate the following types of tangible results:
Improved sector performance and compliance with initial ERP implementation.
This may include increased knowledge of and compliance with regulatory
requirements; increased awareness and application of pollution prevention
practices; and improvements in beyond compliance indicators such as energy
use, water use, and recycling. Note that overtime, states may expect
performance of sectors managed with ERP to plateau. ERP can be used to
maintain a high level of sector performance and compliance over time.
Estimates of emissions reductions or other long-term outcomes associated with
ERP.
Overall improvements in worker health and safety.
Improved understanding of sector performance by regulators.
Streamlined state operations and more efficient use of state resources.
Creation of increased capacity and motivation for improved environmental
performance of facilities in ERP targeted sectors.
92
-------
ERP Evaluation: Final Report August 2009
CHAPTER 4: CONCLUSIONS AND RECOMMENDATIONS
Conclusions and Recommendations for ERP Evaluation
The experiences of the three ERPs reviewed in the course of this evaluation suggest that
ERP is associated with improved business practices in the auto body sector and is
regarded as successful by both state and industry representatives. Quantifying
environmental outcomes associated with ERP is difficult, and those outcomes that we
were able to quantify were relatively small. In addition, sustaining the program has
proven to be difficult given resource constraints and overall regulatory priorities. This
conclusions section summarizes the evaluation team's response to each of the evaluation
questions, while the recommendations section offers the evaluators' suggestions about
how state and federal agencies could apply the lessons learned from this evaluation.
I. CONCLUSIONS
QUESTION 1: TO WHAT EXTENT HAVE THE ERPs IN DELAWARE, MAINE, AND RHODE
ISLAND LED TO ADOPTION OF SELECTED BEST PRACTICES THAT
REDUCE THE ENVIRONMENTAL FOOTPRINT OF AUTO BODY SHOPS?
The findings from this evaluation provide strong evidence that during the three ERP
initiatives, auto body shops changed their behavior on many key indicators of business
practices that affect air emissions, hazardous waste, discharges to water, worker health
and safety, and pollution prevention. Overall, we found statistically significant
improvements on nearly half (29 of 65 indicators, or 45 percent) of the indicators of best
practices that states measured at samples of shops before and after ERP assistance and
self-certification. For the measures where there were statistically significant
improvements, we can infer105 that there was an increase in the proportion of the entire
population of auto body shops106 following best practices. In other words, for nearly half
of measured indicators, we believe that the auto body sector as a whole is shifting to
increased use of best practices, not just those shops that were included in the samples
measured. Furthermore, we found no statistically significant declines in performance.
Despite finding measurable changes in the sample population of facilities, for the 55
percent of indicators (36 of 65 indicators) where we did not detect a statistically
105 We make this inference with 90 percent confidence for indicators measured by Maine and Delaware, and
95 percent confidence for indicators measured by Rhode Island.
106 By "entire population" we mean all auto body shops in the state, except in the case of Maine, where the
program only covered the southern portion of the state. The entire population of shops included those that
opted to participate in ERP (e.g., through voluntarily submitting a self-certification form), as well as shops
that did not actively participate in ERP.
93
-------
ERP Evaluation: Final Report August 2009
significant change in performance, we cannot be certain at the specified confidence level
(90 percent in Delaware and Maine, 95 percent in Rhode Island) that there was a change
in the proportion of the entire population of auto body shops following the best practices.
However, for nearly 70 percent of these indicators, states did observe an increase in the
percentage of auto body shops that were following the best practices, as measured by a
comparison of a sample of shops measured at baseline and an independent sample of
shops measured after ERP assistance and self-certification. This could mean that the
percentage of the entire population following best practices for these indicators is in fact
increasing, but that the improvement is not large enough to produce a statistically
significant change.
There were a number of indicators where multiple states observed statistically significant
improvements in performance. For example, in the air emissions category, two states
observed statistically significant improvements in the percentage of shops controlling
sanding dust. In the hazardous waste category, two states found statistically significant
improvements in the percentage of shops sending waste to an appropriate facility or
documentation that shows where the waste is being sent. In the water discharges
category, two states observed statistically significant improvements in the percentage of
shops that post signs prohibiting the discharge of industrial chemicals to non-industrial
drain outlets. However, overall, the number of indicators where multiple states measured
the same behavior and found statistically significant improvements is relatively small.
The indicators showing the greatest improvements varied by state, and in some cases,
states found conflicting trends for the same indicator. For example, results regarding use
of methylene chloride-based paint strippers were mixed. Rhode Island observed a
statistically significant decline in the percentage of shops using methylene chloride-based
paint strippers in the first cycle of ERP. (Note that a decline in this indicator is beneficial
environmentally and from a worker health and safety perspective.) Rhode Island
attributed this improved performance to the fact that the state focused considerable
outreach effort on this issue. However, in the second cycle of ERP, when there were
fewer resources available for outreach, the state observed a slight increase in the
percentage of shops sampled using methylene chloride-based paint strippers, compared to
the end of the first ERP cycle (the increase was not statistically significant). Delaware's
experience was similar to Rhode Island's second cycle of ERP: Delaware observed a
slight increase in the percentage of shops using methylene chloride-based paint strippers,
though this increase was not statistically significant. The differences in these results may
be partially due to the extent of outreach focused specifically on reducing use of
methylene chloride-based paint strippers.
Note that while the overall trend in improving performance measured by states is
consistent with the hypothesis that ERP leads to adoption of selected best practices, we
cannot be certain that the observed changes can be fully attributed to the presence of
ERP. Neither the states we evaluated nor this evaluation team measured performance in a
control group of auto body shops not subject to ERP; therefore, there may have been one
or more external factors outside of ERP contributing to the observed changes. We asked
states about this possibility, and states did raise a few factors such as pre-existing
regulations and concurrent roll-out of new permitting requirement for the sector. These
factors are discussed in more detail in the response to Evaluation Question 10.
94
-------
ERP Evaluation: Final Report August 2009
In addition to evidence about how behavior changed based on statistical samples, we also
reviewed anecdotal reports from states and industry representatives about their
perspectives on how ERP had influenced behavior. States believe that ERP promoted
learning and behavior changes, based on self-certification rates, and the fact that shops
were more likely to come to the state to ask for help. This is discussed later in more
detail in response to Evaluation Question 11. In addition, the auto body shop
representatives that we interviewed also believed that ERP was valuable in heightening
shop awareness of environmental requirements, offering resources, changing attitudes,
and motivating shops to take actions to achieve compliance.
Given the sum total of the evidence for the three states reviewed, this evaluation team
concludes that in the auto body sector, ERP played an important role in focusing auto
body shop owner and operator attention on best practices and environmental
requirements. In many cases, this focused attention seems to have resulted in improved
performance, while in others ERP simply affirmed and supported actions the shops were
already taking. The program also seems to have created a positive relationship between
state environmental agencies and auto body shops, which paves the way for ongoing
communication and future outreach efforts, should they be necessary.
QUESTION 2: WHAT ENVIRONMENTAL AND HEALTH OUTCOMES ARE ESTIMATED TO
RESULT FROM THE IMPLEMENTATION OF THESE BEST PRACTICES?
The ERPs included in this evaluation measured reductions in air emissions, hazardous
wastes, water discharges, and risks to worker health that are expected to improve
environmental conditions for people that work in and live near auto body facilities. In
addition, the ERPs found that shops were taking actions to prevent pollution, such as
reducing materials usage.
With regard to air emissions, all three ERPs found measurable improvements in the
percentage of shops controlling sanding dust emissions; in two of the three states these
improvements were statistically significant (meaning that we can infer the population as a
whole is improving, not just the shops measured in the samples). Sanding dusts can
contain lead and chromium, which are hazardous and pose health risks. We were also
able to estimate reductions in emissions of VOCs from auto body shops in Maine and
Delaware associated with improving spray gun cleaning methods. Specifically, as
described in the findings chapter, we estimated emissions reductions associated with
switching to automatic spray gun cleaning in Delaware, and use of low-VOC/waterbased
solvents in Delaware and Maine. Of these practices, use of low-VOC/waterbased
solvents had the greater potential for emissions reductions. In both states, we estimated
that VOC emissions reductions associated with the use of low-VOC/waterbased solvents
could be as much as 1.7 tons per year, relative to baseline (although the actual amount
may be less).107 Based on calculations presented in the final Surface Coating Rule, we
estimate that this is equivalent to approximately a 1 percent reduction in total VOC
emissions per shop.
107 Up to 3,355 pounds per year over in Delaware, and up to 3,416 pounds per year in Maine over the course of
the ERP.
95
-------
ERP Evaluation: Final Report August 2009
Regarding hazardous waste management, in one state we were able to develop a first
order approximation of the change in the amount of hazardous wastes subject to proper
hazardous waste determination, which can serve as proxy for proper waste management.
As described in the findings chapter, we calculated that in Delaware, there was an
increase of 102 shops completing hazardous waste determinations. Based on the fact that
these shops were likely CESQGs with a maximum waste generation amount of 220
pounds per month, we estimate that there was a potential increase of as much as 22,440
pounds per month of hazardous waste being properly identified in the state (although the
actual amount is likely to be less). This amount is equivalent to the minimum waste
generated by 10 large quantity generators per month.
We were not able to quantify reductions in water discharges associated with ERP, but two
of the three states did find statistically significant improvements in behaviors that are
expected to reduce water discharges. Specifically, both Delaware and Rhode Island
observed significant improvement in the percentage of shops that post signs prohibiting
the discharge of industrial chemicals to non-industrial drainage outlets, and Delaware
found a statistically significant improvement in the percentage of shops that have
secondary containment for all chemicals. Rhode Island also found a statistically
significant improvement in the percentage of shops that have a proper system in place for
recycling or properly disposing of wastewater, which should substantially reduce the
likelihood of wastewater accidentally leaking into water sources.
Regarding worker health and safety, only Rhode Island measured indicators unique to
this category. The state measured six specific worker health and safety indicators, and
found that all of them improved over the first cycle of ERP. Three of the improvements
were statistically significant: specifically, the indicators related to whether the shop has
established a Personal Protective Equipment Program, whether the shop has a
Lockout/Tagout Program, and whether the employer has posted the Job Safety & Health
Protection poster. All of these improvements are expected to reduce worker risks,
although we are not able to quantify these reductions. Note that in the second cycle of
ERP in Rhode Island, three of the six worker health and safety indicators showed
incremental improvements from Post-Round 1 certification inspections to Post-Round 2
certification inspections, and one indicator showed a marginal decline in performance.
In addition to the indicators unique to the worker health and safety category, it is
important to consider improvements related to methylene chloride usage, since in its
research leading to ERP Rhode Island found that methylene chloride exposures among
automotive repair technicians exceeded OSHA limits in a number of samples.108 As
discussed in the responses to Evaluation Question 1, results for indicators related to
methylene chloride usage were mixed. This indicates that further intervention may be
required to convince shops to stop using methylene chloride-based paint strippers. The
findings related to methylene chloride suggest that when a material is particularly
effective for a certain application, and substitutes do not perform as well, auto body shops
108 Enander, Richard T., Hute, David M., and Missaghian, Richard. "Survey of Risk Reduction and Pollution
Prevention Practices in the Rhode Island Automotive Refinishing Industry." American Industrial hygiene
Association Journal. Vol. 59, 1998, pp. 478-489.
96
-------
ERP Evaluation: Final Report August 2009
may continue using a hazardous material, even when they have been informed about the
dangers it poses to workers.
With regard to pollution prevention, the indicators in this category are a subset of those
described in the air emissions category. We were able to estimate reductions in material
usage associated with increased use of automatic spray gun cleaning methods at shops in
Delaware. Specifically, we calculate that that material usage associated with auto body
shop spray gun cleaning operations declined by between 0.6 tons per year and 1.9 tons
per year in Delaware compared to baseline.
It is difficult to put these environmental and health outcomes in context, since the best
way to do so would be to compare the results from ERP to results from alternate policy
tools. ERP is quite unique in having a design that incorporates measurement as part of
the program; most other policy tools do not have an integrated performance measurement
component, and therefore we do not know whether they are as effective as ERP.
Overall, our ability to quantify environmental and health outcomes resulting from the
behavior changes associated with ERP are limited. The states in this evaluation, like
most ERP states, focused their measurement efforts on tracking changes in key behaviors,
rather than quantifying emissions or other long-term environmental outcomes. Moreover,
using available emissions models to estimate impacts from ERP data requires numerous
assumptions, and the most relevant emissions model (the DfE calculator) is limited to
estimating only air emissions reductions and changes in materials use. Given these
limitations, if it is important to estimate long term outcomes (e.g., emissions estimates)
resulting from changes in behavior, states may need to collect additional data on
emissions reductions, or additional emissions reduction models may be needed to develop
appropriate estimates.
QUESTION 3: WHAT ARE THE COST IMPLICATIONS OF EACH PROGRAM FOR
REGULATORS AND AUTO BODY SHOPS BOTH INITIALLY AND OVER
TIME?
The three states included in this evaluation have spent a range of resources developing
and implementing their ERPs. In Delaware and Maine, program costs (including a
portion of the state staff time involved) ranged from approximately $125,000 to $200,000
for a single cycle of ERP addressing a population in each state of between 100 and
approximately 150 auto body shops. As noted earlier, these costs are not inclusive of the
entirety of resources spent by the states to cover staff costs. Therefore, our cost estimates
are underestimating the entire true cost of the program for these states. In Rhode Island,
the state has spent between $150,000 and $250,000 since the program's inception
(including staff time), which includes two cycles of ERP addressing a population of 367
auto body shops.
Costs for developing and implementing ERP also involve state staff time. In Delaware,
staff time totaled 330 days over approximately 2.5 years (roughly equivalent to half of an
FTE for this time period), while in Maine, 2 FTEs were occupied for 1 year during
design, development and implementation, and staffing dropped to 1.5 FTEs during the
two years of follow-up. In Rhode Island, the design, development, and initial
implementation phases required one to two FTEs; while program follow-up/ongoing
97
-------
ERP Evaluation: Final Report August 2009
support has required less than one FTE. Rhode Island has found that in subsequent rounds
of its ERP, substantially fewer resources have been required to "maintain" the program.
Costs for auto body shop participation generally include time to participate in workshops,
read compliance assistance materials, fill out self-certification forms, and complete return
to compliance plans. Most of the auto body shop representatives interviewed measured
the time needed to participate in each task in hours, although one shop representative said
he spent as much as eight and a half days participating in ERP tasks. In general,
however, auto body representatives interviewed seemed to think that the time required to
participate in ERP was well spent; and in several cases participating in ERP resulted in
cost savings for the auto body shop. In one case, ERP motivated an auto body shop
representative to make a significant ($25,000) capital expenditure on a new vacuum
system that exceeded regulatory requirements.
QUESTION 4: WHAT IS THE COST-EFFECTIVENESS OF EACH PROGRAM?
Overall, the estimated government costs109 expended appear to range from approximately
$800 - $2,000 in government expenses per auto body shop in the population for states
conducting a single cycle of ERP (excluding some state staff costs). As noted earlier,
these states, Delaware and Maine, had populations of 100 and 150 auto body shops,
respectively. For Rhode Island, which conducted two cycles of ERP and had a
population of 367 auto body shops, estimated state costs ranged from approximately $400
- $700 per shop for both cycles of ERP combined. These cost figures are only rough
approximations, and this calculation of cost per shop does not account for the degree of
change in behavior or environmental outcomes associated with ERP. The cost figure for
Rhode Island also does not include the EPA grant money that was utilized by the
University of Rhode Island to support their program. Still, this simple calculation does
highlight that the costs of ERP can be considerable, particularly where the size of the
population targeted is relatively small.
The best assessment of the cost effectiveness of ERP would take into account alternative
approaches to regulating (or not-regulating) the target sector. For the states included in
this evaluation, the alternatives to ERP considered by the states were to: 1) follow-up on
complaints or other information about violations, but otherwise not focus on the auto
body sector, 2) continue inspecting less than five percent of auto body shops for air, water
and hazardous waste compliance each year, or 3) implement a source category permit, but
without the focused compliance assistance efforts associated with ERP.
Compared to the likely outcomes of these less re source-intensive alternatives, states
perceive that ERP resulted in greater improvements in performance. For the first two
alternatives (following up on complaints or conducting infrequent inspections) states
predicted that low levels of compliance would persist, and that ERP provided an avenue
for the states to improve performance. In the case of the third alternative, the state
expected ERP to make the permitting program more effective, since before the ERP, only
a few facilities actually had a permit, and the ERP was designed to ensure all shops had
the required permit. It is more difficult to tease out the relative benefits of ERP and a
These figures include EPA grants and state contributions.
98
-------
ERP Evaluation: Final Report August 2009
source category permit, but the state's feedback suggests that ERP led to more shops
getting a source category permit than would have done so otherwise, which presumably
led to improved performance/compliance. Moreover, in addition to resulting in better
performance than the alternatives considered by these three states, ERP provides far more
data about the results achieved than any of the alternatives.
These benefits of ERP typically come at a higher initial cost, however, compared to these
three less resource-intensive alternatives. Two of the three states in this evaluation have
not found sustainable funding models, and have had to discontinue ERP for the auto body
sector. If states conducted successive rounds of ERP, it's likely that per facility costs
would be reduced, since initial program design and development costs would be reduced
or eliminated in successive rounds. For example, Rhode Island has found that subsequent
rounds of its ERP have required substantially less resources than the initial phases. These
topics are discussed in more detail in the response to Evaluation Question 9.
We recognize that in some cases, ERP may actually present a lower cost alternative, for
example if conducting ERP-type statistically-based inspections is an alternative to
inspecting every facility each year or every few years, particularly in sectors and states
with very large populations of facilities. However, none of the programs included in this
evaluation considered inspections at all facilities as the alternative to ERP.
QUESTION 5: WHAT DO OUR FINDINGS SUGGEST REGARDING THE CIRCUMSTANCES
UNDER WHICH ERPS ARE LIKELY TO PRODUCE COST-EFFECTIVE
RESULTS?
One reason for the substantial per-shop expenditures on ERP is that the population sizes
for all of the states included in this evaluation were relatively small (between 100 and 400
shops). ERP can present economies of scale when applied to a larger population. The
upfront costs of developing a workbook, self-certification forms, and inspector checklists
should be fairly constant, regardless of the size of the population receiving these materials
(with the exception of printing and mailing costs). Moreover, in order to conduct
statistically-based measurement, the number of inspections needed may not increase
considerably, even as the population size increases by an order of magnitude. An
example from the States Produce Results report illustrates this point:
"[I]magine that a state planned to conduct 50 random baseline inspections
among a population of 200 facilities. If they instead had a population of 2000
facilities, they could achieve the same statistical confidence in their results by
doing just 14 more inspections, for a total of 64. With 20,000 facilities, they
would need to do only two more inspections, for a total of 66."110
It is relatively common for states implementing ERP to find that the population size
subject to ERP declines as states comb through databases to eliminate facilities that have
gone out of business and duplicate records. In addition, in some cases the population
originally targeted may be narrowed for policy reasons (e.g., addressing only auto body
shops, and excluding mechanical repair facilities which present a somewhat different set
110 U.S. EPA National Center for Environmental Innovation, ERP States Produce Results 2007 Report: States'
Experience Implementing the Environmental Results Program, December 2007.
http://www.epa.gov/erp/ERPreport.pdf
99
-------
ERP Evaluation: Final Report August 2009
of environmental concerns). At least two of the states included in this evaluation found
that the final population size for the ERP was less than originally predicted. For example,
Delaware initially estimated that there were 300 auto body shops in the state, compared to
the final population size of 152.m Maine initially identified a statewide population of
4,000 auto body and mechanical repair shops and planned to focus on the subset shops in
non-attainment areas;112 ultimately, the project focus was narrowed to a population of
100 auto body shops. Given the final population size in each of these states, it seems that
the potential economies of scale that are possible within ERP have not been realized.
These findings suggest that ERP can be more cost effective when the population targeted
is relatively large. To achieve these savings, the population would need to be relatively
homogenous, with a common set of regulatory requirements and best practices, so that a
shared set of compliance assistance materials and self-certification forms would apply to
all shops.
Another finding is that while the auto body sector seems to offer a promising arena for
ERP, for some types of behavior in some states, shops are already demonstrating strong
performance or changes are more difficult to achieve. In these cases, substantial
improvement may not be observed. In planning for ERP, it is important for states to
consider expected baseline performance (i.e., what percentage of the population is
anticipated to be following best practices at baseline), and likely extent of improvement
(i.e., to what extent are facilities likely to be able or willing to change their behavior). It
can be harder for an ERP to measure statistically significant improvements where the
percentage of shops following a best practice at baseline is already high (e.g., if 90
percent of shops are following a best practice at baseline, there is relatively little room for
improvement). So if a state is considering developing an ERP, it would be best to look to
sectors and issues where a substantial number of shops are not following best practices,
and therefore there is substantial room for improvement. In addition, states should
recognize that some changes (e.g., posting signs, proper storage of materials, or training
employees) are relatively easy to implement, while other changes (e.g., using a new type
of equipment or material and installing emissions control equipment) are more difficult to
make. To the extent that states do not have a good sense of baseline performance levels,
or ability to improve performance, prior to starting ERP, it may be advisable for states to
undertake a limited assessment before committing resources to conducting a full ERP.
For example, states may find it useful to conduct a relatively small, targeted survey to
assess baseline performance on key indicators and to ask facilities about what would
enable or motivate them to change their behavior.
Another factor related to the likelihood of a state achieving cost-effective results through
ERP is the degree to which a staff can build partnerships within and outside state agency
walls. All of the states described in this evaluation forged partnerships within their
agencies and with outside partners to implement the ERP. For example, Delaware
worked with the permitting program staff that was developing the state's source category
111 See State Innovation Grant Quarterly Report for November 2003-February 2004:
http://www.epa.gov/NCEI/stategrants/PDFs/2002DE-Feb2004.pdf.
112 See Maine ERP Pre-Proposal: Implementation of an Automotive Body and Automotive Repair Environmental
Results Program: http://www.epa.gov/NCEI/stategrants/applications/04appllications/maine.htm
100
-------
ERP Evaluation: Final Report August 2009
permit. Maine built strong relationships with its trade associations, and leveraged support
from other regulatory offices within Maine DEP. Rhode Island developed an especially
broad range of partnerships, including a university, state health officials, vocational and
technical schools, and a trade association. These partners not only lent their expertise,
but in the case of the university, they contributed funding toward the project, which made
the overall cost to state regulators smaller. This factor probably contributed to Rhode
Island's relatively smaller program cost per shop.
QUESTION 6: OVERALL, WHAT ARE THE STRENGTHS AND WEAKNESSES OF EACH
OF THE THREE ERPS IN TERMS OF REACHING AUTO BODY SHOPS,
GENERATING ENVIRONMENTAL AND WORKER HEALTH OUTCOMES,
AND ACHIEVING COST-EFFECTIVE RESULTS?
The three states reviewed vary in their strengths and weaknesses. Each state is discussed
in turn below.
The strengths of the Delaware program included the sheer number of statistically
significant results that the program observed (15 out of 19 indicators measured showed
statistically significant improvements; while Rhode Island and Maine each achieved
seven statistically significant improvements.) Delaware also was robust in that it was
implemented in conjunction with a source category permit, and the presence of the ERP
allowed the permitting process to be more efficient and effective, and reach more shops,
than it would have otherwise. The Delaware program achieved a relatively high
participation rate (68 percent of eligible auto body shops that volunteered to participate in
self-certification). A key weakness of the program was the fact that the program was not
sustained beyond one full cycle of ERP (although the state did encourage facilities to
renew their self-certifications). The cost of the Delaware program per shop was
approximately $770, which was between the costs of the other two states.
The Maine program achieved a 42 percent participation rate, which the state viewed as
successful but which is lower than participation rates in the other two ERPs reviewed
here. Among the shops that self-certified, 81 percent self-identified as being out of
compliance. This implies that without the outreach of the ERP program, many of these
shops may have remained unaware of their non-compliance. In addition, the participating
shops reported an increase in their awareness of regulations and best practices in their
shops. Although the state may have had a lower participation rate than Delaware, it
appears that the shops that Maine reached had a great need for the outreach provided. The
costs of the Maine program were relatively high (up to $2,000 per shop) due to the small
population size targeted. As with Delaware, the Maine program was not sustained
beyond a single cycle of ERP.
The Rhode Island program's strengths included its relatively low cost per shop (roughly
between $400 and $700 per shop for two cycles of ERP combined), its strong
partnerships with external stakeholders, and its focus on worker health and safety issues.
In addition, a key strength of the program is that the state has found a way to sustain its
funding, and complete a second cycle of ERP. One relative weakness of the Rhode
Island program is that the participation rate was not as high as that achieved in Delaware
(47 percent).
101
-------
ERP Evaluation: Final Report August 2009
QUESTIONS 7 AND 8: WHAT IS THE CURRENT STATUS OF EACH PROGRAM? WHAT
FACTORS INFLUENCED THE STATES' DECISIONS TO CONTINUE, NOT
CONTINUE, OR MODIFY THEIR PROGRAMS AFTER INITIAL PILOTS?
Of the three ERP programs, only one of them is currently ongoing: the ERP in Rhode
Island. Both ERPs in Delaware and Maine were operational for a specific period of time,
with a specific amount of funding with which to implement the program, and have not
continued. Delaware and Maine each received EPA State Innovation Grants (SIG) to
design and conduct their ERPs. These grants were intended to support the development
and implementation of an ERP in each state, and were not intended to support an ongoing
ERP effort.
Delaware's ERP was designed to work in conjunction with the existing permitting
process for air source categories. Shops were offered compliance assistance, as well as
resources promoting beyond-compliance best management practices and pollution
prevention. One full cycle of the ERP was implemented (including pre and post-program
inspections); after the SIG funding expired, the state did not continue full implementation
of its ERP. In other words, the state did not provide additional compliance assistance
support, nor did it conduct another round of inspections. The state did encourage program
participants to renew their self-certifications in July of 2007, but no further inspections to
measure sector performance occurred after the grant period. The ERP in Delaware was
intended to be a pilot program, in which the state tried the ERP approach as a
complement to its traditional permitting process. Without additional funding, the DNREC
does not currently have the funding or the manpower to continue the program. While the
state cannot continue with a full ERP, it has incorporated an outreach component to its
auto body permitting program, and a self-certification component for the dry cleaning
sector.
Maine's ERP was also considered a pilot project that was designed to address regulations
across all environmental media. The program was designed to increase compliance and
awareness, as well as promote beyond-compliance best management practices and
pollution prevention. Although the ERP was intended to be a pilot program, the
program's manager believed that the program had the potential to serve as a model that
the department could use to apply to other sectors. However, once the SIG funding ended
in March of 2007, the state could no longer sustain the ERP. The ERP is no longer active
due to a lack of funding and a hiring freeze that has prevented the department from
bringing in a new staff person to coordinate the program. The state is currently
implementing another pilot ERP, funded by a separate SIG, to reduce the stormwater
impacts of drive-through commercial establishments.
Rhode Island is the only state to continue its ERP. Its ERP was developed without the
direct assistance of EPA SIG funding. Instead, DEM conducted its own studies prior to
the development of its ERP that revealed significant pollution and health concerns arising
from auto body operations. In addition, its traditional inspections reached less than five
percent of shops in the state. Therefore the department saw a benefit of an ERP as a way
to address the significant occupational health, safety, and environmental issues in the auto
body sector, because of the program's potential to reach more shops. Rhode Island
recently completed its second cycle of ERP, and intends to continue implementing the
program. Moving forward, however, the state plans on spending minimal additional
102
-------
ERP Evaluation: Final Report August 2009
resources in outreach efforts. In addition, the state is adapting its program to be consistent
with the federal Surface Coating Rule for auto body shops, and plans to extend it to a
third and potentially fourth cycle. Finally, DEM is currently implementing ERP programs
in additional sectors, including auto salvage, underground storage tanks, MS4
Construction Site Runoff Control, and reduction of fats/oil/grease discharges from food
processing facilities.
QUESTION 9: WHAT ARE THE PRIMARY IMPLEMENTATION CHALLENGES STATES
FACED IN DEVELOPING AND IMPLEMENTING THEIR PROGRAMS?
The primary challenge states faced in developing and implementing their programs was
access to funding. Both Maine and Delaware relied on State Innovation Grants to provide
the funding to develop and implement their programs. Once the funding ended, the
programs were not able to continue. As discussed earlier, ERPs can require a significant
amount of funds to operate, especially in the initial design phase. While it may vary state
to state, developing the program, including all of the outreach and assistance materials
(e.g., compliance assistance and self-certification workbooks), can demand a lot of effort
and time on the behalf of the state agencies. For example, Delaware reported that a
significant amount of time was required to support the participating auto body shops once
the compliance assistance materials had been sent out. Furthermore, Delaware found that
implementing its ERP in conjunction with the source category permit program required
significantly more resources than the permit program would have incurred on its own. On
the other hand, both Delaware and Maine estimate, and Rhode Island can testify that there
is a steep learning curve with ERPs. In other words, costs for the program taper off as
both state staff and the shops become more familiar with the process. Furthermore, Rhode
Island has found that subsequent rounds of its ERP have required substantially less
resources than the initial phases.
If states see ERP as a one-time trial program, and not as an alternative or complement to
their traditional permitting and inspection process, they may not see the benefit of
continuing to support the program once grant funding has expired. This was the case in
both Delaware and Maine: program managers used SIG funding to conduct an ERP as a
pilot, with the possibility of continuing the program. However, once the funding ended,
the state was unable to commit the necessary resources to continue the program. If, on the
other hand, regulators see ERP as a new way to approach to permitting and compliance,
the state may choose to commit the resources necessary to continue the program. This
was the case in Rhode Island: the state saw the long-term benefit of ERP as an effective
tool to reach more shops and achieve a higher rate of compliance than with its traditional
permitting process, so the state provided the support necessary to maintain the program.
This "buy-in" and support, either from EPA or state agencies, is integral to the program
moving beyond a one-time effort.
Successful design and implementation of ERPs that deal with multiple environmental
media require the cooperation among and buy-in from the various offices that are
affected. For example, in both Delaware and Rhode Island, regulators found that the high
level of input and collaboration with the various media offices made the process much
more successful, primarily because the workbook they developed accurately covered all
of the various regulations involved. While this cooperation is integral to the success of an
103
-------
ERP Evaluation: Final Report August 2009
ERP that deals with multiple environmental media, it can also be a challenge to
coordinate and facilitate the involvement of so many offices.
QUESTION 10: WHAT EXTERNAL FACTORS INFLUENCED THE OUTCOMES OF EACH
PROGRAM
Several factors may have contributed to the outcomes the programs evaluated. In
Delaware, the fact that ERP was developed in tandem with a source category permit may
have made the program more successful. In Maine, the fact that the program was
developed for only part of the state reduced the population size, and increased the
program cost per shop accordingly. In Rhode Island, the fact that the state had passed
regulations specifically addressing the auto body sector, and had conducted outreach
related to this rule, meant that for some indicators, baseline performance was relatively
high. Also in Rhode Island, ERP was implemented in tandem with the auto body license
renewal process, which may have given the program more visibility and clout with auto
body shops. All of the states developed productive partnerships with stakeholders, which
increased the value and credibility of the programs; this was particularly true in Rhode
Island.
Note that the presence of the federal Surface Coating Rule may now be affecting shops'
behavior regarding air emissions, but the vast majority of data analyzed in this report
(with the exception of the Rhode Island's second ERP cycle) was gathered prior to the
promulgation of the federal rule, and therefore we do not expect that the rule had a
substantial impact on shop behavior reported in these ERPs.
QUESTION 11: HOW DOES THE STATE AND/OR EPA REGION INVOLVED IN
IMPLEMENTING EACH PROGRAM VIEW THE PROGRAM'S RESULTS, AND
WHY?
All three states believe their ERPs were successful, although they each had different
reasons for considering the program a success. For example, Delaware said that the ERP
initiative let shops know who to contact when they have questions or need help. Having
a personal connection with the state agency gives the shops a level of comfort and makes
them more willing to ask for help. Maine pointed to the high participation rates in ERP
and the large proportion of self-certifiers who identified themselves as being out of
compliance as evidence that shops are learning. Moreover, for Maine, the measurement
component of ERP was important to knowing that the state made an impact. Rhode
Island pointed to statistically significant improvements across multiple regulatory areas as
evidence of their success.
II. RECOMMENDATIONS
Based on our findings from this evaluation, we offer several recommendations for
consideration. Our recommendations are based on our limited perspective as external
evaluators to EPA, and therefore would clearly need to be viewed within the frame of
what is feasible from a staffing, resources, legal, and policy perspective. Recognizing
these caveats, we suggest the following:
1. Combine forces. ERP offers economies of scale, but until recently states have
been working independently. The Region 5 effort to conduct a multi-state ERP
104
-------
ERP Evaluation: Final Report August 2009
offers great potential to streamline the investment in ERP (e.g., developing
materials and conducting inspections); this should allow for more cost effective
results.
2. Decide on a set of common indicators. Much work on this has already been
done as part of the Common Measures project. It would be helpful to be able to
compare and aggregate ERP data for the same sector across programs. Worker
health indicators like those developed in Rhode Island should be expanded to
other states.
3. Collect quantitative data on facility practices, not just information on the
proportions of shops utilizing specific practices, for a small number of
indicators. Measuring a few key indicators (e.g., quantity of methylene chloride
used, quantity and type of paint used, amount of hazardous waste generated)
could help quantify environmental outcomes. As part of this, discuss key inputs
needed to use the existing DfE calculator that states conducting ERP may wish to
collect.
4. Develop a tool to help states estimate environmental outcomes. Transforming
the DfE calculator into something states could easily use themselves could
encourage more states to make the effort to collect key inputs to the tool so that
they can generate outcome data.
5. Un-package ERP. While ERP as a package offers value, it requires more
resources and effort than some states can provide on an ongoing basis. We
suggest that states consider different ways to apply the measurement component
of ERP (e.g., for initial assessment and longer-term monitoring) alone as well as
in combination with compliance assistance and self-certification. States are also
conducting ERP-style self-certification and outreach without the measurement
component; discuss this as a possibility. In addition, ERP components may be
suitable to apply on a limited scale (e.g., developing self-certification forms for
only one environmental media); some states have already tried this out of
necessity, and while it doesn't offer all of the advantages of a full ERP, it may be
a viable alternative for states interested in the ERP model.
6. Consider implementing ERP primarily where larger populations of facilities
are present. This approach has the potential to reduce per facility expenditures
and increase the cost-effectiveness of the program. The findings suggest that ERP
can be more cost effective when the population targeted is relatively large. To
achieve these savings, the population would need to be relatively homogenous,
with a common set of regulatory requirements and best practices, so that a shared
set of compliance assistance materials and self-certification forms would apply to
all shops.
7. Develop a clearer agreement between EPA and states whether ERP can be
used to address traditional regulatory programs. In circumstances where this
is suitable, develop appropriate guidance and a sustainable funding mechanism.
The Surface Coating Rules relating to auto body shops may represent an
opportunity for the states and EPA to explore this.
105
-------
ERP Evaluation: Final Report
August 2009
APPENDIX A: DATA AVAILABLE FOR DFE CALCULATOR ANALYSIS
Exhibit A-l walks through the available data for each of the three states for each of the five categories for
which the calculator can estimate emissions and materials reductions.
EXHIBIT A-1: DFE CALCULATOR INPUTS
BEST PRACTICE
(1 ) Waterborne paint
usage
(2)HVLPspraygun
usage
(3) Training in spray
gun usage
(4) Spray booth usage
DELAWARE
No data available
9 ppa (not statistically
significant) increase in shops
that use painting techniques
that comply with DE
regulations
No data available*5
6 pp (not statistically
significant) increase in shops
that carry out all painting in a
spray booth
MAINE
No data available
No data available
3 pp (not statistically
significant) decrease in shops
that employ training programs
7 pp (not statistically
significant) increase in shops
that carry out painting in a
spray booth
RHODE ISLAND
No data available
100 percent of shops use HVLP
spray guns in both baseline and
post-certification inspections
(no measurable change)
No data available
No data available
(5) Equipment cleaning improvements
Manual/automatic
spray gun cleaning
Gun cup liners
Low-VOC/
waterbased solvents
11 pp (statistically
significant) increase in shops
that use cleaning methods
that comply with DE
regulations
No data available
23 pp (statistically
significant) increase in shops
that use non-VOC cleaning
methods when possible
7 pp (not statistically
significant) increase in shops
that use an enclosed spray gun
cleaner
No data available
47 pp (statistically significant)
increase in shops that use
low-VOC/HAP solvents
5 pp (not statistically
significant) increase in shops
that use enclosed spray gun
cleaners
No data available
No data available
Note: Observed changes in bold are statistically significant.
a All changes are listed as percentage point changes. These changes are calculated by subtracting the observed percentage of
shops in the sample following the behavior at baseline prior to ERP from the observed percentage of shops in the sample
following the behavior at post-certification. We use percentage point changes, rather than percent changes, to more clearly
show the magnitude of change. For example, suppose the observed proportion of shops following a behavior increased from 6
percent at baseline to 12 percent at post certification: this change could be expressed as a 100 percent improvement, or a 6
percentage point improvement. We believe the latter approach is a clearer, more accurate description of changes in
performance.
b Note that while Delaware did observe a statistically significant increase in the percentage of shops that employ a training
program, the focus of Delaware's training was in the proper use and handling of coatings, solvents and waste products in
minimizing air emissions, rather than training in spray gun usage.
For each of the statistically significant changes described above, we estimate the actual emissions
reductions observed by the state below.
A-1
-------
ERP Evaluation: Final Report August 2009
Emissions reductions associated with increased use of automatic cleaning methods.113 First,
we calculate the confidence interval for the difference in the proportion of shops using cleaning
methods that comply with Delaware regulations at baseline and post-certification using the EPA's
Results Analyzer 2007 Tool.114 The Results Analyzer calculates confidence intervals for the
difference between proportions observed in two different rounds of inspections.115 We are 90
percent confident that the confidence interval for the difference is between 5 and 17 percentage
points. We apply this to the entire population of shops targeted by ERP in Delaware (152 shops),
to estimate the range in the number of shops in the population that changed their behavior on this
indicator (between 8 and 26 shops). According to the DfE calculator, assuming other cleaning
behaviors stay constant at a minimum performance level (i.e., no disposable spray gun cup liners
are used, and low VOC or waterbased cleaners are not used), each shop that switches from
manual to automatic spray gun cleaning methods will reduce VOC emissions by 37 Ibs/year and
material usage by 153 Ibs/year. When extrapolated to the estimated number of shops in the
population that changed this behavior, this means that we expect that VOCs from auto body shop
spray gun cleaning operations declined by between 296 and 962 Ibs/year. relative to baseline.
Likewise, we expect that material usage associated with auto body shop spray gun cleaning
operations declined by between 1.2241bs/vear and 3.978 Ibs/vear. relative to baseline. The DfE
calculator does not estimate any reductions in PM emissions associated with switching automatic
spray gun cleaning methods. Note that each practice is considered individually, but in reality the
practices influence one another (e.g., type of spray gun used and training for spray gun use are
interrelated, and there is overlap in the potential emissions reductions between these two
indicators). Therefore, the results are not additive, i.e., the total potential emissions reductions
associated with adopting all practices is less than the sum of potential emissions reductions for
each practice.
Emissions reductions associated with increase in shops in Delaware that use non-VOC
cleaning methods when possible.116 First, we calculate the confidence interval for the
difference in the proportion of shops that use non-VOC cleaning methods when possible at
baseline and post-certification. We are 90 percent confident that the confidence interval for the
difference is between 13 and 36 percentage points. We apply this to the entire population of
shops targeted by ERP in Delaware (152 shops), to estimate the range in the number of shops in
the population that changed their behavior on this indicator (between 20 and 55 shops).
According to the DfE calculator, assuming other cleaning behaviors stay constant at a minimum
113 To calculate this emissions decrease we assume adoption of automatic cleaning methods (the variable tracked in the DfE
calculator) is equivalent to the indicator tracked by Delaware: use of cleaning methods that comply with Delaware regulations
(specifically, enclosed spray gun cleaning systems that are kept closed when not in use, unatomized discharge of solvent into a
paint waste container that is kept closed when not in use, disassembly of the spray gun and cleaning in a vat that is kept closed
when not in use, or atomized spray into a paint waste container that is fitted with a device designed to capture atomized solvent
emissions).
114These calculations are based on raw data provided by the states (i.e., the total number of facilities sampled at baseline and
post-certification, the number of facilities sampled at each time period for which the answer to the question was "yes," and the
total number of facilities in the population). Raw data was drawn from the State Innovation Grant final reports for these two
states, available online at http://www.epa.gov/NCEI/stategrants/PDFs/DEautobodyfinalreport.pdf and
http://www.epa.gov/NCEI/stategrants/PDFs/AAaine2004Final%20Report.pdf. We then used this information to calculate the
confidence interval for the percentage of shops using cleaning methods that comply with Delaware regulations at baseline and
post certification.
115 The Results Analyzer can be accessed at: http://www.epa.gov/erp/roadmap/matllist.htmfpagecontents. Calculations for the
confidence intervals are based on the following source: Kish, Leslie, 1965. Survey Sampling. John Wiley £t Sons, Inc. New York,
NY. p.41.
116 To calculate this emissions decrease we assume adoption of low VOC-waterbased cleaners is equivalent to shops that use non-
VOC cleaning methods when possible.
A-2
-------
ERP Evaluation: Final Report August 2009
performance level (i.e., no disposable spray gun cup liners are used, and manual spray gun
cleaning methods are used), each shop that increases the percentage of cleaning solvents that are
low-VOC or waterbased from 0 to 100% will reduce VOC emissions by 61 Ibs/year. When
extrapolated to the entire population of shops targeted by ERP in Delaware, this means that we
expect that VOCs from auto body shop cleaning operations declined by up to 3.355 Ibs/vear.
relative to baseline. (This is calculated by multiplying the maximum number of shops to change
performance by the emissions reductions per shop, or 55 x 61.) We include only the maximum of
this range because it is difficult to estimate the minimum emissions reductions, since some shops
may have already begun using non-VOC cleaning methods when possible, but did not completely
switch over to low-VOC/waterbased solvents. It is difficult to estimate to lower end of this range,
it may in fact be zero. When calculating the emissions reductions, we assume that the shops that
use non-VOC cleaning methods are now using those methods 100 percent of the time ._If any of
these shops were only using these non-VOC materials for a small percentage of their cleaning
operations, the emissions reductions would be much lower. The DfE calculator does not estimate
any material reductions or reductions in PM emissions associated with switching to low-
VOC/waterbased solvents.
Emissions reductions associated with increase in shops in Maine that use low VOC/HAP
solvents.117 First, we calculate the confidence interval for the difference in the proportion of
shops that use low VOC/HAP solvents at baseline and post-certification. We are 90 percent
confident that the confidence interval for the difference is between 40 and 55 percentage points.
We apply this to the entire population of shops targeted by ERP in Maine (an average of 102
shops across the baseline and post-certification periods), to estimate the range in the number of
shops in the population that changed their behavior on this indicator (between 41 and 56 shops).
As noted above, the DfE calculator estimates that assuming other cleaning behaviors stay
constant at a minimum performance level (i.e., no disposable spray gun cup liners are used, and
manual spray gun cleaning methods are used), each shop that increases the percentage of cleaning
solvents that are low-VOC or waterbased from 0 to 100% will reduce VOC emissions by 61
Ibs/year. When extrapolated to the whole population in Maine, this means that we expect that
VOCs from auto body shop cleaning operations declined by up to 3.416 Ibs/vear. relative to
baseline. (This is calculated by multiplying the maximum number of shops to change
performance by the emissions reductions per shop, 56 x 61.) We include only the maximum of
this range because it is difficult to estimate the minimum emissions reductions, since some shops
likely began using low VOC /HAP solvents, but did not completely switch over to these
alternative solvents. When calculating the emissions reductions, we assume that the shops that
use non-VOC cleaning methods are now using those methods 100 percent of the time ._If any of
these shops were only using these non-VOC materials for a small percentage of their cleaning
operations, the emissions reductions would be much lower. The DfE calculator does not estimate
any material reductions or reductions in PM emissions associated with switching to low-VOC or
waterbased solvents.
There is no discussion presented here for Rhode Island, as there were no observed statistically
significant increases for any of the five categories for which the DfE calculator can estimate
emissions and materials reductions.
117 To calculate this emissions decrease we assume adoption of low VOC-waterbased cleaners is equivalent to shops that use non-
VOC cleaning methods when possible.
A-3
-------
ERP Evaluation: Final Report
August 2009
APPENDIX B: DFE CALCULATOR INPUTS
Note that all example calculations represent the largest change a shop can make, for some indicators, there may be a continuum of changes that
would result in smaller emissions reductions.
Also note that cells in gray (shaded) represent default values provided by the calculator, or standard values we found in our literature search. Cells
in yellow represent facility practices and are the indicators for which we calculate emissions and materials reductions in Appendix C.
DFEINPUT #
DFE INPUT DESCRIPTION
DATA SOURCE
DATA VALUE (INPUT TO
DFE)
NOTES/ASSUMPTIONS
How many paint jobs per
week are performed at your
shop?
What is the density of the
solvent-based paint you use
in your shop?
What is the VOC content of
the solvent-based paint you
use in your shop?
What is the density of the
waterborne paint you use in
your shop?
What is the VOC content of
the waterborne paint you
use in your shop?
Enander, Richard T., et al. "Chemical
Characterization of Sanding Dust and Methylene
Chloride Usage in Automotive Refinishing:
Implications for Occupational and Environmental
Health." AIHA Journal, Vol. 63, No. 6, November
2002, pp. 741 -749
Default value provided as part of DfE Calculator
Default value provided as part of DfE Calculator
Default value provided as part of DfE Calculator
Default value provided as part of DfE Calculator
6 jobs per week
Primer coats: 10
Ibs/gallon
Basecoats: 7.9
Ibs/gallon
Clear coats: 8.2
Ibs/gallon
Primer coats: 4.8
Ibs/gallon
Basecoats: 5.0
Ibs/gallon
Clear coats: 5.1
Ibs/gallon
Primer coats: 8.3
Ibs/gallon
Basecoats: 8.3
Ibs/gallon
Clear coats: 8.3
Ibs/gallon
Primer coats: 2.1
Ibs/gallon
Basecoats: 3.5
Ibs/gallon
Clear coats: 2.1
Ibs/gallon
Table 1 on page 743, EPA definition of
"small" shop.
B-1
-------
ERP Evaluation: Final Report
August 2009
DFEINPUT #
DFE INPUT DESCRIPTION
DATA SOURCE
DATA VALUE (INPUT TO
DFE)
NOTES/ASSUMPTIONS
9
10
11
12
13
14
What reduction in basecoat
paint usage have you found
with waterborne paint?
How much paint used at
your shop is considered to
be waterborne?
What type of spray gun does
your painter use?
Are your painters trained to
use HVLP spray guns
properly?
What is the efficiency of
your spray booth filter?
How much do your painters
spray in a spray booth?
How much fresh-solvent-
based thinner do you use for
spray gun/cup cleaning in
one week, prior to
improvements?
What is the density of the
solvent-based thinner
(cleaner) you use in your
shop?
What is the VOC content of
the solvent-based thinner
(cleaner) you use in your
shop?
Default value provided as part of DfE Calculator
25% reduction
Example shop calculation
Example shop calculation
Example shop calculation
Default value provided as part of DfE Calculator
Example shop calculation
Enander, Richard T., Gute, David M. and
Missaghian, Richard. "Survey of Risk Reduction and
Pollution Prevention Practices in the Rhode Island
Automotive Refinishing Industry." AIHA Journal.
Vol. 59, 1998, pp. 478-489.
Default value provided as part of DfE Calculator
Default value provided as part of DfE Calculator
Baseline: 0%
Improvement: 100%
Baseline:
Conventional
Improvement:
HVLP
Baseline: no
Improvement: yes
98%
Baseline: 0%
Improvement: 100%
0.875 gallons per
week
7 Ibs/gallon
7 Ibs/gallon
We are calculating the emissions reductions
when a shop changes from using no
waterborne paint to using all waterborne
paint. In other words, this represents the
largest improvement a shop can make.
However, this may be realistic, as
waterborne paint generally requires different
equipment than traditional paint, and a shop
may need to switch completely if at all.
For primer coats, basecoats, and clear coats,
we assume here that a shop changes from
using conventional spray guns to HVLP guns.
Assume that for primer coats, basecoats, and
clear coats, painters go from never spraying
in a spray booth to always spraying in a spray
booth.
Based on average of 42 gallons per year of
solvent - gun cleaning waste in Table VI,
page 484.
B-2
-------
ERP Evaluation: Final Report
August 2009
DFEINPUT #
DFE INPUT DESCRIPTION
DATA SOURCE
DATA VALUE (INPUT TO
DFE)
NOTES/ASSUMPTIONS
15
16
17
18
19
What method is used to
clean spray guns at your
shop?
Do you use disposable spray
gun cup liners?
What is the density of the
low-VOC / waterbased
cleaner used in your shop?
What is the VOC content of
the low-VOC/waterbased
cleaner used in your shop?
How much of your cleaning
solvents are considered to
be low-VOC or waterbased?
Example shop calculation
Example shop calculation
Default value provided as part of DfE Calculator
Default value provided as part of DfE Calculator
Example shop calculation
Baseline: Manual
Improvement:
Automatic
Baseline: No
Improvement: Yes
8.3 Ibs/gallon
0.0083 Ibs/gallon
Baseline: 0%
Improvement: 100%
We assume that a shop goes from using no
low-VOC or waterbased solvent to using all
low-VOC or waterbased solvent. Again, this
represents the largest improvement a shop
can make.
B-3
-------
ERP Evaluation: Final Report August 2009
APPENDIX C: EMISSIONS REDUCTIONS ESTIMATE
While we cannot use the calculator to directly estimate the emissions reductions associated with adoption
of certain best practices in the shops in these three states, we can use the tool to describe the potential
emissions reductions that are possible for a "typical" small auto body shop performing six paint jobs per
week. For each best practice we analyze, we use the calculator to assess the potential change in
emissions if the shop changes from the lowest level of performance (e.g., using conventional spray guns)
to an advanced level of performance (e.g., using FfVLP spray guns). As part of these calculations, we use
several default values to estimate potential emissions reductions. For example, we assume that a typical
small auto body shop performs six paint jobs per week. For a full description of all values used, and their
source, please see Appendix B.
Exhibit C-l below shows the potential emissions and materials reductions possible for each best practice
that is adopted. These are the yellow highlighted items/DfE inputs from Appendix B. Note that each
practice is considered individually, but in reality the practices influence one another (e.g., type of spray
gun used and training for spray gun use are interrelated, and there is overlap in the potential emissions
reductions between these two indicators). Therefore, the results are not additive, i.e., the total potential
emissions reductions associated with adopting all practices is less than the sum of potential emissions
reductions for each practice.
C-1
-------
ERP Evaluation: Final Report
August 2009
EXHIBIT C-1: SUMMARY OF POTENTIAL EMISSIONS AND MATERIALS REDUCTIONS FROM THE DFE CALCULATOR
DFE INPUT
#
7
8
9
11
15
16
19
DFE INPUT DESCRIPTION
How much paint used at your
shop is considered to be
waterborne?
What type of spray gun does
your painter use?
Are your painters trained to
use HVLP spray guns
properly?
How much do your painters
spray in a spray booth?
What method is used to
clean spray guns at your
shop?
Do you use disposable spray
gun cup liners?
How much of your cleaning
solvents are considered to be
low-VOC or waterbased?
DATA VALUE (INPUT TO
DFE)
Baseline: 0% (none)
Improvement: 100%
(all)
Baseline:
Conventional
Improvement: HVLP
Baseline: No
Improvement: Yes
Baseline: 0% (never)
Improvement: 100%
(always)
Baseline: Manual
Improvement:
Automatic
Baseline: No
Improvement: Yes
Baseline: 0% (none)
Improvement: 100%
(all)
Total3
voc
REDUCTION
(LBS/YEAR)
255
71
62
N/A
61
362
% VOC
REDUCTION
66%
18%
20%
N/A
100%
81%
PARTCULATE
REDUCTIONS
(LBS/YEAR)
73
65
57
208
N/A
211
%
PARTICULATE
REDUCTION
34%
31%
39%
98%
N/A
99%
MATERIAL
REDUCTION
(LBS/YEAR)
248
136
119
N/A
207
625
% MATERIAL
REDUCTION
33%
18%
20%
N/A
68%
60%
Note:
a. Reductions for each improvement are calculated individually. Although the total reductions are calculated as the overall reductions of all improvements combined, the
individual reductions are not additive, i.e., the total sum of each individual reduction will be greater than the total presented here.
C-2
-------
ERP Evaluation: Final Report
August 2009
APPENDIX D: DELAWARE COST SUMMARY
PROJECT PHASE
TASK
TASK DESCRIPTION
# STAFF/
#DAYS PER STAFF
# OF SHOPS IN STATE: 152
# OF 4
# OF OF ERP: 1
Design and Development
Grant Submission
Planning and Workbook Development
Prepare and submit grant request to EPA for State Innovation
Grant
Coordinate with consultants, DNREC staff member, and auto
body sector for implementation and development of program
Meeting to explain ERP and get buy-in from other program staff
and managers
Kick-off meeting
Gathering existing materials prior to ERP and finalize ERP
materials
Confirm list of shops
Meeting with auto body shops to gather feedback on program
materials
Follow-up with contractors and auto body shops to implement
suggestions from shops
1/3
1/10
14/1
6/2
6/5
1/2
6/1
1/1
Total Staff Days: 78 days
Initial Implementation
Workshop
Inspections
Mailings
Prepare for workshop
Conduct workshop
Inspector training
Conduct baseline inspections
Conduct follow-up visits for inspector checklists
Draft letter explaining ERP
Mail workbooks
6/3
6/4
6/1
6/12
6/12
1/1
1/1
D-1
-------
ERP Evaluation: Final Report
August 2009
PROJECT PHASE
TASK
Compliance Assistance
TASK DESCRIPTION
Draft and mail workshop notices
Draft and mail reminder postcards for deadline to submit self-
certification forms
Fielding phone calls from auto body shops with questions about
ERP (over 90-day period - more than 50 calls)
Conduct compliance assistance visits of pollution prevention
audits to auto body shops requesting assistance (over 27 shop
visits)
# STAFF/
#DAYS PER STAFF
1/1
1/1
2/2
2/10
Total Staff Days: 220 days
Program Follow-Up/Ongoing
Support
Data Entry and Self -Certification
Submissions
Website
Analysis
Review self-certification forms and make follow-up phone calls
Enter inspector checklists into database
Enter self-certification forms into database
Create website for ERP
Update website as needed
Conduct statistical analysis of data
Collect and prepare information for quarterly reports and final
reports to EPA for grant commitment as well as internal
reporting of project
1/5
2/3
2/3
2/1
2/1
2/3
1/5
Total Staff Days: 32 days
GRAND TOTAL: 330 days
D-2
-------
ERP Evaluation: Final Report August 2009
APPENDIX E: STATE PROGRAM STAFF INTERVIEW GUIDE
INTERVIEW QUESTIONS
ERP Development and Status
1. To begin, what factors encouraged your state to develop an ERP program for the auto
body sector?
2. What did your state hope to achieve by developing the program? What problems or
issues was the ERP designed to address?
3. Would your state have taken any action regarding the auto body sector if it hadn't
implemented ERP? If so, what was the alternative (including no action)?
4. What is the current status of your ERP program?
Experience with ERP
5. What did you learn over the course of developing and implementing your ERP?
What aspects of the program worked well? What aspects of the program presented
challenges?
6. What do you consider are/were the primary challenges in implementing your ERP
program?
7. Did you make any modifications to your program as it unfolded?
8. What factors influenced your decision to continue/not continue/modify (based on
previous responses) your program?
ERP Results
9. What are the most important things your ERP achieved?
10. Is there anything you hoped your ERP would achieve, but didn't? Please describe.
11. Overall, do you consider your ERP to be a success? Why or why not?
12. What factors do you believe may have influenced your success? Please consider both
elements of the ERP itself, as well as external factors such as changes in agency
budgets, changes in regulations, and changes in industry characteristics.
13. Considering the alternative approach that your state might have used with the auto
body sector had it not implemented ERP (per response to Question 3 above), do you
E-1
-------
ERP Evaluation: Final Report August 2009
think ERP required more or less state resources to implement than the alternative
approach to this sector? Please describe.
14. Do you think ERP generated more or less environmental/health results (e.g., better
compliance with requirements, improved worker health, more adoption of pollution
prevention practices) than the alternative approach would have?
15. Overall, what do you see as the advantages and disadvantages of ERP vs. other
approaches for reaching auto body shops, generating environmental and worker
health outcomes, and achieving cost-effective results in the auto body sector?
Details on Resources Invested by the State in ERP
16. In total, what level of staff time was required for the following stages of the
program? (Ideally we would like to measure this in terms of Full Time Equivalents
(FTEs), but it may be easier to think about this in terms of the number of people who
worked on the program, for what percentage of their time, over what period of time.)
a. Design and Development:
b. Initial Implementation:
c. Program Follow-Up/Ongoing Support:
17. Did you use any contractor services to design and implement your program? If so,
what did the contractor do? How much did this cost?
18. What would you estimate as the direct costs (non-labor costs) of designing and
implementing your program, for example, printing and mailing outreach materials?
19. Did you invest other resources in the program (e.g., intern time, EPA grants or
resources, partnerships with other agencies)?
20. Did the resources your state invested in ERP increase or decrease over the course of
ERP development and implementation (e.g., were there relatively high upfront costs,
and lower resources required after that; were costs constant over time; or did costs
increase as the program got under way)?
Future of ERP for this Sector
21. What challenges/opportunities do you see for application of ERP to the auto body
sector in your state in the future?
E-2
-------
ERP Evaluation: Final Report August 2009
22. If you were to advise another state planning to implement ERP for the auto body
sector, what elements would you say are needed to make the project a success?
23. Do you have any other comments that you'd like to share about your ERP?
E-3
-------
ERP Evaluation: Final Report August 2009
APPENDIX F: AUTO BODY REPRESENTATIVES INTERVIEW GUIDE
INTERVIEW QUESTIONS
1. Prior to ERP, had you had any interactions with environmental regulators in your
state? If so, please describe.
2. How did your shop participate in ERP (e.g., attending a workshop, reading a
workbook, filling out a self-certification form, creating a return-to compliance plan,
etc.)?
3. How much time did your shop spend on the following activities:
a. Compliance assistance workshops:
b. Reading compliance assistance materials:
c. Completing self-certification forms:
d. Completing return-to compliance forms:
e. Other ERP-related activities:
4. Other than the time you and your employees spent, did your shop spend any
additional resources to participate in ERP? (Note, we are not trying to account for
any business practices or equipment that you changed to come into compliance).
5. How did participating in ERP change what you do at your shop?
6. Has your company seen any cost savings as a result of the changes you made during
your involvement in the ERP program?
7. Do you think that that ERP program was designed to be sensitive to your operations
and cost structures? In other words, did the program ask you to make changes that
were not cost effective for you from a business standpoint?
8. Did you start implementing any beyond compliance behaviors or pollution
prevention activities after your participation in the ERP program?
9. Has your company seen any cost savings as a result of these additional changes?
10. What aspects of the ERP do you think were well managed by the state?
11. What aspects of the ERP do you think were not well managed by the state?
12. What suggestions would you have for improving your state's ERP?
F-1
-------
ERP Evaluation: Final Report August 2009
APPENDIX G: EPA STAFF INTERVIEW GUIDE
INTERVIEW QUESTIONS
1. What do you consider are the primary implementation challenges often faced during
the design and implementation of an ERP program?
2. What factors do you think influence the success or failure of an ERP?
3. What factors do you think influence states' decision to continue/not continue/modify
their program?
4. What does a successful ERP look like to you?
5. What kind of outcomes would you expect to see with a successful ERP?
6. Are there any common problems you have seen with the design and/or
implementation of ERPs?
7. What challenges do you see in the future management of ERPs?
8. Why do you think an ERP works/does not work for a state?
9. What are the factors that were outside of the ERP that you believe influence the
outcome of a program?
10. What does the ideal candidate (state and business sector) for an ERP look like?
11. What kind of state should avoid an ERP?
12. Are there any business sectors for which ERP would not be appropriate?
G-1
------- |