EPA100-R-07-009
December 2007
Implementing the
Environmental
Results Program
NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
-------
Table of Contents
1 ERP Addresses Small Sources of Pollution Efficiently and Effectively,
1.1 Small Sources Present a Large Challenge
1.3 More States Are Implementing ERP, Delivering Results
2 How Does ERP Work?
2.4 Collect Facility Self-Certification Forms.
: statistical iv oast
Focus on Measures That Summarize Performance...
Environmental Business Practice Indicators (EBPIs).
The Role of Statistics in Decision-Making
Other Factors in Assessing Performance
3 ERP Results Consistently Show Improvement
3.2 "Sector Snapshots" Show Progress, Sustained Performance.
uggest Im
Printers
4 Agencies May Use Resources More Effectively With ERP.
4.1 States Report Enhanced Targeting Capabilities
4.2 ERP May Reduce the Need for Enforcement
4.3 ERP Can Complement or Replace Permitting
4.5 ERP's Results Can Help Meet Demands for Better Measurement
4.6 Many Businesses Value ERP as Fair and Helpful
4.7 ERP Offers the Public Visible Improvements and Broader Accountability
Environmental Performance Under ERP Compared to "Inspect Everyone" Appro.
ERP Resource Use Compared to "Inspect Everyone" Approach
5 ERP's Future Promises Improvement, Experimentation, and Growth,
5.1 More States Work to Produce Results
5.3 Agencies Apply ERP Tools in Innovative Ways
Applying the ERP Tools in New Ways
Alternative to New or More Stringent Regulation
Impaired Water Bodies and Watersheds
5.4 States Convene an ERP Consortium
5.5 EPA Plans to Further Explore Questions Raised by ERP .
-------
ERP Addresses Small Sources of Pollution
Efficiently and Effectively
The Environmental Results Program (ERP) is an integrated approach to addressing environmental
problems associated with various business sectors and other groups having large numbers of small
facilities. While individual facilities within these groups may release
small amounts of pollution, the aggregate impact can be significant.
The U.S. Environmental Protection Agency's (EPA's) National Center
for Environmental Innovation (NCEI) has prepared this report in order
to update the story of ERP, the implementation of which has grown
substantially since Massachusetts first developed it 10 years ago. This
report identifies the many states now using ERP and the environmental
problems they are seeking to address, describes the results and
benefits state ERPs are generating, and discusses some of the new
directions being explored within the ERP community.
"ERP has thrived in
Massachusetts because
it delivers real results: a
cleaner environment, a
better business climate,
and more efficient
government."
— Robert Colledge, Former
Commissioner of Massachusetts'
Department of Environmental Protection1
1.1 Small Sources Present a Large Challenge
Over the past 30 years, U.S. environmental policy has
achieved a great deal of success in improving the quality
of air, water, and land in the United States, especially
with regard to many of the largest sources of pollution.
Increasingly, however, environmental agencies are seeing
critical problems to which multitudes of small sources
of pollution are substantial contributors. For example,
hazardous air pollutants released by dry cleaners and
auto body shops can significantly degrade air quality.
Likewise, leaks from petroleum underground storage
tanks (USTs) and uncontrolled runoff from animal
feedlots, parking lots, and auto salvage yards can worsen
the quality of many water bodies and sources of drinking
water. Such polluting facilities can number in the tens
of thousands nationally. For instance, the United States
has over 130,000 retail gas stations,2 the vast majority of
which are likely to have underground storage tanks.
Consequently, while each small source of pollution may
pose a modest threat, these sources in sum can represent
as important an environmental problem as many major
industrial facilities. In addition, regulatory agencies
typically struggle to both identify all such facilities and
assess their performance. Also, such facilities often
lack sufficient awareness or technical expertise to meet
their regulatory responsibilities and to minimize their
environmental impacts.
Many states are finding that the conventional tools of
permitting, inspections, enforcement, and compliance
assurance may not always be well-suited to small
sources of pollution when applied in a traditional
manner. Resource constraints increasingly confronting
most environmental agencies—often in core regulatory
programs—are driving states to seek more efficient and
sustainable approaches that integrate traditional tools
and other strategies.
-------
1.2 The ERP System
Integrates Several Tools
Increasingly, states report ERP to be a cost-effective
approach to addressing the environmental problems
described previously—an approach that produces
objective data measuring overall sector performance over
time.
As Figure 1 (right) illustrates, ERP is an integrated system
of:
A Plain-language compliance assistance that promotes
pollution prevention (P2);
A Facility self-assessment and self-certification;
A Agency inspections to assess performance; and
A Statistically based performance measurement.
In ERP, regulators also typically conduct a comprehensive
facility inventory and undertake targeted enforcement
actions. The elements of ERP can work together to improve
performance across a business sector or other group of
pollution sources, while deploying government resources
strategically and efficiently.
Figure 1. ERP: Interlocking Tools,
Integrated System
COMPLIANCE
ASSISTANCE
SELF-CERTIFICATION
"[ERP] complements Minnesota's
ongoing efforts to partner with
smaller-scale animal feeding
operations in a way that focuses
on results. We are pleased
with the efforts of producers,
who have worked to learn
more environmentally friendly
business practices that may be
more cost effective as well. With
the success witnessed in our
pilot project, we look forward
to broadening the program's
application."
Brad Moore, C
1.3 More States Are Implementing ERP,
Delivering Results
In 1997, the Massachusetts Department of Environmental
Protection (DEP) first began applying ERP in the dry cleaner
and photo processor sectors. Now, 18 states in eight of EPA's
10 regions have developed or are implementing at least one
ERP. These initiatives aim to address challenging environmental
issues in 11 sectors/groups. (See Table 1, next page.) ERP states
have also banded together with EPA to form a new States ERP
Consortium, an association that aims to promote and facilitate
the use of ERP tools, when appropriate, to solve pressing
environmental problems. (See Section 5.4 of this report for more
information on the Consortium.)
EPA has actively supported the diffusion of ERP across states
since EPA's Innovation Action Council (IAC) endorsed the
approach for "scale up" in 2000.3 In making its decision, the
IAC considered at least three factors: documented evidence
of performance improvements in Massachusetts' first years of
ERP; a favorable evaluation of the initiative by the National
Academy of Public Administration (NAPA); and the significant,
cumulative environmental threat that can be posed by large
groups of small pollution sources, such as in the sectors where
-------
ERP is now being applied. ERP shows high potential
for cost-effectively achieving results with these small
entities, which historically are under-regulated—in
part because their large numbers can seem daunting to
tackle with traditional approaches. For instance, several
states have developed or are planning ERPs for one
such group, auto body shops—which number more
than 30,000 nationwide and are associated with serious
environmental and health impacts.4
Six states have now completed measuring the results
of at least one ERP: Delaware, Florida, Massachusetts,
Maryland, Maine, and Rhode Island. These states have
seen initial average performance improvements of 5 to
30 percentage points on measures tracking the most
important compliance and best management practices,
across environmental media.5 These improvements
have been observed in just the first cycle of ERP
inspections, compliance assistance, self-certification,
and performance measurement. Results from longer-term
ERP efforts show those initial performance improvements
increasing or being maintained over time. The box below
discusses some of the reasons regulators believe ERP
may help improve the performance of sectors where it is
applied, and points readers to more detailed information
on ERP performance data.
Table 1. ERP States, by Sector/Group
Sector/Group
Animal Feedlots (Small)
Auto Salvage Yards
Auto Body Shops
Auto Repair Shops
Dry Cleaners
Gas Stations
Oil & Gas Extraction Facilities
Photo Processors
Printers
Stormwater Dischargers
Underground Injection Wells
ERP States
MN
IN,RI
DE,MD*,ME,NY, RI,WA
FL*, MD*
MA, Ml, NH, NV
RI,VA,VT
LA
MA
MA, NY,WI
ME, Rl
IL
*No longer implementing ERR6 Note that MD had one ERP that covered both
auto body and auto repair.
Evidence of performance improvements reported by
ERP states to date has led EPA to endorse ERP and to
support its state partners. EPA has provided support
through technical assistance, $2.9 million in startup
grant funding, and, in some cases, the flexibility to use
resources that otherwise might be dedicated to more
traditional oversight programs.
Why Do ERP's Performance Improvements Happen?
The six states that have completed at least one ERP have seen average sector improvements of 5 to 30
percentage points for the most important indicators of environmental performance—after just one round of
inspections and self-certification. Why?
Regulators believe that ERP's mix of compliance assistance, self-certification, and agency verification drives
facilities to hold themselves more accountable and gives facilities the capability and incentive to improve
performance. Further, a recent extensive evaluation of the Massachusetts dry cleaners ERP suggests that
the cooperative ERP approach helps engender trust vis-a-vis industry, increasing the willingness of facility
owners and operators to proactively improve their environmental performance.7
For a more detailed example discussing the improvements realized by one state, see the case study discussing
Rhode Island's experience with ERP (next page). For substantial detail on ERP results to date for all six states,
see Section 3 of this report. The appendix to this report, published separately, provides additional performance
data, a detailed list of sources, and explanations ofEPA's data verification and analysis approach for key
analyses conducted by EPA for this report.
-------
Rhode Island Successfully Adapts ERP Approach
Auto Body Shops Present a Problem. In 2002, auto body shops in Rhode Island represented a serious
environmental and public health concern. The sector had historically been ignored by regulators even
though many shops were using highly hazardous methylene chloride paint strippers, as well as releasing
toxic sanding dust and volatile organic compounds (VOCs) to the environment. Further, research by staff
at the state's Department of Environmental Management (DEM) suggested that lead paint dust had been
carried home to workers' families on contaminated work clothes, and that the average level of lead in
auto body workers' blood was greater than five times the national average.8
ERP Offers a Potential Solution. Solving this problem with the usual approaches was not an option,
because of resource constraints. Rhode Island's DEM and business leaders in the sector sawthe
answer in the efficiency of Massachusetts' ERP approach, modified for Rhode Island's circumstances.
While Massachusetts typically requires businesses to submit self-certification forms, Rhode Island
opted to design a voluntary self-certification program. Whether or not they participated, shops needed
to comply with standards. To encourage participation, DEM offered compliance assistance and the
opportunity to come into compliance without fear of enforcement action. Further, non-participants were
informed they were more likely to be targeted for inspection.
Many Facilities Sign Up. Responding to these incentives, and demonstrating their commitment to
resolving worker and community concerns, 47% of the state's 367 facilities submitted self-certification
forms.9
"I'm Doing It Wrong Over Here." Twenty percent of the state's shops
indicated they were out of compliance—particularly with issues
impacting worker health and safety—and submitted plans to address
their problems.10 The state requires that all plans be signed by
responsible company officials committing to take specific actions by
specific dates.
Performance Improves across the Sector. DEM audited random
samples of a//facilities—both certifiers and non-certifiers—before
and after the initiative. Inspectors observed sector-wide improvemen
averaging 21 percentage points across all indicators. (See Section 31
more information.)
DEM Finds More ERP Opportunities. DEM has since expanded its ERP activities to include a mandatory
certification program for underground storage tank (LIST) facilities and voluntary certification programs
for auto salvage yards and small construction sites. The Department also completed a second round of
auto body self-certification in 2006 and conducted post-certification random inspections in late 2007.
Analytical results are expected in 2008.
-------
How Does ERP Work?
ERP combines innovative approaches with
conventional regulatory tools in a rigorous and
repeating process. Synergy among the tools can
help improve environmental performance at each
stage. States have developed some variations on
ERP, but the general approach, tools, and steps in
the process remain the same. Figure 2 (right) is a
stepwise diagram of the major tasks in a typical
ERP. This section describes the ERP process in
more detail, and demonstrates how each step is
intended to help support those that follow.
2.1 Conduct a Comprehensive
Inventory of Facilities
Often, states have not developed a comprehensive
inventory of small-source polluters within the various
sectors that can be effectively targeted by ERP. ERP
implementers strive to better identify the universe of
sources, in order to reach as many businesses as possible
with ERP and to accurately assess the performance of the
sector as a whole. ERP states use a variety of public and
private data sources and data collection approaches to
compile much more complete lists of relevant facilities
than are typically maintained.
Once completed, the ERP inventory often shows that
only a fraction of the businesses in a sector were on
a state's "radar screen." For instance, Massachusetts'
statewide ERP inventories increased the numbers of
facilities in each of DEP's sector databases by about 4 to
20 times. The number of printers in DEP's database grew
from about 250 facilities to around 1,000; its records of
photo processors rose from approximately 1 00 to about
500; and its dry cleaners listings expanded from about
30 facilities to around 600.11
ERP inventories do not always include so many facilities,
such as when ERP is implemented as a pilot program in
a portion of a state. In fact, experience shows that ERP's
Figure 2. A Typical ERP Cycle
Step 1: Inventory. Identify the myriad small facilities
thatare sources of pollution, many of which are
often unknown to regulators.
Step 2: Statistical Baseline Inspections. Conduct
random inspections to accurately measure existing
environmental performance and focus outreach on
the biggest problems.
Step 3: Compliance Assistance. Work with trade
associations to create and provide plain-language,
user-friendly assistance that improves compliance
and promotes pollution prevention.
Step 4: Self-Certification. Facilities conduct self-
assessments using a detailed checklist closely
linked to assistance materials. Responsible officials
certify to their facilities' environmental performance
on each item. If necessary, they submit plans to
return to compliance.
Step 5: Targeted Follow-Up. Identify potential
problem facilities via certification analysis, and target
them for inspections, correspondence, or phone
calls. Provide assistance and/or initiate enforcement,
as needed.
Step 6: Statistical Post-Certification Inspections.
Conduct random inspections to accurately
estimate performance changes and verify facility
certifications.
Step 7: Informed Decision-Making. Assess
performance data and consider whether to adjust
compliance assistance or other strategies directed
atthe sector or, if sufficient progress has been made
over time, target resources elsewhere.
benefits can be reaped on a more local level with smaller
populations of facilities. For example, Maryland's ERP,
discussed later in this report, demonstrated performance
improvements while focusing solely on a single
neighborhood in Baltimore.
o
CD
a.
3
o
21
C/3
CD
CD
CD
-------
2.2 Conduct Statistically Based Baseline Inspections
Once the facilities to be targeted by ERP have been
identified, a state will randomly select a sample of those
facilities. Next, inspectors assess performance at each of
these facilities using detailed checklists. Using the data
from these inspections, states can accurately estimate the
sector/group's performance level before the beginning
of compliance assistance and certification. For instance,
agencies typically calculate the percentage of facilities
achieving key compliance and pollution prevention
practices. This information can help agencies identify
problem areas to highlight during compliance assistance,
and serve as a baseline for understanding whether sector/
group performance changes overtime.
ERP's random sampling approach is intended to help
ensure that the set of facilities visited is representative of
the overall sector or group being targeted. This random
sampling provides agencies with the foundation for
statistically calculating how confident they can be that
the findings from these inspections shed light on overall
sector/group performance. By contrast, many other
policy and measurement approaches collect data from
2.3 Provide Plain-Language Compliance
Assistance Across Environmental Media
Many business owners want to comply with environmental regulations
to protect their workers and communities, but they may not always
know how—or even where—to start. ERP tries to help by reaching out to
facilities with fact sheets, workbooks, workshops, and even videos.
Agencies actively work with key members of the business community,
such as trade associations, to ensure outreach approaches are as helpful
as possible. The best ERP materials are easy to use and intuitive to facility
operators, translating regulatory requirements into plain language. For
instance, ERP workbooks are written in terms the operators are familiar
with, not legalistic regulatory language. They simply tell businesses
what they need to do and how to complete the certification form.
The workbooks typically provide information on requirements for all
environmental media (e.g., air, water, and waste). They also discuss
pollution prevention practices that are good for the environment, worker
health and safety, and/or businesses' bottom line.
samples of facilities that may not be representative,
leaving agencies with greater uncertainty about overall
sector/group performance. (Sections 2.6 and 2.7 discuss
the role of statistics in ERP in greater detail.)
To date, the number of baseline inspections conducted
by states ranges from 35 to more than 250, with most
states conducting 35 to 50 site visits. The precise number
of visits depends on how the state chooses to balance
the resources available for site visits with the need for
increased confidence in the results—because doing more
inspections means having greater certainty in the results.
"Small-busin
operators want to do the
right thing, if we only
know how. We need it
to be simple and easy to
understand. We want to
protect our [real estate]
investment and protect
the environment, too. We
have grandchildren who
we want to enjoy the same
environment as we have.
ERP looks good...."
— George Cart
Auto Parts in PC
o Beach, Nor
-------
2.4 Collect Facility Self-Certification Forms
Reflecting ERP's integrated nature, its compliance
assistance workbooks help facilities conduct
standardized self-assessments of their operations on a
regular basis (usually annually). Facility self-assessment
checklists are typically integrated into certification
forms and correspond closely with the checklists used
by agency inspectors. This correlation is intended to
better enable facilities to address an agency's concerns
and priorities, and to empower the agency to measure
progress and track the accuracy of certification forms.
Following a self-assessment, a responsible official for
each facility signs and submits a certification form
that specifies whether or not the facility is meeting the
requirements and best practices laid out by the agency.
In some ERPs, the submission of certification forms
is mandatory. In others, like the Rhode Island auto
body ERP discussed in the case study box in Section 1,
certification is voluntary. Whatever the approach chosen
by states, all facilities are still expected to comply with
applicable regulatory requirements.
If facilities indicate any noncompliance, they must
also submit return-to-compliance plans. In return-to-
compliance plans, facilities indicate how and within
what time frame they will comply with regulatory
requirements. While agencies retain the authority to take
enforcement action in all cases, they generally refrain
from enforcement against good-faith actors who self-
report problems and act promptly to correct violations.
The next page explains how ERP states consider and
utilize the self-reported data from ERP.
ERP Workbooks Connect to Self-Certification Forms
5.4 Used Coolant/Antifreeze
CAPP QUESTION
REGULATORY REQUIREMENTS
There are two options for management of waste coolant/antifreeze:
Option A - The preferred option is to recycle.
Option B - If you do not recycle your used coolant/antifreeze, IT WILL BE REGULATED AS
HAZARDOUS WASTE unless you are able to prove, through periodk laboratory testing,
that it is not hazardous waste (see Appendix IV), or you must follow all regulatory
Empl
|cftpp7B s | CAPP Compliance Certification Form— continued
II. Waste Batteries
waste ear batteries' D Yes
IS Does your shop comply with
batteries
Refer to page 44 in the Workbook.
III. Used Oil and Oil Filters
manage eerratlner* and tanks ef used ell'
a containers of uiad oil flltars rj yes
IV. Used Antifreeze
DNo
disposing of used oil to a septic tank, storm dram, surface m^[Ma
DNo
2 I Does your shop meet all requirements for reeling and/or Q Yes D No
disputing of used antifreeze' fiw ^ a Kei
Refer to page 53 in the Workbook. Compliance I
DEP FORM 61-73 0.900 (7) (b) 10-10-2002
A page from Florida's ERP workbook (above, left) explains the potential environmental and health hazards of used antifreeze,
and what steps auto repair shops must take to comply with state regulations for handling the substance. A blue arrow at the top
of the page helps indicate that aII facilities must answer question 20 on the self-certification form, as to whether they generate
used antifreeze. The corresponding page in the self-certification form (above, right) shows question 20 under a header related to
used antifreeze. Shops indicating thatthey generate used antifreeze are instructed to answer a subsequent question about their
compliance status, and they are told to referto pages 52 and 53 of the workbook.
-------
s the Role of Self-Reported Information?
Newcomers to the ERP concept often express concerns about letting facilities identify their own compliance levels. Such
concerns generally appear founded on a misconception that ERP primarily relies upon self-reported data to determine
group performance, as well as a correct understanding of the historic difficulty agencies have had in verifying self-
reported data.13 To the contrary, ERP states typically use self-reported data in low-risk ways that can add substantial value
to efforts to improve sector performance. ERP states are comfortable with and confident about the certification process for
many reasons:
A Sector-wide compliance assessments are based on agency site visits, not facility-reported data. Statistical analysis
of data from inspections has allowed states to demonstrate objectively that sector performance is improving. (See
Section 3 for performance data.)
A Random and targeted inspections can enable verification and (when necessary) enforcement. When an inspector
visits a facility that has self-certified, he or she can assess the veracity of its certification form andean follow up with
appropriate enforcement action if certification responses appear to have been falsified.
A Surprising numbers of facilities report noncompliance. In the first cycle of an ERP, up to 34% of facilities have self-
reported that they are out of compliance on at least one issue.14 Further, ERP states typically follow up with each facility
to ensure its plan to return to compliance has been implemented.
A Performance may be improving even if some certification data are inaccurate. Levels of certification accuracy appear
to vary across ERPs, while inspector observations consistently indicate that group performance has improved after ERP
implementation. (Section 3 of this report has substantial detail on objectively measured performance improvements.)
For instance, Massachusetts DEP has seen relatively accurate certification, with sector-wide agreement between
inspector and facility responses as much as 86% of the time.15 Florida's Department of Environmental Protection
(DEP) looked at certification accuracy in a different way. DEP inspectors found that 74% of randomly sampled auto
repair shops were out of compliance on at least one issue on which the facility had previously indicated compliance
on its first self-certification form.16 Despite this inconsistency, DEP inspectors' observations also indicate that the
sector's performance on key indicators improved by an average of 7 percentage points after self-certification, and
that the sector's compliance rate also improved. (See Section 3 for details.) Further, none of the self-certification
inconsistencies involved issues of significant noncompliance.17
A Certification accuracy appears to improve over time. As facilities better understand their obligations and agencies
better communicate their concerns, certification accuracy appears to increase in later certification cycles. For
example, Florida found that the proportion of inspected facilities with completely accurate certification forms increased
by 14 percentage points from the first round of certification to the second.18
A States choose when to trust certification data. Based on inspector observations, professional judgment, and common
sense, agencies are able to decide which kinds of certification data are likely to be accurate and can serve as a basis
for decision-making.
A Inaccuracies can be used to improve outcomes. Some inaccuracy is to be expected, and can even be helpful.
As discussed in the next section, inaccuracies identified during desk analysis of certification data can provide an
opportunity for agencies to find "needles in the haystack," targeting facilities that need compliance assistance or
enforcement.
-------
2.5 Target Compliance Assistance and Enforcement Actions
The self-certification process is not just for helping
facilities improve their performance, but also for
identifying those facilities most in need of further
technical assistance or possible enforcement action.
ERP states can use the large quantity of certification
data made available through the process as one tool to
identify "red flag" facilities—i.e., those which require
follow-up ranging from a simple helpful phone call to a
formal enforcement action.
Figure 3 (right) illustrates how facility return-to-
compliance plans and self-certification "red flags"
might help a state locate many of the scattered facilities
needing the most attention—a targeting process that
might not be possible without ERP certification data.
Widespread inaccuracies with respect to certain self-
certification questions can suggest communication
problems that need to be addressed in future rounds of
compliance assistance and certification.
Such follow-up can be a cost-effective way for the
state agency to identify and improve the performance
of lagging facilities. Follow-up also demonstrates that
an agency is paying attention to self-certification data,
which helps assure facilities that their submissions are
not just a paperwork exercise. This assurance, in turn,
encourages facilities to take great care in preparing
their self-certification forms. Without ERP, this kind of
adaptive management might be more difficult. Section
4.1 provides more information on such targeting, in the
context of discussing ERP's cost-effectiveness.
Figure 3. Needles in a Haystack?
ERP Data Can Help Find Problem Facilities
Non-compliant facilities identified through return-to-compliance plans
Non-compliantfacilities identified through self-certification mistakes
All other facilities in the sector
2.6 Conduct Statistically Based Post-Certification Inspections
The ERP process next calls for more random inspections at another representative sample of all facilities. This round of
inspections serves two primary purposes—one that relates to the actual facilities that are visited, and one that relates to
all facilities in the sector.
First, inspectors are able to observe how well the facilities visited are performing. These findings can be compared
to the performance reported on each participating facility's certification form. Agencies typically undertake standard
enforcement approaches when they encounter violations, particularly those associated with false self-certification.
Second, just like with the baseline inspections before certification, the random sampling approach allows agencies
to use statistical methods to draw inferences about all facilities being targeted by ERP—not just those facilities visited
by inspectors. Consequently, regulators can better understand how well the entire population is performing and how
accurately facilities within the population are certifying. Further, regulators can compare estimates of the population's
post-certification performance to estimates of its baseline performance, and understand how certain they can be that
any observed changes in performance have actually occurred.
-------
2.7 Use Data for
Informed Decision-Making
Each round of random inspections provides substantial
information about sector performance at that particular
point in time. Agencies analyze these data to address
several key questions: the extent to which performance
has improved; whether adequate performance has been
achieved and, if so, whether it has been maintained over
time; and areas in which the sector must improve.
The wealth of data ERP creates to help answer these
questions can be a double-edged sword. These data
can present an opportunity to better understand
environmental problems facing regulatory agencies, but
they can also seem overwhelming if the agency does not
have a clear strategy and a rigorous approach for data
assessment. ERP states' response to this conundrum is
typically to first concentrate data analysis on indicators
that summarize group performance with regard to agency
priorities—pursuing issues in finer detail if data indicate
a closer look is merited. ERP states also apply principles
of statistics, as well as their best professional judgment, to
gauge how confidently to move ahead.
Focus on Measures That Summarize Performance.
ERP states collect detailed data in their inspections,
but regulators don't give equal weight to every single
compliance item. Instead, they focus measurement
activities on what they consider to be the most important
aspects of facility and sector performance. Most
states focus analytical efforts initially on the building
blocks of ERP measurement, Environmental Business
Practice Indicators (EBPIs, described further in the next
paragraph). States can also calculate concise figures that
provide a "snapshot" of sector or group performance,
such as a traditional compliance rate or what can be
called a "group compliance score," a new measure
created by Massachusetts. When feasible, regulators can
also estimate changes in public health or environmental
conditions, such as pollution reductions. Section 3
presents state data on each of these kinds of measures:
EBPIs, "sector snapshots," and measures of environmental
conditions.
Examples of Performance
Improvements on Environmental
Business Practice Indicators
(EBPIs)
A. Do auto body shops dispose of hazardous waste
properly? In Delaware, compliance with hazardous
waste regulations grew from 66% to 91% in one year.
B. Do auto repair shops illegally discharge polluted
wastewater? \r\ its first year of ERP, Florida saw a drop
of 12 percentage points in the incidence of sampled
facilities with illegal discharges.
C. Are dry cleaners properly checking lor equipment
leaks? After the first two rounds of annual self-
certification, Massachusetts observed an increase of
33 percentage points in the use of this key emission-
reduction practice.19
D. Do auto body shops utilize solvents that are low
in volatile organic compounds (VOCs)?Mame's
Department of Environmental Protection (DEP) saw
an increase of 48 percentage points in the use of
environmentally friendly solvents, from 49% to 97% in
one year.
E. To what extent are neighborhood car repair
operations following preferred painting procedures?
Maryland sawthe number of facilities using emission-
reducing painting practices jump from 40% in 2002 to
62% in 2004.
F. Do auto body shops use methylene chloride, a
dangerous paint-stripping chemical? Before ERP
certification, Rhode Island found 33% of facilities were
using methylene chloride; after, they found only 5%
choosing to use it.
Notes: Percentages are of relevant, randomly sampled facilities. Results
from Delaware, Florida, Maine, Massachusetts and Rhode Island are
"statistically significant" at a 95% confidence level, meaning we can be
confidentthat performance changed among all facilities targeted by ERP.
Maryland's result is not significant, because of particularly small sample
sizes. Except where noted above, changes were observed in the first ERP
cycle.
Sources: Unless indicated by endnote, data sources for this box are the
same as for Table 2 in this Report. Sources for Table 2 are presented in the
appendix, Section 1.
-------
Environmental Business Practice Indicators (EBPIs).
For many sectors, there can be well over 100 questions
on a comprehensive ERP inspection form or facility self-
audit checklist, covering both major and minor issues.
This level of detail helps agencies work with individual
facilities to thoroughly improve compliance, but can be
overwhelming to regulators looking at the sector as a
whole, because individual data points can number in the
tens of thousands. Concentrating on the highest-priority
concerns in the sector can conserve analytical resources
and focus management attention. EBPIs can help do
this because they summarize performance data on a
subset of checklist questions regarding what an agency
considers to be the most important compliance and best
management practices.
The sidebar (previous page) gives examples of actual
EBPIs chosen by states, and performance improvements
shown by those indicators. Most ERP states choose
between 10 and 30 EBPIs, depending on the sector
and the state's needs. States typically choose EBPIs that
track facility and sector behavior across all relevant
environmental media categories (e.g., air, water, and
waste). EBPIs generally cover both compliance and
"beyond-compliance" activities, and sometimes practices
related to occupational safety and health.
Figure 4 (right) illustrates the two typical types of EBPIs,
and how they fit with the questions on a comprehensive
checklist. The first type, represented by EBPIs 11 through
13 in the figure, can be called a "roll-up" EBPI, because
it rolls up, or summarizes, the answers to several sub-
questions. For a real-world example, look at EBPI "E"
in the sidebar on the previous page. This EBPI from
Maryland summarizes inspector responses on several
different checklist questions, all of which involve
painting practices that can reduce air emissions from
auto body shops.
The second type of EBPI, represented by EBPI 14 in
Figure 4, can be called a stand-alone EBPI. A stand-alone
EBPI is linked to one important question on the facility
checklist. This kind of EBPI can indicate performance on
a single, salient issue, or it can be used as a "leading"
indicator of performance on other questions that are not
tracked. For instance, EBPI "F" in the sidebar (previous
page) is a stand-alone indicator that examines Rhode
Island auto body shops' use of methylene chloride, a
greenhouse gas and an extremely important worker
health and safety issue.
Figure 4. The Link Between EBPIs
and ERP Checklist Questions
11
-------
The Role of Statistics in Decision-Making. From a
measurement perspective, the ideal way to understand
how a population of facilities is performing is to take
a census of that population, visiting all facilities over a
relatively limited time frame in order to evaluate their
performance. Doing so can provide regulators with little
doubt that they are basing decisions on the very best
available information about the group being investigated.
Such census approaches can be resource-intensive,
however, so regulators using a variety of policy
approaches often base their decisions on data collected
from only a sample of facilities. As a result, regulators
usually are not sure how well the data from their
samples represent the whole group being investigated.
If a sample is not drawn carefully, it may not reflect
the performance of the group as a whole. For instance,
imagine an agency's sample includes a disproportionate
number of facilities that are less likely to be in
compliance than other facilities—such as facilities
with a history of complaints. In such cases, inspectors
may find the performance of sampled facilities to be
worse than typical. Using data from the sample to draw
inferences about the rate of noncompliance in the overall
population—i.e., inferring that the overall population is
performing more poorly than it actually is—might lead to
biased policy decisions.
Using statistics enables regulators to clarify how
certain they can be that data taken from a random,
yet representative, sample of facilities reflects the
performance of the group as a whole. For instance,
the box presented earlier in this section shows several
Using statistics enables regulators
to clarify how certain they can be
that data taken from a random, yet
representative, sample of facilities
reflects the performance of the
group as a whole.
EBPIs for which ERP inspectors observed "statistically
significant" changes between the baseline and post-
certification random samples. "Statistically significant"
means we can be confident that a change occurred in
the group as a whole.
How confident? Each of those changes was statistically
significant at a 95% confidence level. A 95% confidence
level means there is at most a 5% chance we would be
mistaken in saying there was a change in performance for
the group as a whole. Other observed changes that are
not statistically significant may indeed point to genuine
changes in the whole group, but we cannot be as certain
that they occurred—oftentimes because states are basing
their inferences upon small samples of the population.
Smaller sample sizes typically allow only fairly large
observed changes to be deemed "significant" at a 95%
confidence level, a very rigorous level that is used in
this report and is the most common one used by ERP
states. (Some states may choose to use a 90% confidence
level—lower, but still generally considered acceptable as
long as it meets an agency's decision-making needs.)
Other Factors in Assessing Performance. ERP states
do not solely rely upon determinations about statistical
significance and other statistical tests, however. Other
ways of looking at the data can help regulators credibly
assess the performance levels of facilities, by using
professional judgment and common sense.
For instance, certification data can provide valuable
information, even though the information is self-reported.
Return-to-compliance plans are one example. ERP
states feel it is unlikely that facilities will knowingly
claim to be out of compliance when they are actually in
compliance. Consequently, high rates of submission of
return-to-compliance plans for particular EBPIs suggest
performance improvement on those EBPIs, even if the
change in performance observed by inspectors is not
statistically significant.
Similarly, statistically significant change may not be
the best benchmark of success when performance is
already at a high level. Some states have found there are
EBPIs for which inspectors observed 1 00% compliance
among facilities before certification. In such cases of
high-performing facilities, one would not expect (or
desire) statistically significant change, because such
change could only be downward. In this case, a lack of
significant change is a sign that facilities are maintaining
their high performance level.
12
-------
ERP Results Consistently Show
Improvement
As discussed in Section 2, ERP states focus on the most important aspects of performance in three
primary ways: (1) by tracking progress on individual indicators of compliance and best practices; (2) by
compiling "sector snapshots" that summarize, in a single metric, group performance across multiple
measures; and (3) by estimating environmental outcomes, such as emissions reductions.
The individual performance metrics often vary among states—and particularly sectors—in part because
ERP provides states with the flexibility to tailor their measurement to issues of particular interest. This
variety in reporting has made the development of this report challenging and limits comparability among
ERPs. Nevertheless, shared performance improvement is evident among the six states that have completed
a full ERP cycle in one or more sectors: Delaware, Florida, Maine, Maryland, Massachusetts, and
Rhode Island. This section summarizes results from those states. (The appendix to this report, published
separately, provides more detail on the results, data sources and EPA's approach to data verification and
analysis.)
3.1 ERP Groups Improve on Priority Indicators, Across the Board
To date, all ERP states have reported seeing progress
on Environmental Business Practice Indicators (EBPIs)
across multiple regulatory or performance categories. The
observed improvements are statistically significant in many
cases.
Table 2 (next page) summarizes results from each of the
completed ERPs, showing initial average improvements
of 5 to 30 percentage points across each ERP's EBPIs. For
example, the table summarizes results observed by the
Rhode Island Department of Environmental Management
(DEM) for its auto body ERP. OEM's results were reported
in a peer-reviewed publication, American Journal of Public
Health, inMay2007.20
13
-------
DEM inspectors observed improvements on 19 of 24
performance indicators. Seven of those 19 improvements
are statistically significant at a 95% confidence level.
Improvements were observed across each of the
following categories: hazardous waste management,
air pollution control, wastewater discharge, and
worker health and safety. Overall, Rhode Island results
demonstrate an observed performance increase of 21
percentage points, when averaged across all indicators.
This is not to say that every ERP indicator shows
performance improvements. For instance, for two EBPIs,
Rhode Island inspectors observed 100% of facilities
achieving each EBPI in both rounds of inspections,
making it impossible to show improvement. DEM also
observed performance decreases associated with three
EBPIs (although none were statistically significant).
Nevertheless, when considered in aggregate, states'
experience to date demonstrates a net performance
gain across all EBPIs. In fact, the bottom line of Table
2 indicates that, for every EBPI showing decreased
performance, nearly four show improvements. And,
while states found that more than one-third of observed
performance improvements were statistically significant
at a 95% confidence level, they found none of the
observed performance decreases to be statistically
significant at that level.
Table 2. Observed Average EBPI Improvement in First ERP Self-Certification Cycle
K*J fO Irl
DE
FL
MA
MA
MA
MD
ME
Rl
Total
^^FtlrlH 111 ^^1
Auto Body*
Auto Repair
Dry Cleaners
Photo
Processors
Printers
Auto Body/
Repair*
Auto Body*
Auto Body*
BT^H^^El
EBPIs
19
17
15
8
25
5
22
24
135
, .
(# Significant*)
17(13)
13(7)
5(0)
3(1)
17(1)
4(1)
18(3)
19(7)
•MnmiwifSliTiiffl
, .
(# Significant*)
KO)
3(0)
5(0)
2(0)
6(0)
KO)
4(0)
3(0)
B3HI
Number Unchanged
• ilJllLlllUlnlMl!l!*!GB
Performance
1
1
5
3
2
0
0
2
Average Improve-
j. in *.
^1 1 1 1 M 1 1 • 1 • H 1 H H 1 1 T* 1 H ^H
Points)*
30
7
5
12
13
12
10
21
Notes:
*"# Significant" indicates the number of EBPIs, either improving or worsening, for which the observed change reflects a statistically significant difference. The signifi-
cance figures here are based upon a 95% confidence level, which means we are 95% confident that the true percentage of facilities in the population (as opposed to the
sample) achieving such EBPIs has increased.
t For each ERP, the average percentage point improvement represents a simple (not weighted (average of the percentage point changes in EBPI values after the first
round of self-certification (e.g., if 40% of facilities were in compliance with an EBPI in the baseline, with 60% in the post-certification round of inspections, the change
would be 20 percentage points; the average for an ERP is an average of all these percentage point changes, positive or negative). EBPIs for which there was no improve-
ment, but for which there was already 100% performance, were not included in the calculation of average improvement, because improvement is not possible in that
circumstance. This analysis includes both compliance and pollution prevention EBPIs.
t Each of these ERPs utilized a voluntary certification approach. However, readers should recognize that the results presented here are based upon random samples of
the entire population of facilities in the sector targeted by the ERP, not just the facilities that voluntarily submitted self-certification forms.
A detailed list of data sources and methodological notes underlying this table is available in Section 1 of the appendixto this report.
-------
What happens when states do not see progress on key
indicators? One of the potential benefits of ERP is that
its measurement system can allow states to identify
facilities lagging behind in performance, so that agencies
can target their efforts appropriately. For instance, a
key emission-reducing practice among dry cleaners
is to ensure that cleaning machines have completed
all operations before doors are open and clothes are
transferred. After Massachusetts noted that the state's
dry cleaners continued to have poor practices in this
area after the first ERP cycle, DEP communicated its
concern to the trade association, which in turn drew
the attention of facilities to the matter. The next round
of random inspections documented that performance
had improved, because more dry cleaners had installed
lockout mechanisms to prevent premature opening of
machines.21
3.2 "Sector Snapshots" Show Progress, Sustained Performance
In addition to examining individual EBPIs, states
can calculate "sector snapshots" to summarize the
performance of business sectors subject to an ERP.
These benchmarks express performance levels even
more concisely than EBPIs—communicating, in a single
metric, the extent to which businesses are achieving
compliance or other agency goals. Environmental
agencies have historically calculated a sector snapshot
called the "compliance rate" to assess performance in
regulatory programs. This section begins by describing
how the traditional compliance rate is typically
calculated, and provides an example of Florida's
observed change in compliance rates. The section also
discusses how the "group compliance score" developed
by Massachusetts can complement the traditional
compliance rate as a performance management tool.
Compliance Rates. The compliance rate is an easily
calculated summary measure of a regulated group's
compliance level. The compliance rate has been
customarily used by EPA and states to help guide agency
activities. A compliance rate can be calculated in a
few different ways. Most commonly, it is expressed
as the percentage of facilities in full compliance with
all regulatory requirements—a key goal of many
environmental agencies. Compliance rates are often
focused upon a single environmental medium—such as a
compliance rate for air regulations.
Agencies typically calculate compliance rates based on
whatever facility data they have available, which may
not be representative of the population as a whole. ERP's
random, representative sampling approach can allow
states to more confidently express compliance rates—
and their changes over time—for sectors as a whole. For
instance, Florida inspectors observed that the proportion
of what it called "straight-A" auto repair facilities—those
with no violations of any kind, across environmental
media—had risen 1 7 percentage points among randomly
sampled facilities after two rounds of self-certification.22
This improvement is statistically significant at a 95%
confidence level.
Although ERP allows for calculating a statistically
based compliance rate, states implementing ERP have
not substantially relied upon on the compliance rate
in describing the performance of their ERP groups or
sectors, for two primary reasons. First, ERP states tend
to focus more on the most important indicators of
performance (i.e., the EBPIs), arguing that doing so can
help regulators allocate resources and attention to the
highest priority issues. Some kinds of compliance rates
address this concern—focusing only on "significant
noncompliance" and counting facilities as out of
compliance only if they fail to meet one or more core
indicators of performance.
Second, compliance rates—whether indicator-focused
or not—are limited in their ability to show gradations
in performance. Because of how they are calculated,
compliance rates can only indicate the proportions of
facilities that have either perfect records or imperfect
records, and whether those proportions are changing
over time. As such, compliance rates are a valuable
measure for determining the extent to which full
compliance is being achieved. Yet, compliance rates
do not provide information on how imperfect those
imperfect facilities are. Is the average facility achieving
almost nothing, nearly everything, or something in
between? And, over time, is performance among these
imperfect facilities getting better or getting worse?
Mathematically, the traditional compliance rate cannot
answer these questions. For instance, if a facility was
achieving 10% of indicators, then improved to 90%,
the compliance rate calculation would treat the facility
as out of compliance in both cases. The performance
improvement, although substantial, would not be
reflected in the measure.
-------
Group Compliance Scores. Massachusetts has found a
way to begin to address these questions with its "group
compliance score," which is rooted in a mathematical
concept called an index. It expresses the extent to which
facilities are achieving compliance-related EBPIs, as
observed by inspectors during random visits to facilities.
A score of 80%, for instance, would mean that, on
average, each facility is achieving 80% of the indicators
that apply to it.
Massachusetts uses this score to reflect varying degrees
of compliance at individual facilities. The following
example illustrates the potential decision-making value
of the group compliance score. Imagine a situation
in which no facility visited by inspectors during the
baseline round of ERP random inspections is achieving
even a single EBPI. In this case, both the group
compliance score and the compliance rate would equal
0%. But what if inspectors observe facilities achieving an
average of 85% of all indicators after ERP certification,
but still no single facility is achieving all EBPIs? In
this case, the compliance rate would remain at 0%,
showing no facility had as yet achieved the goal of full
compliance. The group compliance score would be 85%,
complementing the compliance rate by reflecting the
substantial progress in the sector.
In practice, what has the group compliance score shown
about the performance of ERP sectors in Massachusetts?
The metric demonstrates that each sector's score is higher
than when Massachusetts began implementing ERP in
1 997. As Figure 5 (right) shows, the photo processing
sector showed a dramatic increase from 57% in 1 997 to
98% just five years later. In the dry cleaning and printing
sectors, performance started at a relatively high level,
and has been maintained or improved over time. In the
case of printers, performance has remained relatively
steady, between 80% and 90%. With dry cleaners, no
change was observed after the first round of certification,
but Massachusetts inspectors saw performance jump
higher in later years, leveling off near 100%.
Figure 5. Massachusetts Group
Compliance Scores, Over Time
1997 1998 1999 2000 2001 2002 2003
Notes:
(1) Massachusetts did not evaluate changes in group compliance scores for
statistical significance. Retroactively doing so was beyond the scope of this
report.
(2) Graph reflects most recent available data. Massachusetts has decreased
inspection frequency over time for these sectors because of their trends toward
sustaining high performance levels. The next round of random inspections is
anticipated later in 2007, for the dry cleaning sector.
(3) Dry cleaners data points: 1997,84%; 1998,84%; 2000,97%; and 2002,98%.
(4) Photo processors data points: 1997, 57 %; 1998,71%; 2000,96%; and 2002,
98%.
(5) Printers data points: 1998,82%; 1999,89%; and 2003,86%.
For data sources and analytical notes, see appendix, Section 2.
16
-------
3.3 Results Suggest Improvements in Environmental Conditions
The leading edge of state ERP efforts involves estimating
changes in environmental conditions, or outcomes,
associated with ERP initiatives. Doing so for any kind
of initiative can be quite challenging, because such
estimation requires adequate underlying data, numerous
assumptions (such as emissions rates associated with
a variety of business practices), and consideration of
extraneous factors that could impact environmental
conditions. In the face of these challenges, Massachusetts
has experimented with a variety of different approaches
for estimating environmental outcomes for each of its
three sectors. Presented below are the results for two
outcome measures, from the dry cleaning and printing
sectors, in which Massachusetts has high confidence.
Dry Cleaners. Traditional dry cleaning relies upon the
use of perchloroethylene, a toxic and volatile solvent
also known as "perc." Massachusetts dry cleaners must
report information on perc usage and waste disposal
on ERP certification forms. Using those data, the state
estimates that dry cleaners reduced average perc
waste by 28% and average perc air emissions by 32%
between the first round of self-certification in 1 997 and
the second round in 1 998. These improvements are
responsible for a 151-ton reduction in annual perc waste
and a 135-ton reduction in annual perc emissions.23
As explained on page 8, ERP measures typically do
not rely on self-reported data. However, Massachusetts
believes these data are reliable because the state does
not perceive that dry cleaners have any incentive to
falsely report—there is no penalty or reward based
upon perc usage and waste levels.24 Further, while the
state's inspectors do not currently verify self-reported
perc usage against on-site facility records, the fact that
they could do so may create a disincentive against
falsification.25
Dry cleaners in Massachusetts
reduced their annual air emissions
of perchloroethylene by an estimated
135 tons by the second round of self-
certification (1998). This amount would
be enough to fill the lungs of over 3.6
million adults.26
17
-------
Printers. Massachusetts explored a number of ways
to measure how changes in the practices of the state's
printing businesses, as measured by several different
EBPIs, could translate into changes in environmental
conditions in the state. DEP inspectors observed varied
performance over time, with EBPIs indicating positive as
well as negative changes. Only one of those estimated
outcomes is presented here, because it is based on the
only statistically significant EBPI performance change (at
a 95% confidence level) among the subset of EBPIs DEP
examined.
Massachusetts inspectors observed that, in 1 998, only 3%
of facilities they visited were utilizing ultraviolet (UV) ink,
so called because it is cured with ultraviolet light. UV ink
is environmentally preferable because it contains little
or no amount of volatile organic compounds (VOCs), a
class of pollutants that can cause respiratory problems
and often contributes to the formation of ground-level
ozone. Although UV ink emits much less (if any) of the
air pollution that can result from the use of standard
printing inks, printers are not compelled by regulation to
use the inks. However, Massachusetts has consistently
encouraged printers, through ERP, to switch to this ink as
a voluntary pollution prevention measure. By 1 999, 9%
of randomly sampled facilities were utilizing UV ink, and
in 2003, the figure had grown to 23%.
Statistical analysis of the inspector findings shows that
Massachusetts can be 95% confident not only that an
improvement occurred in the population between 1 998
and 2003, but also that the proportion of all printers
statewide using UV ink increased by at least 1 % and
perhaps as much as 38% from 1 998 to 2003—a wide
margin due to small sample sizes, but informative
nonetheless. By incorporating this range of performance
improvement data into Massachusetts' environmental
outcome analysis methodology, it can be estimated that
the increased use of UV ink among printers led to a
reduction in statewide VOC emissions of between 157
and 8,011 tons per year (TPY), with a midpoint of 4,084
TPY.27 Massachusetts does not have clear evidence that
the change is solely attributable to ERP, but the measure
nonetheless helps show that printers are moving in the
right direction and helping the environment. Even if ERP
is not fully driving the change, ERP can allow a much
better understanding of whether change is occurring, and
to what extent.
18
-------
Effectively With ERP
The evidence that sectors have improved and maintained performance after ERP has been applied
appears to be a primary factor motivating states to implement new ERPs and to continue with existing
ones. But agencies also indicate they are adopting ERP because they believe it will empower them
to use their increasingly scarce resources much more effectively. As this section discusses in detail,
states implementing ERP have reported enhanced targeting capabilities, reduced long-term need for
enforcement, improved permitting approaches, and an effective voluntary policy option. ERP states
also take advantage of cost savings associated with using statistics and automation, and they benefit
from delivering measurable mission-based results and from better meeting stakeholder demands. These
reported benefits, when considered along with results from ERPs to date, suggest that ERP can represent
a cost-effective option for regulating sectors or groups with large numbers of pollution sources.
4.1 States Report Enhanced Targeting
Capabilities
As discussed earlier, EPA and states have historically focused
traditional regulatory approaches on easier-to-target large businesses,
where a single agency action may have a significant environmental
impact. Currently, however, regulators increasingly face the challenge
of addressing impacts of regulated groups comprising many small
polluting facilities. For these sources, limited information is typically
available to help regulators prioritize their resources.
ERP self-certification data provide a means to efficiently identify
facilities most in need of attention. States can mine certification
data to identify and follow up with facilities that did not return
self-certification forms, that submitted return-to-compliance plans,
and that provided self-certification responses that are internally
inconsistent or otherwise raise "red flags." Such inconsistencies
can mean facilities either do not understand their obligations or are
neglecting them. When data analysis identifies a facility in need of
attention, agencies can choose the most appropriate response, ranging
from providing compliance assistance to initiating enforcement
actions. The box at right describes a Massachusetts targeting success
story that impacted the performance of numerous facilities.
At least one state has found that targeted ERP inspections may be
more likely to find problems than random inspections—although
random inspections are still considered critical for measurement
purposes and for deterring regulatory violations. Massachusetts'
preliminary analysis of ERP data indicates that facilities targeted
for inspection based upon suspicious certification data have been
substantially more likely to have serious problems than facilities
19
Cost-Effective Targeting
Through ERP
Massachusetts' analysis of ERP
certification forms and accompanying
wastewater sampling data allowed the
state to identify and address pervasive
problems at three major retail chains,
each with numerous photo processing
facilities. Without ERP, these persistent
violations may not have been detected.
Massachusetts' efforts led to the payment
of $215,000 in civil and administrative
penalties by the three companies
combined. Further, the state suspended
an additional $131,000 in penalties on
the condition that two of the companies
implement initiatives intended to ensure
against such violations in the future. The
other company also agreed in settlement
to take similar steps to improve the
environmental performance of its stores.28
The president of the Environmental
League of Massachusetts indicated his
satisfaction with the effort: "Thanks to the
DEP, the enforcement picture is becoming
very clear: Photo processors that don't
take care of the environment expose
themselves to fines. This is a very positive
development."29
-------
inspected during random visits. Massachusetts examined four rounds of random and targeted inspections across
three different sectors. Enforcement actions resulted from 63% of targeted inspections, as opposed to 42% of random
inspections—a difference of 21 percentage points. The analysis did not specifically examine the seriousness of the
violations, but anecdotal information from inspectors suggests that more serious violations are found during targeted
inspections.30
ERP inspection data may help improve the focus and effectiveness of compliance assistance as well. Baseline data
from the Maine Department of Environmental Protection's (DEP's) auto body ERP indicated that only 29% of sampled
shops were meeting the requirement to recycle mercury-containing fluorescent light bulbs. Consequently, DEP staff
emphasized in workshops, workbooks, and certification forms why and how to recycle the bulbs—and facilities
responded. Maine's second random sample discovered 85% compliance with this requirement. This 56 percentage
point improvement is statistically significant at a 95% confidence level.31
4.2 ERP May Reduce the Need for Enforcement
In many ERPs, a surprising number of businesses
submit return-to-compliance plans the first time they
self-certify—suggesting that facilities can improve their
compliance even without an inspector at their doors
and with limited direct resource investment by the state.
Three states with voluntary certification programs for
auto body facilities provide evidence to support this
finding. In Rhode Island, 20% of all facilities in the
state volunteered that they were out of compliance and
submitted return-to-compliance plans the first time they
self-certified. In Delaware, the figure was 13%. In Maine,
34% self-declared violations.32
But what happens after that? States track the
implementation of return-to-compliance plans to ensure
that facilities follow through with plan implementation.
Further, data from random inspections demonstrate that
once ERP facilities achieve compliance, they typically
maintain that level of environmental performance, requiring
less and less attention from regulators. This improved
performance means facilities need to submit fewer return-
to-compliance plans after the first cycle of ERP.
Sustained, documented high levels of performance
can offer regulators flexibility in strategically targeting
resources. For instance, regulators in Florida and
Massachusetts saw a substantial drop in return-to-
compliance plan submissions after early certification
cycles, backed up by inspection results showing
sustained performance at high levels. In response, Florida
chose to reallocate inspection resources from the auto
repair sector to other, higher-priority sectors. For its part,
Massachusetts chose to reduce the frequency of random
ERP site visits, believing that facility performance would
be sustained. ERP's measurement approach allows
states to confirm such beliefs by periodically checking
on facility performance. For example, Massachusetts
conducted follow-up random inspections in the dry
cleaning sector in 2007, and the state will be analyzing
the data in order to understand whether the sector has
maintained the high level of performance described in
Section 3.
4.3 ERP Can Complement or Replace Permitting
ERP has the potential to enhance the performance of
traditional permit programs, and in some cases even
replace them. In Massachusetts, ERP's industry-wide
environmental performance standards and annual self-
certification submissions replaced certain state permits in
the dry cleaning, printing, and photo processing sectors.
Although ERP cannot substitute for federally required
permits for large-scale facilities, it can complement
other traditional permitting programs. For instance, ERP
measurement and self-certification could potentially help
core regulatory program offices to improve and verify
performance in some permitted sectors. ERP could also
20
-------
be used as a vehicle for implementing general permitting programs, which require facilities to periodically report on
and certify to specified compliance standards—similar to a mandatory ERP self-certification. To those features, ERP adds
compliance assistance and performance measurement to create an even more robust system.
Finally, ERP may be used as a tool for implementing new regulations that could, but may not necessarily, trigger
permitting requirements—informing the regulated community about new requirements and helping to ensure
compliance. For instance, EPA is actively exploring the extent to which ERP could help implement pending new
regulations facing small "area sources" of air pollution. Some of these categories of small area sources, like auto body
shops, are sectors or groups for which states have already successfully developed ERPs.
4.4 Statistics and Technology Provide Economies of Scale
More and more, ERP is being implemented in sectors
with very large numbers of pollution sources, such as
among oil and gas extraction operations, small animal
feedlots, and facilities with underground storage tanks.
Large populations of facilities can present significant
opportunities for states to take advantage of economies
of scale in automation and statistics.
Information technology can offer states the opportunity
to substantially streamline their processes and readily
assess the performance of individual facilities, groups of
facilities, or entire sectors. For instance, Massachusetts
DEP's ERP tracking and analysis system saves substantial
staff time by automatically screening certification data for
inconsistencies and generating performance results reports.
Statistics plays its role in efficiency by offering what
some consider to be surprising economies of scale. As
populations of facilities grow larger and larger, agencies
only need to undertake relatively small numbers of
additional random inspections to achieve the same
confidence in their results from a statistical point of
view. Figure 6 (right) shows an example of how states
need proportionally fewer random inspections as the
sector population grows, even by orders of magnitude.
For instance, imagine that a state planned to conduct
50 random baseline inspections among a population
of 200 facilities. If they instead had a population of
2000 facilities, they could achieve the same statistical
confidence in their results by doing just 14 more
inspections, for a total of 64. With 20,000 facilities, they
would need to do only two more inspections, for a total
of 66.33
Figure 6. Economies of Scale with Statistics
20,000
Randomly sampled facilities
All other facilities in sector
o 15,000
200 FACILITIES
(50 inspections)
2000 FACILITIES
(64 inspections)
20,000 FACILITIES
(66 inspections)
Note: Figures are based upon a 95% confidence level and maximum margin of
error of approximately+/-12 percentage points.
21
-------
4.5 ERP's Results Can Help Meet Demands for Better Measurement
Increasingly, regulatory agencies are being required to set measurable goals and document their performance against
those goals. Many agencies struggle to meet the requirements of these performance measurement obligations. For
instance, EPA strives to show credible measurable progress in achieving its core mission, as required under the
Government Performance and Results Act (GPRA) of 1993 and the White House Office of Management and Budget's
(OMB's) Program Assessment Rating Tool (PART) initiative.
Agencies often find particular difficulty in tracking mission-oriented outcomes, such as improvements in compliance
or environmental quality. Agencies' customary measures—like numbers of inspections conducted or the total penalties
associated with enforcement actions—fail to tell the full story.
ERP performance data can help states demonstrate project outcomes to those stakeholders demanding credible tracking
of the outcomes of agency efforts. Furthermore, ERP's portraits of group performance can allow for more informed
priority-setting and budgeting.
4.6 Many Businesses Value ERP as Fair and Helpful
ERP can help create a level playing field by holding
all businesses to a uniform standard and following
through with enforcement action. In many ERP
states, business owners actually ask the state to make
certification mandatory rather than voluntary. In
Massachusetts, where regulations allow the state to
reduce the frequency of certification when a sector
has demonstrated sustained high performance levels,
business leaders in at least one sector have urged the
state not to do so.
Businesses also appreciate ERP's consistent emphasis
on all environmental media. Most agencies do not
typically take a multimedia approach in working with
regulated facilities. Instead, agencies often interact with
facilities via several different offices or departments,
each responsible for a separate environmental medium
(e.g., air, water, and waste). ERP, on the other hand,
is a single, focused package that is intended to help
businesses better manage all aspects of environmental
compliance.
Further, ERP's regular certification process may help
businesses routinize environmental management and
reduce compliance problems related to employee
turnover. It may also help save money, because the
plain-language, step-by-step certification process can
substantially reduce the need for consultants.
Praise for ERP from the
Business Community
"[T]he ERP model is working and has
reduced the burden on DEP and on
regulated sectors with absolutely
no reduction in environmental
protection....
"The ERP model, combined with DEP's
continued focus on electronic data
initiatives, has saved the business
community hundreds of thousands of
dollars in fees and consultant time."
— Robert A. Rio, Esq., Senior Vice President
for Government Affairs, Associated
Industries of Massachusetts (AIM). AIM
describes itself as the largest employer
association in Massachusetts.34
-------
4.7 ERP Offers the Public Visible Improvements and Broader Accountability
Residents Credit ERP for
Neighborhood Improvements
The Maryland Department of the Environment
focused its ERP on auto repair and body shops
in the Park Heights neighborhood of Baltimore, a
low-income area with a high minority population.
Regulators were concerned about "environmental
justice" issues in this neighborhood because
these shops were considered by many residents
to be a blight on the community and were in close
proximity to schools and facilities for health care,
child care, and elder care.
Project partners randomly surveyed residents after
the project to gauge how they felt about ERP's
accomplishments. They found that 47% of a large
sample of residents knew about the project, and
87% of those believed the project was working
to improve the neighborhood's environmental
conditions. Further, the survey indicated that
residents perceived a number of neighborhood
improvements over the timeframe of the project.
These improvements include perceived reductions
in discarded parts and tires, used oil spills, odors,
smoke, and nighttime noise.35
The success of this project shows that, while ERP
may be most cost-effective on a larger scale,
it can help create meaningful and noticeable
improvements at the local level as well.
For its part, the public has shown an awareness of
ERP's positive impacts, such as visible improvements
attributed to Maryland's ERP (see box at left). On a
larger scale, reports on sector performance can enable
agencies to show public interest groups and concerned
citizens the extent to which regulators are achieving
their mission and allocating resources appropriately.
States also typically make individual certification forms
available upon public request, allowing citizens to
check on the performance of facilities in their own
neighborhoods. Some agencies envision going further,
using the Internet to make non-confidential certification
information immediately available to the public. Such
information could be bundled with mapping software to
enable each citizen to see what nearby businesses are
reporting about their performance.
4.8 ERP Can Offer an Effective,
Voluntary Option
Many ERP states have found voluntary certification to
be an effective alternative to mandating that facilities
submit certification forms, which is sometimes not
feasible. As Table 2 showed earlier, results from several
voluntary-certification ERPs clearly indicate that
performance has improved post-ERP—not just among
volunteers, but across the sector as a whole.
Why? No thorough analysis has been undertaken
yet, but the data provide clues suggesting that
voluntary-certification ERPs have been effective to
date at engendering trust between state agencies and
industries, which in turn may help drive broad-based
changes in behavior. Experience to date suggests that
higher proportions of facilities may submit return-to-
compliance plans when an ERP has voluntary, rather
than mandatory, certification. Further, ERPs to date have
generally seen high levels of voluntary certification, and
states anecdotally report that many facilities that do not
submit certification forms still appear to conduct self-
audits and improve their performance.
23
-------
It is not yet possible to say whether, and
under what circumstances, a voluntary
certification approach may work better than
a mandatory one. This question may be
explored more in future years. Perhaps it is
telling that state staff managing voluntary
certification programs tend to say they
would prefer that the certification be made
mandatory, for a variety of reasons—such as
to level the playing field for businesses, to
increase the economies of scale associated
with managing certification data, and to
improve their own ability to garner program
management resources, such as by charging
certification fees. Nonetheless, when
mandatory certification is not feasible or
desired, it appears that ERP can offer an
effective, voluntary option.
Maine's Voluntary Approach
Engages Businesses
"[T]he Environmental Results Program has increased
our Department's ability to work with smaller
businesses and gain their engagement in protecting
the Maine environment. By using the ERP compliance
assistance and education model, many more smaller
businesses in the state are aware of compliance
regulations and how to implement them...."
4.9 ERP Appears to Be a Cost-Effective Alternative
ERP results to date, combined with the various other
benefits described elsewhere in this report, strongly
suggest that ERP can be a cost-effective option for
regulating sectors or groups with large numbers of
pollution sources. It would be instructive to compare
each existing ERP with comparable traditional
approaches, specifically in terms of costs and
performance levels. Doing so is challenging, however,
in large part because of the difficulty in directly
comparing traditional compliance programs with ERP:
Traditional programs do not typically have comparable
data depicting sector-wide performance levels, and state
environmental agencies differ significantly in how they
track their resource expenditures.
EPA is working with states to overcome this measurement
challenge. For instance, Rhode Island DEM has received
a grant from the National Center for Environmental
Innovation to conduct a detailed study comparing the
cost-effectiveness of its ERP for underground storage
tanks (USTs) to the cost-effectiveness of one or more
traditional approaches for regulating USTs. Results of that
study are anticipated in 2008.
To more closely assess ERP's cost-effectiveness for the
purpose of this report, EPA examined two dry cleaner
programs, in Massachusetts and Michigan, whose data
lend themselves to comparison. Massachusetts' dry
cleaners have been regulated under the ERP model since
1 997, and achieved very high scores on EBPIs the last
two times Massachusetts conducted random inspections
of the sector. The approach to dry cleaner compliance
assurance taken by the Air Quality Division of Michigan's
Department of Environmental Quality (DEQ) has
been more traditional, with the state inspecting every
dry cleaning facility once per year and providing
compliance assistance upon request. Michigan DEQ's
census-based inspections approach for this sector stems
from particular circumstances: annual inspections of
dry cleaning facilities are required under an industry-
supported Michigan law, and Michigan DEQ's dry
cleaning program is substantially funded through a
license fee paid by dry cleaning facilities.36 The next two
subsections compare these two programs on the basis of
the environmental performance of the dry cleaners they
regulate and the personnel resources each program uses
to implement its goals.
24
-------
Environmental Performance Under ERP Compared
to "Inspect Everyone" Approach. In December 2006,
as part of implementing a pilot ERP, Michigan DEQ
completed random inspections of its
dry cleaner population, using an EBPI-
based checklist to evaluate "baseline"
sector performance. Inspection results
show how well a random sample of 262
Michigan dry cleaners is performing
on the state's EBPIs, after having been
regulated for years under Michigan's
annual-inspection program, but before
ERP self-certification had begun.37
Those data can be compared to results
from Massachusetts DEP's dry cleaners
inspections during the most recent
ERP random sample in 2002. Those
Massachusetts results demonstrate
how well a representative sample of
25 Massachusetts dry cleaners was
performing on that state's EBPIs after the
fifth round of ERP self-certification in 2002
Dry cleaners regulated
under Massachusetts'
ERP appear to perform
as well as those
regulated under an
"inspect everyone,
every year" approach,
while allowing staff
resources to be applied
toward other issues.
EPA worked with staff from the two states to identify six
comparable EBPIs, and examined each state's findings
from random inspections for those EBPIs. The ability to
draw conclusions from the comparison is somewhat
limited because of the small number of indicators
that are comparable—all related to the prevention
and control of air pollution—and also because the
small size of Massachusetts' sample
makes that state's figures much less
precise than Michigan's. The data
analysis nonetheless suggests that
both populations of dry cleaners were
performing fairly well on the selected
indicators, and does not point to
substantial differences in performance
among each state's dry cleaners at
the points in time EPA examined. On
average, 82% of dry cleaners visited by
Michigan inspectors were achieving the
EBPIs in question; for Massachusetts,
the figure is also 82%. A look at
performance on individual EBPIs shows
similar equality: each state's randomly
inspected dry cleaners were performing
better than those of the other state on
three of the six indicators. Also, the largest observed
difference in compliance proportions is six percentage
points, and none of the observed performance
differences are statistically significant.39 (For more
information on data sources and analysis, see Section 5
of the appendix.)
25
-------
ERP Resource Use Compared to "Inspect Everyone"
Approach. Dry cleaners in each state appear to be
performing at similar levels after years of experience
under each state's respective regulatory approach. How
do the two different approaches compare in terms of
resource usage? To examine this question, EPA compared
each state's estimated annual staffing deployment, as a
proxy for overall resource expenditures.40
Michigan DEQ utilizes approximately 4.75 full-time
equivalent (FTE) employees per year to manage its dry
cleaner program, inspecting each of nearly 900 facilities
once a year (including both those using perc and those
using other solvents). For its part, Massachusetts DEP
estimates it expends a maximum of two FTEs per year
to maintain its ERP for roughly 600 perc dry cleaners,
and approximately 1.5 FTEs in years where it does not
conduct random inspections. In the first six years of
ERP (1 997-2002), Massachusetts conducted four rounds
of random inspections. On this basis, EPA estimated
average Massachusetts dry cleaners ERP FTE usage at
1.83 per year.41 Adjusting for differences between the
states in numbers of hours per FTE and numbers of
perchloroethylene dry cleaners, EPA estimates that the
Massachusetts approach utilizes approximately 50%
fewer FTEs than the traditional Michigan approach. (For
more information on data sources and analysis, see
Section 5 of the appendix.)
When considered along with the other information
presented in Sections 3 and 4 of this report, this
comparison suggests that, when allowed by law,
agencies might be able to achieve resource efficiencies
by using an ERP approach, thereby enabling agencies
to shift personnel resources, such as inspectors, to other
pressing priorities. This comparison, combined with the
performance data available on ERPs to date, suggest
states may be able to make this shift while maintaining
existing performance levels in the sector. Before doing
so, however, states must carefully weigh the extent
to which reductions in inspections will increase their
uncertainty about environmental performance in a
sector. For instance, regulators moving from a census
inspections regime to an ERP approach, especially one
with small-sized random samples, may need to grapple
with substantially different levels of uncertainty. In those
cases and others—if resources are sufficient—states may
wish to find a middle ground between efficiency and
measurement certainty.
Greatest Efficiency Requires Manageable, Up-
Front Investment. The apparent efficiency associated
with ERP does not come without some investment,
of course. For instance, a review of workplans for 12
states receiving EPA grant funds to implement ERP
shows an average startup cost for ERP of approximately
$226,000.42 States typically spread out these
expenditures over a period of about three years, the
typical implementation time for the first ERP cycle.
States pursuing advanced data management approaches
have higher-than-average investment costs, but may
generate greater-than-average operating efficiencies.
Currently, data are too limited to develop more precise
estimates of startup and operating costs, but EPA intends
to work with states to improve the data in this area, and
present findings in future reports.
Even without substantial grant funding, states can find
that startup costs are not insurmountable. After all,
Massachusetts developed and increased its use of ERP
largely with in-state resources, even while adapting to
substantial budget and personnel cuts. Rhode Island's
first ERP was also
primarily funded
by the state itself.
Further, now that
Rhode Island and
Delaware have
completed their
first ERP cycles
for the auto body
sector, these states
are continuing to
implement ERP for
this sector using
relatively small
expenditures of their
own resources.
-------
New ERPs Benefit From Peer and EPA Support.
Achieving ERP efficiencies appears to have grown
easier over time as a result of accumulated benefits of
state and EPA investments. For instance, workbooks,
checklists, and helpful information about lessons learned
are available for those sectors that have already been
piloted by one or more states. A large community of
ERP states has formed a States ERP Consortium that
conducts frequent meetings by phone and in person
to share lessons learned. States also report that readily
available state and federal ERP tools and documents have
improved implementation efficiency and reduced startup
time. The box below ("Transferable Tools Streamline ERP
Startup") provides a sample of available resources.
Transferable Tools
Streamline ERP Startup
With so many states implementing ERP,
and with EPA and the recently formed
States ERP Consortium providing long-term
capacity-building support, many tools and
resources are already available to reduce
the costs of ERP startup and to share
lessons learned.
Examples include:
i Sector-specific workbooks, certification
forms, inspector checklists, and other
materials;
i Novice-friendly statistical planning and
analysis tools;
i Automated data storage, processing, and
analysis systems;
i Detailed ERP implementation strategies and
schedules; and
i An online "ERP Roadmap" that provides
implementation guidance for each phase of
ERP, as well as an extensive ERP resource
library.
Since 2002, the
National Center
for Environmental
Innovation has
awarded $2.9
million in State
Innovation Grants,
helping 14 states to
initiate new ERPs.
As mentioned, EPA has
helped to defray upfront
costs with grants and
other support. Since 2002,
the National Center for
Environmental Innovation
has awarded $2.9 million
in State Innovation Grants,
helping 14 states to initiate
new ERPs. Moreover,
EPA staff and contractors
develop tools to share with
states developing ERPs,
and provide states with
free technical assistance,
especially on measurement
approaches and ERP best practices. Performance
partnership agreements with EPA regional offices have
allowed states to utilize enforcement resources on ERP
inspections.
More recently, Massachusetts was awarded resource
flexibility "credit" through EPA's State Review Framework
(part of EPA's enforcement program accountability
system). This flexibility was granted based on the
recognized success of the state's ERP for dry cleaners
and the overall performance of Massachusetts' core
enforcement programs in air and waste. The state
anticipates this arrangement will enable it to focus more
resources on emerging priorities for environmental
improvement while ensuring more traditional
performance objectives are still being met.
27
-------
ERP s Future Promises Improvement,
Experimentation, and Growth
Table 3 (below) demonstrates the expansion of ERP since it was first developed and implemented by
Massachusetts 10 years ago. Now, 18 states have implemented or initiated an ERP in a total of 11
sectors or groups, and 1 6 states currently have active programs. Four states are implementing multiple
ERPs. Eleven of the 25 ERPs involve mandatory certification requirements, typically based on new state
regulations. Eight of the 10 EPA regions now have experience with ERP in at least one of their states.
As the following subsections discuss,
EPA in the coming years anticipates
that the number and type of ERP
applications will grow, and that
ERPs will produce increasingly
more comparable environmental
performance data. EPA plans to
continue supporting states that are
interested in adopting ERP, but also
expects that states will increase their
collaboration with one another,
gathering to share information and
develop new tools through the new
States ERP Consortium. Finally,
EPA plans to work closely with the
Consortium and other interested
parties to continue to explore
important questions about ERP in
future years.
Table 3. ERP Activity in 18 States*
CDDO* * CDD i •*• * ,1 ERPs ERPs with Mandatory
ERP State ERPs nitiated „ . , . „ _.„ ,. '
Completed Certification
Delaware
Floridat
Illinois
Indiana
Louisiana
Maine
Marylandt
Massachusetts
Michigan
Minnesota
Nevada
New Hampshire
New York43
Rhode Island44
Vermont
Virginia
Washington
Wisconsin
1
1
1
1
1
2
1
3
1
1
1
1
2
4
1
1
1
1
1
1
-
-
-
1
1
3
-
-
-
-
-
2
-
-
-
-
-
1
-
-
1
-
-
3
-
-
-
1
2
2
1
-
-
-
* This table defines "ERP Activity" as the implementation of initiatives that use all of the ERP
tools—i.e., combining compliance assistance, self-certification, inspections, and statistically based
performance measurement. Many states are also implementing valuable "ERP-like" initiatives, utiliz-
ing subsets of those four ERP elements, that show promise to achieve demonstrable performance
improvements. (See Section 5.3 for more information).
t No longer implementing ERP.
45
-------
5.1 More States Work to Produce Results
ERP results have thus far accumulated gradually because states typically
take about three years to fully implement their first ERP, from conception
to first post-certification statistical results. Nonetheless, each year more
states finish an initial ERP performance cycle. In the next two years,
EPA expects to see results from several ERPs—and several states—that
are completing full measurement cycles for the first time. These efforts
include ERPs for underground storage tanks, auto salvage yards, and
animal feedlots. The results will be the first ever for these ERP sectors.
EPA also expects to see results from new rounds of random inspections
for the Massachusetts dry cleaner and Rhode Island auto body ERPs,
which will provide additional insights into the long-term efficacy of ERP.
5.2 Agencies Act to Improve Comparability of Results
ERP has appealed to regulatory agencies in part because
of its adaptability to specific state circumstances.
However, this adaptability has led to substantial
variety in reporting that in turn makes assembling and
comparing data from different ERPs—even in the same
sector—quite challenging and resource-intensive. EPA
expects to see increasing convergence among ERP
states and others on indicators to be used for tracking
the performance of business sectors and other groups.
For instance, the States ERP Consortium (discussed in
Section 5.4) plans to develop a set of recommended
ERP reporting standards in order to help ensure that
data reported by all Consortium members meet core
minimum needs for transparency and comparability.
This Consortium effort will draw substantially upon
lessons learned in an important, ongoing ERP-related
measurement initiative: the Common Measures
Project. Explicit goals of this project are to encourage
more widespread adoption of the ERP-style statistical
performance measurement techniques and to allow
participating states to benchmark their performance
against others—whether using ERP initiatives or other
traditional or innovative tools. Under this EPA-funded
project, Massachusetts DEP is leading a group of several
states in developing common measurement indicators
for two regulated groups: auto body shops and small
quantity generators (SQGs) of hazardous waste. SQGs
are facilities—from a variety of sectors—that produce
a moderate amount of waste defined by regulation
as "hazardous." Most states have hundreds, if not
thousands, of SQGs and vary in their approaches for
regulating those facilities.
Common Measures Project participants have just begun
their work on the auto body sector, but their work on
SQGs is well underway, with participants having agreed
upon EBPIs and a statistical approach for measuring
SQG performance. Data from the Common Measures
Project may soon help regulators adapt their approaches
to regulating this important group, based upon the
experience of other states.
5.3 Agencies Apply ERP Tools in Innovative Ways
States are adapting ERP to serve new purposes, address
emerging problems, and juggle competing priorities.
Emerging experiments include applying some (but not all)
ERP tools to address a problem, using ERP approaches as
an alternative to new or more stringent regulations, and
developing watershed-based ERP approaches.
Applying the ERP Tools in New Ways. A number of
states—such as Colorado, Florida, Massachusetts, and
Rhode Island—are implementing initiatives that utilize
more limited sets of the ERP tools. Some states—like
those in the Common Measures Project—use ERP-
style statistical measurement approaches to assess the
efficacy of various policy initiatives, not necessarily
self-certification approaches. Other states are using
certification approaches without random inspections.
-------
For instance, in 2000, Massachusetts began
implementing a certification-based approach to address
poor compliance by the state's roughly 3,000 gas stations
with vapor recovery regulations. Those regulations
are intended to substantially reduce air emissions
that can occur during the refueling of motor vehicles.
Noncompliance
meant that "at-the-
pump" releases of
volatile organic
compounds
(VOCs) were not
being sufficiently
controlled, which in
turn suggested that
the state was at risk of violating national ambient air
quality standards for ozone. EPA could have required
Massachusetts to try to increase compliance through
annual facility inspections, a resource-intensive
proposition. In this case, however, EPA's New England
regional office approved Massachusetts' plan for
mandatory annual self-certification, combined with more
stringent testing and self-inspection requirements. The
state verifies performance through independent, third-
party testing of vapor recovery system integrity, as well
as targeted (not random) inspections. Now, 98% of the
2.7 billion gallons of motor fuel dispensed annually in
the state flows through certified facilities. Massachusetts
estimates that this approach has substantially reduced
emissions by controlling an additional 3,960 tons of
volatile organic compounds (VOCs) each year.46
Mercury Falling
In its first year, a Massachusetts voluntary
certification program for dentists, in lieu of
regulation, reduced estimated annual mercury
discharges by the equivalent of cleaning
up approximately 171,000 broken mercury
thermometers or salvaging about 130,000
mercury-containing automotive switches.47'48
Alternative to New or More Stringent Regulation.
ERP may in some cases offer a promising alternative to
new or more stringent regulation for both the potentially
regulated community and for regulatory agencies.
In 2005, motivated in part by a desire to postpone
regulatory action, 74% of Massachusetts' 3,600 dentists
voluntarily certified that they were meeting new mercury
management standards. This ERP-like initiative uses
self-certification only, without random inspections.
Massachusetts estimates this voluntary action resulted in
an immediate annual reduction in mercury discharges
of approximately 230 pounds, with further reductions
anticipated as the state continues to encourage more
dentists to certify.49 These estimates are based on
facility-reported data, bolstered by requirements that
facilities provide independently verifiable information on
hazardous waste disposal.
Impaired Water Bodies and Watersheds. Some ERP
states have already targeted their efforts in defined
geographic areas impacted by specific environmental
problems, such as air quality degradation or
environmental justice issues. ERP partners are also
exploring the use of ERP tools to demonstrably and
effectively address water body impairments caused by
stormwater runoff. EPA awarded two different State
Innovation Grants in 2007 to explore this use of ERP.
Under one grant, Maine DEP and Massachusetts DEP
will be using a wholly voluntary certification approach
to try to reduce discharges of polluted stormwater from
drive-through facilities and shopping malls, respectively.
Both states will be targeting high-priority watersheds,
and strongly encouraging the adoption of voluntary best
management practices by entities whose stormwater
discharges are not otherwise subject to state regulation.
Under the other grant, Rhode Island DEM will work
with municipal separate storm sewer systems (MS4s) on
a mandatory ERP to reduce stormwater discharges from
certain construction sites.
Targeting multi-state watersheds may be the next step in
this evolution. In a recent report, the National Academy
of Public Administration identifies ERP as one of several
innovative approaches that could have an impact in
improving water quality in the Chesapeake Bay Basin.50
30
-------
5.4 States Convene an ERP Consortium
In October 2006, a large group of states formed a
consortium to explore, develop, promote, and implement
the use of ERP approaches for addressing priority
environmental issues efficiently and effectively (see
Figure 7, below). The States ERP Consortium is officially
organized as a "forum" of the Environmental Council
of States (ECOS), and includes 18 states either currently
using ERP or interested in learning how to use ERP. The
National Center for Environmental Innovation is on
the organization's steering committee and is providing
contractor support.
The Consortium is implementing action plans in four
areas its members deem critical to the growth of ERP:
(1) Communicating results in order to build stakeholder
support;
(2) Sharing information among practitioners;
(3) Promoting ERP as a proven compliance strategy, and
expanding support for ERP within and beyond EPA;
and
(4) Enhancing and disseminating tools that streamline
key aspects of ERP, such as automation and
measurement.
Consortium members hope that providing states with a
common voice and more formal lines of communication
will create new possibilities for identifying and
addressing important problems associated with the use
of ERP and its component tools. EPA is committed to
working with the Consortium in the future to continue
evaluating the many applications of ERP, communicating
ERP results, and finding and developing opportunities to
integrate ERP into the federal regulatory framework. For
more information about the Consortium and its work,
visit www.ERPstates.org.
Figure 7. Growing ERP Community Represented by a New Consortium of States
ERP Implemented
Learning States
Consortium Members
Not currently implementing ERP.
-------
5.5 EPA Plans to Further Explore
Questions Raised by ERP
This document is the first major report on ERP activity
since EPA produced The Massachusetts Environmental
Results Program: User's Guide for Government Agencies
in 2002. Since that time, 17 additional states have
implemented or begun developing ERPs for eight new
groups or sectors. Many of these ERPs involve voluntary
submission of self-certification forms by facilities, an
innovation developed in the last five years.
This report has compiled and presented results for eight
ERPs, covering five sectors. The data from these ERPs,
along with the lessons learned in implementing them,
have helped make a case for regulators to consider using
ERP when faced with problems caused by large numbers
of small sources. Yet the full story of ERP is still unfolding.
Many questions about ERP—particularly those related to
environmental performance and cost-effectiveness—still
bear further examination, especially as states provide
new information.
The box at right provides examples of some of the many
questions EPA plans to explore in the coming years,
using the wealth of data that ERP provides. EPA intends
to work diligently to identify and execute approaches
for addressing these and other questions, and invites the
States ERP Consortium, the academic community, and
other researchers to join in this important work.
Sample Issues for
Further ERP Research
Environmental Performance
A Long-term performance of ERP, using new
data from Massachusetts and other states
that have been using ERP for a relatively
long period
A Performance of ERP in the newest sectors
A Changes to the environment and public
health that may occur as outcomes of ERP
A Efficacy of emerging "ERP-like" approaches
that do not use all of the tools in the
integrated ERP system
Cost-Effectiveness
A Cost-effectiveness for agencies, relative to
other compliance approaches
A Costs of ERP implementation over time
(after startup)
A Cost-effectiveness for the private sector
Efficacy of Voluntary Approaches
A Efficacy of ERP in promoting compliance
versus efficacy in promoting voluntary
best management practices and pollution
prevention measures
A Relative efficacy of voluntary versus
mandatory self-certification, in the same
sector
32
-------
Notes
1 Golledge, Robert, etal. "Regulating Small Business Facilities—
The Environmental Results Program." ECOStates: The lournal of the
Environmental Council of the States. Pages 5-7, 37-40. Fall 2003.
2 Nationwide figure for retail gas stations is 130,515. Source:
U.S. Census Bureau. 2002 Economic Census. NAICS code 447
(Gasoline Stations). Accessed September 26, 2007 http://www.
census.gov/econ/census02/data/industry/E447.HTM.
3 EPA's Innovation Action Council comprises the senior career
leadership of EPA's program and regional offices.
4Nationwide figure for auto body shops is 32,892. Source: U. S.
Census Bureau. 2002 Economic Census. NAICS code 8111211
(Automotive Paint or Body Repair). Accessed September 26, 2007
http://www.census.gov/econ/census02/data/us/USOOO 81 .HTM.
5 Performance changes in this report are expressed in terms of
"percentage points," which is distinct from the expression of
changes in terms of "percent." For instance, imagine sector
performance with regard to a particular regulatory requirement
improved from 50% of facilities in compliance to 75% in
compliance. That change can be described as an improvement
of 25 percentage points, or a 50% increase. This report uses the
former convention.
6 Maryland's Department of the Environment implemented ERPon
a small, pilot basis, but leaders of the initiative left the Department
shortly after project completion and no further attempts to
implement ERP are known to have occurred in the state. Florida's
Department of Environmental Projection (DEP) implemented ERP
as a large pilot in two of the state's six administrative regions.
ERP measurement indicated facilities in those regions were
already performing fairly well prior to self-certification. The sector
showed further improvement after two cycles of mandatory
self-certification in those regions. (See Section 3 of this report for
performance data.) At that point, DEP determined that a statewide,
full ERP for the auto repair sector would be unnecessary and not
cost effective. DEP continues to offer the opportunity to all auto
repair facilities statewide to conduct self-audits of their operations
using ERP-based materials, but does not require self-certification
and does not conduct regular ERP random inspections in the
sector.
7 Lee, Eungkyoon. Why Did They Comply While Others Did
Not?: Environmental Compliance of Small Firms and Implications
for Regulation. Ph.D. dissertation. Massachusetts Institute of
Technology. September 2005.
8 Sources: (1) Enander, Richard T., etal. "Chemical
Characterization of Sanding Dust and Methylene Chloride Usage
in Automotive Refinishing: Implications for Occupational and
Environmental Health." Pages 741-749. Al HA Journal. Volume
63, Issue 6. 2002. (2) Enander, Rich. Rhode Island Department
of Environmental Management. "Rhode Island's ERP Experience:
Automotive Refinishing Sector." Presentation at 2004 California
Environmental Protection Agency Environmental Results Program
Workshop. Slides 6-7. September 30, 2004.
9 Enander, Richard T., etal. "Environmental Health Practice:
Statistically Based Performance Measurement." Pages 819-824.
American lournal of Public Health. Volume 97. 2007.
10 Enander, Richard T, et al. 2007.
11 Initial inventory size: April, Susan and Tim Greiner. Evaluation
of the Massachusetts Environmental Results Program. Learning
from Innovations and Environmental Protection: Research Paper
Number 1. Page 27. National Academy of Public Administration.
June 2000. Maximum inventory size: Peck, Susan. Massachusetts
Department of Environmental Protection, "erp universes by
year.xls," unpublished Microsoft Excel spreadsheet. E-mail
communication with U.S. EPA contractor. March 23, 2007.
12 Go I ledge, Robert, etal. Fall 2003.
13 U.S. Government Accountability Office. Environmental
Enforcement: EPA Cannot Ensure the Accuracy of Self-Reported
Compliance Monitoring Data. Washington: GPO, March 1993.
1434% figure of noncompliance reporting based upon 34 facilities
reporting noncompliance on one or more items, out of a post-
certification universe of 1 00 facilities. Source: Maine Department
of Environmental Protection, Office of Innovation. Auto Body
Environmental Results Program: Final Report. Pages 4, 1 0. May 7,
2007.
^Sources: (1) Massachusetts DEP. ERP Industry Progress Report:
Photo Processing Industry. Page 8. July 2003. (2) Massachusetts
DEP. ERP Industry Progress Report: Dry Cleaning Industry. Page 7.
July 2003. (3) Massachusetts DEP. ERP Industry Progress Report:
Printing Industry. Page 7. July 2003.
16 Florida Department of Environmental Protection. "Compliance
Certification Program (CCP). Compliance Assistance for Auto
Repair Pilot Project (CAPP). Florida 2001-2004 Project." Slide 57.
March 24, 2005.
17 Florida Department of Environmental Protection. "Compliance
Certification Program (CCP). Compliance Assistance for Auto
Repair. PILOT PROJECT Preliminary Results." PowerPoint
presentation, Slide 28. 2003.
18 Florida Department of Environmental Protection. Slide 57.
March 24, 2005.
19TetraTech EM, Inc., for Massachusetts DEP. "Environmental
Results Program for Dry Cleaners and Photo Processors, Round 2
Versus Round 3 and Self-Certification, and Printers Round 1 Versus
Round 2 and Self-Certification Data Analysis. Final Report." Page
S.July 16, 2001.
20 Enander, Richard T, et al. 2007.
21 DeGabriele, Steven. Director, Business Compliance Division,
Massachusetts Department of Environmental Protection. "Re: other
Biennial Report follow-up items." E-mail to U.S. EPA contractor.
Januarys, 2007.
33
-------
22 Florida Department of Environmental Protection. Slide 33.
March 24, 2005.
23 For a description of the calculation and data sources, see
appendix, Section 3.
24 Steve DeGabriele and Susan Peck. Massachusetts DEP.
Telephone interview with U.S. EPA NCEI staff and contractor.
March 12,2007.
25 Reilly, Paul. Dry cleaners ERP manager. Massachusetts
Department of Environmental Protection. "Re: relative perc usage
data." E-mail to U.S. EPA contractor. September 27, 2007.
26 For a description of the calculation and data sources, see
appendix, Section 3.
27 Refer to appendix, Section 4, for more information on data
sources and methodological approach.
28Sources: (1) U.S. Environmental Protection Agency and
Massachusetts Department of Environmental Protection. The
Massachusetts Environmental Results Program: User's Guide for
Government Agencies. Page 12. January 2002. (2) DeGabriele,
Steven. "Re: [No subject.]" E-mail to U.S. EPA contractor. October
31,2007.
29Hartwell, Christopher A. Simplify, Simplify: Alternative
Permitting at the State Level. Reason Foundation. February 1999.
Accessed October 9, 2007 http://www.reason.org/ps253.html.
30 DeGabriele, Steven. "Re: Information Requested." E-mail to U.S.
EPA contractor. November 2, 2006.
31 Sources: (1) Maine Department of Environmental Protection,
Office of Innovation. May 7, 2007. (2) Lippert, Sara. Maine
Department of Environmental Protection. "Re: Information Request
for ERP Biennial Report." E-mail to U.S. EPA contractor. November
27,2006.
32 Delaware data: Delaware Department of Natural Resources and
Environmental Control. Final Report. EPA 2002 State Innovation
Pilot Grant Program. December 2005. Maine data: Maine
Department of Environmental Protection, Office of Innovation.
May 7, 2007. Rhode Island data, based upon 74 of 367 facilities
submitting return-to-compliance plans: Enander, Richard T., et al.
2007.
33This example and figure 6 both assume states are resource-
constrained and can only do a limited number of inspections,
which has been common with the early ERPs. The figures are
based upon a 95% confidence level and a margin of error of
approximately +/- 12 percentage points. For instance, if a state
discovered 50% of facilities to be in compliance with an EBPI,
it could be 95% confident that somewhere between 38% and
62% of all facilities in the population were in compliance with
that EBPI. ERP states have found that margins of error of this
magnitude, and larger ones, have often been sufficient for ensuring
quality policy decision-making. More inspections would be
required to achieve smaller margins of error at each population
size (assuming a confidence level of 95%), but similar economies
of scale would still be observed.
34 Rio, Robert A., Esq. Vice President for Associated Industries of
Massachusetts. Letter to Ms. Karen Regas of Massachusetts DEP
regarding "Amendments to 310 CMR 7.00 - For the Control of Air
Pollution." March 22, 2007.
35 McGrath, Dennis. "Report of the Results of a Survey of the Park
Heights Neighborhood Before and After the Initiation of the Park
Heights Environmental Compliance Assistance Program." Schaefer
Center for Public Policy, University of Baltimore. Attachment 7 to
the Maryland Department of Environment's Park Heights Project
Final Report, prepared by Bernard A. Penner. June 30, 2004.
36Sources: Ostrowski, James. Michigan DEQ. Telephone
interview with U.S. EPA contractor. October 1 8, 2007. (2) Section
133.13307 of the Michigan Public Health Code, Act 368 of 1 978.
37 EPA decided to utilize random sample data, as opposed to data
from the annual inspections of all facilities, for two reasons. First,
Michigan DEQ recently adapted its inspection checklist to an
ERP style checklist, allowing for greater comparability. Second,
random sampling may help remove potential bias caused by
inspectors visiting facilities at a relatively the same timeframe each
year. Michigan's large sample size still leaves high confidence in
estimates of population performance.
38 2002 Massachusetts data were used for this analysis because
results from the latest round, occurring in 2007, were not available
for this report.
39 At either a 90% or 95% confidence level.
40A complete cost analysis of each program was beyond the
scope of this report. Each state's program benefits from additional
resources not reflected in this estimate. For instance, Michigan's
dry cleaners receive technical assistance from a separately funded
Clean Air Assistance Program, and the Massachusetts dry cleaners
ERP has benefited from substantial investments in automation that
have lowered operational costs across all Massachusetts ERPs.
41 The actual, long-term FTE average may be marginally lower,
since Massachusetts has reduced inspection frequency over
time. Once Massachusetts completes its 2007 round of random
inspections in the dry cleaner sector, it will have conducted five
rounds of random sampling over 11 years of ERP. On that basis,
Massachusetts DEP's average FTE usage would be 1.73. EPA chose
to use the higher average (1.83 FTEs), since 2007 performance
data were not available for comparison.
42 Precise average is $225,565. Data from U.S. Environmental
Protection Agency State Innovation Grants work plans for ERP
implementation (2002-2006), as of spring 2007. State work plans
included in analysis: Delaware, Illinois, Indiana, Louisiana, Maine,
Michigan, Minnesota, Nevada, Rhode Island, Vermont, Virginia,
and Wisconsin. Please note that the first full cycle of ERP would
be considered the "implementation phase." The true average
implementation costs may be somewhat higher, since these state
grant workplans may not account for all agency staff time involved
in developing an ERP.
34
-------
43 At present, New York intends that its ERPs for printers and for
auto body shops will have, at minimum, mandatory certification
covering certain hazardous waste aspects of ERR It is also
exploring regulatory and statutory authority for making mandatory
the submission of a broader range of information, across
environmental media. Source: Killeen, Tom. New York State
Department of Environmental Conservation. Telephone interview
with U.S. EPA contractor. September 26, 2007.
44This report presents results for only one completed Rhode Island
ERR (for auto body shops), because results have not yet been fully
analyzed for Rhode Island's other completed ERR (for underground
storage tanks).
45 See note 6.
46 In 1 997, before initiating the program, the state was only able
to demonstrate that 4,860 tons of volatile organic compounds
(VOCs) were under control. As of 2005, the state demonstrated
that 8,820 tons of VOCs were under control. Sources for
vapor recovery program information: (1) Massachusetts
Department of Environmental Protection. "Stage II Goals and
Measures—1997-2005," Microsoft PowerPoint slide. Received
from Steven DeGabriele via e-mail to U.S. EPA contractor on
March 29, 2007. (2) Massachusetts Department of Environmental
Protection. "MA Using ERP Differently Than Full ERPfor Dry
Cleaners. Example 1: Stage II Vapor Recovery Program for Fuel
Dispensers" draft. May 13, 2005.
47Assumes average amount of mercury in mercury-containing,
household thermometer equals 0.61 grams. Source: U.S.
Environmental Protection Agency. Mercury Study Report to
Congress. Volume II: An Inventory of Anthropogenic Mercury
Emissions in the United States. Office of Air Quality Planning
& Standards and Office of Research and Development. EPA-
452/R-97-004. Page 4-45. Washington: GPO, 1997.
48 Assumes average amount of mercury in mercury-containing
automotive switch assemblies equals 0.8 g. Source: Adsit, Dan,
et al. Michigan Mercury Switch Study. Alliance of Automotive
Manufacturers, Michigan Department of Environmental Quality, et
al. 2002.
49The executive summary of this report, published prior to this
report, incorrectly refers to estimated annual mercury reductions
of "several hundred pounds," suggesting 300 pounds or more.
Source for dental mercury program information: Massachusetts
Department of Environmental Protection. "MA Using ERP
Differently Than Full ERP for Dry Cleaners. Example 2: Voluntary
Program for Dental Mercury Facilities" draft. May 13, 2005.
50 National Academy of Public Administration. Taking
Environmental Protection to the Next Level: An Assessment of the
U.S. Environmental Services Delivery System. Pages 33-35. 2007.
This report, along with its executive summary and appendix, were prepared for EPA under Contract No. EP-W-04-023.
35
-------
Readers Notes
-------
State ERP Contacts
Delaware
Kimberly Chesser
Small Business Ombudsman
Department of Natural Resources and
Environmental Control
kimberly.chesser@state.de.us
Florida
Michael Redig
Bureau of Solid and Hazardous Waste
Department of Environmental Protection
michael.redig@dep.state.fl.us
Illinois
Andrew Jankowski
Office of Pollution Prevention
Illinois Environmental Protection Agency
Andrew.Jankowski@illinois.gov
Indiana
Rosemary Cantwell
Industrial Waste Section
Department of Environmental Management
rcantwel@idem.in.gov
Louisiana
Melissa Lantz
Office of Management and Finance
Department of Environmental Quality
melissa.lantz@la.gov
Maine
Julie Churchill
Office of Innovation
Department of Environmental Protection
julie.m.churchill@maine.gov
Massachusetts
Steven DeGabriele
Business Compliance Division
Department of Environmental Protection
Steven.degabriele@state.ma.us
Michigan
James Ostrowski
Environmental Science and Services Division
Department of Environmental Quality
OSTROWSJ@michigan.gov
Minnesota
Alister Innes
Prevention &Assistance Division
Pollution Control Agency
alister.innes@pca.state.mn.us
Nevada
Jim Trent
Bureau of Waste Management
Division of Environmental Protection
jtrent@ndep.nv.gov
New Hampshire
Rudolph Cartier, Jr., PE
Small Business Ombudsman
Department of Environmental Services
rcartier@des.state.nh.us
New York
Paul Counterman
Division of Solid and Hazardous Materials
Department of Environmental Conservation
prcounte@gw.dec.state.ny.us
Rhode Island
Ronald Gagnon
Office of Technical and Customer Assistance
Department of Environmental Management
ron.gagnon@dem.ri.gov
Vermont
Marc Roy
Waste Management Division
Department of Environmental Conservation
marc.roy@state.vt.us
Virginia
Russell P. Ellison III
Underground Storage Tank Compliance
Department of Environmental Quality
rpellison@deq.virginia.gov
Washington
Michelle Underwood
Hazardous Waste and Toxics Reduction
Department of Ecology
mund461@ecy.wa.gov
Wisconsin
Renee Lesjak Bashel
Small Business Clean Air Assistance Program
Department of Commerce
Renee.Bashel@Wisconsin.gov
ERP Contacts at EPA's National Center for Environmental Innovation
EPA welcomes your comments about this document.
Scott Bowles
bowles.scott@epa.gov
Kimberly Green-Goldsborough
green-goldsborough.kimberly@epa.gov
Beth A. M. Termini, Esq.
termini.beth@epa.gov
For more information on ERP, including the executive summary and appendix of this report, visitwww.epa.gov/erp.
-------
------- |