EPA100-R-07-007
May 2007
Implementing the
Environmental
Results Program
NCEI
NATIONAL CENTER FOR
ENVIRONMENTAL INNOVATION
-------
Executive Summary
The Environmental Results Program (ERP) is an innovative approach to improving the environmental performance
of various business sectors and other groups with large numbers of small facilities. While individual facilities
within these groups may release small amounts of pollution, their aggregate impact can be significant. The
Massachusetts Department of Environmental Protection developed ERP 10 years ago for the dry cleaner, photo
processor and printer sectors. Now, 18 states have developed
or are implementing at least one ERP to address challenging
environmental issues in 11 sectors/groups. (See box, right.)
To date, six of those states have completed a full ERP cycle and
generated results demonstrating the success of this approach.
ERP States, by Sector/Group
Animal Feedlots (Small)
Auto Salvage Yards
Auto Body Shops
As Figure 1 (below) illustrates, ERP is an integrated system of:
A Plain-language compliance assistance that promotes pollution
prevention (P2);
A Facility self-assessment and self-certification;
A Agency inspections; and
A Statistically based performance measurement.
Where necessary, regulators also conduct a comprehensive facility
inventory and targeted enforcement actions. These elements work
together to improve performance across a business sector or other
group of pollution sources, while deploying government resources
strategically and efficiently.
Auto Repair Shops
Dry Cleaners
Gas Stations
Oil & Gas Extraction
Facilities
Photo Processors
Printers
Stormwater
Dischargers
Underground Injection
Wells
MN
IN,RI
DE,MD*,ME,NY,
RI,WA
FL*, MD*
MA, Ml, NH,NV
RI,VA,VT
LA
MA
MA, NY,WI
ME,RI
IL
*No longer implementing ERP. Note that MD had one ERP
that covered both auto body and auto repair.
Figure 1. ERP: Interlocking Tools,
Integrated System
The U.S. Environmental Protection Agency (EPA) has actively supported the diffusion of ERP across states
since EPA's Innovation Action Council (IAC) endorsed the approach for "scale up" in 2000. In making its
decision, the IAC considered at least three factors: documented evidence of performance improvements in
Massachusetts' first years of ERP; a favorable evaluation of the
initiative by the National Academy of Public Administration
(NAPA); and the significance of the environmental threat that
can be posed by large groups of small pollution sources. ERP
shows high potential for cost-effectively achieving results with
these small entities, which historically are under- or unregulated.
For instance, six states to date have developed or are planning
ERPs for auto body shops, which number more than 30,000
nationwide and are associated with serious environmental and
health impacts. Since the IAC endorsement, EPA has supported
its state partners with technical assistance and grant funds. In
addition, Massachusetts has received "resource flexibility credit"
through EPA's enforcement program accountability system for the
demonstrated success of the state's ERP for dry cleaners, allowing
Massachusetts to focus more resources on emerging priorities for
environmental improvement.
COMPLIANCE
ASSISTANCE
E-1
-------
How Does an Integrated ERP Work?
An ERP combines several interlocking tools in a cyclical
process to improve overall sector performance. (See Figure
2, below.) Compliance assistance specifies how facilities
should assess their operations and certify compliance,
while agency inspectors document progress against
performance indicators that are linked to self-certification
checklists. Performance data, in turn, inform and improve
the next round of compliance assistance. No two ERPs are
exactly alike, however, because states have adapted this
approach for a wide variety of circumstances. For instance,
many states have successfully implemented ERPs with
voluntary submission of self-certifications, when mandatory
certification was not feasible.
Figure 2. A Typical ERP Cycle
Stepl: Inventory. Identify the myriad small facilities that are
sources of pollution, many of which are often unknown to
regulators.
Step 2: Statistical Baseline Inspections. Conduct random
inspections to accurately measure existing environmental
performance and focus outreach on the biggest problems.
Step 3: Compliance Assistance. Work with trade
associations to create and provide plain-language, user-
friendly assistance that improves compliance and promotes
pollution prevention.
Step 4: Self-Certification. Facilities conduct self-
assessments using a detailed checklist closely linked to
assistance materials. Responsible officials certify to their
facilities' environmental performance on each item. If
necessary, they submit plans to return to compliance.
Step 5: Targeted Follow-Up. Identify potential problem
facilities via certification analysis, and targetthem for
inspections, correspondence or phone calls. Provide
assistance and/or initiate enforcement, as needed.
Step 6: Statistical Post-Certification Inspections. Conduct
random inspections to accurately estimate performance
changes and verify facility certifications.
Step 7: Informed Decision-Making. Assess performance data
and consider whether to adjust compliance assistance or
other strategies directed at the sector or, if sufficient progress
has been made over time, target resources elsewhere.
o
CD
a.
3
o
03
a
CD
CD
3
CD
a.
Purpose of This Report
This report updates the story of ERP,
the implementation of which has grown
substantially in recent years. The report
identifies the states using ERP and the
environmental problems they are seeking to
address, describes the results and benefits state
ERPs are generating, and discusses some of the
new directions being explored within the ERP
community.
ERP States Produce Results
ERP has improved the environmental performance
of businesses in each state that has applied it, and its
measurement system has given regulators credible
evidence of those performance enhancements. Each
of the six states that has completed an ERP cycle
has seen initial average improvements of 5 to 30
percentage points for its top-priority indicators of
performance. (See table on next page, E-3.) What
follows below are highlights of results from these
states.
Environmental Business Practice Indicators (EBPIs).
ERP's key measures are called Environmental
Business Practice Indicators (EBPIs), because they
typically demonstrate facility performance in terms
of both compliance and voluntary best management
practices. For instance, in Delaware, the proportion
of auto body shops complying with disposal
requirements for hazardous waste increased from
66% to 91 % in one year. In a similar time frame,
the voluntary use of "green" solvents among Maine
auto body shops increased from 49% to 97%. These
examples of EBPI improvements, and slightly more
than one third of all observed EBPI improvements,
are statistically significant at a 95% confidence level.
No observed decreases in EBPI performance are
statistically significant at that level.
The Role of Statistics. What is so "significant"
about statistical significance? Regulators often base
decisions upon data from only a sample of facilities,
and usually are not sure how well the data from that
sample represent the whole group. Using statistics,
regulators can understand how certain they can
be about data taken from a random, representative
sample of facilities. For significant performance
improvements like those discussed above, a 95%
confidence level means we know there's at most a
E-2
-------
Sector
Observed Average Indicator Improvement
in First ERP Self-Certification Cycle
_ Avg. Improvement*
(Percentage Points)
Auto Body
Auto Body/Repair
Auto Repair
Dry Cleaners
Photo Processors
Printers
DE
ME
Rl
MD
FL
MA
MA
MA
30
10
21
12
7
5
12
13
*Average of performances changes for each ERP's indicators among randomly sampled
facilities, as observed by inspectors in the first ERP cycle. Includes all observed
changes, regardless of significance level. Sector performance typically continues to
improve in subsequent ERP cycles, then is sustained at high levels. For ERPs with vol-
untary certification (DE, MD, ME and Rl), random samples were drawn from a//facilities
in the ERP inventory, notjustvolunteers. See full report for more detail.
5% chance that we would be mistaken in
saying that there was a change in performance
for the group as a whole. Other observed
changes may indeed point to genuine
changes in the whole group, but we cannot
be as certain that they occurred—oftentimes
because states are basing their inferences upon
small samples of the population, allowing
only large observed changes to be deemed
"significant."
Across-the-Board Performance Improvements.
Changes that are not significant can also
be instructive. Data from ERPs have
consistently shown a net performance gain
across all EBPIs. Most indicators have
risen while some have declined—but the
overall number and size of the increases has
substantially exceeded the number and size
of declines. Furthermore, even where there
are decreases, ERP provides the information
regulators need to target their compliance assistance and enforcement efforts on that small proportion of
lagging indicators.
Sector Snapshots. Aside from examining
how facilities fare on individual EBPIs, ERP
states have also developed higher-level
measures that concisely communicate sector
performance. For instance, Florida inspectors
found that the proportion of "straight-A" auto
repair facilities—those with no violations of
any kind—had risen 17 percentage points after
two rounds of self-certification. Furthermore,
Massachusetts has developed a "group
compliance score," a measure showing
the extent to which a sector's facilities are
achieving compliance-oriented EBPIs. In
Massachusetts, each sector's score has risen
from ERP's inception through the most recent
inspection results, with photo processors
showing a dramatic increase from 57%
in 1997 to 98% just five years later. (See
Figure 3, right.) Results from the other two
sectors suggest that ERP may not always show
substantial improvements right away when
performance is already at a relatively high
level. The dry cleaners case further shows
the potential for improvement over time, after
which a very high performance level can be
sustained.
Figure 3. Long-Term Compliance
Improvements in Massachusetts ERP Sectors
100%
20%
10%
0%
DryClGaners
•Q— Photo Processors
-•— Printers
1997 1998 1999 2000 2001 2002 2003
Notes: (1) "Group compliance score" is a measure of the extent to which
facilities are achieving compliance-related EBPIs, as observed by inspectors
during random visits to facilities. A score of 80%, for instance, would mean
that, on average, each facility is achieving 80% of the indicators that apply to
it. The group compliance score is distinct from traditional compliance rates, in
that it tracks performance only on priority compliance practices and reflects
varying degrees of compliance atindividual facilities. (2) Improvements over
time have not been evaluated for statistical significance. (3) Graph reflects
most recent available data. Massachusetts has decreased inspection
frequency over time for these sectors because of their trends toward sustaining
high performance levels. The next round of random inspections is anticipated
later in 2007, for the dry cleaner sector.
E-3
-------
Estimates of Environmental Outcomes. Massachusetts has experimented with developing measures that
can demonstrate environmental outcomes that result from performance changes. The state has found that
estimating environmental outcomes can provide valuable insights into program success. For instance,
Massachusetts has calculated that its dry cleaners reduced annual emissions and waste of the toxic chemical
perchloroethylene by approximately 30% in the first two years of ERR
Why Do These Performance
Improvements Happen?
States believe that ERP's mix of compliance
assistance, self-certification and agency
verification drives facilities to hold
themselves more accountable and gives
them the capability and incentive to improve
performance. For instance, regulators
are often surprised by the number of
facilities that voluntarily report one or more
non-compliance items on their first self-
certification form and submit plans to return
to compliance. (See box, right.) In fact, the
submission rate of return-to-compliance
plans has become yet another important and
credible measure of ERR success for many
states.
Why Are States Adopting ERP?
Aside from the across-the-board evidence of overall performance improvements, state regulators cite several
other reasons for adopting ERP.
A Most importantly, available evidence suggests that ERP achieves performance at least as good as that of
traditional compliance assurance approaches, at substantially less cost over time.
A State regulators find that rich ERP data sets can help them target resources more efficiently.
"I'm Doing It Wrong over Here!!"
In Rhode Island, 20% of all auto body shops reported
themselves out of compliance with at least one
requirement, and submitted plans to return to
compliance within a reasonable time frame. Typically,
self-reported non-
compliance decreases
in later years of ERP,
consistent with inspector
findings of higher sustained
sector performance. Over
time, this combination of
accountability and sustained
performance can decrease
the need for enforcement.
Baltimore Residents
Appreciate ERP
Maryland's ERP focused on
informal vehicle repair facilities
in an environmental-justice
neighborhood in Baltimore. A
post-ERP survey showed that
nearby residents recognized and
appreciated the project's visible
environmental improvements.
Improved facility accountability may reduce the long-term need for
resource-intensive enforcement actions.
Statistical approaches and technology both offer substantial
economies of scale when ERP is applied to large universes of
facilities.
Agencies report that ERP helps them meet stakeholder demands to
measure how well regulators' actions achieve goals of increasing
compliance and achieving environmental improvements.
Many businesses support ERP because it explains their obligations
in clear terms and because it "levels the playing field" among
facilities in a sector.
The often visible improvements and increased transparency of ERP
provide value to the public. (See box, left.)
E-4
-------
ERP's Future: Further Improvement, Experimentation and Growth Expected
In the coming years, states will report first-time results for several new ERP sectors, including underground storage
tanks, auto salvage yards and small animal feedlots. An EPA-funded, state-led initiative to develop common ERP-type
performance indicators for auto body shops and small quantity generators of hazardous waste is also well underway. Each
state participating in this "Common Measures Project" will track indicators for at least one of these regulated groups using
ERP statistical approaches. The project should allow greater comparability of environmental performance levels across
participating states—whether or not those states are implementing an ERP for those groups. This information will help states
choose the most effective and efficient compliance assurance strategies to meet their needs.
Further, states continue to adapt ERP to address a greater array of environmental issues. For instance, ERP states are pursuing
new applications based on the geographic boundaries of environmental problems—such as watershed-based targeting to
reduce polluted stormwater runoff. When appropriate, states also are implementing other promising variations on ERP, such
as applying individual ERP tools to address problems when a full ERP may not be feasible. Massachusetts calculates that
one application of such "ERP-like" approaches has prevented the yearly wastewater discharge of several hundred pounds of
mercury from dentists' offices. Another such initiative by the state has substantially improved emissions-testing compliance at
Massachusetts gas stations, assuring the capture of thousands of tons of ozone-forming compounds annually.
Finally, awareness of and support for ERP is expected to grow as a result of states forming the new States ERP Consortium.
The Consortium, organized this year as a "forum" of the Environmental Council of the States (ECOS), currently represents
more than a third of all states. (See Figure 4, below.) The Consortium includes states currently using ERP and states learning
how to use ERP to address priority environmental problems. EPA's National Center for Environmental Innovation (NCEI)
is on the Consortium's steering committee and is providing contractor support. The Consortium has four key goals: (1)
communicating results in order to build stakeholder support; (2) sharing information among practitioners; (3) promoting ERP
as a proven compliance strategy and expanding support within EPA; and (4) improving and disseminating tools to further
streamline key aspects of ERP, such as automation and measurement. EPA is committed to working with the Consortium in
the future to continue evaluating the many applications of ERP, and to communicate the results of these efforts.
Figure 4. Growing ERP Community Represented by a New Consortium of States
ERP Implementers
Learning States
Consortium Members
AK
"Not currently implementing ERP.
For a copy of the full report (available Summer 2007) and more information on ERP, visit www.epa.gov/erp.
E-5
------- |