EPA100-R-07-007
                 May 2007
Implementing the
Environmental
Results Program
          NCEI
          NATIONAL CENTER FOR
          ENVIRONMENTAL INNOVATION

-------
Executive  Summary
The Environmental Results Program (ERP) is an innovative approach to improving the environmental performance
of various business sectors and other groups with large numbers of small facilities. While individual facilities
within these groups may release small amounts of pollution, their aggregate impact can be significant.  The
Massachusetts Department of Environmental Protection developed ERP 10 years ago for the dry cleaner, photo
processor and printer sectors. Now, 18 states have developed
or are implementing at least one ERP to address challenging
environmental issues in 11 sectors/groups. (See box, right.)
To date, six of those states have completed a full ERP cycle and
generated results demonstrating the success of this approach.
                                                                 ERP States, by Sector/Group
                                                              Animal Feedlots (Small)
                                                              Auto Salvage Yards
                                                              Auto Body Shops
As Figure 1  (below) illustrates, ERP is an integrated system of:
A  Plain-language compliance assistance that promotes pollution
   prevention (P2);
A  Facility self-assessment and self-certification;
A  Agency inspections; and
A  Statistically based performance measurement.

Where necessary, regulators also conduct a comprehensive facility
inventory and targeted enforcement actions. These elements work
together to improve performance across a business sector or other
group of pollution sources, while deploying government resources
strategically and efficiently.
                                                              Auto Repair Shops
                                                              Dry Cleaners
                                                              Gas Stations
                                                              Oil & Gas Extraction
                                                              Facilities
                                                              Photo Processors
                                                              Printers
                                                              Stormwater
                                                              Dischargers
                                                              Underground Injection
                                                              Wells
MN
IN,RI
DE,MD*,ME,NY,
RI,WA
FL*, MD*
MA, Ml, NH,NV
RI,VA,VT
LA

MA
MA, NY,WI
ME,RI

IL
                                                             *No longer implementing ERP. Note that MD had one ERP
                                                             that covered both auto body and auto repair.
Figure 1.  ERP: Interlocking Tools,
         Integrated System
The U.S. Environmental Protection Agency (EPA) has actively supported the diffusion of ERP across states
since EPA's Innovation Action Council (IAC) endorsed the approach for "scale up" in 2000.  In making its
decision, the IAC considered at least three factors: documented evidence of performance improvements in
                                         Massachusetts' first years of ERP; a favorable evaluation of the
                                         initiative by the National Academy of Public Administration
                                         (NAPA); and the significance of the environmental threat that
                                         can be posed by large groups of small pollution sources.  ERP
                                         shows high potential for cost-effectively achieving results with
                                         these small entities, which historically are under- or unregulated.
                                         For instance, six states to date have developed or are planning
                                         ERPs for auto body shops, which number more than 30,000
                                         nationwide and are associated  with serious environmental and
                                         health impacts. Since the IAC endorsement, EPA has supported
                                         its state partners with technical assistance and grant funds. In
                                         addition, Massachusetts has received "resource flexibility credit"
                                         through EPA's enforcement program accountability system for the
                                         demonstrated success of the state's ERP for dry cleaners, allowing
                                         Massachusetts to focus more resources on emerging priorities for
                                         environmental  improvement.
                  COMPLIANCE
                  ASSISTANCE
                                             E-1

-------
How Does an  Integrated  ERP Work?
An ERP combines several interlocking tools in a cyclical
process to improve overall sector performance.  (See Figure
2, below.) Compliance assistance specifies how facilities
should assess their operations and certify compliance,
while agency inspectors document progress against
performance indicators that are linked to self-certification
checklists.  Performance data, in turn, inform and improve
the next round of compliance assistance.  No two ERPs are
exactly alike, however,  because states have adapted this
approach for a wide variety of circumstances. For instance,
many states have successfully implemented ERPs with
voluntary submission of self-certifications,  when mandatory
certification was not feasible.

         Figure 2. A Typical ERP Cycle

 Stepl: Inventory. Identify the myriad small  facilities that are
 sources of pollution, many of which are often unknown to
 regulators.
 Step 2: Statistical Baseline Inspections. Conduct random
 inspections to accurately measure existing environmental
 performance and focus outreach on the biggest problems.
 Step 3: Compliance Assistance. Work with trade
 associations to create and provide plain-language, user-
 friendly assistance that improves compliance and promotes
 pollution prevention.
 Step 4: Self-Certification. Facilities conduct self-
 assessments using a detailed checklist closely linked to
 assistance materials.  Responsible officials certify to their
 facilities' environmental performance on each item. If
 necessary, they submit plans to return to compliance.
 Step 5: Targeted Follow-Up. Identify potential problem
 facilities via certification analysis, and targetthem for
 inspections, correspondence or phone calls. Provide
 assistance and/or initiate enforcement, as needed.
 Step 6: Statistical Post-Certification Inspections. Conduct
 random inspections to accurately estimate performance
 changes and verify facility certifications.
 Step 7: Informed Decision-Making. Assess performance data
 and consider whether to adjust compliance assistance or
 other strategies directed at the sector or, if sufficient progress
 has been made over time, target resources elsewhere.
o
CD
a.
3
o
03
a
CD
CD
3
CD
a.
        Purpose of This Report
        This report updates the story of ERP,
        the implementation of which has grown
        substantially in recent years. The report
        identifies the states using ERP and the
        environmental problems they are seeking to
        address, describes the results and benefits state
        ERPs are generating, and discusses some of the
        new directions being explored within the ERP
        community.
ERP States Produce Results

ERP has improved the environmental performance
of businesses in each state that has applied it, and its
measurement system has given regulators credible
evidence of those performance enhancements.  Each
of the six states that has completed an ERP cycle
has seen initial average improvements of 5 to 30
percentage points for its top-priority indicators of
performance. (See table on next page, E-3.) What
follows below are highlights of results from these
states.

Environmental Business Practice Indicators (EBPIs).
ERP's key measures are called Environmental
Business Practice Indicators (EBPIs), because they
typically demonstrate facility performance in terms
of both compliance and voluntary best management
practices. For instance, in Delaware, the proportion
of auto body shops complying with disposal
requirements for hazardous waste increased from
66% to 91 % in one year. In a similar time frame,
the voluntary use of "green" solvents among Maine
auto body shops increased from 49% to 97%.  These
examples of EBPI improvements, and slightly more
than one third of all observed EBPI improvements,
are statistically significant at a 95% confidence level.
No observed decreases in EBPI performance are
statistically significant at that level.

The Role of Statistics. What is so "significant"
about statistical significance?  Regulators often base
decisions upon data from only a sample of facilities,
and usually are not sure how well the data from that
sample represent the whole group. Using statistics,
regulators can understand how certain they can
be about data taken from a random, representative
sample of facilities. For significant performance
improvements like those discussed above, a 95%
confidence level means we know there's at most a
                                                     E-2

-------
 Sector
Observed Average Indicator Improvement
   in First ERP Self-Certification Cycle
           _              Avg. Improvement*
                          (Percentage Points)
Auto Body
Auto Body/Repair
Auto Repair
Dry Cleaners
Photo Processors
Printers
DE
ME
Rl
MD
FL
MA
MA
MA
30
10
21
12
7
5
12
13
*Average of performances changes for each ERP's indicators among randomly sampled
facilities, as observed by inspectors in the first ERP cycle. Includes all observed
changes, regardless of significance level. Sector performance typically continues to
improve in subsequent ERP cycles, then is sustained at high levels. For ERPs with vol-
untary certification (DE, MD, ME and Rl), random samples were drawn from a//facilities
in the ERP inventory, notjustvolunteers.  See full report for more detail.
5% chance that we would be mistaken in
saying that there was a change in performance
for the group as a whole. Other observed
changes may indeed  point to genuine
changes in the whole group, but we cannot
be as certain that they occurred—oftentimes
because states are basing their inferences upon
small samples of the  population, allowing
only large observed changes to be deemed
"significant."

Across-the-Board Performance Improvements.
Changes that are not significant can also
be instructive.  Data from  ERPs have
consistently shown a net performance gain
across all EBPIs.  Most indicators have
risen while some have declined—but the
overall number and  size of the  increases  has
substantially exceeded the number and size
of declines.  Furthermore,  even where there
are decreases, ERP provides the information
regulators need to target their compliance assistance and enforcement efforts on that small proportion of
lagging indicators.
Sector Snapshots.  Aside from examining
how facilities fare on individual EBPIs, ERP
states have also developed higher-level
measures that concisely communicate sector
performance. For instance, Florida inspectors
found that the proportion of "straight-A" auto
repair facilities—those with no violations of
any kind—had risen 17  percentage points after
two rounds of self-certification. Furthermore,
Massachusetts has  developed a "group
compliance score," a measure showing
the extent to which a sector's facilities are
achieving compliance-oriented EBPIs. In
Massachusetts, each sector's score has risen
from ERP's inception through the  most recent
inspection results, with photo  processors
showing a dramatic increase from 57%
in 1997 to 98% just five years later.  (See
Figure 3,  right.)  Results  from the other two
sectors suggest that ERP may not always show
substantial improvements right away when
performance is already at a relatively high
level. The dry cleaners case further shows
the potential for improvement over time, after
which a very high performance level can be
sustained.
                                                   Figure 3.  Long-Term Compliance
                                            Improvements in Massachusetts ERP Sectors
                                            100%
                                             20%
                                             10%
                                              0%
                             DryClGaners
                        •Q— Photo Processors
                        -•— Printers
                                                   1997   1998   1999    2000   2001    2002   2003

                                            Notes: (1) "Group compliance score" is a measure of the extent to which
                                            facilities are achieving compliance-related EBPIs, as observed by inspectors
                                            during random visits to facilities. A score of 80%, for instance, would mean
                                            that, on average, each facility is achieving 80% of the indicators that apply to
                                            it. The group compliance score is distinct from traditional compliance rates, in
                                            that it tracks performance only on priority compliance practices and reflects
                                            varying degrees of compliance atindividual facilities. (2) Improvements over
                                            time have not been evaluated for statistical significance. (3) Graph reflects
                                            most recent available data. Massachusetts has decreased inspection
                                            frequency over time for these sectors because of their trends toward sustaining
                                            high performance levels. The next round of random inspections is anticipated
                                            later in 2007, for the dry cleaner sector.
                                                        E-3

-------
Estimates of Environmental Outcomes. Massachusetts has experimented with developing measures that
can demonstrate environmental outcomes that result from performance changes. The state has found that
estimating environmental outcomes can provide valuable insights into program success. For instance,
Massachusetts has calculated that its dry cleaners reduced annual emissions and waste of the toxic chemical
perchloroethylene by approximately 30%  in the first two years of ERR
Why Do These Performance

Improvements Happen?

States believe that ERP's mix of compliance
assistance, self-certification and agency
verification drives facilities to hold
themselves more accountable and gives
them the capability and incentive to improve
performance. For instance, regulators
are often surprised by the number of
facilities that voluntarily report one or more
non-compliance items on their first self-
certification form and submit plans to return
to compliance. (See box, right.) In fact, the
submission rate of return-to-compliance
plans  has become yet another  important and
credible measure of ERR success for many
states.

Why Are States Adopting ERP?

Aside from the across-the-board evidence of overall performance improvements, state regulators cite several
other reasons for adopting ERP.
A  Most importantly, available evidence suggests that ERP achieves performance at least as good as that of
   traditional compliance assurance approaches, at substantially less cost over time.
A  State regulators find that rich ERP data sets can help them target resources more efficiently.
          "I'm Doing It Wrong over Here!!"
          In Rhode Island, 20% of all auto body shops reported
          themselves out of compliance with at least one
          requirement, and submitted plans to return to
          compliance within a reasonable time frame. Typically,
          self-reported non-
          compliance decreases
          in later years of ERP,
          consistent with inspector
          findings of higher sustained
          sector performance. Over
          time, this combination of
          accountability and sustained
          performance can  decrease
          the need for enforcement.
   Baltimore Residents
   Appreciate ERP
   Maryland's ERP focused on
   informal vehicle repair facilities
   in an environmental-justice
   neighborhood in Baltimore. A
   post-ERP survey showed that
   nearby residents recognized and
   appreciated the project's visible
   environmental improvements.
Improved facility accountability may reduce the long-term need for
resource-intensive enforcement actions.

Statistical approaches and technology both offer substantial
economies of scale when ERP is applied to large universes of
facilities.

Agencies report that ERP helps them meet stakeholder demands to
measure how well regulators' actions achieve goals of increasing
compliance and achieving environmental  improvements.

Many businesses support ERP because  it explains their obligations
in clear terms and because it "levels the playing field" among
facilities in a sector.

The often visible improvements and increased transparency of ERP
provide value to the public. (See box, left.)
                                            E-4

-------
ERP's Future:  Further Improvement, Experimentation and Growth Expected

In the coming years, states will report first-time results for several new ERP sectors, including underground storage
tanks, auto salvage yards and small animal feedlots.  An EPA-funded, state-led initiative to develop common ERP-type
performance indicators for auto body shops and small quantity generators of hazardous waste is also well underway.  Each
state participating in this "Common Measures Project" will track indicators for at least one of these regulated groups using
ERP statistical approaches. The project should allow greater comparability of environmental performance levels across
participating states—whether or not those states are  implementing an ERP for those groups.  This information will help states
choose the most effective and efficient compliance assurance strategies to meet their needs.

Further, states continue to adapt ERP to address a greater array of environmental issues. For instance, ERP states are pursuing
new applications based on the geographic boundaries of environmental problems—such  as watershed-based targeting to
reduce polluted stormwater runoff. When appropriate, states  also are implementing other promising variations on ERP, such
as applying individual  ERP tools to address problems when a full ERP may not be  feasible. Massachusetts calculates that
one application of such "ERP-like" approaches has prevented  the yearly wastewater discharge of several hundred pounds of
mercury from dentists' offices. Another such initiative by the state has substantially improved emissions-testing compliance at
Massachusetts gas stations, assuring the capture of thousands  of tons of ozone-forming compounds annually.

Finally, awareness of and support  for ERP is expected to grow as a result of states  forming the new States ERP Consortium.
The Consortium, organized this year as a "forum" of the Environmental Council of the States (ECOS), currently represents
more than a third of all states. (See Figure 4, below.)  The Consortium includes states currently using ERP and states learning
how to use  ERP to address priority environmental  problems.  EPA's National Center for Environmental Innovation (NCEI)
is on the Consortium's steering committee and is providing contractor support. The Consortium has four key goals: (1)
communicating results in order to build stakeholder support; (2) sharing information among  practitioners; (3) promoting ERP
as a proven compliance strategy and expanding support within EPA; and (4) improving and disseminating tools to further
streamline key aspects of ERP, such as automation and measurement. EPA is committed to working with the Consortium in
the future to continue evaluating the many applications of ERP, and to communicate the results of these efforts.

Figure 4.  Growing ERP Community Represented by a New Consortium  of States
                                                                                         ERP Implementers

                                                                                         Learning States

                                                                                         Consortium Members
               AK
                                                                                     "Not currently implementing ERP.
For a copy of the full report (available Summer 2007) and more information on ERP, visit www.epa.gov/erp.
                                                    E-5

-------