i
55
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
EPA Must Improve
Oversight of State
Enforcement
Report No. 12-P-0113
Decembers, 2011
Scan this mobile
code to learn more
about the EPA OIG.
-------
Report Contributors: Dan Engelberg
Kathlene Butler
Charles Brunton
Anthony Chirigotis
Julie Hamann
Tiffme Johnson-Davis
Genevieve Soule
OIG Report No. 12-P-0113, EPA Must Improve Oversight of State Enforcement, was reissued on
January 30, 2012. The original version of the report, issued December 9, 2011, contained an
administrative error in appendix C. On pages 38 and 39, the data underlying the
"RCRA Subtitle C (avg 2003-2009)" column heading, for both the "SNC" and "Penalty"
subheadings, contained a transposition error covering FY2007-2009. The OIG corrected this
error in the appendix. The correction also led to a change in our statement concerning North
Dakota's penalty assessments on page 15, and OIG responses 6 and 18 on pages 45 and 53. The
changes do not affect our findings and recommendations.
Abbreviations
CAA Clean Air Act
CMS Compliance Monitoring Strategy
CWA Clean Water Act
ECHO Enforcement and Compliance History Online
ECOS Environmental Council of the States
EPA U.S. Environmental Protection Agency
FTE Full-time equivalent
FY Fiscal year
GAO U.S. Government Accountability Office
HPV High-priority violation
NPDES National Pollutant Discharge Elimination System
OECA Office of Enforcement and Compliance Assurance
OIG Office of Inspector General
OTIS Online Tracking Information System
RCRA Resource Conservation and Recovery Act
SNC Significant noncompliance
SRF State Review Framework
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
e-mail: OIG Hotline@epa.gov write: EPA Inspector General Hotline
phone: 1-888-546-8740 1200 Pennsylvania Avenue NW
fax: 202-566-2599 Mailcode 2431T
online: http://www.epa.gov/oig/hotline.htm Washington, DC 20460
-------
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
12-P-0113
December 9, 2011
Why We Did This Review
We sought to determine
(1) whether the U.S.
Environmental Protection
Agency (EPA) set clear
national performance
benchmarks for state
enforcement programs, and
(2) to what extent EPA
headquarters holds regions
accountable and supports them
I to ensure that all state
enforcement programs protect
human health and the
environment. The scope of our
I review included selected
programs under three statutes:
Clean Water Act, Clean Air
Act, and Resource
Conservation and Recovery
Act.
Background
EPA is the steward of national
environmental protection, but
states serve as the first line of
enforcement in most cases. Past
reviews identified widespread
problems with state
enforcement.
EPA Must Improve Oversight of State Enforcement
For further information, contact
our Office of Congressional and
Public Affairs at (202) 566-2391.
The full report is at:
www.epa.gov/oiq/reports/2012/
20111209-12-P-0113.pdf
What We Found
EPA does not administer a consistent national enforcement program. Despite
efforts by the Office of Enforcement and Compliance Assurance (OECA) and the
EPA regions to improve state enforcement performance, state enforcement
programs frequently do not meet national goals and states do not always take
necessary enforcement actions. State enforcement programs are underperforming:
EPA data indicate that noncompliance is high and the level of enforcement is
low. EPA does not consistently hold states accountable for meeting enforcement
standards, has not set clear and consistent national benchmarks, and does not act
effectively to curtail weak and inconsistent enforcement by states.
OECA has made efforts to improve state performance and oversight consistency,
but EPA does not manage or allocate enforcement resources nationally to allow it
to intervene in states where practices result in significantly unequal enforcement.
As a result, state performance remains inconsistent across the country, providing
unequal environmental benefits to the public and an unlevel playing field for
regulated industries. By establishing stronger organizational structures, EPA can
directly implement a national enforcement strategy that ensures all citizens have,
and industries adhere to, a baseline level of environmental protection. EPA could
make more effective use of its $372 million in regional enforcement full-time
equivalents by directing a single national workforce instead of 10 inconsistent
regional enforcement programs.
What We Recommend
We recommend that EPA establish clear national lines of authority for
enforcement that include centralized authority over resources; cancel outdated
guidance and policies, and consolidate and clarify remaining enforcement
policies; establish clear benchmarks for state performance; and establish a clear
policy describing when and how EPA will intervene in states, and procedures to
move resources to intervene decisively, when appropriate, under its escalation
policy.
Based on EPA's suggestion in its response to our draft report, we recommend
that EPA develop a state performance scorecard. EPA did not agree with
recommendation 1, agreed with recommendations 2 through 4, and neither agreed
nor disagreed with recommendation 5. All recommendations are unresolved
pending EPA's corrective action plan.
-------
I UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
% WASHINGTON, D.C. 20460
^t PRO^
THE INSPECTOR GENERAL
December 9, 2011
MEMORANDUM
SUBJECT: EPA Must Improve Oversight of State Enforcement
Report No. 12-P-0113
FROM: Arthur A. Elkins, J
Inspector General
TO: Bob Perciasepe
Deputy Administrator
Cynthia Giles
Assistant Administrator for Enforcement and Compliance Assurance
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed-upon
actions, including milestone dates. Your response will be posted on the OIG's public website,
along with our memorandum commenting on your response. Your response should be provided
as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the
Rehabilitation Act of 1973, as amended. The final response should not contain data that you do
not want to be released to the public; if your response contains such data, you should identify the
data for redaction or removal. We have no objections to the further release of this report to the
public. We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Dan Engelberg,
Director for Enforcement and Water Program Evaluations, at (202) 566-0830 or
Engelberg.Dan@epa.gov.
-------
EPA Must Improve Oversight of State Enforcement
12-P-0113
Table of Contents
Chapters
1 Introduction 1
Purpose 1
Background 1
Noteworthy Achievements 3
Scope and Methodology 3
2 EPA Must Improve Oversight of State Enforcement 6
OECA and Regions Have Made Efforts to Improve State Performance 6
State Enforcement Programs Are Underperforming 8
OECA Does Not Have Sole Authority Over Regional Enforcement 11
EPA Headquarters Does Not Consistently Hold Regions Accountable 11
EPA Does Not Shift Resources to Intervene in Problem States 17
Conclusions 19
Recommendations 21
Agency Comments and OIG Evaluation 21
Status of Recommendations and Potential Monetary Benefits 25
Appendices
A Details on Scope and Methodology 26
B Prior GAO and EPA OIG Assessments 32
C State Performance Analysis and Results 35
D EPA Comments on Draft Report and OIG Responses 41
E EPA Attachments to Draft Comment Memorandum and OIG Responses.... 53
F Distribution 74
-------
Chapter 1
Introduction
Purpose
As documented in past reviews by the U.S. Environmental Protection Agency
(EPA) Office of Inspector General (OIG) and others, inconsistent state
enforcement of environmental statutes has been a persistent and widespread
problem. We sought to determine whether EPA has set clear national performance
benchmarks for state enforcement programs. We also sought to determine to what
extent EPA headquarters supports regional enforcement activities and holds
regions accountable for enforcement to ensure that all states protect human health
and the environment.
Background
The Clean Air Act (CAA), Clean Water Act (CWA), and the Resource
Conservation and Recovery Act (RCRA) establish programs to protect Americans
from environmental pollution. EPA is responsible for ensuring that environmental
goals are met and that programs are consistently applied nationwide. While EPA
serves as the steward of many national environmental policies, it relies on states
to do the bulk of environmental enforcement. At EPA, the Office of Enforcement
and Compliance Assurance (OECA) is the primary entity setting national
enforcement policy and monitoring regional oversight of state programs.
National consistency ensures that all Americans live in states that meet minimum
environmental standards. Environmental pollution often does not remain
contained within state borders, so pollution in one state may cause damage in
another. National consistency is also important because it levels the playing field
among regulated entities, ensuring that those regulated facilities that fail to
comply with the law do not have an unfair economic advantage over their law-
abiding competitors. EPA emphasizes that national consistency exists in
conjunction with flexibility to allow national policy and guidance to
accommodate differences in regions and states. For example, the CAA allows a
state to submit implementation plans as long as EPA agrees that the state's
program meets the minimum requirements of the CAA. States may also have
programs that are more stringent than required by federal rules.
Under EPA's organizational structure, regional offices and headquarters program
offices like OECA all report directly to the EPA Administrator. As such, OECA
does not have sole authority over regional enforcement activities.
12-P-0113
-------
EPA Can Authorize States to Act on Its Behalf
The CAA, CWA, and RCRA provide EPA the authority to authorize states to
enact and enforce some programs if states request authorization and EPA
approves their basic program. Most states have acquired this authority. The EPA
Administrator delegated the responsibility for approving state authorization
requests to the Regional Administrators. This arrangement establishes a chain of
authority for enforcement that begins with Congress, extends to the EPA
Administrator, then jointly to the OECA Assistant Administrator and EPA
Regional Administrators, and finally to authorized state environmental agencies.
EPA Oversees States and Retains Independent Enforcement
Authority
Although most states have received authorization to administer most programs
under the CAA, CWA, and RCRA, EPA retains a vital role in ensuring that states
implement nationally consistent programs that meet federal requirements.
According to federal regulations, EPA should provide appropriate oversight so
that it knows when states fail to meet their federally mandated enforcement
commitments. As such, EPA must monitor states to keep apprised of their
enforcement activities, a task largely left to EPA regions.
By authorizing states to enforce portions of these acts, EPA does not forfeit its
authority to continue to conduct its own inspections and take action against
violators, particularly in cases where violations are widespread or related to a
national priority. Not only can EPA engage in independent enforcement activities
in states, but it can also take action against a facility when it determines a state
either did not act or did not take strong enough action against the facility for a
violation (generally referred to as "over-filing"). Ultimately, EPA retains the
authority to withdraw state program authority if a state is not taking appropriate
enforcement actions against violators.
EPA Strives for Shared Accountability With States
EPA Administrator Lisa P. Jackson identified the need to build strong state and
tribal partnerships as one of the Agency's seven priorities. In her October 2009
message to the House Transportation and Infrastructure Committee, she said that
the enforcement performance of states has varied greatly. The Administrator
stated in the Agency's seven priorities that strong partnerships and accountability
are very important. She said EPA must do its part to support state and tribal
capacity and, through strengthened oversight, ensure that EPA delivers programs
consistently, nationwide.
In a 1997 issue paper, Cynthia Giles, then Region 3 Enforcement Director and
currently EPA's Assistant Administrator for OECA, noted that it is EPA's job to
oversee what states are doing, to ensure that all citizens have equal opportunity to
12-P-0113
-------
enjoy the benefits of good health and a clean environment.1 As one of EPA's
2011 enforcement goals, OECA set out to reset EPA's relationship with states, to
better ensure that the Agency and states both meet their environmental protection
commitments.2 OECA intends to achieve this goal through shared accountability
and strengthened oversight. OECA is in the process of establishing new models
for shared accountability and strengthened oversight in the CWA program
through its 2009 Clean Water Act Action Plan. It intends to pursue similar
initiatives for the CAA and RCRA.
Noteworthy Achievements
EPA has taken a number of steps to improve oversight of state enforcement. In
2004, EPA and others established the State Review Framework (SRF), a national
system for reviewing state enforcement performance with respect to the CAA,
CWA, and RCRA. In 2009, OECA began developing a revised CWA reporting
procedure, and developed the Clean Water Act Action Plan. We present more
information on these achievements in chapter 2.
Scope and Methodology
We conducted this evaluation in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform our work to
obtain sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and conclusions
based on our evaluation objectives. We conducted our evaluation from September
2010 to July 2011.
We evaluated EPA's performance under three environmental laws to determine
(1) whether EPA has set clear national performance benchmarks for state
enforcement programs, and (2) to what extent EPA headquarters holds regions
accountable and supports them in their activities. We assessed EPA oversight of
state enforcement under the CAA Title V program, the CWA National Pollutant
Discharge Elimination System (NPDES) program, and the RCRA Subtitle C
program. We assessed EPA oversight for all 50 U.S. states and across all 10 EPA
regions.
To answer our objectives, we looked to several data sources, including EPA
enforcement data, available SRF reports, a 10-region survey, and testimonial and
documentary evidence gathered in interviews with seven states, EPA headquarters
personnel, and enforcement personnel in six regional offices.
1 Cynthia Giles, Region 3 Enforcement Director, "Aiming Before We Shoot," 1997.
2 "EPA Enforcement Goals," http://www.epa.gov/compliance/data/planning/initiatives/goals.html.
12-P-0113
-------
To understand whether state enforcement activities varied over time and
geographically, we evaluated the enforcement data reported to EPA in fiscal years
(FYs) 2003-2009. After consultation with OECA, we identified three key
indicators of state enforcement activity: (1) the percent of facilities states
inspected each year, (2) the percent of inspections that resulted in a state
identifying significant noncompliance or a high-priority violation, and (3) the
percent of violations that resulted in the state issuing a formal enforcement action
with a penalty.
Each of these metrics comes with several caveats about the underlying data.
(Many of these caveats are explained in appendix A, and were previously reported
by OIG.3) However, the OIG conducted a data reliability assessment and
determined that utilizing the metrics offer a valid method for comparing state
activities that took into consideration several key factors. Among them:
• The metrics factor in the size of a state's regulated universe by using
percentage of facilities inspected.
• Because we drew the metrics and their values from EPA databases, they
provide the same information the public sees when viewing state
enforcement performance on EPA websites.4
• The three metrics represent two priority issues EPA identified as national
weaknesses in its initial SRF reviews.
• The metrics allow for state-to-state performance comparison based on
priority issues to EPA. However, without corroboration they do not offer
reliable details about individual state performance.
Over the 7-year period FYs 2003-2009, EPA enforcement data for these metrics
show that some states performed well across the board, some states performed
well in one or two statutes, and some states performed poorly across the board.
Over the same period, state performance remained relatively consistent, though a
few states improved over the period and a few others declined. To better
understand state performance issues, we gathered additional evidence in states
that ranked in the bottom quartile for two or three statutes.
Our in-depth analysis focused on states for which EPA data showed (1) persistent
problems meeting national enforcement goals related to the SRF, (2) states
3 See, for example, EPA Needs to Improve Its Recording and Reporting of Fines and Penalties, Report No. 10-P-
0077; EPA Could Improve RCRAInfo Data Quality and System Development, Report No. 1 l-P-0096; ECHO Data
Quality Audit - Phase I Results: The Integrated Compliance Information System Needs Security Controls to Protect
Significant Non-Compliance Data, Report No. 09-P-0226; and ECHO Data Quality Audit - Phase 2 Results: EPA
Could Achieve Data Quality Rate With Additional Improvements, Report No. 10-P-0230.
4 The public database (Enforcement and Compliance History Online, or ECHO) does not provide data on all three
metrics for all years we included here.
12-P-0113
-------
showing recently improved performance based on the three metrics we chose, or
(3) states showing comparatively high performance. From this analysis, we chose
to interview the states of South Carolina, Illinois, Louisiana, Iowa, Colorado,
North Dakota, and Alaska, and enforcement officials in the associated EPA
Regions, 4, 5, 6, 7, 8, and 10.
Appendix A provides additional details on the scope and methodology, and
appendix B presents information on prior U.S. Government Accountability Office
(GAO) and EPA OIG assessments related to state enforcement issues.
12-P-0113
-------
Chapter 2
EPA Must Improve Oversight of State Enforcement
EPA does not administer a consistent national enforcement program. Despite
OECA efforts to improve state enforcement performance, state enforcement
programs frequently do not meet national goals, and states do not always take
necessary enforcement actions. OECA, the EPA office responsible for
enforcement, does not have sole authority over a national civil enforcement
workforce or state oversight. OECA cannot set clear and consistent benchmarks
for state enforcement performance, or assure consistent oversight of state
enforcement. EPA does not allocate enforcement resources according to the
enforcement workload or high-priority state enforcement problems. As a result,
EPA's enforcement program cannot assure equal and sufficient protection of
human health and the environment to all U.S. citizens or consistent enforcement
of regulated entities.
OECA and Regions Have Made Efforts to Improve State Performance
In response to internal and external critiques of EPA's oversight of state
enforcement activities, OECA has made efforts to improve state performance and
standardize EPA oversight by implementing the SRF, and through several other
headquarters efforts. Meanwhile, some regions have succeeded in improving state
performance in some cases when they used existing tools to intervene in states.
EPA Established the State Review Framework
In 2004, EPA worked closely with the Environmental Council of the States
(ECOS) to establish the SRF, a national system for reviewing state enforcement
performance under the CAA, CWA, and RCRA. Under this system, EPA regions
evaluate states on 12 nationally consistent elements as a means of consistently
judging state performance, and comparing individual states to national goals.5
OECA based many of the metrics for each element on metrics developed in EPA
policies and guidance. The SRF allows regions to recommend improvements to
the states in public reports. EPA conducted the first round of reviews in
FYs 2004-2007, and EPA is conducting the second round beginning in FY 2008
through the end of FY 2012.6
GAO reported that EPA has improved its oversight of state enforcement programs
by implementing the SRF as a consistent approach for overseeing the programs.
Our evaluation verified some benefits from the SRF, and regional enforcement
5 There is a 13th optional element for compliance assistance. For this evaluation, the OIG used three of EPA's SRF
measures as proxies for state enforcement quality, those related to inspection coverage, identification of violations,
and penalty assessment.
6 To see all final SRF reports for Rounds 1 and 2, visit http://www.epa.gov/compliance/state/srf.
12-P-0113
-------
officials said that the program was beneficial. First, posting reports on the EPA
website put public pressure on states. Second, the SRF has raised the enforcement
discussion to the level of state agency commissioners.
An external review of the Round 1 SRF reports identified four major issues
impairing state enforcement: data quality, identification of significant
noncompliances (SNCs) or high-priority violations (HPVs), penalty calculation,
and taking "timely and appropriate" enforcement actions. While OECA indicated
that these were problems, it has yet to issue formal guidance on how to address
the four issues, and has since focused its attention on other activities.
EPA Undertook Additional Efforts
In June 2010, EPA issued the Clean Water Act Action Plan Interim Guidance in
an attempt to target resources toward the biggest threats to water quality
regardless of their size, and to enforce vigorously against unpermitted and illegal
discharges. The action plan commits EPA to establish clear expectations for state
performance, hold states consistently accountable, and hold itself to the same
standards where EPA implements programs. Finally, the action plan commits
EPA to provide more complete, accurate, and timely information to the public. By
doing so, EPA hopes to enlist the public as an ally to press the regulated
community and government for stronger accountability. The interim guidance
directs that (1) EPA regions and states expand NPDES annual planning to include
consideration of enforcement and permitting in an integrated way, and (2) EPA
regions take action in states that have demonstrated long-standing problems with
their permit quality or enforcement programs.
EPA is currently developing an electronic reporting rule that will require NPDES
permittees to submit facility discharge, monitoring, and reporting information
electronically. During the course of this evaluation, EPA was in the process of
modifying the noncompliance and reporting requirements for the NPDES
delegated state programs. In response to a 2010 EPA OIG report, EPA has agreed
to develop criteria and regularly review and update EPA-state memoranda of
agreement.7 In addition, to increase transparency, EPA launched the CWA trends
map and annual noncompliance report tool using FYs 2008 and 2009 CWA data.
This tool allows the public to access basic mapped data on assessed penalties,
inspection rates, and formal actions taken for all major and nonmajor facilities.
Regions Successfully Used Existing Tools
EPA regions have tools to intervene in states that are not enforcing environmental
laws according to EPA's expectations. These tools range from minor (phone calls,
independent EPA inspections) to moderate (objecting to a state permit, taking
independent enforcement actions, or over-filing in the state), to revoking a state's
7 EPA OIG, EPA Should Revise Outdated or Inconsistent EPA-State Clean Water Act Memoranda of Agreement,
Report No. 10-P-0224, September 14, 2010.
12-P-0113
-------
authorization in the most severe cases. Our interviews showed that moderate
intervention tools succeeded more often than other tools in motivating states to
make fundamental changes to enforcement programs:
• Region 4 objected to and assumed issuance authority for over 20 South
Carolina CWA permits in 2004, and took independent enforcement actions
on several other permits because the state legislature passed legislation
that conflicted with federal regulations. The EPA intervention enforced the
law when the state could not, and led to a state legislative change that
brought South Carolina's water protection up to federal standards in this
area.
• Region 8 took over storm water inspections for Colorado in 2006 when
the state was not taking enforcement actions against violators. The region
took its own enforcement actions, and EPA enforcement data show that
the state program has increased its enforcement activity.
• Region 7 took 15 enforcement actions against concentrated animal
feeding operations in Iowa in 2010. In coordination with the Iowa
Department of Natural Resources, Region 7 used aerial flyovers to
identify discharging concentrated animal feeding operations for sampling
inspections. Because of the enforcement effort, many of the operations
applied for NPDES permits and began constructing runoff controls.
With decisive EPA oversight and accountability, state enforcement deficiencies
such as these are more likely to be resolved.
State Enforcement Programs Are Underperforming
Despite EPA's efforts, state enforcement programs frequently do not meet
national enforcement goals.8 We compared state performance against some of
OECA's national enforcement goals, and compared states to each other based on
three enforcement metrics. We found that states did not meet some national goals,
and some states performed far below the average.
Using the three metrics described in chapter 1, we assessed the average
performance reported in EPA databases for each state and each statute over the
FYs 2003-2009 analysis period. These data show that performance was low
across the board, and the range of average performance was wide.
All of the regions except for Region 2 included at least one state that performed in
the bottom quartile in the different program analyses. In some regions, EPA staff
and officials corroborated our assessment by agreeing that particular states in their
region presented oversight challenges, while in other regions, EPA officials did
1 According to 2010 SRF data from EPA's Online Tracking Information System.
12-P-0113
-------
not agree that any state in their region did not perform well. (See appendix C for
state-level data.)
Sfafe Programs Frequently Do Not Meet National Goals
The CAA, CWA, and RCRA, and their associated regulations, establish minimum
federal requirements for authorized state programs. The regulations require that
states have minimum enforcement capabilities to receive authorization for these
programs; for example, states must have the ability to assess civil penalties.
OECA uses policies, guidance, and other programs to communicate additional
expectations to the states on a program-by-program basis. Our review and
analysis of EPA's enforcement data show that state enforcement performance
frequently does not meet national goals set in these documents. (See appendix C,
table C-2 for additional comparison data.)
CAA Title V Performance
• EPA set a national goal that states inspect 100 percent of major CAA
emitters every 2 years.9 States inspected an average of 89 percent of
these facilities in the 2-year period, but only eight states met the
100 percent goal.
• EPA set a national goal that states enter 100 percent of high-priority
CAA violations into EPA data systems within 60 days.10 The national
average indicates that states only enter 35 percent of HPVs in that time
frame, and only two states met the 100 percent goal.
CWA NPDES Performance
• Because the CWA major inspection goal changed in 2007, we looked
at this metric for 2006 (prior to the change) and for 2010. In 2006, the
national goal was that states inspect 100 percent of majors every year.
The national average was only 66 percent of facilities inspected, and
only one state met the goal by inspecting 100 percent of facilities.
• In 2007, EPA issued a new Compliance Monitoring Strategy (CMS)
that reduced the national goal so that states were to inspect 100 percent
of CWA majors every 2 years, starting in 2009. The national average
for 2010, the year after the new goals went into effect, was only
61 percent. While two states met the 100 percent inspection goal,
13 states inspected fewer than 50 percent of major facilities, indicating
that they were not on track to meet the goal of 100 percent every
2 years.
9 EPA, "Issuance of the Clean Air Act Stationary Source Compliance Monitoring Strategy," September 2010.
10 EPA, "Issuance of Policy On Timely and Appropriate (T&A) Enforcement Response to High Priority Violations
(HPVs)," December 1998.
12-P-0113
-------
RCRA Subtitle C Performance
• EPA set a national goal that state agencies inspect 100 percent of large
quantity waste generators every 5 years.11 In 2010, states had only
inspected an average of 62 percent of these facilities for the 5-year
period just ending, and only two states met the 100 percent inspection
goal.
See appendix C for additional comparisons of performance to goals.
Sfafe Enforcement Is Inconsistent
Our analysis shows that in addition to not meeting goals, state enforcement is also
inconsistent among states and between EPA regions. Based on an average of three
enforcement measures for each program (percentage of facilities inspected,
percentage of SNCs or HPVs identified per inspection, and percentage of final
actions with penalties for FYs 2003-2009), we found that state performance
varied widely across the country.12 For example, CAA performance varied by
almost 50 percentage points. This range in state enforcement activity illustrates
that some states inspected facilities, identified violations, and/or assessed
penalties for violations at a much higher rate than other states.
These programs are complex, and differences exist at all levels of analysis. In
addition to the significant variations from state to state, there was often a lack of
uniformity even within states. Many of the highest-performing states in our
overall analysis did not meet all of the national performance goals. For example,
Alabama ranks the highest for CAA but does not meet the SRF stated goal that
states identify HPVs at a rate greater than or equal to half the national average,
and also does not meet the goal that states enter 100 percent of HPVs into EPA
data systems within 60 days.
Average state performance varied by EPA region as well. Because regional
approaches to state oversight varied, state performance varied depending, in part,
on the region overseeing the state. Using the same aggregate performance
measure, we summarized state performance by region and found that performance
at a regional level of analysis varied as well.
11 EPA, "Compliance Monitoring Strategy for the Resource Conservation and Recovery Act (RCRA) Subtitle C
Program," January 2010.
12 Because these measures combine three different metrics, they are most useful for comparing state performance
and showing the performance distribution, and they are not useful for making inferences about an individual state's
performance. Appendix A includes a discussion of the context and limitations associated with measuring and
interpreting these metrics. As described in appendix A, this analysis did not include programs in which EPA retains
sole enforcement authority.
12-P-0113 10
-------
OECA Does Not Have Sole Authority Over Regional Enforcement
Although EPA intends for OECA to be responsible for national enforcement
policy, OECA does not have sole authority over EPA's national enforcement
workforce or state oversight. Instead, EPA's 10 Regional Administrators have
oversight authority over state enforcement; they apply regional criteria about what
constitutes good state performance and determine when and how to intervene if
they feel that states are not performing to expectations.
OECA's Assistant Administrator has direct authority over national criminal
enforcement, but only over the headquarters-based civil enforcement program.
OECA lacks critical region-level program information; OECA officials have
requested information from the regions but the information received or calculated
was incomplete or otherwise known to be inaccurate. Without this basic
knowledge, OECA cannot effectively or efficiently manage its national program.
OECA's lack of direct authority over national enforcement responsibilities and
resources has continued to result in inconsistent enforcement and state oversight.
In the absence of direct lines of authority between OECA and EPA's regional
enforcement workforce, OECA must persuade regions to cooperate with national
initiatives and uniform oversight practices. The OECA Assistant Administrator
stated that OECA must apply persistent pressure to get some regions to intervene
in difficult states.
OECA uses a number of tools such as the SRF, watch lists for problem facilities,
national program managers' guidance, the EPA performance measurement
system, and annual region reviews to encourage regions to participate in and
contribute to a national enforcement program. These may be useful tools for
managing the national program. However, OECA still does not have direct
authority over EPA's regional enforcement workforce or state oversight, and
cannot influence state performance directly. In our opinion, EPA headquarters and
OECA, the office responsible for establishing a national enforcement program,
cannot hold regions accountable for state enforcement of CAA, CWA, and RCRA
requirements.
EPA Headquarters Does Not Consistently Hold Regions Accountable
EPA has not consistently held regions accountable for ensuring that states
adequately enforce environmental laws. EPA has not set clear and consistent
national benchmarks for state performance, and has not held regions accountable
for abiding by national oversight guidance.
OECA Has Not Set Consistent Benchmarks
State enforcement programs are complex and varied. However, as the national
steward of environmental enforcement, EPA must set national benchmarks that
12-P-0113 11
-------
establish the enforcement expectations that states and EPA agree to meet. EPA
has not set consistent benchmarks for state performance. As a result, EPA cannot
hold EPA regions accountable for ensuring that states meet a national standard,
and headquarters does not objectively know which states require immediate
intervention. If state performance exceeds the national benchmarks, states and
regions can then address local priorities.13
Some state officials we spoke with were not opposed to EPA ranking their
performance against that of other states. One state enforcement official said that it
would be interesting to know how his state program compared to others so that he
could ensure that his state program is performing as well as possible.
We interviewed enforcement officials in six EPA regions and seven state
environment agencies about how they evaluate enforcement program quality. We
also reviewed the CAA, CWA, and RCRA for benchmarks contained within the
law, and reviewed hundreds of associated EPA guidance and policy documents.
We found that none of these information sources outlined clear and consistent
benchmarks. Most of the performance requirements established in the laws and
regulations are not easily measurable. For example, the regulations require
appropriate penalties, but do not define "appropriate". To bridge the interpretive
gap, EPA has created policies and guidance to establish expectations for a
national enforcement program. Generally, states are not bound by EPA policy or
guidance that EPA has not codified in the laws and regulations. However, if
authorized states do not to abide by EPA policies and guidance, EPA retains the
authority to step in.
Headquarters Policies and Guidance Not Clear
EPA regulations and guidance documents do not set clear and consistent
benchmarks for state enforcement performance. EPA has not consolidated and
clarified the long lists of documents to show which ones are active and
important to EPA, the documents sometimes conflict with one another, and
EPA regional offices interpret the documents differently.
Through its many policies and guidance documents, EPA has given states
information on how to run their enforcement programs. However, EPA has not
clearly identified the key documents for operating a state program. According
to an OECA official, OECA's internal websites list the most significant
enforcement documents for managing state enforcement programs. In these
lists, we found 9 guidance or policy documents for the CAA, 22 for the CWA,
and 13 for RCRA. However, a public OECA Internet site that contained
enforcement policy and guidance documents listed over 200 documents for the
13 Memoranda of agreement between EPA and states are one tool for setting performance expectations for EPA and
states, but the OIG previously found that CWA memoranda of agreement did not set consistent, national
benchmarks. EPA OIG, EPA Should Revise Outdated or Inconsistent EPA-State Clean Water Act Memoranda of
Agreement, Report No. 10-P-0224, September 14, 2010.
12-P-0113 12
-------
CAA, over 240 for RCRA, and listed 69 as important for the CWA. This
inconsistency makes it difficult for state enforcement officials to know which
documents are the most critical and where to find them; one state official told
us that there was policy "overload."
In addition, some policies have been in interim status for more than a decade,
such as the "Interim Clean Water Act Settlement Penalty Policy," which was
issued as an interim policy in 1995, and is indicated as a key document in
EPA's SRF guidance. EPA has also updated some documents through
memoranda and amendments rather than full updates of the documents. This
approach may confuse states and reduce the efficiency of the day-to-day
program operations as states search for the appropriate guidance.
Policies and Guidance Conflict
In some situations, the guidance and policy documents conflict with one
another. For example, OECA updated the NPDES CMS in 2007 to establish a
new goal that states inspect all major facilities every 2 years rather than
annually. However, SRF reports still measured state performance against the
previous annual goal, and the SRF Plain Language Guide for CWA did not
indicate that the compliance monitoring system changed the inspection
frequency.
In another example, state officials expressed confusion about whether EPA's
CAA and RCRA Enforcement Response Policies and the SRF requirements
were coordinated. The officials were concerned that their state SRF report
could potentially indicate that the state did not meet program requirements
when they were abiding by the relevant EPA enforcement response policy.
Regions Develop Independent Interpretations
Without clear and consistent guidance from headquarters, regions develop
their own guidance and interpretations. Regional officials said that their states
need a lot of guidance because parts of the regulations are confusing, unclear,
or outdated. Enforcement officials in one region said that their states often
interpret EPA's policies differently, especially those that are not clear.
Enforcement officials in this region said that EPA issues numerous new
regulations without guidance for implementation, which makes it difficult for
the states and regions to adopt the new regulations. They said that as a result,
states are not implementing some new regulations. In one region, officials
stated that headquarters should provide more guidance and training to keep
regions and states up to date with continuously changing regulations. Officials
from this region said that headquarters relies too much on the regions to
develop their own guidance and outreach materials, which they said should be
headquarters' responsibility.
12-P-0113 13
-------
The absence of consistent benchmarks creates confusion about what EPA sees as
a good state program and contributes to inconsistent performance. Staff and
officials in EPA regions told us that it would be helpful if OECA clarified its
national policy and guidance, and codified national enforcement standards in
regulations. Staff and officials in states told us it would be helpful if EPA reduced
policy "overload" and "overlap" by centralizing, finalizing, and keeping guidance
up to date, and by codifying pertinent guidance in regulations. Without clear and
consistent national benchmarks, EPA cannot operate a national enforcement
program that consistently interprets and enforces environmental laws.
OECA Has Not Ensured That EPA Regions Consistently Oversee
States
EPA regions vary in the stringency of their oversight. As a result, EPA is not
operating a nationally consistent enforcement program, and EPA headquarters
cannot prioritize state enforcement program improvements based on performance.
Despite issuing guidance on conducting SRF oversight reviews, EPA regions did
not consistently conduct or report on their reviews. In addition, enforcement
officials in three of the seven states we interviewed said that some regions held
their states to a more stringent standard than others did because performance
benchmarks vary by EPA region.
EPA's SRF reviews demonstrate inconsistency among EPA regional approaches
to oversight. EPA designed the SRF to be a nationally consistent review of state
enforcement programs. OECA held training workshops across the nation and
distributed almost 50 guidance documents to regions on how to conduct SRF
reviews and write the associated reports. Despite OECA's efforts to standardize
the process, regions implemented the SRF process inconsistently. OECA did not
successfully enforce consistency in the state reports through its guidance or its
headquarters review processes.
Regions' SRF reports differed in both quantity and quality of information about
the states. The discrepancies limit the reader's ability to find useful information in
the reports or to compare reports for different states. For example, Region 6's
Round 1 SRF report on Oklahoma was 22 pages long, and Region 7's on Kansas
was 225 pages. However, we found that both reports were lacking in information
on certain statutes and metrics. In contrast, Region 6's report on Arkansas was
only 27 pages, but was relatively complete.
After receiving the first 25 Round 1 reports, OECA issued a "Guide to Writing SRF
Reports (Interim Final)" in April 2007. In the accompanying transmittal memo,
OECA said, "To date, inconsistencies in the information available in the SRF
reports inhibit our ability to determine if the reviews are being carried out in a
consistent manner." OECA offered a report formatting guide and eight pages of
guidance on completing the reports, and noted that regions were "encouraged (but
not required) to follow the format of the guide." Despite the guidance, the OIG
12-P-0113 14
-------
found similar inconsistencies in Round 2 SRF reports with the shortest report
(Region 2, New Jersey) at 39 pages and the longest report (Region 7, Iowa)
reaching 220 pages.
The style and content of the reports also varies widely. Some regions were critical
of their states in the SRF reports while others emphasized their states'
accomplishments over their problems. In addition, the level of detail varied. For
example, three Round 2 SRF reports did not include quantitative values for
multiple metrics in their findings tables. Seven of the Round 2 reports did not
provide national average and/or national goal information for many of the
findings. Because these regions did not present the requirements along with the
performance information, it was difficult for the reader to determine whether the
state met requirements.
Regions Have Not Effectively Curtailed Weak and Inconsistent
Enforcement by States
EPA regions do not consistently intervene in states to correct deficiencies. In a
region where states performed better than average based on our analysis, a state
enforcement director said that he felt that his region held its states to a higher
standard of performance than other regions do. A state enforcement manager in a
different region agreed, saying that the state's region held the state to standards
that are higher than those in other regions. Multiple state directors called for
consistent benchmarks and oversight nationwide so that all regions hold all states
to the same standards.
OECA, regional, and state enforcement officials agree that states are
underperforming. However, regional efforts to correct deficiencies have not
consistently led to changes in state performance because regions do not always
assertively intervene in state programs when issues arise. The states highlighted
below were among the lowest-ranked states across programs in FYs 2003-2009
based on the average of the performance metrics we chose. In the following
examples, to date, EPA regions had not acted decisively enough to improve
performance. In each of the states, enforcement officials had a different reason for
ranking low according to these metrics, but in each case, the EPA region did not
substantively change the states' performance.
• North Dakota: EPA data show that North Dakota ranks in the bottom
quartile for two of three statutes (CAA and RCRA). (See maps Cl through
C4 in appendix C.) According to the FY 2009 North Dakota SRF report,
the state inspected 100 percent of its major CWA facilities. However,
despite inspecting facilities, the state reported an SNC rate of 3.8 percent
for its major facilities, well below the national average of 23.2 percent.
EPA data show that the state assessed no penalties against known CWA
violators during the entire period of analysis (FYs 2003-2009). In
response to the state's low CWA performance, Region 8 enforcement
12-P-0113 15
-------
officials told us they increased their inspection coverage and file reviews.
They said that their increased attention in the state resulted in increased
permits for storm water, but that the facilities with permits were not in
compliance when the region returned to inspect them. The state still was
not enforcing the CWA storm water requirements.
Louisiana: EPA data show that Louisiana ranks in the bottom quartile for
two statutes (CAA and RCRA) and the bottom half for all three statutes. In
2001, citizens filed a petition with EPA urging a withdrawal of the state's
CWA NPDES program authority for many reasons, including lack of
adequate enforcement. This was one of many withdrawal petitions filed
for Louisiana. (Citizens filed petitions to withdraw CAA and RCRA
authority as well.) The region responded by conducting audits in the state.
The region found several deficiencies and required the state to change
some policies and develop new measures. Although the state has
completed the recommended actions, the state's poor performance
persisted; our analysis found that Louisiana has the lowest enforcement
activity levels in Region 6 and ranked in the lower half for the CWA and
lowest quartile for CAA and RCRA for FYs 2003-2009.14 (See maps Cl
through C4 in appendix C.) State, EPA regional, and external interview
responses attributed Louisiana's poor performance to several factors,
including a lack of resources, natural disasters, and a culture in which the
state agency is expected to protect industry.
Alaska: EPA data indicate that Alaska ranked in the bottom half for both
of its authorized statutes (CWA and CAA), although it just began phasing
in its authority for the NPDES program in 2008. (See maps Cl through C4
in appendix C.) Therefore, Alaska's enforcement data for the CWA are
largely a reflection of EPA direct implementation in FYs 2003-2007.
However, since program authorization began, all available SRF data show
that the state has not taken any formal enforcement actions nor issued any
penalties for any facilities found to be out of compliance. Regional
directors told us that when the region authorized the state to run the
program, both the region and OECA officials were aware that the state
lacked the capacity to be successful. At the time of our review, the region
had moved to delay the final phase of authorization, but did not ensure that
the state demonstrates a minimum level of performance before it advances
to the next authorization phase.
Illinois: EPA data indicate that Illinois consistently ranked among the
lowest-performing states for two of the three programs (CAA and RCRA)
and was in the bottom half for all three in FYs 2003-2009. (See maps Cl
through C4 in appendix C.) Despite this record, EPA enforcement data
14 We requested documentation about the audit process and results from Region 6 staff, but regional personnel told
us that they were unable to provide documents from that period. In response to the draft report, OECA provided
documentation indicating that the state completed the required tasks.
12-P-0113 16
-------
show that Region 5 has inspected a lower percentage of Illinois CWA and
CAA facilities as compared with some other states in the region, and the
region's RCRA inspection coverage has been declining in recent years. In
2011, the region developed an intervention strategy for this state, which it
was in the process of implementing during this evaluation. It is premature
for us to determine the results of the intervention.
Although these examples suggest different reasons for low enforcement
performance, each state provides a scenario in which EPA's stewardship over
national enforcement has not overcome state deficiencies. The potential causes for
deficiencies vary, and could range from a state philosophically opposed to taking
enforcement action (North Dakota) to a state that cannot enforce because it is
overwhelmed by a natural disaster (Louisiana). Regardless of the cause, when a
state is operating a federal CAA, CWA, or RCRA program, EPA must intervene
to enforce the law when states do not perform satisfactorily.
EPA Does Not Shift Resources to Intervene in Problem States
If an authorized state does not operate its program as outlined in CAA, CWA, and
RCRA, the laws authorize EPA to revoke the state's authority to operate the
program of concern. However, EPA must be able to adjust its resources to take
back state authority in programs. EPA headquarters, regional enforcement
officials, and independent organizations told us that the threat of EPA revoking a
state's authorization was moot because there is a general understanding that no
EPA region has the resources to operate a state program. This reality undercuts
EPA's strongest tool for ensuring that authorized states adequately enforce
environmental laws: de-authorization.15
OECA is constrained from actively managing its resources to direct them to the
most important state enforcement problems. Federal law intends that EPA use its
workforce efficiently and effectively to accomplish its goals.16 Under the current
resource planning structure, EPA regions divide their resources among several
OECA priorities, including state oversight. Figure 1 shows the average allocation
of full-time equivalent (FTE) positions between regions and headquarters from
FY 2000 through FY 2010. If EPA regions report that they are having problems
with state enforcement, OECA cannot reallocate FTEs among regions to address
the problems because OECA does not control enforcement resources in the
regions. Therefore, priority enforcement issues may not receive needed resources.
15 EPA has the power to take a number of steps to help states improve their programs. All of these steps require an
expenditure of resources. The actions taken by Regions 4, 7, and 8, outlined on page 6, took resources away from
other activities. The most severe action that EPA can take, revoking a state's authority to operate a program, takes a
large amount of resources.
16 Government Performance and Results Act (GPRA) Modernization Act of 2010.
12-P-0113 17
-------
Figure 1: Average enforcement FTE allocations, FY 2000-FY 2010
REGIONS: REGION 10:
244 ' 174
REGIONS:
169.
REGION?:
172
REGIONS:
272 "• REGION 1:
174
REGION 2:
267
REGION 4:^ REGIONS:
356 283
Source: Office of the Chief Financial Officer data.
Regional enforcement officials said that EPA had not allocated its enforcement
workforce according to the regions' workloads, and that regional enforcement
resources were insufficient to allow them to conduct their required enforcement
work. In response to our regional survey, eight out of the nine responding regional
enforcement directors cited a lack of resources as a barrier to using the
enforcement tools at their disposal.17 In our interviews, some EPA regions said
they make internal adjustments to redirect resources toward problem states.
Our analysis of FTE data provided by OECA and the Office of the Chief
Financial Officer showed that even though enforcement priorities have changed
and the size of the regulated community has increased, relative allocations to EPA
regions have effectively not changed over the past 10 years. Regional FTE
allocations varied less than 0.3 percent from FY 2000 to FY 2010. It is reasonable
to expect some amount of change in staffing levels between regions during this
time frame to reflect changing priorities and oversight needs.
As an example, we looked at resources using a workload ratio of permits issued to
enforcement employees. We found that the workload varied across regions from
203 permits per employee (in Region 8) to 541 permits per employee (in
Region 4). The data in figure 2 show that Region 4 has the heaviest workload.
Appendix C, which presents the results of our overall analysis, shows that
17 Region 6 enforcement officials declined to respond to all of the survey questions (their eventual response
answered 32 percent of the survey questions). As such, we consider Region 6 nonresponsive, and consider
Region 6's lack of participation a scope limitation. However, we analyzed all survey responses and believe that the
Region's lack of response does not significantly impact the findings or conclusions reached.
12-P-0113 18
-------
Region 4 also has the highest-performing states for the enforcement metrics we
considered.18
Figure 2: Workload assessment (based on 2010 permit levels)
REGIONS
REGION 10
REGIONS
REGION 9
REGION 1
REGIONS
REGION?
REGION 2
REGIONS
REGION 4
0 100 200 300 400 500 600
Workload (Permits/FTE)
Source: OIG analysis of FTE data from the Office of the Chief Financial Officer and enforcement
data from OECA.
The evidence presented in this evaluation is consistent with previous OIG
conclusions regarding EPA workforce management. In February 2010, the OIG
reported that EPA does not enforce a coherent program of position management
to assure the efficient and effective use of its workforce.19 Without an Agency-
wide position management program, EPA leadership lacks reasonable assurance
that it is using personnel in an effective and efficient manner to achieve mission
results. In addition, in September 2008, the OIG reported that EPA does not
assess resources needed and expended to accomplish its priority enforcement
goals.20 EPA has thus far refused to accept our recommendation that it develop a
cost-effective methodology for measuring resource inputs in the national
priorities.
Conclusions
EPA has not implemented a nationally consistent enforcement program. In our
opinion, regions do not consistently take action when states do not enforce the law
according to EPA's policies and the regulations established under federal laws. In
states where enforcement actions are lacking, citizens may be exposed to
inequitable health risks compared to states where EPA or state agencies take
timely and appropriate enforcement action. Inconsistent enforcement can result in
18 There are limitations to using permits issued for estimating workload; for example, some permits require more
time to inspect than others do. However, this method allows for an estimation of workload for the purpose of this
evaluation.
19 EPA OIG, EPA Needs Better Agency-Wide Controls over Staff Resources, Report No. ll-P-0136, February 22,
2011.
20 EPA OIG, EPA Has Initiated Strategic Planning for Priority Enforcement Areas, but Key Elements Still Needed,
Report No. 08-P-0278, September 25, 2008.
12-P-0113 19
-------
unabated pollution in one state causing environmental and public health damage
and cleanup costs in another. Furthermore, inconsistent stringency of enforcement
by some states may give firms in those states a substantial competitive advantage
over firms in the same regulated industries in other states.
A centralized national enforcement program could reduce overhead costs for EPA
by consolidating some functions and decision-making activities. EPA could make
more effective use of its $372 million in regional enforcement FTEs by directing
a single national workforce instead of 10 inconsistent regional enforcement
programs.21 EPA could use its enforcement FTEs more effectively by targeting
decisive interventions in states where enforcement problems require the most
attention.
In addition, reduced pollution due to enforcement actions can lead to large
monetary health benefits. EPA calculated that enforcement actions and the
resulting pollution reduction would have saved an estimated $3.8 billion in 2007
and $35 billion in 2008 in avoided health and lost work costs, including reduced
hospital and emergency room visits, avoided premature death due to heart or lung
disease, and reduced cases of bronchitis, heart attack, and asthma.22 When neither
states nor EPA takes enforcement actions when needed, these health benefits are
not realized and premature deaths and illnesses are not prevented to the extent that
they could be. As a result, EPA cannot assure that Americans in all states are
equally protected from the health effects of pollution or that enforcement of
regulated entities is consistent nationwide.
In our opinion, EPA should act decisively when states do not enforce authorized
programs so that EPA can implement a consistent national enforcement program
that protects all citizens. However, EPA lacks clear, nationwide lines of authority
for enforcement including authority over resource allocations and use. EPA's
current mode of operation has resulted in inconsistent implementation of
environmental regulation and protection. EPA has not managed its enforcement
resources efficiently to allow effective intervention in states where enforcement
practices do not meet its expectations. Additionally, EPA has not provided
standards or consolidated enforcement policies so that state governments and the
regulated community understand EPA enforcement expectations. Particularly
important is establishing a clear escalation policy that shows states when and how
EPA will intervene if state enforcement is deficient under EPA's policies. EPA
can improve its national enforcement through more efficient and effective use of
its resources and authorities to ensure that all citizens and industries enjoy
consistent environmental protection.
21 Dollar amount represents personnel compensation, travel, general, informational technology, and contracts and
grants expenses.
22 EPA, OECA FJ 2007Accomplishments Report, May 2008; and EPA, OECA FY 2008 Accomplishments Report,
December 2008.
12-P-0113 20
-------
Recommendations
We recommend that the Deputy Administrator:
1. Give OECA authority for all nationwide enforcement resources and
workforce allocation.
We recommend that the Assistant Administrator for Enforcement and Compliance
Assurance:
2. Cancel outdated guidance and policy documents, and consolidate and
clarify remaining guidance into EPA documents that are publicly and
easily accessible on the EPA Civil Enforcement website.
3. Establish clear and consistent national enforcement benchmarks
throughout CAA, CWA, and RCRA guidance and policies so that EPA's
enforcement expectations are clear and consistent for state governments
and the regulated community.
4. Establish a clear and credible escalation policy for EPA intervention in
states that provides steps that EPA will take when states do not act to
ensure that the CAA, CWA, and RCRA are enforced.
5. Establish procedures to reallocate enforcement resources to intervene
decisively when appropriate under its escalation policy.
6. Develop a state performance scorecard to publicly track state enforcement
activities and results from year to year.
Agency Comments and OIG Evaluation
In its response to the draft report on this subject, the Agency agreed with the
overall finding that state enforcement performance varies widely across the
country. The Agency also agreed that EPA headquarters and regions can take
steps to improve national consistency.
The Agency expressed concern about the metrics we chose and the methods we
used to summarize enforcement metrics. The Agency was concerned that our
evaluation relied too heavily on the state enforcement activity metrics we
collected from EPA to compare state performance. The Agency argued that
enforcement is a complicated process that ideally relies on analysis of multiple
factors related to state goals and performance. The Agency was concerned that
our choice of metrics to compare state performance could unintentionally
encourage states to focus only on those three metrics: inspection coverage,
identification of SNC/HPVs, and assessing penalties.
12-P-0113 21
-------
The Agency also expressed concern about the quality of the underlying data for
the three enforcement activity metrics. EPA said that the data and goals on which
we based our analysis were unreliable.
We utilized several sources of information to reach our conclusions. Although not
mentioned in its comments, we utilized EPA's own enforcement data, and our
analysis is similar to those that EPA itself conducts. EPA has acknowledged that
these data possess several significant limitations, and we took steps to overcome
and describe these limitations. However, we used several other sources of
information in our evaluation, such as descriptions of states contained in the SRF
reports and interviews with officials and staff of EPA and states. Our conclusions
rest on information contained in all of these sources.
We chose the three measures of state activity after discussions with OECA staff
and officials, and other stakeholders. The use of these measures is supported by
statements contained in OECA's own documents including the SRF, and EPA
displays the data on its publicly available enforcement website. However, because
we realize that no single set of measures is individually definitive, we
corroborated our findings through numerous interviews across six regions and
document review. EPA's agreement with the substance of our findings provides
further support for the validity of our approach. See chapter 1 and appendix A for
details about our methodology.
In response to EPA's comments, we reemphasized our use of multiple sources in
our evaluation in chapter 1 of the final report, and included additional details on
our methodology in report appendix A. We also determined that known
inaccuracies did not prevent using the data. To clarify this in our report, we added
language describing that the OIG was aware of EPA enforcement data issues from
prior reviews, and describing our own accuracy assessments of the data. Based on
data reliability assessments and comparison of "raw" data with data verified by
states for 2008 and 2009, we believe that the data quality is high enough to
provide reliable indicators of state performance. We also believe that the overall
quality of the data submitted to the database will only improve if it is publicly
used.
Specifically, the Agency said that using SNC rates was not a reliable way to
gauge performance. The response stated:
Available data suggests that some states significantly under-report
significant violators. Therefore, it is likely that low rates of
identified significant violators do not reflect actual low rates of
violation, but rather incomplete or mischaracterized data. If this is
correct, then higher reported rates of violation are more likely to be
accurate, and states with such higher rates likely to be higher
performing states, if identification and disclosure of violations is
the metric.
12-P-0113 22
-------
In response, we clarified in our "Scope and Methodology" section that our
method for comparing state performance, likewise, counted higher SNC rates as
indicators of improved performance.
EPA commented that states are not required to report CWA penalties to EPA, so
there is no way to determine whether a state reporting zero penalties assessed zero
or did not report. In response, we reanalyzed CWA data without the penalty
metric. We also reanalyzed the CWA data by including the penalty metric for
states that choose to report their penalties and did not include it for states that did
not choose to report it, in the same way OECA analyzes CWA data on its
Enforcement and Compliance History Online (ECHO) website. The second
method produced more reliable results.
The Agency also disagreed with our method of summarizing enforcement metrics
across statutes. EPA wrote:
Complete data and valid, meaningful measures are key to
understanding state performance. Gaps in our current data make it
difficult to develop measures that tell a complete story across all
media and regulated sectors. Limited resources at both the state
and federal levels make it more difficult to address these gaps.
Measures based on the data that EPA does have may not focus on
the right things. In our opinion, an effective way to address these
issues is to set a new direction in the enforcement program that
takes advantage of the advances in information and monitoring
technologies. This will enable us to get more complete data
through the efficiencies of electronic reporting and field
equipment. This data will allow EPA to more completely identify
regulated sectors, monitor and assess compliance, target resources
more effectively, and measure the overall performance of federal
and state regulators in a more meaningful way.
In response, we eliminated our use of a summary metric for each state. However,
we disagree that presenting a simplified method for comparing state enforcement
performance is invalid. Repeated assessment of indicators allows management to
determine where they should direct additional EPA attention. As with any other
performance measure, the measurement only takes on meaning when it is
compared with benchmarks that clearly outline success. The overarching message
of this report is that EPA must establish clear benchmarks for success so that
states, EPA regions, EPA headquarters, the public, and the regulated community
all understand what is required to ensure a safe and healthy environment. The
report emphasizes that the states we ranked highest are not necessarily
succeeding, but rather that "performance is low across the board." With clear
benchmarks, EPA, states, and the public can clearly understand how states
12-P-0113 23
-------
perform, and EPA can take deliberate steps to improve performance where steps
are most needed.
We agree that ideal performance data are repeatable, reliable, and relevant.
However, we also believe that EPA underrelies on its vast stores of state
enforcement data, which it has collected for decades. As part of our evaluation
process, we conducted a data reliability assessment to determine whether known
data inaccuracies prevented us from relying on data in EPA enforcement
databases. In our "Scope and Methodology" section and in appendix A, we
defined the dates when we drew data from the database.
The Agency did not agree with recommendation 1, generally agreed with
recommendations 2 through 4, and neither agreed nor disagreed with
recommendation 5. EPA suggested an additional recommendation, which is
included above as recommendation 6. Because the Agency did not provide a
detailed plan for implementing recommendations 2-5, we consider all
recommendations unresolved, pending our receipt of the Agency's corrective
action plan.
12-P-0113 24
-------
Status of Recommendations and
Potential Monetary Benefits
Rec.
No.
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (in SOOOs)
Page
No.
Subject
Status1
Planned
Completion
Action Official Date
21 Give OECA authority for all nationwide
enforcement resources and workforce allocation.
21 Cancel outdated guidance and policy documents,
and consolidate and clarify remaining guidance into
EPA documents that are publicly and easily
accessible on the EPA Civil Enforcement website.
21 Establish clear and consistent enforcement
benchmarks throughout CAA, CWA, and RCRA
guidance and policies so that EPA's enforcement
expectations are clear and consistent for state
governments and the regulated community.
21 Establish a clear and credible escalation policy for
EPA intervention in states that provides steps that
EPA will take when states do not act to ensure that
the CAA, CWA, and RCRA are enforced.
21 Establish procedures to reallocate enforcement
resources to intervene decisively when appropriate
under its escalation policy.
21 Establish a state performance scorecard to publicly
track state enforcement activities and results from
year to year.
Deputy Administrator
Assistant Administrator for
Enforcement and
Compliance Assurance
Assistant Administrator for
Enforcement and
Compliance Assurance
Assistant Administrator for
Enforcement and
Compliance Assurance
Assistant Administrator for
Enforcement and
Compliance Assurance
Assistant Administrator for
Enforcement and
Compliance Assurance
Claimed
Amount
Ag reed-To
Amount
$372,0002
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
This dollar amount represents resources that EPA will put to better use when the recommendation is implemented.
12-P-0113
25
-------
Appendix A
Details on Scope and Methodology
We applied the objective questions to three major environmental enforcement programs: the
CAA Title V program, the CWA NPDES program, and the RCRA Subtitle C program. We
conducted our evaluation from September 2010 to July 2011. We included all 50 states and all
10 EPA regions in our evaluation.
To answer the evaluation objectives, we utilized EPA enforcement data and reviewed
environmental laws, regulations, and EPA guidance (enforcement response policies, CMSs, and
other relevant OECA enforcement policies and guidance). We also surveyed EPA's 10 regional
enforcement directors and conducted interviews with enforcement officials in selected EPA
regions and states. We also interviewed OECA officials at EPA headquarters. In addition, we
considered EPA resources for FY 2000 through FY 2010, and assessed EPA's knowledge of the
workload that the regulatory universe presents.
As we analyzed EPA enforcement data, we were specifically interested in states where
enforcement activities began low, compared with other states', and remained low for the duration
of the study period (2003-2009). In these cases, we sought supplemental documentary and
testimonial evidence to determine what factors led to the low enforcement activity levels,
whether EPA undertook efforts to improve those states' performance, and whether EPA efforts
led to improved enforcement activities. In all cases, EPA agreed with our assessments of which
states displayed consistently low enforcement activity. This general agreement validated our use
of these metrics to compare state performance. In all cases, EPA regions engaged states in an
effort to improve their performance. However, EPA's efforts did not improve performance in
those states in all cases.
Interview Methodology
For this evaluation, we selected states and associated regions for interviews based on which
states appeared to have persistent problems meeting national enforcement goals related to the
SRF. To identify these states, the OIGused three information sources:
EPA Enforcement Data Analysis: To assess state performance using enforcement data,
we used EPA's Online Tracking Information System (OTIS) Management Report tool.23
We developed a ranking system that used three enforcement metrics relevant to the SRF:
percentage of facilities inspected, significant noncompliance identification rate, and
percentage of formal actions with penalties. We queried the management reports for
FYs 2003-2006, to correspond to the data years of the first round of SRF reviews, for
each statute and metric. An additional query of information from FYs 2007-2009 was
included to determine whether state performance levels were consistent over time, or
23 According to the OTIS Tool Guide, the most appropriate tool to "determine how your region or state is doing in
relation to other regions and states in regard to inspection coverage, discovery of violations, enforcement actions,
and average penalties," is the Management Report tool, which we chose to use in this analysis.
12-P-0113 26
-------
whether they had improved since the original SRF data years. We analyzed the data using
Microsoft Excel.
For the programs of interest, we looked at state enforcement performance for the CAA,
CWA, and RCRA in FYs 2003-2009 and identified states with low results for these three
metrics in at least two of the three statutes throughout the period.
Round 1 SRF reports: In addition to the data analysis, we reviewed Round 1 SRF reports
for the states and provided an assessment of each state's performance based on the
reports. We compared state results with national averages and national goals, assessed the
overall portrayal of the state in the report, and assessed the comprehensiveness of the
report.
Interviews and Corroborating Evidence: We also corroborated our analysis through
interviews with EPA headquarters and third-party organizations, including ECOS, the
Environmental Integrity Project, and the Environmental Working Group. We also
considered as evidence of underperformance recent petitions filed with EPA asserting
that states were underperforming, and collected and examined the citizen petitions for
states, when available.
From this analysis, we chose to conduct site visits Regions 4, 5, 6, 7, 8, and 10, and the states of
South Carolina, Illinois, Louisiana, Iowa, Colorado, North Dakota, and Alaska. We chose five of
the region/state combinations because the states emerged as persistent underperformers over the
analysis period. We chose the two other states because their performance stood out: in one case,
the state performed poorly in the first analysis period, but significantly improved in the second
(Colorado). In the second case, the team decided to visit a state (South Carolina) in a region
whose states tended to perform better than states in other regions, according to our analysis, to
determine why states in this particular region were performing better.
Data Analysis Methodology
To characterize state performance for FYs 2003-2009 for each of the three programs, we used
the data from the EPA enforcement database (OTIS Management Tool, offered to the public
through ECHO at www.echo.epa.gov). The metrics used for assessment were (1) percentage of
facilities inspected by the state (calculated as number of state facility inspections divided by the
total number of state facilities),24 (2) percentage of SNCs/HPVs identified per inspection by the
state,25 and (3) percentage of formal actions with penalties.26 Because states are not required to
report penalty data for CWA, any state reporting 0 percent was considered a nonreporting state
(even though their actual penalty assessment might actually be 0 percent), and that metric was
24 Actual name of metric as queried in OTIS management reports was the same for all program: "Total # Facil Insp
State".
25 Actual name of metric as queried in OTIS management reports: CWA, "% Facilities in SNC"; CAA, "New State
HPVs Per State Insp Facil"; RCRA, "New State SNCs Per State Insp Facil".
26 Actual name of metric as queried in OTIS management reports was the same for all programs: "% State Actions
With Penalty".
12-P-0113 27
-------
not taken into consideration in their performance average. Appendix C presents the results of the
information downloaded from the EPA database in August 2010.
Data Limitations and Considerations
The use of performance indicators is essential to managing government programs. While
all performance measures are imperfect in some ways, their repeated measurement over
time allows government program managers to gauge changes in performance. Even if the
measure itself does not provide a definitive, precise answer to the question of
performance, it provides a way to gauge performance. In the case of this evaluation, we
looked to nine standard measures (three for each of three statutes) over the period of
seven years to discern performance trends.
EPA Database Considerations
Over time, the OIG has reported about data quality and reliability concerns with
EPA databases.27 Because of our preexisting concerns, the OIG conducted a data
reliability assessment prior to using EPA data for this evaluation.
Data Reliability Assessment
In accordance with generally accepted government auditing standards, we conducted a
data reliability assessment to determine whether the EPA data were sufficiently reliable
for the purposes of this project. The team assessed the reliability of EPA's enforcement
data from the Air Facility System, the Permit Compliance System, the Integrated
Compliance Information System-National Pollution Discharge Elimination System, and
the RCRAInfo database (accessed through OTIS) using the GAO Data Reliability
Assessment guidance and the Government Audit Standards Yellow Book. We found the
EPA enforcement data to be sufficiently reliable for the purposes of the current project.
While there are known issues with the data, we accounted for these in the analysis, and
took steps to diminish the impact of potential data issues. We supported data analysis
results using interviews with EPA and external sources, and document reviews.
Our data reliability assessment indicated that the limitations of EPA data systems should
not prevent us from using the data to compare states.
Considerations About Performance Measures
EPA databases contain enforcement data submitted by state and local agencies and
facilities, and the metrics we chose provide a snapshot of state enforcement activities.
Understanding some underlying assumptions may clarify the use of these data to assess
27 See, for example, EPA Needs to Improve Its Recording and Reporting of Fines and Penalties, Report No. 10-P-
0077; EPA Could Improve RCRAInfo Data Quality and System Development, Report No. 1 l-P-0096; ECHO Data
Quality Audit - Phase I Results: The Integrated Compliance Information System Needs Security Controls to Protect
Significant Non-Compliance Data, Report No. 09-P-0226; and ECHO Data Quality Audit - Phase 2 Results: EPA
Could Achieve Data Quality Rate With Additional Improvements, Report No. 10-P-0230.
12-P-0113 28
-------
performance. The metrics we used included the entire facility universe for all three
statutes.
1. Percent of facilities inspected. EPA establishes goals for facility inspections. The
goals differ from statute to statute, and some states negotiate lower goals with
EPA in exchange for conducting additional activities in other areas. In addition,
EPA databases include the number of permits rather than the number of facilities,
and EPA has not historically frozen facility universe counts. However, OIG
discussions with EPA contractors who manage the databases indicated that, as a
correlate for facility counts, EPA typically uses the permit counts, and that using
the same universe for every year of analysis would not significantly change the
results because the universe does not change very much from year to year. An
important limitation to this measurement is that some permits may require much
more time and expertise to inspect. However, we determined that dividing the
number of inspections by the number of facilities/permits provided a general
method for comparing a state with fewer permits with a state that has many
permits.
Because states are not required to report data on universe tracking for some
nonmajors under CWA and CAA, states reporting more than just major facilities
could possibly appear to be doing worse in this category than they actually are. A
visual assessment of the CWA and CAA facility universe data suggested that this
might be the case for CWA. This is another caveat of the CWA data that users
should consider when interpreting the performance of a given state under that
program.
Because EPA does not require or expect that states inspect 100 percent of their
permitted facilities annually, the OIG did not expect that any state would achieve
a score of 100 percent in this area. However, by comparing state inspection
percentages, we gained a perspective on average inspection coverage, and where
states exceeded these averages. We probed further to understand what factors led
to that state's performance.
2. Percent of inspections identifying SNCs and HP Vs. This metric does not correlate
specific inspections with SNC or HPV identification. Instead, it divides the
number of facilities with a new state-identified SNC or HPV by the number of
facilities inspected. Since CWA did not have a metric for SNC identification by
states, we used the percent of facilities in the SNC metric. We conducted
sensitivity analysis using this and other metrics, found that it yielded comparable
results, and therefore determined it to be an appropriate substitute.
Identifying SNC/HPVs in an inspection may constitute a failure of enforcement
because it means a facility failed to comply with a regulation. However, EPA
more frequently views state identification of an SNC/HPV as a success because it
indicates a rigorous targeting and inspection protocol. We adopted EPA's view by
considering a higher value of this metric as an indicator of better state
enforcement performance. As with inspection coverage, the OIG did not expect
12-P-0113 29
-------
that any state would identify SNC/HPVs at 100 percent of inspections, which
would have indicated that the state had either a perfect targeting strategy or a
regulated community that did not ever comply with regulations. Instead, this
metric offered the OIG a perspective on average SNC/HPV identification, and
allowed us to further probe outliers to determine why those states exceeded or
underperformed compared with their peers.
3. Percent of state actions that included a penalty. This metric also does not track a
specific inspection or violation identification to a penalty. Instead, it divides the
number of state actions that included penalties by the total number of state formal
actions to offer a relative measure of how frequently the state addresses a
violation using a penalty.
There are two considerations regarding this measure for the CWA. First, and most
importantly, states are not required to report penalty information. Therefore, it
was not possible to determine whether a zero in EPA's database is a true value or
a nonresponse. There were 18 states with a zero value. We adjusted for this in our
analysis by utilizing the penalty metric only for (the 28) states that had a greater
than zero value for penalties.
The second consideration is that for two states, the state data contained in the
EPA database were greater than 100 percent. This does not occur in any other
metric or for any other states, and is caused by the way EPA's data system
computes totals. [The system calculates this metric by dividing the number of
state actions with penalties (numerator) by the number of state formal actions
(denominator). However, the database counts one category of formal actions in
the numerator but not in the denominator ("AO Stipulated Penalty 1"). The result
is that the percent of formal actions with penalties can be greater than 100 percent
if a state issued several "AO stipulated penalty 1" actions.] We adjusted for this in
our analysis by capping the metric at 100 percent (thereby assessing the two states
reporting > 100 percent at 100 percent).
Comparison of State Performance to National Goals and Averages
Using the SRF tool in OTIS, we compared actual state performance to the national goals
presented alongside the data metrics (as set in EPA guidance and policies). The SRF data
in OTIS were the most appropriate data for this analysis since they provide national goals
and averages, when available, along with state data. We used this tool to assess CAA,
CWA, and RCRA performance for 2010 to have a current understanding of how state
performance compared to the national goals and averages. We eliminated all metrics that
did not include a national goal, and then eliminated all metrics that were not state-only.
From the remaining metrics, we chose those that were most similar to the metrics used
for the state ranking (inspections, SNCs/HPVs, and penalties). We counted all states that
did not meet the national goal for each metric, and counted all states that did not achieve
12-P-0113 30
-------
the national average for each metric.28 We used the result to calculate a percentage of
reporting states that did not meet the national goal and national average.
Regional Survey Methodology
We surveyed the 10 regional enforcement directors across EPA's regions to solicit their opinions
on the most effective strategies for addressing issues in chronically underperforming state
enforcement programs. We designed the survey as a preliminary research indicator of how
regions intervened in states. We designed the questions to elicit the type and importance of
information sources the regions use for state program assessment, as well as the frequency of
oversight tool usage and barriers to using those tools. We also requested that the regional
enforcement directors describe their most effective strategies. The team requested and reviewed
comments on the survey questions from EPA's lead regional enforcement coordinator, and
OECA's Office of Compliance.
We issued the survey to EPA's 10 regional enforcement directors. Nine of the 10 regions
responded. The team contacted the unresponsive region (Region 6) on nine occasions to request
a response. Region 6 enforcement officials disagreed with the design of the survey instrument
and declined to respond to all of the survey questions (the region eventually responded to
32 percent of the survey questions). We consider Region 6 as nonresponsive, and as such, we
consider Region 6's lack of participation to be a scope limitation. However, we analyzed all
numerical and narrative survey results, and we believe that the region's lack of response does not
significantly affect our findings and conclusions.
1 States with missing values for the metric were not included in the total count of states.
12-P-0113 31
-------
Appendix B
Prior GAO and EPA OIG Assessments
GAO and the EPA OIG identified EPA oversight of states as a management challenge in
FYs 2008, 2009, and 2010. In our 2010 description of EPA management challenges, the
Inspector General said:
While EPA is renewing its attention on the oversight of programs delegated to
states, much remains to be done because the issues are complex and changeable.
Effective oversight of delegations to states is a continuous management
challenge that requires an agile organization, accurate data, and consistent
interpretations of policy.
Both GAO and the EPA OIG have frequently reported on problems with the EPA-state
enforcement relationship, noting key issues such as data quality, identification of violations,
issuing enforcement penalties and other enforcement actions in a timely and appropriate manner,
and general oversight issues. In our most recent report on state oversight, we found that outdated
agreements between EPA and states reduce EPA's ability to maintain a consistent water
enforcement program. A list of selected GAO and EPA OIG reports follows.
GAO
Environmental Protection Agency: Major Management Challenges, GAO-11-422T, March 2,
2011.
Clean Water Act: Longstanding Issues Impact EPA 's and States' Enforcement Efforts, GAO-10-
165T, October 15, 2009.
Environmental Enforcement: EPA Needs to Improve the Accuracy and Transparency of
Measures Used to Report on Program Effectiveness, GAO-08-1111R, September 18, 2008.
Environmental Protection: Collaborative EPA-State Effort Needed to Improve Performance
Partnership System, GAO/T-RCED-00-163, May 2, 2000.
Environmental Protection: More Consistency Needed Among EPA Regions in Approach to
Enforcement, GAO/RCED-00-108, June 2000.
Environmental Protection: EPA 's and States' Efforts to Focus State Enforcement Programs on
Results, GAO/RCED-98-113, May 27, 1998.
Water Pollution: Many Violations Have Not Received Appropriate Enforcement Attention,
GAO/RCED-96-23, March 20, 1996.
12-P-0113 32
-------
EPA OIG
Congressional Testimony Statement of Arthur A. Elkins, Jr., Inspector General, Before the
Subcommittee on Interior, Environment, and Related Agencies Committee on Appropriations
U.S. House of Representatives, "Major Management Challenges at the Environmental Protection
Agency," March 2, 2011.
EPA Needs Better Agency-Wide Controls Over Staff Resources, Report No. 1 l-P-0136,
February 22, 2011.
EPA Could Improve RCRAInfo Data Quality and System Development, Report No. 1 l-P-0096,
February?, 2011.
ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With
Additional Improvements, Report No. 10-P-0230, September 22, 2010.
EPA Should Revise Outdated or Inconsistent EPA-State Clean Water Act Memoranda of
Agreement, Report No. 10-P-0224, September 14, 2010.
EPA Needs to Improve Its Recording and Reporting of Fines and Penalties, Report No. 10-P-
0077, March 9, 2010
EPA Oversight and Policy for High Priority Violations of Clean Air Act Need Improvement,
Report No. 10-P-0007, October 14, 2009.
ECHO Data Quality Audit - Phase I Results: The Integrated Compliance Information System
Needs Security Controls to Protect Significant Non-Compliance Data, Report No. 09-P-0226,
August 31,2009.
Better Enforcement Oversight Needed for Major Facilities with Water Discharge Permits in
Long-Term Significant Noncompliance, Report No. 2007-P-00023, May 14, 2007.
EPA Performance Measures Do Not Effectively Track Compliance Outcomes, Report No. 2006-
P-00006, December 15, 2005.
Limited Knowledge of the Universe of Regulated Entities Impedes EPA 's Ability to Demonstrate
Changes in Regulatory Compliance, Report No. 2005-P-00024, September 19, 2005.
EPA Region 6 Needs to Improve Oversight of Louisiana's Environmental Programs,
Report No. 2003-P-00005, February 3, 2003.
State Enforcement of Clean Water Act Dischargers Can Be More Effective,
Report No. 2001-P-00013, August 14, 2001.
Enforcement - Compliance with Enforcement Instruments, Report No. 2001-P-00006, March 29,
2001.
12-P-0113 33
-------
North Carolina NPDES Enforcement and EPA Region 4 Oversight, Report No. 2000-P-00025,
September 28, 2000.
12-P-0113 34
-------
Appendix C
State Performance Analysis and Results
Using the methodology described in appendix A, we utilized EPA state enforcement data to
determine overall performance and level of consistency from state to state and region to region.
Our analysis of state performance showed that all of the regions included at least one state that
performed in the bottom quartile in the different program analyses (8 for CWA, 6 for CAA, and
7 for RCRA).
Figures C-l through C-3 show how state performance quartiles under the enforcement metrics
for inspections, SNC/HPVs identified and penalties varied both across statutes and
geographically. The figures cover the FYs 2003-2009 period, and divide states with authorized
programs into performance quartiles. The range of average performance was wide, at
approximately 41 percentage points for CWA, 50 percentage points for CAA, and 32 percentage
points for RCRA. This indicates high variability from state to state. The highest average
performance for CAA was over 60 percent (Alabama) and the lowest was almost 11 percent
(North Dakota). The mean performance was 16 percent (out of 100) for CWA, 39 percent for
CAA, and 18 percent for RCRA, which indicates that performance was relatively low across the
board. We calculated rough performance quartiles using EPA data for each statute (table C-l,
below), and modified the quartiles so that they followed natural breaks. The maps presented
herein offer a rough estimation of nationwide performance in conducting inspections, identifying
serious violations, and addressing violations with penalties.
Figure C-1: State CAA performance, FYs 2003-2009, as measured by rates of inspections,
penalties assessed, and HPVs
Legend
1 No Program
| | 1 Top Quartile
| 2 Second Quartile
^^| 3 Third Quartile
^^| 4 Bottom uuartil
# EPA Region
•
Source: OIG analysis of OECA data.
12-P-0113
35
-------
Figure C-2: State CWA performance, FYs 2003-2009, as measured by rates of inspections,
penalties (where states voluntarily reported them), and SNCs
Legend
I | No Program
I | 1 Top Quartile
_] 2 Second Quartile
^H3 Third Quartile
^^|4 Bottom Quartile
* EPA Region
Source: OIG analysis of OECA data.
Figure C-3: State RCRA performance, FYs 2003-2009, as measured by rates of inspections,
penalties assessed, and SNC
yam
Legend
I I No Program
I 1 Top Quartile
| 2 Second Quartile
| | 3 Third Qjartile
I | 4 Bottom Quartile
* EPA Region
Source: OIG analysis of OECA data.
Figure C-4 collates the results of figures C-l through C-3 by assessing which states fell into the
bottom quartile in two of three enforcement programs over the 7-year period of our analysis, and
thus warrant additional EPA oversight and intervention (shown in dark red). The figure also
identifies states that fell into the top quartiles for at least two of three programs (shown in dark
12-P-0113
36
-------
green). Even though this assessment does not indicate that these states are achieving EPA goals
or performing to other EPA expectations (as described in chapter 2), EPA may choose to reduce
its oversight and intervention in these states to refocus resources on states in the bottom quartiles.
To divide states into the categories below, we calculated the most common quartile across the
three statutes. When the three enforcement programs fell into three separate quartiles, we
categorized them as having no consistent pattern (shown in gray).
Figure C-4: Majority top-tier and majority bottom-tier states as measured by rates of inspections,
penalties assessed, and HPVs
Hawaii, Region 9
Legend
1 | 2 or 3 programs in top quartile
| | M osl programs In top half
| | No consistent pattern
I | Most programs in bottom half
^^| 2 or 3 program? in bottom quartiles
= EPA Region
Source: OIG analysis of OECA data.
12-P-0113
37
-------
Table C-l shows the average performance calculation for each state and each statute over the
FYs 2003-2009 analysis period based on information contained in EPA's enforcement database.
Table C-1: CWA, CAA, and RCRA state enforcement performance, FYs 2003-2009
State
AL
AK
AZ
AR
CA
CO
CT
DE
FL
GA
HI
ID
IL
IN
IA
KS
KY
LA
ME
MD
MA
Ml
MN
MS
MO
MT
NE
NV
Region
4
10
9
6
9
8
1
3
4
4
9
10
5
5
7
7
4
6
1
3
1
5
5
4
7
8
7
9
CWA NPDES (avg 2003-2009)
Insp.
14%
a1%
18%
12%
15%
6%
15%
27%
3%
10%
2%
EPA
21%
20%
20%
8%
15%
9%
19%
9%
EPA
9%
25%
7%
7%
4%
13%
7%
SNC
4%
a1%
2%
11%
1%
6%
4%
2%
0%
11%
0%
EPA
3%
16%
11%
<1%
5%
3%
17%
1%
EPA
5%
8%
3%
1%
5%
19%
0%
Penalty
100%
NA
NA
75%
NA
27%
NA
NA
53%
56%
NA
EPA
8%
NA
39%
31%
90%
22%
NA
40%
EPA
NA
36%
39%
15%
5%
NA
NA
Avg
39%
a1%
10%
33%
8%
13%
10%
15%
19%
26%
1%
EPA
11%
18%
23%
13%
37%
11%
18%
17%
EPA
7%
23%
16%
8%
4%
16%
4%
CAA Title V (avg 2003-2009)
Insp.
77%
28%
30%
61%
62%
28%
30%
59%
56%
36%
72%
13%
35%
57%
28%
52%
41%
31%
35%
45%
13%
30%
33%
34%
68%
33%
51%
50%
SNC
8%
3%
11%
3%
24%
6%
7%
5%
6%
5%
8%
10%
8%
4%
2%
2%
6%
15%
6%
6%
9%
2%
11%
5%
2%
9%
4%
10%
Penalty
97%
76%
68%
72%
85%
37%
56%
70%
90%
93%
89%
84%
12%
86%
45%
91%
50%
18%
94%
40%
83%
92%
72%
85%
92%
49%
22%
80%
Avg
60
%
36
%
37
%
45
%
57
%
24
%
31
%
45
%
51
%
45
%
56
%
35
%
19
%
49
%
25
%
48
%
32
%
21
%
45
%
30
%
35
%
41
%
39
%
41
%
54
%
30
%
25
%
47
%
RCRA Subtitle C (avg 2003-2009)
Insp.
5%
EPA
1%
2%
1%
4%
3%
4%
4%
6%
3%
4%
2%
4%
EPA
3%
11%
2%
2%
1%
2%
2%
1%
4%
3%
13%
3%
17%
SNC
2%
EPA
6%
11%
6%
2%
10%
2%
6%
5%
9%
4%
1%
2%
EPA
6%
2%
2%
9%
3%
8%
2%
4%
4%
1%
1%
4%
< 1%
Penalty
44%
EPA
22%
72%
65%
61%
78%
0%
74%
51%
32%
47%
25%
37%
EPA
84%
48%
8%
82%
11%
64%
40%
7%
94%
47%
25%
37%
32%
Avg
17%
EPA
10%
29%
24%
22%
30%
2%
28%
21%
15%
18%
9%
14%
EPA
31%
20%
4%
31%
5%
25%
15%
4%
34%
17%
13%
15%
17%
12-P-0113
38
-------
State
NH
NJ
NM
NY
NC
ND
OH
OK
OR
PA
Rl
SC
SD
TN
TX
UT
VT
VA
WA
WV
Wl
WY
Region
1
2
6
2
4
8
5
6
10
3
1
4
8
4
6
8
1
3
10
3
5
8
CWA NPDES (avg 2003-2009)
Insp.
EPA
5%
EPA
19%
23%
36%
23%
7%
6%
11%
4%
15%
12%
12%
4%
9%
18%
33%
< 1%
5%
33%
16%
SNC
EPA
3%
EPA
7%
1%
1%
21%
10%
<1%
1%
3%
3%
11%
23%
13%
2%
4%
1%
< 1%
4%
18%
0%
Penalty
EPA
28%
EPA
47%
100%
NA
55%
2%
NA
NA
3%
63%
19%
70%
NA
16%
NA
75%
14%
14%
NA
NA
Avg
EPA
12%
EPA
24%
41%
19%
33%
6%
3%
6%
3%
27%
14%
35%
8%
9%
11%
36%
5%
8%
26%
8%
CAA Title V (avg 2003-2009)
Insp.
25%
23%
18%
15%
84%
32%
33%
25%
43%
78%
26%
66%
73%
75%
33%
53%
40%
39%
71%
51%
20%
39%
SNC
9%
15%
5%
9%
3%
0%
19%
12%
6%
5%
17%
4%
1%
5%
43%
3%
1%
2%
7%
6%
8%
8%
Penalty
53%
93%
60%
97%
81%
0%
53%
84%
83%
94%
80%
86%
0%
88%
96%
83%
23%
83%
83%
44%
27%
63%
Avg
29
%
44
%
28
%
40
%
56
%
11
%
35
%
40
%
44
%
59
%
41
%
52
%
25
%
56
%
57
%
46
%
21
%
41
%
53
%
34
%
18
%
37
%
RCRA Subtitle C (avg 2003-2009)
Insp.
0%
4%
3%
1%
9%
3%
4%
2%
3%
4%
2%
4%
2%
4%
3%
2%
2%
3%
2%
7%
2%
6%
SNC
3%
7%
2%
3%
4%
0%
3%
2%
2%
1%
7%
7%
0%
10%
1%
4%
4%
3%
2%
1%
1%
3%
Penalty
64%
58%
64%
51%
22%
5%
69%
36%
42%
70%
45%
78%
29%
13%
48%
59%
24%
54%
50%
44%
44%
18%
Avg
22%
23%
23%
18%
12%
3%
25%
13%
16%
25%
18%
30%
10%
9%
17%
22%
10%
20%
18%
17%
15%
9%
Source: The data used in this assessment are from the OTIS Management Report tool. We downloaded the data in August 2010 and
queried OTIS for each program and each metric for FYs 2003-2009.
Notes: All percentages in this appendix are rounded.
"NA" indicates that the state did not report penalties to EPA. CWA penalty reporting is voluntary.
"EPA" indicates that the average for this statute was not applicable as a measure of state performance because EPA directly
implemented the program.
a EPA directly implemented the Alaska CWA program for FYs 2003-2007 (average CWA = 0.82%), and Alaska directly
implemented the program in FYs 2008 and 2009 (average CWA = 0.00%).
12-P-0113
39
-------
Table C-2 shows a comparison of the national performance average for several metrics to the
national goal and indicates what percentage of the reporting states fell below the goals and the
average using information contained in EPA's enforcement database.
Table C-2: Comparison of national goals and averages to actual state performance
Statute
CAA
CWA
RCRA
Metric
Percent of violations that are
considered significant
Percent of HPVs entered less
than 60 days after designation
Percent of major facilities
inspected every 2 years
HPVs identified per major
source
Percent of actions on HPVs
that include a penalty
Actions taken at major facilities
in violation
Percent of major facilities
inspected annually
Percent of toxic storage
disposal facilities inspected
every 2 years
Percent of large quantity
generators inspected every
5 years
SNCs identified per inspection
Percent of final formal actions
that include a penalty
Total
reporting
50
47
50
50
47
31
50
49
50
47
45
National
goal
<= 50%
100%
100%
>= 1/2 the
national
avg
>= 80%
>= 80%
100%
100%
100%
>= 1/2 the
national
avg
>= 1/2 the
national
avg
National
avg
46%
35%
89%
6%
89%
57%
61%
87%
62%
3%
81%
#below
goal
22
45
42
15
9
14
48
25
48
13
3
% below
goal
44%
96%
84%
30%
19%
45%
96%
51%
96%
28%
7%
#below
avg
28
24
15
32
13
14
22
15
10
27
15
% below
avg
56%
51%
30%
64%
28%
45%
44%
30%
20%
57%
33%
Source: OTIS State Review Framework tool.
and region for FY2010.
We downloaded the data in March 2011 and queried the database for each statute
12-P-0113
40
-------
Appendix D
Agency Comments on Draft Report
and OIG Responses
MEMORANDUM
SUBJECT: Inspector General's (OIG) July 28, 2011 Draft Evaluation Report, Project No.
OPE-FY10-0022, "EPA Must Improve Oversight of State Enforcement"
FROM: Robert Perciasepe
Deputy Administrator
Cynthia Giles
Assistant Administrator
Office of Enforcement and Compliance Assurance
TO: Arthur A. Elkins, Jr.
Inspector General
We appreciate the opportunity to review and comment on the Office of Inspector
General's (OIG) July 28, 2011 draft Evaluation Report entitled: "EPA Must Improve Oversight
of State Enforcement" (Project No. OPE-FY10-0022). We thank you for tackling this
challenging topic, which is critically important to the protection of human health and the
environment and to achieving compliance, as states are the "front lines" of implementing federal
environmental laws.
Summary comments
EPA agrees with your overall finding that state enforcement performance varies widely
across the country. We also agree that there are steps EPA Headquarters and regional offices can
and should take to strengthen our oversight and address longstanding state performance issues.
We strongly support making state performance information publicly available in an easy-to-
understand format to help apply pressure to improve both federal and state government
performance.
EPA's principal concern with the draft Report is the limited number of metrics and the
associated methodology relied on by the OIG to assess state performance. To fairly evaluate a
state enforcement program is a complex undertaking that requires a thorough analysis of multiple
factors considered within the context of a state's overall enforcement activities. Unfortunately,
the methodology of state evaluation adopted by the OIG for this Report - averaging performance
across programs using a very limited number of metrics - oversimplifies what, of necessity, must
be a more comprehensive review process. We are concerned that publication of the data using
these limited metrics could have the unintended consequence of moving states toward a more
simplistic, and less protective, enforcement program. Moreover, within the metrics utilized by
the OIG, data errors and misunderstanding about information in (or missing from) the data have
12-P-0113 41
-------
occurred such that the data and goals presented and relied upon by the OIG are unreliable. EPA
is requesting that these issues be addressed before the Report is released so that the important
conclusions of the Report and the need for further action remain the focus.
OIG Response 1: While generally agreeing with our findings, EPA took issue with some
aspects of our methodology. EPA's extensive comments contained here reflect these concerns.
In summary, EPA was concerned with how we used information about state activities to
compare state program quality to screen states for further in-depth review. Although not
mentioned in its comment, we utilized EPA's own enforcement data within our review, and our
analysis is similar to those that EPA itself conducts. These data possess several significant
limitations that EPA has acknowledged and which we took steps to overcome and describe.
However, we used several other sources of information in our evaluation, such as descriptions
of states contained in the SRF reports and interviews with officials and staff of EPA and states.
Our conclusions rest on information contained in all of these sources.
We affirmed our choice of the three measures of state activity based on discussions with OECA
staff and officials, and other stakeholders. Their use is supported through statements contained
in OECA's own documents including the SRF, and EPA displays the data on its publicly
available enforcement website. However, because we realize that no single set of measures is
individually definitive, we corroborated our findings through numerous interviews across six
regions and document review. EPA's agreement with the substance of our findings provides
further support for the validity of our approach. Details of our methodology are presented in
chapter 1 and appendix A.
We appreciate your acknowledgement that significant work has been done by EPA, with
more underway, to improve our oversight of state enforcement programs within a larger
framework of federal and state shared accountability in the enforcement of our nation's
environmental laws. And yet, new work has occurred since FY2009 that is not well described in
the Report, such as the publication of our state review documents and a web-based state
performance and comparison tool, which have increased transparency and public accountability
of state enforcement programs. Additionally, in June 2010, the Office of Water (OW) and the
Office of Enforcement and Compliance Assurance (OECA) jointly issued guidance to the
regional offices on ensuring consistent enforcement by states of Clean Water Act permits.
Lastly, and importantly, EPA is in the midst of significant advances that could dramatically
change the effectiveness and accountability of federal and state enforcement programs. By way
of example, recent actions include developing a proposed rule for electronic reporting of
National Pollutant Discharge Elimination System (NPDES) permitting and compliance
information, and a new pollutant loadings targeting tool. This work demonstrates EPA's
commitment to the principles that underlie your Report and our willingness to pursue innovative
measures to secure better performance in the future.
OIG Response 2: We describe many of EPA's efforts to improve state performance in the
beginning of chapter 2 of the report.
12-P-0113 42
-------
Methodology, metrics and data
Rarely do states under-perform across the board in all media programs. More
commonly, states perform well in one program area and not well in others. Sometimes
performance issues are related to particular sectors. These layers of performance are difficult to
measure and convey through simple data metrics.
As noted above, the metrics relied on by the Inspector General in this draft Report are
overly simplistic, and in some cases inaccurate, thereby resulting in erroneous conclusions
regarding individual state enforcement performance. The use of limited data presents an
incomplete picture of state enforcement programs, and fails to provide an accurate evaluation of
the quality or other contextual aspects of complex state enforcement performance. Although we
sympathize with the desire to keep it simple, we are concerned the Report, as currently
presented, will give the public a false impression of state performance by publishing both
inaccurately positive and inaccurately negative state evaluations.
OIG Response 3: As described in chapter 2, in response to EPA's comments, we eliminated our
use of a summary metric for each state. However, we disagree that presenting a simplified
method for comparing state enforcement performance is invalid. Repeated assessment of
indicators allows management to determine where EPA should focus additional attention. As
with any other performance measure, the measurement only takes on meaning when it is
compared with benchmarks that clearly outline success. The overarching message of this report
is that EPA must establish clear benchmarks for success so that states, EPA regions, EPA
headquarters, the public, and the regulated community all understand what is required to ensure a
safe and healthy environment. The report emphasizes that the states we ranked highest are not
necessarily succeeding, but rather that "performance is low across the board." With clear
benchmarks, according to these or other performance indicators, EPA, states, and the public can
clearly understand performance, and EPA can take deliberate steps to improve performance
where it is needed.
Our specific comments on the metrics and methodologies used by the Inspector General
and the text of the draft Report are listed in the Attachment to this memorandum, but some
overall points are listed below.
1. The metrics chosen by OIG overemphasize major facilities and inspections.
Large facilities often are significant sources of pollution. Consequently, compliance at
these facilities is, and remains, a priority. However, EPA increasingly is realizing that large
cumulative impacts and locally significant pollution is associated with violations occurring
collectively at smaller facilities. In the water program, for example, stormwater and agricultural
sources are often the most significant sources of water quality impairment. In RCRA, there is
increasing evidence that smaller quantity generators are disproportionately responsible for
violations that could pose a risk of release. In addition, pollution from large facilities not
technically classified as "majors" can be very important. An example from the water program is
mining facilities, which often contribute large pollution loads but are not captured in the
definition of majors. In our work on state oversight we have been pushing for greater
12-P-0113 43
-------
accountability for these other categories of sources, and moving away from the historic and
nearly exclusive focus on majors. States that have responded to this shift in emphasis and
refocused resources toward the most significant pollution problems would fare badly on the
metric OIG uses, which may include some minor source data, but in fact relies almost entirely on
data about majors. This will give an inaccurate and unfairly negative picture of these states'
performance, potentially harming the overall effectiveness of their enforcement programs.
As the compliance program has matured, and our understanding of pollution problems
has grown, we have acknowledged that in addition to inspections there are other ways to
ascertain compliance. As we have discussed, we are moving toward a new paradigm where self-
monitoring and electronic reporting will be increasingly important tools to supplement
inspections. Government resources will be targeted to the most serious problems and we will
have a much better idea overall about the compliance picture than we do today. We can be more
effective and efficient, both at the federal and state level, through better monitoring and reporting
and public disclosure of pollution and compliance information. These approaches also
encourage better compliance performance through the power of public accountability. While it
is fair to evaluate states based on their performance in inspecting facilities as required in the
various program compliance monitoring strategies, we are concerned that the Inspector General's
overly heavy emphasis on inspections of majors as a measure of state performance weights the
scale in the wrong direction.
OIG Response 4: We note again that the metrics the OIG utilized are the same ones that OECA
uses to manage its programs, and we selected them in coordination with OECA. We agree that
enforcement should be targeted at the most significant sources of pollution, whether these
sources are majors or a collection of minors. We would use information about nonmajors if
reliable information on this were available. However, EPA does not consistently collect
information about compliance and enforcement at nonmajor sources. The OIG has reported on
data quality numerous times in the past and it is an acknowledged weakness of EPA reporting
systems. As such, it is extremely difficult to assess how well states ensure compliance and
conduct enforcement at smaller facilities.
2. Rates of identification of significant violations are not an informative metric.
In the abstract, comparing rates of violation would be an excellent metric for evaluating
state performance. However, available data suggests that some states significantly under-report
significant violators. Therefore, it is likely that low rates of identified significant violators do not
reflect actual low rates of violation, but rather incomplete or mischaracterized data. If this is
correct, then higher reported rates of violation are more likely to be accurate, and states with
such higher rates likely to be higher performing states, if identification and disclosure of
violations is the metric. As a consequence, self-reported rates of significant violation by states is
an ambiguous metric at best, since either low rates of violation or high rates of violation could be
associated with a high performing state compliance program and vice versa. If the Inspector
General uses this metric as one of just three, this significantly oversimplifies an otherwise
complex evaluation. In our judgment, greater reliance on self-monitoring, electronic reporting,
and public disclosure of enforcement data will result in more complete and credible compliance
information in the years ahead, greatly increasing public transparency and the deterrent effect of
12-P-0113 44
-------
compliance oversight. With the addition of the new data mentioned above, the Agency should
be able to use this metric with greater confidence in future years.
OIG Response 5: As with any measurement, SNC identification and reporting requires
additional information to provide comprehensive understanding of performance. However, SNC
rates are a valid measure of state performance, and EPA uses them as one measure of
performance. Our analytical method recognized underreporting as a potential issue, and as such,
we used higher SNC identification rates as an indicator that the state targeted inspections and
identified violations. We came to this decision based, in part, on conversations with EPA
enforcement staff.
Moreover, it is uncertain whether a greater reliance upon self-reporting will improve data
quality. EPA's recent comparison of randomly generated compliance rates with results of
targeted inspections and self-reported violations indicated that self-reported violations were
underreported.
1. Some of the data presented by OIG appears to be incorrect
EPA's regions have been unable to replicate the OIG data pulls, and it appears that OIG
researchers have misunderstood some of the data, leading to inaccurate statements regarding the
meaning and significance of EPA's data on state enforcement. The draft OIG Report also
contains incorrect descriptions of several specific instances of state oversight by EPA, e.g., the
Agency's response to a 2001 petition to withdraw the Louisiana NPDES program, the Agency's
reaction to program performance issues in Illinois, and EPA's review and approval of the NPDES
permitting program in Alaska. A full discussion of these cases, along with the corrected
information, appears in the Attachments.
OIG Response 6: In response to this comment, we expanded the description of our
methodology and quality assurance steps in appendix A. EPA's enforcement database allows
corrections to be made by states. Because we downloaded the data used in our analysis from
EPA databases in August 2010, if states subsequently modified their data, the specific results
may vary slightly if the analysis were repeated today.
We reviewed the additional information about Louisiana's NPDES program provided in the
supplied attachment and determined that it did not change our overall interpretation of events.
We based our descriptions of specific instances of state oversight on several analytical methods,
including interviews with the regions and the states, and data and document review. We believe
our interpretation of the events accurately portrays the information that we gathered from all
stakeholders. However, where appropriate we updated the narrative to elaborate and accurately
reflect the additional information provided to us. In fact, this new information more starkly
demonstrates the ineffectiveness of EPA intervention in the state. Despite the region and
headquarters involvement, the state performance has not improved substantially. This failure
underlines the importance of recommendations 1 and 5, which would allow OECA to direct a
national workforce to respond to enforcement crises like those experienced in Louisiana.
For responses regarding Louisiana, Illinois, and Alaska, see subsequent OIG responses.
12-P-0113 45
-------
We appreciate OIG's willingness to meet to help us better understand the information
utilized in the OIG analyses. It is important that data errors be identified and corrected, and
information presented consistently for all states. For example, there are several instances where
percentage scores for states exceeded 100%, clearly an error. This and other examples of data
inaccuracies in the draft Report, as identified in this memorandum and its attachment, should be
corrected. Importantly, the metrics identified by the OIG should accurately reflect the data in
EPA's information systems, because this data is central to the conclusions that OIG reaches in
the draft Report.
OIG Response 7: The information in the report accurately reflects the data contained in EPA's
information systems. In terms of the example provided, we discovered instances in EPA's
database where state data exceed 100 percent for a category of enforcement action. This is an
artifact of how EPA's enforcement data system calculates this measure. [The database
calculates this metric automatically by dividing the number of state actions with penalties
(numerator) by the number of state formal actions (denominator). However, the database counts
one category of formal actions in the numerator but not in the denominator ("AO Stipulated
Penalty 1"). The result is that the percent of formal actions with penalties can be greater than
100 percent if a state issued a lot of "AO stipulated penalty 1" actions.] Because this calculation
method was consistent across states, in our draft report we used the number as presented in the
database.
EPA's new approaches to enforcement and state evaluation
Complete data and valid, meaningful measures are key to understanding state
performance. Gaps in our current data make it difficult to develop measures that tell a complete
story across all media and regulated sectors. Limited resources at both the state and federal
levels make it more difficult to address these gaps. Measures based on the data that EPA does
have may not focus on the right things. In our opinion, an effective way to address these issues is
to set a new direction in the enforcement program that takes advantage of the advances in
information and monitoring technologies. This will enable us to get more complete data through
the efficiencies of electronic reporting and field equipment. This data will allow EPA to more
completely identify regulated sectors, monitor and assess compliance, target resources more
effectively, and measure the overall performance of federal and state regulators in a more
meaningful way.
An example of this approach is the Clean Water Act Action Plan. In the NPDES
program, EPA has focused its resources and attention on the biggest facilities, the majors.
Despite the focus on majors, compliance at those facilities is not what it should be. In addition,
over time, wet weather and other sources have been identified as having significant impacts on
water quality. EPA needs to expand its knowledge and data of these smaller regulated sources
and target attention on addressing the environmental damage and health impacts they cause. By
utilizing electronic reporting, better monitoring, and new compliance strategies, the Agency will
eventually build the capacity to develop new performance measures that include these sources.
EPA has begun to implement this new strategic vision by taking near-term actions such
as the development of new pollutant loadings targeting tool, electronic reporting of NPDES
12-P-0113 46
-------
Discharge Monitoring Reports (NetDMR), and piloting the use of third party vendors to build
electronic reporting tools for facilities to report DMRs. The Agency also has initiated longer
term actions such as developing the aforementioned NPDES e-reporting proposed rule, building
e-reporting and new compliance strategies into rules, and integrating permit and enforcement
annual planning and reviews. This vision will take some years to achieve. Until then, we must
recognize the limitations of the data and metrics we currently have and not put so much weight
on what is an admittedly incomplete, and in some cases potentially misleading picture.
OIG Response 8: We agree that ideal performance data are repeatable, reliable, and relevant.
However, we also believe that EPA underrelies on its vast stores of state enforcement data,
which it has collected for decades. Our scope and methodology descriptions in chapter 1 and
appendix A describe the steps we took to ensure the data we used were reliable for our purpose.
EPA supports the use of national maps to compare state enforcement performance. This
past spring, EPA released Clean Water Act state dashboards that display data in a similar fashion
across states. EPA is, however, concerned that, by not using the right metrics and data for
comparison purposes, the OIG draft Report will lead to false conclusions regarding state
performance. For instance, states that have had deficiencies noted in State Review Framework
evaluations, based on extensive file reviews and other contextual information beyond the data in
EPA's systems, may look good in the limited data set pulled and displayed by the Inspector
General. In this case, the unintended consequences are that the careful reviews done by the
regions may be undermined by the analysis presented by the Inspector General, and the public
may be falsely reassured that the state program is stronger than it in fact is. Another possible
unintended consequence is that there may be states that perform well across their regulated
universes, but our data system only contains required reporting on majors, making these states
appear to have performance problems. Another issue is the Report's failure to note that the
flexibility provided in our Compliance Monitoring Strategies allows states to negotiate
alternative compliance monitoring plans annually. These plans are specifically negotiated to
ensure that states address their most important sources of pollution and noncompliance. The
Agency is working on how to make these alternative plans transparent to the public so states are
held accountable to their plans. It is important for us to recognize these issues as we work to
improve consistency, transparency, and performance.
OIG Response 9: The OIG reviewed the OECA dashboard and map system under
development. At the time of our evaluation, this information was under development for the
CWA and not yet available for the CAA or RCRA programs, which were included in the scope
of this evaluation. The OIG applauds EPA's efforts in finding new approaches to address state
enforcement needs. We did not base our analysis on the data alone, but also on several
additional information sources, described in our detailed methodology.
At the same time, in response to EPA's concerns we eliminated our use of a summary metric for
each state. However, we disagree that presenting a simplified method for comparing state
enforcement performance is invalid. Repeated assessment of indicators allows management to
determine where EPA should direct the most attention. As with any other performance measure,
the measurement only takes on meaning when it is compared with benchmarks that clearly
outline success. The overarching message of this report is that EPA must establish clear
12-P-0113 47
-------
benchmarks for success so that states, EPA regions, EPA headquarters, the public, and the
regulated community all understand what is required to ensure a safe and healthy environment.
The report emphasizes that the states we ranked highest are not necessarily succeeding, but
rather that "performance is low across the board." With clear benchmarks, EPA, states and the
public can clearly understand performance, and EPA can take deliberate steps to improve
performance where it is needed.
EPA's ability to measure and its oversight of state program performance is evolving as the
Agency continues to stress the need for transparency and a focus on the most important sources
of pollution affecting environmental quality. We are working to develop better tools and
approaches to improve the consistency of regional oversight of states and address known
performance issues. Examples include:
• The State Review Framework (SRF) has been a big step forward in establishing a
structured review process for state enforcement programs and promoting regional
consistency in conducting oversight. Nevertheless, there are still inconsistencies in how
regions draft reports and recommendations, and in their responses to performance issues.
To improve the SRF, we are streamlining the data metrics, verifying annual data sets,
improving reporting, and integrating the SRF with NPDES permit quality reviews.
• Increasing use of new web products that provide easy-to-use summary information using
national maps comparing state enforcement and permit program performance. The
Agency posted a CWA state dashboard in the spring, and is working on similar maps for
the CAA, RCRA, SDWA and state pesticide programs. We think this information goes a
long way toward the kind of useful comparison and transparency that the OIG is
promoting in this Report, with significantly less opportunity for mischaracterization.
• The CWA Action Plan has a detailed implementation plan covering four primary changes
in how we do business:
o Mandating electronic reporting from regulated facilities for DMRs, notices of
intent to discharge, and various program reports in a rule.
o Reviewing and revising CWA guidance and policies to ensure the ability of EPA
and states to address the most significant environmental problems, and address
them in an appropriate manner using a tiered approach to addressing violations.
o Building new compliance approaches into rules such as electronic reporting; self
monitoring, certification and reporting; use of new technology to monitor; and
third party certification programs.
o Integrating permit quality and enforcement annual planning and reviews.
OIG Response 10: We describe many of EPA's efforts to improve state performance in the
beginning of chapter 2 of the report.
The regions have a range of tools and approaches to address state performance that go
beyond the SRF, and which are important in looking at issues not contemplated in the
SRF. Region 5's actions in Illinois (which are publicly available at
http://www.epa.gov/Region5/illinoisworkplan) are one example, as they address both
permitting and enforcement issues and go well beyond the analyses and
recommendations under SRF. These actions have already yielded significant results and
12-P-0113 48
-------
meaningful improvements to Illinois' program and are a direct result of the Region's
active engagement with the state.
OIG Response 11: We commend EPA's efforts to improve Illinois' program, and described
our knowledge of the process in chapter 2 of the report.
Responses to specific recommendations
Despite serious concerns with the metrics and methodology of the draft Report, and the
unintended consequences should these be included in the final Report, EPA agrees that states
should be held accountable for addressing state performance issues that are identified using the
SRF and other available tools while the Agency develops and implements a new enforcement
paradigm. Specific comments on the recommendations made in the draft Report are as follows:
Recommendation 1: Give OECA authority for all nationwide enforcement resources and
workforce allocation.
EPA does not agree that centralizing resources and workforce allocation will address the
concerns raised by the Inspector General concerning national inconsistency in state performance
and regional oversight. OECA currently exercises significant central authority for enforcement
resources and general workforce allocation, provides national direction through the Strategic
Plan, National Program Managers Guidance and the ACS commitment process, and holds
regular meetings and calls with regions to discuss performance and oversight. The degree of
control over a dispersed workforce anticipated under this recommendation would not be
substantially changed from what currently exists and would not lead to the improvements
envisioned by the Inspector General.
The national perspective, necessary to achieve a level playing field for businesses and
states and equal protection for the public, needs to be balanced against the value of applying
local, on-the-ground knowledge and necessary, ongoing relationships to solving specific state
and regional environmental issues. Regions need the discretion to tailor the national approaches
to their and their states' unique and individual needs. OECA cannot appropriately be positioned
to maintain the level of specific knowledge across the country necessary to ensure that each state
is addressing its most important sources and to tailor fixes to identified problems that make
sense.
OIG Response 12: Maintaining oversight of a national enforcement program requires a
centralized capability to manage a national enforcement workforce. EPA does not currently
possess this capability because most enforcement resources are managed by the 10 regional offices
rather than the OECA. Because of this, EPA does not adjust resources to meet changes in EPA's
enforcement mission and goals. This leads to inefficiency, which is one of EPA's future challenges
if resource constraints increase. In addition, dispersed decision-making leads to disparities in state
expectations and consequently in their performance. These are concerns that EPA shares as
evidenced in its comments to our draft report. These disparities limit the protection of the public
from environmental pollution in states and regions where standards are not the same. As a result,
we disagree with EPA's response and consider recommendation 1 unresolved.
12-P-0113 49
-------
Recommendation 2: Cancel outdated guidance and policy documents, and consolidate and
clarify remaining guidance into EPA documents that are publicly and easily accessible on
the EPA Civil Enforcement website.
EPA agrees that it can separate current guidance and policy from historic policy that may
still need to be accessible to regulators and the public, and clarify what is currently applicable for
state programs. We reserve judgment until further review as to how to do this and where to post
appropriate documents for public access.
OIG Response 13: We look forward to reviewing the Agency's action plan for addressing
recommendation 2 in the final report response.
Recommendation 3: Establish clear and consistent national enforcement benchmarks
throughout CAA, CWA, and RCRA guidance and policies so that EPA's enforcement
expectations are clear and consistent for state governments and the regulated community.
EPA agrees that we should have clear national enforcement goals and benchmarks so
regions and states are clear on what expectations apply. OECA's national expectations are laid
out in current compliance monitoring, enforcement and penalty guidance and policies. These
expectations are generally identified as program goals, with reviewers of state performance
looking at how individual states match up both to the goals and to comparison performance
across states. As EPA refines and streamlines the metrics for the third round of the State Review
Framework, we will seek ways to ensure that these goals and benchmarks are clear and balance
national consistency with regional and state flexibility. As we move to the new paradigm, we
will work collaboratively with regions and states to develop appropriate expectations and
benchmarks.
As part of our improvements in increasing public access and understanding of
government performance, EPA will explore how to make specific expectations, benchmarks and
commitments visible to the public through our state dashboards and other public web sites. Part
of the lack of clarity of guidance is that while guidance changes, the evaluation of performance is
done retroactively against the policies in place at the time of the performance. This may lead to
confusion as to which guidance the state is being held accountable for in any given year. EPA
agrees to try and clarify which guidance applies to each performance year as reviews continue
and to make that information available to the public.
EPA seriously questions the value of codifying guidance applicable to state enforcement
programs in rules. Unlike rules, guidance does not have the force and effect of law, is intended
to be more flexible in its application, and can be modified and improved more quickly to meet
changing needs and circumstances. In contrast, the rulemaking process is both resource and time
intensive. A rule may be promulgated, and modified, only in accordance with well-defined
public comment and hearing procedures. The rulemaking process, thus, is ill-suited for dynamic,
case-specific situations like state enforcement program reviews. The use of guidance in the
context of state oversight is particularly appropriate in that it allows the EPA to quickly respond
to state requests for clarification on various enforcement-related issues, identify data and
12-P-0113 50
-------
information the Agency will rely on as part of the review process, and provide insight as to
EPA's priorities when undertaking individual state enforcement reviews.
OIG Response 14: We believe the Agency is on the right track when it comes to addressing
recommendation 3. We look forward to seeing the complete action plan containing the steps the
Agency intends to take to address this recommendation in the final report response.
Recommendation 4: Establish a clear and credible escalation policy for EPA intervention
in states that provides steps that EPA will take when states do not act to ensure that the
CAA, CWA, and RCRA are enforced.
EPA agrees that a national strategy that describes escalating actions for regions to take to
address state performance issues would be beneficial. We want to be clear that the purpose of
escalation is to resolve performance issues and improve state programs, and to do so at the
lowest management level possible. We do not agree that when state performance is a problem
that EPA's response should always be to step in and initiate an enforcement action. Our goal is
to work to strengthen state performance so that the state can again administer a strong
enforcement program. Moreover, enforcement cannot be viewed in isolation from the
performance of a state's delegated or authorized programs as a whole. The ability to run an
effective state enforcement program is greatly compromised where state media programs,
standards, plans or permits are weak or otherwise ineffectual.
While program withdrawal may be seen as a necessity in cases of severe program
deficiencies, program withdrawal is not the goal of escalating oversight. The environmental
laws of this country clearly envision the states be in the forefront of day-to-day program
implementation. While not the goal, we recognize that program withdrawal is a necessary tool in
the oversight toolbox.
OIG Response 15: We believe that EPA was nonresponsive to recommendation 4. We agree
that program withdrawal should be a tool in the oversight toolbox. We will expect the Agency
to address this recommendation by providing a plan of action in the final report response.
Recommendation 5: Establish procedures to reallocate enforcement resources to intervene
decisively when appropriate under its escalation policy.
This is a concept that warrants further Agency consideration. The budget constraints
EPA is currently under present a challenge to implementing this recommendation. EPA believes,
however, that if program withdrawal is warranted because of significant and severe issues across
a delegated or authorized program, the Agency needs to have the capacity to carry it out. The
Agency will explore further what options may be available regarding program withdrawal when
it is truly in order to make it a credible feature of EPA oversight.
OIG Response 16: We believe that EPA was nonresponsive to recommendation 5. Contrary to
EPA's point of view, we feel that EPA adopting the procedures noted here especially at the
present time will enhance budgetary efficiency. We consider it unresolved, and expect the
Agency to address this recommendation by providing a plan of action in the final report
response.
12-P-0113 51
-------
Conclusion
EPA agrees with the OIG that state enforcement performance varies significantly and that
EPA Headquarters and regional oversight can be improved. EPA generally agrees with the
direction and recommendations in the draft Report that seek to update enforcement guidance and
policy, clarify expectations and identify escalating steps for regions to take to address
performance issues. OECA has already taken steps to improve national consistency in the
enforcement program across regions and states:
• Developing and implementing the State Review Framework (SRF)
• Making SRF metrics and documents public
• Developing state performance dashboards, currently for the CWA but
including CAA and RCRA in the future
• Discussing state performance at national meetings with regions and states and
having state oversight as a top priority goal for enforcement
• Working with regions to identify steps to address state performance issues
EPA acknowledges its limitations in establishing meaningful measures for evaluating
performance. With this in mind, the Agency is pursuing a new paradigm for enforcement through
the use of 21st Century technology to improve data collection and to monitor compliance. Once
e-reporting, the use of new technologies in compliance monitoring and new compliance
strategies in our rules are in place, we are confident the Agency will have more and better data
with which to evaluate state enforcement performance. Until that time, we are in transition and
need to be cautious about putting too much weight on the limited metrics for which we do have
data. We need to avoid the unintended consequences that occur when making inaccurate,
misinterpreted or confusing data available to the public without adequately explaining the
limitations of the data and its significance. For these reasons, EPA has serious concerns with the
metrics and methodology utilized by the Inspector General in conducting this review.
EPA agrees more needs to be done. More complete data and better measures will help us
do a better job of holding states accountable and portraying the complexities of state
performance to the public. This is a process of continual learning and improvement, the fruits of
which may not be evident for a number of years. Regardless, it is critically important that we
take steps now if we are to achieve nationally consistency and shared accountability in the
enforcement of the nation's environmental laws by EPA Headquarters, the regions and our state
partners.
Should you have any questions or concerns regarding this response, please contact
OECA's Audit Liaison, Gwendolyn Spriggs, at 202-564-2439.
Attachments
12-P-0113 52
-------
Appendix E
EPA Attachments to Draft Comment Memorandum
and OIG Responses
Attachment: Detailed Comments on OIG Draft Report OPE-FY10-0022
"EPA Must Improve Oversight of State Enforcement"
Substantial Issues with the State Performance Analysis Methodology
EPA feels that the key conclusions reached in the OIG analysis - that there is demonstrable
inconsistency in state performance and regional oversight, and that EPA should take action to
raise state performance where it is poor - are valid conclusions supportable with pre-existing
EPA data and analysis. However, there are a number of issues associated with the data, metrics,
and methodology used by the OIG to produce the results shown in Table B-l in Appendix C of
this Report. EPA feels that the map on page 7, the regional comparison chart on page 9, and
Table B-l all provide an inaccurate picture of state performance.
OIG Response 17: The OIG stands by its methodology. In response to EPA's comments, we
provided additional detail about our overall methods in chapter 1 and appendix A. We described
in chapter 1 how the metrics allow for state-to-state performance comparison based on priority
issues to EPA. However, without corroboration they do not offer reliable details about
individual state performance. In addition, we made some modifications to the presentation of
data in this final report. These changes are described in chapter 2, and below, in additional OIG
responses.
Issues: First, there are errors in the data collection and presentation that casts many of the raw
figures in the table into doubt. Second, the three metrics chosen to support the overall analysis
are ambiguous when examined individually and without additional context: each has several
variables which affect their meaning and create uncertainty for the purpose of being able to draw
conclusions or inferences. Third, rolling up metric averages into one average obscures
differences and hides complexity; and, averaging the averages across programs — which treat
those program elements differently - further compounds the errors. Lastly, assessing program
performance is complex. Data alone cannot adequately describe performance. EPA uses a
number of metrics, both federal and state data, file reviews, and interviews with the states to
understand state programs.
OIG Response 18: In response to this comment, we expanded the description of our
methodology and quality assurance steps in appendix A. EPA's enforcement database allows
corrections to be made by states. We also eliminated our use of a summary metric for each state
On August 31, 2011, OECA and the OIG met to discuss these significant issues with the draft
Report identified by OECA and the regions. OECA felt that the discussion was very constructive
and that the OIG was engaged and receptive to the feedback. EPA requests that the OIG address
these issues by working with experts in OECA's Office of Compliance (OC) on more accurate
12-P-0113 53
-------
data pulls and utilization and display of the data the OIG is interested in using. While OECA
recognizes the independence of the OIG in doing their reviews and analyses, OC staff are
available to the OIG to help produce and share accurate data, demonstrate current tools and data
displays, and provide knowledge and experience regarding the metrics chosen in the draft
Report.
OIG Response 19: We appreciate OECA discussing its concerns with our report. However, this
criticism is unfounded. Subsequent to the August 31 meeting, the OIG requested the data
mentioned in the OECA comment. The data that OECA provided to us covered 2 years for only
one statute (CWA data for 2008-2009). Because the scope of this evaluation covers 7 years
(2003-2009) and CAA and RCRA statutes in addition to CWA, this was not a comparable
substitute for the years of data contained in EPA databases. Moreover, for the limited data
provided (the 2008-2009 CWA data), we conducted statistical analyses and determined that an
assessment of state performance using the new information was materially no different from the
data we used in our analysis.
Inaccurate Data
EPA regions have commented that they were unable to reproduce the data or analyses as
presented in the Report. EPA recommends that OIG and the regions work together to understand
how the data was collected and if there are errors that can be corrected.
OIG Response 20: In response to EPA's comments, we included additional details about how
we collected and summarized EPA enforcement data in appendix A.
Average Percentage of Facilities Inspected ('03- '09)
It is not possible to draw valid conclusions from the inspection coverage metric as presented in
Table Bl because of the sheer number of variables that affect the data. Some examples of the
types of data errors in the Report include:
• The calculation method used for the percentage of facilities inspected misrepresents state
inspection performance. For each statute that was reviewed in this Report (CWA, CAA,
and RCRA), it is not clear how the data was pulled and if it included the entire universe
for each media in the corresponding EPA data system. Universe numbers in both the
CWA and RCRA Subtitle C are not complete or accurate in EPA systems.
• In the NPDES program, states are only required to report detailed information
about major facilities, which represent about 6,700 facilities out of a universe of
close to a million. Yet, some states that use the federal system for their own
database enter in additional sectors that may have been mistakenly included in the
totals used for averaging. This could introduce inconsistency into the analyses of
these numbers across states, and could serve as a disincentive for states to report
information above what is required.
• The RCRA C universe is known to fluctuate greatly as facilities may periodically
change status from a Large Quantity Generator (LQG) to a Small Quantity
Generator (SQG). This introduces a margin of error that distorts analyses without
correction.
12-P-0113 54
-------
• The OIG did not control for the inspection frequency goals that relate to facility
size, choosing to lump together facilities - big, medium and small. Because the
overwhelming number of facilities are small, the inspection percentages are
skewed and do not dovetail with inspection goals.
OIG Response 21: We note again that some of the metrics OIG utilized are the same ones that
OECA uses to manage its programs, and we selected them after consultation with OECA. We
agree that enforcement should be targeted at the most significant sources of pollution, whether
these sources are majors or a collection of minors. We would use information about nonmajors
if reliable information on this were available. However, EPA does not consistently collect
information about compliance and enforcement at nonmajor sources. The OIG has reported on
data quality numerous times in the past, and the issue is an acknowledged weakness of EPA
reporting systems. As such, it is extremely difficult to assess how well states ensure compliance
and conduct enforcement at smaller facilities. The metrics factor in the size of a state's
regulated universe by using percentage of facilities inspected. In response to EPA's comments,
we included additional details about how we collected and summarized EPA enforcement data
in appendix A.
Percent of Significant Noncompliance Identified per Inspection
• For the CWA program, most SNC is identified via self-reports rather than inspections.
This is not an appropriate comparison.
OIG Response 22: We described in chapter 1 how the metrics allow for state-to-state
performance comparison based on priority issues to EPA. However, without corroboration they
do not offer reliable details about individual state performance. The database calculates the
SNC metric via self-reports for states, so this was a consistent metric to use for a cross-state
comparison.
Percentage of Formal Actions with Penalties
• CWA penalties are not required to be entered by states at this time. Eighteen states were
listed at 0%, though it cannot be determined if there are actually no penalties assessed or
if the data was not reported.
• Some of the numbers appear to be erroneous (e.g., Alabama has 165% of formal
enforcement with penalties, which is not possible and inflates their performance score).
• Analysis of appropriateness of penalties (amount assessed and whether there was a return
to compliance) was not analyzed using the OIG methodology.
OIG Response 23: In response to EPA's comments, we clarified our scope and methodology.
There are two considerations regarding this measure for the CWA. First, and most importantly,
states are not required to report penalty information. Therefore, it was not possible to determine
whether a zero in EPA's database is a true value or a nonresponse. There were 18 authorized
states with a zero value. We adjusted for this in our analysis by utilizing the penalty metric only
for the 28 authorized states that had a greater than zero value for penalties.
The second consideration is that the EPA database shows that several states exceed 100 percent.
This is caused by the way EPA's data system computes totals. [The system calculates this
12-P-0113 55
-------
metric by dividing the number of state actions with penalties (numerator) by the number of state
formal actions (denominator). However, the database counts one category of formal actions in
the numerator but not in the denominator ("AO Stipulated Penalty 1"). The result is that the
percent of formal actions with penalties can be greater than 100 percent if a state issued several
"AO stipulated penalty 1" actions.] We adjusted for this in our analysis by capping the metric at
100 percent (so, we assessed the two states reporting > 100% at 100% in our analysis).
The scope of our evaluation did not include penalty appropriateness.
Metrics Selected are Ambiguous Without Additional Information and Context
The three metrics selected all require additional information to interpret.
Average Percentage of Facilities Inspected ('03- '09)
The goals in the CWA, CAA, and RCRA Compliance Monitoring Strategies mostly reflect major
facilities and inspections, while allowing the negotiation of alternative compliance monitoring
plans that include non-major inspections for which there generally are no specific goals. This
makes it difficult to analyze performance against one goal. States who are appropriately utilizing
our policies to focus on the most important sources of pollution and noncompliance will appear
to be performing poorly under the analyses in the draft Report.
• Under the CAA CMS, states can negotiate lower coverage of major sources in
exchange for greater coverage of minor sources that are contributing to priority
environmental problems.
• Under the CWA CMS, we encourage states to shift inspection resources from
major sources to non-majors that pose significant environmental harm or have
known compliance issues.
• Under RCRA, a large percentage of the universe can shift between SQG and LQG
in any given year so it is not possible for EPA to determine the precise universe.
Beyond statutorily defined coverage frequencies for Treatment, Storage and
Disposal facilities, states can negotiate different coverage levels according to
options in the RCRA CMS.
OIG Response 24: The EPA's first two examples, above, underscore the confusion states
expressed to the OIG about EPA benchmarks and goals for states. Because there are not clear,
consistent benchmarks for nationwide environmental protection, EPA holds different states to
different standards. As we stated in the report, this leads to confusion on the part of states, and
impedes the public's ability to discern whether states meet national goals. The OIG concurs that
flexibility is valuable; however, EPA's performance measures should accurately reflect what
EPA expects from all states. This improves transparency for state performance for the American
public, Congress, states, and the regulated community.
In response to EPA's comments, we added an explanation to chapter 1. The metrics allow for
state-to-state performance comparison based on priority issues to EPA. However, without
corroboration they do not offer reliable details about individual state performance.
12-P-0113 56
-------
Percent of Significant Noncompliance Identified per Inspection
By itself, there is no way to know whether a low or high SNC rate is the sign of good or poor
program performance. A low SNC rate can indicate that a program is not identifying SNC that it
should and is, therefore, performing poorly. On the other hand, low SNC numbers can mean a
high rate of facility compliance as a result of a program performing well. In this instance, a state
could have a high field presence and be active in taking enforcement actions that are creating
high deterrence (which may or may not relate to numbers of enforcement actions). To understand
the significance of the SNC rate, you need to understand the quality of the state's targeting,
inspections, and enforcement.
OIG Response 25: Based on EPA's comment, we provided additional explanation in
appendix A.
Percentage of Formal Actions with Penalties
The percentage of penalties per action does not provide a real indicator of program quality.
While the number of actions is important, it is also important that the actions are appropriate,
meaning that they bring a facility back into compliance and provide deterrence. States with a
high number of actions with penalties, but insignificant penalty amounts "score" high under the
OIG analysis, but may not actually result in facility compliance or general or specific deterrence.
OIG Response 26: We agree that the ultimate success of an enforcement action is bringing a
facility back into compliance and providing deterrence to future noncompliance. The metrics we
selected for this evaluation are not comprehensive, but rather serve as important indicators of
state enforcement activity. Assessing the effectiveness of enforcement actions at achieving
sustained compliance was outside the scope of this evaluation.
Roll-up of Data into "Overall Performance Averages" to Characterize State Performance
To fully appreciate the meaning of a percentage, one first needs to understand the universe and
how variations might affect the percentage. In the OIG analysis, there does not appear to be any
consideration of large versus small states. Looking within the universe of regulated facilities of a
state can be informative as to the quality of the job they are doing in terms of coverage or dealing
with violations. The relevant universe for a metric is also important. A state with a small
universe may be capable of dealing with its universe in a more thorough manner than a state with
a very large universe. For example, inspecting 100% of a total universe of eight facilities is much
easier than inspecting 100% of a total universe of 8,000 facilities.
The program performance analysis throughout the Report, and featured in the national map on
page 7, is the result of rolling up the problematic figures in each of the metrics by media program
and then rolling them up again across the programs. This approach obscures some errors or
anomalies and compounds others. The end result provides an inaccurate impression of program
performance for any particular state, which does not comport to EPA's more in depth
understanding of state performance.
The variability produced by this approach also does not help to distinguish performance. This
methodology results in an array of numbers that largely cluster between 20 and 30 per cent with
two obvious outliers at nine and 46 per cent. One of the outliers, Alabama, achieves its high
12-P-0113 57
-------
score based on an incorrect data result: 165% of formal actions with penalties - which is
compounded through the averaging process. It is not clear what these percentages describe,
whether a high or low score is better, or what the distribution of scores means.
OIG Response 27: EPA raised two points in this comment. Its first point is that comparing all
states masks the different circumstances faced by large and small states. However, this criticism
does not account for our choice of metrics. In response to the comment, we provided additional
information in chapter 1 regarding our choice of metrics. We said, "Each of these metrics comes
with several caveats about the underlying data. (Many of these caveats are explained in
appendix A, and were previously reported by OIG.29)" However, we conducted a data reliability
assessment and determined that utilizing the metrics offer a valid method for comparing state
activities that took into consideration several key factors. Among them:
• The metrics factor in the size of a state's regulated universe by using percentage of
facilities inspected.
• Because we drew the metrics and their values from EPA databases, they provide the
same information the public sees when viewing state enforcement performance on EPA
websites.
• The three metrics represent two priority issues EPA identified as national weaknesses in
its initial SRF reviews.
• The metrics allow for state-to-state performance comparison based on priority issues to
EPA. However, without corroboration they do not offer reliable details about individual
state performance.
EPA's second point was that we made an error by comparing states by a grouped measure of the
three statutes. We conducted this comparison to determine which states performed poorly across
the board. The maps contained in appendix C demonstrate the results of this exercise.
Program Complexity Can Not be Captured Using Three Metrics Drawn from Existing
Data
Gaps in our current data make it difficult to develop measures that tell a complete story across
the regulated universes in each program. EPA has developed its measures for state oversight
based on the data that we have, knowing that our current data may not focus on the right things.
While EPA also seeks more straightforward means to portray program performance to the public,
we had to begin to assess programs based on the information that is available.
As described above and stated in the transmittal memo for these comments, the 3 metrics used by
the OIG are overly simplistic, sometimes inaccurate and/or misleading, and are not adequate for
drawing conclusions about program performance. The full suite of metrics used by OECA is a
set of indicators that serve as a place to start to evaluate performance. EPA also looks at state
data, file reviews and interviews with state management and staff to help inform reviewers as to
29 See, for example, EPA Needs to Improve Its Recording and Reporting of Fines and Penalties, Report No. 10-P-
0077; EPA Could Improve RCRAInfo Data Quality and System Development, Report No. 1 l-P-0096; ECHO Data
Quality Audit - Phase I Results: The Integrated Compliance Information System Needs Security Controls to Protect
Significant Non-Compliance Data, Report No. 09-P-0226; and ECHO Data Quality Audit - Phase 2 Results: EPA
Could Achieve Data Quality Rate With Additional Improvements, Report No. 10-P-0230.
12-P-0113 58
-------
whether issues exist that warrant more examination. Each metric is not used in isolation to draw
a conclusion. If a metric warrants cause for concern, then the issue is explored before any
inference and recommendation for improvement is made. The OIG's use of only three metrics
without the more in depth review and context does not lead to accurate conclusions of program
performance.
OIG Response 28: We agree that enforcement programs are complex. We employed these
metrics the same way as EPA does: "as a place to start to evaluate performance." As described
in our scope and methodology descriptions in chapter 1 and appendix A, we also looked at,
"state data, file reviews, and interviews with state management and staff." For more
information, please see OIG Response 1.
EPA Recommends Modifying the State Performance Analysis
As stated in EPA's memorandum, the Agency agrees with the OIG findings that state
performance varies widely across the country and that regional oversight can be strengthened
and improved. This conclusion is supported by previous evaluations, many cited in Appendix B,
and by EPA's own oversight experience. EPA feels that the methodology used by the OIG in this
evaluation takes away from these primary findings and conclusions because of the many issues
surrounding the metrics and the averaging across programs and states. Because EPA's own data
support the conclusions, it is not necessary to create confusion and unintended consequences by
publishing the problematic methodology. EPA requests the OIG modify its analysis and display
of results to address these issues.
Because the findings are not in dispute or lacking in evidence, because of the extensive issues
associated with the data utilized, the limited selection of metrics and the method of analysis, and
because of the incorrect and misleading conclusions drawn from the analysis, EPA requests that
this analysis and references to it be modified before the OIG publishes this Report. This request
applies to:
1. The discussion on page 3 of the Scope and Methodology section;
2. The "State Enforcement Programs are Underperforming" section on pages 6 and 7,
including the map on page 7;
3. The "State Programs Frequently Do Not Meet National Goals" section on pages 7 and 9;
4. The "State Enforcement Is Inconsistent" section on pages 8 and 9, including the graphic
on page 9;
5. The state examples provided on pages 14 and 15;
6. The "Data Analysis and Methodology" section on page 23; and
7. Appendix C
Alternative 1: Utilize Existing State Performance Data and Maps
OECA believes that developing overall state performance metrics is something to strive for, but
recognizes that current metrics are not sufficiently developed. OECA has begun setting the stage
for this in the development of state comparative maps and dashboards, which are now available
to the public for the CWA. OECA has also provided comparative national graphs for any
individual SRF metric within the Enforcement and Compliance History Online (ECHO) database
so the public can look at individual performance indicators and compare across states. During the
early phases of the evaluation, OECA provided the OIG with several draft tables under
12-P-0113 59
-------
development. Any of these sources of information (state dashboards, comparative graphs in
ECHO, or the comparison tables under development) could provide a starting place for the
OIG's analysis of state performance and consistency.
OIG Response 29: Because the data in the SRF system come from the same databases as the
tool used in our analysis, the results of this change would be substantially the same. In addition,
we conducted a reanalysis utilizing the data that OECA provided to us, and the difference in the
results was not statistically significant.
Alternative 2: OIG could direct EPA to develop a data-based scorecard that better reflects the
complexity of state performance
The OIG could use EPA's experience and their own analysis to identify the complexities of
determining and communicating state performance to the public, and direct EPA to develop a
comparison score card that relays the data caveats and context necessary while making this
information transparent.
OIG Response 30: We agree that this would be a preferable exercise to our periodic assessment
of public data. Based on OECA's suggestion, we have added this suggestion as
recommendation 6.
Alternative 3: Potential re-Analysis of Three Metrics
If the OIG determines it must retain the three metrics selected, EPA recommends that the
information for each of the programs analyzed be independently displayed, rather than rolled into
a single average across programs. While this would not address some of the more significant
issues raised above, it would reduce the confusing results created by averaging across programs
and put the focus on individual programs. EPA has found that program quality can vary
significantly within a state and, if performance is poor, EPA generally addresses issues on a
program by program basis rather than a state by state basis.
OIG Response 31: In response to EPA's comments, we presented the data for each program
separately while also including a general assessment of where states performed poorly across
programs.
Page by Page Detailed Comments
Page 1 Background - "EPA is responsible...." We recommend changing the sentence to
read "EPA is responsible for ensuring that federal environmental programs are implemented by
EPA regions and states consistent with the requirements of the CAA, CWA and RCRA. EPA
establishes goals as a means to monitor the implementation of these programs, including
enforcement."
OIG Response 32: We retained the original wording of this sentence.
Page 1 Background - "National Consistency ensures that all Americans..." Throughout
12-P-0113 60
-------
the document the word "consistency" does not acknowledge that flexibility is built into national
policy and guidance to accommodate differences in regions and states. For example, the CAA
allows a state to submit implementation plans so long as EPA agrees that the state's program
meets the minimum requirements of the CAA. States may also have programs that are more
stringent than required by federal rules.
OIG Response 33: We added a statement to explain that national consistency exists in
conjunction with some measure of flexibility.
Page 1 "EPA Can Authorize States...." We recommend changing the sentence to read:
"Most states have acquired this authority for many programs (not all states are authorized or
delegated for all programs)."
OIG Response 34: We retained the original wording of this sentence.
Page 2 please add text"... authority for enforcement that begins with Congress, extends
to the EPA Administrator, then to the Office of Enforcement and Compliance Assurance
Assistant Administrator, EPA Regional Administrators...
OIG Response 35: We changed the text to reflect the joint authority of OECA and EPA
Regional Administrators.
Page 2 "EPA Oversees States " We recommend changing the first sentence to read:
"Although most states have received authorization or delegation to administer most programs..."
| OIG Response 36: As requested, we added "most" to the sentence.
Page 2 The Report does not articulate a full appreciation of the relationship between a
state program and the federal program in the enforcement area. When a state is "delegated"
authority or authorized to run a federal environmental program, it enforces using its own state
authorities, and EPA does not relinquish its enforcement authorities. Most effectively, the two
authorities partner on how to assure compliance and enforce in the regulated community, and
such partnerships tend not to be uniform in all respects.
OIG Response 37: We retained the original wording for this portion.
Page 2 The Report also does not seem to appreciate that program withdrawal is an
appropriate response when there are broad and pervasive problems across a program, generally
extending beyond just the enforcement program. Enforcement concerns can be an important
component of a withdrawal process, but generally, in and of themselves, are not the sole reason
behind a program withdrawal.
12-P-0113 61
-------
OIG Response 38: We understand that program withdrawal should be a last resort for failing
programs. However, it should be available for EPA to use at some point. We do not believe that
is presently the case. We understand enforcement is just one component of a successful
program. However, enforcement is one of the most critical aspects for ensuring program
effectiveness. Regulations clearly state that if states do not take appropriate enforcement action,
EPA retains the authority to withdraw state program authority.
Page 4 EPA Established the State Review Framework - Please modify the language in
the opening sentence: "In 2004, EPA worked closely with states to establish the SRF... Under
this system, EPA regions evaluate states on 12 nationally consistent..." Element 13 is optional
and not nationally consistent. On the footnote to this sentence, please add ".. .the OIG used 3 of
EPA's many SRF metrics as proxies..."
OIG Response 39: Because the evidence indicates that EPA worked predominately with ECOS,
we added, "worked with the Environmental Council of the States (ECOS)" to this sentence. We
revised the SRF sentence to read "12 nationally consistent elements." We revised the footnote to
read, "used three of EPA's SRF metrics." Many of the metrics were not data metrics, but file
review metrics, so we could not include them in our data analysis. For example, one of the
metrics looks at whether or not files were complete with penalty calculations, and that would not
lave been reasonable for us to consider in this analysis. However, we reviewed all of the SRF
reviews and utilized them in developing our findings.
Page 5 EPA Undertook Additional Efforts - please add "In this way, EPA hopes to enlist
the public... stronger accountability from the regulated community and from government."
OIG Response 40: We made this requested change.
Page 6-7 See EPA comments on Methodology above. EPA does not agree with averaging
the data in each metric, averaging data across programs, the rolling up of averages into one
number per state and the use of a color-coded map of averages. EPA does support the use of
maps for numeric metrics, with appropriate caveats and context for the data provided. EPA does
not agree with many of the state-specific conclusions reached by the OIG.
OIG Response 41: We appreciate these comments and made the following changes to the final
report. We removed the average across three statutes, replaced the single map with four maps,
and augmented the discussion of our methodology in chapter 1 and appendix C.
Pages 6-9 As indicated in EPA's comments on Methodology above, OIG's analysis of state
program performance against inspection goals for each of the media misapplies those metrics:
• Data may not reflect full regulated universes or all inspections conducted. Data is focused
on majors in all three programs reviewed, and states are not required to fully report
information on regulated non-major facilities. Non-major information is inconsistently
reported in EPA data systems.
12-P-0113 62
-------
OIG Response 42: In response to EPA's comment, we included additional information in our
scope and methodology to describe what portions of the regulated universe are contained in the
metrics we use.
• Each of the three programs have Compliance Monitoring Strategies (CMSs) that establish
goals (not requirements) to reflect the diverse range of industries, facilities and state
conditions. CMS guidelines are not equally applicable across all circumstances. For
example, in the CAA, some states (and regions) have a greater number of mega sites. The
guidance in the existing CMS establishes goals for state/local agencies (not requirements)
that a Full Compliance Evaluation (FCE) should be conducted once every two federal
fiscal years at all Title V major sources except those classified as mega-sites for which
the minimum evaluation frequency is once every three federal fiscal years. The CMS also
allows for alternative frequencies to be established through CMS alternative plans, with
consideration of the following: facilities on tribal lands, major source universes, major
source universe inaccuracies, classification changes within the CMS cycle, and number
of majors classified as mega-sites.
OIG Response 43: We considered the CMS guidelines for all three programs in its analysis of
enforcement activities, goals, and accomplishments. Though individual state goals may vary,
we believe it is reasonable to compare state accomplishments against national goals.
• For the vast majority of SRF reviews, the regions have concluded that the delegated
agencies are successful regarding evaluation coverage and are meeting their CMS
commitments. OECA is encouraging states to expand coverage to non-majors that cause
environmental harm or have known compliance issues.
• The Report does not address the planning processes in which regions discuss inspection
coverage and commitments with states. EPA can and does step in, where possible, to
improve inspection coverage where states are unable to meet regulatory requirements or
address their most significant sources. The objectives of the planning discussions are to
fully utilize limited resources while tailoring inspection priorities to the range of source
universes and environmental issues unique to individual state circumstances.
OIG Response 44: We acknowledge that enforcement activities are part of a larger program,
which includes an annual planning process. However, the planning process is outside the scope
of this assignment.
• For example, the October 2007 CWA CMS has a goal of 100% coverage every two years
to inspect NPDES major facilities, but allows that percentage to be adjusted based on
state coverage of non-major facilities of interest, such as CAFOs or stormwater. The
analysis used by the OIG measured performance with a rigid 100% minimum standard. A
specific example is the data on the inspection of majors related to Alabama and a number
of other Region 4 states. Since many of the Region 4 states negotiated a NPDES CMS
that was less than the 100% (national goal) inspection of majors (for Alabama it was
50%), it is inappropriate to evaluate them against the national goal of 100%. The
reduction in the inspection of NPDES majors was offset by the increase in inspections of
12-P-0113 63
-------
NPDES minors and this negotiated agreement was part of the 106 Work plan. Alabama
exceeded the negotiated inspection amount by performing inspections at 54% of their
NPDES major universe. The Report states that 13 states inspected fewer than 50% of
major CWA facilities in 2010, but does not explain that states may have conducted
inspections at non-major facilities or other resource trade-offs available to states through
the flexibility provisions of the CWA CMS.
OIG Response 45: EPA's comment demonstrates the type of scenario states describe when
expressing confusion over EPA expectations. In chapter 2 of the report, we describe the
confusion, saying, "The absence of consistent benchmarks creates confusion about what EPA
sees as a good state program and contributes to inconsistent performance. Staff and officials in
EPA regions told us that it would be helpful if OECA clarified its national policy and guidance,
and codified national enforcement standards in regulations."
The statistic about RCRA Subtitle C performance is confusing and not accurate. The
ACS commitment requires inspections at 20% of the large quantity generators in each
state per year (100% every five years). However, this is a combined state and EPA goal.
As stated in the NPM guidance, "at least 20% of the LQG universe should be covered by
combined federal and state inspections unless an alternative plan is approved under the
RCRA CMS." EPA questions whether the 62% cited by OIG includes EPA inspections.
Also, as the sentence quoted above indicates, the CMS allows for deviations from the
20% per year goal to allow states to do more small quantity generator inspections because
of suspected compliance issues within that universe, the 62% cited by OIG includes EPA
inspections. Also, as the sentence quoted above indicates, the CMS allows for deviations
from the 20% per year goal to allow states to do more small quantity generator
inspections because of suspected compliance issues within that universe.
OIG Response 46: To clarify, the CMS limits EPA inspections to fewer than 10 percent of the
required 20 percent, so if a state is required to complete 40 inspections, only four of those could
be EPA inspections. Our assessment looked at state-only inspection rates. The EPA database
reports state inspections for RCRA. We used this metric. We are not arguing for inflexibility,
but rather for ensuring that all states meet a minimum standard. Above the minimum standard,
flexibility in achieving overall goals may be appropriate.
This can be found in the RCRA SRF Plain Language guide for Metric 5c (Five-year
inspection coverage - Large Quantity Generators) which states "...NPM guidance states
100% of LQGs should be inspected every five years. The BR [Biennial Reporting]
universe, while not perfect, in many cases offers the most accurate LQG count. However,
because this is a difficult universe to gauge, the region and state may agree upon and
substitute an alternative universe count. In addition, states with approved plans may
substitute other facility inspections for LQGs per the Guidance for FY08 RCRA Core
LQG Pilot Projects; commitments that vary from the national goal may be reviewed
under Element 4. Further review is needed when states do not meet the goal, although
due to universe changes, coming close to but not reaching the 100% goal is not cause for
concern." Given that the LQG universe changes on a monthly basis and that only 60 to
70% of the LQGs appear in each BR report over a 3 BR reporting cycle, it is unlikely a
12-P-0113 64
-------
state will reach 100% of the universe unless it is a small state with a stable generator
universe. The closer a state is to 100% the less concern, while the further from the 100%
the more the issue should be investigated further.
OIG Response 47: OECA may explore using this detailed information when a state scorecard
(per report recommendation 6) indicates that there are issues with its RCRA program. However,
the detail was outside the scope of this OIG evaluation.
None of the OIG analyses take into account state resource limitations, while regional
resources are considered. This doesn't seem reasonable.
OIG Response 48: We understand that states fund their environmental protection programs to
varying degrees and that in some instances funding may not be adequate to meet agreed-upon
responsibilities. However, when this happens it is EPA's responsibility to step in and ensure
that the state protects its citizens from the resulting pollution by enforcing the law. EPA must be
able to adjust resource allocations to respond to these deficiencies in order to fulfill its statutory
responsibility as the steward for environmental protection.
Page 8 "Based on an overall average of the three enforcement measures... we found that
the state performance varied across the country almost 40 percentage points from lowest to
highest performing state." Per the discussion in EPA's comments on Methodology above, EPA
recommends that this comparison be removed from the Report. However, if not, the results of
this array of averages on page 9 also show that, other than the two regions with the highest and
lowest averages, the remaining eight regions all fall within a narrow range of about 8 percentage
points, which would suggest a high degree of consistency among regions.
OIG Response 49: We modified this section. Specifically, we removed the figure comparing
regional performance. We also modified the report to discuss state performance by statute,
rather than overall. However, our analysis using EPA data and interviews across six EPA
regions indicates significant variation in regional approaches to state oversight.
Page 9 "Because regional approaches to state oversight varied, state performance varied
depending, in part on the region overseeing the state" is not accurate. State performance varies
because of a range of issues including state resources, whether the state agency has independent
administrative authority, influence of regulated community within the state, etc. These factors
have greater influence on state performance than regional oversight. The Report does not provide
an analysis of the relative importance of regional oversight against these other factors. We
recommend changing the sentence to read, "Our evaluation indicated that state performance
varied depending, in part, on the region overseeing the state."
12-P-0113 65
-------
OIG Response 50: Our analysis of EPA data and interview results across six EPA regions and
seven states does not support EPA's comment that other factors have a greater influence than
regional oversight. We found that some states with resource constraints or states lacking
administrative penalty authority performed well, while others did not. The laws clearly state
that EPA has a responsibility to assist states or step in when states are not adequately
performing their enforcement programs, be it from resource constraints, penalty limitations, or
other factors. Therefore, regional oversight is critical. We did not change the text as we believe
the existing sentence in the report is sufficient.
Page 9 EPA does not agree that rolling up state averages is a legitimate way of evaluating
regional oversight performance.
OIG Response 51; We removed this from the final report.
Page 9 "Instead EPA's 10 Regional Administrators have authority over state
enforcement..." is not correct: RAs do not have authority over state enforcement, rather they
have oversight authority.
| OIG Response 52: We made this change in the final report.
Page 10 EPA does not believe that a (one) national baseline or standard is
appropriate for judging state or regional performance. Performance is complex and must be
viewed with appropriate context. National expectations are laid out in the goals that are set
through our national guidance, which allow flexibility to balance national consistency with
regional, state and local conditions.
OIG Response 53: We describe the importance of national benchmarks in chapter 2 of the
report, saying, "State enforcement programs are complex and varied. However, as the national
steward of environmental enforcement, EPA must set national benchmarks that establish the
enforcement expectations that states and EPA agree to meet. EPA has not set consistent
benchmarks for state performance. As a result, EPA cannot hold EPA regions accountable for
ensuring that states meet a national standard, and headquarters does not objectively know which
states require immediate intervention. If state performance exceeds the national benchmarks,
states and regions can then address local priorities."
Page 11 Headquarters Policies and guidance Not Clear — EPA has consolidated all
current guidance that serves as the basis for an SRF review on the SRF OTIS web site. This is
available to both EPA and states. We acknowledge that the public web site is not as clear.
Page 11 "We ... reviewed hundreds of associated EPA guidance and policy
documents. We found that none of these information sources outlined clear and consistent
baselines." It is not clear what standard the OIG is applying for "clear and consistent baselines."
While EPA agrees that guidance is numerous and can be improved, we have substantial
documentation that does establish clear baselines and we continue to make improvements when
12-P-0113 66
-------
issues arise. For example, under the CAA program, the CAA CMS has been updated to: 1)
clarify minimum frequencies based on federal fiscal years; 2) incorporate the Federally
Reportable Violations Clarifications memo; 3) address new guidance focusing on the array of
new air toxics area source rules; 4) update Stack Testing Guidance and Section 112(r) Guidance.
In addition, the HPV Policy and the CAA Penalty Policy are both currently under review to
update them consistent with changes in the CAA program.
OIG Response 54: Our interviews with seven states and six EPA regions indicated that despite
all of these documents, enforcement personnel did not clearly understand how benchmarks and
flexibility worked to ensure that all states met the same basic requirements. The OIG is not
arguing for inflexibility, but rather for ensuring that all states meet a minimum standard. Above
the minimum standard, flexibility in achieving overall goals may be appropriate. The OIG
report emphasizes that these information sources have not explicitly outlined performance
benchmarks.
Page 12 In the first paragraph, the OIG appears to have used an old inspection
frequency benchmark in determining the appropriate percentage for coverage NPDES major
coverage for 2010. EPA recognizes that the plain language guide used a previous version of the
CMS. This was because the new CMS went into effect in October 2008 (FY2009), after the start
of SRF Round 2. Reviews done using FY2008 data would use the old CMS while reviews using
the FY2009 data would use the new CMS. EPA appreciates that this could cause some
confusion, but this practice reflects prior agreement with states that they are held accountable to
guidance and policy in effect during the year of performance. The plain language guide for
Round 3 is being updated to reflect current policy.
OIG Response 55: We described this scenario in a footnote in both the draft and final reports.
We used the goal that EPA presented alongside the data in the SRF data, but included the
footnote because of the discrepancy between the goal presented with the data and the CMS.
This provides another example of how benchmarks are not clear. However, in response to this
comment, we compared state performance to this goal based on 2006 data (prior to the new
CMS), as well as 2010 data. The results are presented in chapter 2.
Page 12 "In another example, a state official pointed out a discrepancy between
EPA's RCRA Enforcement Response Plan and the SRF review process. Specifically, the RCRA
Enforcement Response Plan allows states to miss a timeliness requirement 20% of the time. In
contrast, the SRF review process does not allow this. Therefore, the SRF report could potentially
indicate that the state did not meet program requirements if the state did not meet the 100%
timeliness goal." This is inaccurate. The state official has either misunderstood the requirement
or the SRF review process. The RCRA Enforcement Response Policy flexibility allowing
exceedance of timeliness requirements for 20% of cases per year is for cases involving unique
factors and is not a blanket exemption. Nevertheless, the associated SRF data metric has a goal
of 80% timeliness to most closely reflect the policy. This is noted on the data metrics report and
in the Plain Language Guide for the RCRA SRF reviews.
12-P-0113 67
-------
OIG Response 56: We rephrased the statement to clarify that the state officials expressed
confusion about whether the goals under the SRF and Enforcement Response Policy were
coordinated.
Page 12 Independent Interpretations — States may interpret the regulations
differently for RCRA since they adopt their own regulations (even if they use EPA language)
and as long as they are not less stringent than EPA.
OIG Response 57: This EPA comment supports the OIG argument about ensuring all states
meet national benchmarks, above which there may be flexibility in establishing additional
goals. We are not arguing for inflexibility, but rather for ensuring that all states meet a
minimum standard. Above the minimum standard, flexibility in achieving overall goals may be
appropriate. Our report emphasizes that these information sources have not explicitly outlined
performance benchmarks.
Page 12 The IG suggests that, by codifying policy in regulations, EPA would
operate a national enforcement program that consistently interprets and enforces environmental
laws. While this would support consistency, it may not allow sufficient flexibility to focus on
state and regional priorities for addressing the most serious water quality issues. Also, the
rulemaking process is resource-intensive and time-consuming. By the time a rule would be final,
it is likely that the policy or guidance would need to be modified or changed.
OIG Response 58: The report does not recommend codifying policy in regulations; however,
we heard from state and external sources that codification would make it easier for the states to
implement the enforcement programs.
Page 13 The IG incorrectly uses report length as an indicator of the quantity and
quality of information. Report length varies for a number of simple reasons — such as the number
of attachments, the size and complexity of the state program, the number of issues identified —
and does not correlate with the quality of the report. In addition, the specific examples provided -
Region 7's lengthy reports versus other regional reports - are not comparative as the Region 7
reports are comprehensive media program reviews (including permitting), which incorporate the
SRF elements for reviewing the enforcement components of the overall program.
OIG Response 59: Our description of SRF report inconsistencies does not focus on the length
of the reports. We say, "Regions' SRF reports differed in both quantity and quality of
information about the states." In addition, we point out EPA's own conclusions about
inconsistency, saying, "After receiving the first 25 Round 1 reports, OECA issued a 'Guide to
Writing SRF Reports (Interim Final)' in April 2007. In the accompanying transmittal memo,
OECA said, 'To date, inconsistencies in the information available in the SRF reports inhibit our
ability to determine if the reviews are being carried out in a consistent manner.'"
Page 14 See EPA comments on Methodology above. Because of the
methodological errors, the analyses of state performance in this section are incorrect and
misleading and should be removed from the Report.
12-P-0113 68
-------
OIG Response 60: Our methodology was sound and based on EPA data systems. Please see
OIG Response 1 for additional details.
For example, North Dakota's NPDES and RCRA inspection rates far exceed both the
national average and the national goals. North Dakota ranks second in the nation for
percentage coverage of inspections at facilities classified at majors, and first for
percentage inspection coverage at those facilities with reportable minor permits. North
Dakota has inspected 100% of its TSDs over the last 7 years and 68% of its LQG's, yet
according to the Report, their inspection coverage percentage is 3%.
OIG Response 61: Our analysis included all facilities. This means that it included all of these
inspections. As described in chapter 2, North Dakota's inspection rate does not reflect the
state's enforcement activity when it finds a violation.
For Louisiana: the OIG states "In 2001, citizens filed a petition with EPA urging a
withdrawal of the state's CWA NPDES program authority for many reasons, including
lack of adequate enforcement. The region responded by conducting audits in the state.
Even though the region found multiple parts of Louisiana's NPDES program to be
deficient, it decided not to withdraw the program. [Footnote omitted.] The state's poor
performance has persisted."
• This paragraph implies that the Agency did not take significant action in response to
the petition, and that program withdrawal is the action EPA should have taken; we
disagree with both points.
• However, as a threshold matter, it is important to recognize that at the time of the
OIG inquiry, Region 6 was unable to provide OIG with copies of relevant documents
from the 2001 period, and thus OIG did not have access to the full set of facts. The
Agency regrets and apologizes for that omission. We have now located the relevant
documents (attached), and want to share them so that the IGs recommendations can
be based upon the fullest and most accurate information, and so that the public is well
informed.
• The record shows that although the Agency decided not to withdraw Louisiana's
NPDES program (the "LPDES" program); EPA undertook an extensive process to
evaluate, oversee, and improve the State's program.
• After the citizens' petition was filed in 2001, the Regional Administrator and the
Governor Foster of Louisiana decided to jointly review the administration of the
LPDES program. The Governor convened a special task force, and EPA began an
informal investigation pursuant to 40 CFR §123.64(b). Region 6's informal
investigation included on-site reviews of LPDES files, interviews with LDEQ
management and staff, and an evaluation of information and data concerning program
implementation. The Governor's task force findings were sent to the EPA
administrator, who responded with a letter on February 10, 2003. (See Attachment 2,
Enclosure 1.) Both the special task force and EPA's informal investigation found
12-P-0113 69
-------
deficiencies in the LPDES program. The EPA, through its then Assistant
Administrator for Water, and the Assistant Administrator for Enforcement and
Compliance Assurance, subsequently responded by detailing a list of seven
performance measures that needed to be completed by LDEQ within a specified
timeframe in order to address the program deficiencies. (See Attachment 2, Enclosure
2.) Region 6 also sent a letter commenting on the findings of the Governor's special
task force. (See Attachment 2, Enclosure 3.) In reply, the Governor committed to
complete the seven performance measures, and Louisiana submitted a revision to its
NPDES program that was published in the Federal Register on August 13, 2004. In
May 2004, Region 6 found that Louisiana had successfully completed all seven
performance measures (see Enclosure 4), and in June 2004, the Region performed a
follow-up review of LDEQ's processes and procedures outlined in the revised
LPDES program authorization documents. The Region found that LDEQ was
implementing the changes agreed to, and the Louisiana NPDES program showed
marked improvement. The EPA then approved the revisions to the LPDES program
which was published in the Federal Register on January 5, 2005.
Therefore, contrary to its description in the draft OIG Report as a failure of EPA
oversight, Agency action in this case actually demonstrates the effectiveness of EPA
oversight in resolving state program deficiencies."
OIG Response 62: We appreciate the additional information provided by EPA, since Region 6
was unable to provide the information during the course of our evaluation. We reviewed and
analyzed the additional information provided, and modified the text in chapter 2 related to
Louisiana accordingly. We said, "The region found several deficiencies and required the state to
change some policies and develop new measures. Although the state completed the
recommended actions, the state's poor performance persisted; our analysis found that Louisiana
has the lowest enforcement levels in Region 6 and is ranked in the lower half for CWA and the
lowest quartile for CAA and RCRA."
For Alaska, the OIG uses data from 2003-2009. EPA ran the full NPDES program in the
Alaska until October 2008. In 2009, the State DEC administered the program for only
part of the NPDES universe, so the SRF metrics run for that year are confused by this
shared lead. An SRF review of the State's newly authorized NPDES program has not
been done.
OIG Response 63: In response to this request, we added a sentence clarifying Alaska's 2003-
2007 performance, when EPA was directly implementing the program. The continuously poor
performance record spanning EPA and Alaska's implementation across the 2003-2009 period
underscores the challenges Alaska faces in operating an effective enforcement program.
Page 15 While the Report's methodological errors also apply to the analysis of
Louisiana, Alaska and Illinois, in each of these states EPA has identified issues in its SRF reports
and in other public documents and, most importantly, is taking extensive action to address them.
The Report should acknowledge these actions.
12-P-0113 70
-------
OIG Response 64: We present the actions regions took in the report.
• In the aftermath of Hurricane Katrina, Louisiana worked closely with Region 6 to re-
evaluate environmental priorities and redirect staff to deal with a multitude of unforeseen
environmental problems. The IG appears to have disregarded the workforce and
workload implications of these facts.
OIG Response 65: We agree that many external factors create and increase resource strain, in
particular the disasters affecting Louisiana and Region 6. However, we reiterate that when states
are not adequately operating their enforcement programs, EPA should take corrective actions.
We intended report recommendations 1 and 5 to allow OECA to move resources at a national
level when crises like these occur so that EPA can respond to a compounded enforcement crisis
in a state like Louisiana using the weight of its national workforce.
Region 10 had continuing concerns about Alaska's capacity and performance in
enforcement at the time of authorization in 2008 and shared these concerns with the
State. It was determined that Alaska's program did meet the minimum regulatory and
statutory requirements for program authorization despite those concerns. The phased
approach was itself a way to help the State build capacity in permit and enforcement
work and for EPA to assess progress as the State started a new NPDES permit and
enforcement program from the ground up. As expected, the State has encountered a
number of challenges. Even absent an SRF review, the Region has both formally and
informally informed the State of improvements that need to be made in both their permits
and enforcement work. Region 10 also proposed in the Federal Register, a delay of one
year for the final phase of the NPDES universe.
OIG Response 66: We understand that Region 10 has sought to assist Alaska build capacity.
However, the actions taken by the region have thus far not brought about improved performance
in the state program. We intended report recommendations 1 and 5 to allow OECA to move
resources at a national level when crises like these occur so that EPA can respond to a
compounded enforcement crisis in a state like Alaska using the weight of its national workforce.
Region 5's actions in Illinois (which are publicly available at
http://www.epa.gov/Region5/illinoisworkplan) address both permitting and enforcement
issues and go well beyond the analyses and recommendations under SRF; and these
actions have already yielded significant results and meaningful improvements to lEPA's
program which go beyond the scope of SRF. Specific results to date include:
• Air Enforcement - IEPA has completed the following measures pursuant to its Work
Plan commitments: a new Compliance Monitoring Report has been drafted and is
currently being field tested by IEPA inspectors for completion of inspection reports,
three committees are now consulting with field inspectors prior to and following
inspections planned for the upcoming quarter, IEPA is issuing Violation Notices
containing a recommended technical remedy to resolve violations, and an EPA
review of HPV cases was conducted with IEPA on March 11, 2011. Additionally,
IEPA has referred sixteen cases to the Illinois Attorney General this year. This
compares with fourteen for FY 2010, and appears to be on track to meeting the
12-P-0113 71
-------
required twenty-five percent increase. Currently, IEPA is meeting or is on track to
meet all of its enforcement related Work Plan requirements.
• Title V Permit Issuance - To date, IEPA has met every Clean Air Act permitting-
related commitment in the Work plan, including efforts to issue timely permits, hire
additional staff to work on Title V, and the evaluation of its application completeness
process. On June 30, 2011, IEPA submitted its strategy for reducing the Federally
Enforceable State Operating Permits backlog for sources that would otherwise be
subject to Title V. IEPA has met both November 30, 2011 commitments to issue six
final permits and public notice eleven permits in an effort to reduce the Title V
backlog. IEPA has hired nine additional permit analysts to work on Title V permitting
in last six months.
• Water Program: CAFO Inspections - IEPA is on course to meet the enforcement and
inspection commitments specified in the Concentrated Animal Feeding Operation
(CAFO) work plan. IEPA conducted 38 CAFO inspections, surpassing its
commitment to complete 25. IEPA met the hiring commitment by hiring two
additional inspectors. EPA and IEPA collaborated to provide inspection training for
IEPA field inspectors and enforcement staff. IEPA drafted and submitted a long-term
CAFO National Pollutant Discharge Elimination System (NPDES) inspector training
curriculum. IEPA also developed and submitted a plan to create and maintain a
comprehensive inventory of large CAFOs and is on schedule to submit an inventory
of large CAFOs by the end of calendar year 2011. IEPA developed a citizen
complaint standard operating procedure (SOP) and a database for animal feeding
operations and CAFOs. IEPA prepared and submitted a CAFO NPDES inspection
SOP and a revised Enforcement Response Guide. Since February 2011, IEPA issued
approximately 10 violation notices, received and investigated 4 complaints of spills
and releases, and sent 3 referrals to its Attorney General's Office (IL AGO). IEPA,
the IL AGO, and EPA participate in quarterly docket review conference calls to help
ensure all referred CAFO matters are moving forward.
Water Program: Priority Permits Issuance - IEPA has done a good job on priority
permits issuance. Performance is 114% of the goal. The State has exceeded its
commitment of 42 issuances for FY 2011.
OIG Response 67: We drew our analysis from available EPA data. We applaud Region 5 and
Illinois for increasing their enforcement oversight efforts in Illinois. At some future point, we
may revisit the progress made in Illinois and evaluate how well the Region 5 intervention
program worked.
Page 15 The OIG Report concludes that "regardless of the cause," when a state
cannot fully operate its program, EPA should step in and fill the gap. It is important to
understand that when a state is overwhelmed by the need to address a national disaster — as
Louisiana was during Hurricane Katrina — the Regional Office was similarly required to devote
significant resources to the response activities (with the help of all of the other Regions and HQ).
It should also be noted that in the wake of a natural disaster like Hurricane Katrina, many
12-P-0113 72
-------
industrial operations were curtailed for months, during which time inspections and enforcement
actions would logically decrease.
| OIG Response 68: Please see OIG Responses 65 and 66.
Page 16 "OECA lacks significant control over EPA enforcement resources
nationwide." OECA centrally manages the national enforcement program through the Strategic
Plan, GPRA national program measures, EPA's Enforcement Goals, the NPM Guidance and
ACS Commitment system, semi-annual regional progress and planning meetings during which
state oversight is a major topic, and regular meetings, conference calls and video conferences.
OECA generally directs enforcement resources to the regions for broad purposes, with regions
having some discretion as to how those resources are deployed. Additional transparency on how
regions utilize enforcement resources would be helpful.
OIG Response 69: We do not dispute the many actions that OECA takes within the constraints
of the present organizational structure. In fact, we point many of them out in the report. Our
general finding was that OECA was taking actions within the current structure, and only
through a structural change along the lines of our recommendations may EPA improve its
oversight of state enforcement.
Page 17 The workload ratio of permits issued to enforcement employees is not a
valid measure of enforcement workload. EPA would need to know which permits were included
in the analysis to better understand the OIG's approach. Does it include both state and federally-
issued permits? Permits are issued by media program staff in both the regions and in states.
Permits vary in complexity, and not all permits are required to be reviewed or lend themselves to
federal enforcement. Regions generally take actions based on national enforcement initiatives or
regional priorities.
OIG Response 70: EPA's data dictionary indicates that the count received from the database
includes all permits. We agree that permits vary in complexity and the scope of the report does
not include detailed analysis of permits. Creating a ratio of all permits to FTEs brings awareness
of general workload levels across regions where there were gaps in that knowledge previously.
As an aside, OECA suggested this metric. We applied it across all states, so the permit count
includes the same information nationwide.
Attachments
12-P-0113 73
-------
Appendix F
Distribution
Office of the Administrator
Deputy Administrator
Assistant Administrator for Enforcement and Compliance Assurance
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Associate Administrator for Policy
Audit Follow-Up Coordinator, Office of Enforcement and Compliance Assurance
12-P-0113 74
------- |