$
<
3)
O
%
^ PRO^
CD
T
s*
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
The EPA Should Assess the
Utility of the Watch List as a
Management Tool
Report No. 13-P-0435
September 30, 2013
Scan this mobile
code to learn more
about the EPA OIG.

-------
Report Contributors:	Dan Engelberg
Khadija Walker
Tiffine Johnson-Davis
Genevieve Soule
Andre von Hoyer
Abbreviations
ECHO	Enforcement and Compliance History Online
EPA	U.S. Environmental Protection Agency
OECA	Office of Enforcement and Compliance Assurance
OIG	Office of the Inspector General
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
email:	OIG Hotline@epa.gov	write: EPA Inspector General Hotline
phone:	1-888-546-8740	1200 Pennsylvania Avenue, NW
fax:	202-566-2599	Mailcode 2431T
online:	http://www.epa.gov/oiq/hotline.htm	Washington, DC 20460

-------
*	' U.S. Environmental Protection Agency	13-P-0435
§	\ Office of Inspector General	September 30, 2013
s
—'—'J"
\ vjv "
At a Glance
Why We Did This Review
We sought to determine
how the U.S. Environmental
Protection Agency (EPA) uses
the Watch List as a tool to
target enforcement actions and
whether there are opportunities
for improvement.
Used by the EPA since 2004,
the Watch List is designed to
provide EPA regions and states
with a list of facilities that are in
significant violation of
environmental laws and that
appear not to have been
addressed by timely and
appropriate enforcement.
The Watch List initially was
just used as an in-house
management tool, but in 2011
the EPA made its data
available to the public.
This report addresses
the following EPA theme:
• Embracing EPA as a high
performing organization.
The EPA Should Assess the Utility of the
Watch List as a Management Tool
For further information,
contact our public affairs office
at (202) 566-2391.
The full report is at:
www.epa.aov/oia/reports/2013/
20130930-13-P-0435.pdf
What We Found
According to the EPA, the Watch List is a management tool used to identify
long-standing significant violations. However, through interviews, regional
questionnaires, and data analysis, we found that:
•	Use of the Watch List differs among headquarters and regional enforcement
programs.
•	Trends in the number of facilities on the Watch List differ among
enforcement programs.
•	EPA staff suggest multiple opportunities for improvement in the Watch List.
•	The public version of the Watch List has limited search capabilities and
information.
Wthout a proper assessment of the Watch List, the agency runs the risk of
maintaining a management tool that does not assist in tracking facilities with long-
standing significant violations, has limited transparency and utility to the public,
and does not meet the needs of EPA users.
Recommendations and Planned Agency Corrective Actions
We recommend that the EPA Office of Enforcement and Compliance Assurance
assess the Watch List's utility as a management tool. If the agency determines
that the tool is useful, it should:
•	Ensure that Watch List criteria are consistent with relevant enforcement
response policies, and reassess criteria to determine relevance.
•	Develop an approach for identifying and/or removing facilities on the list that
have been referred elsewhere or are under a consent decree.
•	Improve transparency of the publicly available Watch List to allow users to
search and receive information similar to what is available internally.
•	Identify and implement other improvements to the Watch List identified in
the EPA's assessment.
The agency agreed with the recommendation to assess the Watch List by
September 30, 2014. If the agency decides to retain the list, it will act on the
remainder of the recommendations by December 30, 2014. The
recommendations are resolved with corrective actions pending.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 30, 2013
MEMORANDUM
SUBJECT: The EPA Should Assess the Utility of the Watch List as a Management Tool
Report No. 13-P-0435
FROM: Arthur A. Elkins Jr.
TO:
Cynthia Giles, Assistant Administrator
Office of Enforcement and Compliance Assurance
This is a report on the subject evaluation conducted by the Office of Inspector General (OIG) of the
U.S. Environmental Protection Agency (EPA). This report describes issues the OIG identified and
makes recommendations to address those issues. This report represents the opinion of the OIG and does
not necessarily represent the final EPA position. Final determinations on matters in this report will be
made by EPA managers in accordance with established audit resolution procedures.
Action Required
The report recommendations are resolved with corrective actions pending. Therefore, the agency is not
required to provide a final response to this report. However, if you choose to provide a final response,
we will post your response on the OIG's public website, along with our memorandum commenting on
your response. You should provide your response as an Adobe PDF file that complies with the
accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended.
We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Assistant Inspector General
for Program Evaluation Carolyn Copper at (202) 566-0829 or copper.carolyn@epa.gov. or Director for
Water Issues Dan Engelberg at (202) 566-0830 or engelberg.dan@epa.gov.

-------
Purpose
The purpose of this review was to determine how the U.S. Environmental
Protection Agency (EPA) uses the Watch List as a tool to target enforcement
actions and whether there are opportunities for improvement.
Background
In 2003, the assistant administrator for the Office of Enforcement and Compliance
Assurance (OECA) requested that a tool be developed that would allow senior
enforcement officials to track facilities that were in noncompliance with regulations
(specifically, the Clean Air Act, Clean Water Act, and Resource Conservation and
Recovery Act) without timely and appropriate enforcement actions. This request
resulted in the implementation of the Watch List in 2004. The Watch List, one of
several management tools used by the EPA enforcement programs, was designed to
provide OECA and the EPA's regions, as well as states, with a list of facilities that
data indicate are in significant violation of these environmental laws and have not
been addressed by timely and appropriate enforcement. The system identifies
facilities as high-priority violators under the Clean Air Act or significant
noncompliers under the Clean Water Act's National Pollutant Discharge
Elimination System and the Resource Conservation and Recovery Act.
The Watch List contains a subset of violations identified under EPA enforcement
response policies. The policies define when a violation of environmental laws and
regulations becomes significant. This automated management tool is generated
using compliance information from the EPA's data systems, including the Air
Facility System, the Integrated Compliance Information System-National
Pollutant Discharge Elimination System (previously the Permit Compliance
System), and the Resource Conservation and Recovery Act Information System
(RCRAInfo). In 2011, in response to a Freedom of Information Act request, the
EPA made a modified version of the Watch List available to the public on its
Enforcement and Compliance History Online (ECHO) website; the Watch List
had previously only been used as an in-house management tool.
Scope and Methodology
We conducted this evaluation in accordance with generally accepted government
auditing standards. Those standards require that we plan and perform our work to
obtain sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that the
evidence obtained provides a reasonable basis for our findings and conclusions
based on our evaluation objectives. We conducted our evaluation from February
through August 2013.
To answer our objective, we identified the laws, regulations, policies, procedures
and guidance used by the EPA for enforcement. We also reviewed documents
13-P-0435
1

-------
related to the EPA's roles and responsibilities, oversight/management
requirements, performance tracking and goals for the Watch List. We interviewed
the EPA program and enforcement staff about data reliability. We interviewed
agency officials and sent a questionnaire to staff in all 10 EPA regions to
determine their use of the Watch List and the areas for which it could be
improved. We also conducted an analysis of Watch List data to identify potential
trends and anomalies for further discussion with OECA and the regions and
comparison with agency information.
Results of Review
According to OECA, the Watch List, which the agency has used for almost
10 years, is a management tool to identify long-standing significant
noncompliance. We found the use of the Watch List differs among headquarters
and regional enforcement programs. Further, we observed that trends in the number
of facilities on the Watch List differ among enforcement programs. Additionally,
staff suggested multiple opportunities for improvement. We also found that the
public version of the Watch List has restricted capabilities. Without a proper
assessment of the Watch List, the agency runs the risk of maintaining a
management tool that does not assist in tracking facilities without timely and
appropriate enforcement actions. In its current state, the list also has limited
transparency to the public and may not meet the needs of its users, potentially
limiting its utility for public users.
Enforcement Programs Utilize the Watch List Differently
Information we gathered from interviews with headquarters and questionnaires
sent to all 10 regions showed that some enforcement staff in regional program
offices use the Watch List to assist in their daily work (e.g., tracking facilities to
determine whether they have returned to compliance and/or proceeded to
follow-up with the state), while others only use it as a part of quarterly meetings
with states and/or EPA headquarters to update the data on the list. Interviews and
questionnaires indicated that the following factors might have affected the level of
utilization of the Watch List:
•	Seven of 10 regions reported the availability of duplicative/alternate data
sources.
•	Four of 10 regions and two headquarters program offices reported
facilities remaining on the Watch List that have been referred to other
offices/programs (such as the Department of Justice or the Superfund
program) or are under a consent decree.
•	Three of 10 regions and one headquarters program office reported
inaccurate reporting and timeliness problems with the data.
•	OECA headquarters and one region reported the Watch List not keeping
up with technological advancements or changes in criteria definitions
since it was created nearly a decade ago.
13-P-0435
2

-------
Enforcement Programs Have Differing Trends in Watch List Data
If the Watch List was an effective and key management tool for addressing
facilities in longstanding noncompliance, we would expect to see some decreasing
trends in the number of facilities on the list. However, our review of national
Watch List data, pertaining to three acts (the Clean Air Act, Clean Water Act, and
Resource Conservation and Recovery Act), from headquarters programs and all
10 regions, showed different trends in the number of facilities on the Watch List
from 2004 to 2013. Some regions had decreasing numbers of facilities on the
Watch List in certain programs, others had increasing numbers, and others
showed no obvious pattern of increase or decrease. The information gathered
through interviews with OECA and questionnaires sent to the regions cited
several reasons for the differences. They included states not reporting in a timely
manner or having data entry issues, interface issues resulting from the switch
from the Permit Compliance System to the Integrated Compliance Information
System-National Pollutant Discharge Elimination System, and states or the
agency not updating status codes timely and appropriately. Since most of the
reasons cited were data quality issues, it is possible that, absent these issues, the
trends would look different. Therefore, we are not able to draw a conclusion from
the data about whether the Watch List has assisted EPA in addressing long-term
noncompliance.
Headquarters and Region Staff Identified Opportunities for
Improvement
Through interviews and regional questionnaires, Watch List users identified
opportunities for improvement, including the following:
•	Incorporate a Watch List status in the ECHO facility search outputs
(e.g., a "Watch List" column).
•	Create an efficient process to identify and/or remove facilities that are on
the list in error or that the agency has referred to other programs or have
been placed under a consent decree.
•	Ensure that criteria on the Watch List match the criteria in the relevant
enforcement response policy.
•	Finalize Clean Water Act criteria.
•	Add the Safe Drinking Water Act to the Watch List
•	Allow OECA and regions to create region-specific "Watch Lists" based on
mutually determined criteria while continuing to identify violations that
meet overall Watch List criteria.
•	Initiate Web-based quarterly reporting to accept status codes and
explanations from the regions.
•	Make the data on Watch List real time.
•	Enhance the Watch List standard operating procedures.
•	Add additional status codes that better describe the status of the current
situation of a facility (make codes more specific).
13-P-0435
3

-------
Publicly Available Watch List Lacks Many of the Features and
Information Available Internally
The Watch List capabilities on the public website (ECHO) are very different from
those of the internal Watch List. The public site does not provide users with the
ability to search the Watch List to return only information that they are interested
in. In addition, the information provided is in a "PDF" format and therefore
cannot be easily sorted or analyzed by users, decreasing the utility of the list.
The public Watch List provides limited information - only facility name, ID and
location. The internal Watch List tool (available through the Online Tracking
Information System) provides a substantial amount of additional enforcement
information that allows the user to put the Watch List designation into context
with the facility's enforcement history. Given the agency's long-stated goal of
transparency, the information available to the public should contain all
information that is not deemed enforcement sensitive.
Conclusion
Given the initial reasons for the Watch List implementation, its current differing
use among regions, and differing data trends, the EPA should determine whether
the Watch List is still a useful management tool. Our review identified
opportunities for improvement, including updating and reviewing its approach for
listing facilities to ensure transparency and accuracy of information as well as the
public utility of the information. OECA is currently modernizing its enforcement
data system, which is scheduled to be completed by the end of fiscal year 2013.
We believe this is an opportune time for OECA to make changes to the Watch
List should that be its decision.
Recommendation
We recommend that the assistant administrator, Office of Enforcement and
Compliance Assurance:
1. Assess the Watch List's utility as a management tool in assisting the
agency in monitoring long-standing alleged significant violators. If the
agency determines that the tool is useful, it should:
a.	Ensure that Watch List criteria are consistent with relevant
enforcement response policies and reassess the criteria to
determine relevance.
b.	Develop an approach for identifying and/or removing facilities on
the list that have been referred to other offices/programs or are
under a consent decree.
13-P-0435
4

-------
c.	Improve transparency of the publicly available Watch List to allow
users to query and receive information similar to what is available
through internal data systems.
d.	Identify and implement other improvements to the Watch List
identified in the EPA's assessment.
Agency Comments and OIG Evaluation
The agency agreed with the recommendation to assess the Watch List by
September 30, 2014. OECA intends to assess the overall appropriateness and
utility of the Watch List as a multi-media tool as it revises some of the media-
specific compliance and enforcement policies. In follow-up communications with
OECA staff, we were told that if OECA decides to retain the Watch List, it will
implement the remainder of the recommendations by December 30, 2014. The
recommendations are resolved with corrective actions pending.
13-P-0435
5

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Planned
Completion
Date
Claimed
Amount
Ag reed-To
Amount
Assess the Watch List's utility as a management
tool in assisting the agency in monitoring long-
standing alleged significant violators. If the agency
determines that the tool is useful, it should:
a.	Ensure that Watch List criteria are consistent
with relevant enforcement response policies
and reassess the criteria to determine
relevance.
b.	Develop an approach for identifying and/or
removing facilities on the list that have been
referred to other offices/programs or are
under a consent decree.
c.	Improve transparency of the publicly
available Watch List to allow users to query
and receive information similar to what is
available through internal data systems.
d.	Identify and implement other improvements
to the Watch List identified in the EPA's
assessment.
Assistant Administrator, 9/30/14
Office of Enforcement and
Compliance Assurance
12/30/14
12/30/14
12/30/14
12/30/14
1 O = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
13-P-0435
6

-------
Appendix A
Agency Response
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
ENFORCEMENT AND
COMPLIANCE ASSURANCE
July 25,2013
MEMORANDUM
SUBJECT: Comments on "Quick Reaction Report: The EPA Should Assess the Utility of the
Watch List as a Management Tool"
Thank you for the opportunity to review the Office of Inspector General (OIG) "Quick Reaction
Report: The EPA Should Assess the Utility of the Watch List as a Management Tool". We agree
that it is an opportune time to discuss the utility of the Watch List as a management tool in
assisting the Agency in monitoring long-term alleged significant violators.
We are currently in the process of revising some of the media-specific compliance and enforcement
policies. As part of this process, we can better assess the overall appropriateness and utility of the
Watch List as a multi-media management tool, As we are conducting this assessment, we will keep
in mind the findings and initial recommendations of the OIG and commit to the following corrective
action:
Recommendation	Lead Office CA Target Date Corrective Action
If you have any questions regarding this response, please contact Gwendolyn Spriggs, the OECA
Audit Liaison, at 202-564-2439.
FROM: Cynthia Giles, Assistant Administrator
TO:
Carolyn Copper, Assistant Inspector General
Office of Program Evaluation
Assess the Watch List's OC/OCE
1 9/30/2014	Evaluate the utility of the
Utility as a management
tool.
the Watch List as a
Management tool, and
identify appropriate next
steps.
13-P-0435
7

-------
cc: Lisa Lund
Susan Shinkman
Mamie Miller
Lauren Kabler
Gwendolyn Spriggs
13-P-0435
8

-------
Appendix B
Distribution
Office of the Administrator
Assistant Administrator for Enforcement and Compliance Assurance
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Principal Deputy Assistant Administrator for Enforcement and Compliance Assurance
Audit Follow-Up Coordinator, Office of Enforcement and Compliance Assurance
Audit Follow-Up Coordinator, Office of Solid Waste and Emergency Response
Audit Follow-Up Coordinator, Office of Water
Audit Follow-Up Coordinator, Office of Air and Radiation
13-P-0435
9

-------