^ T-
s
3
%
s Office of Inspector General
% PR0 ^ Audit Report
INFORMATION TECHNOLOGY
Unreliable Data Affects Usability
of DOCKET Information
Report Number 2002-P-00004
January 18, 2002
-------
Inspector General Division
Conducting the Audit
Information Technology Audits Staff,
Washington, DC
Regions Covered
Program Office Involved
Audit Team Members
All
Office of Enforcement and Compliance Assurance
Ed Shields
Chuck Dade
John Ewan
Rudolph Brevard
Teresa Richardson
Judy Tsai
David Cofer
Abbreviations
AO
Administrative Order
CCDS
Case Conclusion Data Sheet
CDETS
Consent Decree Tracking System
DOCKET
EPA's National Enforcement Docket System
EPA
United States Environmental Protection Agency
GAO
United States General Accounting Office
GPRA
Government Performance and Results Act
ICIS
Integrated Compliance Information System
IDEA
Interactive Data Extraction and Analysis Software
IG
Inspector General
OECA
Office of Enforcement and Compliance Assurance
OIG
Office of Inspector General
Regions
Ten Regional Offices, Headquarters' Mobile Source Enforcement Branch, and
Headquarters' Toxics and Pesticides Enforcement Division
SEP
Supplemental Environmental Project
-------
*/""¦ - * >
- #% ,
' 4
*
UN!TED STATES ENVIRONMENTAL PROTECTION AGENCV
«AaH!NGTOtH O.K ;;?460
JAM I 8 2002
SUBJECT:: Final Report: UerelkN* After® Usability of DOCKET
Information
Audit No :000-tH)00813
Report Kb. 2'.)02-P-(WM>4
¦'/
v6*l t* <*£"_ '<*!
FROM; Prawn* H. Hill. Director
Information Technology Audits Staff (242!)
TO; Sylvia k ' ov»t,»nce A>v^tam Wmmistrafeir
OftVc M* i.ts*Qr<.e>neiU .uirf Compliant \jiur.uKc ;2201 A)
Attached i? iJiii final report tti'itfetl '*L'nre*iabte D«t!a Affects U Mobility of (XX KF. i
Iufcnaatkm." Xht objective of this aasitt was to mm whether DOCKET cQOtattned accurate
asd rfi'.ible Jhu t^f jtwusj-ng ent»;«Bfa u^ans, jw vvr'l a% reporting on ctivtronmudu :r> n uifliai timtof? Oat dewnhc problems the Office o» Inspector OwkhU
iOIG) has identified and cwwciiw actions the OlCf recommend*. Phis audit resort represents
ifee opinion of the OKI, mi tt» finttogt wmaml w »to {«pott not weemriiy
represent if* final EPA position. Ftral determinations '-He mitten \n the Ju.ii: tepoft mil tie
dUKk b$ EP A man **«s »ft 4cewd#r«e wtd» established EP \ audit resolute procedufW-
Mfkals*^
In jjMsRfK* With EPA Order 27W. ym, as as ittitm official, are inquired to jjtovWte a*
v,.tb k\ written resporwe w the jUhJ reivrt within 90 day* ot Use final wpon %hid» wit assist us to deciding whether to aase
«jis report- in additWBv piease track M »ttda ?4b» «hi m.leswoe sJa£» » EFA s M4R*•«««
Audit Tracking System,
* , .' ~-ijLrf* ».'5pM.k:
-------
We appreciate the cooperation afforded us during the course of this audit by the Regions
and OECA. We have no objections to the further release of this report to the public. Should you
or your staff have any questions regarding this report, please contact Ed Shields, Audit Manager,
Information Technology Audits Staff, at (202) 260-3656.
Attachment
-------
EXECUTIVE SUMMARY
INTRODUCTION The National Enforcement Docket System (DOCKET) is
the official United States Environmental Protection Agency
(EPA) database for tracking and reporting on civil judicial
and administrative enforcement actions. EPA developed
DOCKET to track the life-cycle of an enforcement action
from its initiation to its conclusion. The Office of
Enforcement and Compliance Assurance (OECA) provides
policies and procedures for the DOCKET system, while
EPA regions and Headquarters' offices are responsible for
entering and maintaining information within the system.
OBJECTIVE The audit objective was to assess whether DOCKET
contained accurate and reliable data for managing
enforcement actions, as well as reporting on environmental
progress.
RESULTS IN BRIEF EPA under-reported the environmental impact of
enforcement actions under the Government Performance
and Results Act (GPRA). Our audit disclosed that 69
percent of the applicable fiscal 2000 pollutant records
contained blank pollutant amount fields. The pollutant
amount field should have contained amounts used to
compute and report pollutant reductions under GPRA.
Additionally, our audit determined the DOCKET system, as
a whole, contained significant instances of inaccurate and
incomplete data. Specifically, 94 percent of the cases
reviewed within DOCKET contained at least one error in a
key field. Until management takes decisive steps to address
particular data weaknesses, neither Congress, the public,
nor EPA management should depend on the DOCKET
system as a reliable source of environmental data.
OECA is implementing a replacement system for DOCKET,
the Integrated Compliance Information System (ICIS), for
the express purpose of tracking and managing enforcement
compliance cases. However, if improvements are not made
Report No. 2002-P-00004
-------
so that ICIS can avoid the problems encountered with
DOCKET, enforcement data quality will continue to suffer.
In order for ICIS to adequately meet user needs and be a
meaningful source of information, in accordance with
GPRA, the multiple causes that contributed to incomplete
and inaccurate data in DOCKET need to be addressed.
Factors contributing to the poor data quality included a
cumbersome system that lacks the functionality needed to
manage cases; inadequate and outdated policies,
procedures, and guidance; and the absence of
comprehensive reviews.
RECOMMENDATIONS Due to the nature of the audit findings, OECA, as well as
the 10 regions, Headquarters' Mobile Source Enforcement
Branch, and Headquarters' Toxics and Pesticides
Enforcement Division, will need to implement corrective
actions to effectively address these weaknesses. In
particular, we recommend that the Assistant Administrator
for Enforcement and Compliance Assurance improve overall
polices and procedures for: computing pollutant related
information, performing quality assurance, disseminating
training, and monitoring the implementation of the policies.
The Assistant Administrator also needs to (1) develop and
implement a process to ensure system requirements
identified by the users are included in the design and
implementation of the system being developed to replace
DOCKET, and (2) develop and apply a scrubbing process
to the DOCKET data used to populate the new system to
ensure it is accurate and complete.
We also made several recommendations to the Regional
Administrators and the Director of Toxics and Pesticides
Enforcement Division and the Chief of the Mobile Source
Enforcement Branch. These recommendations entail
establishing regional policies and procedures that (1) ensure
compliance with Headquarters' policies and procedures;
(2) outline required cross-organizational lines of authority
and communication; and (3) implement a monitoring
process to ensure regional policies and procedures are
followed, continuously reevaluated, and improved.
li
Report No. 2002-P-00004
-------
AGENCY COMMENTS In general, the Agency officials agreed with the conditions
AND OIG EVALUATION identified in the report. However, the Agency believes the
report overstates the unreliability of the data. In a
memorandum dated December 19, 2001, the Agency
proposed separating Judicial (Civil) cases from
Administrative cases. The Agency believed this would
provide a better representation of DOCKET data reliability
because certain fields only apply to Judicial cases.
We employed a methodology that conservatively assessed
the overall reliability of the data within DOCKET. We
evaluated each field based on whether it was applicable to
the particular case. For example, if a field was not
applicable and was left blank, we accepted it as correct.
This methodology provided a very conservative error rate in
comparison to separately evaluating cases applicable to a
specific situation. We obtained concurrence on the
methodology from Agency officials and the General
Accounting Office (GAO) prior to beginning the review.
We believe that data unreliability is not overstated, based on
the significant error rates uncovered for most of the key
fields reviewed.
Agency officials stated they would respond to the specific
recommendations upon receipt of the final report.
111
Report No. 2002-P-00004
-------
Report No. 2002-P-00004
-------
TABLE OF CONTENTS
Page
EXECUTIVE SUMMARY i
CHAPTERS
1- INTRODUCTION 1
Purpose 1
Background 1
Scope and Methodology 3
Prior Audit Coverage 3
Criteria 4
2- GPRA ENVIRONMENTAL IMPACT UNDER-REPORTED 5
EPA Under-Reported Environmental Impact 5
GPRA-Related Fields Inaccurate, Incomplete, or Unsupported 6
Existing Problems Affect Future Reporting 7
Agency Comments and OIG Evaluation 7
3- UNRELIABLE DATA IMPAIRS USABILITY OF SYSTEM INFORMATION 11
DOCKET Data Inaccurate, Incomplete, or Unsupported 11
Deletion of Cases Not Properly Controlled 14
Input Errors Do Not Play Significant Role 14
Agency Comments and OIG Evaluation 15
Report No. 2002-P-00004
-------
4- CORRECTIVE ACTIONS NEEDED PRIOR TO
IMPLEMENTING REPLACEMENT SYSTEM 19
Poor Data Quality May Continue In ICIS 19
Causes for Unreliable DOCKET Data Need to be Addressed 20
System Not Used To Manage 21
DOCKET Policies, Procedures, and Guidance Were Outdated
and Inconsistent 21
Guidance Not Provided for Addressing Pollutant Information 22
Quality Assurance Reviews Not Performed 23
Policies and Procedures Need Improvement 23
Recommendations 24
Agency Comments and OIG Evaluation 27
EXHIBIT
DETAILS ON SCOPE, METHODOLOGY, AND CRITERIA 29
APPENDICES
1- OECA COMMENTS TO DRAFT REPORT 33
2- REGIONAL INPUT FOR SYSTEM ENHANCEMENTS FOR ICIS 39
3- REPORT DISTRIBUTION 41
Report No. 2002-P-00004
vi
-------
CHAPTER 1
INTRODUCTION
Purpose The audit objective was to assess whether the National
Enforcement Docket System (DOCKET) contained
accurate and reliable data for managing enforcement
actions, as well as reporting on environmental progress.
Background In October 1998, several Committee Chairmen in the House
of Representatives tasked the Inspector General (IG)
community with verifying and validating selected data
sources and information systems used to report on the
Agency's Government Performance and Results Act
(GPRA) annual performance goals. In December 1998, the
Senate also suggested the IG community verify and validate
the accuracy and reliability of the performance data used to
report on the Agency's GPRA annual performance goals.
DOCKET is EPA's official database for tracking and
reporting on civil judicial and administrative enforcement
actions. EPA developed this national, automated
information system to track civil litigation and
administrative actions from case development to closure
under all environmental statutes. DOCKET includes facility
and defendant information, and can be used to track all
major milestone events of a case. For example, DOCKET
can track complaints, settlements, and Supplemental
Environmental Projects (SEP). The Office of Enforcement
and Compliance Assurance (OECA) has responsibility for
providing policies and procedures for the DOCKET system,
while both EPA Headquarters and regional offices are
responsible for entering and maintaining information within
the system.
The Consent Decree Tracking System (CDETS) is a
national database closely related to the DOCKET system.
CDETS tracks facility and compliance schedule information
of consent decrees. The system assists the tracking,
1
Report No. 2002-P-00004
-------
managing, and reporting of compliance status information
from the point of initiation until facilities are in compliance
and the decree is terminated. Because of this close
relationship, CDETS was included in the data accuracy and
reliability work of this audit.
In 1999, Congress asked the National Academy of Public
Administration to conduct an independent evaluation of
state and federal enforcement data and to make
recommendations to Congress, EPA, and the states
regarding actions necessary for ensuring public access to
accurate, credible, and consistent enforcement data. The
Academy's June 2001 report, "Evaluating Environmental
Progress: How EPA and the States Can Improve the
Quality of Enforcement and Compliance Information,"
pointed out the increasing importance of reliable
enforcement and compliance data. The report indicated
that good information was rapidly becoming an essential
element for environmental protection, and that reliable
enforcement and compliance data should be considered a
prerequisite for all environmental programs. The
Academy's report emphasized that reliable data was
essential for EPA and the state environmental agencies to
identify environmental problems, set goals, select tools to
remedy problems, measure their progress, analyze the
effectiveness of their programs, and adjust their strategies
accordingly.
Furthermore, the report stated:
"Good information is fundamental to effective
management and public confidence in government
agencies. For the last three decades, however, EPA
and most state environmental agencies have relied
on data about enforcement activities that do not
actually show how well the environment is doing, or
how well the regulated community is obeying
environmental laws. To the extent that these data
measure enforcement or other governmental
performance, they are much more likely to be
misleading than useful."
2
Report No. 2002-P-00004
-------
The Academy panel concluded that current data on
enforcement and compliance, and the systems used to
collect and disseminate that data, were inadequate to meet
the needs of the next generation of environmental programs.
The primary focus of the audit was to evaluate the reliability
of specific DOCKET data elements used to track and
manage enforcement activities, as well as support
externally-reported GPRA performance measures and
annual performance goals. We conducted audit fieldwork
from November 2000 through June 2001, visiting all
10 EPA regional offices, Headquarters' Mobile Source
Enforcement Branch, and Headquarters' Toxics and
Pesticides Enforcement Division (hereafter referred to
collectively as "regions"). We conducted this audit in
accordance with Government Auditing Standards, and
requested and reviewed applicable documentation governing
DOCKET. For further discussion of our scope and
methodology, refer to the scope and methodology section
included in the Exhibit.
In March 2001, the Office of Inspector General (OIG)
issued a report entitled "Enforcement: Compliance with
Enforcement Instruments" (Report No. 2001-P-00006).
The audit found that:
OECA's annual accomplishment reports did not
accurately represent the actual environmental
benefits resulting from enforcement activities, and
associated performance measures were not sufficient
to determine the program's actual accomplishments.
Regions did not always adequately monitor
compliance with enforcement instruments nor did
they always consider further enforcement actions.
For more information, refer to the prior audit coverage
section included in the Exhibit.
The Agency is in the process of responding to this report.
3
Report No. 2002-P-00004
-------
We used Agency policies and procedures to form a
framework of prudent practices for processing DOCKET
data. The following criteria were of particular importance
to our audit field work and analysis:
• Procedures for Maintenance of Enforcement Docket
System
• Enforcement DOCKET Maintenance
Quality Assurance of Active Civil Enforcement Case
DOCKET
• Support of the Enforcement DOCKET for
Information Management in OECA
Clarification of Administrative Order Tracking
Requirements
• Tracking EPA Administrative Action in DOCKET
Guidance for Case Conclusion Data Sheet (CCDS)
Reporting Improvements
For summaries of each of these policy memorandums, refer
to the criteria section included in the Exhibit.
4
Report No. 2002-P-00004
-------
CHAPTER 2
GPRA ENVIRONMENTAL IMPACT UNDER-REPORTED
EPA Under-Reported
Environmental Impact
EPA under-reported the environmental impact of
enforcement actions under GPRA. This occurred due to
incomplete data in the DOCKET System (see Chapter 3).
In particular, DOCKET data was not accurate and complete
for providing information on enforcement actions that
resulted in pollutant reductions. As a result, the information
provided to Congress, the public, and EPA management
was not reliable.
Incomplete pollutant information prevented EPA from
accurately measuring environmental impact. According to
Section II of EPA's fiscal 2000 Annual Report, EPA
enforcement actions resulted in a reduction of
approximately 714 million pounds of pollutants. We
reviewed the methodology used to report pollutant
reductions stemming from enforcement actions and found
pollutant amount data incomplete.
EPA uses DOCKET data to estimate the amount of
pollutant reduced as a result of enforcement actions. This
process requires pollutant amount information to be
completed for all enforcement actions reported as SEP or
compliance action pollutant reductions. EPA calculates
pollutant reduction estimates by selecting records that have
pollutant reductions associated with a SEP or compliance
action. EPA only includes a record in the calculation if it
has both the Pollutant Name and Pollutant Amount.
We selected the pollutant records associated with SEP or
compliance action reductions. For the records selected, we
tested the relationship between the Pollutant Name and
Pollutant Amount fields to determine how many pollutant
records did not have either a Pollutant Name or Pollutant
Amount. All pollutant records had Pollutant Names.
However, a significant number of records did not have
5
Report No. 2002-P-00004
-------
Pollutant Amounts recorded, and consequently were not
used in the calculation of pollutant reductions. The
following table shows our results for fiscals 1998, 1999, and
2000.
Fiscal Year
Number of Pollutant
Records Associated with SEP &
Compliance Action Pollutant
Reductions Identified *
Percent of Records
Without Pollutant
Amounts Entered
1998
1,738
62%
1999
1,561
56%
2000
1,476
69%
* NOTE: The number of pollutant records is different from the number of enforcement actions because
each enforcement action case can generate multiple pollutant records.
As noted in the table, in fiscal 2000, 69 percent of the
pollutant records identified as SEP reductions or
compliance action reductions did not contain pollutant
amount information in DOCKET. As a result, EPA could
not accurately quantify the estimated environmental impact
of enforcement actions and, most likely, understated
progress made in reducing pollutants.
Our review of 53 key fields disclosed that several fields
used for computing GPRA results were inaccurate,
incomplete, or unsupported. We found that the following
GPRA-related fields contained among the highest
occurrences of inaccurate, incomplete, or unsupported data.
GPRA-Related Fields
Inaccurate, Incomplete,
or Unsupported
6
Report No. 2002-P-00004
-------
DOCKET Screen Input Field Name
Projected Statistical Error Rate (%)
Units
54%
Amount Pollutant
52%
Media
52%
Pollutant ID/case
51%
Percent
51%
Frequency
51%
Pollutant Name
43%
NOTE: These fields are defined within the table in Chapter 3.
Existing Problems Affect EPA's current understatement of actual performance results
Future Reporting could result in management setting lower, future
performance targets. Moreover, the high instance of
missing data prevents EPA from (1) measuring the
effectiveness of environmental policy, and (2) optimally
directing national environmental efforts. With more reliable
measurement data, EPA will have the potential to set more
realistic and progressive targets for enforcement actions that
result in pollutant reductions. Finally, management will be
able to focus on enforcement actions that generate the
greatest environmental impact.
Management intends to use DOCKET as the baseline for
populating a replacement system - the Integrated
Compliance Information System (ICIS). Therefore, if
management does not address missing and inaccurate
DOCKET data in a prompt manner, the problems will
continue in ICIS.
Chapter 4 provides further details, including causes of the
DOCKET inadequacies and recommendations for corrective
actions.
The Agency officials agreed with the problems
we identified. They also concluded there has been
significant under-reporting of pollutant reductions.
However, they believe that the understatement does not
Agency Comments
and OIG Evaluation
7
Report No. 2002-P-00004
-------
warrant calling the data unreliable. They believe that
understating the impact is preferable and more credible than
overstating it, and that the data should just be considered an
underestimate.
We concur that it is preferable and more credible to be
conservative when providing estimates. However, we
believe that estimates should be based on a consistent
methodology that:
provides a reasonable assurance that the majority of
the applicable data is included, and
the absence of any data would not have a material
impact on decision-making.
This is not the case based on the Agency's own belief that
the pollutant reduction estimates have been significantly
under-reported. For example, GPRA data is used to assess
what EPA accomplishes over a set period of time and is also
used for establishing a baseline for future improvements.
By setting the baseline so low, perceived performance
improvements can be achieved, without actually improving
performance, by including a larger percentage of the
applicable data in the future year's calculation. The same
scenario applies to using the data internally for managing.
At the Agency's request, we made changes to Chapter 2 to
clarify that the data contained in pollutant-related data fields
are estimates rather than actuals.
Additionally, Agency officials indicated they were unsure
how inaccurate, incomplete, or unsupported DOCKET data
included within fields utilized for computing GPRA results
(GPRA-related fields) could affect those results.
We believe that employing inaccurate, incomplete, or
unsupported DOCKET data to compute GPRA results will
render inaccurate outcomes.
The Agency indicated that they developed a training module
to disseminate the information contained in the pollutant
8
Report No. 2002-P-00004
-------
reduction estimating guidance. Management assured us that
regional training would be completed by the fourth quarter
of fiscal 2002.
9
Report No. 2002-P-00004
-------
Report No. 2002-P-00004
-------
CHAPTER 3
UNRELIABLE DATA IMPAIRS USABILITY OF
SYSTEM INFORMATION
DOCKET contained significant instances of inaccurate and
incomplete data. Many crucial fields were inaccurate and
incomplete, thereby creating a questionable baseline for
managing the national enforcement program and assessing
EPA's environmental performance. Specifically, 94 percent
of the cases reviewed within DOCKET contained errors in
key fields. Therefore, neither Congress, the public, nor
EPA management should depend on the DOCKET system
as a reliable source of environmental data on enforcement
actions. The seriousness of the problem is compounded by
the fact that EPA is making an increasing amount of
DOCKET data available to the public via EPA Internet
websites. Factors contributing to poor data quality included
a cumbersome system that lacks functionality needed to
manage cases; inadequate and outdated policies,
procedures, and guidance; and the absence of
comprehensive quality assurance reviews.
Analysis of statistical sample results showed that
94 percent of DOCKET cases contained at least one error
within the 53 key fields reviewed. Both the OIG and OECA
management believed these 53 fields were critical to
managing the enforcement programs. These fields identified
key milestone dates, as well as quantitative information
related to pollutants, penalties, SEPs, and cost recovery.
The 53 key fields included three fields which OECA
management specifically requested us to review.
DOCKET Data Inaccurate,
Incomplete, or Unsupported
11
Report No. 2002-P-00004
-------
We verified these data elements to source documentation
(detailed information available upon request). Our field
work disclosed that more than 88 percent (88.7% = 67.9%
+ 20.8%)) of the key fields had an error rate greater than
3 percent, and just under 68 percent had an error rate
greater than 10 percent, as shown in the accompanying
chart.
Percent Distribution of Field Error Rates
11.3%
20.8%
67.9%
D Error Rate > 10%
D Error Rate > 3% & <= 10%
I Error Rate <= 3%
Our review disclosed that most key data fields were
inaccurate, incomplete, or unsupported. As a result, the
system often lacked data needed by management to
(1) identify and track cases, and (2) report accurate
performance information to Congress. The following table
identifies the data elements reviewed with error rates
greater than 30 percent.
12
Report No. 2002-P-00004
-------
DOCKET Screen Input
Field Name
Projected Statistical
Error Rate (%)
Field Definition
Case Summary
71%
A summary of the violation(s), environmental
problem(s), and a description of the cause(s) of action
(basis of legal action). The Case Summary should
provide:
A brief summary of the nature of the matter (civil
or administrative) and the problem to be addressed.
The name of principal defendant(s) and type and
number of defendant(s).
A summary of relief requested.
Any special facts or circumstances relevant to the
case.
Units *
54%
The measure used for the amount of reduction or
elimination of each pollutant or waste.
Closed
54%
When all conditions, terms, and monetary requirements
of the order or the settlement document have been met,
the case is considered closed. A case may also be
closed if no further action is contemplated by EPA.
Amount Pollutant *
52%
The amount of reduction or elimination of each
pollutant or waste in pounds, kilograms, or liquid
measure.
Media *
52%
The media where the pollutants or waste were
emitted/discharged (e.g., land, water, air).
Pollutant ID/case *
51%
The pollutant identification number or code assigned to
each pollutant selected.
Percent *
51%
The percent of pollutant or waste reduced or
eliminated.
Frequency *
51%
The frequency measure (in units of time) used for the
amount of reduction or elimination of each pollutant or
waste.
Pollutant Name *
43%
The waste material, substance, or chemical involved at
the facility in violation at the case level. At the
settlement level, the field is used to identify the
nnllnlnnK imnacterl bv the nhvsiral actions taken
* Fields used for GPRA purposes.
The error rates for the three fields specifically requested by
OECA Senior Management were: 71 percent for Case
Summary, 29 percent for Court Docket Number, and
21 percent for Judicial District.
13
Report No. 2002-P-00004
-------
Only reliable information should be shared with Congress
and the public. These error rates are unacceptable if EPA
plans to use this data to address Congressional and public
needs. Moreover, the degree of incomplete or inaccurate
system data hampers the ability to successfully measure
enforcement actions.
Deletion of Cases Not OECA and regional management did not enforce consistent
Properly Controlled procedures to ensure deletions of enforcement cases were
properly documented, authorized, and controlled. OECA
did not have a policy that required written justification and
management authorization prior to deleting a case from the
system. Consequently, no regional policies existed to
ensure internal controls were applied when deleting case
records. Our regional field work disclosed numerous
instances where cases were deleted without proper
authorization and support.
Currently, regions use various local systems to track the
status of cases, rather than relying on DOCKET data as a
management tool. However, OECA is implementing a
replacement system, ICIS, for the express purpose of
tracking and managing enforcement actions. With the
introduction of ICIS, regions will be expected to eliminate
the redundant systems. As a result, the reliability of data in
the new system will play an even larger role for properly
tracking the status of cases. Therefore, management must
establish and implement consistent, effective controls over
the deletion of case records or ICIS will inherit and
perpetuate the same data problems.
We noted that input errors were not instrumental in causing
the inaccuracy and incompleteness of the data. To
determine the input error rate, we verified the contents of
162 fields to the applicable input documents. The 162 fields
were selected based on our knowledge of DOCKET and
CDETS, and on discussions with OECA representatives.
These fields include the 53 key fields previously discussed.
A field was counted as an error if the contents of the
DOCKET field did not match the input document.
Input Errors Do Not
Play Significant Role
14
Report No. 2002-P-00004
-------
Less than 14 percent of the fields selected had an error rate
greater than or equal to 3 percent, and less than 2 percent
had error rates greater than or equal to 5 percent. The
following table shows the three fields with error rates
greater than or equal to 5 percent. Based on these results,
we concluded that input errors did not have a substantial
impact on the inaccuracy and incompleteness of DOCKET
data.
DOCKET Screen Input
Field Name
Projected
Statistical Error
Rate (%)
Complaint/AO (Administrative
Order) Issued (Date Field)
5.86%
Law/Section
5.52%
Lead EPA Attorney
5.17%
Details regarding the results of this review will be provided
upon request.
Chapter 4 provides further details, including causes of the
DOCKET inadequacies and recommendations for corrective
actions.
Agency officials agreed with the problems we identified.
However, management believes the report overstates the
unreliability of the data. In a memorandum dated December
19, 2001, the Agency proposed separating Judicial (Civil)
cases from Administrative cases. The Agency believed this
would provide a better representation of DOCKET data
reliability because certain fields only apply to Judicial cases.
Management is correct in stating that certain fields only
apply under specific circumstances related to each case. In
addition to the fields that are only applicable to Judicial
cases, some fields are only applicable to Administrative
cases, cases with SEPs, Multi-Media cases, etc..
We employed a methodology that conservatively assessed
the overall reliability of the data within DOCKET. We
15
Report No. 2002-P-00004
-------
evaluated each field reviewed based on whether it was
applicable to the particular case. If a field was not
applicable and was left blank, then we accepted it as correct.
This methodology provided a very conservative error rate in
comparison to separately evaluating cases applicable to each
specific situation. We obtained concurrence on the
methodology from Agency officials and the General
Accounting Office (GAO) prior to beginning the review.
We disagree with the Agency's specific examples to support
separating Judicial from Administrative cases because the
fields listed in its response are not solely applicable to Civil
Judicial Cases. For example, the Case Summary field is also
required for Administrative Cases based on section 2.1 of
the Agency's DOCKET User's Manual and the Agency's
Administrative Case Data Form, issued by Headquarters.
The Pollutant Name field is optional (not required) for both
Judicial and Administrative Cases according to sections 3.4,
3.25, 3.28, & 3.29 of the same manual. Likewise, the
Penalty By Statute field applies to both Judicial and
Administrative cases as long as they are a multi-media case
with a penalty. To ensure that we did not overstate any of
the reported error rates, we double-checked our work. We
did not find problems specific to this situation. However,
we did find and correct errors affecting the Penalty by
Statute field caused when the results data was transferred
from Word Perfect to Lotus.
We believe the unreliability of data is not overstated, based
on the significant error rates uncovered for most of the key
fields reviewed.
We provided the results of our review at the case level so
the Agency could break out the results by Civil and
Administrative Cases. As such, management can perform
further analysis that they deem to be of additional benefit.
The Agency also stated that they believe the data in
DOCKET is adequate to track the status of Civil Judicial
and Administrative cases.
16
Report No. 2002-P-00004
-------
We believe that for the data to be useful for managing
purposes, it needs to be reliable. Based on the significant
error rates uncovered for most of the key fields reviewed,
we believe the DOCKET data is not adequate for tracking
the status of Civil Judicial and Administrative caseload.
The results of our fieldwork indicate that Headquarters is
the primary user of collected DOCKET data, and that most
regions use DOCKET predominately as a reporting system
to EPA Headquarters. Regions input the information into
DOCKET, even though it does not meet their needs,
because they have been directed to do so by Headquarters.
The data fields reviewed were key elements for managing
enforcement actions, based on a consensus between OECA
management and the OIG. In addition to quantitative
information, these fields included data used specifically for
tracking the status of the cases, such as milestone dates.
The fact that resources are dedicated to collecting data that
is primarily used by Headquarters, rather than what the
regions need to manage, substantiates the importance of this
data to Headquarters.
The Agency requested the report be clear about the source
of the problems so that corrective actions could be targeted
appropriately. We identified what we believe to be the root
causes within Chapter 4 of the report. We focused our
recommendations, also included in Chapter 4, on corrective
actions to address those causes.
Agency officials also indicated that they have never
experienced problems due to the absence of procedural
controls for deleting cases from the system. They stated
that when a case is deleted, it is not critical to managing
their enforcement program.
We found that the Agency does not have compensating
controls in place to determine if cases were improperly
deleted. We believe it is important to apply proper internal
controls to ensure the integrity of system data. Instituting a
standard control practice becomes even more critical with
the advent of ICIS, because the regions will be expected to
17
Report No. 2002-P-00004
-------
use it to manage and track the status of cases rather than
using stovepipe systems.
18
Report No. 2002-P-00004
-------
CHAPTER 4
CORRECTIVE ACTIONS NEEDED PRIOR TO IMPLEMENTING
REPLACEMENT SYSTEM
OECA is implementing a replacement system for DOCKET,
the Integrated Compliance Information System (ICIS), for
the express purpose of tracking and managing enforcement
actions. If improvements are not made to keep ICIS from
having the same problems as DOCKET, enforcement data
quality will continue to suffer. Due to DOCKET'S lack of
functionality, regions only used it as an information
reporting system to EPA Headquarters, and not as an
enforcement case management system. If ICIS does not
fulfill that role, then regions will continue to expend
additional resources to maintain redundant systems for their
particular managing needs.
Poor Data Quality Poor enforcement data quality will continue if EPA's new
May Continue In ICIS enforcement system, ICIS, does not meet the regions' needs
for managing enforcement actions on a day-to-day basis.
Because of the difficulties encountered with DOCKET,
users primarily relied on their regional systems for managing
cases and placed less importance on the quality of data
entered in DOCKET. The regions have identified critical
functions they deem necessary to meet management
requirements and improve data quality in ICIS. With the
introduction of ICIS, regions will be expected to eliminate
the redundant systems. However, if EPA does not address
the user's needs within ICIS, the new system will perpetuate
the same data quality concerns.
Regional personnel consistently acknowledged the
importance of having a single system for tracking and
managing enforcement actions. As such, all regions
participated in OECA focus group discussions to identify
ICIS system requirements. In addition, some regions
provided us with input regarding system improvements. A
few of the regional suggestions were:
19
Report No. 2002-P-00004
-------
Establish single point of entry to satisfy all
environmental systems' data requirements.
Establish security controls to protect high profile
litigation or high dollar cases.
Establish automatic notification of missed milestones
(e.g., e-mail).
See Appendix 2 for a complete listing of regional
suggestions.
If ICIS is not designed to meet user needs, then regions will
continue to expend additional resources to maintain
redundant systems. We surveyed EPA to determine the
extent regions were developing and maintaining independent
systems for enforcement and compliance tracking. We
received responses from all regions and found that they have
already spent $290,000 to develop 19 redundant, "stove-
piped" systems to meet their needs, and have budgeted
approximately $1.3 million over the next 5 years to maintain
those 19 systems. We believe the regions could better use
these funds to support environmental programs.
Our review disclosed that multiple causes contributed to
incomplete and inaccurate data in DOCKET. Specifically,
we identified the following main factors:
DOCKET was not used to manage, because it was
considered cumbersome and lacked functionality.
DOCKET'S policies, procedures, and guidance were
outdated and inconsistent.
OECA did not provide guidance on identifying and
calculating pollutant information.
Regions did not comprehensively review
enforcement data.
20
Report No. 2002-P-00004
-------
Regional policies and procedures were inadequate
and inconsistent.
Details on each of these areas follow.
System Not Used to Manage Regions did not use DOCKET on a daily basis for
enforcement case management, because users said they
considered the system to be cumbersome, non-user-friendly,
and lacking the functionality needed to manage. Regions
were primarily using DOCKET as an information reporting
system to EPA Headquarters.
DOCKET'S Policies, OECA had not updated the DOCKET User's Guide and
Procedures, and Guidance Data Dictionary to identify critical data elements essential
Were Outdated and for performance reporting. Many crucial data elements
Inconsistent were designated as optional, resulting in many database
fields being unpopulated. For example, the current
DOCKET User's Guide states the pollutant fields are
optional. Therefore, pollutant information is often not
reported, even though such data factors into
Congressionally-reported performance measures. The error
rates for these particular fields are listed in Chapter 2.
In addition, inconsistencies between DOCKET'S User's
Guide, Data Dictionary, and input forms could lead to
confusion by users. Some examples of inconsistencies are:
The Administrative Case Data Form indicates that
fields are required, although the detailed section of
the User's Guide indicates they are not required.
The Administrative Case Data Form indicates that
certain fields included on the form are not required,
even though the general section of the User's Guide
indicates they all are required.
The required fields per the Data Dictionary do not
always coincide with the User's Guide required
fields.
21
Report No. 2002-P-00004
-------
Guidance Not Provided
for Addressing
Pollutant Information
Furthermore, DOCKET does not force users to input data
into many key fields. OECA indicated that the User's
Guide and Data Dictionary define required fields as those
for which the system demands data input. However,
prudent practices should ensure that fields which senior
management have identified as key to managing the
program and reporting results coincide with the mandatory
entry fields defined in the application system. Because
DOCKET'S system requirements do not always correspond
with management's desires, users cannot always trust the
User's Guide or Data Dictionary to determine what data
they must enter into the system.
Although EPA recently issued an extensive training manual
regarding pollutant information, the regions have not
implemented the guidance. Our field work concluded that
personnel consistently provided incomplete or inaccurate
pollutant information and needed to be trained. OECA
reached similar conclusions as a result of independent
studies, although they did not issue any comprehensive
training guidance until January 2001. During the last
several years, OECA has enhanced DOCKET to
accommodate input of CCDS information, created a new
CCDS, and initiated additional reviews to address
weaknesses in their information reporting methodology.
OECA's latest review1 found that incomplete data was a
major issue for several key data elements, such as pollutant
name, amount, and units.
Whereas OECA's recent manual included guidance on how
to identify and calculate pollutant information, management
did not identify an implementation date for using the new
procedures. Consequently, the regions did not apply the
guidance. OECA stated that a program for delivering onsite
training is being developed.
1 Report entitled, "Review, Analysis, and Recommendations
for Improvement of Office of Enforcement and Compliance Assurance
Case Conclusion Data Sheets - Final Report," dated November 29,
1999, Page 1.
22
Report No. 2002-P-00004
-------
To accurately and consistently compute pollutant
information associated with compliance actions, field
inspectors must be aware of the information needed for the
computations outlined in the guidance. In some cases, these
field inspectors are state employees. If the state field
inspectors are not advised of the data necessary for the
computations associated with potential compliance actions,
then it is conceivable that not all of the necessary
information will be collected during the inspection. Without
this information, EPA may still have problems accurately
and consistently computing pollutant information.
Quality Assurance Reviews Regions relied on inconsistent, limited reviews rather than
Not Performed systematic and comprehensive quality assurance reviews.
Although OECA published procedures for conducting
monthly quality assurance reviews of all DOCKET fields,
the regions were not adhering to the policy. The policy
requires that the lead attorneys review and update all data
fields using the Monthly Case Update Report. In addition,
the User's Guide states that the information for completing
some of the key fields will be provided on the Monthly Case
Update Report. We found that the regions were not using
this report and, as a result, the information for these fields
were often not completed or reviewed.
None of the regions had formal policies and procedures
which assigned responsibility and accountability for
conducting quality reviews. The informal reviews merely
consisted of reconciling case names in stovepipe systems to
DOCKET and/or performing high-level reviews of limited
fields. Furthermore, we found that whenever regions
distributed reports for quality assurance purposes, the
reports did not contain all data fields. The inconsistent and
limited nature of quality assurance reviews perpetuated high
error rates in DOCKET.
The regions did not have formal policies and procedures
to implement OECA's policies, procedures, and guidance
for all phases of enforcement data processing (i.e.,
identification, collection, inputting, deletion, quality
assurance, etc.). As a result, regional management had not
Policies and Procedures
Need Improvement
23
Report No. 2002-P-00004
-------
formalized its expectations, nor implemented a formal
mechanism to ensure that personnel processed enforcement
information in a consistent manner. The lack of formalized
policies and procedures led to inconsistencies in the way
enforcement data was processed from region to region, as
well as within regions.
We recommend the Assistant Administrator for
Enforcement and Compliance Assurance:
1. Develop and implement a process to ensure system
requirements, identified by the users at Headquarters
and within the regions, are included in the design
and implementation of the system being developed
to replace DOCKET.
2. With regards to OECA's policies, procedures, and
guidance related to DOCKET, as well as guidance
related to pollutant information:
a. Revise, consolidate, issue, and monitor the
implementation of all policies, procedures,
and policy memorandums related to
DOCKET. The policies and procedures
should:
(1) Be based on management's needs and
how the data will be used;
(2) Address all phases of DOCKET/new
system (ICIS) data processing, e.g.,
identification, collection, input,
deletion, quality assurance, etc.;
(3) Assign authority, responsibility, and
accountability for each of the phases
and processes; and
(4) Ensure all data elements applicable to
an enforcement action are completed
timely, accurately, and consistently.
24
Report No. 2002-P-00004
-------
b. Develop and implement training coinciding
with the issuance of new policies and
procedures for all applicable personnel
(e.g., attorneys, data entry, technical
representatives, etc.).
c. Revise the User's Guide, Data Dictionary,
and input forms to eliminate inconsistencies
and to identify management requirements
related to the data elements.
d. Formally communicate and reach agreement
with the states regarding what information
needs to be collected during inspections that
may result in compliance actions. This
information is necessary, per the guidance, to
complete the CCDS.
3. In regard to the Quality Assurance Process:
a. Develop and implement policies and
procedures that establish a Quality
Assurance process. OECA management
should periodically monitor and review the
Quality Assurance process. The Quality
Assurance process should:
(1) Assign authority, responsibility, and
accountability for Quality Assurance
for each case to the person most
knowledgeable about the case
(attorney);
(2) Establish and document how often
the procedures should be performed.
The timing of the Quality Assurance
reviews should be tied to
management's information needs; and
(3) Be monitored and periodically
reviewed by OECA management.
25
Report No. 2002-P-00004
-------
b. To assist in the identification of missing data,
develop standard reports that will identify
missing or potentially missing data for each
type of case and each relevant segment of
information, (e.g., SEPs, Audit Policy,
pollutant-related information, etc.), and have
the regions review these reports as part of
the Quality Assurance process.
4. With regard to populating and deploying the new
system:
a. Identify those cases that need to be brought
into the new system (ICIS); and
b. Develop and apply a scrubbing process to
the cases identified (prior to populating the
system) to ensure that the data used to
populate the new system is accurate and
complete.
We recommend the Regional Administrators and EPA's
Director of the Office of Regulatory Enforcement:
5. Develop and implement formal regional policies and
procedures that:
a. Implement OECA's policies and procedures
effectively and efficiently at each site;
b. Establish lines of authority and
communication across organizational
segments to ensure that the data collected
and maintained within the system is accurate,
complete, timely, and consistent; and
c. Establish a monitoring process to:
(1) Verify the policies and procedures
are being followed,
26
Report No. 2002-P-00004
-------
(2) Continuously evaluate the efficiency
and effectiveness of the processes,
and
(3) Identify opportunities to improve
processes.
Agency officials agreed with the causes identified in this
report. They pointed out that it would be helpful if we
could identify the inconsistencies we found during our
review.
We provided them with examples of inconsistencies we
found during our cursory review of the DOCKET User's
Guide, Data Dictionary, and input forms. However, we
believe that fixing the ones we noted does not fulfill
management's responsibility to perform their own review
and make applicable corrections.
Agency officials noted that our report does not present
analysis based on data produced after the issuance of the
guidance (related to estimating pollutant reduction results)
in January 2001.
The sample cases selected were all opened prior to
September 29, 2000. However, we verified the accuracy of
the data, associated with the selected cases, as of the dates
we performed the field work. We conducted field work for
nine of the ten regions after the guidance was issued. The
nine regions we visited after it was issued stated they had
not started using the guidance yet.
The Agency asked us to clarify that Recommendation 2.d.
refers to the information on the Case Conclusion Data
Sheets for pollutant information. We clarified the
recommendation, as requested.
Agency officials stated that they would respond to the
specific recommendations upon receipt of the final report.
27
Report No. 2002-P-00004
-------
EXHIBIT
DETAILS ON SCOPE, METHODOLOGY, AND CRITERIA
Scope And Methodology
Congress asked the Inspector General community to both verify and validate the accuracy and
reliability of performance data used to report on the Agency's GPRA annual performance goals.
Verifying is the process of determining whether system data is accurate and complete (i.e.,
reliable), whereas validating entails determining whether an agency's key performance measures
are consistent with its strategic and annual performance goals. Our audit was limited to verifying
the accuracy and reliability of the data within DOCKET.
We selected two random attribute samples to help assess the accuracy and completeness of
DOCKET data. We selected one sample for active cases and a second sample for deleted cases.
We used the sample for active cases to perform source document verification for 53 key fields and
input document verification analysis. We used the second sample to review support documents
associated with the justification and authorization of deleted cases.
We also performed analysis of the complete DOCKET Database for edit checks and field
relationships, as defined by the DOCKET Data Dictionary.
OECA management disclosed they were developing a replacement system for DOCKET entitled
Integrated Compliance Information System (ICIS). As a result, we requested that regions identify
what improvements they would like implemented in the replacement system. We also conducted
a survey to determine the resources regions committed to develop and maintain systems for
managing enforcement actions. We performed this survey because these systems were used in
lieu of DOCKET.
Prior Audit Coverage
In March 2001, the OIG issued a report entitled "Enforcement: Compliance with Enforcement
Instruments," (Report No. 2001-P-00006). The audit found that:
OECA's annual accomplishment reports did not accurately represent the actual
environmental benefits resulting from enforcement activities. For example, an OECA
report stated that fiscal 1999 enforcement actions resulted in the reduction of more
than 6.8 billion pounds of pollutants. However, this may have been an understatement
28
Report No. 2002-P-00004
-------
or overstatement since: (1) violators did not always comply with the enforcement
instruments, and (2) data was not comprehensive. Also, OECA's performance
measures were not sufficient to determine the program's actual accomplishments.
Consequently, Congress had less useful performance data upon which to base its
decisions.
Regions did not always adequately monitor compliance with enforcement instruments
nor did they always consider further enforcement actions. Ineffective monitoring was
due primarily to the lack of: (1) guidance detailing how or when to monitor
enforcement instruments, and (2) emphasis OECA placed on monitoring. Ineffective
monitoring may have contributed to the regions not considering further enforcement
actions for noncompliance with enforcement instruments. Consequently, there was a
risk that (1) violations would continue and contribute to environmental harm or
increased health risks, and (2) EPA's effectiveness, through deterrence, would be
adversely impacted. The report listed instances where EPA had no evidence that
significant violations had been corrected.
Criteria
Included below are summaries of criteria that were of particular importance to our audit field
work and analysis:
Procedures for Maintenance of Enforcement Docket System
This memorandum, dated April 21, 1983, established the Enforcement Docket System as
the official system for tracking and maintaining summary and status information about civil
and criminal enforcement actions. The document was the official record of all litigation
activity used for:
Reporting to the Administrator regarding accountability measures,
Responding to Congressional inquiries,
Answering Freedom of Information Act requests, and
Performing special analyses.
According to this source, the Lead EPA Attorney is responsible for completing the case
input forms and giving them to the regional data analyst for assignment of a case number
and initial data entry. The Lead EPA Attorney also has primary responsibility for the
review and update of all active cases. In addition, the document establishes and assigns
the responsibility for the quality assurance function to the Lead EPA Attorney.
29
Report No. 2002-P-00004
-------
Enforcement DOCKET Maintenance
This policy memorandum, dated April 8, 1988, identified specific responsibilities for the
regional offices and EPA Headquarters. The document stated the primary responsibility
for timeliness, accuracy, and completeness of DOCKET data resided with the Offices of
Regional Counsel. Specifically:
Regions were responsible for accurate updates at least monthly;
Headquarters was responsible for accurate monthly update of Headquarters-
initiated data fields;
Headquarters would not amend regional data entry;
Headquarters would continue to monitor overall data quality on a monthly
basis for the balance of fiscal 1988, and thereafter on a quarterly basis.
Headquarters would be responsible for bringing discrepancies to the attention
of the Regional Counsel; and
DOCKET maintenance would be considered part of the annual performance
assessment discussion with Regional Counsels.
Quality Assurance of Active Civil Enforcement Case DOCKET
This policy, dated March 2, 1989, addressed the quality of the data in DOCKET. The
document emphasized EPA's reliance on DOCKET information to (1) manage
enforcement programs, and (2) respond to internal and external inquiries (e.g., Congress
and the public). Furthermore, the policy decentralized the Agency's enforcement process
by giving regions the authority and responsibility for initiating and following through on
enforcement actions.
Support of the Enforcement DOCKET for Information Management in OECA
On October 3, 1994, this policy assigned the Regional Counsels primary responsibility for
entering and maintaining data on all civil judicial and administrative enforcement actions.
In particular, Regional Counsels were to ensure that:
Every civil judicial and formal administrative penalty enforcement case was
entered into DOCKET;
All new formal administrative penalty actions, starting with fiscal 1995, were
entered into DOCKET; and
By the end of the second quarter of fiscal 1995, all administrative penalty
orders that had been issued or filed but not yet concluded, were entered into
DOCKET.
30
Report No. 2002-P-00004
-------
Clarification of Administrative Order Tracking Requirements
This policy memorandum, dated May 9, 1995, clarified administrative orders and, in
particular, addressed the inefficiencies of dual data entry - that is, entering the same data
into DOCKET and various program databases. This policy emphasized that all
administrative penalty actions were to be entered into DOCKET. The document also
reemphasized that DOCKET was to be the primary source for Enforcement data.
Tracking EPA Administrative Action in DOCKET
This policy memorandum, dated March 6, 1996, clarified that all EPA administrative
penalty orders and orders for cost recovery, issued in fiscal 1996 and subsequent years,
were to be entered into DOCKET. The document also reaffirmed that DOCKET was the
official system for tracking and reporting on EPA civil judicial and administrative
enforcement actions.
Guidance for Case Conclusion Data Sheet (CCDS) Reporting Improvements
This guidance, dated January 2, 2001, provided instructions and examples for collecting
and computing pollutant-related enforcement data, used to describe the impact of
enforcement actions on the environment.
31
Report No. 2002-P-00004
-------
APPENDIX 1
OECA COMMENTS TO DRAFT REPORT
T. .,-V
-'C PRCp-*-
a *
UNITED STATES ENVIRONME
WASHINGTO
TECTION AGENCY
¦8Q
OFFICE OF
ENFORCEMENT AND
COMPLIANCE ASSURANCE
MEMORANDUM
SUBJECT: OECA Comments on Draft Report of Quality of Data in the Enforcement Docket
Thank you for the opportunity to provide comments oil your Draft Report. Your review
of the Docket was timely as we are preparing to implement an enlot cement management system
called the Integrated Compliance Information S) stem (I CIS") that w ill leplace the cunent
Enforcement Docket. Lessons learned from the Docket System will pun e valuable as we go
forward with the new system, I have attached specific comments on) oui lepoi I to this
memorandum.
We think that your report overstates the unreliability of environmental benefit
information in Docket while overlooking the fact that the Docket is reliable for the main purpose
for which it is used-tracking and reporting EPA enforcement actions. Ha\ii2 nil hat one of
the primary values of the pollutant reduction calculations is demonstrating th tm nonmental
benefit of our enforcement cases. The first essential stage in that demonstt ition is c t thli ring
that there was a pollution reduction-which the data from Docket does. Willie me ponuiant
reductions or other benefits claimed are probably understated in Docket (an issue we will
continue to address), we believe that demonstrating the environmental results of our enforcement
program is a major step for moving our program management measures towards environmental
outcome tnd one that has been emulated by a number of states. The development of ICIS will
fati I tak unproved reporting of outcome-oriented information, but the information Docket
cuiitnlh provides has been of great value in identifying and analyzing the results we achieve.
System
FROM; r_, Sylvia K. Lowrance'l *7*"
A ' Acting Assistant Administrator
TO:
Patricia H Hill. D hector
Information Technology Audits Staff
Internet Address (URL) • http://www.epa.soy
RecycledlRKydafote • Printed with C*l Based Inks on Recycled Paper (Minimum 30% Poaiconsymer)
32
Report No. 2002-P-00004
-------
We «Mt AM «f juHattmt ndosttoni em and wiii be i«paw«| iIkm#
van** ¦«» n.i"««emert As aowd. wf tfcirt l?»t that has, ten =.Knif t.« cndtr-
* " • (MlMfaK. Tluu hottuv«, |g peferails a the enter, but less credible,
_ jung irnills. Ow efpsMioi li iist the pncMM wit, whiei time
t -rfi MMSBUi!« mpwm over i® wtta mmaqprioK^ gartwesaiiit
IfteBMWgr ewmwBitsci qttesfaM about «w i^w, «*»» cman Frederick F.
SlKil, Director, blMM FtattiBt, Ingrttaj md MMsiatm, at (302) 564-2290.
AtetmenE
33
Report No. 2002-P-00004
-------
Comments m Draft Report - Audit No. 2QQO<0000813
White ws agree with »t»«y of the problems identified in the audit, we believe that the
statement of audit result* is overly bma4 aod that it creates an impression that the data
avuiltiblc in Docket, based particularly on pollutant reductions, is so inaccurate as to be
useless A fairer representation of the findings, again on pollutant reductions, is that the
Aria ill Docket is likely to significantly understate the actual environmental benefits
because pollutant amount fields are not completed for many cases where they could be.
Likewise, the audit findings convey the impression that the data From Docket cannot be
relied upon to provide the basic Wotmation thai Docket was created to track - the status
of the civil judicial, and now administrative, caseload. We think that the data in Docket
is adequate to serve that purpose. Docket was created as a system to track and manage
the Agency.'s civil enforcement actions, hut it was not intended to track and manage the
fill life cycle of a ca$e--it functions awkwardly in that role, In the last few yean. Regions
have- developed a need for such systems and have started to create their own. The new
enforcement and compliance system under development by OBCA, ICIS, is being
designed both to trade case status and to serve as a management tool for Regions and
I l«dq«artcra.
In evaluating data quality, the auditors apparently applied the same assumptions about
what is repaired to administrative cases as to judicial cases. A fatter representation of
data quality would 1* made the report specifically differentiated between civil judicial
and administrative cases. The requirements ate ma the same for both, For example, case
summaries and pollutants at the case level are iwl mandatory fields for administrative
eases whereas they are required for judicial actions Another example (applicable to both
administrative and judicial multimedia actions) is the penalty amount by statute. It is not
mandatory to track the penalty by statiue- only the total assessed penalty. As a result, the
extent of error identified in the report is overstated. We hud asked that you specifically
differentiate the errors between the.* two areas in our comments on a previous draft of
this report; However, this was not changed in this version
Finally, the Docket system is Wanted for what aire arguably, management deficiencies.
Hie report acknowledges that data entry is not the reason for the error rates, awl this
implies thai data entry clerics ere provided with inadequate or inaccurate data in the first
place. We think that the report should be clear about the source of the problem so that
corrective actions wilt be targeted appropriately, It doesn't matter what system is used if
the data provided is inadequate in the fust place. In all instances, where yon can provide
the details regarding results of this review, please provide them »that we can take steps
to correct lie problem.
OECA Comments - Docket Data Quality I December 20, 2001
34
Report No. 2002-P-00004
-------
S&c£ifi.c CamnaiBte
KPA I titter-Reported f.nvtronmeiual lmp»il
OECA concurs that enforcement actions recorded in Ducket that indicate expected
environmental improvements - pollutant reductions treatment of fomtantBiaied material*
or preventative management of poten-.tally damaging subatalKttMahoalil include
quantitative information.
To encourage more complete and accurate reporting of environmental benefits, OECA
has worked with the Regions lo ensure thai pollutant reduction information is included in
case conclusion data .sheet*. In .Ian u«ry 2001. She Office of Compliance (OC) distributed
specific guidance to support regional and headquarters staff in preparing quantitative
estimates of pollutant reductions and m applying -qoality control processes to their
enforcement-related envirnnir.ehial data overall. Additionally, vro have developed a
training module that will be delivered to managers and staff in Headquarters and Regions
who arc rcifK>£»iblc for concluding and reporting an enforcement actions, The first
session is December 18,2001, and we pirn to complete training for all 10 regions by the
end of the 4" quarter
While we agree that many cases that should have pollutant reductions reported do not.
we do think that your opening siaeinetrt that "information provided to Congress, the
public, and EPA management was Dot reliable," is a substantial overstatement. The
failure to report may result from a failure of management to sufficiently oversee or audit
the work of those estimating the environmental benefits or it may be a result of a
conservative approach to claiming environmental betwfitis. In any case, it does nor
suggest that what is reported is wrong - only that it is an undc-estimate.
We do suggest that in your report you m Ac clear that the pollutant reduction, treatment,
and' preventative substance management quantities identified are estimates, sothat readers
do not misunderstand this data as redactions that have already occurred, a. paint we have
recently clarified. at the IG's suggestion - in documents intended to be released to the
public,
GPRA Fields Inaccurate. in«umpl tit. or Unsupported:
Met a comprehensive reading of the draft, we sane sot safe how your graph of
"inaccurate, incomplete, or unsupported" GPJLVrdated fields were derived nor what thai
allegation means. We have conceded above flat there is under-reporting, and
acknowledge that the data may include some incorrect reporting (which our training and
guidance is intended to minimize). We will be glad to review and incorporate into our
processes, as needed, sip&iifie information you can share in this area which would allow
us to make corrections.
(MCA Cmmertts - Docket Data Quality 2 December 20, 200!
35
Report No. 2002-P-00004
-------
Deletion of < a«e\ No l Properly Cimfrt-lk'tl
In pmrtiee, this 'has never beat a problem. We give access to Docket personnel for lie
puptise of crating m wsR asdeleting cases, What• owe mm Metedbootc-riiical to
managing our fflfwegnjeiif program.
('»#«« for I'arelfetblc Docket Data Nerd to it* Addresei
£k
-------
He GBCA response wi!t be made upou receipt of the Final Report However, the
recommendation thai OECA " Formally communicate anJ rcwlv agreement with the
itates regarding the required information flat needs to be collected during tasjwctbns,"
should be revised «that it's clear that this refers to inferaiattOB on i m
Date Sheets for pollutant iafoitnation.
OECA Comments - iteefaf Data Qualty 4 Dtctmbtr 20, 200!
37
Report No. 2002-P-00004
-------
APPENDIX 2
REGIONAL INPUT FOR SYSTEM ENHANCEMENTS FOR ICIS
From December 2000 through May 2001, we visited each EPA Regional Office and
Headquarters. During these visits, we asked each location to provide suggested improvements
they would like to see incorporated into the Integrated Compliance Information System (ICIS).
Regions participated in OECA Focus Groups on the new system (ICIS), where they provided
suggestions for design enhancements to meet their needs. Most regions did not provide a
comprehensive list of suggestions because they had already provided them directly to OECA.
We received a total of 38 responses. All regions did not provide comments. The table below is
not a complete listing of requirements from all locations visited. We grouped similar responses
into categories and ordered the listing based on the number of responses received for each
category. The table below summarizes the comments received.
Additional Considerations for the Development of ICIS
(* Input Provided By EPA Regions and Headquarters)
Number of Responses
Received
Category
10
Improve Report Capabilities
a. Improve report formats to reduce paper
b. Add capability to detect and report missing data
c. Integrate quality control process into system
7
Integrate Regional Systems' Capabilities
d. Integrate the capabilities of regional systems such as Compliance
Activity Tracking System (CATS), Super DOCKET, etc..
6
Improve Interface
e. Make data easier to access the data
f. Make Ad-Hoc Queries easier to run
g. Make the screens more user-friendly
5
Improve Search Capabilities
h. Make it easier to use
i. Add the capability to see Defensive Cases
j. Add the capability for managers to see milestones
38
Report No. 2002-P-00004
-------
Additional Considerations for the Development of ICIS
(* Input Provided By EPA Regions and Headquarters)
4
Add New Features
k. Add automatic notification of missed milestones (e.g., e-mail)
1. Add security to protect high-profile/high-dollar cases
m. Add fields for supplemental referrals
n. Add ability to hotlink to a source document
3
Reduce Duplicate Entry of Data in Multiple Systems
o. Develop an integrated system that shares the data with all other
related systems so that each applicable data element only has to
be entered one time
3
Improve Policy and Procedures
p. Require regions to use
q. Allow entry of data by those most knowledgeable about the case
information
r. Standardize data codes
* NOTE: Not all regions and Headquarters offices provided comments. This list is not a complete listing
of regional and Headquarters requirements.
39
Report No. 2002-P-00004
-------
APPENDIX 3
REPORT DISTRIBUTION
Recipient
Acting Assistant Administrator for Enforcement and Compliance Assurance (2201 A)
Office of Inspector General
Inspector General (2410)
Assistant Inspector General for Audits (2421)
Media and Congressional Liaison (2410)
Counsel (2411)
Editor (3A100)
EPA Headquarters
Agency Audit Foilowup Official (271 OA)
Agency Followup Coordinator (2724A)
OECA Audit Liaison (2201 A)
Director, Office of Compliance (2221 A)
Director, Enforcement Planning, Targeting, and Data Division (2222A)
Director, Office of Regulatory Enforcement (2241 A)
Director, Toxics and Pesticides Enforcement Division (2245A)
Director, Air Enforcement Division (2242A)
Regional Offices
Region I: Regional Administrator (RAA)
Director, Office of Environmental Stewardship
Audit Follow-up Coordinator (MIO)
Region II: Regional Administrator
Regional Counsel
Audit Follow-up Coordinator
Region III: Regional Administrator (3RA00)
Regional Counsel (3RC00)
Audit Follow-up Coordinator (3PM70)
Report No. 2002-P-00004
40
-------
Region IV: Regional Administrator
Director, Environmental Accountability Division
Audit Follow-up Coordinator
Region V: Regional Administrator (R-19J)
Regional Counsel (C-14J)
Audit Follow-up Coordinator (MF-10J)
Region VI: Regional Administrator (6RA)
Regional Counsel (6RA-D)
Director, Compliance Assurance and Enforcement Division (6EN)
Audit Follow-up Coordinator (6MD-RC)
Region VII: Regional Administrator (RA)
Regional Counsel (CNSL)
Audit Follow-up Coordinator (PLMGRFMB)
Region VIII: Regional Administrator (RA)
Regional Counsel (RC)
Assistant Regional Administrator, Office of Enforcement, Compliance,
and Environmental Justice (ENF)
Audit Follow-up Coordinator (TMS-ISP)
Region IX: Regional Administrator (ORA-1)
Regional Counsel (ORC-1)
Audit Follow-up Coordinator (PMD-4)
Region X: Office of Regional Administrator (RA-140)
Office of Regional Counsel
Audit Follow-up Coordinator
41
Report No. 2002-P-00004
------- |