s A
Iffil
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Audit Report
ECHO Data Quality Audit -
Phase 2 Results: EPA Could
Achieve Data Quality Rate With
Additional Improvements
Project No. 10-P-0230
September 22, 2010

-------
Abbreviations
DMR
Discharge Monitoring Report
ECHO
Enforcement Compliance and History Online
EPA
U.S. Environmental Protection Agency
GPRA
Government Performance and Results Act of 1996
ICIS-NPDES
Integrated Compliance Information System - National Pollutant Discharge

Elimination System
IDEF
Interim Data Exchange Format
OECA
Office of Enforcement and Compliance Assurance
OIG
Office of Inspector General
OMB
Office of Management and Budget
PCS
Permit Compliance System

-------
U.S. Environmental Protection Agency	10-P-0230
9	\ Office of Inspector General	September 22,2010
I fi

At a Glance
Why We Did This Review
We sought to assess the
quality of key data elements
reported through the
Enforcement Compliance and
History Online (ECHO)
Website. KPMG, LLP,
performed the review.
Background
ECHO provides a single
source of detailed compliance
history of U.S. Environmental
Protection Agency (EPA)-
regulated facilities. EPA
developed ECHO to provide
the public compliance and
inspection data under its
environmental programs, as
well as demographic data of
the surrounding areas. This
report focuses on the quality
of data elements entered into
ECHO source systems.
For further information,
contact our Office of
Congressional, Public Affairs
and Management at
(202) 566-2391.
To view the full report,
click on the following link:
www.epa.aov/oia/reports/2010/
20100922-10-P-0230.pdf
Catalyst for Improving the Environment\
ECHO Data Quality Audit - Phase 2 Results:
EPA Could Achieve Data Quality Rate With
Additional Improvements
What KPMG Found
EPA mandates that data elements reported to the public through the ECHO
Website have a 95 percent accuracy rate. KPMG found a 91.5 percent data
accuracy rate for key data elements entered into two primary ECHO source
systems: the legacy Permit Compliance System (PCS) and the newer Integrated
Compliance Information System - National Pollutant Discharge Elimination
System (ICIS-NPDES). Although the 91.5 percent data quality rate is close to
EPA's goal, EPA and the State environmental offices could take additional steps
to increase the quality of data reported through the ECHO Website.
What KPMG Recommends
KPMG made several recommendations to the Assistant Administrator for
Enforcement and Compliance Assurance. These included:
•	Establishing an internal control structure to help manage the conversion of
PCS to ICIS-NPDES.
•	Including language in the National Program Manager Guidance requiring
the use of the Environmental Information Exchange Network for reporting
data to EPA.
•	Developing a plan to share data quality best practices implemented at
State environmental offices with all States.
•	Completing new rules requiring reporting ECHO data for minor facilities
and notifying ECHO Website users that the site does not contain data on
minor facilities.
•	Reviewing procedures used to test ICIS-NPDES programming code
before it is placed into production.
The Agency generally agreed with the report findings. EPA felt it has extensive,
documented procedures in place to test ICIS-NPDES programming code before it
is placed into production. Tests disclosed that programming errors directly
resulted in incomplete data on the ECHO Website and, as such, management
should review these processes to prevent future occurrences. The Agency's
response is included in Appendix A.

-------
* o \
I® I
V PRO^0
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 22, 2010
MEMORANDUM
SUBJECT: ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve
Data Quality Rate With Additional Improvements
Report No. 10-P-0230
FROM: Arthur A. Elkins, Jr.
Inspector General
TO:
Cynthia Giles
Assistant Administrator for Enforcement and Compliance Assurance
This is the report on the subject audit conducted by KPMG, LLP, on behalf of the Office of
Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report
contains findings KPMG LLP identified and corrective actions recommended. This report
represents the conclusions of KPMG and does not necessarily represent the final position of the
OIG on the subjects reported. Final determinations on matters in this report will be made by
EPA management in accordance with established audit resolution procedures.
The estimated cost of this audit, which includes contract costs and OIG contract management
oversight, is $331,361.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed-upon
actions, including milestone dates. Your response will be posted on the OIG's public Website,
along with our comments on your response. Your response should be provided in an Adobe
PDF file that complies with the accessibility requirements of section 508 of the Rehabilitation
Act of 1973, as amended. If your response contains data that you do not want to be released to
the public, you should identify the data for redaction. We have no objections to the further
release of this report to the public. This report will be available at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact
Patricia H. Hill at (202) 566-0894 or hill.patricia@epa.gov. or Rudolph M. Brevard at
(202) 566-0893 or brevard.rudv@epa.gov.

-------
September 22, 2010
SUBJECT: Audit Report:
ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve
Data Quality Rate With Additional Improvements
THRU: Arthur A. Elkins, Jr.
Inspector General
TO:	Cynthia Giles
Assistant Administrator
Office of Enforcement and Compliance Assurance
This memorandum is to inform the U.S. Environmental Protection (EPA) of critical findings by
KPMG, LLP, that require management action regarding the improvement of the data quality for
the Enforcement Compliance History Online Website.
If you or your staff have any questions regarding this report, please contact the
U.S. Environmental Protection Agency, Office of Inspector General, point of contact
identified on the report transmittal memorandum.

-------
ECHO Data Quality Audit - Phase 2 Results:
EPA Could Achieve Data Quality Rate With
Additional Improvements
10-P-0230
Table of C
Purpose		1
Background		1
Scope and Methodology		2
Findings		4
Identified Data Quality Exceptions		4
Factors That Contributed to Data Quality Exceptions		5
No Automated Data Quality Processes		5
No Consistent State Quality Review Processes		6
No Standardized DMR Form		6
No Documentation of Data Entry		7
No Requirements to Report Minor Facility Data		7
Michigan Data Processing		7
Data Quality Exceptions Affect EPA's Representation of Data		8
Recommendations		8
Agency Comments and OIG Evaluation		9
Status of Recommendations and Potential Monetary Benefits		10
Appendices
A Agency Response	 11
B Distribution	 18

-------
10-P-0230
Purpose
The U.S. Environmental Protection Agency (EPA) Office of Inspector General (OIG)
initiated this review to evaluate the quality of data reported through the EPA
Enforcement Compliance and History Online (ECHO) Website. KPMG LLP (KPMG)
completed the review in the following two phases:
•	Phase 1: Evaluate the automated process EPA uses to display ECHO data
transmitted from select source systems. We reviewed the automated processes for
the legacy Permit Compliance System (PCS) and the newer Integrated
Compliance Information System - National Pollutant Discharge Elimination
System (ICIS-NPDES). We provided the Phase 1 results in OIG report, ECHO
Data Quality Audit - Phase I Results: The Integrated Compliance Information
System Needs Security Controls to Protect Significant Non-Compliance Data,
Report No. 09-P-0226, August 31, 2009.
•	Phase 2: Test the quality of select PCS and ICIS-NPDES data elements against
Discharge Monitoring Reports (DMRs), which are the primary source documents
for PCS and ICIS-NPDES data.
Background
ECHO provides a single source of detailed and up-to-date environmental compliance
history of EPA-regulated facilities. EPA developed ECHO in partnership with State
environmental offices to provide compliance and inspection data under air, water, and
hazardous waste programs, as well as demographic data of the surrounding areas. Since
ECHO'S November 2002 launch, hundreds of thousands of citizens, government
officials, investors, and staff at regulated facilities and companies have asked over
3,000,000 questions about the environmental records of the more than 800,000 facilities.
EPA and the state environmental offices need to maintain a high level of ECHO data
quality to: 1) provide accurate and complete information to the public, States, facilities,
and other stakeholders, and 2) maintain their reputations as effective public stewards and
information providers.
The Government Performance and Results Act of 1996 (GPRA) requires federal agencies
to provide managers with accurate information about program results and service quality.
EPA's 2009 Performance and Accountability Report (PAR), developed in accordance
with GPRA, specifically identifies data quality as a key element of EPA's efforts to
improve management and results.
The Office of Management and Budget (OMB) Circular A-130, Management of Federal
Information Resources, authorized in part by GPRA and the Paperwork Reduction Act,
guides that federal agencies should ensure that systems maximize the usefulness of
information, minimize the burden on the public, and preserve the appropriate integrity,
1

-------
10-P-0230
usability, availability, and confidentiality of information throughout the life cycle of the
information.
EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states:
•	It is very important that PCS and NPDES data be complete, accurate, and up-to-
date, and that all users be consistent in the way they define and use various data
elements because the systems play such a central role in program management.
•	Quality is achieved through effective management and commitment. Responsive
management should provide the support and training that is necessary to achieve
good data quality; and recognize, reward, and encourage quality service and
performance.
•	Quality must be tracked and performance of the program evaluated at regular
intervals. Data quality must be measurable so that the causes of poor data quality
can be identified and corrected.
•	In order for the quality of PCS data to be "good," the data must meet quality
targets for timeliness, accuracy, completeness, and consistency. For example, a
quality target for accuracy is that 95 percent of the data elements in the system of
record be identical to the source document. The failure to meet these targets is
especially serious when the data elements affect public access.
•	Ninety-five percent of the data elements entered into PCS should be identical with
those reported on the DMR permit or other input document.
EPA's PCS Policy Statement states that to achieve national consistency and uniformity in
the NPDES program, the required data in PCS must be complete and accurate.
The Office of Compliance is responsible for identifying, preventing, and reducing
noncompliance and environmental risks by providing leadership in program planning,
priority setting, and expanding the use of compliance data. In this regard, the office's
Enforcement Targeting and Data Division (ETDD) supports EPA's national enforcement
and compliance information and reporting needs and manages the ECHO Internet site.
Scope and Methodology
We coordinated with the OIG to select major high-risk ecosystems and corresponding
discharge facilities as the population for our testing. We and the OIG selected the major
ecosystems and facilities, in part, based on the EPA's 2006-2011 Strategic Plan, which
identifies high-risk ecosystems in the United States. For the selected ecosystems and
facilities, we reviewed 2008 calendar year DMR data extracted from PCS and ICIS-
NPDES.
2

-------
10-P-0230
We performed this audit in accordance with Generally Accepted Government Auditing
Standards issued by the Comptroller General of the United States. These standards
require that we plan and perform the audit to obtain sufficient and appropriate evidence to
provide a reasonable basis for their findings and conclusions based on the audit
objectives. We believe the evidence obtained provides a reasonable basis for our findings
and conclusions.
To assist in selecting the ecosystems, we used the EPA Website, which provided a map
of watersheds and Hydrologic Unit Codes, as well as the ECHO map feature. We then
selected a judgmental sample of facilities supporting each ecosystem. For these
facilities, we compared data elements reported in ECHO to related DMR records. We
included the following types of data elements in our review:
•	Facility name
•	Permit number and approval
•	Pollutant levels
•	Facility status (e.g., major or minor)
•	Specific DMR data (e.g., types of pollutants)
We focused our review on the more critical data elements, such as pollutant levels and
facility status, rather than less critical elements, such as facility addresses and phone
numbers. Our testing was based on a judgmental sample, and the results are not
statistically valid and should not be projected to other States or to all facilities in the
States we visited (for the State of Florida, we did test the entire population of 41 non-
Federal managed facilities).
We conducted the following site visits:
Table 1: State Environmental Offices Visited During the Review
State
Visited
ECHO Source
System Used
Dates of Visit
Ecosystem
No. of Facilities
Reviewed
Maryland
ICIS-NPDES
November 2- 10, 2009
Chesapeake Bay
50 (sample)
Indiana
ICIS-NPDES
November 16-20, 2009
Great Lakes
50 (sample)
Michigan
PCS
December 7- 11, 2009
Great Lakes
50 (sample)
Florida
PCS
January 11-15, 2010
South Florida
42 (entire population)
Source: KPMG-compiled data
3

-------
10-P-0230
Findings
EPA mandates that data elements reported to the public through the ECHO Website have
a 95 percent accuracy rate. During this review, we found a 91.5 percent data accuracy
rate for key data elements entered into two primary ECHO source systems - the legacy
PCS and newer ICIS-NPDES. Although the 91.5 percent data quality rate is close to
EPA's goal, EPA and the State Environmental Offices can make additional
improvements to further increase data quality. Specifically, we identified several types of
data quality exceptions and a processing issue in the State of Michigan that affected the
quality of data ultimately presented through ECHO. We cite them on the following
pages.
identified Data Quality Exceptions
During each of our four site visits to State environmental offices, we found four types of
data element exceptions, as follows:
•	Data elements were reported in ECHO but not in DMRs. In these cases, we noted
that data elements were reported through ECHO but the supporting source DMRs
did not reflect the same data element information.
•	Data elements were reported in DMRs but not in ECHO. In these cases we noted
that data elements were reported in the source DMRs (e.g., pollutant values), but
the same data element information was not reported through ECHO.
•	Data elements were inaccurately input into ECHO source systems (PCS or ICIS-
NPDES) from DMRs. For example, at one facility, the pollutant phosphorous had
a value of 1.7 in ECHO but a value of 0.7 on the supporting DMR. At another
facility, the pollutant nitrogen had a value of 2.64 in ECHO but a value of 2.22 on
the DMR.
•	Data elements had missing DMRs. In these cases, we were not able to review the
applicable source DMRs because they were not available at the state
environmental offices.
4

-------
10-P-0230
The following table summarizes the data quality exceptions by state.
Table 2: Data Quality Exceptions











Data













Data


Data


Elements













Elements


Elements


Inaccurately


Data







No. of


in ECHO


on DMRs


Entered Into


Elements


Total Data




Data
Elements


Not
Supported


Not
Entered


Source
Systems


With
Missing


Elements
With

Exception
%
State

Tested


by DMRs


Into ECHO


From DMRs


DMRs


Exceptions

(By State)
Maryland
6,199
94
386
174
133
787
12.7%
Indiana
8,850
297
16
16
8
337
3.8%
Florida
12,590
62
501
85
565
1213
9.6%
Totals
27,639
453
903
275
706
2,337
8.5%
Source: KPMG-compiled results from documentation review
Michigan had data quality exceptions in the same four categories. However, Michigan's
results are not included in Table 2 because we were not able to fully quantify the
exceptions. Michigan had a data processing issue that resulted in many data elements not
being displayed in ECHO. See the Michigan Data Processing section later in this report
for details.
We provided detailed data quality exceptions to the visited State environmental offices,
in which we identified the facility names and types of exceptions identified at each
facility.
Factors That Contributed to Data Quality Exceptions
No Automated Data Quality Processes
The DMR data entry process is a manual-oriented process for many State environmental
offices, which significantly elevates the risk of data exceptions. The EPA 2009 PAR
addresses this issue in part by stating, "data quality has been improving and will continue
to improve as existing data entry requirements and procedures are being reevaluated and
communicated with data entry practitioners." The PAR further states that in FY 1999
EPA acknowledged PCS as a weakness and recognized that the Agency needed to
revitalize or replace PCS to ensure complete and accurate water data. EPA has been
continually enhancing DMR data entry and quality review processes, including efforts to
(1) convert from PCS to ICIS-NPDES, (2) increase use of the Interim Data Exchange
Format (IDEF) quality review process, and (3) implement eDMR and NetDMR
automation capabilities. According to EPA officials, competing resources have limited
full implementation of these enhancements.
5

-------
10-P-0230
Regarding the conversion from PCS to ICIS-NPDES, the Office of Enforcement and
Compliance Assurance (OECA) Office of Compliance, with support from the Office of
Water and the Office of Environmental Information, has been working with States and
regions to develop data requirements, design the new system, test the developed software,
and prepare PCS data for migration. Now 28 States, 2 tribes, and 10 territories are using
ICIS-NPDES instead of PCS. The full migration from PCS to ICIS-NPDES is scheduled
for completion in FY 2013. These and other automation efforts are critical to improve
data quality. The Environmental Information Exchange Network estimates that
"electronic DMR reporting has the highest documented return on investment of any other
EPA data automation project - implementation by all or most NPDES facilities could
save industry, states, and EPA around $100 million per year."1
Using eDMR and NetDMR capabilities will assist with Full Batch State conversion.
OECA can accelerate the Full Batch State conversion by using the Environmental
Information Exchange Network as a preferred method of transmitting data to the Agency.
OECA's FY 2010 & 2011 National Program Manager Guidance does not contain any
language or plans for using the Environmental Information Exchange Network. In this
regard, more liaison with respective EPA offices would help ensure Agency grants given
to States would more effectively align with EPA data.
No Consistent State Quality Review Processes
Closely linked to the above factor, ECHO data quality efforts are reliant upon resource
capabilities in EPA and the State environmental offices. During our site visits, we noted
that the States' data quality resources vary. For example, the State of Maryland has one
dedicated person focused on the ECHO data quality program, while the State of Indiana
had four data entry personnel and three data quality reviewers. As noted earlier in Table
2, Indiana had the smallest number of exceptions.
EPA's IDEF function provides the State environmental offices with the capability to
audit their submission of ECHO data for completeness and accuracy, but the States are
not consistently reviewing IDEF. In fact, State environmental office officials confirmed
that some of the data quality findings identified during this review would not have been
identified through existing EPA data quality processes, such as IDEF.
No Standardized DMR Form
State environmental offices use various types of DMR forms for data entry. Some of
these offices use their own State-specific reporting forms, and forms can vary within the
same State.
EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states:
1 http://www.exchangenetwork.net/benefits/NetDMR SuccessStorv.pdf
6

-------
10-P-0230
All DMRs submitted to EPA Regional Offices (including DMRs
submitted by NPDES States for EPA entry into PCS) must be
preprinted using the Office of Management and Budget (OMB)
approved DMR form. NPDES States directly using PCS are not
required to use the OMB-approvedform; however, its use is strongly
encouraged.
Although some similarity exists between the various input forms, the lack of full
standardization increases the risk of data quality exceptions. Enhanced automation with
ICIS-NPDES, eDMR, and NetDMR could improve this issue. There is a standard form
for entering NPDES data (OMB Form Number 2040-0004, National Pollutant Discharge
Elimination System (NPDES) Discharge Monitoring Report), but we found that not all
State environmental offices were consistently using this form.
No Documentation of Data Entry
State environmental office officials informed us their data quality personnel sometimes
will input data into PCS or ICIS-NPDES but not record the information in supporting
documentation (e.g., DMRs, permits). The PCS Quality Assurance Manual states that the
regions should work closely with their NPDES States using PCS to address similar data
entry problems with State-issued NPDES permits. The primary reason for not updating
the source documentation was that the State officials believed the changes were so minor
they would not affect overall data quality.
No Requirements to Report Minor Facility Data
Existing legislation does not require regulated facilities to send EPA all minor facility
data elements displayed in ECHO. Only submission of major facility data elements is
required.
EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states:
If States and Regions wish to enter NPDES data beyond what has been required,
they may do so. For example, if States want to enter DMR data for minor
facilities, the options is available in PCS and the States may use it as their
resources allow.
According to EPA officials, a proposed rule is being developed to require States and
regulated facilities to provide additional minor facility data. EPA expects to have the rule
finalized during FY 2012.
Michigan Data Processing
During our site visit to Michigan, we identified a data processing error where DMR data
for 18 facilities were transmitted to PCS but the data was not completely populated into
ECHO. Specifically, ECHO did not completely receive PCS data for facilities with
7

-------
10-P-0230
multiple discharge points. EPA had previously identified this issue, as the ECHO Data
Alerts Website (http://www.epa-echo.uov/echo/data alerts.html). last updated on July 6,
2009, identified the issue and stated that EPA is working with the State of Michigan to
resolve the issue. ETDD determined that software code used to pull data elements from
PCS into ECHO was not pulling all necessary data. ETDD developed, tested, and
implemented new software code to correct the issue. The processing error contributed to
a significant number of 2008 data quality exceptions for Michigan. As noted in the
Permit Compliance System (PCS) Quality Assurance Guidance Manual, PCS data must
be complete and accurate to assist EPA with program management activities.
While these efforts correct the identified issues with the Michigan data, ultimately non-
compliance with prescribed EPA System Management Life Cycle (SMLC) Guide
contributed to the code being placed into production without testing. EPA SMLC
guidance states that the Test Subphase results in proof that the system satisfies the
requirements. As such, internal controls over system testing would have identified the
error in the Michigan programming logic before it was placed into production.
Data Quality Exceptions Affect EPA's Representation of Data
The exception rates increase the risk that ECHO data presented to internal EPA
stakeholders and the public is not accurate. EPA's Permit Compliance System (PCS)
Quality Assurance Guidance Manual notes that PCS and ICIS-NPDES data must be
complete and accurate to assist EPA with program management activities. Further,
inaccurate ECHO data could misrepresent facility information, so data accuracy is
significant for EPA and State environmental offices in their roles as good public
stewards.
EPA officials informed us that the Agency partially mitigates the risk of data exceptions
by relying on the public and facilities to notify the Agency of inaccurate data. The EPA
2009 PAR emphasizes this point by stating, "an important means for catching potential
data errors is to obtain stakeholder feedback. EPA has a formal error correction process
in place to investigate and react to public notification of errors in its publications." While
relying on the public serves as a detective control, it is still incumbent upon EPA and the
State environmental offices, as good public stewards, to have more effective preventive
controls in place to better ensure data quality.
Recommendations
We recommend that the Director, Office of Compliance, within the Office of
Enforcement and Compliance Assurance:
1. Establish a management control structure to facilitate PCS to ICIS-NPDES
conversion. The management control structure should support plans to meet
milestone dates.
8

-------
10-P-0230
2.	Insert language that requires the use of the Environmental Information Exchange
Network in the National Program Manager (NPM) Guidance to assist with PCS to
ICIS-NPDES conversion for FY 2012.
3.	Gather key leading data quality practices used by the States and share these
practices with all States.
4.	Provide reminders to EPA regional offices and State environmental offices to use
the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for
State-specific reporting requirements), EPA should require completely reporting
DMR data by using standardized Form 2040-0004 or alternative entry forms.
Further, EPA should remind the States of the importance of documenting changes
made to data elements after entry into the source systems.
5.	Complete new rules that require States to report minor facility data.
6.	Place a notice on the EPA ECHO Website stating that ECHO does not contain all
minor facility data. The wording and placement of the notice should be designed
to help provide the public with transparency into the actual data elements
maintained in ECHO.
7.	Conduct a review of the procedures used to test ICIS-NPDES programming code
before it is placed into production.
Agency Comments and OIG Evaluation
EPA agreed with the report findings. EPA felt that it has extensive, documented
procedures in place to test ICIS-NPDES programming code before it is placed into
production. Management cited that they follow best practices used by the information
technology industry as well as EPA system life cycle policies and procedures and that
their primary contractor is certified by the Software Engineering Institute to be Level 3
on the Capability Maturity Model Integration scale. Management also stated their
procedures have been refined and improved over the years, and, because of this, new and
revised software is typically implemented into production with few, if any, defects.
We agree that management should use industry best practices whenever feasible and
adopt an industry recognized method for controlling system development activities.
However, we believe as part of an effective internal control structure management must
continually review these practices to ensure they operate as intended. For the State of
Michigan, EPA recognized the error in the State's ECHO data and it took the Agency
close to one year to make changes to the programming code to correct the problem. As
such, we believe it is incumbent upon management to conduct a root cause analysis to
identify the causes for the programming code error and update the testing procedures to
prevent similar errors from happening in the future.
9

-------
10-P-0230
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Action Official
Establish a management control structure to
facilitate PCS to ICIS-NPDES conversion.
The management control structure should
support plans to meet milestone dates.
Insert language that requires the use of the
Environmental Information Exchange
Network in the National Program Manager
(NPM) Guidance to assist with PCS to ICIS-
NPDES conversion for FY 2012.
Planned
Completion
Date
Claimed
Amount
Agreed To
Amount
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
9 Gather key leading data quality practices
used by the States and share these practices
with all States.
9 Provide reminders to EPA regional offices
and State environmental offices to use the
standard Form 2040-0004 for data entry.
Where this is not feasible (e.g., for State-
specific reporting requirements), EPA should
require completely reporting DMR data by
using standardized Form 2040-0004 or
alternative entry forms. Further, EPA should
remind the States of the importance of
documenting changes made to data elements
after entry into the source systems.
9 Complete new rules that require States to
report minor facility data.
Place a notice on the EPA ECHO Website O
stating that ECHO does not contain all minor
facility data. The wording and placement of
the notice should be designed to help provide
the public with transparency into the actual
data elements maintained in ECHO.
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
Conduct a review of the procedures used to test
ICIS-NPDES programming code before it is placed
into production.
O Director, Office of
Compliance, Office of
Enforcement and
Compliance Assurance
O = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is undecided with resolution efforts in progress
10

-------
10-P-0230
Appendix A
^tD SX
* A \
I® i
MEMORANDUM
SUBJECT: Response to the Office of the Inspector General Draft Evaluation Report, "ECHO
Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate
with Additional Improvements" Project Number OMS-FY09-00011, dated
July 28, 2010
FROM:	Cynthia Giles /s/
Assistant Administrator
TO:	Rudolph M. Brevard
Director, Information Resources Management Assessments
Office of Mission Systems
Office of Inspector General
Thank you for the opportunity to review and comment on the draft evaluation report
entitled, "ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate
with Additional Improvements," (Report) Project Number OMS-FY09-00011. This Report
focuses on improving the quality and transparency of the enforcement and compliance
information EPA provides to the public through our Enforcement and Compliance History Online
(ECHO) website.
OECA agrees that providing high quality transparent data to the American public is a
fundamental responsibility of the government. Recently, OECA has taken several steps to
expand transparency of enforcement and compliance information through ECHO with the release
of the Clean Water Act Annual Noncompliance 2008 Report and the enhanced Air, Water and
Waste Resource pages.
Additionally, OECA is expanding our efforts to ensure the accuracy of the information in
ECHO. A key new resource we are providing the public is consolidated information by program
area (Clean Water, Clean Air and Hazardous Waste). Included on these pages are reports such as
the "2009 Annual CAA Majors Report" and the "2009 State Review Framework Frozen National
Report". Prior to updating these annual reports, we provide the States an opportunity to verify the
data and provide additional information. OECA is working with our State partners to ensure that
the public receives the best information available.
Agency Response
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
August 20, 2010	office of
ENFORCEMENT AND
COMPLIANCE ASSURANCE
11

-------
10-P-0230
Report Recommendation 1:
• Establish a management control structure to facilitate PCS to ICIS-NPDES
conversion. The management control structure should support plans to meet
milestone dates.
OECA Response:
OECA agrees with the reports attention to strong management controls to ensure a
continued quality conversion from PCS to ICIS-NPDES. OECA has strong management
controls in place to facilitate the PCS to ICIS-NPDES conversion. There are three
control structures used for this purpose including an Integrated Project Team using
established best practices for IT project management, the Environmental Information
Exchange Network (EN) governance structure which controls electronic exchange of
data, and the permanent EPA management team of the Office of Compliance. In addition
there are state-specific Data Migration Workgroups which are formed in advance of each
state's conversion; to work through data migration issues and ensure readiness for the
conversion process.
The ICIS-NPDES Batch Integrated Project Team (IPT) was formed to bring
together management and staff with the necessary business and technology skills from
EPA (OECA, OEI, and EPA Regions) and our state partners through the Environmental
Council of States to implement the successful transition from PCS to ICIS-NPDES. The
IPT serves as the primary management control structure for the conversion. The
members oversee development processes and software, including finalization and
implementation of the NPDES schemas for the electronic flow of data from state NPDES
systems to ICIS-NPDES. This IPT was initially convened in March of 2007 to move
forward the electronic submission of Discharge Monitoring Report data from state
systems to ICIS-NPDES. This was successfully completed in May of 2008. The ICIS-
NPDES Batch IPT was reconvened in October of 2009 to move forward with the
electronic transmittal of the remaining NPDES data families and complete the conversion
of the remaining 22 states to ICIS. There are currently 38 states and 6 regions
participating on the IPT. Part of the responsibility of the IPT is to set project milestones
and regularly review them. To date, the work of the project remains on schedule. Pilot
state(s) are drawn from the IPT (as volunteers) to test out the schemas and migrate to
ICIS-NPDES per the agreed upon schedule.
The governance structure of the Environmental Information Exchange Network
(EN) requires that all software schema to be used for flowing data across the EN,
including the products developed under the leadership of the IPT, be reviewed and
approved by the Network Technology Group (NTG) which is made up of information
technology specialists that have expertise in data conversion, exchange, and publishing.
The NTG operates under the auspices of the Network Operations Board (NOB), a group
that is responsible for the day-to-day operations of the EN and its operational policies and
procedures. Any issues that are not resolved at the NOB level may be elevated to the
12

-------
10-P-0230
higher Exchange Network Leadership Council. All three groups have EPA, state, and
tribal partners and decisions are made collaboratively.
Within the Office of Compliance, the activities of the IPT and project milestones
are reported to Office of Compliance executives at biweekly management meetings.
Both the Division Director and the Office Director and their deputies review
accomplishments, address issues, and ensure that funding and other resources are
available to assist states with conversion of their data.
Finally, an ICIS-NPDES Data Migration Workgroup is initiated each time there is
a group of states scheduled to migrate from legacy PCS to ICIS-NPDES. Participants
include management and staff from each migrating state, each state's respective region,
and OECA. Work must be done by each migrating state to ready their data in legacy PCS
for the migration to ICIS-NPDES. The purpose of holding the workgroup is to walk the
state and regional participants through the data migration process, explain the data
differences between legacy PCS and ICIS-NPDES, walk through the mapping rules for
data being migrated from legacy PCS to ICIS-NPDES, review test data migration results
from each test data migration run, and answer questions concerning the process, the data
and ICIS-NPDES. Work groups typically last a year and the status of activities is
reported to the IPT and the management chain within the Office of Compliance.
Report Recommendation 2:
• Insert language that requires the use of the Environmental Information
Exchange Network in the National Program Manager (NPM) Guidance to
assist with PCS to ICIS-NPDES conversion for FY 2012
OECA Response:
OECA currently has guidance language regarding the use of the Environmental
Information Exchange Network in its "FY-2011 OECA National Program Manager
(NPM) Guidance" dated April 30, 2010 that reads:
"Implement the use of NetDMR or other e-DMR tools for the electronic transfer
of Discharge Monitoring Reports (DMR) to ICIS-NPDES, supported by use of the
National Environmental Information Exchange Network (Exchange Network), by
all of their NPDES permitted facilities. Administrator Jackson , in her July 7,
2009 memorandum, "Achieving the Promise of the National Environmental
Information Exchange Network," requested cross-Agency commitment to make
the Network the preferred way EPA, States, Tribes, and others share and
exchange data while supporting an aggressive timetable to phase out other data
submission and exchange methods. EPA Regions need to demonstrate leadership
in implementing this strategic vision for the Agency." (See page 29.)
OECA commits to continuing to provide this information in future National
Program Manager Guidance documents.
13

-------
10-P-0230
Report Recommendation 3:
•	Gather key leading data quality practices used by the states and share these
practices with all states.
OECA Response:
OECA agrees that sharing of best practices across the states will improve the
overall quality of information available to EPA and the American public. OECA is
committed to assuring data quality. It will, as permitted under the Paperwork Reduction
Act, seek to identify a few states that are willing to share and post their quality control
and quality assurance methods for achieving high quality enforcement and compliance
information. OECA also commits to using National Meetings as forums for panel and
other discussions to allow states to share their data quality best practices.
In addition, there are a number of existing examples of states sharing and using
data quality practices for both ICIS and PCS. In 2009 at the ICIS National Meeting,
regions and states shared their Discharge Monitoring Report (DMR) data quality
practices. States also shared their Quality Assurance and Quality Control (QA/QC)
processes as they related to the standard reports available in ICIS. These processes
involved quality control of data for limits, overdue schedules, and violations. The
Quarterly Non-Compliance Report is also used by states as a data quality tool. On
occasion, data quality issues are addressed on the ICIS monthly user calls.
For PCS, states use the "Dummy" audit capability to review data and reject
transactions before transferring the data into the system. They follow up by reviewing
the rejected transactions from the "Live" audit reports.
Report Recommendation 4:
•	Provide reminders to EPA regional offices and state environmental offices to
use the standard Form 2040-0004for data entry. Where this is not feasible (e.g.,
for state-specific reporting requirements), EPA should require completely
reporting DMR data by using standardized Form 2040-0004 or alternative entry
forms. Further, EPA should remind the states of the importance of
documenting changes made to data elements after entry into the source systems.
OECA Response:
OECA agrees with the report that we should strive to better ensure states and
regions are well informed on the proper forms and procedures for collecting and
submitting data to EPA. OECA will take every available opportunity to remind EPA
regional offices and states to use the Standard Form 2040-0004. This will include
mechanisms such as the quarterly ICIS newsletter, monthly user calls, and electronic mail
14

-------
10-P-0230
messages sent to individuals on the current ICIS list serve. Please note, however, that
this form is also called Standard Form 3320-1 and both numbers appear on the physical
document. Because our current user community and documentation refer to SF-3320-1
and we do not wish to confuse submitters, we will continue to refer to SF-3320-1 in our
communications.
Additionally, all the State and Federal electronic DMR systems are 3320-1
compliant.
Report Recommendation 5:
•	Complete new rules that require states to report minor facility data.
OECA Response:
OECA strongly agrees with the recommendation to complete the rulemaking
process requiring the submission of minor facility data. OECA is currently developing a
rule which will significantly improve the NPDES data quantity and quality nationally. A
key component of this rulemaking is the requirement for electronic reporting of certain
NPDES information from the permittees, including non-major facilities, to states and
EPA. This information is currently required to be reported by the permittees to the states,
but there is no regulatory requirement for states to provide that specific information to
EPA. This NPDES information is generally obtained on paper forms rather than
electronically, although some states have developed electronic reporting tools.
Therefore, this rule may likely change the mode by which the permittees report, without
requiring additional information to be collected and submitted by the permittees, and
reducing the burden on state programs caused by the handling of paper forms..
Key EPA management decisions regarding the specifics of the rulemaking are
still pending, awaiting ongoing detailed analyses. However, this proposed requirement
for electronic reporting of NPDES information from the permittees may likely include
information such as discharge monitoring reports (DMRs), program reports, notices of
intent to discharge, etc. As a supplement to that information from permittees, other
information will likely be required to be reported by the states to EPA regarding the
states' implementation activities (such as inspections, violation determinations,
enforcement actions, and permit issuance) for major and non-major facilities as well as
for NPDES subprograms (e.g., pretreatment, biosolids, CAFOs, etc.).
EPA currently projects issuance of this proposed rule in April 2011. Following a
comment period, EPA expects to issue a final rule regarding this matter circa September
2012.
Report Recommendation 6:
•	Place a notice on the EPA ECHO website stating that ECHO does not contain
all minor facility data. The wording and placement of the notice should be
15

-------
10-P-0230
designed to help provide the public with transparency into the actual data
elements maintained in ECHO.
OECA Response:
OECA agrees that the public needs to be aware that not all minor facility data is
currently represented in ECHO. OECA does provide several notices to the public on the
limitations of the data provided by ECHO. OECA has recently added several new
features that allow ECHO users to find additional detailed information about the quality
and completeness of information, including an interactive map that shows the data entry
percentage for minor discharge monitoring data in each state. However, this information
can be difficult to find, and OECA agrees this information can be better organized. For
example we currently use grey text to indicate that there are data caveats. We will
enhance this grey text so it is more recognizable to ECHO users. OECA will evaluate
each area where the existing notice is displayed and determine if it needs to be enhanced.
OECA will also evaluate its ECHO site as a whole and determine if additional notices
need to be added. OECA regularly enhances the ECHO website, and will continue its
practice of providing the public with caveats concerning the data.
Report Recommendation 7:
• Develop and implement procedures to test ICIS-NPDES programming code
before it is placed into production.
OECA Response:
OECA has extensive, documented procedures in place to test ICIS-NPDES
programming code before it is placed into production. It follows best practices used by
the information technology industry as well as EPA system life cycle policies and
procedures. The prime contractor for the ICIS system is certified by the Software
Engineering Institute (SEI) to be Level 3 on the Capability Maturity Model Integration
(CMMI) scale. This is the same high level that is required for Department of Defense
contractors. This certification requires the contractor to follow repeatable, standard
processes for verification of all products. In addition, OECA completes and documents
all items that are required by the EPA National Computing Center's Automated
Deployment Checklist prior to each software release.
As part of the standard life cycle process, procedures for thoroughly testing the
software for ICIS-NPDES were implemented during the 2004/2005 timeframe. This was
the period during which the software for ICIS-NPDES was being initially developed and
tested prior to its release into production. These procedures have been refined and
improved over the years, and, because of this, new and revised software is typically
implemented into production with few, if any, defects. The software goes through Unit
Testing, Functional Testing, and End-to-End or System Integration Testing by the ICIS
contractor to ensure it both meets stated requirements and executes correctly before it is
turned over to EPA and the user community for User Acceptance Testing. For new
16

-------
10-P-0230
software relating to major new capability, the software also goes though Beta Testing
with experts on the ICIS Project Team prior to User Acceptance Testing. This ensures
that as many defects as possible are found and fixed prior to user community testing. All
testing is conducted according to Test Plans which include documenting test cases and,
upon execution, the results of executing the test cases. Problems identified during each
test phase are corrected and retested as necessary prior to the software being released into
Production. Testing results are reviewed during the Bi-weekly Project Management
meetings held between EPA and the ICIS contractor.
17

-------
10-P-0230
Appendix B
Distribution
Office of the Administrator
Assistant Administrator for Enforcement and Compliance Assurance
Assistant Administrator for Environmental Information and Chief Information Officer
Regional Administrator, Region 3
Regional Administrator, Region 4
Regional Administrator, Region 5
Director, Office of Compliance, Office of Enforcement and Compliance Assurance
Acting Director, Office of Information Collection, Office of Environmental Information
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Agency Follow-up Coordinator, Office of Enforcement and Compliance Assurance
Inspector General
18

-------