s A Iffil U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Audit Report ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements Project No. 10-P-0230 September 22, 2010 ------- Abbreviations DMR Discharge Monitoring Report ECHO Enforcement Compliance and History Online EPA U.S. Environmental Protection Agency GPRA Government Performance and Results Act of 1996 ICIS-NPDES Integrated Compliance Information System - National Pollutant Discharge Elimination System IDEF Interim Data Exchange Format OECA Office of Enforcement and Compliance Assurance OIG Office of Inspector General OMB Office of Management and Budget PCS Permit Compliance System ------- U.S. Environmental Protection Agency 10-P-0230 9 \ Office of Inspector General September 22,2010 I fi At a Glance Why We Did This Review We sought to assess the quality of key data elements reported through the Enforcement Compliance and History Online (ECHO) Website. KPMG, LLP, performed the review. Background ECHO provides a single source of detailed compliance history of U.S. Environmental Protection Agency (EPA)- regulated facilities. EPA developed ECHO to provide the public compliance and inspection data under its environmental programs, as well as demographic data of the surrounding areas. This report focuses on the quality of data elements entered into ECHO source systems. For further information, contact our Office of Congressional, Public Affairs and Management at (202) 566-2391. To view the full report, click on the following link: www.epa.aov/oia/reports/2010/ 20100922-10-P-0230.pdf Catalyst for Improving the Environment\ ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements What KPMG Found EPA mandates that data elements reported to the public through the ECHO Website have a 95 percent accuracy rate. KPMG found a 91.5 percent data accuracy rate for key data elements entered into two primary ECHO source systems: the legacy Permit Compliance System (PCS) and the newer Integrated Compliance Information System - National Pollutant Discharge Elimination System (ICIS-NPDES). Although the 91.5 percent data quality rate is close to EPA's goal, EPA and the State environmental offices could take additional steps to increase the quality of data reported through the ECHO Website. What KPMG Recommends KPMG made several recommendations to the Assistant Administrator for Enforcement and Compliance Assurance. These included: Establishing an internal control structure to help manage the conversion of PCS to ICIS-NPDES. Including language in the National Program Manager Guidance requiring the use of the Environmental Information Exchange Network for reporting data to EPA. Developing a plan to share data quality best practices implemented at State environmental offices with all States. Completing new rules requiring reporting ECHO data for minor facilities and notifying ECHO Website users that the site does not contain data on minor facilities. Reviewing procedures used to test ICIS-NPDES programming code before it is placed into production. The Agency generally agreed with the report findings. EPA felt it has extensive, documented procedures in place to test ICIS-NPDES programming code before it is placed into production. Tests disclosed that programming errors directly resulted in incomplete data on the ECHO Website and, as such, management should review these processes to prevent future occurrences. The Agency's response is included in Appendix A. ------- * o \ IŪ I V PRO^0 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 THE INSPECTOR GENERAL September 22, 2010 MEMORANDUM SUBJECT: ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements Report No. 10-P-0230 FROM: Arthur A. Elkins, Jr. Inspector General TO: Cynthia Giles Assistant Administrator for Enforcement and Compliance Assurance This is the report on the subject audit conducted by KPMG, LLP, on behalf of the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings KPMG LLP identified and corrective actions recommended. This report represents the conclusions of KPMG and does not necessarily represent the final position of the OIG on the subjects reported. Final determinations on matters in this report will be made by EPA management in accordance with established audit resolution procedures. The estimated cost of this audit, which includes contract costs and OIG contract management oversight, is $331,361. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 90 calendar days. You should include a corrective actions plan for agreed-upon actions, including milestone dates. Your response will be posted on the OIG's public Website, along with our comments on your response. Your response should be provided in an Adobe PDF file that complies with the accessibility requirements of section 508 of the Rehabilitation Act of 1973, as amended. If your response contains data that you do not want to be released to the public, you should identify the data for redaction. We have no objections to the further release of this report to the public. This report will be available at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact Patricia H. Hill at (202) 566-0894 or hill.patricia@epa.gov. or Rudolph M. Brevard at (202) 566-0893 or brevard.rudv@epa.gov. ------- September 22, 2010 SUBJECT: Audit Report: ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements THRU: Arthur A. Elkins, Jr. Inspector General TO: Cynthia Giles Assistant Administrator Office of Enforcement and Compliance Assurance This memorandum is to inform the U.S. Environmental Protection (EPA) of critical findings by KPMG, LLP, that require management action regarding the improvement of the data quality for the Enforcement Compliance History Online Website. If you or your staff have any questions regarding this report, please contact the U.S. Environmental Protection Agency, Office of Inspector General, point of contact identified on the report transmittal memorandum. ------- ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements 10-P-0230 Table of C Purpose 1 Background 1 Scope and Methodology 2 Findings 4 Identified Data Quality Exceptions 4 Factors That Contributed to Data Quality Exceptions 5 No Automated Data Quality Processes 5 No Consistent State Quality Review Processes 6 No Standardized DMR Form 6 No Documentation of Data Entry 7 No Requirements to Report Minor Facility Data 7 Michigan Data Processing 7 Data Quality Exceptions Affect EPA's Representation of Data 8 Recommendations 8 Agency Comments and OIG Evaluation 9 Status of Recommendations and Potential Monetary Benefits 10 Appendices A Agency Response 11 B Distribution 18 ------- 10-P-0230 Purpose The U.S. Environmental Protection Agency (EPA) Office of Inspector General (OIG) initiated this review to evaluate the quality of data reported through the EPA Enforcement Compliance and History Online (ECHO) Website. KPMG LLP (KPMG) completed the review in the following two phases: Phase 1: Evaluate the automated process EPA uses to display ECHO data transmitted from select source systems. We reviewed the automated processes for the legacy Permit Compliance System (PCS) and the newer Integrated Compliance Information System - National Pollutant Discharge Elimination System (ICIS-NPDES). We provided the Phase 1 results in OIG report, ECHO Data Quality Audit - Phase I Results: The Integrated Compliance Information System Needs Security Controls to Protect Significant Non-Compliance Data, Report No. 09-P-0226, August 31, 2009. Phase 2: Test the quality of select PCS and ICIS-NPDES data elements against Discharge Monitoring Reports (DMRs), which are the primary source documents for PCS and ICIS-NPDES data. Background ECHO provides a single source of detailed and up-to-date environmental compliance history of EPA-regulated facilities. EPA developed ECHO in partnership with State environmental offices to provide compliance and inspection data under air, water, and hazardous waste programs, as well as demographic data of the surrounding areas. Since ECHO'S November 2002 launch, hundreds of thousands of citizens, government officials, investors, and staff at regulated facilities and companies have asked over 3,000,000 questions about the environmental records of the more than 800,000 facilities. EPA and the state environmental offices need to maintain a high level of ECHO data quality to: 1) provide accurate and complete information to the public, States, facilities, and other stakeholders, and 2) maintain their reputations as effective public stewards and information providers. The Government Performance and Results Act of 1996 (GPRA) requires federal agencies to provide managers with accurate information about program results and service quality. EPA's 2009 Performance and Accountability Report (PAR), developed in accordance with GPRA, specifically identifies data quality as a key element of EPA's efforts to improve management and results. The Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, authorized in part by GPRA and the Paperwork Reduction Act, guides that federal agencies should ensure that systems maximize the usefulness of information, minimize the burden on the public, and preserve the appropriate integrity, 1 ------- 10-P-0230 usability, availability, and confidentiality of information throughout the life cycle of the information. EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states: It is very important that PCS and NPDES data be complete, accurate, and up-to- date, and that all users be consistent in the way they define and use various data elements because the systems play such a central role in program management. Quality is achieved through effective management and commitment. Responsive management should provide the support and training that is necessary to achieve good data quality; and recognize, reward, and encourage quality service and performance. Quality must be tracked and performance of the program evaluated at regular intervals. Data quality must be measurable so that the causes of poor data quality can be identified and corrected. In order for the quality of PCS data to be "good," the data must meet quality targets for timeliness, accuracy, completeness, and consistency. For example, a quality target for accuracy is that 95 percent of the data elements in the system of record be identical to the source document. The failure to meet these targets is especially serious when the data elements affect public access. Ninety-five percent of the data elements entered into PCS should be identical with those reported on the DMR permit or other input document. EPA's PCS Policy Statement states that to achieve national consistency and uniformity in the NPDES program, the required data in PCS must be complete and accurate. The Office of Compliance is responsible for identifying, preventing, and reducing noncompliance and environmental risks by providing leadership in program planning, priority setting, and expanding the use of compliance data. In this regard, the office's Enforcement Targeting and Data Division (ETDD) supports EPA's national enforcement and compliance information and reporting needs and manages the ECHO Internet site. Scope and Methodology We coordinated with the OIG to select major high-risk ecosystems and corresponding discharge facilities as the population for our testing. We and the OIG selected the major ecosystems and facilities, in part, based on the EPA's 2006-2011 Strategic Plan, which identifies high-risk ecosystems in the United States. For the selected ecosystems and facilities, we reviewed 2008 calendar year DMR data extracted from PCS and ICIS- NPDES. 2 ------- 10-P-0230 We performed this audit in accordance with Generally Accepted Government Auditing Standards issued by the Comptroller General of the United States. These standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for their findings and conclusions based on the audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions. To assist in selecting the ecosystems, we used the EPA Website, which provided a map of watersheds and Hydrologic Unit Codes, as well as the ECHO map feature. We then selected a judgmental sample of facilities supporting each ecosystem. For these facilities, we compared data elements reported in ECHO to related DMR records. We included the following types of data elements in our review: Facility name Permit number and approval Pollutant levels Facility status (e.g., major or minor) Specific DMR data (e.g., types of pollutants) We focused our review on the more critical data elements, such as pollutant levels and facility status, rather than less critical elements, such as facility addresses and phone numbers. Our testing was based on a judgmental sample, and the results are not statistically valid and should not be projected to other States or to all facilities in the States we visited (for the State of Florida, we did test the entire population of 41 non- Federal managed facilities). We conducted the following site visits: Table 1: State Environmental Offices Visited During the Review State Visited ECHO Source System Used Dates of Visit Ecosystem No. of Facilities Reviewed Maryland ICIS-NPDES November 2- 10, 2009 Chesapeake Bay 50 (sample) Indiana ICIS-NPDES November 16-20, 2009 Great Lakes 50 (sample) Michigan PCS December 7- 11, 2009 Great Lakes 50 (sample) Florida PCS January 11-15, 2010 South Florida 42 (entire population) Source: KPMG-compiled data 3 ------- 10-P-0230 Findings EPA mandates that data elements reported to the public through the ECHO Website have a 95 percent accuracy rate. During this review, we found a 91.5 percent data accuracy rate for key data elements entered into two primary ECHO source systems - the legacy PCS and newer ICIS-NPDES. Although the 91.5 percent data quality rate is close to EPA's goal, EPA and the State Environmental Offices can make additional improvements to further increase data quality. Specifically, we identified several types of data quality exceptions and a processing issue in the State of Michigan that affected the quality of data ultimately presented through ECHO. We cite them on the following pages. identified Data Quality Exceptions During each of our four site visits to State environmental offices, we found four types of data element exceptions, as follows: Data elements were reported in ECHO but not in DMRs. In these cases, we noted that data elements were reported through ECHO but the supporting source DMRs did not reflect the same data element information. Data elements were reported in DMRs but not in ECHO. In these cases we noted that data elements were reported in the source DMRs (e.g., pollutant values), but the same data element information was not reported through ECHO. Data elements were inaccurately input into ECHO source systems (PCS or ICIS- NPDES) from DMRs. For example, at one facility, the pollutant phosphorous had a value of 1.7 in ECHO but a value of 0.7 on the supporting DMR. At another facility, the pollutant nitrogen had a value of 2.64 in ECHO but a value of 2.22 on the DMR. Data elements had missing DMRs. In these cases, we were not able to review the applicable source DMRs because they were not available at the state environmental offices. 4 ------- 10-P-0230 The following table summarizes the data quality exceptions by state. Table 2: Data Quality Exceptions Data Data Data Elements Elements Elements Inaccurately Data No. of in ECHO on DMRs Entered Into Elements Total Data Data Elements Not Supported Not Entered Source Systems With Missing Elements With Exception % State Tested by DMRs Into ECHO From DMRs DMRs Exceptions (By State) Maryland 6,199 94 386 174 133 787 12.7% Indiana 8,850 297 16 16 8 337 3.8% Florida 12,590 62 501 85 565 1213 9.6% Totals 27,639 453 903 275 706 2,337 8.5% Source: KPMG-compiled results from documentation review Michigan had data quality exceptions in the same four categories. However, Michigan's results are not included in Table 2 because we were not able to fully quantify the exceptions. Michigan had a data processing issue that resulted in many data elements not being displayed in ECHO. See the Michigan Data Processing section later in this report for details. We provided detailed data quality exceptions to the visited State environmental offices, in which we identified the facility names and types of exceptions identified at each facility. Factors That Contributed to Data Quality Exceptions No Automated Data Quality Processes The DMR data entry process is a manual-oriented process for many State environmental offices, which significantly elevates the risk of data exceptions. The EPA 2009 PAR addresses this issue in part by stating, "data quality has been improving and will continue to improve as existing data entry requirements and procedures are being reevaluated and communicated with data entry practitioners." The PAR further states that in FY 1999 EPA acknowledged PCS as a weakness and recognized that the Agency needed to revitalize or replace PCS to ensure complete and accurate water data. EPA has been continually enhancing DMR data entry and quality review processes, including efforts to (1) convert from PCS to ICIS-NPDES, (2) increase use of the Interim Data Exchange Format (IDEF) quality review process, and (3) implement eDMR and NetDMR automation capabilities. According to EPA officials, competing resources have limited full implementation of these enhancements. 5 ------- 10-P-0230 Regarding the conversion from PCS to ICIS-NPDES, the Office of Enforcement and Compliance Assurance (OECA) Office of Compliance, with support from the Office of Water and the Office of Environmental Information, has been working with States and regions to develop data requirements, design the new system, test the developed software, and prepare PCS data for migration. Now 28 States, 2 tribes, and 10 territories are using ICIS-NPDES instead of PCS. The full migration from PCS to ICIS-NPDES is scheduled for completion in FY 2013. These and other automation efforts are critical to improve data quality. The Environmental Information Exchange Network estimates that "electronic DMR reporting has the highest documented return on investment of any other EPA data automation project - implementation by all or most NPDES facilities could save industry, states, and EPA around $100 million per year."1 Using eDMR and NetDMR capabilities will assist with Full Batch State conversion. OECA can accelerate the Full Batch State conversion by using the Environmental Information Exchange Network as a preferred method of transmitting data to the Agency. OECA's FY 2010 & 2011 National Program Manager Guidance does not contain any language or plans for using the Environmental Information Exchange Network. In this regard, more liaison with respective EPA offices would help ensure Agency grants given to States would more effectively align with EPA data. No Consistent State Quality Review Processes Closely linked to the above factor, ECHO data quality efforts are reliant upon resource capabilities in EPA and the State environmental offices. During our site visits, we noted that the States' data quality resources vary. For example, the State of Maryland has one dedicated person focused on the ECHO data quality program, while the State of Indiana had four data entry personnel and three data quality reviewers. As noted earlier in Table 2, Indiana had the smallest number of exceptions. EPA's IDEF function provides the State environmental offices with the capability to audit their submission of ECHO data for completeness and accuracy, but the States are not consistently reviewing IDEF. In fact, State environmental office officials confirmed that some of the data quality findings identified during this review would not have been identified through existing EPA data quality processes, such as IDEF. No Standardized DMR Form State environmental offices use various types of DMR forms for data entry. Some of these offices use their own State-specific reporting forms, and forms can vary within the same State. EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states: 1 http://www.exchangenetwork.net/benefits/NetDMR SuccessStorv.pdf 6 ------- 10-P-0230 All DMRs submitted to EPA Regional Offices (including DMRs submitted by NPDES States for EPA entry into PCS) must be preprinted using the Office of Management and Budget (OMB) approved DMR form. NPDES States directly using PCS are not required to use the OMB-approvedform; however, its use is strongly encouraged. Although some similarity exists between the various input forms, the lack of full standardization increases the risk of data quality exceptions. Enhanced automation with ICIS-NPDES, eDMR, and NetDMR could improve this issue. There is a standard form for entering NPDES data (OMB Form Number 2040-0004, National Pollutant Discharge Elimination System (NPDES) Discharge Monitoring Report), but we found that not all State environmental offices were consistently using this form. No Documentation of Data Entry State environmental office officials informed us their data quality personnel sometimes will input data into PCS or ICIS-NPDES but not record the information in supporting documentation (e.g., DMRs, permits). The PCS Quality Assurance Manual states that the regions should work closely with their NPDES States using PCS to address similar data entry problems with State-issued NPDES permits. The primary reason for not updating the source documentation was that the State officials believed the changes were so minor they would not affect overall data quality. No Requirements to Report Minor Facility Data Existing legislation does not require regulated facilities to send EPA all minor facility data elements displayed in ECHO. Only submission of major facility data elements is required. EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual states: If States and Regions wish to enter NPDES data beyond what has been required, they may do so. For example, if States want to enter DMR data for minor facilities, the options is available in PCS and the States may use it as their resources allow. According to EPA officials, a proposed rule is being developed to require States and regulated facilities to provide additional minor facility data. EPA expects to have the rule finalized during FY 2012. Michigan Data Processing During our site visit to Michigan, we identified a data processing error where DMR data for 18 facilities were transmitted to PCS but the data was not completely populated into ECHO. Specifically, ECHO did not completely receive PCS data for facilities with 7 ------- 10-P-0230 multiple discharge points. EPA had previously identified this issue, as the ECHO Data Alerts Website (http://www.epa-echo.uov/echo/data alerts.html). last updated on July 6, 2009, identified the issue and stated that EPA is working with the State of Michigan to resolve the issue. ETDD determined that software code used to pull data elements from PCS into ECHO was not pulling all necessary data. ETDD developed, tested, and implemented new software code to correct the issue. The processing error contributed to a significant number of 2008 data quality exceptions for Michigan. As noted in the Permit Compliance System (PCS) Quality Assurance Guidance Manual, PCS data must be complete and accurate to assist EPA with program management activities. While these efforts correct the identified issues with the Michigan data, ultimately non- compliance with prescribed EPA System Management Life Cycle (SMLC) Guide contributed to the code being placed into production without testing. EPA SMLC guidance states that the Test Subphase results in proof that the system satisfies the requirements. As such, internal controls over system testing would have identified the error in the Michigan programming logic before it was placed into production. Data Quality Exceptions Affect EPA's Representation of Data The exception rates increase the risk that ECHO data presented to internal EPA stakeholders and the public is not accurate. EPA's Permit Compliance System (PCS) Quality Assurance Guidance Manual notes that PCS and ICIS-NPDES data must be complete and accurate to assist EPA with program management activities. Further, inaccurate ECHO data could misrepresent facility information, so data accuracy is significant for EPA and State environmental offices in their roles as good public stewards. EPA officials informed us that the Agency partially mitigates the risk of data exceptions by relying on the public and facilities to notify the Agency of inaccurate data. The EPA 2009 PAR emphasizes this point by stating, "an important means for catching potential data errors is to obtain stakeholder feedback. EPA has a formal error correction process in place to investigate and react to public notification of errors in its publications." While relying on the public serves as a detective control, it is still incumbent upon EPA and the State environmental offices, as good public stewards, to have more effective preventive controls in place to better ensure data quality. Recommendations We recommend that the Director, Office of Compliance, within the Office of Enforcement and Compliance Assurance: 1. Establish a management control structure to facilitate PCS to ICIS-NPDES conversion. The management control structure should support plans to meet milestone dates. 8 ------- 10-P-0230 2. Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS-NPDES conversion for FY 2012. 3. Gather key leading data quality practices used by the States and share these practices with all States. 4. Provide reminders to EPA regional offices and State environmental offices to use the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for State-specific reporting requirements), EPA should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the States of the importance of documenting changes made to data elements after entry into the source systems. 5. Complete new rules that require States to report minor facility data. 6. Place a notice on the EPA ECHO Website stating that ECHO does not contain all minor facility data. The wording and placement of the notice should be designed to help provide the public with transparency into the actual data elements maintained in ECHO. 7. Conduct a review of the procedures used to test ICIS-NPDES programming code before it is placed into production. Agency Comments and OIG Evaluation EPA agreed with the report findings. EPA felt that it has extensive, documented procedures in place to test ICIS-NPDES programming code before it is placed into production. Management cited that they follow best practices used by the information technology industry as well as EPA system life cycle policies and procedures and that their primary contractor is certified by the Software Engineering Institute to be Level 3 on the Capability Maturity Model Integration scale. Management also stated their procedures have been refined and improved over the years, and, because of this, new and revised software is typically implemented into production with few, if any, defects. We agree that management should use industry best practices whenever feasible and adopt an industry recognized method for controlling system development activities. However, we believe as part of an effective internal control structure management must continually review these practices to ensure they operate as intended. For the State of Michigan, EPA recognized the error in the State's ECHO data and it took the Agency close to one year to make changes to the programming code to correct the problem. As such, we believe it is incumbent upon management to conduct a root cause analysis to identify the causes for the programming code error and update the testing procedures to prevent similar errors from happening in the future. 9 ------- 10-P-0230 Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFITS (In $000s) Rec. No. Page No. Subject Status1 Action Official Establish a management control structure to facilitate PCS to ICIS-NPDES conversion. The management control structure should support plans to meet milestone dates. Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS- NPDES conversion for FY 2012. Planned Completion Date Claimed Amount Agreed To Amount Director, Office of Compliance, Office of Enforcement and Compliance Assurance Director, Office of Compliance, Office of Enforcement and Compliance Assurance 9 Gather key leading data quality practices used by the States and share these practices with all States. 9 Provide reminders to EPA regional offices and State environmental offices to use the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for State- specific reporting requirements), EPA should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the States of the importance of documenting changes made to data elements after entry into the source systems. 9 Complete new rules that require States to report minor facility data. Place a notice on the EPA ECHO Website O stating that ECHO does not contain all minor facility data. The wording and placement of the notice should be designed to help provide the public with transparency into the actual data elements maintained in ECHO. Director, Office of Compliance, Office of Enforcement and Compliance Assurance Director, Office of Compliance, Office of Enforcement and Compliance Assurance Director, Office of Compliance, Office of Enforcement and Compliance Assurance Director, Office of Compliance, Office of Enforcement and Compliance Assurance Conduct a review of the procedures used to test ICIS-NPDES programming code before it is placed into production. O Director, Office of Compliance, Office of Enforcement and Compliance Assurance O = recommendation is open with agreed-to corrective actions pending C = recommendation is closed with all agreed-to actions completed U = recommendation is undecided with resolution efforts in progress 10 ------- 10-P-0230 Appendix A ^tD SX * A \ IŪ i MEMORANDUM SUBJECT: Response to the Office of the Inspector General Draft Evaluation Report, "ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate with Additional Improvements" Project Number OMS-FY09-00011, dated July 28, 2010 FROM: Cynthia Giles /s/ Assistant Administrator TO: Rudolph M. Brevard Director, Information Resources Management Assessments Office of Mission Systems Office of Inspector General Thank you for the opportunity to review and comment on the draft evaluation report entitled, "ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate with Additional Improvements," (Report) Project Number OMS-FY09-00011. This Report focuses on improving the quality and transparency of the enforcement and compliance information EPA provides to the public through our Enforcement and Compliance History Online (ECHO) website. OECA agrees that providing high quality transparent data to the American public is a fundamental responsibility of the government. Recently, OECA has taken several steps to expand transparency of enforcement and compliance information through ECHO with the release of the Clean Water Act Annual Noncompliance 2008 Report and the enhanced Air, Water and Waste Resource pages. Additionally, OECA is expanding our efforts to ensure the accuracy of the information in ECHO. A key new resource we are providing the public is consolidated information by program area (Clean Water, Clean Air and Hazardous Waste). Included on these pages are reports such as the "2009 Annual CAA Majors Report" and the "2009 State Review Framework Frozen National Report". Prior to updating these annual reports, we provide the States an opportunity to verify the data and provide additional information. OECA is working with our State partners to ensure that the public receives the best information available. Agency Response UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 August 20, 2010 office of ENFORCEMENT AND COMPLIANCE ASSURANCE 11 ------- 10-P-0230 Report Recommendation 1: Establish a management control structure to facilitate PCS to ICIS-NPDES conversion. The management control structure should support plans to meet milestone dates. OECA Response: OECA agrees with the reports attention to strong management controls to ensure a continued quality conversion from PCS to ICIS-NPDES. OECA has strong management controls in place to facilitate the PCS to ICIS-NPDES conversion. There are three control structures used for this purpose including an Integrated Project Team using established best practices for IT project management, the Environmental Information Exchange Network (EN) governance structure which controls electronic exchange of data, and the permanent EPA management team of the Office of Compliance. In addition there are state-specific Data Migration Workgroups which are formed in advance of each state's conversion; to work through data migration issues and ensure readiness for the conversion process. The ICIS-NPDES Batch Integrated Project Team (IPT) was formed to bring together management and staff with the necessary business and technology skills from EPA (OECA, OEI, and EPA Regions) and our state partners through the Environmental Council of States to implement the successful transition from PCS to ICIS-NPDES. The IPT serves as the primary management control structure for the conversion. The members oversee development processes and software, including finalization and implementation of the NPDES schemas for the electronic flow of data from state NPDES systems to ICIS-NPDES. This IPT was initially convened in March of 2007 to move forward the electronic submission of Discharge Monitoring Report data from state systems to ICIS-NPDES. This was successfully completed in May of 2008. The ICIS- NPDES Batch IPT was reconvened in October of 2009 to move forward with the electronic transmittal of the remaining NPDES data families and complete the conversion of the remaining 22 states to ICIS. There are currently 38 states and 6 regions participating on the IPT. Part of the responsibility of the IPT is to set project milestones and regularly review them. To date, the work of the project remains on schedule. Pilot state(s) are drawn from the IPT (as volunteers) to test out the schemas and migrate to ICIS-NPDES per the agreed upon schedule. The governance structure of the Environmental Information Exchange Network (EN) requires that all software schema to be used for flowing data across the EN, including the products developed under the leadership of the IPT, be reviewed and approved by the Network Technology Group (NTG) which is made up of information technology specialists that have expertise in data conversion, exchange, and publishing. The NTG operates under the auspices of the Network Operations Board (NOB), a group that is responsible for the day-to-day operations of the EN and its operational policies and procedures. Any issues that are not resolved at the NOB level may be elevated to the 12 ------- 10-P-0230 higher Exchange Network Leadership Council. All three groups have EPA, state, and tribal partners and decisions are made collaboratively. Within the Office of Compliance, the activities of the IPT and project milestones are reported to Office of Compliance executives at biweekly management meetings. Both the Division Director and the Office Director and their deputies review accomplishments, address issues, and ensure that funding and other resources are available to assist states with conversion of their data. Finally, an ICIS-NPDES Data Migration Workgroup is initiated each time there is a group of states scheduled to migrate from legacy PCS to ICIS-NPDES. Participants include management and staff from each migrating state, each state's respective region, and OECA. Work must be done by each migrating state to ready their data in legacy PCS for the migration to ICIS-NPDES. The purpose of holding the workgroup is to walk the state and regional participants through the data migration process, explain the data differences between legacy PCS and ICIS-NPDES, walk through the mapping rules for data being migrated from legacy PCS to ICIS-NPDES, review test data migration results from each test data migration run, and answer questions concerning the process, the data and ICIS-NPDES. Work groups typically last a year and the status of activities is reported to the IPT and the management chain within the Office of Compliance. Report Recommendation 2: Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS-NPDES conversion for FY 2012 OECA Response: OECA currently has guidance language regarding the use of the Environmental Information Exchange Network in its "FY-2011 OECA National Program Manager (NPM) Guidance" dated April 30, 2010 that reads: "Implement the use of NetDMR or other e-DMR tools for the electronic transfer of Discharge Monitoring Reports (DMR) to ICIS-NPDES, supported by use of the National Environmental Information Exchange Network (Exchange Network), by all of their NPDES permitted facilities. Administrator Jackson , in her July 7, 2009 memorandum, "Achieving the Promise of the National Environmental Information Exchange Network," requested cross-Agency commitment to make the Network the preferred way EPA, States, Tribes, and others share and exchange data while supporting an aggressive timetable to phase out other data submission and exchange methods. EPA Regions need to demonstrate leadership in implementing this strategic vision for the Agency." (See page 29.) OECA commits to continuing to provide this information in future National Program Manager Guidance documents. 13 ------- 10-P-0230 Report Recommendation 3: Gather key leading data quality practices used by the states and share these practices with all states. OECA Response: OECA agrees that sharing of best practices across the states will improve the overall quality of information available to EPA and the American public. OECA is committed to assuring data quality. It will, as permitted under the Paperwork Reduction Act, seek to identify a few states that are willing to share and post their quality control and quality assurance methods for achieving high quality enforcement and compliance information. OECA also commits to using National Meetings as forums for panel and other discussions to allow states to share their data quality best practices. In addition, there are a number of existing examples of states sharing and using data quality practices for both ICIS and PCS. In 2009 at the ICIS National Meeting, regions and states shared their Discharge Monitoring Report (DMR) data quality practices. States also shared their Quality Assurance and Quality Control (QA/QC) processes as they related to the standard reports available in ICIS. These processes involved quality control of data for limits, overdue schedules, and violations. The Quarterly Non-Compliance Report is also used by states as a data quality tool. On occasion, data quality issues are addressed on the ICIS monthly user calls. For PCS, states use the "Dummy" audit capability to review data and reject transactions before transferring the data into the system. They follow up by reviewing the rejected transactions from the "Live" audit reports. Report Recommendation 4: Provide reminders to EPA regional offices and state environmental offices to use the standard Form 2040-0004for data entry. Where this is not feasible (e.g., for state-specific reporting requirements), EPA should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the states of the importance of documenting changes made to data elements after entry into the source systems. OECA Response: OECA agrees with the report that we should strive to better ensure states and regions are well informed on the proper forms and procedures for collecting and submitting data to EPA. OECA will take every available opportunity to remind EPA regional offices and states to use the Standard Form 2040-0004. This will include mechanisms such as the quarterly ICIS newsletter, monthly user calls, and electronic mail 14 ------- 10-P-0230 messages sent to individuals on the current ICIS list serve. Please note, however, that this form is also called Standard Form 3320-1 and both numbers appear on the physical document. Because our current user community and documentation refer to SF-3320-1 and we do not wish to confuse submitters, we will continue to refer to SF-3320-1 in our communications. Additionally, all the State and Federal electronic DMR systems are 3320-1 compliant. Report Recommendation 5: Complete new rules that require states to report minor facility data. OECA Response: OECA strongly agrees with the recommendation to complete the rulemaking process requiring the submission of minor facility data. OECA is currently developing a rule which will significantly improve the NPDES data quantity and quality nationally. A key component of this rulemaking is the requirement for electronic reporting of certain NPDES information from the permittees, including non-major facilities, to states and EPA. This information is currently required to be reported by the permittees to the states, but there is no regulatory requirement for states to provide that specific information to EPA. This NPDES information is generally obtained on paper forms rather than electronically, although some states have developed electronic reporting tools. Therefore, this rule may likely change the mode by which the permittees report, without requiring additional information to be collected and submitted by the permittees, and reducing the burden on state programs caused by the handling of paper forms.. Key EPA management decisions regarding the specifics of the rulemaking are still pending, awaiting ongoing detailed analyses. However, this proposed requirement for electronic reporting of NPDES information from the permittees may likely include information such as discharge monitoring reports (DMRs), program reports, notices of intent to discharge, etc. As a supplement to that information from permittees, other information will likely be required to be reported by the states to EPA regarding the states' implementation activities (such as inspections, violation determinations, enforcement actions, and permit issuance) for major and non-major facilities as well as for NPDES subprograms (e.g., pretreatment, biosolids, CAFOs, etc.). EPA currently projects issuance of this proposed rule in April 2011. Following a comment period, EPA expects to issue a final rule regarding this matter circa September 2012. Report Recommendation 6: Place a notice on the EPA ECHO website stating that ECHO does not contain all minor facility data. The wording and placement of the notice should be 15 ------- 10-P-0230 designed to help provide the public with transparency into the actual data elements maintained in ECHO. OECA Response: OECA agrees that the public needs to be aware that not all minor facility data is currently represented in ECHO. OECA does provide several notices to the public on the limitations of the data provided by ECHO. OECA has recently added several new features that allow ECHO users to find additional detailed information about the quality and completeness of information, including an interactive map that shows the data entry percentage for minor discharge monitoring data in each state. However, this information can be difficult to find, and OECA agrees this information can be better organized. For example we currently use grey text to indicate that there are data caveats. We will enhance this grey text so it is more recognizable to ECHO users. OECA will evaluate each area where the existing notice is displayed and determine if it needs to be enhanced. OECA will also evaluate its ECHO site as a whole and determine if additional notices need to be added. OECA regularly enhances the ECHO website, and will continue its practice of providing the public with caveats concerning the data. Report Recommendation 7: Develop and implement procedures to test ICIS-NPDES programming code before it is placed into production. OECA Response: OECA has extensive, documented procedures in place to test ICIS-NPDES programming code before it is placed into production. It follows best practices used by the information technology industry as well as EPA system life cycle policies and procedures. The prime contractor for the ICIS system is certified by the Software Engineering Institute (SEI) to be Level 3 on the Capability Maturity Model Integration (CMMI) scale. This is the same high level that is required for Department of Defense contractors. This certification requires the contractor to follow repeatable, standard processes for verification of all products. In addition, OECA completes and documents all items that are required by the EPA National Computing Center's Automated Deployment Checklist prior to each software release. As part of the standard life cycle process, procedures for thoroughly testing the software for ICIS-NPDES were implemented during the 2004/2005 timeframe. This was the period during which the software for ICIS-NPDES was being initially developed and tested prior to its release into production. These procedures have been refined and improved over the years, and, because of this, new and revised software is typically implemented into production with few, if any, defects. The software goes through Unit Testing, Functional Testing, and End-to-End or System Integration Testing by the ICIS contractor to ensure it both meets stated requirements and executes correctly before it is turned over to EPA and the user community for User Acceptance Testing. For new 16 ------- 10-P-0230 software relating to major new capability, the software also goes though Beta Testing with experts on the ICIS Project Team prior to User Acceptance Testing. This ensures that as many defects as possible are found and fixed prior to user community testing. All testing is conducted according to Test Plans which include documenting test cases and, upon execution, the results of executing the test cases. Problems identified during each test phase are corrected and retested as necessary prior to the software being released into Production. Testing results are reviewed during the Bi-weekly Project Management meetings held between EPA and the ICIS contractor. 17 ------- 10-P-0230 Appendix B Distribution Office of the Administrator Assistant Administrator for Enforcement and Compliance Assurance Assistant Administrator for Environmental Information and Chief Information Officer Regional Administrator, Region 3 Regional Administrator, Region 4 Regional Administrator, Region 5 Director, Office of Compliance, Office of Enforcement and Compliance Assurance Acting Director, Office of Information Collection, Office of Environmental Information Agency Follow-up Official (the CFO) Agency Follow-up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for External Affairs and Environmental Education Agency Follow-up Coordinator, Office of Enforcement and Compliance Assurance Inspector General 18 ------- |