$ < 73 \ Ml (T PRQrt^ b 2 ui (J T J OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Evaluation Report Limited Knowledge of the Universe of Regulated Entities Impedes EPA's Ability to Demonstrate Changes in Regulatory Compliance Report No. 2005-P-00024 September 19, 2005 ------- Report Contributors: Erin Barnes-Weaver Kim Bryant Manju Gupta Jeff Hart Ben Webster Abbreviations CAA Clean Air Act CWA Clean Water Act ECHO Enforcement and Compliance History Online EPA Environmental Protection Agency FIFRA Federal Insecticide, Fungicide, and Rodenticide Act GAO Government Accountability Office NPDES National Pollutant Discharge Elimination System OECA Office of Enforcement and Compliance Assurance OIG Office of Inspector General RCRA Resource Conservation and Recovery Act SDWA Safe Drinking Water Act TSCA Toxic Substances Control Act ------- £ < 33 V ^60S% V PRO^4-0 o LU a U.S. Environmental Protection Agency Office of Inspector General At a Glance 2005-P-00024 September 19, 2005 Why We Did This Review To enforce its regulations and achieve maximum compliance, a regulatory agency must know its entire regulated universe. We sought to determine how well the U.S. Environmental Protection Agency (EPA) Office of Enforcement and Compliance Assurance (OECA) knows the composition and size of its regulated universe, as well as how OECA determines and reports compliance levels across the regulated universe. Background OECA compiled its regulated universe table to provide consistent numbers when presenting compliance information to Congress, the public, and other stakeholders. The information also aids EPA in making management decisions about compliance and enforcement resource allocations. In the universe table issued in September 2001, OECA reported an inventory of approximately 41.1 million regulated entities. For further information, contact our Office of Congressional and Public Liaison at (202) 566-2391. To view the full report, click on the following link: www.epa.qov/oiq/reports/2005/ 20050919-2005-P-00024.pdf Catalyst for Improving the Environment Limited Knowledge of the Universe of Regulated Entities Impedes EPA's Ability to Demonstrate Changes in Regulatory Compliance What We Found OECA has limited knowledge of the diverse regulated universe for which it maintains responsibility. OECA has not updated its universe table since generating it in 2001, even though some universe figures for reviewed program areas have changed substantially. EPA has used the 2001 table as a source for describing the size of its regulated universe in public documents. Various data quality issues impact OECA's ability to adequately identify the size of its regulated universe and associated compliance information. OECA concentrates most of its regulatory activities on large entities and knows little about the identities or cumulative impact of small entities. OECA cannot effectively use universe figures to assist with its regulatory activities. OECA does not develop programmatic compliance information, adequately report on the size of the universe for which it maintains responsibility, or rely on universe figures to assist with planning. OECA's limited universe knowledge prevents it from determining overall compliance levels in five of the six regulatory program areas we reviewed. This hinders OECA's ability to generate valid programmatic compliance information and effectively determine program success. In addition, OECA lacks adequate transparency in publicly reporting some currently available compliance information. What We Recommend We recommend that OECA biannually update publicly released universe figures, and produce complete, reasonably accurate, and current universe data. Further, OECA should better describe its enforcement and compliance role, develop an objective to obtain better reporting from States, and request EPA program offices to analyze and report on the cumulative impact of violations from small entities. Also, we recommend that OECA develop and publish information that demonstrates changes in compliance levels, and better share existing compliance data and analyses that will provide external stakeholders with an improved understanding of programmatic compliance levels. EPA agreed with some of our recommendations, but not those related to biannually updating universe figures, developing an objective to obtain better reporting from States, or for developing programmatic compliance information. ------- ^£0SX i - UNITED STATES ENVIRONMENTAL PROTECTION AGENCY | X\\/V 3 WASHINGTON, D.C. 20460 V OFFICE OF INSPECTOR GENERAL September 19, 2005 MEMORANDUM SUBJECT: Limited Knowledge of the Universe of Regulated Entities Impedes EPA's Ability to Demonstrate Changes in Regulatory Compliance Report No. 2005-P-00024 FROM: Jeffrey K. Harris /s/ Director for Program Evaluation, Cross-Media Issues TO: Granta Y. Nakayama Assistant Administrator Office of Enforcement and Compliance Assurance This is our final report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This evaluation report contains our findings that describe the problems we have identified and corrective actions we recommend. This evaluation report represents the opinion of the OIG and the findings contained in this report do not necessarily represent the final EPA position. EPA managers will make final determinations on matters in this report in accordance with established procedures. We met with Office of Enforcement and Compliance Assurance managers on June 13, 2005, to discuss our preliminary findings, and provided our draft report on June 30, 2005. EPA did not concur with all of our recommendations. EPA provided its official written comments in two documents. We have included EPA's summary memorandum response in its entirety as Appendix D. EPA's second document - the Agency's response, including detailed comments on our recommendations - is available on OIG's Web site, with the report. We are also including lists of publicly available compliance and enforcement measures, provided to us in EPA's response, in Appendix F. ------- Action Required EPA Manual 2750 requires you as the action official to provide this office with a written response to this report within 90 calendar days of the final report date. Your response should address all recommendations and must include your concurrence or nonconcurrence with all recommendations. For corrective actions planned but not completed by the response date, please describe the actions that are ongoing and provide a timetable for completion. If you do not concur with a recommendation, please provide alternative actions addressing the findings reported. For your convenience, this report will be available at http://www.epa.gov/oig/. Attachment cc: Phyllis Harris, Principal Deputy Assistant Administrator, OECA Michael M. Stahl, Director, Office of Compliance, OECA Walker B. Smith, Director, Office of Civil Enforcement, OECA Greg Marion, Audit Followup Coordinator, OECA ------- Table of C At a Glance Chapters 1 Introduction 1 Purpose 1 Background 1 Scope and Methodology 3 2 OECA Has Limited Knowledge of the Regulated Universe 6 Universe Table Not Updated Since 2001 Despite Significant Changes 6 Data Quality Problems Affect OECA's 2001 Universe Figures 7 OECA Could More Clearly Describe Its Role 10 OECA Does Not Focus Its Compliance Monitoring and Enforcement Activities on a Greater Part of the Regulated Universe 11 OECA Does Not Know the Cumulative Impact of Small Entities 14 Recommendations 15 Agency Comments and OIG Evaluation 15 3 OECA Cannot Determine Changes in Compliance for Five of Six Sample Program Areas 16 OECA Cannot Generate Programmatic Compliance Information for Five of Six Program Areas 16 OECA Is Not Sufficiently Transparent with Available Compliance Data 19 Recommendations 20 Agency Comments and OIG Evaluation 20 Appendices A OECA's Regulatory Universe Identification Table - Executive Summary 22 B Environmental Statute Information and Program-Specific Facility Size Classifications 26 C Details on Scope and Methodology 28 D Agency Response Memorandum to Draft Report 32 E Agency Comments to OIG Recommendations and OIG Evaluation 36 F Publicly Available Compliance and Enforcement Measures 43 G Distribution 48 ------- Chapter 1 Purpose To enforce its regulations and achieve maximum compliance, a regulatory agency must know its entire regulated community. This knowledge enables an agency to make more informed management decisions about compliance and enforcement resource allocations, and provides Congress and the public with data on whether compliance is increasing. Further, if entities realize they are unknown to a regulatory agency, they may be less likely to comply with environmental regulations, and enforcement actions by the U.S. Environmental Protection Agency (EPA) may be unlikely to occur. The EPA is responsible for implementing a large number of environmental regulations over a diverse universe of regulated entities of differing size, nature of operations, and complexity. To evaluate the effectiveness of EPA's enforcement and compliance efforts, we plan to evaluate several interrelated issues. This report addresses the nature and composition of the regulated universe, and answers the following questions: • How well does EPA's Office of Enforcement and Compliance Assurance (OEC A) know the composition and size of its regulated universe? • How does OECA determine and report levels of compliance in the regulated universe? In subsequent reports, we plan to address how to best measure compliance status, determine EPA's tools and strategies for improving compliance, and consider the changes that occur as a result of those tools and activities. Background Importance of Knowing the Regulated Universe Knowledge of the size and character of the regulated community is fundamental to a regulatory agency's effectiveness. Identifying the regulated universe enables regulatory agencies to develop effective enforcement and compliance strategies, as well as establish deterrence among the regulated community. Current and complete universe information facilitates evaluation, highlights the scope of responsibilities, increases transparency, and improves the ability to manage enforcement and compliance efforts. 1 ------- We developed the following list of the benefits of a known universe based on interviews with OECA staff and reviews of environmental policy literature (see Table 1-1): Table 1-1: Benefits of Knowing the Full Regulated Universe Evaluation Current and complete universe figures can allow for improved internal and external analysis and reporting of the impact of a regulatory agency's activities. Scope Universe data discloses the size and nature of a regulatory agency's responsibilities. Knowledge of the universe provides the agency with a definitive baseline on the number, size, location and character of entities subject to particular regulations and statutes. This is especially important as entities shut down, start up, move, or change their operations. Transparency EPA has a longstanding commitment to transparency - to publicly account for its decisions and explain why it took or did not take certain actions. During his confirmation hearing, current EPA Administrator Stephen Johnson listed one of his guiding principles as pursuing "as open and transparent a decision making process as possible." Increased information disclosure allows external stakeholders to better understand a regulatory agency's activities. Since the passage of the Emergency Planning and Community Right-to-Know Act in 1986, EPA has used information disclosure as a means to improve environmental performance. OECA documents note how continued and increased release of information keeps with the Administrator's emphasis on greater transparency. Public disclosure also holds the Agency accountable for all lessons learned, including both successes and failures. Management Current and complete universe information can assist with the following management activities: • Targeting: Updated universe data provides information on how many entities can and will be affected by regulatory agencies' enforcement and compliance activities, and thus enables better targeting strategies. • Retrospective Analyses: Comprehensive regulated universe information can assist a regulatory agency in conducting retrospective analyses on the success of their enforcement and compliance activities in certain program areas. • Rule-making: OECA staff said EPA has used universe data in regulatory decisions and analyses. • Priority-Setting: Knowledge of the regulated universe enables regulatory agencies to more easily establish defensible priorities in focusing resources. OECA's 2001 Universe Table In September 2001, OECA compiled figures detailing the size of the regulated universe for which it was responsible. OECA produced its 2001 Regulatory Universe Identification Table (see Appendix A) after receiving criticism for not having an adequate knowledge of its regulated universe. According to senior 2 ------- OECA managers, OECA also generated the universe figures to provide a single definitive source for internal briefings and external presentations. For example: • Prior to 2001, OECA staff cited a variety of inconsistent numbers when discussing the regulated universe. During congressional budget hearings, OECA could not provide consistent accounts of its regulated universe. • In a July 2001 report,1 the Government Accountability Office (GAO) stated OECA could not demonstrate "the universe of entities subject to regulation under federal environmental laws." GAO also noted OECA could not adequately explain variations in enforcement activity across regions, or how it distributed resources and determined priorities. The 2001 Regulatory Universe Identification Table shows that OECA's universe totaled approximately 41.1 million entities. OECA developed the table by using Agency databases and consulting 36 sources for the 58 program areas included in the table. In developing the table, OECA produced a methodology that could be used to generate future universe data. Staff in the Office of Compliance coordinated with the Office of Regulatory Enforcement (now the Office of Civil Enforcement) and program offices to obtain numbers for universe program areas. OECA staff asked relevant program staff to provide program descriptions, a data source to generate the universe number, and any data caveats. OECA staff emphasized that a source, database, or document had to support the universe number used in the table; institutional knowledge or a "gut" feeling were not adequate. OECA staff described the methodology as a resource-intensive process. Since 2001, EPA has described the number of regulated entities as approximately 41 million based on OECA's universe table. For example, EPA cited this figure in public documents, such as the EPA Strategic Plan 2003-2008. Scope and Methodology We conducted our evaluation field work on EPA's knowledge of its regulated universe and compliance status between January and April 2005. We generally performed our evaluation in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. To answer both the universe and compliance objectives, we judgmentally selected six sample program areas, as case studies, from OECA's 2001 Regulatory Universe Identification Table. We based this sample on information gathered from our preliminary research. An initial analysis of the Regulatory Universe Identification Table showed that OECA's universe consisted of a total of approximately 41 million entities. These included a diverse and complex mix of entities of varying sizes and types. Small entities made up a much greater part of 1 GAO-Ol-812, Human Capital: Implementing an Effective Workforce Strategy Would Help EPA to Achieve Its Strategic Goals, July 2001. 3 ------- the universe than major and large entities. A review of OECA's planning documents also showed that a handful of programs were selected as national priorities, and others were described as core programs. While OECA is ultimately responsible for regulating all 41 million entities, it focuses its regulatory attention on national priority program areas, while the States are primarily responsible for core programs. To conduct a balanced evaluation of OECA's knowledge of the regulated universe, it was important to select a sample that reflected the mix of program areas for which OECA is both ultimately responsible and it had represented as falling under its regulatory authority in its Regulatory Universe Identification Table. We selected the sample program areas using the following criteria: • Program areas from across major environmental statutes; • Program areas including a mix of large and small entities because OECA included both groups in its overall universe count; and • A mix of national priority and core programs. Applying these criteria, we selected the sample program areas listed in Table 1-2. We included details on the applicable statutes, program areas, and distinctions between the different sizes of entities in Appendix B. Table 1-2: Environmental Statutes and Sample Program Areas Statute Program Area Clean Air Act (CAA) Minor Stationary Sources and Synthetic Minors * Clean Water Act (CWA) Stormwater Permits (National Pollutant Discharge Elimination System) Federal Insecticide, Fungicide, Total Number of Farms and Business Sites and Rodenticide Act (FIFRA) Regulated under Pesticides Programs Toxic Substances Control Act Core TSCA - Other Manufacturers, Processors, (TSCA) Distributors, Users, and Exporters Safe Drinking Water Act (SDWA) Public Water Systems: Community Systems Resource Conservation and Small Quantity Generator Facilities Recovery Act (RCRA) * Synthetic minors are stationary sources that have the capacity to emit at or above the major source threshold but are considered minor sources because they place physical or operational limitations on their capacity to emit pollution. We did not quantitatively extrapolate the results from our judgmental sample across OECA's entire regulated universe. Similarly, we did not use our judgmental sample to generalize to OECA's entire regulated universe. During the course of our review, however, we did find conditions that were systemic. We interviewed OECA managers to determine whether a condition was limited to the program area under review, or whether it was an OECA-wide policy or practice. 4 ------- When OECA managers confirmed that it was an OECA-wide policy or practice, we presented it as a finding with an appropriate recommendation. These findings and recommendations may therefore be broader than the program areas under review within the sample. However, they are accompanied by appropriate explanatory caveats. Both Chapters 2 and 3 include such findings and recommendations. We did not independently verify the accuracy or reliability of data provided by OECA staff for our program area sample. We conducted limited work regarding fraud, and did not test internal controls. We followed the guidance and definitions provided in the GAO guidance document, Assessing the Reliability of Computer-Processed Data, to answer our evaluation objectives. We reviewed existing information on OECA's data problems, limitations, and corrective actions, and conducted interviews with OECA staff knowledgeable about enforcement and compliance data systems. We designed our scope to examine universe data at a national level and, as such, did not trace documents or data from original points of regional, State, or local data entry to final database output. See Appendix C for more details on our scope and methodology. 5 ------- Chapter 2 OECA Has Limited Knowledge of the Regulated Universe OECA has limited knowledge of the regulated universe for which it maintains responsibility. OECA has not updated its Regulatory Universe Identification Table since generating that table in 2001, even though some universe figures have changed substantially. In addition, various data quality issues impact OECA's ability to adequately determine reliable and current information on the size and composition of its regulated universe. OECA concentrates most of its compliance monitoring and enforcement activities on large entities, and knows little about the identities or cumulative pollution effects of small entities. Therefore, OECA cannot effectively use universe figures to assist with its regulatory activities. It cannot develop programmatic compliance information, adequately report on the size of the universe for which it maintains responsibility, or rely on these numbers to assist with management and regulatory planning. Further, in publications, OECA could more clearly qualify its role in relation to the roles played by States and others. Universe Table Not Updated Since 2001 Despite Significant Changes While the number of entities in our universe sample has changed, OECA has not updated the universe table as a whole. To determine whether any significant changes in the size of the sampled universe have occurred, we requested that OECA staff provide us with the current number of regulated entities for our sampled program areas. Between 2001 and 2005, the universe for these program areas increased by 35 percent. Although we had limited our review of the size increase to only the six sampled areas, this indicates that the numbers of entities changed over time and OECA needs to update the numbers. Table 2-1 illustrates, for example, how some areas changed very little, while others changed significantly: 6 ------- Table 2-1: Change in Sample Universe Figures by Program Area, 2001 and 2005 Program Area 2001 2005 Percent Change CAA - Minor Stationary Sources CAA - Synthetic Minor Sources 96,866 17,248 101,370 20,795 + 5 % + 21 % CWA - Stormwater Permits 380,000 >550,000 + 45 % FIFRA - Farms and Businesses, etc. 2,246,512 2,168,241 -4% TSCA - Other Manufacturers, etc. 3,758,176 6,063,948 + 61 % SDWA - Public Water Systems: Community Systems 54,101 52,838 -2% RCRA - Small Quantity Generator Facilities 202,965 178,771 - 12 % Total 6,755,868 9,135,963 + 35 % Following the release of the Regulatory Universe Identification Table in September 2001, OECA planned to begin updating the table in December 2001. OECA intended future iterations to provide regional breakouts of the regulated universe in addition to national figures. However, OECA has not produced new universe figures since it released the original table in 2001, and continues to cite the number of entities from the 2001 table. OECA staff attributed this to a lack of resources. However, they indicated they are considering updating information on certain areas selected as national priority programs. OECA selected a handful of programs as national priorities based on patterns of noncompliance or the threat of significant environmental risk. In a December 8, 2004, response to our questions, senior managers from OECA's Office of Compliance indicated they did not believe the universe numbers had changed dramatically. Some OECA staff, however, said it is difficult to provide precise universe figures for some program areas because they fluctuate frequently. Regulatory agencies should attempt to maintain a reasonably accurate count of the regulated entities, by program area, under their purview. Knowledge of the regulated universe allows an environmental regulatory body like OECA to better understand the patterns of noncompliance in a sector or population, assess the production process and practices used by entities, and determine the state of environmental management sophistication in the sector or population. Without reliable universe information, OECA lacks both a definitive baseline on the number, size, and character of entities subject to regulation, as well as the information necessary to provide a denominator for compliance rates. Data Quality Problems Affect OECA's 2001 Universe Figures The data that constitute OECA's numbers in the universe table are subject to several data quality issues. With the exception of SDWA, we found universe data for the sampled program areas was not reliable. The EPA Office of Inspector 7 ------- General (OIG), GAO, and the National Academy of Public Administration have previously reported data quality problems. We based our conclusions using definitions from GAO's Assessing the Reliability of Computer-Processed Data, which defines reliable data as being both complete and accurate, and provides the following definitions: • Complete: The data contain all of the data elements and records needed for the engagement. • Accurate: The data reflect the data entered at the source or, if available, in the source documents. Table 2-2 shows where we determined that a number of program areas do not have reliable or current data. Some program areas rely on estimated or extrapolated data, and the estimates were not based on supportable information. According to OECA, its inability to ensure desired data quality occurred due to data gaps. Data gaps occur in cases where regulations do not require States to report data to OECA. OECA managers said that obtaining additional reporting from States would require approvals by the Office of Management and Budget, or additional regulatory actions. Table 2-2: Data Quality of Sample Program Areas Program Area Reliable Data * Current Data CAA - Minor Stationary Sources CAA - Synthetic Minor Sources No No No No CWA - Stormwater Permits No Unknown FIFRA - Farms and Businesses, etc. No Unknown Core TSCA - Other Manufacturers, etc. No No SDWA - Public Water Systems: Community Systems Yes Yes RCRA - Small Quantity Generator Facilities No Unknown * Reliable data must be both complete and accurate. Results from Table 2-2 are explained below: • CAA Minor Stationary Sources: CAA minor source data are incomplete because States are not required to report CAA minor stationary source data. In addition, State and local agencies do not evaluate all minor source entities regularly, so the data are not kept current. • CAA Synthetic Minor Sources: According to OECA staff, the Agency does not have a complete, accurate, or current universe of CAA synthetic minor sources even though State and local regulatory agencies are required to report the data to EPA. The staff also noted that State and local agencies do not evaluate all synthetic minor sources regularly, so the data are not kept current. 8 ------- • CWA Stormwater: EPA's Office of Water staff said they estimated the figure used in the 2001 table, but they could not provide a supportable basis for the estimate. They suggested the number might have come from extrapolated projections from the seven States for which EPA had not delegated the CWA stormwater program. It, therefore, does not necessarily represent a reliable estimate for the true national universe of entities subject to stormwater regulations. • FIFRA: Approximately 99 percent of the FIFRA universe of 2,246,512 entities is estimated. It includes three subgroups: pesticide production establishments (12,442), commercial pest control firms (40,000), and farms (2,194,070). OECA has limited knowledge of the largest subgroup: farms. FIFRA provides primacy to States, and OECA has limited involvement with the FIFRA universe. The statute requires commercial establishments to register pesticides, and this component of the universe includes only those who register. The number of non-registrants is not known. • TSCA: The numbers for the TSCA universe are not reliable for the sample program area. The sample included a large group (3,758,176) described as "other manufacturers, processors, users, distributors, and exporters of chemical substances in the U.S...." OECA estimated this number from the Dun and Bradstreet sector data of entities, which is composed of manufacturing, wholesale, and retail trades. However, according to OECA staff, its compliance and enforcement activities only focus on a small subset of the total Core TSCA universe - the 13,513 chemical manufacturers. This number for chemical manufacturers was generated from the 1997 U.S. Economic Census. This 1997 figure was used in the 2001 universe table, and was also provided as the current universe number. Therefore, both the 2001 and 2005 figures are outdated. • SDWA: We take no exception to the published universe figures for SDWA. The sample for SDWA included Public Water Systems: Community Systems. This universe is well defined. We did not independently test the SDWA data. • RCRA: The RCRA small quantity generator facility universe number is incomplete. The designation of large quantity versus small quantity depends on the amount of waste generated and stored. According to OECA staff, these facilities are not required to notify EPA if their status changes from a small to large facility. The facilities can easily change from small to large generators, and the staff explained how it is difficult for OECA to generate a reliable universe figure because of this flux. 9 ------- The EPA OIG, GAO, and the National Academy of Public Administration have previously reported problems with OECA's data quality.2 High quality data in EPA's national databases are essential for credible measures, reports, and analyses. In response to weaknesses identified by external sources, OECA issued guidance in March 2002 on improving data quality, titled Final Enforcement and Compliance Data Quality Strategy, which states: EPA managers and members of Congress rely on this data to hold EPA programs accountable and to inform their decision-making... Accurate data, in an accessible and usable form, is essential to support Agency planning and targeting... Documenting performance [for the Government Performance and Results Act] requires timely, high quality data... Proof of high quality gives rise to confidence among the public, industry, and other users of enforcement and compliance data. OECA Could More Clearly Describe Its Role Of the 41.1 million entities listed in the 2001 universe table, EPA maintains only 1.6 million facility or entity records in its compliance and enforcement databases. This represents less than 4 percent of the total facilities for which OECA is responsible. Many of those entities represent those for which EPA has direct regulatory authority. For most entities, EPA has authorized States to implement environmental programs and conduct enforcement activities in accordance with environmental laws. Among the program areas we evaluated, OECA is the primary regulatory authority only for TSCA. For the other program areas, the States are the primary regulators, with OECA providing an oversight role, although in certain instances EPA may become involved in compliance activities and enforcement actions. Some documents in which EPA includes the 41.1 million figure do not clearly indicate OECA's shared role in conducting its regulatory responsibility. These include the Agency's 2005-2007 National Program Guidance, and EPA responses to prior EPA OIG and GAO reports. EPA's 2003-2008 Strategic Plan and OECA's response to the 2006 Office of Management and Budget Program Assessment Rating Tool do explain that EPA works cooperatively with States, as well as local and tribal agencies, to secure and maintain compliance. OECA staff stated they never intended to imply that OECA had direct responsibility for 41.1 million entities. However, without appropriate explanatory caveats, external stakeholders might misinterpret the figure to mean that OECA actually ensures environmental protection by directly regulating all 41.1 million entities. 2 These documents include OIG's EPA's Key Management Challenges (2003), GAO's Human Capital: Implementing an Effective Workforce Strategy Would Help EPA to Achieve Its Strategic Goals (2001), and the National Academy of Public Administration's Evaluating Environmental Progress: How EPA and the States Can Improve the Quality of Enforcement and Compliance Information (2001). 10 ------- OECA Does Not Focus Its Compliance Monitoring and Enforcement Activities on a Greater Part of the Regulated Universe According to OECA staff, EPA's enforcement and compliance monitoring activities focus on major and large entities or sources, which represent only a small fraction of the total universe shown in OECA's universe table. OECA has mostly focused on larger and major entities, and has not conducted or obtained analyses showing the cumulative impact of the vast number of entities that emit pollution below the threshold of major or larger entities. EPA has focused on major and larger entities because any one of the larger entities can have a greater individual impact than any of the smaller entities by itself. However, given the much greater number of small entities in the sample, and the potential cumulative impact from this vast part of the regulated universe, we find it is important for OECA to know the cumulative environmental impact of entities that fall below the major or large threshold. Information on small entities could help OECA better prioritize where to focus resources and facilitate effective management. The following charts illustrate the proportion of large and small entities in the CAA, CWA, RCRA, and SDWA program areas (FIFRA and TSCA do not formally distinguish between large and small entities). We generated each chart using data from OECA's 2001 Regulatory Universe Identification Table. The charts demonstrate that smaller entities comprise a much larger percentage of OECA's total universe than large entities. We include general statutory information and program-specific facility size classifications in Appendix B. CAA: OECA concentrates its CAA resources on major stationary sources and large polluters. Using 2001 universe figures, major stationary sources, minor sources, and synthetic minor sources total 137,982 entities. Major stationary sources represent only 4 percent of OECA's CAA regulated universe when including other CAA entities such as mobile sources. Chart 2-1: CAA Regulated Universe - 2001 ~ Large Facilities (Major Stationary Sources) ¦ Small Facilities (Minor Stationary Sources) ¦ Small Facilities (Synthetic Minor Stationary Sources) ~ Other Facilities (includes mobile sources) 11 ------- CWA Stormwater Permits: The CWA stormwater program is a subset of the National Pollutant Discharge Elimination System (NPDES) program. The stormwater permits program was instituted in two phases. The first phase required permits for large municipal waste water systems, industrial plants, and large commercial construction sites. Phase two permit requirements cover a much greater number of smaller municipal waste water systems and smaller construction sites. In a majority of cases, States issue general permits, and only a small percent of these permits are in OECA's databases. Analyses have shown that stormwater overflow is a major source of water pollution. The universe of stormwater sources is not well known even though stormwater has been a priority area since fiscal year 2001. Stormwater continues as a national enforcement priority program in fiscal year 2005, and OECA has developed a national strategy. The strategy puts emphasis on providing compliance assistance and developing an inventory of watersheds. OECA plans to devote more resources to focus on these sources. Chart 2-2: CWA-NPDES Regulated Universe - 2001 1% ~ Large Facilities (Major Permitted Facilities) ¦ Small Facilities (Minor Permitted Facilities) ~ Other Facilities (includes wet weather facilities (stormwater permits)) RCRA: OECA concentrates most of its regulatory resources on large quantity generator facilities, although they comprise less than 10 percent of the combined total of RCRA facilities, according to OECA's 2001 universe figures. The percentage of large quantity generator facilities in the total RCRA universe drops to 4 percent when including other facilities such as conditionally-exempt small quantity generators. 16% 83% 12 ------- Chart 2-3: RCRA Regulated Universe - 2001 42% 35% 19% ~ Large Facilities (Large Quantity Generators) 1 Small Facilities (Small Quantity Generators) ¦ Small Facilities (Conditionally-Exempt Small Quantity Generators) ~ Other Facilities SDWA: While OECA is familiar with the SDWA public water system universe, OECA does not expend its regulatory resources on all types of drinking water entities. The majority of the U.S. population gets its drinking water from a small number of public community water systems. EPA's annual report to Congress on national public water system compliance includes information on large public community water systems. The States, not OECA, are responsible for most compliance activities across drinking water systems. SDWA was a national priority enforcement area for OECA until 2005. Chart 2-4: SDWA Regulated Universe - 2001 84% 1 ~ Large Facilities (Public Water Systems - Community Water Systems) ¦ Small Facilities (Non- Transient and Transient Community Water Systems) ~ Other Facilities (includes Class l-V Wells) TSCA: While Core TSCA does not formally distinguish between large and small entities, OECA concentrates its compliance and enforcement resources on 13,513 13 ------- chemical manufacturing entities out of 3,771,689 entities in the universe table. This is 0.4 percent of the stated Core TSCA universe. OECA staff mentioned that they use compliance assistance to help smaller entities become compliant. They also provided various reasons why they generally focus compliance monitoring and enforcement resources on large entities. OECA officials said they should concentrate resources on large entities because: Large entities are often priority enforcement and compliance monitoring areas; States address small entities through core program implementation and OECA specifies to States how frequently they should inspect small entities; The large size of program area universes, and an accompanying lack of resources; and The difficulty in identifying and analyzing RCRA small quantity generator facilities because of their transient nature. OECA Does Not Know the Cumulative Impact of Small Entities In most program areas in our sample, OECA does not know the cumulative effects of pollution from small entities. OECA did not provide information on any analyses that study the cumulative environmental or health effects of small entities. Of our sample program areas, OECA focuses compliance and enforcement resources on small and medium entities in only the CWA stormwater and Core TSCA program areas. EPA's concern that stormwater runoff significantly impairs water quality contributed to designating the stormwater program as a national enforcement priority in prior years, and continuing it as a priority in fiscal 2005. While Core TSCA regulations do not formally distinguish between large and small entities, OECA TSCA staff recently focused on smaller chemical manufacturers because the staff believed they are less likely to understand their Core TSCA obligations than large chemical manufacturers. While no formal impact assessments have been undertaken, some States and EPA regions have argued that RCRA small quantity generator facility inspections represent some of the most environmentally significant activities that regions and States conduct. RCRA enforcement staff said they are collecting small quantity generator facility information, and may consider shifting OECA's RCRA resources in coming years. We acknowledge that analyzing the cumulative environmental effect of small entities is not OECA's direct responsibility. However, given that OECA crosscuts all of the Agency's program offices, we believe OECA should request that program offices conduct these studies if OECA does not do the analyses themselves. The knowledge generated by these analyses would allow OECA to establish more defensible priorities of its own activities, as well as better organize and coordinate the efforts of its State partners. 14 ------- Recommendations We recommend that the Assistant Administrator for Enforcement and Compliance Assurance: 2-1 Biannually update publicly released universe figures by tracking and recording the number of entities over which OECA has oversight and primary regulatory responsibility. 2-2 When producing its biannual universe update, use reliable data to generate complete and current universe numbers that meet national data quality standards similar to those outlined in OECA's Final Enforcement and Compliance Data Quality Strategy. 2-3 Describe OECA's enforcement and compliance role in relation to States and other partners when the Agency publicly releases universe figures. 2-4 Develop an objective of having the most up-to-date and reliable data on all entities that fall under its regulatory responsibility. OECA should adopt the goals of requiring States to track, record, and report data for entities over which States have regulatory responsibility. To achieve this goal, OECA should develop a multi-State, multi-program pilot program of collecting data that States track, record, verify, and report. 2-5 Request that EPA program offices analyze and report to OECA the cumulative impact of violations by regulated entities that pollute below the thresholds of major or large entities. OECA should use any cumulative impact analyses conducted by program offices to support OECA's management decisions. Agency Comments and OIG Evaluation Even though OECA agreed with the premise that knowing the size and character of the regulated universe is a fundamental activity for a regulatory agency, OECA has not agreed to update its knowledge of the universe of entities for which it maintains regulatory responsibility. Specifically, OECA agreed with Recommendations 2-2 and 2-3, but disagreed with Recommendations 2-1, 2-4, and 2-5. Appendix D includes the Agency memorandum that contains summary comments. Appendix E includes the Agency's comments on specific recommendations and other general comments, and our evaluation of those comments. The Agency's detailed comments are available on the EPA OIG Web site. We have made some appropriate changes to the report in response to the Agency's comments. 15 ------- Chapter 3 OECA Cannot Determine Changes in Compliance for Five of Six Sample Program Areas OECA could not determine or report on the levels of compliance with environmental regulations for five of our six sample regulatory areas. OECA's National Program Guidance for 2005-2007 states that "OECA's national enforcement and compliance assurance program is responsible for maximizing compliance." OECA conducts several activities to determine and report compliance. However, in five of the six sample program areas, OECA could not determine and report overall programmatic compliance levels because it lacks current and complete knowledge of the regulated universe. Data quality problems further hindered OECA's ability to generate valid programmatic compliance information. The lack of programmatic compliance information impedes OECA's ability to determine which programs are working and how effective they are in achieving goals. As a result, OECA cannot demonstrate changes in overall compliance in the regulated universe. OECA generates region- and State-specific noncompliance statistics and data for internal reports, but lacks sufficient transparency by not publicly sharing some of this compliance-related information with external stakeholders. OECA Cannot Generate Programmatic Compliance Information for Five of Six Program Areas Reliable compliance information is essential for a regulatory agency to plan, set goals, evaluate the results of its strategy, and demonstrate results to external stakeholders. Statutes and regulations provide the basic framework of regulatory compliance, along with various policies. OECA cannot generate programmatic compliance information for five of the program areas in our sample due to its limited knowledge of the full universe of regulated entities. In these five sample areas, OECA maintained only facility- specific noncompliance information for a limited part of that universe: the larger or major entities in the program. OECA cannot determine the compliance of most minor and synthetic minor stationary air sources, small quantity waste generator facilities, and stormwater dischargers because OECA does not currently require States to report data on minor and small sources. As discussed in Chapter 2, minor and smaller entities constitute a much greater number of OECA's total regulated universe than large entities. Lack of knowledge of the number, location, and levels of compliance of this significant portion of the universe leaves a large gap in OECA's knowledge of overall compliance in each program. Within most of our sample, OECA has not 16 ------- been able to quantitatively demonstrate the success of its strategies, nor show an increase in the percent of entities in compliance, because OECA does not know the full universe of regulated entities. In Table 3-1, we summarize whether compliance information for the program area was available, and provide additional information after the table, based on discussions with OECA staff, about available programmatic compliance information. During discussions with OECA managers, we found that the lack of data was systemic and not limited to only the program areas under review. Table 3-1 shows by program area whether: OECA could provide programmatic compliance information, OECA has defined a significant noncompliance or high priority violator category, and the area is or has been a national priority program since 2001. Table 3-1: Summary of Programmatic Compliance Information for Sample Areas Sample Program Area Programmatic Compliance Information SNC/HPV * Applicable National Priority Program 2001 2005 CAA - Minor Stationary Sources CAA - Synthetic Minor Sources No No Yes Yes No No No No CWA - Stormwater Permits No No Yes Yes FIFRA - Farms and Businesses, etc. No No No No Core TSCA - Other Manufacturers, etc. No No No No SDWA - Public Water Systems: Community Systems Yes Yes Yes No RCRA - Small Quantity Generator Facilities No Yes No No * SNC/HPV: Significant Noncompliance/High Priority Violator CAA Minor Stationary Sources: OECA did not have information on levels of compliance for minor sources because it did not require States to report them in the national Air Facilities Subsystem database or provide associated compliance and enforcement information. CAA Synthetic Minor Sources: OECA does not have complete data on the synthetic minor sources, even though States and local regulatory agencies are required to report on these entities. OECA staff said they have focused mainly on major sources because OECA considers them a greater source of air pollution. In April 2001, EPA issued the Clean Air Act Stationary Source Compliance Monitoring Strategy, which recommends that State and local agencies focus compliance monitoring activities on a subset of the synthetic minor universe. However, OECA staff indicated that they do not have current or complete knowledge of compliance status of all synthetic minor sources. 17 ------- CWA: According to OECA staff, there is a high level of noncompliance with stormwater regulations. Even though this has been a priority area since 2001, OECA cannot provide programmatic or facility-specific compliance information because it does not have a good knowledge of this universe. In most cases, States issue general permits for stormwater regulations, and EPA does not have detailed data on these permits. In fiscal year 2005, OECA again selected the Stormwater program area as a national enforcement priority program due to runoff concerns. FIFRA: OECA cannot provide overall compliance information for the FIFRA program area because it does not have data for the universe of 2,246,512 regulated entities. OECA staff said that, according to statutes, States have primacy in implementing the FIFRA enforcement and compliance program. OECA provides grants to States, and States then report the number of inspections and violations found. However, OECA cannot demonstrate levels of programmatic compliance based on the results of States' targeted inspections. TSCA: OECA cannot determine overall compliance for the Core TSCA universe because OECA can only make compliance determinations by conducting inspections. According to OECA staff, OECA has not inspected even a small portion of the 3,758,176 Core TSCA regulated entities. Core TSCA enforcement staff uses the Enforcement Response Policy to assess the gravity of violations and determine appropriate enforcement actions on a case-by-case basis. After concentrating compliance monitoring on larger manufacturing entities for many years, OECA staff indicated that they would pay more attention to providing compliance assistance to smaller entities. However, OECA cannot presently provide compliance information on the Core TSCA universe. SDWA: EPA reports on the status and progress of the SDWA program to Congress annually, and SDWA has been a national enforcement and compliance priority for many years. OECA generated programmatic compliance rates to report changes in compliance in the SDWA Public Water Systems - Community Systems universe. However, OECA generated the rates with incomplete violations data, resulting in underreporting. States conduct the majority of SDWA inspections and enter information and violations in the Safe Drinking Water Information System database. Both EPA OIG and OECA audits3 of that database have shown that States do not provide consistently high quality violations data and do not enter many violations in the database. RCRA: OECA does not know the universe or compliance levels for the small quantity generator universe and therefore cannot provide information to show levels of RCRA compliance. As discussed in Chapter 2, RCRA facilities can change generator size status from small to large. However, if they do not notify EPA, the status change will not be discovered until the facility is inspected. 3 EPA OIG Report No. 2004-P-0008, EPA Claims to Meet Drinking Water Goals Despite Persistent Data Quality Shortcomings, March 5, 2004; and OECA 2002 National Public Water System Compliance Report, Providing Safe Drinking Water in America, December 2004. 18 ------- According to OECA staff, not all RCRA facilities are inspected annually, and a significant percentage of small quantity generator facilities have never been inspected. OECA Is Not Sufficiently Transparent with Available Compliance Data While OECA generates many types of compliance reports for management use from the data provided by States and EPA regions, OECA does not release many of these reports to external stakeholders. The internal reports include monthly management reports, watch lists, and performance analyses. From the information provided by States and regions in Agency databases, OECA develops data, statistics, rates, and analyses for internal OECA and regional management. OECA organizes analyses and data by region and State. Some of the rates that measure noncompliance include significant noncompliance rates, recidivism rates, time to attain compliance, and "hit rates" that show the number of significant noncompliance violations found from inspections. OECA provides performance results annually to external audiences in measures such as pounds of pollutants reduced, dollars of fines and penalties, number of inspections, and compliance assistance provided. The external reports and publicly reported data do not demonstrate results in terms of levels of compliance or noncompliance across regulated program universes. OECA provides facility- specific compliance information on approximately 800,000 facilities in its public Enforcement and Compliance History Online (ECHO) database, but the system does not provide programmatic compliance information. Transparent regulatory information allows Congress and the public the opportunity to review and analyze compliance data. Also, State compliance data can show regional and geographical differences in environmental problems, approaches, and solutions. EPA has a longstanding commitment to transparency, which can improve data quality because States might more timely and carefully enter enforcement and compliance data that is open to public scrutiny. Through fiscal 2002, OECA publicly released most of the rates and data in its Measures of Success Reports, but OECA currently releases this information only to Agency personnel. According to OECA staff, they do not publicly release internal rates and data because: • People may misunderstand and misrepresent the rates and data derived from targeted inspections and compliance monitoring, since the data do not represent the noncompliance levels of the whole regulated universe. • Congress and the public may be unduly alarmed by the high level of noncompliance because inspections are targeted based on high risk or suspected noncompliant activity. 19 ------- • Experts could create programmatic compliance rates from the facility-specific information in the ECHO database; therefore, OECA does not need to generate programmatic information. • States do not want to publish statistics because people may make comparisons among States and draw incorrect inferences without the proper context. • OECA must keep some information confidential for enforcement actions. While we agree that some information must be kept confidential for conducting enforcement actions, we do not agree with the other reasons. If the data is based on targeted inspections, or only represents some States or regions, OECA can provide explanatory notes to avoid misrepresentation and prevent Congress and the public from misunderstanding what the data represents. Providing significant noncompliance rates, with appropriate qualifying explanations, can still demonstrate the success of pursuing known violators. The credibility of EPA's decisions can increase when external audiences understand how the Agency came to its decisions. Further, even experts cannot generate programmatic compliance information from the facility specific compliance information in the ECHO database because, according to OECA statements, this database only contains compliance data on approximately 800,000 entities, whereas OECA has described its regulated universe as approximately 41 million entities. Recommendations We recommend that the Assistant Administrator for Enforcement and Compliance Assurance: 3-1 To show the results of its national enforcement and compliance program in maximizing compliance with environmental statutes, develop and publish information that demonstrates changes in compliance levels within the regulated universe, by program areas. Include any appropriate explanations of data quality issues or data caveats. 3-2 Share compliance data and analyses with external stakeholders to provide a better understanding of programmatic compliance levels; include explanatory notes as needed to ensure proper representation and understanding. Agency Comments and OIG Evaluation OECA did not concur with Recommendation 3-1, and as an alternative proposed that it would share with the public any statistically valid noncompliance rates developed in the past year and any that OECA will develop in the future. This alternative is not adequate considering the limitations OECA itself acknowledged in its September 29, 2004, memo to us requesting assistance in developing methodologies for producing statistically valid noncompliance rates. The Agency concurred with Recommendation 3-2 to increase transparency. OECA included 20 ------- two tables with lists of reports that we included as Appendix F. However, OECA did not agree to publish other compliance information that shows significant noncompliance rates, 'hit' rates, recidivism, and the time taken to bring violators to compliance. Appendix E includes the Agency's comments on specific recommendations and other general comments, and our evaluation of those comments. 21 ------- Appendix A OECA's Regulatory Universe Identification Table - Executive Summary Our analysis of OECA's September 2001 Regulatory Universe Identification Table found that OECA listed 14 statutes that had one or more programs (or programmatic descriptions). OECA included "federal facilities," "tribes," "wetlands," "UST" (underground storage tanks), and "multi-media" as statutes even though they are not, by themselves, statutes. We identified 58 separate program areas under the 14 statutes. Further, OECA consulted 36 different sources to obtain the universe numbers for the 58 programs. Data sources came from both within and outside of EPA and included enforcement and media-specific databases, final rule documents, Bureau of Indian Affairs data, and Census information. The universe table also included estimates when no database or source existed for a given regulatory area. In this case, according to OECA staff, the estimate served as the best source for the universe number. In the cover letter that accompanied OECA's universe table, OECA explained that the total universe of entities in EPA's compliance and enforcement databases is approximately 1.6 million. OECA derived the rest of the universe numbers from various other sources, including Agency publications and databases. We noted that OECA incorrectly totaled its universe of regulated entities. While the Executive Summary states that OECA's universe is 41.1 million entities, we found, after reviewing the figures, that OECA's 2001 regulatory universe should have been listed as 41.8 million entities. We also found other math transcription errors while analyzing OECA's universe table, including two instances where the number listed in OECA's universe table for a particular program did not match the number OECA listed in the executive summary for that same program. For example, under the CWA National Pollutant Discharge Elimination System statute, OECA lists a number of 29,688 for "biosolids/sludge active POTWs" (Publicly Owned Treatment Works) in the universe table, but lists that figure as 29,668 in the Executive Summary. We transcribed the following table and endnotes directly from OECA's September 2001 Regulatory Universe Identification Table. To maintain the integrity of the source document, we did not spell out abbreviations or correct any spelling, punctuation, or spacing errors. 22 ------- OECA Regulatory Universe Identification Table - Executive Summary Statute Programmatic Description Universe Estimate Universe in Database CAA Major Stationary Sources (AFS) 23,868 Minor Stationary Sources (AFS) 96,866 Synthetic Minor Stationary Sources (AFS) 17,248 Chlorofluorocarbon Sources (AFS) 4,292 Asbestos Work Practice Standards Notifications (NARS) 96,954 Section 112(r) Risk Management Plans 15,081 Mobile Sources 324,830 CWA Major permitted facilities (PCS) 6,599 NPDES Minor Permitted facilities (PCS) 87,844 Pretreatment significant industrial users 30,000 Biosolids (Sludge) (PCS) 29,668 5,143 Wet Weather facilities - Stonnwater permits 380,000 41,613 CAFOs 12,660 Wetlands Total acreage 105.5 M RCRA Treatment, Storage, and Disposal (TSD) Facilities* 2,393 2,973 Large Quantity Generator (LQG) Facilities* 20,876 30,231 Small Quantity Generator (SQG) Facilities 202,965 Conditionally Exempt Small Quantity Generator (CESQGs) Facilities 108,780 Transporter Facilities 13,223 RCRA Non-Notifier Facilities 7,076 "Other" Facilities in RCRAInfo 230,870 SDWA Public Water Systems: Community Systems 54,101 Public Water Systems: Non-Transient Non- Community Systems 20,429 PWS: Transient Non-Community Water Systems 93,034 UIC: Class I Wells 533 UIC: Class II Wells 156,215 UIC: Class III Wells 15,452 UIC: Class IV Wells 4 UIC: Class V Wells 650,000 207,429 23 ------- Statute Programmatic Description Universe Estimate Universe in Database FIFRA Total Registered Pesticides Total Registrants with active pesticide registrations 20,000 1,963 Pesticide Production Establishments (Active) 12,442 Commercial Pest Control Finns 40,000 Agricultural Sector Total Number of Farms Total Amount of Farm Acreage 2,194,070 947,340,000 Total Number of Agricultural Workers 4,135,315 Total Number of Farms And Business Sites Regulated Under Pesticide Programs 2,246,512 EPCRA Toxic Release Inventory (TRI) Reoortine Universe - Total number of facilities 22,639 Total Number of 311/312 Facilities 559,600 TSCA Core TSCA - Chemical Manufacturers - Other manufacturers, processors, distributers, users and exporters 13,513 3,758,176 Polvchlorinated Biohenvls (PCBs) - Storase & Disposal Sectors Commercial Storers Commercially Permitted Disposal Companies Commercial Decontamination Facilities Scrap Metal Oven Facilities 89 48 4 7 PCBs- Utilitv/Non-utilitv Waste Generators Electrical Utility Industry Total Number of Non-Utility Establishments with PCB-contaminated Equipment Company-registered Transformer Facilities - Transformers in operation 3,215 6,404,244 2,500 20,742 Lead-based (Pb) Paint - 402/404 Total Training Program Providers Lead-based Paint activities - Finns - Individual Workers 177 4,069 17,249 Pb-based oaint - 1018 - Targeted Housing - Real Estate Finns - Real Estate Agents - Property Managers 99,500,000 92,000 352,000 243,000 24 ------- Malulc Program malic Description I ni\cisc I'.siimalc Pb-based Paint - 406 - Annual Number of Target Units Subject to Renovation Requirements - Total Number of affected Renovation Firms - Total Number of Contractor Personnel most likely to be involved in Pb-based paint renovation activities: 18,500,000 482,000 2,272,000 Total Number of Entities Regulated under Asbestos AHERA/MAP Programs 5,356,984 Federal Facilities Number of Federal Facilities Subject to Environmental Regulations 11,670 Tribes Number of Federally Recognized Tribes 556 Total Population 1,400,000 Total Acreage of Tribal Lands 54,893,267 Federal, industrial, commercial, and municipal facilities located on Tribal areas 1,645 CERCLA National Priorities List (NPL) Sites (non-federal facility) - Proposed, Final & Deleted 1,349 Total number Non-NPL Sites (non-federal facility) 8,884 NPL Federal Facilities - Total number of sites (Proposed, Final & Deleted) 176 Non-NPL Federal Facilities - Total number of sites 781 OPA Oil Storage Facilities Subject to SPCC Requirements 469,289 UST Active Underground Storage Tanks 713,666 Multi- media Facilities tracked by two or more media programs (CAA, CWA, or RCRA) 15,733 The total universe estimate of entities under the responsibility of EPA's enforcement and compliance programs is approximately 41.1 million (the sum of all bolded numbers in the universe estimate and universe in database columns). The total universe of entities maintained in EPA databases is approximately 1.6 million. This is the sum of the numbers in the universe in database column, except for the RCRA TSD and LQG numbers, which were summed from the universe estimate column (see the asterisked footnote below regarding the RCRA data), and excluding the federal facility, tribal, and multimedia numbers, which are accounted for in the media-specific counts. Note: In summing entities across media categories, an unknown amount of multiple-counting (i.e., of the same facility regulated by several EPA programs) is inevitable. Universe Estimate provides a documented estimate of all facilities/entities, taken from Agency publications, ICRs, databases, and other sources. Universe in Database provides a snapshot of the number of facilities/entities that are tracked by EPA program databases. Please refer to the full version of this table for detailed information about each universe category and estimate. *: Both universe estimate and universe in database numbers are generated from RCRAInfo data. The universe estimate number represents the subset of handlers with inspection, violation, and/or enforcement activity within the past five years ("active" per FY2001 RECAP definition). September 21, 2001 Version 4.exec USEPA / OECA / OC / EPTDD / IUTB 25 ------- Appendix B Environmental Statute Information and Program-Specific Facility Size Classifications The following briefly summarizes the environmental statutes that cover our sample program areas. Where applicable, we described how OECA defined large and small, or major and minor entities (FIFRA and TSCA programs do not distinguish between large and small entities). CAA Background The Clean Air Act (CAA) created a national program to control the damaging effects of air pollution. Large/Small Entity Classifications Stationary sources of air pollution include factories, processing plants, chemical plants, refineries, and utilities. Major Stationary Source: Any stationary source or group of sources located in a contiguous area, with potential to emit more than 100 tons of pollutant per year, or, for hazardous air pollutants, 10 tons per year. Minor Stationary Source: Any stationary source that has the potential to emit air pollutants at less than the major stationary source thresholds, or is not classified as a synthetic minor source, and is subject to Federal regulations. Synthetic Minor Source: Any stationary source that has the capability to emit air pollutants at or above the major stationary source threshold but is considered a minor source because it places physical or operational limitations on its capacity to emit pollution. CWA Background The Clean Water Act (CWA) is the primary Federal law that protects the Nation's waters, including lakes, rivers, and coastal areas. Large/Small Entity Classifications The National Pollutant Discharge Elimination System (NPDES) program controls water pollution by regulating point sources that discharge into U.S. waters. The majority of NPDES facilities are classified as either: Major Permitted Facilities: Municipal or industrial facilities that directly discharge effluent, based on design flow or a qualifying permit rating score. Minor Permitted Facilities: Municipal or industrial facilities that directly discharge effluent, but are not designated as major dischargers. Stormwater Permits: Permits issued through a two-phased national program for regulating non-agricultural sources of stormwater discharges and run-off. Phase I required stormwater permits for large municipalities, industrial plants, and commercial construction sites. Phase II requires permits for smaller municipalities and smaller construction sites. FIFRA Background The Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) regulates the sale, distribution, and use of pesticides through a registration system. 26 ------- RCRA Background The Resource Conservation and Recovery Act (RCRA) regulates hazardous and municipal and industrial solid waste generated nationwide. RCRA Subtitle C establishes a system for controlling hazardous waste from the time of generation until ultimate disposal and includes regulations for the generation, transportation, treatment, storage, or disposal of hazardous wastes. Large/Small Entity Classifications The statute defines three categories of generator facilities based on volume: Large Quantity Generator Facilities: • Greater than 1,000 kg of hazardous waste per calendar month (approximately 2,200 lbs); or • Greater than 1 kg of acutely hazardous waste per calendar month (approximately 2.2 lbs). Small Quantity Generator Facilities: • Between 100 kg (approximately 220 lbs) and 1,000 kg of hazardous waste per calendar month; and • Accumulate less than 6,000 kg (approximately 13,200 lbs) of hazardous waste at any time. Conditionally-Exempt Small Quantity Generator Facilities: • Less than 100 kg of hazardous waste per calendar month; or • Less than 1 kg of acutely hazardous waste per calendar month; and • Accumulate less than 1,000 kg of hazardous waste, 1 kg of acute hazardous waste, or 100 kg of any residue from the cleanup of a spill of acute hazardous waste at any time. SDWA Background The Safe Drinking Water Act (SDWA) ensures public health protection by public water systems complying with all health-based standards, including monitoring and reporting requirements. Through the Public Water System Supervision program, EPA implements and enforces drinking water standards to protect public health. Large/Small Entity Classifications EPA defines Public Water Systems, by type and number of people they serve, as either community systems or non-transient non-community systems. A community system is a public water system that supplies water to at least 15 service connections or 25 people year-round in their primary residences. Non-transient non-community systems serve at least 25 of the same persons over 6 months per year. Transient non-community water systems serve at least 25 persons (not the same 25 persons) over 6 months per year. EPA does not regulate drinking water wells that supply water to fewer than 25 people. TSCA Background The Toxic Substances Control Act (TSCA) authorizes EPA to secure information on and regulate all new and existing chemical substances, as well as control any substances determined to cause unreasonable risk to public health or the environment. "Core" TSCA regulates industrial chemicals, exclusive of Polychlorinated Biphenyls, lead-based paint, and asbestos. 27 ------- Appendix C Details on Scope and Methodology To gain a general understanding of OECA's regulated universe and associated enforcement and compliance activities and resources, we reviewed policy and guidance documents, including: ~ Supplemental Information for EPA Fiscal Year 2003 Annual - Integrity Act Report and Key Management Challenges ~ Memoranda of Agreement Guidance Priorities - Fiscal Years 1996-2007 ~ Fiscal Year 2005 Budget ~ Fiscal Year 2005-2007 National Program Guidance ~ Enforcement Response Policies for program area sample ~ Fiscal Year 2005 Annual Performance Plan and Congressional Justification ~ End-of-Year Accomplishments Reports We also interviewed staff in the following OECA offices: Office of Site Remediation Enforcement Office of Civil Enforcement ~ Air Enforcement Division ~ Water Enforcement Division ~ RCRA Enforcement Division ~ Toxics and Pesticides Enforcement Division Office of Compliance ~ National Planning Measures and Analysis Division ~ Enforcement Targeting and Data Division ~ Compliance Assessment and Media Programs Division ~ Compliance Assistance and Sector Programs Division We reviewed relevant reports, including: EPA Office of Inspector General ~ EPA's Key Management Challenges (2001, 2002, 2003) ~ Information Technology: Unreliable Data Affects Usability of DOCKET Information (2002) Government Accountability Office ~ Environmental Information: EPA Needs Better Information to Manage Risks and Measure Results (2000) ~ Human Capital: Implementing an Effective Workforce Strategy Would Help EPA to Achieve Its Strategic Goals (2001) ~ Major Management Challenges and Program Risks - Environmental Protection Agency (2003) National Academy of Public Administration ~ Evaluating Environmental Progress: How EPA and the States Can Improve the Quality of Enforcement and Compliance Information (2001) ~ Environmental Performance Measures in a Federal System (2000) Environmental Council of States ~ Report to Congress - State Environmental Agency Contributions to Enforcement and Compliance (2001) 28 ------- Sample of Six Regulatory Program Areas We reviewed and summarized environmental statutes for our sample program areas (see Appendix B for descriptions). The table below lists the 2001 universe numbers for the six environmental statutes covering our sample program areas, with specific program areas reviewed in bold type. Our six sample areas comprise just over 16 percent of OECA's 2001 total universe of regulated entities. We obtained the numbers from the Executive Summary OECA included with its 2001 Regulatory Universe Identification Table (see Appendix A). Statute Programmatic Description 2001 Universe Number CAA Major Stationary Sources 23,868 Minor Stationary Sources 96,866 Synthetic Minor Stationary Sources 17,248 Others (includes mobile sources) 441,157 CAA Total 579,139 CWA NPDES - Major Permitted Facilities 6,599 NPDES - Minor Permitted Facilities 87,844 NPDES - Stormwater Permits 380,000 NPDES - Others (includes biosolids and Concentrated Animal Feeding Operations) 72,328 CWA Total 546,771 RCRA Treatment, Storage, and Disposal Facilities 2,393 Large Quantity Generator Facilities 20,876 Small Quantity Generator Facilities 202,965 Conditionally Exempt Small Quantity Generator Facilities 108,780 Others (includes transporters, non-notifiers, and other facilities included in EPA's RCRA data system) 251,169 RCRA Total 586,183 SDWA Public Water Systems: Community Water Systems 54,101 Public Water Systems: Transient and Non-Transient Non-Community Systems 113,463 Others (includes underground injection control wells) 822,204 SDWA Total 989,768 FIFRA Total Number of Farms and Business Sites Regulated under Pesticide Programs 2,246,512 FIFRA Total 2,246,512 TSCA Core TSCA - Chemical Manufacturers 13,513 Core TSCA - Other Manufacturers, Processors, Distributors, Users and Exporters 3,758,176 Others (includes facilities related to Polychlorinated Biphenyls and lead-based paint) 31,341,591 TSCA Total 35,113,280 Sample Prograrr Area Total 6,755,868 Table Total 40,061,653 29 ------- To answer both the universe and compliance objectives, we judgmentally selected the six sample program areas, as case studies, from OECA's 2001 Regulatory Universe Identification Table. This sample was based on information gathered from our preliminary research. A preliminary analysis of the Regulatory Universe Identification Table showed that OECA's universe of 41 million entities included a diverse and complex mix of entities of varying sizes and types. Small entities made up a much greater part of the universe than the major and large entities. A review of OECA's planning documents also showed that a handful of programs were selected as national priorities, and others were described as core programs. While OECA is ultimately responsible for regulating all 41 million entities, it focuses its regulatory attention on national priority program areas, while the States are primarily responsible for core programs. To conduct a balanced evaluation of OECA's knowledge of the regulated universe, it was important to select a sample which reflected the mix of program areas for which OECA is both ultimately responsible and it had represented as falling under its regulatory authority in its Regulatory Universe Identification Table. OECA's Knowledge of Its Regulated Universe To determine OECA's knowledge of the composition and size of its regulated universe, we analyzed OECA's 2001 regulated universe table. We compared 2001 universe figures to 2005 data provided by OECA staff for six sample program areas. We met with staff in EPA's Office of Water to discuss OECA's universe number for the CWA-NPDES stormwater program. We also interviewed OECA staff to understand any challenges associated with obtaining accurate universe numbers for our sample regulatory programs, as well as to discuss the benefit of knowing the composition and size of the regulated universe. We did not independently verify the accuracy of data provided by OECA. We did review EPA's fiscal 2005 Annual Performance Plan for Goal 5, where EPA identified data problems, limitations, and reliability issues in OECA's databases and data. We used that information to identify limitations in the reliability of OECA's universe numbers. In addition, we applied a methodology described in GAO's October 2002 report, Assessing the Reliability of Computer Processed Data (GAO-03-273G). We reviewed additional data quality materials, including OECA's: ~ "Enforcement and Compliance Reporting Process for FY 2004" Memorandum (March 12, 2004) ~ "Final Enforcement and Compliance Data Quality Strategy" (March 25, 2002) ~ Data quality quarterly reviews ~ "Ensuring Integrity of Reporting Enforcement and Compliance Data" Memorandum (May 6, 2003) ~ Information from OECA's Web site on various enforcement and compliance databases Changes in Regulatory Compliance To determine how OECA determines and reports levels of compliance in its regulated universe, we reviewed general information on compliance monitoring as well as how OECA defines compliance through its enforcement response policies. We interviewed representatives from the National Academy of Public Administration, the Environmental Law Institute, and the Environmental Compliance Consortium on how OECA measures changes in regulatory compliance. We interviewed OECA managers and staff to understand how they determine and 30 ------- report compliance in our sample program areas, as well as any challenges associated with demonstrating changes in regulatory compliance. We discussed conditions we identified during the review of the sample with OECA managers to determine whether the problems were limited to the program areas, or whether they were systemic and OECA-wide conditions. In instances where the condition was systemic, we reported it as a general condition. We analyzed the cause of the problem, and developed recommendations to address the cause of the condition. 31 ------- Appendix D Agency Response Memorandum to Draft Report as o \ c ''L PRCrt^ UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF ENFORCEMENT AND COMPLIANCE ASSURANCE August 1, 2005 MEMORANDUM SUBJECT: The Office of Enforcement and Compliance Assurance's Agency Response to the Draft, "Evaluation Report: Limited Knowledge of the Universe of Regulated Facilities Impede's EPA's Ability to Demonstrate Changes in Regulatory Compliance," dated June 30, 2005 FROM: Thomas V. Skinner Acting Assistant Administrator TO: Jeffrey K. Harris Director Program Evaluation, Cross-Media Issues Office of Inspector General Introduction Today, on behalf of the Office of Enforcement and Compliance Assurance (OECA) and as the designated "Action Official," I am forwarding to you our consolidated Agency Response (Response) regarding the Office of Inspector General's (OIG) draft "Evaluation Report: Limited Knowledge of the Universe of Regulated Facilities Impede's EPA's Ability to Demonstrate Changes in Regulatory Compliance," dated June 30, 2005. In accordance with the instructions provided in your June 30, 2005 memorandum, the Response addresses the factual accuracy of the draft Evaluation Report. Consistent with those instructions, the Response also specifically indicates whether OECA concurs with each of the recommendations proposed by the OIG. Further, to the extent that action has already been initiated or planned to address issues identified in the draft Evaluation Report, the Response 32 ------- specifically identifies those actions that have been initiated or planned. Finally, your June 30, 2005 memorandum expressly states that the "final report will include an assessment of [the] comments" made in the Response. Consequently, I am specifically requesting that this memorandum and the attached Response be attached to, and be made a part of, the final version of the draft Evaluation Report. Principal Concerns While we appreciate the effort made by the OIG to provide a useful review of the data we had developed in 2001 to better understand the regulatory universe, we were troubled by a number of practices which were used in this evaluation. We would ask that the OIG consider our concerns both for purposes of possible revisions to this evaluation and as suggestions about the conduct and content of future evaluations. Given the amount of time that the staff and managers of OECA and OIG are investing in these evaluations, it is important that the evaluations focus on the most important issues, provide clear and meaningful findings, and yield recommendations that add value and improve the effectiveness of our Agency programs. Here are the general comments we offer about the report as a whole. Comments on specific findings and recommendations are included in the attached response. 1. Limitations of Judgmental Sampling. We are very concerned that judgmental sampling of six segments of the regulated universe produced results that are biased (i.e., not representative of the larger population being examined). Judgmental sampling can only characterize the types of problems that might exist in the full set of universe data, but it cannot quantify the size of the problem for the full set of universe data. While the OIG states that it "cannot use our judgmental sample to generalize to OECA's entire population," that caveat did not constrain the OIG from making a very sweeping set of recommendations to address the issues it identified through the use of a technique with very significant limitations. (See pages 2 and 3 of our attached response for a full description of our concerns.) 2. Findings Broader Than the Supporting Evidence. In several instances, the evaluation presents findings that are much broader than the issue the OIG is raising or the evidence it is providing. The primary example of this problem is the finding entitled, "OECA Does Not Focus Activities on Majority of Facilities," on page 8 of the draft evaluation. While it may be true that in recent years enforcement actions have focused increasingly on larger facilities associated with OECA's national enforcement priorities (though we note that the OIG presents no quantitative evidence of this), OECA's compliance assistance program since its inception has been focused on serving smaller facilities which often lack the resources or expertise to achieve and maintain compliance. Of the 14 OECA-sponsored online Compliance Assistance Centers (visited more than one million times last year), 12 are serving sectors comprised primarily of small businesses. In addition, other compliance assistance initiatives in sectors dominated by small businesses (e.g., dry cleaners, construction, auto repair) have See OIG Response in Appendix E, Note 1 See OIG Response in Appendix E, Note 2 33 ------- been undertaken to provide tailored assistance to meet the specialized needs of small businesses. (See pages 8 and 9 of our attached response for a full description of our concerns.) A second example of the IG drawing findings which are too broad is its assertion that "OECA Lacks Transparency in Sharing Data," on page 15 of the draft report. In our response we point out the many different types of compliance data we make available to the public. We describe the Enforcement and Compliance History Online (ECHO) web site which has been used by two million visitors to find facility-specific inspection and enforcement data. We also detail in Table 3 (pages 19-21), the various types of compliance data we make publicly available for all the major EPA programs. The OIG may want more transparency, but to say that OECA lacks transparency in sharing data is inaccurate. 3. Recommendations Broader than the Findings Being Addressed. Several of the recommendations seem disproportionate to the problem they are meant to solve. The primary example of this disproportionality is Recommendation 2.4, specifically the portion of the recommendation directing OECA to require states to "track, record, and report data for facilities over which states have regulatory responsibility... [and] develop a multi-state, multi-program pilot program for tracking, recording, verifying, and reporting of state data." Because the OIG has identified that OECA has not updated its universe table since 2001 and a determination (through a non-representative judgmental sample of six regulated populations) that the universe table must be updated, it now recommends a very large and resource-intensive goal of increasing state reporting. In our response, we describe the significant regulatory, procedural and resource barriers to increasing state reporting. We believe that the "solution" (i.e., significant new data reporting requirements for states) is a complete mismatch and way out of proportion to the "problem" (i.e., the need for more reliable universe data). (See page 6 of our response for a full description of our concerns about increasing the reporting burden on states.) 4. Exclusion of Relevant Information. We believe the evaluation at times ignores important information about OECA programs and leads to poorly conceived findings and recommendations. For example, in the section about OECA's description of its role, the evaluation does not provide a complete and accurate description of the relative roles of EPA and the states, opting instead for a very simplistic view which seems to color the OIG's judgment about the findings and recommendations in the report. Additionally, as described above, the OIG fails to take into account that OECA's compliance assistance program is and always has been focused on sectors comprised of small facilities. As a result, the OIG makes an inaccurate finding that "OECA Does Not Focus Activities on Majority of Facilities." See OIG Response in Appendix E, Note 3 See OIG Response in Appendix E, Note 4 34 ------- 5. Disconnect from Procedural /Resource Barriers and Competing Demands. In some of the recommendations in this evaluation, the OIG does not take into account the regulatory and procedural obstacles that would be encountered and the significant resource commitments that would be necessary for implementation. Similarly, the OIG seems never to consider existing demands already placed on organizations and the relative importance of those demands versus proposed OIG recommendations. The OIG's recommendation for increased state reporting of data would require a large investment of personnel and energy to overcome the regulatory and procedural barriers and resource shortfalls described in our response (see page 6). Moreover, it could well disrupt OECA's and the states' progress toward modernizing single-media data systems and incorporating them into the Integrated Compliance Information System (ICIS). We look forward to working with you to finalize this report and on future evaluations of OECA programs. cc: Phyllis Harris Michael Stahl Walker Smith See OIG Response in Appendix E, Note 5 35 ------- Appendix E Agency Comments to OIG Recommendations and OIG Evaluation The Agency comments consisted of two parts - a transmittal memorandum with summary discussion, and a document with detailed comments. We included the memorandum as Appendix D; the detailed comments are available on the EPA OIG Web site at www.epa.gov/oig. Two lists of reports that were part of OECA's detailed comments are provided in Appendix F. Appendix E includes each specific OIG recommendation, the Agency's comments on those recommendations, and our evaluation of comments from both Agency comment documents. In this appendix, we first evaluate the Agency's comments from the detailed comment document to our recommendations, and follow this in Notes 1 through 5 with our responses to the five points discussed in the Agency's memorandum in Appendix D. OIG Recommendation 2-1: Biannually update publicly released universe figures by tracking and recording the number of entities over which it has oversight and primary regulatory responsibility. Agency Comments: Non-concur. As an alternative, only universe figures updated within the previous two years will be released to the public. OECA will begin the process of updating universe figures for populations associated with its national priorities and complete that update within six months. Further, OECA will remove the 41 million universe figure from its National Program Guidance and not use it in any future public documents. OIG Evaluation: Updating the universe numbers for populations associated with the handful of national priority program areas is a beginning. However, it will not indicate the complete universe of regulated entities for which OECA is responsible for ensuring compliance. Not releasing the number of regulated facilities will subject EPA to the same July 2001 GAO criticism noted in Chapter 1 of this report. GAO had stated that OECA could not demonstrate "the universe of entities subject to regulation under federal environmental laws." OECA has not provided any reasons to show there is more value to updating the universe of the five national priority areas in lieu of all program areas. Additionally, OECA has not provided any timetable for updating universe information on a regular basis. OECA officials said OECA management has assigned other projects higher priority. Nevertheless, as OECA itself has recognized, knowledge of the size and character of the regulated universe is fundamental for a regulatory body, and we continue to recommend that OECA assign a higher priority to regularly updating that universe. OIG Recommendation 2-2: When producing its biannual universe update, use reliable data to generate complete and current universe numbers that meet national data quality standards similar to those outlined in OECA's Final Enforcement and Compliance Data Quality Strategy. 36 ------- Agency Comments: Concur. When updating universe figures as described in response to recommendation 2.1, OECA will use complete, accurate, and current data in accordance with national data standards including those in OECA's Data Quality Strategy. OIG Evaluation: OECA qualified its concurrence to ensure the quality of the universe data by referencing its response to Recommendation 2-1 on updating the numbers for national priority program areas only. We consider this a good start, but it will not meet the objective of knowing the full universe of regulated entities. OECA also commented that we should describe the data problems in universe numbers as data gap problems, explaining that data gaps exist when the statutes do not specifically require States to report data to EPA. OECA indicated that requiring more data would be burdensome to States due to shortfalls in State resources. OECA explained its use of estimates where data gaps existed, and objected to our using GAO criteria to evaluate data quality. We have modified the text to provide additional sources we used to find weaknesses in data quality. We also revised the discussion of the problems we found in the estimated numbers in OECA's universe table. We cannot provide an opinion on the issue of resources as we have not analyzed OECA's resource allocations and the process by which management assigns relative weight to tasks as part of this evaluation. OIG Recommendation 2-3: Describe OECA's enforcement and compliance role in relation to States and other partners when the Agency publicly releases universe figures. Agency Comments: Concur. OECA will develop and add a more precise description to relevant documents to more clearly explain the respective roles and responsibilities of EPA and the states in maximizing compliance in the regulated universe. OIG Evaluation: We accept the concurrence. OIG Recommendation 2-4: Develop an objective of having the most up-to-date and reliable data on all entities that fall under its regulatory responsibility. OECA should adopt the goals of requiring States to track, record, and report data for entities over which States have regulatory responsibility. To achieve this goal, OECA should develop a multi-State, multi-program pilot program of collecting data that States track, record, verify, and report. Agency Comments: Non-concur. While OECA believes that it currently subscribes to an objective of having current and accurate facility data, we believe this recommendation fails to take into account the significant procedural and resource barriers which impede expanded collection and reporting of data from states. Further, we believe that given the resource constraints under which the states are operating, a multi-state, multi-program pilot would be resisted vehemently by the states. As an alternative, OECA and the states will continue on their current path of modernizing single-media data systems and integrating them into ICIS (Integrated Compliance Information System). The modernization process is being done in full 37 ------- consultation with states, on a schedule developed jointly by EPA and the states, and will achieve many of the improvements sought by the OIG. OIG Evaluation: We are aware of resource and procedural barriers in collecting additional information from States, and therefore recommended that OECA obtain knowledge in incremental steps by conducting pilot data collection programs. The pilot might reveal that data are readily available in some of the States. Also, Recommendation 2-4 addresses the problem of smaller and minor entities. OECA still needs to indicate how the modernization of OECA's data systems will address the condition under discussion. OIG Recommendation 2-5: Request that EPA program offices analyze and report to OECA the cumulative impact of violations by regulated entities that pollute below the thresholds of major or large entities. OECA should use any cumulative impact analyses conducted by program offices to inform OECA's management decisions. Agency Comments: Non-concur. As an alternative, OECA will request from program offices any and all currently available data and analyses of cumulative impact of small facilities. OECA will make this request within sixty days of the date of the final version of this OIG evaluation. OECA will use this information to identify emerging problems in sectors comprised of smaller facilities. OIG Evaluation: Collecting available analyses from program offices, and using the information to identify emerging problems in smaller facilities, is a move in the right direction. OECA may find that many such analyses exist within program offices. However, we recommend that OECA also request analyses for program areas where such analyses have not been done. As shown in Chapter 2, small and minor entities comprise a large part of the regulated universe; thus, it is important that OECA know their cumulative impact. OECA occupies a distinctive position in EPA in that it cuts across the Agency's program offices. At a minimum, OECA can use these data and analyses to support its decisions to focus its enforcement and monitoring activities. OIG Recommendation 3-1: To show the results of its national enforcement and compliance program in maximizing compliance with environmental statutes, develop and publish information that demonstrates changes in compliance within the regulated universe, by program areas. Include any appropriate explanations of data quality issues or data caveats. Agency Comments: Non-concur. As an alternative, OECA will share with the public any statistically-valid compliance rates it has developed in the past year and any statistically valid rates it will develop in the future. OIG Evaluation: We reiterate that OECA should develop and publish information that shows changes in the levels of compliance within various program universes. OECA's proposed alternative to publish only statistically valid rates falls short of the recommendation, considering the limitations OECA itself stated in its request for assistance from us in developing methodologies for producing statistically valid 38 ------- noncompliance rates. In the detailed response to this report (available on the EPA OIG Web site at www.epa.gov/oig). OECA listed a total of 10 projects to develop statistically valid noncompliance rates from Fiscal Years 2000 through 2006. Each of the projects deals with a very limited sector or population. Given the limited number and scope of OECA's statistically valid noncompliance rates, releasing only those rates would show extremely limited results of OECA's efforts to maximize compliance in the regulated universe. Additionally, we note that OECA's response to Recommendation 3-1 is inconsistent with its response to Recommendation 2-1. In our opinion statistically valid noncompliance rates cannot be generated without knowledge of the size of a program area's universe. Therefore, OECA's response to this recommendation is untenable. OIG Recommendation 3-2: Share compliance data and analyses with external stakeholders to provide a better understanding of programmatic compliance levels; include explanatory notes as needed to ensure proper representation and understanding. Agency Comments: Concur. OECA will expand the amount of compliance data it will make available to the public on the EPA web site. The data which will be added to the web site is listed in Table 4 and includes a wide variety of information about compliance monitoring, enforcement cases, citizen complaints, cases resulting from voluntary disclosures, and more. OECA will post this data on the web site within 60 days of the release of the final version of this evaluation. OIG Evaluation: We commend OECA for its concurrence with Recommendation 3-2 and its expansion of the amount of compliance data it plans to make available on the EPA Web site. We have provided, in Appendix F, two tables from OECA's response. Releasing this additional information will increase transparency. However, we found that OECA did not include some other compliance-related rates that OECA generates. We reiterate the recommendation that OECA should publish the other compliance information that shows significant noncompliance rates, 'hit' rates, recidivism, and the time taken to bring violators to compliance. If OECA presents the information with appropriate explanations, readers should understand what the data represents. This information more directly links to compliance than most other measures OECA currently releases or plans to release. They show the results of OECA's enforcement and compliance monitoring activities on noncompliant entities and facilities. Notes on Principal Concerns In this section we provide a summary of OECA's comments (italicized) from the Agency memorandum (included in full in Appendix D), followed by our evaluation. NOTE 1 - Limitations of Judgmental Sampling: We are very concerned that judgmental sampling of six segments of the regulated universe produced results that are biased (i.e., not representative of the larger population being examined). 39 ------- OIG Response: We do not consider OECA's critique of our sampling approach and subsequent analysis valid. We used a qualitative - not quantitative methodology in this evaluation. Therefore, OECA's concerns about quantifying from our sample are not applicable. To clarify any questions on our methodology, we added more details in the report and provide some additional detail and rationale below. The use of a judgmental sample was appropriate to answer our objectives. We selected a small judgmental sample to obtain case studies for a 'qualitative' and not a 'quantitative' evaluation. We did not quantify, or extrapolate, the results from our sample across the entire universe. While we did determine that the size of the sample had changed by 35 percent between 2001 and the present, this figure clearly did not apply to the entire universe. During the course of our review, when we found a condition that was systemic, we interviewed OECA managers to determine whether the condition was limited to the program area under review or was an OECA-wide policy or practice. Only when OECA managers confirmed that it was an OECA-wide policy or practice did we present it as a broader finding with a recommendation. We included appropriate explanatory caveats in those cases. We determined that case study analysis was an appropriate research design for our review because our objective sought to determine how well OECA knew the composition and size of universes within program areas. Statistical models are best used to measure across cases, while case studies are best to investigate within a case since they allow one to explore more in depth. While a judgmental sample limits generalization, the accompanying in-depth analysis can allow the evaluator to pursue questions that arise and determine whether a problem is endemic across the universe or confined to the sample. We do not agree with OECA's opinion that the results of a judgmental sample are automatically biased. Selection bias usually occurs when the sampling includes some type of systematic error. In designing the sampling plan, we took care to capture an array of program areas for which OECA bears responsibility: small and large, across media areas, national priority and core programs, and those in which OECA shares regulatory responsibility with States versus those it does not. This sampling technique explicitly acknowledged the heterogeneity of OECA program areas. This sample is more representative of the program areas OECA included in the universe totals than if we had included only major programs or national priority programs in a sample. NOTE 2 - Findings Broader than the Supporting Evidence: In several instances, the evaluation presents findings that are much broader than the issue the OIG is raising or the evidence it is providing. OIG Response: We do not agree with OECA's comment. We revised some of the report language to ensure statements may not be interpreted to imply more than intended. We also expanded the description of our scope and methodology to explain when the finding or recommendation is broader than the specific program area in our sample. As discussed regarding Note 1, during the course of our review we found certain conditions 40 ------- that could be systemic across the entire universe for which OECA is responsible. After confirming with OECA managers, we presented each as a finding that may be broader than the specific program area in our sample. We added language to explain this. Also, as suggested by OECA, we added discussion on OECA's use of compliance assistance activities to target small businesses and entities. While OECA provides compliance assistance to many small entities, it nevertheless does not focus the majority of its compliance monitoring resources on small entities. In reviewing our sample areas we found that many of OECA's databases do not include compliance monitoring histories, or even facility identification information for many of the small entities over which OECA has oversight responsibility. OECA's response does not address this point. A second example of the IG drawing findings which are too broad is its assertion that 'OECA Lacks Transparency in Sharing Data...." OIG Response: We added Tables 3 and 4 from OECA's response to Appendix F of our report (the tables are shown in our report as Tables 1 and 2). These tables list the types of compliance information that OECA now releases. However, OECA does not publicly release compliance information that can inform Congress and the public as to whether compliance has increased across programs. While OECA makes facility-specific compliance information available on its ECHO Web site, Congress and the public do not have the ability to determine whether OECA is achieving its central goal of maximizing compliance across the regulatory programs for which it is responsible. Programmatic compliance information will allow OECA to know whether compliance is increasing or decreasing, whether its targeting strategies are successful, and whether its mix of compliance tools and methods is effective. It also provides Congress and the public the means to determine whether there are regional and geographic differences in environmental problems, approaches, and solutions. OECA already produces some compliance rates in Online Tracking Information System management reports. We urge OECA to release to the public the information it currently possesses internally, with the necessary caveats. NOTE 3 - Recommendations Broader than the Findings Being Addressed: Several of the recommendations seem disproportionate to the problem they are meant to solve. The primary example of this disproportionality is Recommendation 2-4, specifically the portion of the recommendation directing OECA to require states to "track, record, and report data for facilities over which states have regulatory responsibility... [and] develop a multi-state, multi-program pilot program for tracking, recording, verifying, and reporting of state data. " OIG Response: We do not agree with OECA's comment that the recommendations are disproportionate to the problems they are supposed to remedy. We developed the recommendations to match the breadth of the conditions and causes. If a condition is due to a systemic cause, the recommendation addresses the cause of the problem. OECA has oversight responsibilities over State compliance and enforcement activities of many small entities. OECA agrees in its response that the knowledge of the size and character of the regulated community is fundamental to a regulatory agency's effectiveness. Our analyses show that some of OECA's universe has changed. OECA has explained it has data gaps 41 ------- because States do not report on smaller entities. Therefore we concluded that OECA does not know the size, nature, or character of its full universe because of these gaps. This limits OECA's ability to exercise its oversight responsibilities, and we developed recommendations addressing this problem. Also, OECA needs a broad range of knowledge of regulated entities for which it is responsible to fully exercise oversight, and again we made recommendations accordingly. This data will provide OECA, Congress, and the public with information on entities for which EPA is ultimately responsible. NOTE 4 - Exclusion of Relevant Information: We believe the evaluation at times ignores important information about OECA programs and leads to poorly conceived findings and recommendations. OIG Response: We do not agree with OECA's comments. We gathered and reviewed extensive amounts of documentation and information, and held numerous meetings with OECA staff and managers. We also met with external stakeholders and experts and reviewed documents and reports from external sources (see Appendix C). We reported the significant conditions that we identified as a result of the evaluation, and included the information pertinent to presenting the findings and recommendations. We do not believe we omitted important information in our draft report. Nonetheless, as suggested by OECA, we added information on OECA's use of compliance assistance for interacting with small entities, as well as additional information on various other issues. OECA also commented that we offered a simplistic view of OECA's relationship with the States that resulted in mischaracterized findings and recommendations. We recognize that OECA has broad and varied relationships with States. One of these central relationships is OECA's oversight of States' program activities, and without obtaining knowledge of the full universe, OECA cannot fully exercise this oversight authority. NOTE 5 - Disconnect from Procedural/Resource Barriers and Competing Demands: In some of the recommendations in this evaluation, the OIG does not take into account the regulatory and procedural obstacles that would be encountered and the significant resource commitments that would be necessary for implementation. OIG Response: We recognize OECA's regulatory and procedural constraints in obtaining data from States that States are not required by statute to provide. However, in our opinion, it is fundamental for OECA to have an adequate range of knowledge about the entities it regulates, and OECA agrees with this central premise. Since it is critical for OECA to have this knowledge, it should pursue the best possible procedures to get this data. Due to the procedural constraints faced by OECA, we recommended that they accomplish this incrementally. Expanded universe data will ultimately provide OECA, and therefore Congress and the public, with increased information on entities for which EPA is responsible for regulating compliance. 42 ------- Appendix F Publicly Available Compliance and Enforcement Measures OECA provided the following two lists of reports that it plans to make publicly available during 2005. Tablel. Enforcement and Compliance Measures and Reports Data FY(s) Available Reports/Website in Which Measure Appears Acres of Wetlands Mitigated 2000-2004 Five-Year Trend Charts1, Numbers-at-a-Glance1 Administrative Compliance Orders 2000-2004 Results Summary1, Five-Year Trend Charts1, Numbers-at-a-Glance,ECH02 Administrative Penalties 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance, ECHO Administrative Penalty Orders 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance, ECHO Civil Case Highlights 2004 Case Highlights1, ECHO Civil Judicial Referrals 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance Civil Judicial Settlements 2000-2004 Five-Year Trend Charts, ECHO Compliance Assistance Activity 2004 Case Highlights Compliance Incentives Program 2000-2004 Results Summary Complying Actions 2003-2004 Results Summary, ECHO Criminal Case Highlights 2004 Case Highlights Criminal Defendants Charged 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Criminal Fines and Restitution 2000-2004 Results Summary, Numbers-at-a-Glance Criminal Investigations 2000-2004 Five-Year Trend Charts Criminal: Judicial Mandated Projects 2000-2004 Numbers-at-a-Glance Criminal: Pounds of Pollution Reduced, Treated or Properly Managed 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Cubic Yds of Contaminated Soil to be Cleaned Up 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Cubic Yds of Contaminated Water to be Cleaned Up 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Entities Reached through Compliance Assistance 2000-2004 Results Summary Environmental Crime Cases Initiated 2000-2004 Five-Year Trend Charts 43 ------- Data FY(s) Available Reports/Website in Which Measure Appears Environmental Homeland Security Cases Initiated 2000-2004 Numbers-at-a-Glance Facilities Resolved 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Final Administrative Penalty Orders 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance, ECHO Gallons of Wastewater/ Groundwater T reated 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance GPRA Goal 5 Civil Investigations 2004 Annual Performance Report3, Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance GPRA Goal 5 Complying Actions 2004 Annual Performance Report, Results Summary, ECHO GPRA Goal 5 Criminal Investigations 2004 Annual Performance Report, Results Summary, Numbers-at-a- Glance GPRA Goal 5 Develop and Use Compliance Rates 2004 Annual Performance Report, Results Summary GPRA Goal 5 Entities Reached through Compliance Assistance 2004 Annual Performance Report, Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance GPRA Goal 5 EPA-Assisted Inspections 2004 Annual Performance Report, Results Summary, ECHO GPRA Goal 5 Facilities with Voluntary Disclosures 2004 Annual Performance Report, Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance GPRA Goal 5 Federal Inspections 2004 Annual Performance Report, Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance, ECHO GPRA Goal 5 Pollutant Reductions 2004 Annual Performance Report, Results Summary, ECHO Incarceration 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Injunctive Relief 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a- Glance, ECHO Judicial Penalties 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance, ECHO Pounds of Contaminated Soil/Sediment to be Cleaned Up 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Pounds of Pollutants Reduced, Treated or Properly Managed 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance Linear Feet of Stream Mitigated 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Notices of Determination (NODs) 2000-2004 Five-Year Trend Charts, Number at a Glance People Protected by SDWA Enforcement 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance Referrals 2000-2004 Five-Year Trend Charts Stipulated Penalties 2002-2004 Five-Year Trend Charts, Numbers-at-a-Glance Superfund: % of Cost Recovery Statute of Limitation Cases Addressed with Total Past Greater than or Equal to $200,000 2000-2004 Five-Year Trend Charts, Numbers-at-a-Glance 44 ------- Data FY(s) Available Reports/Website in Which Measure Appears Superfund: % of Remedial Action (RA) Starts Where Settlement Reached or Enforcement Taken by the Time of the RA Start (during the FY) at Non-Federal Superfund Sites that Have Known Viable, Liable Parties 2004 Five-Year Trend Charts, Numbers-at-a-Glance Superfund: Private Party Commitments ($ in millions for past cost) 1995-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance Superfund: PRP-financed RA Starts 2000-2003 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance Superfund: PRP-lead RA Starts (%) 2000-2003 Numbers-at-a-Glance Superfund Private Party Commitments ($ in millions for future response work including cash outs) 1995-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance Supplemental Environmental Projects (SEPs) 2000-2004 Results Summary, Five-Year Trend Charts, Numbers-at-a-Glance, ECHO Statistically Valid Noncompliance Rates for Combined Sewer Overflows 2002,2004 2004 Combined Sewer Overflow Statistically Valid Noncompliance Rate Study4 'The six reports comprising OECA's annual press release for FY 2004 are: Results Summary, Numbers-at-a-Glance, Criminal Enforcement Highlights, Civil Enforcement Highlights, Compliance Assistance Highlights, and Five-Year Trends. The reports are available onEPA's Web site: http://cfpub.epa.gov/compliance/resources/reports/endofVear 2The Enforcement and Compliance History Online (ECHO) Web site (http://www.epa. gov/echo) provides facility- level compliance monitoring, compliance status, enforcement action, and penalty data from 2002-2004 for facilities regulated as Clean Air Act (CAA) stationary sources, Clean Water Act (CWA) permitted dischargers (under the National Pollutant Discharge Elimination System, and Resource Conservation and Recovery Act (RCRA) hazardous waste sites. 3The Annual Performance Reports for 1999 through 2004 are available at http://cfpub.epa.gov/compliance/resources/reports/gpra/ 'Available at http://www.epa.gov/compliance/resources/publications/data/planning/priorities/cwacsosvnrstudv.pdf 45 ------- Table 2. Enforcement and Compliance Measures and Reports Data FY(s) Available Reports in Which Measure Appears Acres of wetlands restored 2002-2003 FY 2002 - FY 2003 National Estimates of Environmental Benefits Administrative Penalty Order (APO) complaints by statute/program 1991-2003 FY 1991 - FY 2003 Administrative Penalty Order Complaints, FY 1999 - FY 2003 Case Initiations- Administrative Orders Administrative Compliance Orders by statute 1999-2003 FY 1999 - FY 2003 Case Initiations-Administrative Orders, FY 1999 - FY 2003 Case Conclusions Administrative penalties by statute ($) 1974-2003 1974 - FY 2003 Enforcement Penalties, FY 1999 - FY 2003 Administrative and Civil Judicial Penalties, FY 1999- FY 2003 Penalties Cases against facilities initiated as a result of voluntary disclosure under audit policy 1999-2003 FY 1999 - FY 2003 Voluntary Disclosure Policy Cases against companies initiated as a result of voluntary disclosure under audit policy 1999-2003 FY 1999 - FY 2003 Voluntary Disclosure Policy Cases with SEPs by statute 1999-2003 FY 1999 - FY 2003 Supplemental Environmental Projects (SEPs) Citizen complaint responses by program area 2003 FY 2003 Citizen Complaints Citizen complaints received by program area 2003 FY 2003 Citizen Complaints Civil investigations by national priority and non-priority program area 2002-2003 FY 2002 - FY 2003 Civil Investigations Civil judicial settlements by statute 1999-2003 FY 1999 - FY 2003 Case Conclusions Civil judicial referrals 1973-2003 FY 1973 - FY 2003 Civil Judicial Referrals and Penalties, FY 1999 - FY 2003 Case Initiations-Civil Judicial Referrals Civil judicial penalties ($) 1973-2003 FY 1973 - FY 2003 Civil Judicial Referrals and Penalties, FY 1974 - FY 2003 Enforcement Penalties, FY 1999 - FY 2003 Administrative and Civil Judicial Penalties, FY 1999 - FY 2003 Penalties Criminal referrals 1983-2003 FY 1983 - FY 2003 Criminal Referrals and Penalties, FY 1998 - FY 2003 Criminal Enforcement Program Activities Criminal defendants charged 1998-2003 FY 1998 - FY 2003 Criminal Enforcement Program Activities Criminal cases initiated 1998-2003 FY 1998 - FY 2003 Criminal Enforcement Program Activities Criminal penalties ($) 1974-2003 1974 - FY 2003 Enforcement Penalties, FY 1983 - FY 2003 Criminal Referrals and Penalties, FY 1999 - FY 2003 Penalties 46 ------- Data FY(s) Available Reports in Which Measure Appears Expedited penalty order complaints by program area 2003 FY 2003 Expedited Administrative Penalty Orders Expedited penalty order settlements by program area 2003 FY 2003 Expedited Administrative Penalty Orders Federal inspections and evaluations by program 1999-2003 FY 1999 - FY 2003 Federal Inspections and Evaluations Federal inspections and evaluations by statute 1994-2003 FY 1994 - FY 2003 Federal Inspections and Evaluations Final Administrative Penalty Orders by statute 1999-2003 FY 1999 - FY 2003 Case Conclusions Gallons of wastewater/ground water treated 2002-2003 FY 2002 - FY 2003 National Estimates of Environmental Benefits Incarceration (years) 1998-2003 FY 1998 - FY 2003 Criminal Enforcement Program Activities Injunctive relief by case type ($) 1999-2003 FY 1999 - FY 2003 Injunctive Relief Injunctive relief by statute ($) 1999-2003 FY 1999 - FY 2003 Injunctive Relief Notices of Determination as a result of voluntary disclosure under audit policy 1999-2003 FY 1999 - FY 2003 Voluntary Disclosure Policy People protected by Safe Drinking Water Act (SDWA) enforcement 2002-2003 FY 2002 - FY 2003 National Estimates of Environmental Benefits Pounds of pollution reduced, treated, or properly managed 2002-2003 FY 2002 - FY 2003 National Estimates of Environmental Benefits Pounds of contaminated soil/sediment 2002-2003 FY 2002 - FY 2003 National Estimates of Environmental Benefits Resolved cases against companies as a result of voluntary disclosure under audit policy 1999-2003 FY 1999 - FY 2003 Voluntary Disclosure Policy Resolved cases against facilities as a result of voluntary disclosure under audit policy 1999-2003 FY 1999 - FY 2003 Voluntary Disclosure Policy Settlements with/without complying actions by region 2003 FY 2003 Complying Actions Statistically-valid noncompliance rates for RCRA inspections of foundries 2004-2005 Statistically-Valid Noncompliance Rates for RCRA Inspections of Foundries Supplemental Environmental Projects by statute ($) 1999-2003 FY 1999 - FY 2003 Supplemental Environmental Projects (SEPs) Total penalties by case type ($) 1999-2003 FY 1999-FY 2003 Penalties These reports will be posted to EPA's Web site at http://www.epa.gov/compliance/data/results/index.html 47 ------- Distribution Appendix G Office of the Administrator Assistant Administrator for Enforcement and Compliance Assurance Principal Deputy Assistant Administrator for Enforcement and Compliance Assurance Director, Office of Compliance, Office of Enforcement and Compliance Assurance Director, Office of Civil Enforcement, Office of Enforcement and Compliance Assurance Agency Followup Official (the CFO) Agency Followup Coordinator Audit Followup Coordinator, Office of Enforcement and Compliance Assurance General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for Public Affairs Inspector General 48 ------- |