U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL Efficiency of EPA's Rule Development Process Can Be Better Measured Through Improved Management and Information Report No. 13-P-0167 February 28, 2013 ------- Report Contributors: Laurie Adams Daniel Carroll Carolyn Copper Jerri Dorsey Gabby Fekete Jeffrey Harris Olga Stein Abbreviations ADP Action Development Process EPA U.S. Environmental Protection Agency OIG Office of Inspector General OMB U.S. Office of Management and Budget OP Office of Policy RAPIDS Rule and Policy Information and Development System Hotline To report fraud, waste, or abuse, contact us through one of the following methods: e-mail: OIG Hotline@epa.gov write: EPA Inspector General Hotline phone: 1-888-546-8740 1200 Pennsylvania Avenue, NW fax: 202-566-2599 Mailcode 2431T online: http://www.epa.gov/oiq/hotline.htm Washington, DC 20460 ------- ^eDsx ' JL \ i®i U.S. Environmental Protection Agency Office of Inspector General At a Glance 13-P-0167 February 28, 2013 Why We Did This Review The purpose of this review was to evaluate whether the U.S. Environmental Protection Agency's (EPA's) Action Development Process (ADP) results in the timeliest, most efficient, and most effective method for rule development. This review was requested by the Agency. We focused on the key aspects of efficiency with which program offices implement the ADP's guidance for rule development. EPA's Office of Policy coordinates the ADP. This report addresses the following EPA Goals or Cross-Cutting Strategies: • Taking action on climate change and improving air quality • Protecting America's waters • Cleaning up communities and advancing sustainable development • Ensuring the safety of chemicals and preventing pollution Efficiency of EPA's Rule Development Process Can Be Better Measured Through Improved Management and Information What We Found Rule development is one of the Agency's principal tasks. EPA develops rules to carry out the environmental and public health protection laws passed by Congress. Efficient EPA rulemaking may accelerate the progress of protecting human health and the environment. However, due to limitations in EPA rulemaking documentation and guidance, the Agency is unable to evaluate the efficiency of the rulemaking process or identify potential delays in its rulemaking activities. For example, EPA has limited information on the time and resources used to complete the various stages of the rule development process. The development and implementation of management controls to ensure that the rulemaking process is progressing efficiently and that resources are accurately accounted for will enhance EPA's ability to assure efficiency during the development process and accelerate the progress of protecting human health and the environment. Recommendations and Planned Agency Corrective Actions We recommend that the Associate Administrator for EPA's Office of Policy establish guidance, maintain database documentation, and track resources, to enhance the Agency's ability to determine the efficiency of the rulemaking process. The Associate Administrator concurred with the first and second recommendations but disagreed with the third recommendation. We consider the third recommendation unresolved. As required by EPA Manual 2750, the Agency will need to initiate the audit resolution process. For further information, contact our Office of Congressional and Public Affairs at (202) 566-2391. The full report is at: www.epa.qov/oiq/reports/2013/ 20130228-13-P-0167.pdf ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 THE INSPECTOR GENERAL February 28, 2013 MEMORANDUM SUBJECT: Efficiency of EPA's Rule Development Process Can Be Better Measured Through Improved Management and Information Report No. 13-P-0167 This is a report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 60 calendar days. You are not required to provide a written response to the first and second recommendations because you provided agreed-to corrective actions and planned completion dates. The OIG may make periodic inquiries on your progress in implementing these corrective actions. The OIG and Agency have not agreed on a course of action to address the third recommendation. We consider that recommendation unresolved. As required by EPA Manual 2750, the Agency will need to initiate the audit resolution process. Should you choose to provide a response, we will post your response on the OIG's public website, along with our memorandum commenting on your response. You should provide your response as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should not contain data that FROM Arthur A. Elkins Jr. TO Michael Goo, Associate Administrator Office of Policy ------- you do not want to be released to the public; if your response contains such data, you should identify the data for redaction or removal along with corresponding justification. We have no objections to the further release of this report to the public. We will post this report to our website at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact Carolyn Copper, Assistant Inspector General for Program Evaluation, at (202) 566-0829 or copper.carolvn@epa.gov; or Jerri Dorsey at (919) 541-3601 or dorsev.ierri@epa.gov. ------- Efficiency of EPA's Rule Development Process Can Be 13-P-0167 Better Measured Through Improved Management and Information Table of Chapters 1 Introduction 1 Purpose 1 Background 1 Scope and Methodology 3 2 Additional Controls Needed to Measure Efficiency 5 Enhanced Tracking and Document Retention Needed 5 Rulemaking Costs Are Not Tracked 6 Conclusion 8 Recommendations 8 Agency Comments and OIG Evaluation 8 Status of Recommendations and Potential Monetary Benefits 10 Appendices A Completed Rules Sampled and Reviewed by OIG 11 B Agency Response and OIG Comments 12 C Distribution 18 ------- Chapter 1 Introduction Purpose The purpose of this review was to evaluate whether the U.S. Environmental Protection Agency's (EPA's) Action Development Process (ADP) ensures the timeliest, most efficient, and most effective method for rule development. Specifically, we focused on key aspects of the efficiency1 with which the program offices implement the Agency's ADP guidance. The initial impetus for this review was a request by the Agency to improve the process.2 Background EPA protects public health and the environment in a variety of ways, but one of the most important ways is through regulation development. EPA regulations cover a range of environmental and public health protection issues, from setting standards for clean water and establishing requirements for proper handling and reductions of toxic wastes to controlling air pollution from industry and other sources. According to EPA, it is one of the most active regulatory agencies in the federal government. Each year, the Agency develops a considerable number of rules which address highly technical, scientific, and complex environmental problems. On average, EPA issues over 122 Administrator-signed regulations each year.3 Between 2005 and 2010, EPA published 735 major or Administrator- signed rules, of which approximately 40 percent were final rules.4 According to the Agency, much of its environmental success and organizational credibility is directly linked to the quality of its work under the ADP. EPA designed the ADP to develop rules based on sound scientific, economic, legal, and policy analyses. The ADP is designed to be a multi-disciplinary, collaborative, cross-office, and cross-media approach to rule development and accelerate the progress of protecting human health and the environment. 1 For the purposes of this review, "efficiency" is defined as the resources used to achieve program results. This is the U.S. Government Accountability Office's Government Auditing Standards definition. 2 OIG management received a suggestion to review the Agency's rule development process in an effort to identify areas of redundancy, waste, and inefficiency. 3 EPA's Administrator-signed rules are published actions which modify or propose to modify the Code of Federal Regulations and, therefore, contain rule text. The Code of Federal Regulations is the official record of all regulations created by the federal government. 4 A final rule is the conclusion of the Agency's regulatory development. A final rule represents a range of documents that are published in the Rules section of the Federal Register. These include final rules that amend the Federal Register by adding new text or by revising or removing existing text. These also include other types of final actions or documents that have no regulatory text and do not amend the Federal Register but affect the Agency, such as general policy statements, final guidance documents, or petitions for reconsideration. 13-P-0167 1 ------- The ADP serves as a framework to ensure that rules are developed using quality information and all scientific, economic, and policy issues are addressed during the appropriate rule development stages. The ADP also provides opportunities for senior management's early involvement and provides guidance to rulemaking staff at key points of the rule's development. EPA's Office of Policy (OP) leads the regulatory development process via its Office of Regulatory Policy and Management, which provides support and guidance to EPA's program and regional offices as they develop regulations. EPA's OP coordinates the ADP and also serves as liaison to other agencies involved in the rule development, such as the Office of Management and Budget (OMB). There are five major stages of the ADP: 1. Tiering the action and obtaining commencement approval. 2. Developing the proposed rule or draft action. 3. Requesting OMB review (if needed) for proposed and final actions. 4. Requesting signatures, publishing an action in the Federal Register, and soliciting and accepting public comment. 5. Developing the final action and ensuring congressional review. Figure 1 illustrates the key steps related to the five stages of the ADP. This figure represents the development process for all Tier 1 and 2 actions. Figure 1: Flowchart of the Action Development Process in EPA 1 0»iissfor \ y 2 < /Ticniiti LJ» j * 'iM-rkyrobp pWte- Ar.aMic 4 Ear^v GuiJsmv X . ¦'%. ,r' 5 Detail! Artal^c S'ueiwit 8, Mw^ierivnl 1,)uhilertsUsartsk**ar^ \ refjulatervoptoGrts f v- 9 fftMgwu?* pPfientipitarnHe nils- »wl iufpsitna dotuBiei.b "X X, 1J Fna: Rfv'fir ¦x. Source: EPA's Action Development Process training presentation. Note: Step 7 of the illustration involves the Agency's necessary analyses and an assessment of all applicable statutes and executive orders, of which there are many. AA/RA: Assistant Administrator/Regional Administrator EO: Executive Order 13-P-0167 2 ------- Scope and Methodology We conducted our work from August 2011 through August 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform our work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our objectives. Our review focused on the key aspects of efficiency with which program offices implement the ADP's guidance for rule development. Efficiency can refer to not only the timeliness in producing rules but the management of the resources used in the rule development process. In an effort to evaluate the efficiency of the rulemaking process, we identified rules that had final regulatory actions in 2010 and 2011. We reviewed these final rules to determine the adequacy with which the program offices' staff complied with preparing documents that evidenced the "products of a quality action."5 We reviewed information in the Rule and Policy Information and Development System (RAPIDS), EPA's database to manage the creating, tracking, approving, and reporting of the Agency's rulemaking actions. 6 We also interviewed EPA staff to determine how the Agency tracks the resource cost for rule development activities. During field work, we took an initial sample of four completed rulemaking actions to determine the feasibility of conducting a larger sample of completed rules.7 We attempted to document progress for each rule throughout the entire ADP process using a data collection instrument developed by the EPA Office of Inspector General (OIG). We sought to obtain key documentation and dates both from RAPIDS and the workgroup chairs. We were eventually able to obtain some documentation for each rule sampled; however, the documentation was not always the final signed product. This impeded our ability to track the progress of the rules though the ADP process in an effort to identify areas to increase efficiency. After completing the limited sample work, we discovered fundamental gaps in the ADP tracking process that made further data collection unnecessary. These gaps are discussed in detail in chapter 2 of this report. 5 "Products of a quality action" are defined by EPA's ADP guidance document as those actions that include the following elements: Tiering Form; Preliminary Analytic Blueprint; Early Guidance Briefing Package; Early Guidance Memorandum; Detailed Analytic Blueprint; Options Selection Briefing Package; Draft Action (e.g., rule text, report to Congress, policy or guidance); Action Memorandum; and Communications Plan, coordinated with the Office of Public Affairs. 6 During our field work, RAPIDS was the database used at the time to manage all ADP activities and the one to which we were directed for our evaluation's data collection activities. 7 See appendix A for details on these four actions. 13-P-0167 3 ------- Prior Reports The U.S. Government Accountability Office reviewed federal rulemaking in two recent reports.8 One of the reports concluded that agencies had little data on the time and resources used to comply with regulatory requirements, making it difficult to evaluate the effects of requirements on rulemaking. In the other report, EPA was cited for not following key steps in its guidance before finalizing a major rule. There were no previous EPA OIG reviews of this subject matter. 8 GAO-08-128: EPA Actions Could Reduce Environmental Information Available to Many Communities, November 2007; GAO-09-205: Improvements Needed to Monitoring and Evaluation of Rules Development as Well as to the Transparency of OMB Regulatory Reviews, April 2009. 13-P-0167 4 ------- Chapter 2 Additional Controls Needed to Measure Efficiency EPA established the ADP to ensure the quality of actions developed, but ADP guidance does not contain adequate controls to assure that the process is completed in the most efficient manner. Inefficiencies in the rule development process may result in delays that could postpone new or revised rules and impede progress to protect human health and the environment. To determine the efficiency of the rulemaking process, the Agency needs adequately maintained documentation, defined milestone goals that measure and monitor rule development progress, and cost information. Adequate system controls and the development of a method to track resource use will provide EPA with the tools necessary to measure and assure the efficiency of the development process. Enhanced Tracking and Document Retention Needed We found that EPA's database for creating and tracking rules was sparsely populated and did not contain the necessary documents or information to allow us to complete our review.9 We found program offices were not adequately utilizing the Agency's database due to a lack of standardized procedures from the OP describing who was responsible for uploading the developmental documents to the databases used to manage the ADP process. The development documents called for in the ADP guidance represent tools for both the rulemaking workgroup as well as for management's oversight responsibilities. The availability of these documents and associated milestone completion data would provide management with the information needed to oversee the rule development process and assure the rulemaking process progresses in a timely and efficient manner. In February 2012, EPA implemented a new database, ADP Tracker. However, there remain some limitations in regards to tracking and documentation which challenge the Agency's ability to monitor, evaluate, and assure the efficiency of EPA rulemaking. According to the Agency, the new system will provide improved capability to track milestones, manage workgroups, and track work flow, as well as provide better security and access.10 According to OP, the Agency conducted nearly 20 training sessions for staff throughout the Agency on proper use of ADP Tracker. However, there are still no standardized procedures describing who is responsible for uploading the key developmental documents to the database used to manage the ADP process. OP, the office that manages the 9 Because Agency staff had not populated RAPIDS with the milestone dates and other data on the ADP, efficiently determining any potential sources of delay in the rulemaking process for our sampled rules was not possible. 111 We were unable to access or conduct a full evaluation of the new system because it was not fully implemented during our evaluation field work. 13-P-0167 5 ------- ADP, should provide additional instruction to the program offices involved in rulemaking regarding the roles and responsibilities for uploading the various documents into the new database. Furthermore, OP should assure that the program offices comply with the data entry requirements. This will provide OP with the ability to ensure data reliability and the rulemaking process progresses in a timely and efficient manner. Rulemaking Costs Are Not Tracked The Agency does not maintain cost data associated with action development and rulemaking. Rule development requires input from various offices in the Agency with an associated time commitment, and can take several years to complete. The lack of cost data means that EPA has limited information on the resources used to complete the ADP process. Rulemaking Activities Represent Time Commitment Across Agency EPA prepares and releases hundreds of actions each year that define the technical and operational details of environmental programs. These actions can be complex. As illustrated previously in Figure 1, the rulemaking process includes a number of steps, including internal and external comment and review. According to the Agency, the development time (i.e., time from initiation of a rulemaking to final rule stage) for publication of an Administrator-signed rule for the period 2005- 2010 was a total average time of 2 years and 6 months (930 days). Figure 2 illustrates how rule development time varies by program office and by year.11 11 The variation in the development time across the five main offices with published Administrator-signed rules is greatly influenced by a variety of internal and external factors. These factors include the complexity of the rulemaking, whether the rulemaking is under a court-schedule or legislative mandate, the interests of other program offices, and the extent of public involvement during the rule development process. 13-P-0167 6 ------- Figure 2: Administrator-signed rules, development time Administrator-Signed Rules: Average Development Time (Days) by EPA Program Offices, 2005-2010 6000 5500 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0 4,690 1,714 2010 2009 2008 2007 2006 2005 OCSPP OSWER OW Source: EPA's RegStat website. OAR: Office of Air and Radiation OEI: Office of Environmental Information OCSPP: Office of Chemical Safety and Pollution Prevention OSWER: Office of Solid Waste and Emergency Response OW: Office of Water Rule development requires input and time from offices across the Agency, neither of which are tracked for budgetary or management purposes. It should be the Agency's goal to ensure that rulemaking is done in the most cost effective and efficient manner. Because EPA does not have input measures that track time spent on the development of individual rules, rulemaking cost information is not available to EPA management to determine efficiency or return on investment. OMB recommends using input measures (e.g., time, employee hours, funding, equipment) to determine the resources used to achieve an output (activity) or outcome (impact). When compared to the achieved outputs and outcomes, the information can be used to determine program efficiency and return on investment. Resource Data Can Enhance Budgeting and Forecasting of Resources EPA has limited data on the time and resources used to complete the various stages of the rule development process. By tracking both costs and milestones, management can monitor the rulemaking progress and better assess where delays are occurring and make the necessary corrections. Tracking and documenting costs would also provide management with historical data that could be used for 13-P-0167 7 ------- budgeting and forecasting resource needs in future rulemaking work. For budgetary considerations, management can have better control on resource needs in the future based on the expected workload of new rules and how much of their budget is allocated to rulemaking activities. Conclusion Developing environmental regulations is one of EPA's principal tasks. The ADP was designed to deliver actions that are based on sound science, promote economic efficiency, and accelerate the progress of protecting human health and the environment. EPA should take steps to track and document the actions of the rule development process. Additionally, the development of a method to track resource use in key rulemaking activities will provide the Agency with the tools necessary to assess the key aspects of efficiency in the rule development process. A commitment to assessing efficiency of the rulemaking process shows the Agency values accelerating the protection of human health and the environment. Recommendations We recommend that the Associate Administrator, Office of Policy: 1. Establish guidance that clarifies roles and responsibilities in ADP implementation, including data entry, record keeping, and the status of action development. 2. Ensure that ADP Tracker has established clear roles, responsibilities, and requirements for the program offices to upload development documents; and that entries are updated in a timely manner, are monitored for data quality, and all features available are used to evaluate the efficiency of the rule development process. 3. Develop and implement a method to track resource use in key rulemaking activities. Agency Comments and OIG Evaluation The Associate Administrator concurred with the first and second recommendations. The Agency provided high-level corrective actions in response to these recommendations with an estimated milestone completion date of June 2013. These recommendations are considered open with agreed-to corrective actions pending. OP disagreed with the third recommendation because it does not have the expertise to develop a method to track resources. The OIG and Agency have not agreed on a course of action to address this recommendation. We consider this recommendation unresolved with actions underway as required by EPA Manual 2750. 13-P-0167 8 ------- OP believes the report does not present an accurate picture of rulemaking in the Agency. Additionally, OP believes that clarification of the purpose of the report is critical to assessing what appropriate conclusions can be fairly reached regarding the ADP process. OP also questioned the emphasis on the RAPIDS database, which was replaced by ADP Tracker in February 2012. We made changes to the report as appropriate. The Agency's complete response, along with the OIG's evaluation, is in appendix B. 13-P-0167 9 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFITS (In $000s) Rec. No. Page No. Subject Status1 Action Official Establish guidance that clarifies roles and responsibilities in ADP implementation, including data entry, record keeping, and the status of action development. Ensure that ADP Tracker has established clear roles, responsibilities, and requirements for the program offices to upload development documents; and that entries are updated in a timely manner, are monitored for data quality, and all features available are used to evaluate the efficiency of the rule development process. Develop and implement a method to track resource use in key rulemaking activities. Associate Administrator, Office of Policy Planned Completion Date Claimed Amount Ag reed-To Amount Associate Administrator, June 2013 Office of Policy Associate Administrator, June 2013 Office of Policy 1 0 = Recommendation is open with agreed-to corrective actions pending. C = Recommendation is closed with all agreed-to actions completed. U = Recommendation is unresolved with resolution efforts in progress. 13-P-0167 10 ------- Appendix A Completed Rules Sampled and Reviewed by OIG Rule title Tier Program office Transport Rule (CAIR Replacement Rule) - Rule No. 2060-AP50 (SAN: 5336) Tier 1 Office of Air and Radiation Federal Requirements under the Underground Injection Control (UIC) Program for Carbon Dioxide (C02) Geologic Sequestration (GS) Wells - Rule No. 2040-AE98 (SAN: 5211) Tier 2 Office of Water Mercury; Significant New Use Rule for Elemental Mercury in Flow Meters, Manometers, and Pyrometers - Rule No. 2070-AJ36 (SAN: 5238) Tier 3 Office of Chemical Safety and Pollution Prevention Protection of Stratospheric Ozone: Listing of Substitutes for Ozone- Depleting Substances-Hydrocarbon Refrigerants - Rule No. 2060- AP54 (SAN: 5339) Tier 3 Office of Air and Radiation Source: OIG analysis. 13-P-0167 11 ------- Appendix B Agency Response and OIG Comments October 12, 2012 MEMORANDUM SUBJECT: Response to Draft Report: Efficiency of EPA's Rule Development Process Cannot Be Assessed, Project No. OPE-FY 11-0021 FROM: Michael L. Goo /s/ Associate Administrator TO: Carolyn Copper Assistant Inspector General for Program Evaluation Office of Inspector General In accordance with U.S. Environmental Protection Agency (EPA) Manual 2750,1 hereby submit the Office of Policy's (OP) response to the findings and recommendations cited in the Office of Inspector General's (OIG) draft evaluation report "Efficiency of EPA's Rule Development Process Cannot Be Assessed, Project No. OPE-FY11-0021." Thank you for providing the Office of Policy with an opportunity to share our feedback on the draft evaluation report. We believe the report seeks to examine important features of the EPA Action Development Process (ADP) and we are eager to engage with the Office of Inspector General in crafting an accurate and relevant report that fairly characterizes the complete ADP context. As currently drafted, we believe the report does not present an accurate picture. We believe that additional data bearing on this issue will be forthcoming in the near future and that such data will be extremely helpful in accurately describing the situation. We also believe that clarification of the purpose of the report is critical to assessing what appropriate conclusions can fairly be reached regarding the ADP process. OIG Response: As communicated to the Agency, the purpose of this review was to evaluate whether EPA's ADP resulted in the timeliest, most efficient, and most effective method for rule development. This basis for our review was a suggestion by the Agency to look at the process and potential ways to increase efficiency. We focused on the efficiency with which program offices implemented the ADP guidance for rule development. 13-P-0167 12 ------- Finally, we believe that the draft findings contemplated in the draft report relate largely to a database that is no longer in use by EPA and therefore such conclusions are of, at best, diminished relevance to the ongoing work of the Agency. In light of these concerns we would appreciate the opportunity to consult further with the Office of the Inspector General regarding this Draft Report prior to its finalization. OIG Response: The OIG addressed suggested corrections as appropriate. Nevertheless, OP agrees with the OIG's conclusion that opportunities exist to expand the usefulness and applicability of the ADP Tracker and concurs with the intent of OIG's second recommendation to "ensure that ADP Tracker has established clear roles, responsibilities, and requirements for the program offices to upload development documents; and that the rules are updated in a timely manner, entries are monitored for data quality, and all features available are used to evaluate the efficiency of the rule development process." The Draft report states, under the heading "Why We Did This Review" that: The purpose of this review was to evaluate whether the U.S. Environmental Protection Agency's Action Development Process results in the timeliest, most efficient and most effective method for rule development. The initial impetus for this review was a request by the Agency to look at the process and potential ways to increase efficiency. We focused on the efficiency with which program offices implement the ADP's guidance for rule development. Except for the description above, the details of the "request by the Agency" remain largely unclear to the Office of Policy at present. Although the Office of Policy welcomes the opportunity to explore ways to improve the efficiency of EPA's rulemaking process, the language above seems to suggest a concept of "efficiency" in which the speed of ADP is the primary focus of the inquiry. Moreover the concept of "efficiency" as "speed" is then incorporated into the draft findings which state "the ADP Guidance does not contain controls to ensure that the process is completed in the most efficient manner. Inefficiencies can result in delays that postpone new or revised rules." As a result, the report concludes that "The Efficiency of EPA's Rule Development Process Cannot be Assessed" and that "[The] ADP Does Not Include Controls to Ensure Efficiency." OIG Response: The OIG had multiple communications with the Agency regarding the basis for this evaluation and the details of the request. Our review was requested by Agency officials. At no time during our review did anyone within the Agency question or raise concerns with our scope. As the draft report also acknowledges, "the ADP serves as a framework to ensure that rules are developed using quality information and that all scientific, economic and policy issues are addressed during the appropriate rule development stages." Given this purpose for the ADP, it should be evident that mere hastening of the rulemaking process, in such a way as to produce rulemaking products in the shortest amount of time, and using the fewest amount of Agency 13-P-0167 13 ------- resources, may not ultimately be consistent with a more comprehensive and appropriate definition of "efficiency." OIG Response: OIG's report does not advocate for speed in the rulemaking process over effectiveness of the process. OIG's report recommends that additional controls are needed to measure efficiency. As OP staff conveyed during our May 30, 2012 meeting and in several conference calls, the purpose of the ADP is to guide the process for developing a wide variety of rulemakings, public notices, and guidance documents. Each such action type can vary significantly with regard to Agency priority, level of input required from other EPA offices, need for inter-agency review, and for technical and/or scientific complexity. The goal is to produce quality actions that are timely, clear, scientifically sound, technically accurate, effective, and legally defensible. It would serve no public purpose to produce a rule quickly, using few resources, if it fails to withstand legal challenge and/or produce social benefits. OIG Response: The OIG report does not advocate that speed or any other factor is more or less important than efficiency. The report presents the aspects of efficiency that we were unable to assess due to limitations in the Agency's data systems. Perhaps more importantly, the rapid development of rulemaking products in ways which fail to pass either scientific, legal or policy muster cannot credibly be conceived of as "efficient," since the end result is to produce products that must either be substantially reworked or possibly even abandoned. Paradoxically, the consequence of moving potentially flawed products through the system, with speed as the main goal, may in fact result in greater delay in achieving environmental benefits, thereby undermining even the relatively narrow value of increased speed at a particular point in the process. OIG Response: In this review, we did not assess the effectiveness of the ADP nor the potential for delayed environmental benefits due to the process. For example, a rule may be "delayed" at a particular ADP stage while important policy and analytical issues are resolved. While this may appear to be a "delay," well crafted changes to the rule can speed later stages of review and successful implementation. And of course, rapid action can result in an inefficient uses of the agency's resources negatively, impact transparency, and potentially undermine the Agency's credibility. Thus, the ADP process is designed not only with timeliness in mind, but perhaps more importantly, with the paramount goal of quality decision making at its heart. We believe the ADP, as currently configured and implemented, plays a critical role in achieving that goal that should not be overlooked. OIG Response: Based on the information available to us during our review, we concluded that the efficiency of the rule development process could not be assessed. We could not conduct a detailed review of the Agency's rulemaking process due to the lack of data and documents in support of the process. We believe that the intent of the ADP is to ensure effective actions are developed. 13-P-0167 14 ------- Accordingly, any evaluation of the ADP process should begin with an assessment of whether the goals of clarity, scientific validity, technical accuracy, effectiveness and legal defensibility are systematically being met and should examine the extent to which ADP process requirements have contributed to their fulfillment. OP's review of the draft report leads us to conclude that OIG did not fully consider either the complexity or the needed flexibility of the ADP as it considered its draft recommendations and conclusions. In the attached redline strikeout of the draft report we provide suggested edits to areas that we believe reflect misunderstandings about the ADP. OIG Response: It is reasonable and appropriate to assess the efficiency of EPA's rulemaking process. Our report does not address effectiveness issues or advocate that those factors are more or less important than efficiency concerns. We also note that the draft report directs much of its attention toward the RAPIDS system, which is no longer in use. A more recent, system, ADP Tracker is in use and we are beginning to have data available from that system that could help EPA and the OIG in drafting this report. RAPIDS was developed over 15 years ago and was designed as a forward-looking management information system to help Agency and program-level leadership track and schedule decision meetings. Contrary to the Draft Report's suggestion, RAPIDS was not intended to be an aggregator for all Agency records relating to action development, nor was it intended to be used to assess the extent to which the documents supporting EPA's rules are efficiently produced. These activities are managed within individual EPA programs. In any event, RAPIDS has been replaced. OIG Response: When we initiated our field work, RAPIDS was the database used by the Agency to manage the regulatory development activities. The Agency launched ADP Tracker in February 2012, approximately a month after we met with the Agency to discuss our final evaluation objectives. At that meeting we were advised that since our sample was from 2010 and 2011, RAPIDS would have all the information for our selected rules. The Agency said that we should not have to get involved with the new database. Consequently, we based our review on the data available to us in RAPIDS. The issues we uncovered with RAPIDS were not things that the Agency said would be addressed with ADP Tracker, particularly issues of responsibility for data entry and attaching documents. Therefore the first and second OIG recommendations are directed toward strengthening the controls in place governing ADP Tracker. The replacement system for RAPIDS is ADP TRACKER. This system does include features that can be used to track process milestones. For example, ADP TRACKER allows us to track whether ADP milestones were waived by senior management. It also allows us to track the time workgroups were given to review milestone material. By early next year, a full year of these data will be available for review. Even so, it should be noted again that even when fully utilized, ADP TRACKER will not allow us to measure the "efficiency" of the Agency's rulemaking process and whether the Agency is allocating resources optimally. Again, the system was not designed for this purpose. A fair notion of "efficiency" would examine whether the ADP process has been efficient at enhancing the quality of EPA rulemaking. Such an assessment cannot credibly be performed by a tracking or database management system alone. 13-P-0167 15 ------- OIG Response: While we did not conduct a detailed review of ADP Tracker, the issues we uncovered with RAPIDS were not things that the Agency said would be addressed with ADP Tracker, particularly issues of responsibility for data entry and attaching documents. During our review we were advised that a subset of the data was moved from RAPIDS to ADP Tracker. If that is the case, the data in ADP Tracker could have the same problems that RAPIDS had. The Agency did not disclose what it defines as the "fair notion of efficiency" in its response. Apart from these concerns, I agree that opportunities exist to expand the usefulness and applicability of the ADP TRACKER. Therefore, OP concurs with the draft report's first recommendation to "establish guidance that clarifies roles and responsibilities in ADP implementation, including data entry, record keeping, and the status of action development." OP has addressed many of the concerns with regard to clearly delineated roles in recording data and key action development documents. Additionally, ADP TRACKER (unlike RAPIDS) utilizes controls to ensure that program offices enter key fields before updating later milestones. OP plans to issue the updated ADP TRACKER user guides that will clarify these roles and responsibilities by June 2013. OP also concurs with the intent of OIG's second recommendation to "ensure that ADP-Tracker has established clear roles, responsibilities, and requirements for the program offices to upload development documents; and that the rules are updated in a timely manner, entries are monitored for data quality, and all features available are used to evaluate the efficiency of the rule development process." Along with the updated ADP TRACKER user guides, OP will issue a memo clearly outlining the responsibility of program offices for uploading development documents. We will also use this opportunity to ensure that program offices are working to achieve to a high degree of data and record-keeping integrity in ADP TRACKER. However, we believe the draft report incorrectly suggests that OP is responsible for maintaining records associated with each rulemaking either in ADP TRACKER or otherwise, when it is, in fact, the Agency policy for the originating office to maintain such records. As OIG states on page 6 of the draft report, "program offices are required to maintain electronic records in the enterprise-wide electronic content management system to allow for timely access and retrieval." We believe this distinction concerning recordkeeping should be more clearly and consistently communicated in OIG's final report and we suggest edits to this end. OIG Response: The OIG acknowledges the Agency's concurrence with the first and second recommendation. While it is the program offices' responsibility to maintain records, the OIG notes that OP still has oversight responsibility over the ADP process and program office responsibility does not diminish that oversight role. With regard to OIG's third and final recommendation, OP disagrees with OIG's suggestion that it "develop and implement a method to track resource use in key rulemaking activities." OP does not have the expertise to develop or implement such a method. OP's resources and expertise are best suited to overseeing the regulatory development process to ensure that the Agency produces effective regulatory actions to protect human health and the environment. Monitoring the use of 13-P-0167 16 ------- internal Agency costs associated with rule development may fall under the purview of the Office of the Chief Financial Officer or the relevant program office. OIG Response: The OIG position is that, without the controls outlined in recommendation 3, the Agency will continue to be challenged in assessing the efficiency of the rulemaking process and therefore the recommendations stands. We consider this recommendation unresolved. As required by EPA Manual 2750, the Agency will need to initiate the audit resolution process. OP welcomes the opportunity to discuss the comments and suggested revisions detailed here and in the attached markup. If you or your staffs have questions regarding this response, please feel free to contact Nathaniel Jutras at (202) 564-0301 or iutras.nathaniel@epa.gov. Attachment 13-P-0167 17 ------- Distribution Office of the Administrator Associate Administrator, Office of Policy Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for External Affairs and Environmental Education Audit Follow-Up Coordinator, Office of the Administrator Audit Follow-Up Coordinator, Office of Policy 13-P-0167 ------- |