OIG V "'l PRO**6 OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Audit Report EPA Needs to More Actively Promote State Self Assessment of Environmental Programs Report No. 2003-P-00004 December 27, 2002 ------- ------- s UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL December 27, 2002 MEMORANDUM SUBJECT: Report Number: 2003-P-00004 EPA Needs to More Actively Promote State Self Assessment of Environmental Programs FROM: Michael A. Rickey, Director Assistance Agreement Audits TO: Edward Krenik Associate Administrator for Congressional and Intergovernmental Relations We completed our preliminary research of the effectiveness of state self assessments. The objective of the assignment was to answer the following questions: • Are states using self assessments to develop their performance partnership agreement and performance partnership grant priorities and activities? • What have been the benefits and obstacles to conducting and using self assessments? • Have states that have based their performance partnership agreement and performance partnership grant priorities and activities on a self assessment achieved environmental results superior to those states that have not conducted such an assessment? The purpose of preliminary research is to gather information to define the scope and methodology for an audit. During preliminary research, we determined, as discussed below, that the U.S. Environmental Protection Agency (EPA) has done little to develop and promote greater reliance on state self assessment. We met with the Deputy Associate Administrator for Intergovernmental Relations and discussed the results of our preliminary research. Based on our work to date and the meeting, we decided to report on what we found and not to perform additional audit work. This report addresses the first two objectives. Based on our conclusions on the first two objective questions, we did not believe that there was sufficient information to answer the third objective question. Details on the scope and methodology are in Appendix A. ------- Background In 1995, EPA and the states established the National Environmental Performance Partnership System (NEPPS) to strengthen the management, efficiency, and effectiveness of the nation's state and Federal environmental programs by improving the partnership between EPA and the states. Self assessment is one of the principal components of NEPPS. Self assessment was introduced as a new approach to program assessment, to share with the public information about environmental goals, priorities, and prior year achievements. In the self assessment, states would provide information identifying: • What the state sees as the key environmental problems, opportunities, and priorities facing the state; • The recent performance of the state's programs based on available measures of program success; • An analysis of current program weaknesses from the state's perspective; • An assessment of basic fiscal accountability, along with an identification of any areas needing capacity building; • The state's proposed action plan for maintaining and improving its program performance, and identifying specific action and approaches the state plans to take in the coming year; and • How well the state carried out the plan agreed to in the environmental performance agreement. While states are not required to participate in the NEPPS process, 40 Code of Federal Regulations Part 35.115 requires regions and states to have a joint evaluation process for all continuing environmental grants awarded after January 2001. The joint evaluation process includes a discussion of existing and potential problem areas and suggestions for improvement, including, where feasible, schedules for making improvements. The joint evaluation process could provide EPA and the state with information similar to the self assessment that could be used in setting priorities for each state.1 EPA and States Not Effectively Using Self Assessments EPA and states had not widely adopted the self assessment concept. Many states were not performing self assessments, their content varied, and they had little impact on environmental performance agreements. This occurred because EPA had not taken a leadership role to define to staff and states its expectations for self assessments. As a result, Federal resources may not be directed at the state level to where they would have the most vital environmental impact. Most states did not conduct self assessments. According to region officials, only 23 states performed self assessments. Also, self assessments did not always address the elements identified in the NEPPS policy. We reviewed self assessments for two states in Region 1 and compared the contents of the documents to the elements prescribed in the NEPPS policy. Each ' As part of this audit, we did not evaluate whether regions and states were performing joint evaluations. 2 ------- state's self assessment varied from NEPPS policy. In one state, the self assessment addressed five of the six elements. The other state's self assessment fully addressed two elements and partially addressed two elements. The checkmarks in the table denote the elements included in each state's self assessment. Elements of a Self Assessment State 1 State 2 Key problems and priorities / Limited Recent performance / / Program weaknesses / Assessment of fiscal accountability and capacity building Plan accomplishment / / Proposed action plan for improving areas / Limited Self assessments did not include all of a state's programs. According to information provided by the regions, 9 of 23 states did not include all programs in their self assessment. According to officials in one state, all programs were not included because of the extensive coordination and resources required to perform a comprehensive assessment. According to state officials, the self assessment had little impact on the negotiation of the environmental performance agreement and the allocation of Federal funds. State and regional officials both stated that the negotiation process was dominated by EPA priorities. The NEPPS policy states: The self-assessment, in concert with EPA 's perspective on environmental conditions and program performance, wouldform the basis for negotiating the Environmental Performance Agreement. Regional staff stated that the self assessment was of limited value to them. The self assessment was used as a reference for state accomplishments and outputs, but was not very timely for negotiation of the performance agreement. Further, some regional staff stated that the self assessment will be of little value until EPA reaches a consensus on (a) what will be in the self assessment, (b) how it should be conducted, and (c) how the results will be used. Since the NEPPS policy was issued in 1995, EPA had done little to develop and promote greater reliance on the self assessment. EPA had not issued any additional guidance or training on self assessments. As a result, few regions and states were performing and effectively using self assessments. EPA had not taken a leadership role in defining or re-defining the purpose of the self assessment, what should be included, and how it should be used. The NEPPS policy envisioned the self assessment as one of the tools for directing scarce public resources toward improving environmental results and allowing states greater flexibility to 3 ------- achieve those results. By identifying and reporting on state priorities, performance, areas of weakness, and fiscal accountability, the self assessment could provide valuable information on where environmental resources are most needed in a state. Without a comprehensive self assessment, Federal resources may not be directed to the state level where they will have the most vital environmental impact. Recommendations We recommend that the Associate Administrator for Congressional and Intergovernmental Relations: 1. Clarify the purpose of self assessments and how they are to be used by EPA and states. EPA needs to work with states to develop guidance and training on self assessments. Questions to consider include: What information should be included in the self assessment? When should the self assessment be completed? How should EPA combine or link self assessments with other reporting requirements, such as the joint evaluation process required by 40 Code of Federal Regulations Part 35.115? How should EPA use the results of the self assessment to prioritize and allocate Federal funds? 2. Work with other EPA offices and states to gain their acceptance and involvement with the self assessment concept. Agency Response In the Associate Administrator's November 20, 2002, response to our draft report (Appendix B), he generally agreed with all of our findings and recommendations. The Associate Administrator indicated that states that conducted self assessments considered the exercise worthwhile, but expended considerable resources without EPA feedback. The Associate Administrator agreed that a joint evaluation process similar to that required under 40 Code of Federal Regulations Part 35.115 would improve joint planning and priority setting, and would provide information similar to the self assessment. EPA will continue discussions about advancing performance partnership agreements with states. Based on the discussions, the self assessment process may be overtaken by improved priority setting and joint evaluation processes. Office of Inspector General (OIG) Comments We agree with the Associate Administrator's response and that a joint evaluation process could serve the same purpose as a self assessment process. At the exit conference with the Office of Congressional and Intergovernmental Relations, we indicated that regardless of whether the Agency chooses to use a joint evaluation or a self assessment, the Agency needs to clarify the 4 ------- process. Further, the Office of Congressional and Intergovernmental Relations should work with EPA offices and states to gain acceptance of the process, as stated in the recommendations. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 90 calendar days of the date of this report. You should include a corrective actions plan for agreed upon actions, including milestone dates. We have no objections to the further release of this report to the public. This audit report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This audit report represents the opinion of the OIG and the findings contained in this report do not necessarily represent the final EPA position. Final determinations on matters in this audit report will be made by EPA managers in accordance with established audit resolution procedures. If you or your staff have any questions about this report, please contact me at (312) 886-3037, or Ricardo Martinez at (212) 637-3045. 5 ------- 6 ------- Appendix A Scope and Methodology In order to accomplish our objectives, we collected information from EPA's Office of Congressional and Intergovernmental Relations, and from regions and states. Specifically: • We sent questionnaires to staff in EPA Region 1 and its six states regarding self assessment use, benefits, and obstacles. For other than two states, we did not verify the information provided in the questionnaires. • We interviewed NEPPS staff from the Office of Congressional and Intergovernmental Relations, Regions 1 and 2, and the States of Massachusetts and New Jersey. • We obtained information on the number of states conducting self assessments through a questionnaire to the NEPPS coordinators for all 10 EPA regions, and reviewed their responses. We did not verify the information the regions provided. • We compared the content of the self assessments for Massachusetts and New Hampshire with the NEPPS policy. We conducted our fieldwork from August through October 2002. We performed our audit in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We limited our preliminary research to objective questions 1 and 2; and state participation, benefits, and obstacles in conducting self assessments. Based on our preliminary results and a meeting with the Office of Congressional and Intergovernmental Relations, we decided to report on what we found and not to perform a full audit. 7 ------- 8 ------- Appendix B Page 1 of 7 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, DC 20460 November 20, 2002 OFFICE OF CONGRESSIONAL AND INTERGOVERNMENTAL RELATIONS MEMORANDUM SUBJECT: Draft Report - EPA Needs to More Actively Promote State Self-Assessment of Environmental Programs Assignment Number 2002-000877 Edward D. Krc^iLB~c0 £>¦ Associate Administrator for Congressional and Intergovernmental Relations FROM: TO: Michael A. Rickey, Director Assistance Agreement Audits Thank you for the opportunity to comment on the report about your preliminary research on the effectiveness of state self-assessments. Thank you also for your earlier meeting with Dona DeLeon, Deputy Associate Administrator for Intergovernmental Relations, to discuss the results of your research. We appreciate the time you devoted to this important aspect of State/Federal relations and generally agree with your findings. Since January 2002, the Office of Congressional and Intergovernmental Relations (OCIR) has coordinated discussions between EPA senior managers and state commissioners relative to how the performance partnership system is working. This general evaluation and recommended next steps are in the attached draft, "The State-EPA Performance Partnership for Cleaner Air, Purer Water and Better Protected Land." You will note that on page 3 of the document the States conveyed that those that prepared self-assessments invested considerable resources with no feedback from EPA; they believed the exercise was worthwhile. The paper suggests that identifying and reporting on state priorities, performance, areas of weakness, and fiscal accountability could be accomplished through a joint assessment or as part of a State-Regional Office joint strategic planning and priority-setting process. We agree with your point that a joint evaluation process similar to that required under 40 Code of Federal Regulations Part 35.115 would improve joint planning and priority-setting. We appreciate your bringing this important requirement to our attention and we plan to formally include joint evaluation in our work with the States. This joint evaluation process could include discussions of existing and potential problem areas, roles and responsibilities of each level of government in Internet Address (URL) • http://www.epa.gov Recycled/Recyclable • Printed with Vegetable Oil Based Inks on Recycled Paper (Minimum 25% Postconsumer) 9 ------- Appendix B Page 2 of 7 2 solving the problem and resources needed. We see this joint evaluation process providing EPA and the State with information similar to the State self-assessment and it could be used in setting priorities for each State. EPA will continue discussions about advancing PPAs with the ECOS Strategic Planning Committee. It is unclear at this time what role, if any, a State self-assessment will play. It very well may be overtaken by an improved priority-setting process and a much needed joint- evaluation process, as you point out in your review. Should you or your staff have any questions about these comments, please contact the Deputy Associate Administrator for Intergovernmental Relations, Dona DeLeon, at (202) 564- 7178 or Steve Wilson at (202) 564-3646. Attachment 10 ------- Appendix B Page 3 of 7 DRAFT The State-EPA Performance Partnership for Cleaner Air, Purer Water and Better Protected Land For Discussion at the ECOS Partnership Meeting April 23, 2002 4:15 to 6:15 PM Co-chaired by ECOS Secretary-Treasurer Karen Studders, Commissioner, Minnesota Pollution Control Agency & Dona DeLeon, EPA Deputy Associate Administrator for Intergovernmental Relations Successes: What Have We Accomplished ? States and EPA agree that the May 17, 1995 Performance Partnership agreement put forth a sound set of principles to maintain a core level of environmental protection for all citizens. The principles guides a flexible process that adapts environmental goals to local conditions in a way that builds trust between states and EPA and serves both of us as we work together. The National Environmental Performance Partnership System (NEPPS) greatly improved trust and communication between EPA and State Environmental Agencies by encouraging more frequent discussions between Regional Administrators and State Environmental Commissioners and beginning to break down organizational and media- program barriers at both the EPA regional and state offices. One result of these improved communications was improved priority setting and planning among state environmental agencies and EPA Regional Offices. This has been demonstrated through: 1) improved internal priority planning by some state agencies; 2) increased joint EPA Regional and state priority setting and planning efforts (i.e., EPA Region V State's Environmental Performance Partnership Agreements); and 3) 1 11 ------- Appendix B Page 4 of 7 DRAFT development of self-assessments and strategic plans by some state agencies that improved internal or joint priority setting and planning. In some instances these efforts led to funding shifts and work sharing. The improvement in priority setting and planning in turn better positioned EPA and the states to focus national environmental protection efforts on results, creating a strong management partnership. It created demand for better integration of environmental data that yielded EPA's grant program to states, known as the "One Stop" grant program, the subsequent state/EPA Information Management Working Group, and EPA's structural reorganization creating the Office of Environmental Information. The states and EPA are now building an internet based Information Exchange Network, which will increase the speed of sharing data while driving down cost and improving efficiency, and accuracy. Also, the EPA/State Regulatory Innovation Agreement developed in part from these efforts, and for the first time, states and EPA agreed on performance measures to track progress towards results. Challenges: Impediments to Advancing Performance Partnerships EPA's multiple priority setting and planning processes including performance partnerships, development of national program guidance, and budgeting and accountability requirements of GPRA developed too independently of each other and at different times, and are not aligned in a way that fosters joint planning and priority setting across media program lines. Staff in both EPA and state environmental agencies have limited experience with a non- traditional, collaborative approach to environmental problem solving. Also, formal training for staff on how to manage programs to better focus on outcomes never occurred and tools are limited. A strong media program perspective and loyalties still dominate many aspects of state/EPA relationships and there are few incentives for state and federal staff to take on new ways of doing business or to face new risks. A Performance Partnership Agreement (PPA) is "in addition to" and many times conflicts with delegation agreements, national program guidance or other documents dictating some aspect of state -federal management of environmental programs. Transaction costs for developing performance partnership agreements are believed to be too high because of the perceived lack of flexibility in the process (# of hours spent planning exceeds number of hours doing environmental work). The expected benefits of a reduction of oversight and reporting were not realized. 2 12 ------- Appendix B Page 5 of 7 DRAFT Some states invested considerable resources in developing self-assessments about which they received no EPA feedback. Vision: What Should Performance Partnerships Be? A Performance Partnership embraces a data-driven system of joint state/EPA planning and priority setting that articulates: 1) federal and state government roles and responsibilities; 2) environmental goals; 3) performance measures and time frames; and, 4) resource needs. Ultimately, it provides a framework that can guide delivery of environmental services and environmental problem solving at all levels of government. This data-driven joint state-EPA priority setting and planning process would yield a set of agreed upon national, regional, and state and local priorities that build on the core responsibilities of each. It would provide the information foundation for individual states/Environmental Council Of the States (ECOS) to be further involved in improving EPA's budgeting and accounting systems, strategic planning, and development of environmental indicators and information management systems. It would be an evolving, learning partnership between EPA and the states that rewards innovation and innovators, encourages experimentation, and documents reduction in transaction costs for both sides. Joint EPA/state assessments of priorities and implementation strategies and annual or biennial performance evaluations would take place to measure progress, as well as the joint development of agreed upon performance measures. Accountability, oversight and reporting would be based on a data-sharing relationship between EPA and the states. From this we would jointly determine performance measures, measures performance against them, provides feedback and continuously improve them. Technological advancements that make information accessible to EPA, states and the public would be readily adopted. [For states with PPAs, jointly established priorities and plans for results would be articulated in a single operating document ~ the performance partnership agreement. This PPA would reflect the state's strategic plan. It would be a multi-media, integrated, holistic agreement between the state and EPA that addresses environmental performance expectations and needs. Criteria for what all such agreements should contain would be developed by these states and EPA. In addition, EPA and states would identify incentives 3 13 ------- Appendix B Page 6 of 7 DRAFT and rewards for PPA states to continue to improve this process. Further Discussion Necessary.] A Performance Partnership Advancement Agenda: Overcoming the Challenges To bridge some of the gaps between where performance partnerships are now and our vision for the future, there are several areas that EPA and the states can begin to address. Joint Priority Setting and Planning The state/EPA performance partnership will be based on joint priority setting and planning. Beginning the summer of 2002, in either a national forum or state-specific regional processes, EPA and states will meet to identify a set of national, state and regional priorities. These priorities will consider early information from our collective environmental indicators and performance work. They will be incorporated into EPA's national strategic planning process, and annual planning, budgeting and accountability processes for FY 2004. States are encouraged to participate in these annual EPA processes to ensure continuous improvement. These priorities will continue to inform and refine the development of national and state indicators, and the identification of investment needs for necessary data and monitoring needs. State priorities also will inform and influence the development of the FY 2004 Annual Performance Goals, targets, and national program guidance, which in turn will encourage flexible and innovative approaches to regional and state problem solving. Joint Capacity Building Based on Performance Partnership Principles EPA and the states jointly will review specific environmental program functions as identified by EPA national program managers (i.e., NPDES, RCRA Permitting) in order to improve efficiency and environmental impact of work. This would include a renegotiation of roles and responsibilities for that specific function, as well as a review of potential resource shifts. Based on EPA's Innovation Action Agenda, EPA and states will actively search for opportunities that display flexible and innovative solutions that produce measurable environmental results. As regulatory or statutory barriers are identified and agreed upon, they should be presented by the states and EPA to the EPA Administrator for further action within an agreed upon time frame. 4 14 ------- Appendix B Page 7 of 7 DRAFT To improve the states' and EPA's ability to solve environmental problems, we should provide incentives for continuous improvement and advancement of state and federal programs to achieve environmental results. Among the incentives that could be used are: o Developing strategies for collection of necessary data and monitoring information for development of environmental indicators. States and EPA jointly should identify barriers to achieving results due to lack of data, including the need for additional funding. o Accelerate development of system for shared data that would drive environmental results. o Market the successes and benefits and establish acknowledgment and rewards for long-term civil servants who support the cultural change. o EPA and the states should provide opportunities and incentives for staff to engage in continuous improvement. Among the incentives could be: Joint state-EPA evaluation of employees/managers; State and EPA senior leadership celebrating and sharing successes; Successful state and federal employees mentoring others; Regular state and EPA assessment of performance partnership implementation; More joint training opportunities for staff and senior managers; Incorporating performance partnership principles into everyone's expected job performance to include personal goal setting, performance evaluation and incentives, both financial and other; annual personnel performance reviews. A joint annual evaluation of individual performance partnership agreements should be institutionalized to ensure accountability and proper management. These evaluations should include discussions of program performance, delegations and oversight responsibilities as well as other kinds of appropriate process issues. EPA and state staff should be accountable in performance evaluations for implementing performance partnership agreements. EPA and the states with PPAs may chose to perform joint performance assessments annually or biennially in lieu of the self-assessments described in the May 17th Performance Partnership Agreement. 5 15 ------- 16 ------- Appendix C Distribution Headquarters Office Associate Administrator for Congressional and Intergovernmental Relations (1301 A) Deputy Associate Administrator for Intergovernmental Relations (1301 A) Comptroller (2731 A) Director, Grants Administration Division (3903R) Director, Office of Regional Operations (1108 A) Agency Followup Official (271 OA) Agency Audit Followup Coordinator (2724A) Audit Followup Coordinator, Office of the Administrator (1104) Regional Office Regional Administrators Office of Inspector General Inspector General 17 ------- |