OFFICE OF INSPECTOR GENERAL GOVERNMENT PERFORMANCE AND RESULTS ACT (GPRA) REVIEW GUIDE Assessing EPA and States' Systems For Measuring and Reporting on Environmental Performance U.S. EPA Headquarters, Ubrary Mai) oocte 3£01 AVeaue NW 20460 •» EPA 350/ 1999.4 JUNE 1999 • -.. ..•..•• • • "•':: '• • '''^niflteS'imfiyiT'T'"''''•'''"rT ------- GPRA REVIEW GUIDE PREPARED BY: Financial Audit Division CONTACT PERSON: GAIL SAUNDERS (202) 260-1476 ------- E PA \s fc (K UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 1999 OFFICE OF THE INSPECTOR GENERAL MEMORANDUM SUBJECT: FROM: TO: Government Performance and Results Act Implementation Review Guide James O. Rauch Assistant Inspe eneral for Audit Assistant AdnthTistrators General Counsel Regional Administrators Attached is our Government Performance and Results Act (Results Act) Review Guide (Guide), "Assessing EPA's and States' Systems For Measuring and Reporting on Environmental Performance" outlining the Office of Audit's overall strategy for reviewing and assisting the Agency in implementation of the Results Act. The Guide supplements our annual audit plan and issue area plans. It provides general background information and recommended review procedures that will be incorporated into our audit and advisory assistance assignments. The Guide provides our staff with another means of assisting EPA in reporting on environmental outcomes and measuring progress towards the Agency's mission and environmental goals. You may also want to consider using the Guide to assess your office's implementation activities. If you would like our staff to either perform an assessment of your Results Act activities, or assist your staff in performing a review, please contact Melissa Heist, Divisional Inspector General for Financial Audits at (202) 260-1479, or Gail Saunders at (202) 260-1476. Attachment cc: Chief Financial Officer (2710) US- EPA Headquarters Ubrary Ma my Ma" code 3201 Recycled/Recyclable «Printed with Vegetable Oil Based Inks on 100% Recycled Paper (40% Poslconsumer) ------- ------- TABLE OF CONTENTS MESSAGE FROM THE INSPECTOR GENERAL i INTRODUCTION 1 Purpose of the Plan 1 Relationship to OIG's and EPA's Missions 1 Background on the Results Act 2 Background on EPA's GPRA Implementation Efforts 3 Scope and Approach 5 Prior Audit Coverage 6 Reporting '. 6 AUDITING GPRA 7 Program/Performance Audits 8 Assistance and Other Agreement Audits 8 Contract Audits 9 Financial Statement Audits 9 PARTNERSHIP WITH STATE AUDITORS 11 Purpose 11 Background II Benefits 12 Pilot Partnership With Florida State Auditors 12 Procedures For Developing Partnerships With State Auditors 13 ATTACHMENT 1 - Procedures For Assessing EPA's GPRA Implementation ATTACHMENT 2 - Summary of GPRA Audit Procedures by Type of Audit ATTACHMENT 3 - GPRA Audit Checklist ATTACHMENT 4 - Reference Material ------- ------- MESSAGE FROM THE INSPECTOR GENERAL I am pleased to present the Office of Inspector General's (OIG) Government Performance and Results Act (GPRA) Implementation Review Guide. The audit work outlined in this guide will not only benefit EPA, but the nation as well, by providing: (I) assurances about the environmental data and systems EPA and states have in place to measure environmental progress, and (2) recommendations in areas where improvements are needed. In presenting the Fiscal 1999 Audit Plan, I stated that "the Plan not only continues our journey toward reaching our vision of being agents of positive change but also assists EPA to evaluate the accomplishment of its environmental goals and ensure the adequacy of accountability systems needed to measure performance." The GPRA Implementation Review Guide is an extension of that commitment. Our goals "as agents of positive change striving for continuous improvement in our Agency's management and program operations , " are to: (1) help EPA achieve its environmental goals by improving the performance and integrity of EPA programs and operations; safeguarding and protecting the Agency's resources; and, clearly reporting the results of our work; (2) foster strong working relationships; and (3) operate at the highest performance level. By implementing this guide, with an emphasis on results, we are assisting EPA in reporting on environmental outcomes and measuring progress towards the Agency's mission and environmental goals. This approach facilitates the OIG's contribution to the efficient and effective achievement of the Agency's mission and goals. JU. Nikki L. Tinsley Inspector General ------- ------- INTRODUCTION PURPOSE OF THE PLAN The Government Performance and Results Act (GPRA or the Results Act) was designed to improve the overall management and effectiveness of Federal agencies. The Results Act requires agencies to shift their emphasis from measuring activities (inputs and outputs) to measuring results (outcomes) and to focus on achieving desired results in making management decisions. Implementing the Results Act is expected to improve agencies' planning, budgeting and accountability processes by linking them together so that planning drives the budgeting process. Actual program results and the cost of achieving these results can be used to influence future planning. GPRA was intended to move the Government to a results oriented management approach. The purpose of the OIG GPRA Implementation Review Guide is to provide a coordinated approach for reviewing the Agency's GPRA implementation efforts as a part of our ongoing work. Congress attaches great importance to effective implementation of GPRA and believes Inspectors General have an important role to play in informing agency heads and the Congress on issues concerning efforts to implement the Act. This plan provides general background information and recommended procedures that can be integrated into performance audits, including performance audits of contracts and assistance agreements; financial statement audits; and other assignments, such as consulting services. The GPRA Implementation Review Guide supplements the OIG's annual audit plan and other issue area plans. The procedures contained in this plan provide audit teams with the necessary steps for incorporating GPRA concepts and requirements into their work. This guide focuses primarily on Agency/State systems for measuring progress towards EPA's goals and is not designed to evaluate the overall effectiveness of Agency programs. All audits and assistance assignments are different; therefore, the review procedures contained in this guide may not cover all circumstances or conditions encountered, and in some cases not all the procedures will necessarily apply. Auditors are to use their professional judgement to tailor the procedures to their audit objectives, taking into consideration the time and resources available to conduct the audit and the benefits to the Agency in completing the steps. RELATIONSHIP TO OIG'S AND EPA'S MISSIONS EPA-OIG was established to provide independent and objective information, advice, and recommendations to Agency managers based on audits, investigations, and other activities. The OIG mission is defined in the Inspector General Act of 1978 (Public Law 95-452), as amended. The Act requires the Inspector General to: (1) conduct and supervise audits and investigations relating to programs and operations in the Agency; (2) provide leadership and make 1 ------- recommendations designed to: (i) promote economy, efficiency, and effectiveness, and (ii) prevent and detect fraud and abuse in Agency programs and operations: and (3) fully inform the Agency Head and the Congress about problems and deficiencies identified by the OIG relating to the administration of Agency programs and operations. Our vision is: We are agents of positive change striving for continuous improvement in our Agency's management and program operations, and in our offices. Goal 10 of EPA's strategic plan, "Effective Management," states that EPA will seek to establish a management infrastructure that will set and implement the highest quality standards for effective internal management and fiscal responsibility within the Agency. Our GPRA Implementation Review Guide supports this goal, as well as EPA's other nine environmental goals. Effectively incorporating the Results Act concepts and requirements into the day-to-day management of Agency programs will facilitate the efficient and effective achievement of the Agency's goals. BACKGROUND ON THE RESULTS ACT Seeking to promote improved government performance and greater public confidence in government through better planning and reporting on the results of federal programs, Congress enacted the Government Performance and Results Act of 1993. As Congress and the Administration seek to reduce the cost and improve the performance of the federal government. there is broad consensus that holding agencies accountable for results is a key to better management of programs. Performance-based management, as envisioned by the Results Act. is a dynamic and complementary process of setting a strategic direction, defining annual goals and measures, and reporting on performance. Under the Results Act, agencies prepare multi-year strategic plans that set the general direction for their efforts. Then, beginning with fiscal year 1999, agencies are required to prepare annual performance plans that establish the connections between the long- term strategic goals outlined in the strategic plans and the day-to-day activities of managers and staff. Finally, the Act requires that each agency report annually on the extent to which it is meeting annual performance goals, and the actions needed to achieve or modify those goals that have not been met. Strategic planning is an effort to establish long-term goals and objectives that will shape and guide activities and programs to fulfill an organization's mission. Performance measures are a key tool to help managers assess progress towards the goals or objectives stated in their strategic plans. They are also an important accountability tool to communicate agency progress to Congress and the public. Program managers should analyze and use performance information to inform and influence their: ------- 1. program implementation decisions: for example, determining which environmental programs or activities to undertake; 2. planning decisions: for example, using information on performance trends to set challenging yet realistic program goals; 3. management decisions: for example, making decisions about aligning management systems and structures to support the organization's desired outcomes; and, 4. budget and resource allocation decisions: for example, using information about the connection between a given level of resources and a given level of results for a program to decide if additional budgetary expenditures will be beneficial. Most importantly, performance measures build a common results language among all decision- makers. Selected measures define what is important to an organization, what it holds itself accountable for, how it defines success, and how it structures its improvement efforts. Since passage of the Results Act Congress has expressed an interest in having Inspectors General involved in ensuring the Act was effectively implemented. On March 19,1998, the House of Representatives passed the "Government Performance and Results Act Technical Amendments" (HR 2883); however, no further action was taken on the bill. The amendments would have required OIG verification and validation of selected supporting data sources and information systems used for performance measures outlined in agency performance reports and strategic plans. A summary of the findings, results, and recommendations from completed OIG reviews would have been required to be included in the OIG's semiannual reports. On October 7. 1998. the leadership of the House of Representatives instructed the Inspectors General Community to develop and implement a Results Act review plan as envisioned by HR 2883. irrespective of whether or not the bill becomes law. In addition, in a letter to IGs on December 17,1998, the Chairman, Senate Committee on Governmental Affairs, recommended two areas IGs should consider for GPRA work in the future: (1) determining if key performance measures the Agency has chosen are consistent with the goals in its strategic and annual performance plans, and (2) verifying the accuracy and reliability of the underlying performance data. BACKGROUND ON EPA'S GPRA IMPLEMENTATION EFFORTS In the spring of 1997, the Agency formally reorganized several of its key management functions to tighten the links among planning, budgeting, and accountability. The agency created the Office of the Chief Financial Officer (OCFO). One of the functions of the OCFO is to lead the Agency's efforts to implement the Results Act. The Act requires the Agency to adopt the tools of performance-based management: a five-year strategic plan that includes a mission, a set of goals. ------- measurable objectives to achieve those goals, and performance measures to identify progress against the objectives. In addition to the strategic plan, the Agency is required to develop annual performance plans tied to the strategic plan and budget requests. By* March 31, 2000 and at the same time in all subsequent years, the Agency is required to submit performance reports comparing performance against goals. The Act also requires Federal agencies to include findings of program evaluations completed during the fiscal year in their performance reports. Strategic Plan. The Agency issued its 5 year strategic plan in September 1997, charting the Agency's course for protecting human health and the environment. The guide was developed with input from EPA's senior leadership and employees, as well as from environmental stakeholders. This guide describes the Agency's 10 goals and related objectives and discusses strategies for achieving them in the years ahead. The plan must be updated by September 30, 2000. Annual Performance Plan. The Act also requires federal agencies to prepare annual performance plans covering each program activity set forth in the Agency's budget. The fiscal 1999 President's Budget/Annual Performance Plan was the first budget submission prepared to meet the GPRA requirements by identifying EPA's activities and requesting resources by goal and objective. It was submitted to Congress February 1998. Annual Performance Reports. An accountability workgroup was convened in December 1997 to develop a new accountability process for EPA. The workgroup recommended a two track approach to implementing GPRA: Track A, beginning in fiscal 1998, was defined as a project to plan, develop and carry out the Agency's initial accountability process to ensure ftill compliance with GPRA requirements for the fiscal 1999 annual performance report. Track B, also beginning in fiscal 1998, was defined as a project to plan, develop, and carry out the Agency's longer-range accountability process, which will support efforts to manage for environmental results and produce annual performance reports for fiscal 2000 and beyond. Program Evaluations. The Chief Financial Officer (CFO) issued guidance on program evaluations to program offices on September 30, 1998. In defining what a program evaluation is. the CFO quoted OMB Circular A-l 1, Part 2 - Program Performance Reports (revised July 1998). The criteria defines a program evaluation as "an assessment, through objective measurement and systematic analysis, of the manner and extent to which federal programs achieve intended objectives." OMB also includes process evaluations in its definition of a program evaluation (i.e., evaluations of the mechanisms by which programs achieve their goals or objectives). The CFO's guidance states that EPA's National Program Managers (NPMs) are to conduct program evaluations and to provide input on how these evaluations were used in revising the strategic plan and annual performance plan; and to provide a schedule for future evaluations. NPMs are also to provide the findings of their program evaluations for inclusion in the Agency's annual performance reports. ------- Data Quality. The Agency's Data Quality Strategic Plan was developed in response to the Deputy Administrator's request for an Agency-wide approach to ensuring the quality of EPA's data. The plan recommends several items to improve data quality, including: 1. the development of data quality performance standards for each of EPA's major data systems to track and improve data quality over time. 2. an error correction process to ensure that discrepancies in EPA data are routed to the appropriate data managers. 3. the establishment of customer service performance standards for each major data system to ensure that discrepancies are addressed promptly and appropriately. In a report accompanying EPA's Appropriations Bill for fiscal 1999, the Senate Committee on Appropriations expressed concerns about EPA's data quality in reporting on performance measurement and right-to-know programs. The committee instructed EPA to report quarterly on its progress in addressing data quality and stewardship concerns. The first quarterly report was issued on March 22, 1999. To meet the information challenges of the 21st century, the Administrator announced on October 28,1998 her intention to create a new information management organization. The new office will offer the advantage of consolidated management to implement the data quality strategic plan recommendations. The Data Quality Staff and Data Quality Board contemplated in the new organization offer the opportunity to champion implementation of the plan. SCOPE AND APPROACH The OIG is committed to helping Agency managers use Results Act principles in their day-to-day activities in order to more effectively deliver environmental protection. Our goal is to work with Agency managers so implementation of the Act achieves the desired results and does not become a paperwork exercise. Audit teams, by integrating GPRA concepts and requirements into their work, can help the Agency in making progress towards achieving its strategic goals, objectives, and mission. The generic review procedures contained in this guide are designed to assist OIG staff in carrying out this work. In August 1998, the OIG Office of Audit held a nationwide GPRA strategy meeting to gain an understanding of the purposes and requirements of GPRA; learn about what EPA is doing to implement the Act; discuss ways to integrate the Act's requirements into our planned audit and assistance work; and discuss ways to measure our success in helping the Agency meet its goals. We anticipate having additional meetings to get updates from Agency managers on the status of their implementation efforts and areas where we could be of assistance to them. These meetings will also allow us to share information and better coordinate our work. ------- A related ongoing OIG effort is the Data Quality Audit Strategy Workgroup formed in October 1998. with the objective of developing a strategy for auditing data quality in EPA's mission- critical information systems. The workgroup plans to identify major-application systems and, more specifically, data elements which are being relied on: (1) to assess Agency progress on the ten designated GPRA goals, or (2) for other crucial reporting purposes. The workgroup expects to develop and issue the Strategic Audit Plan for Data Quality during the third quarter of fiscal 1999. Additionally, we are proposing to form partnerships between EPA OIG and state auditors to have them assess the reliability and validity of the environmental performance data provided by state agencies to EPA. The partnerships would benefit both the state agencies and the EPA by providing assurances about the data used to make decisions about the environment. PRIOR AUDIT COVERAGE The Financial Audit Division (FAD) conducted a survey of "EPA's Implementation of the Government Performance and Results Act," (Audit Report No, E1AMF5-20-0013-6100297, - issued September 25, 1996) in response to the Agency Chief Financial Officer's request that we review EPA's ongoing and planned efforts to implement GPRA. The survey identified some areas of concern with the Agency's performance and financial data it planned to use in reporting on the Agency's progress in achieving it goals and objectives, and the need to develop and implement program evaluations at EPA. Numerous audits have identified a longstanding history of problems with EPA's information systems. Audits of EPA programmatic areas often have a component relating to environmental data information systems, and we have reported on numerous data systems deficiencies. In responding to a request from the House of Representatives leadership, we reported environmental data information systems as one of the ten management challenges confronting the EPA. REPORTING The audit procedures contained in this plan were designed to help ensure GPRA coverage for audit and assistance work conducted in fiscal 1999 (after issuance of this plan) and beyond. The procedures will help focus our work on environmental outcomes and provide a means of evaluating and giving the Agency feedback on its implementation of the Results Act. Findings and recommendations should tie into the appropriate EPA goals, objectives, and/or sub-objectives for the entity being audited. To the extent possible, findings and recommendations should be tied into audit objectives dealing with program effectiveness/results rather than being reported in a separate chapter or an attachment to the report. This reporting approach will support our goal of helping the Agency use Results Act principles in its ongoing environmental program activities. ------- AUDITING GPRA To address the wide variety of audit and assistance services we provide the Agency, we have categorized our GPRA review work into four functional areas: 1. Program/Performance Audits: During these audits, we determine the extent to which the desired environmental results or benefits envisioned by the Administration and Congress are being achieved; review the economy, efficiency and effectiveness of Agency operations; and determine the extent of compliance with applicable laws and regulations. 2. Assistance and Other Agreement Audits: These are audits of EPA'sinteragency agreements, Performance Partnership Grants, and other assistance agreements. An Interagency Agreement is the mechanism used to obtain or provide services to or from another Federal agency. Performance Partnership Grants and other assistance agreements provide assistance to state, local and tribal governments, universities and nonprofit recipients. Our objectives are to ensure that the scope and goals of the assistance agreements are being achieved (which will indicate progress the Agency is making towards achieving its goals); EPA funds are being properly accounted for and effectively managed; and, the Agency is receiving what it is paying for. j. Contract Audits: EPA gets much of its work "protecting human health and the environment" done and produces many of its major products through contracts. These are audits to assist EPA in ensuring that it pays a fair and reasonable price for services and products it requested. We also plan to do some work in the future, looking at contractors' performance in relation to EPA's environmental mission. 4. Financial Statement Audits: These audits consist of evaluations of the Agency's financial systems and statements to ensure that the Agency's accounting information is accurate, reliable • and useful; adequate accounting controls are in place; and EPA complied with applicable laws and regulations. Our objective in performing these annual audits is to assist EPA in making improvements in the financial management processes and controls which will provide better information for decisions promoting the greatest possible environmental results. For each of the above audits categories, the OIG is committed to helping Agency management identify how it can improve the use of its resources and authority for the greatest environmental protection. By incorporating Results Act principles and requirements into our audits, we can better focus our work on environmental results and assist the Agency in carrying out its mission. The following sections of this guide deal with how we will do this in each type of audit. The same or similar steps should also be considered for inclusion in our advisory and assistance work. ------- 1. PROGRAM/PERFORMANCE AUDITS Performance audits focus primarily on issues based on their relative risk, materiality, and importance to EPA's mission. These audits evaluate and report on the Agency's accomplishments of its environmental mission, thereby helping to increase EPA's efficiency and effectiveness and to identify and prevent fraud, waste, and abuse. The OIG is focusing its program audit resources in the upcoming years in the following "issue areas": Enforcement and Compliance Assurance, Performance Partnership Grants, Resource Conservation and Recovery Act, Water Quality, and Superfund. Issue areas allow us to provide comprehensive audit coverage which can significantly improve the Agency's effectiveness and credibility. Performance audits are conducted by OIG divisional audit offices nationwide. The divisional office conducting the audit will be responsible for reporting the results of the audit, including work related to the Results Act. Auditors should incorporate the following recommended GPRA review procedures (that relate to the scope and objectives of the area being audited) into their audit guide. Recommended GPRA review procedures: Sections A through C, D, E and I. NOTE: The GPRA review procedures referenced above (and in the subsequent sections) can be found at Attachment 1 - Procedures for Assessing EPA's GPRA Implementation. '2. ASSISTANCE AND OTHER AGREEMENT AUDITS EPA accomplishes a large part of its environmental mission through assistance agreements awarded to states, local and tribal governments, universities, and nonprofit recipients. Our GPRA related audit work in this area will focus on: Performance Partnership Grants (PPGs). other Assistance Agreements and Interagency Agreements. a. Performance Partnership Grants - This assistance program gives states and tribes the option to combine funds from two or more grants, so they have the flexibility to address their highest environmental priorities while continuing to address core program commitments. PPGs are part of the National Environmental Performance Partnership System (NEPPS) which gives states a stronger role in priority setting; focuses scarce resources on their highest priorities; and tailors the amount and type of EPA oversight to an individual state's performance. Our goal is to provide EPA program managers with useful information to set up and monitor the PPG and NEPPS programs, so they achieve maximum environmental results. Recommended GPRA review procedures: Sections B, C, F and I. ------- b. Interagency/Assistance Agreements - Interagency Agreements are services provided to and from another Federal agency, and Assistance Agreements are the transfer of anything of value for a public purpose of support or stimulation authorized by law. Our audit objectives are to determine whether the activities funded by EPA were within the Agency's assistance authorities as described in the relevant statutes and to assess the environmental benefits received for the assistance provided. Recommended GPRA review procedures: Section F. 3. CONTRACT AUDITS GPRA reviews conducted as a part of performance audits of EPA's contracts are intended to provide the Agency with an assessment of the contractors' performance in relation to EPA's environmental mission. The Agency can use this information to determine the environmental benefits received for the EPA resources expended. Recommended GPRA review procedures: Sections A and G. 4. FINANCIAL STATEMENT AUDITS In accordance with the requirements of the CFO Act. as amended by the Government Management Reform Act, the OIG is required to audit the Agency's annual financial statements. OMB Bulletin 97-01, "Form and Content of Agency Financial Statements" requires that the Overview section of the financial statements contain a brief narrative discussion which describes the reporting entity and its mission, activities, program and financial results, and financial condition. At a minimum, the Overview should contain sections which address, among other things, performance goals and results. OMB Bulletin 98-08, "Audit Requirements for Federal Financial Statements" requires auditors to obtain an understanding of the components of internal control relating to the existence and completeness assertions relevant to the performance measures included in the Overview of the Reporting Entity. Those internal controls that have not been properly designed and placed in operation should be reported. OMB Bulletin 98-08 does not require that we test whether the controls are effective. Since the Agency has acknowledged weaknesses in the quality of the environmental data it has available to manage its environmental programs, in the past we have performed limited work in this area. Beginning in fiscal 1999, EPA implemented new cost accounting procedures in order to accumulate and report total EPA costs expended on each goal. The cost information is reported ------- in the Statement of Net Costs, one of the principal financial statements. As a part of the audit of this statement, we will evaluate the systems, processes and controls the Agency has in place to accumulate and report on the costs of its programs. Specifically, we will audit the cost reported for each of the Agency's goals. This will allow us to assist the Agency in improving the accuracy and timeliness of cost information. Recommended GPRA review procedures for Overview: Section H. 10 ------- PARTNERSHIP WITH STATE AUDITORS PURPOSE The purpose of a partnership between EPA OIG and state auditors is to have state auditors assess the reliability and validity of the environmental performance data provided by the state agencies to EPA and make recommendations for improvement, if necessary. This work would benefit both the state agencies and EPA since it would provide assurances about the quality of the environmental data that is being used in making decisions about the nation's environmental protection. It would also provide a means of identifying and correcting data quality problems. BACKGROUND In broad terms, EPA is a regulatory agency assigned the task of controlling pollution that threatens human health and the environment. Much of the responsibility for environmental protection has been delegated to the states. EPA headquarters is responsible for developing national policies, standards, and guidance; the regional offices implement federal programs or monitor the states' implementation of them. Much of the data that EPA uses to manage its programs and measure progress towards its environmental goals comes from the states, and Congress holds the Agency accountable for state actions in managing environmental programs delegated to them. The Results Act requires the Agency to describe the procedures that will be used to verify and validate performance data in its annual performance plan. The procedures should be credible and specific to ensure that performance information is sufficiently complete, accurate and consistent to document performance and support decision-making. Validation is an effort to ensure that data are free of systematic error or bias and that what is intended to be measured is actually measured. Verification is a process of checking or testing performance information to assess other types of errors, such as errors in keying data. OMB Circular No. A-l 1, Part 2, "Preparation and Submission of Annual Performance Plans (revised 1998)," section 220.12 - Verification and Validation, states: x* "Federal agencies are not required to develop an independent capacity for verifying or validating performance data received from, or based oh, sources outside the agency. Data sources outside the agency should be identified in the plan. Available information from an outside source regarding the accuracy and reliability of its data should be collected, but need not be included in the plan." 11 ------- As EPA gives more discretion and decision-making authority to states, EPA will need to revise its accountability methods. This shift will change the nature of how-EPA reviews state performance. At the heart of EPA's change is the use of accountability measures that will encourage states, firms and individuals to be more innovative by focusing on results. EPA and state leaders agreed in August 1997 to use a national set of core performance measures for assessing the nation's progress in protecting public health and the environment. Although core performance measures are not perfect, a well-balanced set of indicators gives state program managers. EPA officials and the public a useful sense of how a state is doing relative to both its own aspirations and the performance of other states. BENEFITS Forming a partnership between EPA OIG and state auditors enables EPA to, "potentially" forgo review of state repotted performance data in order to ensure the quality of state environmental data used in carrying out national environmental programs. Poor data quality, as measured by state auditors, would be subject to additional oversight and EPA control. EPA could negotiate tighter regional and state agreements and review the use of Federal funds more closely. PILOT PARTNERSHIP WITH FLORIDA STATE AUDITORS Florida has a law similar to the Federal GPRA. in that it requires each department to develop a performance results driven budget and system to measure and report progress against the planned goals. As pan of this plan, individual departmental Offices of Inspector General, including the OIG for the Department of Environmental Protection, are responsible for auditing performance measures and data reported to support the state's performance goals. Implementing a partnership between EPA OIG and Florida OIG Department of Environmental Protection would provide a means to validate the accuracy of the performance data received from the state of Florida. We will use the lessons learned in this partnership to decide how. and if, we will pursue similar partnerships with other state audit organizations. A pilot partnership with Florida state auditors is proposed for fiscal year 1999. Representatives from EPA OIG and Region 4 should meet with Florida's OIG for the Department of Environmental Protection to consider a partnership to exchange and share audit procedures and results on the assessment of environmental measures, data and results. Successful implementation of the partnership would result in the state auditors reporting the results of their findings and recommendations to the Southern Audit Division. The auditors could then report the results to the appropriate EPA program officials for needed action. 12 ------- PROCEDURES FOR DEVELOPING PARTNERSHIPS WITH STATE AUDITORS Following are recommended procedures for developing and implementing a partnership with state auditors that may be modified at a later date based on our experience with Florida. EPA auditors should determine if there is an audit organization that has responsibility for reviewing the performance data the state reports to EPA. If it is determined that such an audit organization exists: 1. Obtain and review a representative sample of EPA assistance agreements with the state. 2. Identify the national core performance measures in assistance agreements selected. 3. Meet with the state auditors and determine if they have plans to audit or have some other group audit the national core performance measurement data outlined in the EPA/state assistance agreements? 4. If an audit program has been developed, obtain and review the scope and objectives. Identify those objectives that could provide valuable feedback to EPA program managers. If no audit program has been developed, discuss with state auditors an audit scope and objectives that will benefit both EPA and state program managers. 5. Discuss with the state auditors their reporting requirements and ways this information could be provided to EPA managers. ------- ------- Attachment 1 Procedures for Assessing EPA's ' GPRA Implementation PURPOSE The purpose of the following audit procedures is to provide Office of Inspector General staff with the necessary guidance (steps) to assess the auditee's implementation of applicable Results Act requirements and the effect of implementation on the efficiency and effectiveness of the Agency's or the state's activities in meeting EPA's environmental goals (These steps are to be incorporated into audit programs as previously outlined in each functional area discussion). The method and extent of verification for any of these steps should be based on the auditor's professional judgement. AUDIT OBJECTIVES AND STEPS A. Agency Program Reviews - The purpose of performing these audit procedures is to . provide the auditors) with an understanding of how the audit entity fits into the Agency's overall mission and its 10 goals (and the supporting performance measures) and to gain an understanding of the system of internal controls as it relates to the data in support of EPA's GPRA reporting. In addition, this information will provide the foundation for auditing GPRA in subsequent years. Audit objectives: (1) How does the audited activity contribute to the Agency successfully accomplishing it environmental goal? (2) What systems are in place to track, manage and report on this contribution? 1. Identify and determine how the audit area relates to the Agency's goals, objectives, and/or sub-objectives. 2. If there is a direct link between the audit area and the Agency's goals, objectives, and/or sub-objectives, determine the following: a. what information/measures are being reported in support of the Agency's goals, objectives, and/or sub-objectives (GPRA reporting requirements); are the measures consistent with the goals, objectives and sub-objectives; and how is the performance information being used to plan and manage the day-to-day operations of the program? b. are goals established for regional offices? If so, how are they set? c. what is the source of the information being reported in support of EPA's GPRA reporting requirements? ------- B. Attachment 1 d. identify any automated and/or manual systems used to gather and report performance data. e. gain an understanding of the system of internal controls relevant to the performance information being reported for GPRA. 3. If there is no link between the audited activity and the Agency's goals, objectives, and/or sub-objectives: a. where are the costs of the activity being charged? b. discuss with program managers the environmental benefits of the program's activities, and c. with the assistance of program managers, assess the benefits of continuing the program's activities. 4. How is performance data made available to management? a. how has this data been used by management for policy-making, and for management and evaluation of the agency's progress toward meeting its goals? b. have/will any changes in funding, effort, or priorities result from the reported performance? c. has the agency clearly communicated the performance data/results to citizens, elected officials and other stakeholders? State Program Reviews - EPA is a regulatory agency that provides guidance and oversight to the states for implementing environmental protection within their respective states. The purposes of performing these audit procedures is to ensure the Agency is obtaining relevant, timely, accurate and reliable performance data for managing its environmental programs. Audit objective: Are states providing the Agency with data needed to assess progress towards its environmental goals and objectives? 1. Obtain and review any relevant state/EPA agreements (issued after August 1997) to determine if the agreements contain the required EPA performance measures. Determine the following: a. does the agreement contain performance measures for the audit entity? ------- Attachment 1 i. if no, why not? b.'are the required state related "national core measures" included in the agreement? If not, why? What are the region/state plans to include the measures? c. are the measures quantifiable, measurable, and verifiable? Do they measure environmental results? Are they tied to EPA's goals, objectives and sub- objectives? If not, why? Determine what data is used to measure performance. a. is the data accurate? i. does data support the associated measure? ii. is data based on statewide and systematic assessments? iii. if data is not being captured, what are the state's plans for capturing the data? iv. is there an existing data base to measure future results? b. how is data captured? Who captures it? When is it captured? Systematically? Ad-hoc? Do external sources provide performance data, and is the source recognized and generally acceptable as a reliable source for this data? c. does the state have adequate procedures/controls to verify and validate the data (is the data audited)? d. what resources are dedicated to collecting, verifying, and validating performance data? Do they appear insufficient/excessive? Determine the methods used for reporting performance. a. how does the state report performance measures to the region? b. are the reports complete, accurate, and timely? Do they include any qualifiers on the quality of the data? i c. did the state report on all of the required performance measures? ------- Attachment 1 4. Identify whether the state uses any performance measures other than those included in the state/EPA agreement. a. what measures does the state use to measure performance/manage the area? b. where there are differences between these measures and those identified in state/EPA agreements, consider whether the state measure is appropriate for measuring environmental performance. C. Regional Program Reviews - The purpose of performing these audit procedures is to ensure the regional program offices are providing appropriate state monitoring for implementing their respective environmental programs. In addition, these audit procedures will ensure the regional program offices are collecting timely, relevant and reliable performance data needed to assess how well their programs are progressing towards EPA's goals and objectives. Audit objective: Are EPA regional offices providing adequate oversight and monitoring of states to ensure effective and efficient implementation of the environmental programs delegated to them? 1. Obtain and review the most current memorandum of agreement between the regional program office and headquarters for the audit area. Determine the following: a. does the agreement contain performance measures for the audit area? b. are annual performance measures included in the agreement consistent with the Agency's GPRA reporting requirements? c. will the measures help assess the region's progress in achieving the relevant Agency goals, objectives and sub-objectives? d. are the required state related "national core measures" included in the agreement? If not, why, and what are the region/headquarters plans to include the measures? e. are the measures quantifiable, measurable, and verifiable? Do they measure environmental results? If not, why? 2. Determine what data is used to measure performance. a. specifically, what data does the region use to measure its performance in the audited area and is the measure consistent with the relevant EPA goal, objective ------- Attachment 1 and sub-objective? b. does the data support the associated measure? c. are the regional program offices capturing the data needed to measure their performance? If not, what are the region's plans for capturing the data? d. how is the data captured (by whom, when, systematically or manually)? Do external sources provide performance data, and is the source recognized and generally acceptable as a reliable source? e. how do regional program officials ensure the accuracy of the data used to measure performance? f. does the region have adequate procedures/controls to verify and validate the data, or assess the state's process? g. what resources are dedicated to collecting, verifying, and validating performance data? Do they appear insufficient/excessive? Determine the methods used for reporting performance. a. how does the region report the results of the performance measure to headquarters? b. are the reports complete, accurate, and timely? Do they include any qualifiers on the quality of the data? c. did the region report on all of the measures in the memorandum of agreement? Determine if the region uses any measures outside the ones included in the memorandum of agreement with headquarters. a. what measures does the region use to measure performance/manage the audited entity? b. where there are differences between these measures and those identified in the memorandum of agreement, consider whether the region's measures are appropriate for measuring environmental performance? ------- Attachment 1 D. Data Quality Reviews - Sound information resources and efficient, responsive information systems are critical to EPA's ability to achieve its environmental mission. The information that EPA uses to manage its environmental programs must be accurate, reliable and accessible to authorized users in order for the Agency to meet its organizational goals. Good data will assist the Agency in making sound regulatory and program decisions. Data quality reviews focus on Agency systems that support externally reported performance measures associated with EPA's "vital few" annual performance goals and measures. Audit objective: Will the Agency's information systems provide relevant, timely, accurate, complete, and consistent data for reporting on GPRA? 1. Determine if the reported data is a meaningful measure of progress being made on the relevant annual performance goal. 2. Identify and test key controls designed to ensure the timeliness, accuracy, completeness and consistency of the relevant data. 3. Test the reliability of the data (i.e. timely, accurate, consistent and complete). 4. If necessary, identify steps that need to be taken to improve the reliability of the data. E. Accounting for costs under GPRA - In response to GPRA. the Agency prepared the fiscal 1999 annual plan and budget linking resources with the Agency's goals, objectives, and sub-objectives. The Agency replaced its program element structure with program results codes, which specify the goal, objective, sub-objective and National Program Manager. Accounting will occur at the sub-objective level. Audit objective: Will the Agency's accounting system accurately capture the costs associated with its environmental goals, objectives and sub-objectives? 1. Obtain a copy of the relevant sub-objective description(s). 2. Compare the activities described in the sub-objective description(s) with the activities being performed. 3. Test a sample of the time-sheets to ensure time is being charged against the appropriate program results code. 4. Discuss with program managers of the audited entity whether they collect and use cost information to manage their program activities. If no, discuss what cost information would be beneficial for managing their program activities. 5. Provide the Financial Audit Division with copies of any position papers, memoranda. ------- Attachment 1 or reports that are issued to regional offices, so they can be shared with Headquarters staff responsible for the Agency's overall financial operations. Assistance Reviews - Assistance agreements account for about one half of the Agency's total budget. Therefore, the ability of assistance agreement recipients to achieve the agreement objectives and results will affect the Agency's achievement of its goals. The purpose of performing these audit procedures is to determine the environmental benefits received for EPA resources expended for assistance agreements. Audit objective: What are the environmental benefits of assistance agreements? Obtain and review the assistance agreement and determine: 1. How the assistance agreement relates to the Agency's goals and objectives (what are we accomplishing or what are the expected outcomes)? 2. Whether the assistance agreement's deliverables are clearly communicated and quantifiable? 3. Whether the deliverables were received timely and supported EPA's environmental goals as outlined in the assistance agreement? 4. How program officials used assistance agreement deliverables to further EPA's mission and goals? Contract Audits/Reviews - EPA gets much of its work done and produces many of its major products through contracts. The OIG is committed to assisting EPA in ensuring that it pays only for the services and products it requests and receives. Audit objective: What are the environmental benefits of contracts? Obtain and review the contract and determine: H. 1. How the contract relates to the Agency's goals and objectives (what are the expected environmental outcomes)? 2. Whether the contract deliverables are clearly communicated and quantifiable? 3. Whether the deliverables were received timely and supported EPA's environmental goals as outlined in the contract? 4. How program officials used contract deliverables to further EPA's mission and goals? Financial Statement Audits - OMB Bulletin 97-01, "Form and Content of Agency Financial Statements," states that "performance measures presented in the overview and as ------- I. Attachment 1 other accompanying information should be consistent with the measures used by the agency as_part of their Government Performance and Results Act (GPRA) implementation efforts." The Financial Audit Division will perform these audit procedures to ensure that the Agency is consistently and accurately reporting its program accomplishments under GPRA. Audit objectives: Is the performance data reported in the financial statements overview consistent with the Agency's GPRA reporting, as required by OMB Bulletin 97-01? Are there controls in place to ensure the reliability of the data? Identify and review the performance goals and results reported for each of the reporting entities and: 1. Determine if all program offices have included performance goals and results in the overview that are consistent with the Agency's goals and objectives. 2. Identify the source of the performance data reported, including automated and manual systems, whether from regions, state or tribes, or other reporting institutions. 3. Review prior audit reports which included a review of the performance data reported in the overview to the financial statements. 4. Annually select two or more reporting entities, time permitting, and obtain an understanding of the system of internal controls used to monitor and report performance data. Selection of the reporting entities will be based on the dollars budgeted for the programs, beginning with the highest dollar programs. This process is to continue until internal controls have been identified for all major reporting entities. 5. Annually update the auditors understanding of the reporting entity's internal control system for those that have been reviewed in prior years and follow-up on prior-year recommendations. Program Evaluations - GPRA requires the agency to include in its strategic plan a description of the program evaluations used in establishing or revising general goals and objectives, with a schedule for future program evaluations. The Agency's strategic plan states that program evaluations are one approach to evaluating and reporting progress towards goals and objectives. The strategic plan further states that program evaluations will analyze relationships between activities being conducted and environmental results. Audit objective: Are program offices conducting and using program evaluations to assess program effectiveness? 1, Determine if a program evaluation covering the audit entity has been conducted by the Agency, OIG, GAO or some other organization. If yes, proceed with a. below, if no, proceed with 2. below. ------- Attachment 1 a. what methodology was used, and what was the general scope of the evaluation, as well as particular issues addressed? b. identify the results of the program evaluation and determine if and how the results affected the audit entity's planning and day-to-day operations. c. do the results warrant revisions to the program's goals, objectives and performance measures? 2. What are management's plans for future program evaluations? ------- Attachment 2 GPRA Review Procedures by Type of Audit Auditing Procedures A. Agency Program Reviews B. State Program Reviews C. Regional Program Reviews D. Data Quality Reviews E. Accting for Cost Under GPRA F. Assistance Reviews G. Contract Audits/Reviews H. Financial Statement Audits I. Program Evaluations Performance / / / / ^ / Assistance / / / / IAGs - / Contracts / / Financial Stmt. / ------- Attachment 3 GPRA Audit Checklist - ftuirtc Refprenrf £frurfur0 fnr Assessment and Rpnnrlintr Components Questions Goals - Are (or do) they: (Strategic and Performance) S clear, precise and results oriented? /expressed in a way to allow for future assessment or measurement? / realistic in terms of given resources and timeframes? / relate to the mission statement and cover the major functions and operations of the Agency (or program)? / consistent with the statutory requirements and Congressional priorities? - Are (or do) they / clear, specific and measurable (Agency and Program) / relate to either outputs or outcomes? / practical and cost beneficial? / attainable within a specific period? / relate to an Agency performance goal? / relate to any internal or external benchmarks or base levels? Data- / Does it exist & is the source identifiable? / Is it relevant, reliable, valid, accurate, complete? / Is it free from influence and bias in collection and presentation? / Are significant limitations identified and accounted for? / Is it consistent (if recreated, counted or calculated or would it give the same results)? Data Collection System/Controls / How reliable are the systems or methods used to collect data (External vs. internal, manual vs. automated management, questionnaires vs. telephone interview)? / What plans/procedures (internal management controls) exist to verify and validate performance data? S Are adequate resources (technology or personnel)available to develop systems needed to obtain and manage the data? ------- Attachment 4 Reference Material EPA Strategic Plan; EPA/190-R-97-002, September 1997 (http://www.epa. eov/ocfopaeeA EPA Fiscal 1999 Performance Plan/Agency Budget Government Performance and Results Act (GPRA), PL 103-62,1993 Chief Financial Officers (CFO) Act, PL 101-576,1990 Government Management and Reform Act (GMRA), PL 103-356, 1994 OIG Issue Area Plans, OIG Intranet Enforcement and Compliance Assurance, September 1998 Performance Partnership Grants and National Environmental Performance Partnership System, September 1998 Resource Conservation and Recovery Act Program, May 1998 Superfund Program, July 1997 Water Quality, September 1997 Core Performance Measures for FY 1998, June 1997 OMB Circular No. A-l 1, Part 2, "Preparation and Submission of Annual Performance Plans (revised 1998)" Executive Guide - Effectively Implementing The Government Performance And Results Act, GAO/GGD-96-118, June 1996 & many other GAO reports. ------- |