OFFICE OF INSPECTOR GENERAL
GOVERNMENT PERFORMANCE
AND RESULTS ACT
(GPRA)
REVIEW GUIDE
Assessing EPA and States' Systems For
Measuring and Reporting on Environmental Performance
U.S. EPA Headquarters, Ubrary
Mai) oocte 3£01
AVeaue NW
20460
•» EPA
350/
1999.4 JUNE 1999
• -.. ..•..•• • •
"•':: '• • '''^niflteS'imfiyiT'T'"''''•'''"rT
-------
GPRA REVIEW GUIDE
PREPARED BY:
Financial Audit Division
CONTACT PERSON: GAIL SAUNDERS
(202) 260-1476
-------
E PA
\s
fc
(K
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
1999
OFFICE OF
THE INSPECTOR GENERAL
MEMORANDUM
SUBJECT:
FROM:
TO:
Government Performance and Results Act
Implementation Review Guide
James O. Rauch
Assistant Inspe
eneral for Audit
Assistant AdnthTistrators
General Counsel
Regional Administrators
Attached is our Government Performance and Results Act (Results Act) Review Guide
(Guide), "Assessing EPA's and States' Systems For Measuring and Reporting on Environmental
Performance" outlining the Office of Audit's overall strategy for reviewing and assisting the
Agency in implementation of the Results Act. The Guide supplements our annual audit plan and
issue area plans. It provides general background information and recommended review
procedures that will be incorporated into our audit and advisory assistance assignments. The
Guide provides our staff with another means of assisting EPA in reporting on environmental
outcomes and measuring progress towards the Agency's mission and environmental goals. You
may also want to consider using the Guide to assess your office's implementation activities.
If you would like our staff to either perform an assessment of your Results Act activities,
or assist your staff in performing a review, please contact Melissa Heist, Divisional Inspector
General for Financial Audits at (202) 260-1479, or Gail Saunders at (202) 260-1476.
Attachment
cc: Chief Financial Officer (2710)
US- EPA Headquarters Ubrary
Ma my
Ma" code 3201
Recycled/Recyclable «Printed with Vegetable Oil Based Inks on 100% Recycled Paper (40% Poslconsumer)
-------
-------
TABLE OF CONTENTS
MESSAGE FROM THE INSPECTOR GENERAL
i
INTRODUCTION 1
Purpose of the Plan 1
Relationship to OIG's and EPA's Missions 1
Background on the Results Act 2
Background on EPA's GPRA Implementation Efforts 3
Scope and Approach 5
Prior Audit Coverage 6
Reporting '. 6
AUDITING GPRA 7
Program/Performance Audits 8
Assistance and Other Agreement Audits 8
Contract Audits 9
Financial Statement Audits 9
PARTNERSHIP WITH STATE AUDITORS 11
Purpose 11
Background II
Benefits 12
Pilot Partnership With Florida State Auditors 12
Procedures For Developing Partnerships With State Auditors 13
ATTACHMENT 1 - Procedures For Assessing EPA's GPRA Implementation
ATTACHMENT 2 - Summary of GPRA Audit Procedures by Type of Audit
ATTACHMENT 3 - GPRA Audit Checklist
ATTACHMENT 4 - Reference Material
-------
-------
MESSAGE FROM THE INSPECTOR GENERAL
I am pleased to present the Office of Inspector General's (OIG) Government
Performance and Results Act (GPRA) Implementation Review Guide. The audit work outlined
in this guide will not only benefit EPA, but the nation as well, by providing: (I) assurances
about the environmental data and systems EPA and states have in place to measure
environmental progress, and (2) recommendations in areas where improvements are needed. In
presenting the Fiscal 1999 Audit Plan, I stated that "the Plan not only continues our journey
toward reaching our vision of being agents of positive change but also assists EPA to evaluate
the accomplishment of its environmental goals and ensure the adequacy of accountability
systems needed to measure performance." The GPRA Implementation Review Guide is an
extension of that commitment.
Our goals "as agents of positive change striving for continuous improvement in our
Agency's management and program operations , " are to: (1) help EPA achieve its
environmental goals by improving the performance and integrity of EPA programs and
operations; safeguarding and protecting the Agency's resources; and, clearly reporting the results
of our work; (2) foster strong working relationships; and (3) operate at the highest performance
level.
By implementing this guide, with an emphasis on results, we are assisting EPA in
reporting on environmental outcomes and measuring progress towards the Agency's mission and
environmental goals. This approach facilitates the OIG's contribution to the efficient and
effective achievement of the Agency's mission and goals.
JU.
Nikki L. Tinsley
Inspector General
-------
-------
INTRODUCTION
PURPOSE OF THE PLAN
The Government Performance and Results Act (GPRA or the Results Act) was designed to
improve the overall management and effectiveness of Federal agencies. The Results Act
requires agencies to shift their emphasis from measuring activities (inputs and outputs) to
measuring results (outcomes) and to focus on achieving desired results in making management
decisions. Implementing the Results Act is expected to improve agencies' planning, budgeting
and accountability processes by linking them together so that planning drives the budgeting
process. Actual program results and the cost of achieving these results can be used to influence
future planning.
GPRA was intended to move the Government to a results oriented management approach. The
purpose of the OIG GPRA Implementation Review Guide is to provide a coordinated approach
for reviewing the Agency's GPRA implementation efforts as a part of our ongoing work.
Congress attaches great importance to effective implementation of GPRA and believes
Inspectors General have an important role to play in informing agency heads and the Congress
on issues concerning efforts to implement the Act. This plan provides general background
information and recommended procedures that can be integrated into performance audits,
including performance audits of contracts and assistance agreements; financial statement audits;
and other assignments, such as consulting services. The GPRA Implementation Review Guide
supplements the OIG's annual audit plan and other issue area plans. The procedures contained
in this plan provide audit teams with the necessary steps for incorporating GPRA concepts and
requirements into their work. This guide focuses primarily on Agency/State systems for
measuring progress towards EPA's goals and is not designed to evaluate the overall
effectiveness of Agency programs.
All audits and assistance assignments are different; therefore, the review procedures contained in
this guide may not cover all circumstances or conditions encountered, and in some cases not all
the procedures will necessarily apply. Auditors are to use their professional judgement to tailor
the procedures to their audit objectives, taking into consideration the time and resources available
to conduct the audit and the benefits to the Agency in completing the steps.
RELATIONSHIP TO OIG'S AND EPA'S MISSIONS
EPA-OIG was established to provide independent and objective information, advice, and
recommendations to Agency managers based on audits, investigations, and other activities. The
OIG mission is defined in the Inspector General Act of 1978 (Public Law 95-452), as amended.
The Act requires the Inspector General to: (1) conduct and supervise audits and investigations
relating to programs and operations in the Agency; (2) provide leadership and make
1
-------
recommendations designed to: (i) promote economy, efficiency, and effectiveness, and (ii)
prevent and detect fraud and abuse in Agency programs and operations: and (3) fully inform the
Agency Head and the Congress about problems and deficiencies identified by the OIG relating to
the administration of Agency programs and operations.
Our vision is:
We are agents of positive change striving for continuous improvement in our Agency's
management and program operations, and in our offices.
Goal 10 of EPA's strategic plan, "Effective Management," states that EPA will seek to establish a
management infrastructure that will set and implement the highest quality standards for effective
internal management and fiscal responsibility within the Agency. Our GPRA Implementation
Review Guide supports this goal, as well as EPA's other nine environmental goals. Effectively
incorporating the Results Act concepts and requirements into the day-to-day management of
Agency programs will facilitate the efficient and effective achievement of the Agency's goals.
BACKGROUND ON THE RESULTS ACT
Seeking to promote improved government performance and greater public confidence in
government through better planning and reporting on the results of federal programs, Congress
enacted the Government Performance and Results Act of 1993. As Congress and the
Administration seek to reduce the cost and improve the performance of the federal government.
there is broad consensus that holding agencies accountable for results is a key to better
management of programs.
Performance-based management, as envisioned by the Results Act. is a dynamic and
complementary process of setting a strategic direction, defining annual goals and measures, and
reporting on performance. Under the Results Act, agencies prepare multi-year strategic plans that
set the general direction for their efforts. Then, beginning with fiscal year 1999, agencies are
required to prepare annual performance plans that establish the connections between the long-
term strategic goals outlined in the strategic plans and the day-to-day activities of managers and
staff. Finally, the Act requires that each agency report annually on the extent to which it is
meeting annual performance goals, and the actions needed to achieve or modify those goals that
have not been met.
Strategic planning is an effort to establish long-term goals and objectives that will shape and
guide activities and programs to fulfill an organization's mission. Performance measures are a
key tool to help managers assess progress towards the goals or objectives stated in their strategic
plans. They are also an important accountability tool to communicate agency progress to
Congress and the public. Program managers should analyze and use performance information to
inform and influence their:
-------
1. program implementation decisions: for example, determining which environmental
programs or activities to undertake;
2. planning decisions: for example, using information on performance trends to set
challenging yet realistic program goals;
3. management decisions: for example, making decisions about aligning management
systems and structures to support the organization's desired outcomes; and,
4. budget and resource allocation decisions: for example, using information about the
connection between a given level of resources and a given level of results for a program to
decide if additional budgetary expenditures will be beneficial.
Most importantly, performance measures build a common results language among all decision-
makers. Selected measures define what is important to an organization, what it holds itself
accountable for, how it defines success, and how it structures its improvement efforts.
Since passage of the Results Act Congress has expressed an interest in having Inspectors General
involved in ensuring the Act was effectively implemented. On March 19,1998, the House of
Representatives passed the "Government Performance and Results Act Technical Amendments"
(HR 2883); however, no further action was taken on the bill. The amendments would have
required OIG verification and validation of selected supporting data sources and information
systems used for performance measures outlined in agency performance reports and strategic
plans. A summary of the findings, results, and recommendations from completed OIG reviews
would have been required to be included in the OIG's semiannual reports. On October 7. 1998.
the leadership of the House of Representatives instructed the Inspectors General Community to
develop and implement a Results Act review plan as envisioned by HR 2883. irrespective of
whether or not the bill becomes law.
In addition, in a letter to IGs on December 17,1998, the Chairman, Senate Committee on
Governmental Affairs, recommended two areas IGs should consider for GPRA work in the
future: (1) determining if key performance measures the Agency has chosen are consistent with
the goals in its strategic and annual performance plans, and (2) verifying the accuracy and
reliability of the underlying performance data.
BACKGROUND ON EPA'S GPRA IMPLEMENTATION EFFORTS
In the spring of 1997, the Agency formally reorganized several of its key management functions
to tighten the links among planning, budgeting, and accountability. The agency created the
Office of the Chief Financial Officer (OCFO). One of the functions of the OCFO is to lead the
Agency's efforts to implement the Results Act. The Act requires the Agency to adopt the tools of
performance-based management: a five-year strategic plan that includes a mission, a set of goals.
-------
measurable objectives to achieve those goals, and performance measures to identify progress
against the objectives. In addition to the strategic plan, the Agency is required to develop annual
performance plans tied to the strategic plan and budget requests. By* March 31, 2000 and at the
same time in all subsequent years, the Agency is required to submit performance reports
comparing performance against goals. The Act also requires Federal agencies to include findings
of program evaluations completed during the fiscal year in their performance reports.
Strategic Plan. The Agency issued its 5 year strategic plan in September 1997, charting the
Agency's course for protecting human health and the environment. The guide was developed
with input from EPA's senior leadership and employees, as well as from environmental
stakeholders. This guide describes the Agency's 10 goals and related objectives and discusses
strategies for achieving them in the years ahead. The plan must be updated by September 30,
2000.
Annual Performance Plan. The Act also requires federal agencies to prepare annual
performance plans covering each program activity set forth in the Agency's budget. The fiscal
1999 President's Budget/Annual Performance Plan was the first budget submission prepared to
meet the GPRA requirements by identifying EPA's activities and requesting resources by goal
and objective. It was submitted to Congress February 1998.
Annual Performance Reports. An accountability workgroup was convened in December 1997
to develop a new accountability process for EPA. The workgroup recommended a two track
approach to implementing GPRA: Track A, beginning in fiscal 1998, was defined as a project to
plan, develop and carry out the Agency's initial accountability process to ensure ftill compliance
with GPRA requirements for the fiscal 1999 annual performance report. Track B, also beginning
in fiscal 1998, was defined as a project to plan, develop, and carry out the Agency's longer-range
accountability process, which will support efforts to manage for environmental results and
produce annual performance reports for fiscal 2000 and beyond.
Program Evaluations. The Chief Financial Officer (CFO) issued guidance on program
evaluations to program offices on September 30, 1998. In defining what a program evaluation is.
the CFO quoted OMB Circular A-l 1, Part 2 - Program Performance Reports (revised July 1998).
The criteria defines a program evaluation as "an assessment, through objective measurement and
systematic analysis, of the manner and extent to which federal programs achieve intended
objectives." OMB also includes process evaluations in its definition of a program evaluation
(i.e., evaluations of the mechanisms by which programs achieve their goals or objectives). The
CFO's guidance states that EPA's National Program Managers (NPMs) are to conduct program
evaluations and to provide input on how these evaluations were used in revising the strategic plan
and annual performance plan; and to provide a schedule for future evaluations. NPMs are also to
provide the findings of their program evaluations for inclusion in the Agency's annual
performance reports.
-------
Data Quality. The Agency's Data Quality Strategic Plan was developed in response to the
Deputy Administrator's request for an Agency-wide approach to ensuring the quality of EPA's
data. The plan recommends several items to improve data quality, including:
1. the development of data quality performance standards for each of EPA's major data
systems to track and improve data quality over time.
2. an error correction process to ensure that discrepancies in EPA data are routed to the
appropriate data managers.
3. the establishment of customer service performance standards for each major data
system to ensure that discrepancies are addressed promptly and appropriately.
In a report accompanying EPA's Appropriations Bill for fiscal 1999, the Senate Committee on
Appropriations expressed concerns about EPA's data quality in reporting on performance
measurement and right-to-know programs. The committee instructed EPA to report quarterly on
its progress in addressing data quality and stewardship concerns. The first quarterly report was
issued on March 22, 1999.
To meet the information challenges of the 21st century, the Administrator announced on October
28,1998 her intention to create a new information management organization. The new office
will offer the advantage of consolidated management to implement the data quality strategic plan
recommendations. The Data Quality Staff and Data Quality Board contemplated in the new
organization offer the opportunity to champion implementation of the plan.
SCOPE AND APPROACH
The OIG is committed to helping Agency managers use Results Act principles in their day-to-day
activities in order to more effectively deliver environmental protection. Our goal is to work with
Agency managers so implementation of the Act achieves the desired results and does not become
a paperwork exercise. Audit teams, by integrating GPRA concepts and requirements into their
work, can help the Agency in making progress towards achieving its strategic goals, objectives,
and mission. The generic review procedures contained in this guide are designed to assist OIG
staff in carrying out this work.
In August 1998, the OIG Office of Audit held a nationwide GPRA strategy meeting to gain an
understanding of the purposes and requirements of GPRA; learn about what EPA is doing to
implement the Act; discuss ways to integrate the Act's requirements into our planned audit and
assistance work; and discuss ways to measure our success in helping the Agency meet its goals.
We anticipate having additional meetings to get updates from Agency managers on the status of
their implementation efforts and areas where we could be of assistance to them. These meetings
will also allow us to share information and better coordinate our work.
-------
A related ongoing OIG effort is the Data Quality Audit Strategy Workgroup formed in October
1998. with the objective of developing a strategy for auditing data quality in EPA's mission-
critical information systems. The workgroup plans to identify major-application systems and,
more specifically, data elements which are being relied on: (1) to assess Agency progress on the
ten designated GPRA goals, or (2) for other crucial reporting purposes. The workgroup expects
to develop and issue the Strategic Audit Plan for Data Quality during the third quarter of fiscal
1999.
Additionally, we are proposing to form partnerships between EPA OIG and state auditors to have
them assess the reliability and validity of the environmental performance data provided by state
agencies to EPA. The partnerships would benefit both the state agencies and the EPA by
providing assurances about the data used to make decisions about the environment.
PRIOR AUDIT COVERAGE
The Financial Audit Division (FAD) conducted a survey of "EPA's Implementation of the
Government Performance and Results Act," (Audit Report No, E1AMF5-20-0013-6100297, -
issued September 25, 1996) in response to the Agency Chief Financial Officer's request that we
review EPA's ongoing and planned efforts to implement GPRA. The survey identified some
areas of concern with the Agency's performance and financial data it planned to use in reporting
on the Agency's progress in achieving it goals and objectives, and the need to develop and
implement program evaluations at EPA.
Numerous audits have identified a longstanding history of problems with EPA's information
systems. Audits of EPA programmatic areas often have a component relating to environmental
data information systems, and we have reported on numerous data systems deficiencies. In
responding to a request from the House of Representatives leadership, we reported environmental
data information systems as one of the ten management challenges confronting the EPA.
REPORTING
The audit procedures contained in this plan were designed to help ensure GPRA coverage for
audit and assistance work conducted in fiscal 1999 (after issuance of this plan) and beyond. The
procedures will help focus our work on environmental outcomes and provide a means of
evaluating and giving the Agency feedback on its implementation of the Results Act. Findings
and recommendations should tie into the appropriate EPA goals, objectives, and/or sub-objectives
for the entity being audited. To the extent possible, findings and recommendations should be tied
into audit objectives dealing with program effectiveness/results rather than being reported in a
separate chapter or an attachment to the report. This reporting approach will support our goal of
helping the Agency use Results Act principles in its ongoing environmental program activities.
-------
AUDITING GPRA
To address the wide variety of audit and assistance services we provide the Agency, we have
categorized our GPRA review work into four functional areas:
1. Program/Performance Audits: During these audits, we determine the extent to which the
desired environmental results or benefits envisioned by the Administration and Congress are
being achieved; review the economy, efficiency and effectiveness of Agency operations; and
determine the extent of compliance with applicable laws and regulations.
2. Assistance and Other Agreement Audits: These are audits of EPA'sinteragency agreements,
Performance Partnership Grants, and other assistance agreements. An Interagency Agreement is
the mechanism used to obtain or provide services to or from another Federal agency.
Performance Partnership Grants and other assistance agreements provide assistance to state, local
and tribal governments, universities and nonprofit recipients. Our objectives are to ensure that
the scope and goals of the assistance agreements are being achieved (which will indicate progress
the Agency is making towards achieving its goals); EPA funds are being properly accounted for
and effectively managed; and, the Agency is receiving what it is paying for.
j. Contract Audits: EPA gets much of its work "protecting human health and the environment"
done and produces many of its major products through contracts. These are audits to assist EPA
in ensuring that it pays a fair and reasonable price for services and products it requested. We also
plan to do some work in the future, looking at contractors' performance in relation to EPA's
environmental mission.
4. Financial Statement Audits: These audits consist of evaluations of the Agency's financial
systems and statements to ensure that the Agency's accounting information is accurate, reliable
• and useful; adequate accounting controls are in place; and EPA complied with applicable laws
and regulations. Our objective in performing these annual audits is to assist EPA in making
improvements in the financial management processes and controls which will provide better
information for decisions promoting the greatest possible environmental results.
For each of the above audits categories, the OIG is committed to helping Agency management
identify how it can improve the use of its resources and authority for the greatest environmental
protection. By incorporating Results Act principles and requirements into our audits, we can
better focus our work on environmental results and assist the Agency in carrying out its mission.
The following sections of this guide deal with how we will do this in each type of audit. The
same or similar steps should also be considered for inclusion in our advisory and assistance work.
-------
1. PROGRAM/PERFORMANCE AUDITS
Performance audits focus primarily on issues based on their relative risk, materiality, and
importance to EPA's mission. These audits evaluate and report on the Agency's
accomplishments of its environmental mission, thereby helping to increase EPA's efficiency and
effectiveness and to identify and prevent fraud, waste, and abuse. The OIG is focusing its
program audit resources in the upcoming years in the following "issue areas": Enforcement and
Compliance Assurance, Performance Partnership Grants, Resource Conservation and Recovery
Act, Water Quality, and Superfund. Issue areas allow us to provide comprehensive audit
coverage which can significantly improve the Agency's effectiveness and credibility.
Performance audits are conducted by OIG divisional audit offices nationwide. The divisional
office conducting the audit will be responsible for reporting the results of the audit, including
work related to the Results Act. Auditors should incorporate the following recommended GPRA
review procedures (that relate to the scope and objectives of the area being audited) into their
audit guide.
Recommended GPRA review procedures: Sections A through C, D, E and I.
NOTE: The GPRA review procedures referenced above (and in the subsequent sections) can be
found at Attachment 1 - Procedures for Assessing EPA's GPRA Implementation.
'2. ASSISTANCE AND OTHER AGREEMENT AUDITS
EPA accomplishes a large part of its environmental mission through assistance agreements
awarded to states, local and tribal governments, universities, and nonprofit recipients. Our GPRA
related audit work in this area will focus on: Performance Partnership Grants (PPGs). other
Assistance Agreements and Interagency Agreements.
a. Performance Partnership Grants - This assistance program gives states and tribes the
option to combine funds from two or more grants, so they have the flexibility to address
their highest environmental priorities while continuing to address core program
commitments. PPGs are part of the National Environmental Performance Partnership
System (NEPPS) which gives states a stronger role in priority setting; focuses scarce
resources on their highest priorities; and tailors the amount and type of EPA oversight to
an individual state's performance. Our goal is to provide EPA program managers with
useful information to set up and monitor the PPG and NEPPS programs, so they achieve
maximum environmental results.
Recommended GPRA review procedures: Sections B, C, F and I.
-------
b. Interagency/Assistance Agreements - Interagency Agreements are services provided to
and from another Federal agency, and Assistance Agreements are the transfer of anything
of value for a public purpose of support or stimulation authorized by law. Our audit
objectives are to determine whether the activities funded by EPA were within the
Agency's assistance authorities as described in the relevant statutes and to assess the
environmental benefits received for the assistance provided.
Recommended GPRA review procedures: Section F.
3. CONTRACT AUDITS
GPRA reviews conducted as a part of performance audits of EPA's contracts are intended to
provide the Agency with an assessment of the contractors' performance in relation to EPA's
environmental mission. The Agency can use this information to determine the environmental
benefits received for the EPA resources expended.
Recommended GPRA review procedures: Sections A and G.
4. FINANCIAL STATEMENT AUDITS
In accordance with the requirements of the CFO Act. as amended by the Government
Management Reform Act, the OIG is required to audit the Agency's annual financial statements.
OMB Bulletin 97-01, "Form and Content of Agency Financial Statements" requires that the
Overview section of the financial statements contain a brief narrative discussion which describes
the reporting entity and its mission, activities, program and financial results, and financial
condition. At a minimum, the Overview should contain sections which address, among other
things, performance goals and results.
OMB Bulletin 98-08, "Audit Requirements for Federal Financial Statements" requires auditors to
obtain an understanding of the components of internal control relating to the existence and
completeness assertions relevant to the performance measures included in the Overview of the
Reporting Entity. Those internal controls that have not been properly designed and placed in
operation should be reported. OMB Bulletin 98-08 does not require that we test whether the
controls are effective. Since the Agency has acknowledged weaknesses in the quality of the
environmental data it has available to manage its environmental programs, in the past we have
performed limited work in this area.
Beginning in fiscal 1999, EPA implemented new cost accounting procedures in order to
accumulate and report total EPA costs expended on each goal. The cost information is reported
-------
in the Statement of Net Costs, one of the principal financial statements. As a part of the audit of
this statement, we will evaluate the systems, processes and controls the Agency has in place to
accumulate and report on the costs of its programs. Specifically, we will audit the cost reported
for each of the Agency's goals. This will allow us to assist the Agency in improving the accuracy
and timeliness of cost information.
Recommended GPRA review procedures for Overview: Section H.
10
-------
PARTNERSHIP WITH STATE AUDITORS
PURPOSE
The purpose of a partnership between EPA OIG and state auditors is to have state auditors assess
the reliability and validity of the environmental performance data provided by the state agencies
to EPA and make recommendations for improvement, if necessary. This work would benefit
both the state agencies and EPA since it would provide assurances about the quality of the
environmental data that is being used in making decisions about the nation's environmental
protection. It would also provide a means of identifying and correcting data quality problems.
BACKGROUND
In broad terms, EPA is a regulatory agency assigned the task of controlling pollution that
threatens human health and the environment. Much of the responsibility for environmental
protection has been delegated to the states. EPA headquarters is responsible for developing
national policies, standards, and guidance; the regional offices implement federal programs or
monitor the states' implementation of them. Much of the data that EPA uses to manage its
programs and measure progress towards its environmental goals comes from the states, and
Congress holds the Agency accountable for state actions in managing environmental programs
delegated to them.
The Results Act requires the Agency to describe the procedures that will be used to verify and
validate performance data in its annual performance plan. The procedures should be credible and
specific to ensure that performance information is sufficiently complete, accurate and consistent
to document performance and support decision-making. Validation is an effort to ensure that
data are free of systematic error or bias and that what is intended to be measured is actually
measured. Verification is a process of checking or testing performance information to assess
other types of errors, such as errors in keying data.
OMB Circular No. A-l 1, Part 2, "Preparation and Submission of Annual Performance Plans
(revised 1998)," section 220.12 - Verification and Validation, states:
x* "Federal agencies are not required to develop an independent capacity for verifying or
validating performance data received from, or based oh, sources outside the agency. Data
sources outside the agency should be identified in the plan. Available information from an
outside source regarding the accuracy and reliability of its data should be collected, but
need not be included in the plan."
11
-------
As EPA gives more discretion and decision-making authority to states, EPA will need to revise
its accountability methods. This shift will change the nature of how-EPA reviews state
performance.
At the heart of EPA's change is the use of accountability measures that will encourage states,
firms and individuals to be more innovative by focusing on results. EPA and state leaders agreed
in August 1997 to use a national set of core performance measures for assessing the nation's
progress in protecting public health and the environment. Although core performance measures
are not perfect, a well-balanced set of indicators gives state program managers. EPA officials and
the public a useful sense of how a state is doing relative to both its own aspirations and the
performance of other states.
BENEFITS
Forming a partnership between EPA OIG and state auditors enables EPA to, "potentially" forgo
review of state repotted performance data in order to ensure the quality of state environmental
data used in carrying out national environmental programs. Poor data quality, as measured by
state auditors, would be subject to additional oversight and EPA control. EPA could negotiate
tighter regional and state agreements and review the use of Federal funds more closely.
PILOT PARTNERSHIP WITH FLORIDA STATE AUDITORS
Florida has a law similar to the Federal GPRA. in that it requires each department to develop a
performance results driven budget and system to measure and report progress against the planned
goals. As pan of this plan, individual departmental Offices of Inspector General, including the
OIG for the Department of Environmental Protection, are responsible for auditing performance
measures and data reported to support the state's performance goals. Implementing a partnership
between EPA OIG and Florida OIG Department of Environmental Protection would provide a
means to validate the accuracy of the performance data received from the state of Florida. We
will use the lessons learned in this partnership to decide how. and if, we will pursue similar
partnerships with other state audit organizations.
A pilot partnership with Florida state auditors is proposed for fiscal year 1999. Representatives
from EPA OIG and Region 4 should meet with Florida's OIG for the Department of
Environmental Protection to consider a partnership to exchange and share audit procedures and
results on the assessment of environmental measures, data and results. Successful
implementation of the partnership would result in the state auditors reporting the results of their
findings and recommendations to the Southern Audit Division. The auditors could then report
the results to the appropriate EPA program officials for needed action.
12
-------
PROCEDURES FOR DEVELOPING PARTNERSHIPS WITH STATE AUDITORS
Following are recommended procedures for developing and implementing a partnership with
state auditors that may be modified at a later date based on our experience with Florida. EPA
auditors should determine if there is an audit organization that has responsibility for reviewing
the performance data the state reports to EPA. If it is determined that such an audit organization
exists:
1. Obtain and review a representative sample of EPA assistance agreements with the
state.
2. Identify the national core performance measures in assistance agreements selected.
3. Meet with the state auditors and determine if they have plans to audit or have some
other group audit the national core performance measurement data outlined in the
EPA/state assistance agreements?
4. If an audit program has been developed, obtain and review the scope and objectives.
Identify those objectives that could provide valuable feedback to EPA program managers.
If no audit program has been developed, discuss with state auditors an audit scope and
objectives that will benefit both EPA and state program managers.
5. Discuss with the state auditors their reporting requirements and ways this information
could be provided to EPA managers.
-------
-------
Attachment 1
Procedures for Assessing EPA's '
GPRA Implementation
PURPOSE
The purpose of the following audit procedures is to provide Office of Inspector General staff with
the necessary guidance (steps) to assess the auditee's implementation of applicable Results Act
requirements and the effect of implementation on the efficiency and effectiveness of the
Agency's or the state's activities in meeting EPA's environmental goals (These steps are to be
incorporated into audit programs as previously outlined in each functional area discussion). The
method and extent of verification for any of these steps should be based on the auditor's
professional judgement.
AUDIT OBJECTIVES AND STEPS
A. Agency Program Reviews - The purpose of performing these audit procedures is to .
provide the auditors) with an understanding of how the audit entity fits into the Agency's
overall mission and its 10 goals (and the supporting performance measures) and to gain an
understanding of the system of internal controls as it relates to the data in support of
EPA's GPRA reporting. In addition, this information will provide the foundation for
auditing GPRA in subsequent years. Audit objectives: (1) How does the audited activity
contribute to the Agency successfully accomplishing it environmental goal? (2) What
systems are in place to track, manage and report on this contribution?
1. Identify and determine how the audit area relates to the Agency's goals,
objectives, and/or sub-objectives.
2. If there is a direct link between the audit area and the Agency's goals, objectives,
and/or sub-objectives, determine the following:
a. what information/measures are being reported in support of the Agency's goals,
objectives, and/or sub-objectives (GPRA reporting requirements); are the measures
consistent with the goals, objectives and sub-objectives; and how is the
performance information being used to plan and manage the day-to-day operations
of the program?
b. are goals established for regional offices? If so, how are they set?
c. what is the source of the information being reported in support of EPA's GPRA
reporting requirements?
-------
B.
Attachment 1
d. identify any automated and/or manual systems used to gather and report
performance data.
e. gain an understanding of the system of internal controls relevant to the
performance information being reported for GPRA.
3. If there is no link between the audited activity and the Agency's goals, objectives,
and/or sub-objectives:
a. where are the costs of the activity being charged?
b. discuss with program managers the environmental benefits of the program's
activities, and
c. with the assistance of program managers, assess the benefits of continuing the
program's activities.
4. How is performance data made available to management?
a. how has this data been used by management for policy-making, and for
management and evaluation of the agency's progress toward meeting its goals?
b. have/will any changes in funding, effort, or priorities result from the reported
performance?
c. has the agency clearly communicated the performance data/results to citizens,
elected officials and other stakeholders?
State Program Reviews - EPA is a regulatory agency that provides guidance and
oversight to the states for implementing environmental protection within their respective
states. The purposes of performing these audit procedures is to ensure the Agency is
obtaining relevant, timely, accurate and reliable performance data for managing its
environmental programs. Audit objective: Are states providing the Agency with data
needed to assess progress towards its environmental goals and objectives?
1. Obtain and review any relevant state/EPA agreements (issued after August 1997)
to determine if the agreements contain the required EPA performance measures.
Determine the following:
a. does the agreement contain performance measures for the audit entity?
-------
Attachment 1
i. if no, why not?
b.'are the required state related "national core measures" included in the
agreement? If not, why? What are the region/state plans to include the measures?
c. are the measures quantifiable, measurable, and verifiable? Do they measure
environmental results? Are they tied to EPA's goals, objectives and sub-
objectives? If not, why?
Determine what data is used to measure performance.
a. is the data accurate?
i. does data support the associated measure?
ii. is data based on statewide and systematic assessments?
iii. if data is not being captured, what are the state's plans for capturing the
data?
iv. is there an existing data base to measure future results?
b. how is data captured? Who captures it? When is it captured? Systematically?
Ad-hoc? Do external sources provide performance data, and is the source
recognized and generally acceptable as a reliable source for this data?
c. does the state have adequate procedures/controls to verify and validate the data
(is the data audited)?
d. what resources are dedicated to collecting, verifying, and validating
performance data? Do they appear insufficient/excessive?
Determine the methods used for reporting performance.
a. how does the state report performance measures to the region?
b. are the reports complete, accurate, and timely? Do they include any qualifiers
on the quality of the data?
i
c. did the state report on all of the required performance measures?
-------
Attachment 1
4. Identify whether the state uses any performance measures other than those
included in the state/EPA agreement.
a. what measures does the state use to measure performance/manage the area?
b. where there are differences between these measures and those identified in
state/EPA agreements, consider whether the state measure is appropriate for
measuring environmental performance.
C. Regional Program Reviews - The purpose of performing these audit procedures is to
ensure the regional program offices are providing appropriate state monitoring for
implementing their respective environmental programs. In addition, these audit
procedures will ensure the regional program offices are collecting timely, relevant and
reliable performance data needed to assess how well their programs are progressing
towards EPA's goals and objectives. Audit objective: Are EPA regional offices
providing adequate oversight and monitoring of states to ensure effective and efficient
implementation of the environmental programs delegated to them?
1. Obtain and review the most current memorandum of agreement between the
regional program office and headquarters for the audit area. Determine the
following:
a. does the agreement contain performance measures for the audit area?
b. are annual performance measures included in the agreement consistent with the
Agency's GPRA reporting requirements?
c. will the measures help assess the region's progress in achieving the relevant
Agency goals, objectives and sub-objectives?
d. are the required state related "national core measures" included in the
agreement? If not, why, and what are the region/headquarters plans to include the
measures?
e. are the measures quantifiable, measurable, and verifiable? Do they measure
environmental results? If not, why?
2. Determine what data is used to measure performance.
a. specifically, what data does the region use to measure its performance in the
audited area and is the measure consistent with the relevant EPA goal, objective
-------
Attachment 1
and sub-objective?
b. does the data support the associated measure?
c. are the regional program offices capturing the data needed to measure their
performance? If not, what are the region's plans for capturing the data?
d. how is the data captured (by whom, when, systematically or manually)? Do
external sources provide performance data, and is the source recognized and
generally acceptable as a reliable source?
e. how do regional program officials ensure the accuracy of the data used to
measure performance?
f. does the region have adequate procedures/controls to verify and validate the
data, or assess the state's process?
g. what resources are dedicated to collecting, verifying, and validating
performance data? Do they appear insufficient/excessive?
Determine the methods used for reporting performance.
a. how does the region report the results of the performance measure to
headquarters?
b. are the reports complete, accurate, and timely? Do they include any qualifiers
on the quality of the data?
c. did the region report on all of the measures in the memorandum of agreement?
Determine if the region uses any measures outside the ones included in the
memorandum of agreement with headquarters.
a. what measures does the region use to measure performance/manage the audited
entity?
b. where there are differences between these measures and those identified in the
memorandum of agreement, consider whether the region's measures are
appropriate for measuring environmental performance?
-------
Attachment 1
D. Data Quality Reviews - Sound information resources and efficient, responsive
information systems are critical to EPA's ability to achieve its environmental mission.
The information that EPA uses to manage its environmental programs must be accurate,
reliable and accessible to authorized users in order for the Agency to meet its
organizational goals. Good data will assist the Agency in making sound regulatory and
program decisions. Data quality reviews focus on Agency systems that support externally
reported performance measures associated with EPA's "vital few" annual performance
goals and measures. Audit objective: Will the Agency's information systems provide
relevant, timely, accurate, complete, and consistent data for reporting on GPRA?
1. Determine if the reported data is a meaningful measure of progress being made on the
relevant annual performance goal.
2. Identify and test key controls designed to ensure the timeliness, accuracy,
completeness and consistency of the relevant data.
3. Test the reliability of the data (i.e. timely, accurate, consistent and complete).
4. If necessary, identify steps that need to be taken to improve the reliability of the data.
E. Accounting for costs under GPRA - In response to GPRA. the Agency prepared the
fiscal 1999 annual plan and budget linking resources with the Agency's goals, objectives,
and sub-objectives. The Agency replaced its program element structure with program
results codes, which specify the goal, objective, sub-objective and National Program
Manager. Accounting will occur at the sub-objective level. Audit objective: Will the
Agency's accounting system accurately capture the costs associated with its
environmental goals, objectives and sub-objectives?
1. Obtain a copy of the relevant sub-objective description(s).
2. Compare the activities described in the sub-objective description(s) with the activities
being performed.
3. Test a sample of the time-sheets to ensure time is being charged against the appropriate
program results code.
4. Discuss with program managers of the audited entity whether they collect and use cost
information to manage their program activities. If no, discuss what cost information
would be beneficial for managing their program activities.
5. Provide the Financial Audit Division with copies of any position papers, memoranda.
-------
Attachment 1
or reports that are issued to regional offices, so they can be shared with Headquarters staff
responsible for the Agency's overall financial operations.
Assistance Reviews - Assistance agreements account for about one half of the Agency's
total budget. Therefore, the ability of assistance agreement recipients to achieve the
agreement objectives and results will affect the Agency's achievement of its goals. The
purpose of performing these audit procedures is to determine the environmental benefits
received for EPA resources expended for assistance agreements. Audit objective: What
are the environmental benefits of assistance agreements? Obtain and review the assistance
agreement and determine:
1. How the assistance agreement relates to the Agency's goals and objectives (what are
we accomplishing or what are the expected outcomes)?
2. Whether the assistance agreement's deliverables are clearly communicated and
quantifiable?
3. Whether the deliverables were received timely and supported EPA's environmental
goals as outlined in the assistance agreement?
4. How program officials used assistance agreement deliverables to further EPA's
mission and goals?
Contract Audits/Reviews - EPA gets much of its work done and produces many of its
major products through contracts. The OIG is committed to assisting EPA in ensuring
that it pays only for the services and products it requests and receives. Audit objective:
What are the environmental benefits of contracts? Obtain and review the contract and
determine:
H.
1. How the contract relates to the Agency's goals and objectives (what are the expected
environmental outcomes)?
2. Whether the contract deliverables are clearly communicated and quantifiable?
3. Whether the deliverables were received timely and supported EPA's environmental
goals as outlined in the contract?
4. How program officials used contract deliverables to further EPA's mission and goals?
Financial Statement Audits - OMB Bulletin 97-01, "Form and Content of Agency
Financial Statements," states that "performance measures presented in the overview and as
-------
I.
Attachment 1
other accompanying information should be consistent with the measures used by the
agency as_part of their Government Performance and Results Act (GPRA) implementation
efforts." The Financial Audit Division will perform these audit procedures to ensure that
the Agency is consistently and accurately reporting its program accomplishments under
GPRA. Audit objectives: Is the performance data reported in the financial statements
overview consistent with the Agency's GPRA reporting, as required by OMB Bulletin
97-01? Are there controls in place to ensure the reliability of the data? Identify and
review the performance goals and results reported for each of the reporting entities and:
1. Determine if all program offices have included performance goals and results in the
overview that are consistent with the Agency's goals and objectives.
2. Identify the source of the performance data reported, including automated and manual
systems, whether from regions, state or tribes, or other reporting institutions.
3. Review prior audit reports which included a review of the performance data reported
in the overview to the financial statements.
4. Annually select two or more reporting entities, time permitting, and obtain an
understanding of the system of internal controls used to monitor and report performance
data. Selection of the reporting entities will be based on the dollars budgeted for the
programs, beginning with the highest dollar programs. This process is to continue until
internal controls have been identified for all major reporting entities.
5. Annually update the auditors understanding of the reporting entity's internal control
system for those that have been reviewed in prior years and follow-up on prior-year
recommendations.
Program Evaluations - GPRA requires the agency to include in its strategic plan a
description of the program evaluations used in establishing or revising general goals and
objectives, with a schedule for future program evaluations. The Agency's strategic plan
states that program evaluations are one approach to evaluating and reporting progress
towards goals and objectives. The strategic plan further states that program evaluations
will analyze relationships between activities being conducted and environmental results.
Audit objective: Are program offices conducting and using program evaluations to assess
program effectiveness?
1, Determine if a program evaluation covering the audit entity has been conducted by the
Agency, OIG, GAO or some other organization. If yes, proceed with a. below, if no,
proceed with 2. below.
-------
Attachment 1
a. what methodology was used, and what was the general scope of the evaluation,
as well as particular issues addressed?
b. identify the results of the program evaluation and determine if and how the
results affected the audit entity's planning and day-to-day operations.
c. do the results warrant revisions to the program's goals, objectives and
performance measures?
2. What are management's plans for future program evaluations?
-------
Attachment 2
GPRA Review Procedures by Type of Audit
Auditing Procedures
A. Agency Program Reviews
B. State Program Reviews
C. Regional Program Reviews
D. Data Quality Reviews
E. Accting for Cost Under GPRA
F. Assistance Reviews
G. Contract Audits/Reviews
H. Financial Statement Audits
I. Program Evaluations
Performance
/
/
/
/
^
/
Assistance
/
/
/
/
IAGs
-
/
Contracts
/
/
Financial
Stmt.
/
-------
Attachment 3
GPRA Audit Checklist -
ftuirtc Refprenrf £frurfur0 fnr Assessment and Rpnnrlintr
Components
Questions
Goals - Are (or do) they:
(Strategic and Performance)
S clear, precise and results oriented?
/expressed in a way to allow for future assessment or measurement?
/ realistic in terms of given resources and timeframes?
/ relate to the mission statement and cover the major functions
and operations of the Agency (or program)?
/ consistent with the statutory requirements and Congressional
priorities?
- Are (or do) they / clear, specific and measurable
(Agency and Program) / relate to either outputs or outcomes?
/ practical and cost beneficial?
/ attainable within a specific period?
/ relate to an Agency performance goal?
/ relate to any internal or external benchmarks or base levels?
Data-
/ Does it exist & is the source identifiable?
/ Is it relevant, reliable, valid, accurate, complete?
/ Is it free from influence and bias in collection and presentation?
/ Are significant limitations identified and accounted for?
/ Is it consistent (if recreated, counted or calculated or would it
give the same results)?
Data Collection
System/Controls
/ How reliable are the systems or methods used to collect data
(External vs. internal, manual vs. automated management,
questionnaires vs. telephone interview)?
/ What plans/procedures (internal management controls) exist to
verify and validate performance data?
S Are adequate resources (technology or personnel)available to
develop systems needed to obtain and manage the data?
-------
Attachment 4
Reference Material
EPA Strategic Plan; EPA/190-R-97-002, September 1997 (http://www.epa. eov/ocfopaeeA
EPA Fiscal 1999 Performance Plan/Agency Budget
Government Performance and Results Act (GPRA), PL 103-62,1993
Chief Financial Officers (CFO) Act, PL 101-576,1990
Government Management and Reform Act (GMRA), PL 103-356, 1994
OIG Issue Area Plans, OIG Intranet
Enforcement and Compliance Assurance, September 1998
Performance Partnership Grants and National Environmental Performance Partnership
System, September 1998
Resource Conservation and Recovery Act Program, May 1998
Superfund Program, July 1997
Water Quality, September 1997
Core Performance Measures for FY 1998, June 1997
OMB Circular No. A-l 1, Part 2, "Preparation and Submission of Annual Performance Plans
(revised 1998)"
Executive Guide - Effectively Implementing The Government Performance And Results Act,
GAO/GGD-96-118, June 1996 & many other GAO reports.
------- |