Office of Inspector General
Audit Report
Additional Efforts Needed
to Improve EPA's Oversight
of Assistance Agreements
Report No. 2002-P-00018
September 30, 2002
-------
Inspector General Human Resource
Center Conducting the Audit:
Regions Covered:
Mid-Atlantic Audit and Evaluation
Resource Center, Philadelphia, PA
Regions 1-10
EPA Offices Covered:
Administration and Resources
Management
Administrator
Air and Radiation
Enforcement and Compliance
Assurance
International Affairs
Prevention, Pesticides and
Toxic Substances
Research and Development
Solid Waste and Emergency Response
Water
Abbreviations
EPA: Environmental Protection Agency
GAD: Grants Administration Division
GAO: General Accounting Office
GMO: Grants Management Office
MER: Management Effectiveness Review
MOR: Management Oversight Review
OARM: Office of Administration and Resources Management
OIG: Office of Inspector General
SRO: Senior Resource Official
Report No. 2002-P-00018
-------
-\cuirjj
' '*>
* UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
1 WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
September 30, 2002
MEMORANDUM
SUBJECT: Audit Report
Additional Efforts Needed to Improve EPA's Oversight
of Assistance Agreements
Report No. 2002-P-00018
FROM: Michael A. Rickey /S/ Michael A. Rickey
Director for Assistance Agreement Audits
TO: Morris X. Winn
Assistant Administrator for Administration and
Resources Management (3101)
Attached is a copy of the final audit report on EPA's Oversight of Assistance
Agreements. This is one in a series of audits that shows the need for improved EPA
management of assistance agreements. This audit report contains issues that
describe problems the Office of Inspector General (OIG) has identified and corrective
actions the OIG recommends. This audit report represents the opinion of the OIG.
Final determinations on matters in this report will be made by EPA managers in
accordance with established EPA audit resolution procedures. Accordingly, the
issues contained in this report do not necessarily represent the final EPA position,
and are not binding upon EPA in any enforcement proceeding brought by EPA or
the Department of Justice.
ACTION REQUIRED
In accordance with EPA Order 2750, you are requested to provide a written response
within 90 days of the date of this report. A corrective action plan for agreed upon
actions, including milestone dates, should be included. We have no objections to the
further release of this report to the public. If you or your staff have any questions,
please contact me at 312-886-3037 or Lisa White, Project Manager, at 215-814-2391.
For your convenience, this report will be available at
http://www.eDa.sov/oisearth/erooni.htm.
-------
-------
Executive Summary
Purpose
The Environmental Protection Agency's (EPA's) 1996 Integrity Act
Report included "Closeout and Oversight of Assistance Agreements" as
a material weakness. Following this designation, EPA developed
corrective actions to address the weakness. In 1999, EPA determined
closeout was no longer an issue and redesignated oversight as an
Agency-level weakness. We conducted this audit to determine the
effectiveness of the corrective actions EPA has taken to improve its
oversight of assistance agreements. Our specific objectives were to
determine whether:
• Grant specialists and project officers effectively provide oversight of
projects according to established guidelines.
• Senior Resource Officials effectively fulfill their responsibilities
concerning oversight of assistance agreements.
Results of Review
Although EPA developed corrective actions to improve oversight
controls for assistance agreements, a number of EPA Office of Inspector
General, Agency, and General Accounting Office reviews determined
that oversight continued to be a weakness. Therefore, to assist the
Agency in implementing effective corrective action, we sought to
determine the root cause of the problem. We concluded the root cause
was that EPA had not sufficiently prioritized oversight as a necessary
and important part of managing assistance agreements.
Specifically, corrective actions taken by EPA did not ensure grant
specialists and project officers effectively monitored assistance
agreements. EPA's corrective actions included the: (1) development of
post-award monitoring policies; (2) establishment of training
requirements for project officers; and (3) performance of management
effectiveness reviews. However, weaknesses continued to exist in each
area.
Also, Senior Resource Officials did not fully meet their responsibilities
as stewards of government resources, including ensuring adequate
controls over assistance agreement funds and compliance with policies.
Report No. 2002-P-00018
-------
Further, these officials, who are generally Deputy Assistant
Administrators in headquarters offices and Assistant Regional
Administrators, did not emphasize the importance of effective post-
award monitoring. These officials must ensure a sufficient level of
personnel, training, and travel funds are available.
Without adequate oversight controls for assistance funds, EPA and the
public may not be receiving anticipated benefits from EPA-funded
projects. As a result, EPA's ability to achieve its environmental mission
and goals through these assistance agreements is limited. Further,
assistance agreement funds may not be safeguarded against misuse.
Recommendations
We recommend that the Assistant Administrator for the Office of
Administration and Resources Management ensure all corrective
actions are effectively implemented, documents required through post-
award management policies are submitted timely and meet policy
requirements, and timely and effective guidance and oversight is
provided to each headquarters program and regional office. We also
made a number of recommendations regarding SRO responsibilities
and improvements to EPA's internal management reviews.
Agency Response and OIG Evaluation
EPA agreed with all of our recommendations and outlined actions it
intends to take to correct the deficiencies noted. EPA indicated it will
prepare a long-term strategic plan for grants management, for which it
intends to have a final plan by January 2003. The plan will address
SRO roles and responsibilities, workload analyses, and SRO resource
commitments to ensure effective oversight. Also, EPA indicated the
2002 post-award monitoring plans have significantly improved over
prior year plans. The proposed actions should be effective if EPA places
sufficient emphasis on ensuring those actions are fully implemented.
Report No. 2002-P-00018
ii
-------
Table of Contents
Executive Summary i
Introduction 1
Purpose 1
Background 1
Roles and Responsibilities 3
Corrective Action Could Be More Effective 5
Overview 5
EPA Action 6
Post-Award Monitoring Policies 7
Training Requirements 13
Management Effectiveness Reviews 14
Recommendations 16
Agency Response and OIG Evaluation 18
Senior Resource Officials Did Not Ensure Compliance with
Policies 21
Recommendations 23
Agency Response and OIG Evaluation 24
Internal Management Reviews Need Improvement 25
Management Oversight Reviews 25
Post-Award Validation Reviews 26
Recommendations 27
Agency Response and OIG Evaluation 28
Appendices 29
A: Scope and Methodology 29
B: Agency Response 31
C: Distribution List 49
Report No. 2002-P-00018
iii
-------
-------
Introduction
Purpose
The Environmental Protection Agency's (EPA's) 1996 Integrity Act
Report included "Closeout and Oversight of Assistance Agreements" as
a material weakness. Following this designation, EPA developed
corrective actions to address the weakness. In 1999, EPA determined
closeout was no longer an issue and redesignated oversight as an
Agency-level weakness. EPA continued to treat oversight as an
Agency-level weakness into fiscal year 2002. We conducted this audit
to determine the effectiveness of the corrective actions EPA has taken
to improve its oversight of assistance agreements. Our specific
objectives were to determine whether:
Grant specialists and project officers
effectively provide oversight of
projects according to established
guidelines.
Senior Resource Officials effectively
fulfill their responsibilities
concerning oversight of assistance
agreements.
Post-Award Monitoring (Oversight):
Occurs after an award is made and includes
the evaluation of grantee:
* performance
4 compliance with terms and conditions
* scope of work
EPA Policy 98-6 - May 14, 1998
Background
When EPA transfers funds for a public purpose, it uses a legal
instrument called an assistance agreement, which may be in the form of
a grant or cooperative agreement. In fiscal year 2001, EPA awarded
$4.5 billion in assistance agreements to state and local governments,
tribes, universities, nonprofit recipients, and other entities. This
amount accounted for more than half of the Agency's $7.8 billion
budget. Assistance agreements are the primary vehicles through which
EPA delivers environmental and human health protection.
In September 1995, the Office of
Inspector General (OIG) issued an
^ ^ '
audit report, EPA'S Controls Over
Assistance Agreements, that disclosed
significant weaknesses regarding
T-IT-I A > • T i r • i
EPA S Oversight Of assistance
... .. , „., _ , n
Findings from EPAs Controls Over
Assistance Agreements audit included:
4 lmproved oversight by grant specialists
and project officers needed
* Insufficient number of site visits.
4 Inadequate nie documentation.
* Incomplete projects - no final reports.
-------
agreements. In June 1996, EPA's Inspector General testified before a
Congressional subcommittee that EPA did not fulfill its obligation to
properly monitor assistance agreements. As a result, EPA's 1996
Integrity Act Report indicated a
material weakness existed regarding
EPA's closeout and oversight of
assistance agreements. The report
included corrective actions to remove
the material weakness such as the:
1) development of post-award
monitoring policies; 2) establishment
of project officer training
requirements; and 3) performance of
management effectiveness reviews.
EPA's fiscal year 1999 Integrity Act Report indicated the material
weakness had been corrected. However, in November 1999, the
Inspector General, in testimony before a Congressional subcommittee,
indicated recent audits disclosed the same oversight weaknesses as
reported in the 1996 testimony. Additionally, over the past few years,
the OIG has listed EPA's Use of Assistance Agreements to Accomplish its
Mission as a key management challenge confronting the Agency.
Material Weakness - A management control
weakness that the Administrator determines is
significant enough to report to the President
and Congress pursuant to the Integrity Act.
Agency-Level Weakness - A management
control weakness that merits the attention of
the Administrator on a periodic basis.
EPA Order 1000.24 - May 13, 1998
To provide assurance that the corrective
actions were successful, EPA created a
new Agency-level weakness in fiscal year
2000 - "Validation of Corrective Actions
to Improve EPA's Oversight of
Assistance Agreements." Subsequently,
EPA performed management oversight
reviews and post-award validation
reviews. Up on completion of these
reviews, EPA determined further
improvement was needed in the
oversight of assistance agreements and
decided to carry over the weakness into
fiscal year 2002. The corrective actions
planned for fiscal year 2002 consisted of
six new validation reviews and follow-up
on the 2001 validation reviews.
Types of EPA Internal Reviews
Management Oversight Reviews - Part
of the quality assurance efforts of the
Grants Administration Division's annual
regional oversight program; these reviews
evaluate assistance agreement management
for the grants management offices.
Generally, each region is reviewed by the
Grants Administration Division every third
year.
Post-Award Validation Reviews -
Conducted for the first time in 2001 with
the purpose of validating the corrective
actions developed to fix oversight
weaknesses; these reviews focus on post-
award monitoring activities.
Management Effectiveness Reviews -
A self-assessment performed by each
headquarters and regional program office of
their management of assistance agreements.
Report No. 2002-P-00018
-------
Roles and Responsibilities
EPA officials are responsible for providing oversight throughout the
various phases of the assistance agreement process:
4 Pre-award - reviewing the application paperwork and making an
award decision.
4 Award - preparing the assistance agreement documents and
instructing the recipient on technical requirements.
4 Post-award - providing technical assistance and oversight.
4 Closeout - ensuring project completion and initiating closeout
documents.
The Office of Administration and Resources Management (OARM) is
responsible for enhancing public health and environmental protection
through effective management of EPA's human, financial, and physical
resources. Within OARM, the Office of Grants and Debarment is
responsible for ensuring EPA's financial resources are protected
against waste, fraud, abuse, mismanagement, and poor performance.
The Office's Director is the National Program Manager for assistance
agreement management. Further, within the Office of Grants and
Debarment, the Grants Administration Division (GAD) is responsible
for issuing all assistance program policies and assuring quality
assistance agreement information.
Various offices and officials at EPA headquarters and regional levels
are involved in managing assistance agreements:
Grants Management Offices - Located in each of EPA's regions,
personnel at these offices are responsible for the review, negotiation,
award, and administration of assistance agreements, including audit
resolution and final closeout. GAD has this responsibility for all
headquarters-administered assistance.
Senior Resource Officials (SROs) - These officials, charged with
strengthening Agency-wide fiscal resource management, are located in
each headquarters program and regional office. They are responsible
for ensuring compliance with laws and regulations, while furthering
program mission. SROs are typically Deputy Assistant Administrators
in headquarters offices and Assistant Regional Administrators.
Report No. 2002-P-00018
-------
Grant Specialists - These specialists work within grants
management offices, including GAD, and are primarily responsible for
administration of assistance agreements from application to closeout.
They are also responsible for developing and maintaining the official
grant file, which should consist of programmatic and fiscal information
on the purpose, performance, and history of an assistance agreement.
Project Officers — These officers work within regional and
headquarters program offices, and are responsible for the review,
negotiation, award, and administration of assistance agreements as
they relate to the programmatic and technical requirements. They are
also responsible for developing and maintaining the official technical
project file.
On-going communication between the project officers, grant specialists,
and recipients regarding all aspects of the assistance agreement is
essential.
Report No. 2002-P-00018
-------
Corrective Action Could Be More Effective
Overview
Although OAEM developed corrective actions to improve oversight
controls for assistance agreements, oversight continued to be a
weakness throughout the Agency. Because of these continued
weaknesses, the corrective actions became a major focus of our audit.
We concluded the root cause was EPA had not sufficiently prioritized
oversight as a necessary and important part of managing assistance
agreements. Specifically: (1) corrective actions were often ineffective
and their implementation not enforced, and (2) SROs did not ensure
compliance with policies. In addition, some of EPA's guidance needs
improvement. Without adequate oversight controls for assistance
funds:
4 EPA and the public may not be receiving anticipated benefits from
EPA-funded projects. As a result, EPA's ability to achieve its
environmental mission and goals through these assistance
agreements is limited.
4 Assistance agreement funds may not be safeguarded against
misuse.
The corrective actions taken since 1996 by EPA did not ensure grant
specialists and project officers effectively provided oversight of
assistance agreements. Several reports issued by the EPA OIG and the
General Accounting Office (GAO) indicated oversight continued to be a
weakness. Further, internal reviews conducted by EPA also noted
oversight weaknesses. Details follow:
Year Review Type: Weaknesses identified included:
Internal Management • Files lacked documentation of post-award monitoring.
1997 Effectiveness Reviews • Minimal on-site evaluations performed.
Training needed by staff on monitoring assistance agreements.
Internal Management • Files lacked documentation of post-award monitoring.
1998 Oversight Reviews • Minimal on-site evaluations performed.
Training needed by staff on monitoring assistance agreements.
Report No. 2002-P-00018
-------
Year
1999
2000
2001
2002
Review Type:
Internal Management
Oversight Reviews
and
Internal Management
Effectiveness Reviews
Internal Management
Oversight Reviews
OIG Audit on Tribal
Assistance Program
GAO Audit on EPA's
Oversight of Nonprofit
Grantees
Internal Management
Oversight Reviews
and
Internal Post-Award
Validation Reviews
OIG Audit of Surveys,
Studies, Investigations,
and Special Purpose
Grants
OIG Audit of
Procurements Made by
Assistance Agreement
Recipients
Weaknesses identified included:
Files lacked documentation of post-award monitoring.
Minimal on-site evaluations performed.
Training needed on conducting on-site evaluations and tracking
expenditures.
Files lacked documentation of post-award monitoring.
Minimal on-site evaluations performed.
Training needed on conducting on-site evaluations.
The Region did not hold recipients accountable for grant requirements.
Inadequate post-award monitoring of expenditures and progress reports.
Minimal on-site evaluations performed.
Training needed on how to conduct on-site evaluations and determine
unallowable costs.
Files lacked documentation of post-award monitoring.
Minimal on-site evaluations performed.
Training needed on conducting on-site evaluations and tracking
expenditures.
EPA did not measure whether some awards achieved results and
contributed to the Agency's overall mission.
Minimal communication between project officers and the recipients.
Files lacked documentation of post-award monitoring of recipient
procurements.
We recognize that OAEM has taken action to correct the weakness in
oversight of assistance agreements through the development and
deployment of corrective actions. We found these corrective actions
could be more effective. Also, while several EPA Senior Resource
Officials indicated assistance agreement oversight is important for the
Agency to accomplish its mission, these officials did not ensure policies
established to improve oversight were effectively implemented.
EPA Action
In its 1999 Integrity Act Report, EPA removed the material weakness
based on corrective actions it had taken. However, EPA's
determination was based solely on the development of the corrective
actions, not its success or effectiveness. The corrective actions included:
4 Development of post-award monitoring policies.
Report No. 2002-P-00018
6
-------
Establishment of training requirements for project officers.
Performance of management effectiveness reviews.
Post-Award Monitoring Policies
OARM developed post-award
monitoring policies (see box) as a
corrective action. However, we noted
instances of noncompliance with
these policies, as well as evidence
that the policies were not adequately
enforced. Examples included:
4 Inconsistent performance of
baseline/post-award monitoring
responsibilities.
4 Inadequate preparation of post-
award monitoring plans.
4 Incomplete submission of
quarterly compliance assistance
reports.
4 Insufficient performance of on-site evaluations.
4 Inconsistent usage of the grantee compliance activities database.
Baseline Monitoring
Post-Award Monitoring Policies require
grants management and program offices to
perform various oversight activities.
Policy 98-6 — Policy, Procedures and
Guidelines for the Post-Award Management of
Grants and Cooperative Agreements by Grants
Management Offices — May 14, 1998
Policy 5700.3 — EPA Policy for Post-Award
Management of Grants and Cooperative
Agreements by Headquarters and Regional
Offices —April 5,1999
Policy 5700.4 — Interim Grantee Compliance
Assistance Initiative Policy — June 7, 2001
Proposed Policy — Policy on Compliance
Review and Monitoring— currently in draft
and consolidates the previous three policies
Under Policy 98-6, grant specialists
are required to perform baseline
monitoring (i.e., post-award
monitoring) for every assistance
award. Baseline monitoring is the
minimum routine oversight of a
recipient's performance and
compliance with the requirements
of assistance agreements. Without
sufficient baseline monitoring, EPA
has little assurance assistance
Document files \/ *\
f^/
Monitor compliance witr
. terms and conditions
V
/
Baseline
Monitoring
Requirements for
Grant Specialists
include:
r \
Monitor payments
and draw-downs
/
Communicate wtih proje
v officer and recipient
N
X N
Report No. 2002-P-00018
-------
agreement recipients are following regulations and are appropriately
using EPA funds.
Many grant specialists did not perform baseline monitoring when
administering their awards. For example, according to the 2001 post-
award validation reviews, some grants management officials said they
believed baseline monitoring adds little value and should only be
conducted once during the life of the award. One official said baseline
monitoring need not be conducted at all. Some grants management
officials reported to GAD that they only intend to conduct baseline
monitoring for a specified number of assistance awards. All of these
beliefs are contrary to the policy requiring monitoring for all assistance
awards.
GAD is responsible for enforcing compliance with policies, but we found
no evidence of any action taken when baseline monitoring was not
performed. Further, SROs are responsible for ensuring implementation
of post-award monitoring policies, however, we found a lack of
compliance with baseline monitoring requirements. If grants
management officials believe baseline monitoring adds little value,
GAD and the SROs must take action to address these misconceptions.
The 1995 OIG audit report disclosed EPA grant specialists were not
performing the minimum requirements for monitoring their assistance
agreements. Seven years later they are still being cited for not
performing this monitoring and debating whether baseline monitoring
adds any value.
EPA has been working on a new policy to consolidate existing post-
award monitoring policies. In the proposed monitoring policy, baseline
monitoring is defined as the minimum basic monitoring that should
take place on every award on an ongoing basis throughout the lifetime
of the award. We agree baseline monitoring must be ongoing, and at
any given time during the project, evidence of monitoring should exist
in the file.
Post-A ward Monitoring Plans ,,n
•* Post-award monitoring is a crucial component in
ensuring the success of the Agency's many assistance
Policies 98-6 and 5700.3 require agreement programs....the Region can only be
successful in its mission to protect the environment if
each headquarters program and its award recipients are meeting the terms and
regional office SRO to submit a conditions of their assistance a9reements
Post-Award Monitoring Plan Region 10's 2000 Post Award Monitoring Plan
listing what, how, and when
Report No. 2002-P-00018
8
-------
post-award monitoring activities will be performed by grant specialists
and project officers. These Plans, if prepared properly, could assist
EPA's programs and regions with improving the effectiveness of post-
award monitoring activities. However, when we reviewed seven Plans
submitted in January 2000, we noted deficiencies such as the Plans
being:
Untimely Not Finalized Insufficient Not Inclusive
For example, several of the Plans did not include the specific office's
planned course of action regarding post-award monitoring, but rather
restated policy requirements. While the Plans indicated project officers
and grant specialists would perform on-site evaluations, the Plans did
not indicate which recipients or how many would be evaluated.
Also, GAD did not provide written feedback to the SROs about the
content of their Plans. When we spoke to several SROs, they generally
recalled receiving minimal feedback on their Plans. It was not until the
performance of the management oversight reviews and post-award
validation reviews, occurring more than a year after the Plans were
due, that GAD provided some written feedback. The remaining offices
or regions who were not reviewed, but submitted Plans, did not receive
written feedback.
Further, the SROs are required to approve their specific regional or
program Plan. However, due to the quality of the Plans, it is unlikely
that all SROs were ensuring adequate preparation of these Plans. In
effect, the SROs were not prioritizing the importance and need for
effective post-award monitoring of assistance agreements.
After we discussed our concerns about the Plans with EPA officials,
GAD provided the Agency with draft guidance on how to prepare future
Plans. GAD told us the guidance has been widely accepted by the
headquarters program and regional offices.
Report No. 2002-P-00018
9
-------
Quarterly Reports on Post-Award Activities
EPA's post-award monitoring Policy 5700.4 requires each headquarters
program and regional office (includes grants management and program
offices) to submit quarterly Grantee
Compliance Assistance Initiative / Assistance
reports to GAD indicating the number
and type of assistance activities each
pp. p i /-v i pp. I • pre-award workshops/training
office performed. Once each office / . on-Site reviews
submits its quarterly report. GAD ( * technical assistance site visits
J \ • desk reviews
Consolidates them and forwards the \ • management assistance forums
grants management offices' totals to
Congress, which has requested this
information because of EPA's
continuing grants management weaknesses. Regional offices reported
incomplete and inaccurate information to GAD, which limits the
usefulness of the reports as tools to monitor the level of assistance
agreement oversight. Subsequently, EPA reported the same inaccurate
assistance agreement activity information to Congress.
Each SRO designates an official to report all assistance activities
performed by the office. However, at the regional level, we found the
designee was usually from the grants management office and generally
did not incorporate the activities performed by project officers.
Although the policy requires all activities be reported, one region
indicated it was not the grants management office's responsibility to
track activities performed by project officers.
Specifically, of the 10 regions, only 4 complied with the policy by
submitting separate totals for grants management office and program
office activities. Four regions only reported the grants management
office activities. The remaining two regions combined the number of
grants management and program office activities into one report, but
did not distinguish which activities were reported by which office.
Reporting accurate information on the number of assistance activities
is important because GAD can monitor whether adequate oversight is
being performed by the Agency nationwide and report accurate
information to Congress.
Report No. 2002-P-00018
10
-------
On-Site Evaluations
On-site evaluations are comprehensive reviews of recipients'
management and financial systems. These reviews ensure recipients'
systems are capable of documenting and accounting for funds under
EPA assistance awards. Despite the importance of these reviews, EPA
continued to be cited for conducting a minimal number of on-site
evaluations:
4 The OIG 1995 oversight report indicated EPA needed to improve
the number of on-site evaluations performed of assistance recipients.
4 In 1999, a Congressional subcommittee questioned EPA on why it
had only conducted 84 on-site evaluations during fiscal year 1999
when there were approximately 8,800 award transactions.
4 In 2001, GAO reported EPA seldom conducted on-site evaluations of
non-profit assistance agreement recipients.
4 Also in 2001, GAD reported through its post-award validation
reviews that a number of offices were unlikely to meet the minimum
requirement of 5 to 10 percent for conducting assistance activities,
including on-site evaluations.
In 2001, EPA reported there were 466 on-site evaluations performed.
However, 265 of those evaluations, or 57 percent, were performed by 2 of
the 19 headquarters program and regional offices - GAD and the Office
of Research and Development. The remaining 17 headquarters
program and regional offices performed the rest. Two hundred on-site
evaluations by the vast majority of the Agency does not represent
significant improvement. Agency leaders must prioritize the need to
conduct on-site evaluations as part of effective post-award monitoring.
By doing so, EPA will be providing significantly more assurance that
assistance agreements are adequately monitored.
On-site evaluations are necessary to learn if work is on schedule and is
of adequate quality. Moreover, good business practice would dictate
performing on-site evaluations. They provide the EPA presence. It is
not reasonable to give recipients hundreds of thousands of dollars,
sometimes even millions, to complete projects and not visit them to
review progress. Not performing on-site evaluations weakens EPA's
Report No. 2002-P-00018
II
-------
ability to monitor progress, to ensure recipients comply with the terms
and conditions of their agreements, and to safeguard Federal assets.
Not getting out there sends the message to the
grantees that they are in control. Increasing the
percentage of evaluative on-site visits sends the
message to the grantees that they are going to be
physically evaluated and follow up will be done.
Subcommittee on Oversight, Investigations, and
Emergency Management — November 4, 1999
The limited number of on-site
evaluations may continue to be a
weakness unless EPA reevaluates
the proposed post-award
monitoring policy. This policy
currently requires both GMOs and
program offices to conduct at least
one on-site evaluation each year.
We believe EPA should use risk
analysis techniques when selecting recipients for on-site evaluations.
Grantee Compliance Activities Database
Under Policy 5700.4, the Grantee Compliance Activities Database was
developed for the grants management offices to track all their
assistance initiatives. The grants management office personnel are
required to list recipients selected for on-site evaluations, as well as
planned and actual evaluations performed. However, they are not
consistently reporting information in the database. As shown, the
number of post-award activities included in the quarterly reports
significantly exceeded the totals contained in the database.
2001 Assistance Activities
On-site Evaluations
On-site Technical Evaluations
Desk Reviews
Total Activities
Reported
Quarterly Activities
126
53
93
272
Activities in
Database
70
4
44
118
We believe these discrepancies resulted because GAD, as well as the
SROs, had not taken sufficient action to ensure the grants management
offices consistently reported assistance activities in the database. As a
result, minimal benefits are received from the database.
Although not currently required, the proposed post-award monitoring
policy will require program offices to enter their assistance activity
Report No. 2002-P-00018
12
-------
information into the database. The policy also states GAD will provide
necessary direction for using the database. If EPA officials ensure all
assistance activities are entered in the database, we believe EPA will
have a good tool for obtaining information about assistance recipients,
and comparing the level of activities throughout the Agency.
Training Requirements
Project Officers
EPA's 1999 Integrity Act Report indicated, as a corrective action, that
each project officer must take Basic Project Officer Training, as well as a
refresher course. However, SROs and GAD did not ensure adequate
controls were in place to determine project officer compliance with the
training requirements. Additionally, GAD did not have an appropriate
system in place to monitor the training of project officers, and did not
enforce training policies.
Grant Policy Issuance 01-1, Continued Training Requirements for EPA
Project Officers, states an individual must complete the basic course
"Managing Your Financial Assistance Agreement - Project Officer
Responsibilities" prior to becoming a project officer. It also requires
taking a refresher course. It is the SRO's responsibility to ensure
project officers receive the basic course prior to managing an assistance
agreement, and are re-certified every 3 years.
However, during a 2001 management
oversight review, GAD found that a «|t is important that you have a mechanism for
region did not track the dates and ensurin9that educati°n and trainin9are °ne °f
_ __ . . the stones in your foundation for improving
names Ol Stall WhO received project organizational performance
officer training, nor report this
' f, Baldrige A ward Winning Quality, 11th
information tO GAD. AISO, Several Edition -Copyright ® 2001 Mark Graham
SROs disclosed they did not know who Brown' Prod"ctivity Pre"
received project officer training. Not
having an adequate system in place
impedes the ability of SROs to ensure designated project officers are
certified prior to managing an assistance agreement.
GAD is to serve as the Agency's focal point for this training and should
receive the dates of training sessions and names of participants.
Report No. 2002-P-00018
13
-------
However, we found GAD's Agency-wide database on training for project
officers was inaccurate. For example, we compared one regional project
officer list to GAD's national database and found conflicting totals of
certified project officers (see illustration). Duplicate entries and names
of former EPA employees were /\ ^^
found in GAD's database. We also
found the functionality of this /0nTer.RiX°"
-, -, •!• • ii • i \Database=170 y/O^sam e Region = 238
database is limited because it does s^ect officers <^ project vm^s .,
not indicate staff who still need \ I ' i /
project officer training, as well as ^
which project officers are actively
managing assistance agreements.
Further, while project officers can take the refresher course on-line, the
course does not have adequate controls to prevent someone from
skipping through the training and still receiving the certification. For
example, an OIG auditor accessed the on-line course and within
60 seconds was able to get an inappropriate certification that she
completed the course. This was accomplished without ever reading the
on-line course material.
Grant Specialists
Although not part of the corrective action, training for grant specialists
is crucial for successful oversight of assistance agreements. EPA has
not specified training requirements for grant specialists, although we
found that most grant specialists do take the basic project officer
training course. Their proficiency in performing the business aspects of
assistance administration is as crucial as project officer functions. A
workgroup has been formed to develop a core curriculum, but this has
yet to be finalized. The Agency should finalize a core curriculum for
grant specialists and ensure required training is provided and taken.
Management Effectiveness Reviews
As part of the corrective action, OARM required each headquarters
program and regional office to conduct a management effectiveness
review (MER), which is a self-assessment of their management of
assistance agreement programs. However, EPA did not evaluate the
results of these MERs prior to removing the oversight material
weakness from the 1999 Integrity Act Report. The 1999 MERs
disclosed weaknesses that would have justified keeping the material
Report No. 2002-P-00018
U
-------
weakness intact. A vast majority of EPA's headquarters program and
regional offices reported continued weaknesses in how they manage
their assistance agreements. Weaknesses included:
Specific Post-Award Monitoring Weaknesses Identified in 1999 MERs
Insufficient training on conducting J Minimal communication between
on-site evaluations and tracking project officers, grant specialists, and
expenditures with project progress. recipients.
Limited on-site evaluations being / Minimal to no evidence in files of
performed. post-award monitoring activities.
In addition to the post-award monitoring weaknesses identified by the
Agency in the MERs, we found weaknesses in the oversight and
performance of these reviews. Collectively, these weaknesses support
our belief that oversight of assistance agreements is not adequately
prioritized throughout EPA. Details on weaknesses follow.
OARM's Guidance and Feedback
OARM provided limited guidance and feedback on the performance of
MERs. OARM issued one correspondence regarding MER
requirements, which provided each headquarters program and
regional office with a list of five topics
for their MERs. However, this
guidance only required one of the five Topic Choices for 1999 MERS
topics be reviewed and did not require n A
, • • i •£ i Pre-Award
they Select a tOplC identified aS a 2 Statutory and Delegation Authority
material weakness. OARM should 3 SRO Review Level
4. Post-Award
require a consistent MER focus that 5 cioseout
ensures systemic weaknesses are
._. 111 i mi -a/TT-IT-. Memorandum from Assistant
identified and addressed. The MER Administrator of OARM - May 20,1999
results can then be used as adequate
support for any decisions regarding
EPA's material weaknesses for assistance agreements. For example,
only 10 of the 18 MERs we reviewed looked at post-award monitoring,
even though OARM knew this was a material weakness (and 9 of the
10 MERs still reported weaknesses in the area). Therefore, OARM
should have ensured all MERs evaluated post-award monitoring.
OARM also has a responsibility to provide timely and appropriate
feedback on the MERs, which should include an emphasis on the
importance of correcting any reported weaknesses. OARM informed us
Report No. 2002-P-00018
15
-------
they issued no written feedback for the 1999 MERs, but did meet with
program office and regional officials to discuss the MER results.
However, we learned from several of these officials that the feedback
was limited and often untimely.
SRO Role in the Performance of the 1999 MERs
We believe SROs are responsible for ensuring MERs are of sufficient
quality. This includes taking steps to ensure the MERs are
appropriately planned and corrective action is taken. The planning
should consist of identifying the appropriate focus or scope of the MER,
which should address potentially
weak areas, not the strengths, of ,,CD_ ^ . „. „ „
7 CT ' SROs are required to monitor, periodically, the
the program Or region. For the effectiveness of management control practices to
1999 MERS this foCUS Should have safe9uard Programs from mismanagement and
!»»» IVlJ^XVb, Llllb lOLUb bllOUlU IldVe promote the achievement of environmental results."
included post-award monitoring of
. , , i SRO Responsibility Statement - May 31,1994
assistance agreements; however,
only half of the MERs focused on
this topic.
Of the MERs that reported weaknesses in post-award monitoring, most
did not clearly identify the causes for the specific weaknesses, or how
the proposed corrective action would remedy the identified weaknesses
and improve oversight of assistance agreements. Instead, the
corrective action consisted of emphasizing the post-award management
responsibilities. For example, some reports identified weaknesses that
were also identified in the respective 1997 MERs, without addressing
the cause for why these findings were uncorrected or allowed to
continue for more than 2 years. Without addressing the causes for the
weaknesses, it is unlikely the corrective action will be successful. By
providing the appropriate level of priority to the quality of the MERs,
the SROs will be providing a higher level of priority to making sure
grant specialists and project officers effectively monitor assistance
agreements.
Recommendations
We recommend that the Assistant Administrator for the Office of
Administration and Resources Management ensure:
Report No. 2002-P-00018
16
-------
1. All corrective actions cited in the Integrity Act Report are effectively
implemented and the importance and benefits of such actions are
clearly stated.
2. Documents required by post-award management policies are
submitted timely and meet policy requirements.
3. Timely and effective guidance and oversight is provided to each
headquarters program and regional office. Specifically:
a. The proposed post-award monitoring policy should require
baseline monitoring be ongoing for all awards.
b. The proposed policy should increase the minimum requirement
for the number of on-site evaluations conducted by grants
management and program offices.
c. An appropriate and reasonable time frame should be
established for providing feedback to the Agency on submitted
documents and reports.
d. Controls should be established to ensure information is
consistently and accurately reported by all headquarters
program and regional offices in the Grantee Compliance
Activities Database.
e. Controls should be established to ensure project officers receive
the required training prior to managing an assistance
agreement. These controls must include documenting training
information.
f. Controls should be established to ensure the on-line project
officer refresher course prevents the user from obtaining an
unearned certification.
g. A core curriculum for grant specialists should be finalized and
all required training should be provided and taken.
h. The management effectiveness review guidance should require
a consistent focus to identify and address systemic weaknesses.
i. The management effectiveness reviews should be appropriately
planned. Once these reviews are performed, adequate corrective
action should be taken to address identified weaknesses.
Report No. 2002-P-00018
11
-------
Agency Response and OIG Evaluation
OAEM agreed with all our recommendations and outlined action it
intends to take to correct the deficiencies noted. OAEM indicated that
our draft report did not always describe the good things EPA was doing
in grants management. We agree that positive actions were being
taken, but generally those actions were made as a result of our
continual communication with EPA officials during our review. For
example, OARM required baseline monitoring to be ongoing for every
award after we noted the need for such monitoring.
Regarding the specific recommendations, OARM agreed with
Recommendation 1 and stated that the analysis for the FY 2002
Integrity Act Report will include an evaluation of EPA's progress in
improving post-award monitoring.
OARM also agreed with Recommendation 2 advising them to ensure
documents required by post-award management policies are submitted
timely and meet policy requirements. OARM stated they are pleased
with the quality of the final 2002 post-award monitoring plans and will
continue to work to address timeliness issues. However, we believe the
true success of post-award monitoring plans should be measured by how
effectively they are implemented. OARM should provide details on how
they intend to address timeliness issues.
Regarding Recommendation 3a and 3c-i, OARM agreed by indicating
they will or already have taken actions to:
4 Require baseline monitoring be performed for all awards.
4 Establish a 60-day feedback standard.
4 Require all Regional and Headquarters program offices and GMOs
to report post-award activities in the Grantee Compliance Assistance
Database.
4 Develop an enhanced training database that will eliminate
duplicate names or the names of individuals no longer with the
Agency and provide notification of the need for the refresher
training.
Report No. 2002-P-00018
18
-------
4 Establish controls to the on-line refresher course that should
prevent unearned certifications.
4 Develop a core curriculum and pilot training for grant specialists
and fully deploy the training to the Regional GMOs in June 2003.
4 Develop guidance for the next set of management effectiveness
reviews to assure programs identify any weaknesses in grants
management and correct them.
Regarding Recommendation 3b, OARM responded that the proposed
post-award monitoring policy will increase the level of advanced
monitoring by GMOs and the program offices from a minimum of
5-10 percent of active recipients to a minimum of 10 percent of active
recipients annually. The response further stated this will include, for
both GMOs and program offices, a requirement for conducting at least
one on-site evaluation each year. In addition, each GMO will be
required to conduct a minimum number of desk reviews equal to the
number of its grant specialists. As stated in our comments, the
proposed post-award monitoring policy does not indicate how minimum
levels of advanced monitoring were developed, and may not be
sufficient to reduce the risk of failed projects and misspent funds.
Therefore, we cannot determine whether the proposed level of advanced
monitoring is sufficient to improve oversight. We believe EPA should
use risk analysis techniques when selecting recipients for on-site
evaluations and desk reviews.
OARM stated that our report suggests on-site evaluations should occur
for virtually every major EPA grant. OARM further indicated they
have reservations about conducting on-site evaluations when other less
costly forms of monitoring, such as desk reviews, may suffice. The
report does not suggest OARM perform on-site evaluations on every
major grant. We believe EPA should use risk analysis to establish
criteria to determine when to use on-site evaluations or desk reviews.
Desk reviews do not provide the same level of review as on-site
evaluations.
Report No. 2002-P-00018
19
-------
20
-------
Senior Resource Officials Did Not Ensure
Compliance with Policies
SROs did not fulfill their responsibilities as stewards of government
resources to ensure adequate controls
over assistance funds and compliance
With policies. Further, SROS did not "... results-based planning and budgeting, fiscal
emphasize the importance of effective a-°untabilityand <*«*"' stewardship of our
-1 -1 resources provide the foundation for
pOSt-award monitoring tO their everything EPA does to advance the protection
,•11 , i • i of human health and the environment."
respective headquarters and regional
Offices. In effect, SROS allowed EPA's Strategic Plan - September 2000
noncompliance with post-award
monitoring requirements, including:
4 Inconclusive and vague post-award monitoring plans.
4 Nonspecific or incomplete quarterly reports of grantee compliance
assistance initiatives taken by program and grants management
offices.
4 Inconsistent use of the grantee compliance activities database.
4 Insufficient monitoring of project officer training.
Some SROs stated that the level of post-award monitoring is affected by
the limited availability of resources, including: (1) funding for a
sufficient number of people to monitor awards; (2) travel costs for on-site
evaluations; and (3) training for project officers and grant specialists.
SROs must ensure a sufficient level of personnel, training, and travel
funds are available to safeguard EPA's assistance funds. Resources
must be available for the accomplishment of administrative activities
because the success of programmatic activities is often dependent on
adequate administration. Without the appropriate resource mix, it is
likely that assistance programs will not yield the intended results and
ultimately complement EPA's environmental mission.
Several SROs said another factor affecting the level of oversight is the
minimal control they have regarding regional program office oversight
Report No. 2002-P-00018
21
-------
activities. Regional SROs
said they have little control
over the priorities established
by the regional programs,
including the level of
assistance agreement
oversight these offices
administer. Headquarters
SROs also claim limited
control over the regional
program offices' priorities for
oversight. This leaves the
Agency with no one
specifically assuming
responsibility for monitoring the regional program offices' priorities as
they relate to oversight of assistance agreements. We believe the
regional SROs should be responsible for managing the assistance
agreement resources for both the program offices and grants
management office in their region.
Regional SROs are responsible for ensuring local resources are used
appropriately and assistance funds are adequately monitored. Since a
majority of the Agency's budget is used for assistance agreements, an
appropriate level of resources should also be dedicated for effective
oversight controls. These resources may be in the form of additional
employees available for oversight, increased training, and additional
travel funds. Even if additional resources are not available, SROs
should work within the resource constraints that exist. For example,
one region, recognizing its resource limitations, decided to analyze the
specific functions of the grants management office and develop
"operation efficiencies." As a result of these efficiencies, the region
redesignated or revised many grant specialist functions to improve
their efficiency.
SRO responsibilities include working with managers to ensure
sufficient staff are devoted to resources management. One way this
could be accomplished, as recommended by the SRO Responsibility
Statement, is by periodically performing workload analyses to
determine baseline workload expectations and where additional
resources may be needed. Of the seven SROs we interviewed, all
indicated they had not established workload limits for their grant
specialists and/or project officers.
Report No. 2002-P-00018
22
-------
Average Agreements Per Grant Specialist
R7
R4
R6
R8
R9 GAD
R10
Regional GMOs and GAD
We contacted personnel from
GAD and each regional
grants management office
(GMO) to determine the
average number of
assistance agreements per |
grant specialist; all but one I
region provided data. As §
ro
shown in the chart, we 1
" (f>
learned that regional and ?
headquarters grant |
specialists are responsible for z
managing anywhere from 80
to 150 assistance
agreements. We believe it
would be beneficial for the
SROs to conduct a workload
analysis to determine the appropriate number of assistance agreements
grant specialists, as well as project officers, can effectively manage. We
recognize the level of difficulty varies with the type of assistance
agreement. However, this analysis, coupled with the SROs adjusting
resources accordingly, will assist the Agency in its efforts to correct the
weaknesses associated with oversight of assistance agreements.
SROs are responsible for continuously monitoring and improving the
effectiveness of management controls associated with their regions and
programs. This continuous monitoring, as well as other evaluations,
provide the basis for the Administrator's annual statement and report
to the President as required by the Integrity Act. Therefore, it is
crucial that SROs take necessary action to ensure their regional and
program resources are properly managed. EPA limits its ability to
deliver environmental results to the public whenever it does not
adequately administer assistance agreements.
Recommendations
We recommend that the Assistant Administrator for the Office of
Administration and Resources Management:
4. Clearly define which SRO (headquarters or regional) is responsible
for the oversight priorities of the regional program offices.
Report No. 2002-P-00018
23
-------
5. Direct SROs to ensure compliance with post-award monitoring
policies.
6. Require SROs to conduct workload analyses to determine the
appropriate number of assistance agreements that their grant
specialists and project officers can effectively manage.
7. Require SROs to dedicate an adequate level of resources to ensure
effective oversight of assistance agreements.
Agency Response and OIG Evaluation
OARM agreed with Recommendations 4, 6, and 7 and indicated they
will prepare a long-term strategic plan for grants management. This
plan will address SRO roles and responsibilities, workload analyses, and
SRO resource commitments to ensure effective oversight. The Agency
anticipates having a final plan in place by January 2003.
OARM's efforts to create a long-term strategic plan is a positive step
forward. However, implementation as well as creation of the strategic
plan is critical to improving EPA's accountability and oversight of
assistance agreements.
Regarding Recommendation 5, OARM believes the SROs are firmly
committed to post-award monitoring due to the high quality of the post-
award monitoring plans submitted in 2002. However, ensuring
compliance with the policies does not only include submitting post-
award monitoring plans. A true commitment by the SROs would
include compliance with all post-award monitoring policy requirements
along with effective implementation.
Report No. 2002-P-00018
24
-------
Internal Management Reviews
Need Improvement
We noted several areas in which EPA's internal management reviews
could be improved. These reviews include management oversight
reviews and post-award validation reviews, which can be valuable tools
for EPA to identify any vulnerable areas in the management of
assistance agreements. In particular, EPA can determine whether
post-award monitoring needs improvement and then take necessary
steps to correct the weaknesses.
Management Oversight Reviews
During our evaluation of management oversight reviews (MORs)
dating back to fiscal year 1998, we identified the following areas for
improvement:
+ Focus of the MORs.
4 Addressing prior MOR findings.
4 Using MOR results for the Integrity Act Report.
Focus of MORs
MORs are intended to evaluate the quality of EPA's assistance
agreement management. As a result, we believe these reviews should
include a cradle-to-grave evaluation of assistance agreement
management. However, when we examined the review protocols used
for the 2001 MORs, we learned that GAD did not satisfactorily address
completed or closed projects. Of the approximately 63 assistance
agreements sampled by GAD for the MORs, only 6 agreements were
closed. Further, the checklists completed by GAD for these agreements
did not indicate an assessment of whether the assistance agreement
objectives were achieved or the individual projects were successful. A
project's level of success could provide a good indication of the quality of
post-award monitoring.
We also found that although in many cases a weakness was apparent,
review comments were often vague and GAD did not draw conclusions
about the quality of post-award monitoring performed. Further, GAD
Report No. 2002-P-00018
25
-------
representatives indicated the MORs are strictly of the grants
management office oversight activities. We believe these reviews also
need to include program offices' oversight activities to get a complete
picture of how well the Agency is monitoring assistance agreements.
Prior MOR Findings
In the MORs conducted from 1998 through 2001, GAD identified
weaknesses in EPA's oversight of assistance agreements that were
repeat findings for certain regional offices. We learned that there were
several instances of previously-reported weaknesses going uncorrected
and GAD took no action to enforce compliance. Additionally, GAD did
not always address how the weaknesses reported in MORs should be
corrected. We believe GAD should make recommendations to correct all
weaknesses reported in the MORs.
Using MOR Results for the Integrity Act Report
In the 1999 Integrity Act Report, the 1999 MORs were listed as "results
indicators" to support the correction of the oversight material weakness.
We examined these reviews and found that the issues noted in the
reports identified several oversight areas where the Agency needed
improvement, and the decision to report the oversight weakness as
corrected was contrary to the review findings. In the future, MOR
results should be used when making decisions regarding Integrity Act
weaknesses.
Post-Award Validation Reviews
In 2001, GAD performed 10 post-award validation reviews (6 in
headquarters and 4 in the regions). These reviews were performed to
verify that corrective actions were effective in improving post-award
monitoring. We have concerns with: (1) the scope of the reviews, and (2)
how the results were reported.
Scope of the Validation Reviews
GAD limited its reviews of headquarters programs to project officer files.
To get a complete picture of post-award monitoring activities and
determine where oversight weaknesses may exist, GAD should have
also reviewed the corresponding grant specialist files. In addition, GAD
also limited its reviews of headquarters programs to active assistance
Report No. 2002-P-00018
26
-------
agreements. Like the MORs, closed or completed awards should have
been reviewed. EPA's determination that a final product was
unsatisfactory could be indicative of the quality or level of oversight
provided.
Validation Review Reports
A key concern with the 10 reports we reviewed was that GAD did not
draw overall conclusions on the effectiveness of post-award monitoring
by the grants management offices and program offices. Also, GAD did
not quantify its review results in the reports. During each validation
review, GAD reviewed a sample of assistance agreements. However, the
reports did not stipulate how many of those agreements received post-
award monitoring or whether the monitoring was effective. GAD's
conclusions regarding the quality of oversight, including the severity of
the weaknesses, are crucial for determining improvements needed to
correct the current Agency-level weakness.
Recommendations
We recommend that the Assistant Administrator for the Office of
Administration and Resources Management improve the internal
management reviews. Specifically for the:
8. Management Oversight Reviews —
a. Require a cradle-to-grave review of assistance agreements,
including an evaluation of closed awards. This review should
also include the grants management office responsibilities, as
well as program offices.
b. For prior review findings that continue to exist, ensure
corrective action is implemented.
c. Ensure the results are used to support decisions regarding
Integrity Act weaknesses.
Report No. 2002-P-00018
27
-------
9. Post-Award Validation Reviews -
a. Include an evaluation of closed or completed awards to
determine the effectiveness and usefulness of the final
products.
b. Quantify the results of the reviews and provide conclusions
concerning the effectiveness of post-award monitoring.
10. Conduct a study to determine whether the existing internal
management reviews can be combined into one review to improve
their effectiveness.
Agency Response and OIG Evaluation
OARM agreed with Recommendations 8a and 10, indicating it will
combine the Management Oversight Reviews with the Post-Award
Validation Reviews to perform a cradle-to-grave review of assistance
agreements.
For Recommendation 8b, OARM stated follow-up of prior review
findings has always been part of the MOR process and the MOR reports
have identified corrective actions needed for any prior review findings.
Based on this information, we are modifying the recommendation to
include the need for OARM to ensure the corrective actions are
implemented and the prior MOR findings are corrected.
For Recommendation 8c, OARM indicated they will use the results of
the internal reviews in making decisions on Integrity Act weaknesses.
However, it should be noted that while the 2001 Regional post-award
validation reviews were conducted by OARM in April - June 2001, the
final reports were not issued to the Regional offices until March 2002.
OARM must ensure the reports are prepared timely in order for the
results to be used in making decisions on Integrity Act weaknesses.
OARM agreed with Recommendation 9a and 9b and stated the
validation reviews being performed in FY 2002 will include a limited
review of product accomplishment. OARM also stated that, in 2003, it
will provide more quantitative data in the internal review reports.
Report No. 2002-P-00018
28
-------
Appendix A
This audit included tests of the program records and other auditing procedures we
considered necessary. We performed this audit in accordance with the Government
Auditing Standards, issued by the Comptroller General of the United States.
We did not take a sample of assistance agreements; therefore, we did not review
project officer or grant specialist files. However, we reviewed GAD files and
interviewed staff pertaining to MERs, MORs, and post-award validation reviews and
relied on the results of these reviews. We also interviewed SROs from headquarters
program and regional offices. We did not review the internal controls associated with
the input and processing of information in EPA's Grant Information Control System
or any other system.
We reviewed EPA's fiscal year 1995 and 1997 through 2001 Assurance Letters that
were prepared to comply with the Federal Managers'Financial Integrity Act. In
addition to the Assurance Letters, we reviewed EPA's fiscal year 1996, 1998, and
1999 Integrity Act Reports to the President and Congress; EPA's fiscal year 2000
Annual Report; the fiscal year 2001 and 2002 Annual Plans; and EPA Order
1000.24 on Management Integrity. We did not evaluate the corrective action taken
to remove "Closeout of Assistance Agreements" from the Integrity Act Report.
We also examined EPA's post-award monitoring policies, as well as several post-
award monitoring plans. As part of this review, we looked at fiscal year 2001
assistance activity quarterly reports, and information stored in the Grantee
Compliance Activities Database. We reviewed the Project Officer's Handbook and
project officer training policies, and also interviewed 65 project officers and grant
specialists. Regarding SRO responsibilities, we reviewed the following: EPA Order
1130.2A- SRO Responsibility Statement, dated May 31, 1994; the President's
Management Agenda, issued August 2001; and OARM's Management Action Plan
through Fiscal Year 2000.
Our fieldwork was conducted from October 16, 2001, to April 5, 2002. On February
25, 2002 we provided a finding outline with recommendations to the Office of Grants
and Debarment and GAD for review. They did not provide a written response, but
requested a conference call to discuss the finding outline. During our conference call
on April 2, 2002, these officials indicated they believe EPA has made many
improvements in their oversight of assistance agreements, but also agreed more is
needed. We issued the draft report on July 3, 2002 and received OARM's response
Report No. 2002-P-00018
29
-------
on September 9, 2002. We held an exit conference with OARM on September 26,
2002. Agency comments and our evaluation are summarized at the end of each
chapter, and a copy of the Agency response is provided in Appendix B. Due to the
volume of the attachments to the Agency response, we did not include them in the
final audit report. These attachments are available upon request.
Prior Audit Coverage
4 EPA's Controls Over Assistance Agreements
EPA-OIG - Report No. E1FMF4-03-0141-5100513, September 28, 1995.
4 Increased Focus on Grant Management and Internal Relationships
Would Improve Region 8's Tribal Assistance Program
EPA-OIG - Report No. 2000-P-000615-00021, September 29, 2000.
4 EPA's Oversight of Nonprofit Grantee's Cost is Limited
GAO - Report No. GAO-01-366, April 6, 2001.
4 EPA's Competitive Practices for Assistance Awards
EPA-OIG - Report No. 200 l-P-00008, May 21,2001.
4 Surveys, Studies, Investigations, and Special Purpose Grants
EPA-OIG - Report No. 2002-P-00005, March 21, 2002.
4 Procurements Made by Assistance Agreement Recipients
Should Be Competitive
EPA-OIG - Report No. 2002-P-00009, March 28, 2002.
Report No. 2002-P-00018
30
-------
Appendix B
Agency Response
September 6, 2002
MEMORANDUM
SUBJECT: Response to Office of Inspector General Draft Audit Report,
"EPA's Oversight Controls for Assistance Agreements,"
Assignment #2001-001105
FROM: Morris X. Winn
Assistant Administrator for Administration and
Resources Management
TO: Michael A. Rickey
Director for Assistance Agreement Audits
This provides the response of the Office of Administration and Resources Management
(OARM) to the Office of the Inspector General's (OIG) draft audit report (report) on EPA's Oversight
Control's for Assistance Agreements dated July 3, 2002. As noted in the report, the objectives of the
audit were to determine whether: 1) Grants specialists and project officers effectively provide oversight
of projects according to established guidelines; and 2) Senior Resource Officials (SRO) effectively fulfill
their oversight responsibilities for assistance agreements.
A. Background
In Fiscal Year (FY)1996, based on findings made by the OIG, EPA declared Grants Closeouts
and Oversight of Assistance Agreements a Material weakness. In response, the Agency implemented a
post-award corrective action strategy, including closing out a backlog of approximately 20,000 grants,
providing extensive project officer training, and issuing formal post-award management policies. As a
result of these efforts, EPA, with the concurrence of the OIG, redesignated this weakness as an Agency
weakness in FY 1999 and eliminated it in FY 2000.
However, in FY 2001, the OIG included assistance agreements on its list of management
problems requiring high-level Agency attention, citing audits suggesting the need for EPA to
validate its post-award management strategy. EPA therefore designated a new Agency weakness for
FY 2001 entitled "Improved Management of Assistance Agreements" and conducted a formal
Report No. 2002-P-00018
31
-------
validation study covering five Headquarters program offices and four Regional offices. The study
indicated that while the Agency had made progress in grants management, further improvement was
needed. Based on the results of the study, EPA carried forward the weakness into FY 2002.
The OIG conducted this audit to evaluate the effectiveness of the Agency's corrective strategy
to improve the oversight of assistance agreements. Examining the Agency's policies and procedures for
post-award monitoring, training, management effectiveness reviews and other internal reviews, the
report finds that the corrective action taken by EPA since 1996 was often ineffective and its
implementation not enforced. The report also reviews the role of SROs and finds that they did not
ensure compliance with post-award monitoring policies. Based on these findings, the report presents a
number of recommendations for my consideration.
B. EPA's Progress in Grants Oversight
At the outset, let me state that I fully agree with the OIG that strong, sustained oversight
controls are needed to ensure that taxpayers receive the benefit of the government's investment in
environmental protection and that funds are safeguarded against misuse. I also agree with many of the
report's recommendations. At the same time, I believe that report's central premise - that the
Agency's oversight controls have been generally ineffective — does not provide an accurate picture of
grants oversight at EPA. In my opinion, the Agency has made, and continues to make, forward
progress in this area and with continued improvement, will become a "best practices" agency.
In response to the FY 1996 Material weakness designation, EPA has worked hard to create an
effective oversight program composed of post-award monitoring policies, grants management training,
and formal internal management reviews. Considering that there was minimal oversight prior to 1996,
and given the absence of best practices for grants oversight in the Federal Government,1 the transition
to a new system has not been easy. Nevertheless, as discussed below, there is clear evidence that the
Agency is moving in the right direction.
First, a March 2001 report issued by the OIG on non-profit grants awarded by headquarters
and Region IV exemplifies the progress EPA is making in grants oversight. See OIG Report No.
2001-P-00005 (March 29, 2001). There, the OIG found that Headquarters and Region IV had
undertaken initiatives to improve the grants administration process, including training of grant specialists
and project officers, issuance of new or revised policy guidance, selective on-site reviews of recipient
organizations to assess their performance, and implementation of an internal review process that
analyzed specific aspects of grant programs on an ongoing basis. It also found that the Agency
1 As noted in the General Accounting Office report entitled "Environmental
Protection: EPA's Oversight of Nonprofit Grantees' Cost is Limited," (GAO-
01-366, April 2001), Federal agencies have experienced difficulties in identifying
best practices for grantee oversight.
Report No. 2002-P-00018
32
-------
maintained appropriate relationships with recipient organizations and avoided conflicts of interest
situations, and that specific grants reviewed complied with the Federal Grant and Cooperative
Agreement Act. In light of these findings, the audit concluded that a review of additional grant
agreements based on the same objectives was not warranted.
Second, while the OIG takes issue with the Agency's system for training project officers, the
fact remains that the Agency's basic three-day project officer training course has been, and continues
to be, highly successful, as evidenced by the number of employees trained and course evaluations.
Nationwide, using the training manual developed by the Office of Grants and Debarment (OGD), the
Agency has trained more than 5000 EPA employees. The course covers the entire grants process and
gives project officers a thorough grounding in their responsibilities for post-award monitoring.
Moreover, based on a validation study conducted in FY 1998, OGD made improvements to the course
in FY 1999 and is in the process of expanding it to provide greater detail on application, budget and
procurement review as well as planning for environmental results. To date, at least twenty-five
individuals from the OIG have taken the course, and the course evaluations on file from OIG staff are
very favorable.
Third, although the report leaves the impression that little or no post-award monitoring is
occurring, the results of OGD's internal reviews demonstrate otherwise. It is clear from OGD's Post-
Award Validation Reviews, for example, that project officers generally are very familiar with their
projects, and frequently monitor project progress through meetings and conference calls with recipients,
as well as through review of progress reports. While monitoring is in fact occurring, OARM recognizes
that project officers must do a better job of documenting the work in the project file.
Fourth, despite the report's criticism of the Agency's 2000 post-award monitoring plans, it
must be noted that the quality of post-award monitoring plans increased dramatically in 2002. The vast
majority of the final 2002 plans received by OGD are very comprehensive and demonstrate a
commitment to post-award monitoring. For example, the Office of Air and Radiation (OAR) prepared
its plan based on a template developed by OGD. The plan outlines an aggressive program of baseline
monitoring by requiring project officers, upon the receipt of progress reports, to perform a review that
answers the following questions:
• Is the recipient work progress to date satisfactory?
• Are the recipient's funds expenditures consistent with work performed to date?
• Does the information available in EPA's Financial Data Warehouse indicate any areas of
concern?
• Is the recipient in compliance with programmatic terms and conditions?
• Did the recipient encounter any difficulties during the period reported?
• If difficulties were encountered, did the recipient propose satisfactory remedies?
• Did the recipient discuss activities planned for the next reporting period?
• Did the recipient procure equipment?
Report No. 2002-P-00018
33
-------
• If the recipient did procure equipment, did the recipient provide appropriate information, such
as the make, model, and serial number?
In addition to baseline monitoring requirements, the plan commits OAR to performing advanced
monitoring of 10% of its recipients. It also contains an innovative feature providing for special reviews
of recipients receiving Federal funding for the first time, and requires project officers with grants of
$750,000 or more to develop a monitoring strategy for the life of the award.
Two other excellent examples of post-award monitoring plans include the plans submitted by
EPA Regions n and X. These plans cover both grants management and program personnel. The
Region X plan contains an extensive discussion of baseline monitoring, while the Region n plan
identifies non-profit organizations as the primary focus for advanced monitoring. I am attaching for your
information copies of the OAR (Attachmentl), Region n (Attachment 2)and Region X (Attachment 3)
plans.
Fifth, while the report questions the effectiveness of EPA's system for internal assessments, I
believe that the system has served the Agency well in many cases by identifying oversight problems or
other grant vulnerabilities and facilitating necessary corrective action. For example, OGD's FY 2002
Post-Award Validation Review/Management Oversight Review for Region I examined the Region's
post-award management policies and procedures and provided clear recommendations on needed
changes, including changes in baseline monitoring, the development of a Region-wide post-award
management plan, and grants specialist training. In response, the Region: 1) fully embraced the
baseline monitoring program by developing a baseline monitoring checklist for both grants specialists
and project officers; 2) submitted a post-award management plan that applies to both the grants
management and program offices and identifies specific agreements subject to advanced monitoring;
and 3) provided training to all of its grants specialists on post-award management.
Similarly, in FY 2001, OGD conducted validation reviews of a number of Headquarters
program offices to determine their compliance with the Agency's post-award policies. Among other
things, the reviews identified problems in documenting post-award monitoring, including documenting
files to reflect conversations or correspondence with recipients and placing copies of award and
monitoring materials in the project file. In FY 2002, OGD is conducting follow-up reviews to
determine whether corrective action has been taken. The preliminary results of these reviews reveal
significant improvements in file documentation and integrity. Further, with regard to Management
Effectiveness Reviews (MERs), a number of program offices have used the results of MERs to
implement necessary corrective actions. OSWER, for example, responded to the documentation
weakness identified in its 1999 MER by developing a documentation checklist be used by all OSWER
project officers. Additionally, based on the results of its 1997 MER, the Office of Water adopted a
policy requiring quarterly reporting on all of its Headquarters assistance agreements in order to
strengthen post-award monitoring.
Report No. 2002-P-00018
34
-------
Sixth, any assessment of the risk to EPA and the taxpayers of mismanagement by EPA grantees
must be viewed in the context of the Agency's overall assistance program. Significantly, none of the
OIG or General Accounting Office reviews listed in the report cite EPA's administration of State and
local government program grants, which comprise over 87% of the grant funds awarded by EPA
annually, as an oversight weakness. This means that the primary area of oversight risk involves other
categories of grants that receive relatively small amounts of funds (e.g., grants to non-profit
organizations, which receive about 6% of EPA's grant dollars each fiscal year.) As the OIG is aware,
EPA is strengthening its internal management controls to mitigate this risk through new initiatives on
grant competition, post-award monitoring, and training/technical assistance.
While I believe the report minimizes, and in some instances does not describe, the good things
that EPA is doing in grants oversight, I personally concur with its conclusion that further work remains
to be done. For that reason, OARM is developing a long-term strategic plan to strengthen all aspects
of grants management. The plan will focus on: developing and retaining a skilled grants management
workforce; promoting grant competition; further enhancing grants oversight; providing training and
technical assistance to non-profit and Tribal recipients; and improving accountability, coordination and
resource management in the grants area. The Agency anticipates having a final plan in place by January
2003. Needless to say, implementation of the strategic plan is the most critical part of our oversight
efforts. Successful implementation will require that OGD, the GMOs, the SROs and the program
offices double their efforts to plan their work and work their plan in order to provide high level,
effective and sustained oversight.
C. Specific Comments
We have the following comments on specific sections of the report.
Overview (page 6)
We recommend that the second sentence be revised as follows: Although this corrective
action has resulted in improvements in grants oversight, we found that further improvement
and management attention is needed to strengthen EPA's oversight program."
Roles and Responsibilities (pages 3-4)
This discussion should be modified to accurately reflect the relative roles of the Grants
Management Offices (GMOs), grant specialists and project officers in post-award monitoring. The
report states that "project officers are responsible for monitoring project performance under the terms
of the agreement," while the GMO is "responsible for the review, negotiation, award, and
administration of assistance agreements, including audit resolution and final closeout." In fact, both
project officers and staff from the GMO are responsible for the review, negotiation, and administration
(including post-award monitoring and close-out) of assistance agreements. Project officers are
primarily responsible for these functions as they relate to the programmatic and technical requirements
Report No. 2002-P-00018
35
-------
of the assistance agreement while grant specialists are principally responsible for the administrative and
financial aspects.
The last sentence of the description of "Grants Specialist" should be revised to make clear that
specialists are responsible for developing and maintaining the official grant file. The following sentence
should also be added to "Project Officer" description: "They are also responsible for developing
and maintaining the official technical project file."
Baseline Monitoring (pages 7-8)
The report criticizes concerns expressed by a number of GMOs over the utility of baseline
monitoring. However, those concerns were not, as the report suggests, aimed at reducing the Agency's
commitment to grants oversight. Instead, they reflected an honest difference of opinion over the
effectiveness of baseline monitoring vis-a-vis other forms of monitoring. EPA formally included baseline
monitoring as a type of monitoring in EPA Order 5700.3 with the expectation that it would be an
efficient way of detecting problems. Initially, baseline monitoring did not meet that expectation, since
most reviews indicated that grantees were in compliance with administrative requirements. This
prompted some GMOs to ask whether baseline monitoring was a good use of grants specialists' time
and whether the Agency should rely instead on alternative methods of monitoring to identify problem
grantees. After carefully considering the views of the GMOs, and based on further experience, OARM
has determined that baseline monitoring is critical to an effective oversight program and has
incorporated it in the proposed consolidated post-award monitoring policy. That policy is currently in
the Directives Clearance process. Under the new policy, baseline monitoring will be required on an
ongoing basis for every award.
Post-Award Monitoring Plans (pages 9-10)
The report states that post-award monitoring plans, if properly prepared, can assist programs
and Regions in improving the effectiveness of post-award monitoring activities. Based on a review of
seven plans submitted in 2000, the report finds that plans are not always timely or complete, may omit
required information, and may not include program office activities. In the OIG's opinion, these
deficiencies suggest a lack of commitment on the part of the SROs to grants oversight.
I agree with the report that post-award monitoring plans are the cornerstone for effective post-
award management. However, I understand that the OIG has not had the benefit of reviewing the final
2002 plans submitted by the SROs. As I explained previously, the vast majority of these plans are
significantly improved over previous year plans. Although there are still some timeliness issues, which
OARM will take steps to address, the 2002 plans, on balance, provide a solid basis for the
implementation of post-award monitoring activities. In short, these plans are proof of the importance
that SROs attach to post-award monitoring.
Report No. 2002-P-00018
36
-------
Quarterly Reports on Post-Award Activities (page 10)
The report questions the usefulness of the quarterly reports to Congress on post-award
activities prepared by the Grants Administration Division (GAD). It is true that these reports have
focused almost exclusively on GMO, as opposed to program office, activities. However, GAD
structured the reports in this fashion to respond to Congress' overriding concern with grantee violations
of allowable cost principles (e.g., the costs of lobbying or suing the Federal Government) or other
administrative requirements, areas that fall within the purview of the GMOs. OARM agrees with the
OIG that it is appropriate to expand the reports to identify programmatic activities so that Congress has
a complete picture of EPA's oversight efforts. In the recent third quarter report to Congress
(Attachment 4), the Agency provided a partial listing of program activities and will include a full listing
beginning with the fourth quarter report.
On-Site Evaluations (pages 11-12)
The report states that on-site reviews are necessary to ensure that Federal assets are
safeguarded, funded work is on schedule and of adequate quality, and recipients comply with award
terms and conditions. I agree with the OIG that on-site reviews are important. At the same time, I
have significant and strong reservations about the report's suggestion that on-site reviews must be
conducted on virtually every major EPA grant, even when other, less costly forms of monitoring, such
as desk reviews, may suffice. As described below, in those cases where desk reviews or other less
costly forms of monitoring identify significant problems, Agency managers will be expected to take
appropriate corrective action or refer the recipient for an on-site review.
Baseline monitoring is the minimum monitoring performed by grants specialists and project
officers. For grants specialists, baseline monitoring consists of ensuring, to the best of their ability, that
recipients are: 1) complying with the terms and conditions of the award, including the submission of
progress reports; 2) filing audits, if applicable, under Office of Management and Budget Circular A-
133; and 3) submitting financial status reports in a timely fashion. Additionally, grants specialists review
expenditures by the recipient against project progress. Conversely, project officers focus baseline
monitoring on compliance with programmatic award terms and conditions, including but not limited to,
receipt of progress reports and Quality Assurance.
In contrast to baseline monitoring, advanced monitoring involves a more detailed process for
measuring a recipient's compliance with applicable administrative and programmatic requirements.
GMO advance monitoring includes an extensive evaluation of a recipient's general financial and
administrative management systems. This entails a review of all the assistance agreements with the
recipient as well as a search of the recipient's web pages for policies and procedures related to financial
management, property, procurement, personnel, and other systems (e.g., timekeeping, payroll,
overtime, travel). If the web site does not contain this information or no web site is available, the GMO
asks the recipient to provide necessary documentation. The GMOs conduct these reviews and invite
participation by the project officer and other interested parties. In conducting the reviews, GMOs
follow established protocols to ensure all relevant areas are covered with consistency.
Report No. 2002-P-00018
37
-------
In carrying out advanced monitoring, project officers review the project work plan, recipient
progress reports and any products produced to date. They also review the EPA role under the
agreement and the recipient's payment history, assess whether the recipient has met programmatic
reporting requirements, and determine whether the recipient's progress is commensurate with payments
made by the Agency. As with GMO advanced monitoring, project officers follow established
protocols.
Off-site evaluations (i.e., desk reviews) include the above areas of review and are
accomplished in the office, telephonically. GMOs/Project officers provide recipients with advance
notice and ask them to provide documents, as appropriate. They then schedule the call and identify
key participants. Once the review has been completed, the Agency develops findings, generates a
report, and either institutes corrective action or takes other appropriate follow-up action, including
referral for an on-site evaluation.
Performed at the recipient organization, on-site evaluations are the most resource-intensive
form of post-award monitoring. For the GMOs, they include an assessment of a recipient's
management and financial systems, with transaction testing for unallowable costs performed during the
review. These reviews are conducted by a team similar in composition to that used in a desk review.
Reviews are typically three to five days in length. Travel costs can be substantial, particularly for
recipients in remote locations.
Given the large number of EPA assistance agreements and budgetary constraints, it is crucial
for the Agency to prioritize which recipients are subject to on-site reviews. On-site reviews may be
appropriate, for example, if there is clear potential for the misuse of EPA grant funds. Nonetheless, not
every recipient or project warrants an on-site review. In many cases, the Agency can identify problems
or issues through baseline monitoring or desk reviews, and then determine whether an on-site review
with transaction testing is warranted. Moreover, as in the case of an on-site review, the scheduling,
substantive planning and actual performance of an off-site review sends a strong signal to recipients that
EPA takes its fiduciary obligations seriously.
I believe that the proposed consolidated policy on post-award monitoring creates a monitoring
program that is both strong and cost-effective by: 1) requiring GMOs and program offices to increase
the level of advanced monitoring from a minimum of 5-10% of active recipients to a minimum of 10%
of active recipients; 2) requiring each GMO and program office to perform at least one on-site review a
year; and 3) requiring each GMO to perform annually a minimum number of desk reviews equal to one
per grant specialist. Where desk reviews identify significant problems, GMOs and program offices will
be expected to take necessary corrective action or target the matter for an on-site review.2 OGD is
currently evaluating the comments submitted on the proposed policy, and will have discussions with you
before the policy is finalized.
2 For example, in FY 2001, GAD referred five recipients for on-site reviews as
a result of significant issues identified during desk reviews. Thus far in FY 2002,
GAD has referred six recipients based on the results of desk reviews.
Report No. 2002-P-00018
38
-------
Training requirements (pages 13-14)
The report finds inefficiencies in GAD's system for monitoring projects officer training, citing
problems with GAD's Agency-wide training database and the lack of adequate controls over the on-
line project officer refresher course. While I agree that the current system should be upgraded, I want
to emphasize that it does contain safeguards to help prevent uncertified project officers from managing
an assistance agreement. Specifically, before an award is made, GMOs check the database to ensure
the project officer has been certified (or re-certified as necessary). If there is any doubt about a listing
(or no listing at all), the GMO contacts the program office for either verification of current training status
or the name of a new project officer whose training is complete. In no event is an award made without
the designation of a (re)certified project officer as the program manager for the agreement. With regard
to the on-line refresher course, although the OIG has not provided any evidence to OARM of misuse
by project officers, GAD has made changes that should prevent unauthorized certifications.
The report also states that EPA has not specified training requirements for grant specialists, but
acknowledges that most specialists do take the basic project officer training course. In addition to the
basic course, the GMOs have provided a significant amount of contractor-delivered training to
specialists and encouraged them to participate in a private sector grants management certification
program. OARM recognizes that the core body of knowledge required for grants specialists has
increased significantly over the years. To address this issue, GAD has updated the core body of
knowledge and incorporated it in a curriculum that was presented at a pilot training session in June
2002 for thirty specialists from Headquarters and the Regions. Based on the results of the pilot, GAD
is in the process of making adjustments to the training program and will have it fully deployed in the
GMOs by June 2003.
Management Effectiveness Reviews (pages 15-17)
The report criticizes the Agency's process for internal Management Effectiveness Reviews
(MERs) that Headquarters program and Regional offices periodically perform of their grant programs.
As noted above, I believe that the Agency's system for internal assessments, including the MERs, has
had the salutary effect of focusing senior management attention on post-award management issues.
Nevertheless, I agree with much of the report's analysis, particularly the need for OARM to provide
more frequent guidance and timely feedback. Under the proposed consolidated policy on post-award
monitoring, GAD will address that issue by providing comprehensive guidance in advance of the next
round of MERs in 2004.
I would like to comment on two points raised by the report's discussion of MERs, namely: 1)
that EPA should have reviewed the results of the 1999 MERs before removing the Material weakness
designation in FY 1999; and 2) that OARM should have required programs to focus the 1999 MERs
exclusively on post-award monitoring.
FY 1999 Integrity Act Process: The 1999 MERs were still ongoing at the time of the Agency's FY
1999 Integrity Act deliberations. Thus, the results of the MERs were not available when the Agency
Report No. 2002-P-00018
39
-------
decided to redesignate grants management from a Material to an Agency weakness.3 Although that
decision did not have the benefit of the MER results, it did reflect the Agency's considered judgment —
a judgment concurred in by the OIG-- that EPA had substantially eliminated a closeout backlog of
nearly 20,000 grants and had put in place sufficient internal controls to address post-award
management, including requirements for MER corrective action plans and post-award monitoring
plans.
Scope of the 1999 MERs: OARM allowed program offices to choose one of the five topics for the FY
99 MERs. These topics included: 1) Pre-Award Negotiations; 2) Post-Award Monitoring; 3)
Closeouts; 4) Statutory Authority/Delegations of Authority; and 5) SRO Review Levels. OARM did
not require programs to limit the MERs to post-award monitoring for two reasons.
First, in 1999, OARM developed EPA Order 5700.3 prescribing requirements for post-award
monitoring plans, and also directed the Directors of OGD and GAD to meet with all SROs to discuss
and evaluate their post-award monitoring activities. OARM believed these measures would ensure
continued program focus on post-award monitoring independent of the MERs process.
Second, OARM determined that it was critical for the Agency, as it addressed post-award
monitoring, to ensure that problems did not arise in other important areas of grants management. It
therefore gave programs the flexibility in the MERs process to address issues besides post-award
monitoring. Looking at the additional topics authorized by OARM, I am sure you would agree that the
Agency must continue to be vigilant (1) in reviewing its grants closeouts procedures, which prompted
the original Material weakness designation, and (2) in addressing statutory authority and pre-award
management, which have been, or currently are, the subject of OIG audits.4
While I will not second-guess the judgment made by my predecessor Assistant Administrator
in 1999,1 will consider the concerns of the OIG as GAD develops the guidance for the 2004 MERs.
3 By memorandum dated May 20, 1999, OARM asked program offices to
submit a report on their 1999 MERs by December 31, 1999. Typically, Agency
decisions on Material weakness candidates are made in the September-November
time frame.
Statutory Authority for EPA Assistance Agreements, E3AMF8-
11-0008-8100209, issued September 18, 1998, and Awarding Assistance
Agreements, Assignment No. 2002-000346.
Report No. 2002-P-00018
40
-------
Senior Resource Officials Did Not Ensure Compliance with Policies (pages 19-21)
The report states that the SROs did not fulfill their stewardship responsibilities for assistance
management, failed to emphasize the importance of effective post-award monitoring, and allowed non-
compliance with post-award monitoring requirements. According to the report, the inability of SROs to
meet their responsibilities may be attributable to a number of factors such as a lack of resources and
limited control of Regional/Headquarters SROs over Regional program oversight priorities. The report
offers a number of potential solutions, including more resources, more efficient use of existing resources,
giving Regional SROs the responsibilities to manage assistance agreement workloads for both program
offices and the GMOs, and the development of workload limits for grants specialists and project
officers.
Notwithstanding the report's criticism, I am convinced that the SROs are firmly committed to
effective post-award monitoring, as evidenced by their submission in 2002 of excellent post-award
management plans. Fundamentally, the issues raised by the OIG are a subset of the larger challenge
faced by the Agency of ensuring effective regulatory programs and appropriate grants oversight in an
era of limited resources. It is precisely because of that challenge, and the fact that grants comprise over
50% of the Agency's budget, that OARM is preparing a long-term strategic plan for grants
management. Two of the major objectives of the plan will be to clarify the roles and responsibilities of
the SROs for post-award monitoring and establish a partnership between OARM and the program
offices to improve accountability, coordination and resource/workload management in the grants area.
OARM will consult with the OIG as it develops and implements the plan.
Management Oversight Reviews (pages 23-24)
The report criticizes the quality and usefulness of EPA's Management Oversight Reviews
(MORs), claiming that: 1) their scope is too narrow by excluding closed or completed projects and
program offices; 2) they are vague and fail to address prior review findings; and 3) their results are not
used in the Integrity Act process.
The Agency instituted MORs to provide GAD with a vehicle to assess grants management
practices in the Regional GMOs. GAD conducts three or four MORs every year on a rotating basis.
While I agree that there is room for improvement in the MORs process, I believe that a number of the
OIG's criticisms are misplaced.
Scope of the MORs: GAD properly limited the scope of the MORs to active projects because at the
time the post-award monitoring policies were relatively new and thus would have minimal or no impact
on closed or completed projects. It was also proper for GAD to generally exclude program offices
from the review since their activities were the subject of a separate analysis under the Post-Award
Validation Review (PAVR) process. In this regard, although the MORs did not evaluate program
activity, they did include interviews with program representatives to determine the extent of
communication between the programs and the GMOs and opportunities for GMO customer service
improvements.
Report No. 2002-P-00018
41
-------
Vagueness and Prior Review Findings: The MORs represent a detailed assessment of GMO
operations and place great emphasis on prior review findings. For example, the 2001 MOR for
Region I carefully analyzed whether all prior review recommendations had been implemented and found
that the Region had done so. It then proceeded to examine the GMO's organizational effectiveness, its
commitment to quality grants management, and whether the GMO was complying with national grants
policies. The document concluded with six specific recommendations for corrective action addressing
customer service, training, monitoring provisional indirect cost rates, and program coordination. While
the document did not scrutinize the quality of post-award monitoring by grants specialists, it clearly
explained that monitoring would be evaluated under the parallel PAVR process.
Use in Integrity Act Reports: In FY 2001, OARM used the results of the MORs and other internal
reviews to evaluate the existing grants management weakness. Based on those results, OARM carried
over the weakness into FY 2002. OARM will also consider the results of the MORs when making
decisions in the FY 2002 Integrity Act process.
Post-Award Validation Reviews (pages 24-25)
The report examines ten Post-Award Validation Reviews (PAVRs) conducted by GAD in
2001 to verify compliance with post-award monitoring policies. It makes three major findings: 1) the
reviews of headquarters program offices should have covered corresponding grants specialist files and
examined closed assistance agreements for product accomplishment; 2) the reviews failed to draw
overall conclusions on the effectiveness of post-award monitoring by project officers and grant
specialists; and 3) the reviews did not quantify the number of assistance agreements receiving adequate
post-award monitoring. OARM's response to these findings is as follows.
Scope of Headquarters Program Office Reviews: GAD specifically designed the Headquarters
Program reviews to measure program office monitoring. The reviews therefore did not cover
Headquarters grant specialist monitoring, which was the subject of a separate GAD review. In
addition, as in the case of the MORs, GAD did not include closed or completed agreements in the
reviews because the post-award monitoring policies were relatively new.
Conclusions on Monitoring Effectiveness: The PAVRs both contained and facilitated overall
conclusions as to the adequacy of post-award monitoring. For example, findings common to virtually
all of the Headquarters PAVRs were that project officers needed to engage in more proactive
monitoring and pay greater attention to progress reports. Similarly, the 2001 Regional PAVRs
concluded that Regional project officers needed to address all of the core areas identified by the
Grantee Compliance Assistance Initiative and that grant specialists needed to place greater emphasis on
baseline monitoring. More importantly, OARM used the results of all the PAVRs to develop
conclusions on the state of post-award management at EPA. These findings were summarized in
OARM's Agency's FY 2001 Integrity Act Assurance letter and served as a basis for carrying forward
the Agency-level weakness into FY 2002.
Report No. 2002-P-00018
42
-------
Quantified Results: The Headquarters PAVRs do contain quantitative data, including the number of files
reviewed and the number of deficient files. That information is supported by additional quantitative data
contained in the work papers of the GAD review teams. In addition, the Regional PAVRs identified
the number of files reviewed and are supported by work papers providing detailed information noting
the deficiencies in individual files.
In short, I believe that the FY 2001 PAVRs served their intended purpose by influencing the
Agency's FY 2001 Integrity Act process. GAD is continuing to perform PAVRs in FY 2002 and is
including in them a limited review of product accomplishment. For FY 2003, GAD will expand its
analysis of product accomplishment, provide more quantitative data in the reports themselves, and, in
the interest of streamlining, combine the MORs and PAVRs into one review.
D. Response to Recommendations
The report contains a number of recommendations directed to the Assistant Administrator for
OARM. OARM's response to each recommendation is outlined below.
Recommendation 1 (page 17): All corrective action cited in the Integrity Act Report is effectively
implemented and the importance and benefits of such actions are clearly stated.
OARM Response: OARM agrees with this recommendation. We are currently developing the
analysis for FY 2002 Integrity Act Report, which will include an evaluation of EPA's progress in
improving post-award management.
Recommendation 2 (page 17): Documents required by post-award management policies are submitted
timely and meet policy requirements.
OARM Response: OARM agrees with this recommendation and is pleased with the quality of the final
post-award monitoring plans that it received in 2002. OARM will continue to work to address
timeliness issues.
Recommendation 3 (pages 17-18): Timely and effective guidance and oversight is provided to each
headquarters program and regional office. Specifically:
a. The proposed post-award monitoring policy should require that baseline monitoring be ongoing for
all awards.
OARM Response: The proposed policy submitted by OARM in the Directives Clearance process
includes this requirement.
b. The proposed policy should increase the minimum requirement for the number of on-site evaluations
conducted by the grants management and program offices.
Report No. 2002-P-00018
43
-------
OARM Response: As discussed in our comments, the proposed policy will increase the level of
advanced monitoring by GMOs and the program offices from a minimum of 5-10% of active recipients
to a minimum of 10% of active recipients annually. This will include, for both GMOs and program
offices, conducting at least one on-site evaluation each year, and for each GMO, conducting a minimum
number of desk reviews equal to the number of its grant specialists. GMOs and program offices may
conduct additional on-site reviews where justified by the results of baseline monitoring or desk reviews.
OARM believes the policy will strengthen post-award monitoring in a cost-effective way. GAD is
currently reviewing the comments received on the policy during the Directives Clearance process and
will consult with the OIG before the policy is finalized.
c. An appropriate and reasonable time frame should be established for providing feedback to the
Agency on submitted documents and reports.
OARM Response: OARM agrees with this recommendation and will establish a minimum 60 day
feedback standard.
d. Controls should be established to ensure information is consistently and accurately reported by all
headquarters and regional offices in the Grantee Compliance Activities Database.
OARM Response: OARM agrees with this recommendation. GAD has already put procedures in
place to ensure that reports submitted by the Regional GMOs include activities conducted by the
Regional program offices. The proposed consolidated post-award monitoring policy will require all
Regional and Headquarters program offices and GMOs to report post-award activities in the Grantee
Compliance Assistance Database. As pointed out in the report, GAD will be providing guidance on the
format, content and timing of the reporting.
e. Controls should be established to ensure project officers receive the required training prior to
managing an assistance agreement. Those controls must include documenting training information.
OARM Response: OARM agrees with this recommendation. GAD is developing an enhanced
database that will eliminate duplicate names or the names of individuals no longer with the Agency and
provide notification of the need for refresher training.
f Controls should be established to ensure that the on-line refresher course prevents the user from
obtaining an unearned certification.
OARM Response: OARM has already made changes to the course that should prevent unearned
certifications.
g. A core curriculum for grant specialists should be finalized and all required training should be
provided and taken.
Report No. 2002-P-00018
44
-------
OARM Response: As discussed in our comments, GAD has developed a core curriculum and pilot
training and will fully deploy the training to the Regional GMOs in June 2003.
h. The management effectiveness review guidance should require a consistent focus to identify and
address systemic weaknesses.
OARM Response: The next set of management effectiveness reviews will be performed in 2004.
GAD will develop guidance for the reviews to assure programs identify any weakness in grants
management and correct them. GAD will consult with the OIG before issuing the guidance.
i. The management effectiveness reviews should be appropriately planned. Once these reviews are
performed, adequate corrective action should be taken to address identified weaknesses.
OARM Response: OARM agrees with this recommendation and will address the need for corrective
action in the 2004 MER guidance.
Recommendation 4 (page 21): Clearly define which SRO (headquarters or regional) is responsible for
the oversight of the regional program offices.
OARM Response: OARM is addressing the issue of the roles and responsibilities of the SROs for
post-award monitoring in its long-term strategic plan, for grants management. OARM will consult with
the OIG as it develops and implements the strategic plan.
Recommendation 5 (page 22): Direct the SROs to ensure compliance with post-award monitoring
policies.
OARM Response: As indicated by the high quality of final post-award monitoring plans submitted in
2002, OARM believes the SROs are firmly committed to effective post-award monitoring. OARM is
addressing the issue of the roles and responsibilities of the SROs for post-award monitoring in its long-
term strategic plan for grants management. We will consult with the OIG as we develop and implement
the plan.
Recommendation 6 (page 22): Require SROs to conduct workload analyses to determine the
appropriate number of assistance agreements that their grant specialists and project officers can
effectively manage.
OARM Response: The GMOs have been compiling data on grant specialist and project officer
workloads. The workload issue is complex one. For example, some grant specialists and project
officers may also be responsible for duties other than assistance management. Other important
variables include the complexity and size of assistance agreements. OARM is addressing this issue in
the strategic planning process. We will consult with the OIG as we develop and implement the strategic
plan.
Report No. 2002-P-00018
45
-------
Recommendation 7 (page 22): Require the SROs to dedicate an adequate level of resources to ensure
effective oversight of assistance agreements.
OARM Response: OARM is addressing this issue in the strategic planning process. We will consult
with the OIG as we develop and implement the strategic plan.
Recommendation 8 (page 25): Management Oversight Reviews -
a. Require a cradle-to-grave review of assistance agreements, including an evaluation of closed
awards. This review should also include grants management office responsibilities, as well as program
offices.
OARM Response: OARM agrees with this recommendation. Viewing the Management Oversight
Reviews (MORs) and Post-Award Validation Reviews (PAVRs) together, OARM believes it has
conducted cradle-to-grave reviews of assistance agreements. Based on two years of experience of
conducting these reviews simultaneously, we have now determined that they can be combined, which
will make the cradle-to-grave coverage more apparent. The combined review will begin in FY 2003
and address both project officers and the GMOs and include an evaluation of closed awards.
b. Include corrective action that should be taken for prior review findings that continue to exist.
OARM Response: Follow-up of prior review findings has always been part of the MOR process and
the MOR reports have identified corrective actions needed for any prior review findings. GAD will
continue this practice in future reviews. No further action is required.
c. Ensure the results are used to support decisions regarding Integrity Act weaknesses.
OARM Response: OARM has used, and will continue to use, the results of its internal reviews in
making decisions on Integrity Act weaknesses. No further action is required.
Recommendation 9 (pages 25-26): Post-Award Validation Reviews -
a. Include an evaluation of closed or completed awards to determine the effectiveness and usefulness
of the final products.
OARM Response: In FY 2002, GAD is focusing its reviews on post-award monitoring of open
agreements, but will also include a limited review of product accomplishment. In FY 2003, GAD will
take a broader look at product accomplishment.
b. Quantify the results of the review and provide conclusions concerning the effectiveness of post-
award monitoring.
Report No. 2002-P-00018
46
-------
OARM Response: OARM will continue to provide conclusions on post-award monitoring
effectiveness in its internal review reports and Integrity Act Reports. While internal reports are
currently supported by quantitative data, in FY 2003, OARM will provide more quantitative data in the
report themselves.
Recommendation 10 (page 26): Conduct a study to determine whether the existing internal
management reviews can be combined into one review to improve their effectiveness.
OARM Response: OARM will combine the existing reviews (see response to Recommendation 8.a.).
E. Conclusion and Commitment
It is clear that the findings in this report reflect longstanding OIG concerns that the Agency's
efforts to date have not, as a factual matter, been effective and or given sustained management attention
and priority by OGD, GAD, the Regions and Headquarters program offices.
This response intends to provide a balanced report of the actual practices that are in place and
the Agency's progress in accomplishing many of the recommendations made by the OIG. It also
recognizes that further progress is needed and desired to eliminate documented or perceived
weaknesses. This response also provides a necessary glimpse into the compliance activities already
underway and, in our opinion, already working.
New to this corrective action equation is a further commitment to accountability demonstrated
by the strategic planning process. This accountability starts with: 1) the assignment and redeployment
by OARM of resources to the oversight process; 2) requiring programs and Regions to develop and
implement effective post-award monitoring plans; 3) cooperative efforts between OARM and the
Regions and Headquarters programs to identify additional resources needed to address oversight
problems; and 4) holding offices singularly responsible for providing the leadership to accomplish this
effort.
We have cited several areas where we firmly believe efforts to correct previously identified
weaknesses are well underway or working. As the Assistant Administrator for OARM, I believe that
excellent stewardship of Federal assistance dollars is critical. In this regard, I have redeployed
resources to OGD to assist with these oversight efforts and stand ready to work with the OIG and
others to strengthen the Agency's internal system for evaluating its oversight strategy, including the
performance of internal audits.
Thank you for the opportunity to comment on the report. If you have any questions about
OARM's comments, please contact Howard Corcoran, Director, OGD at (202) 564-1903.
Attachments
cc (w/o attachments):
Report No. 2002-P-00018
47
-------
Nikki Tinsley
Assistant Administrators
Regional Administrators
Dave O'Connor
Deputy Regional Administrators
Marty Monell
Marguerite Pridgen
Sherry Kaschak
Sandy Womack-Butler
Howard Corcoran
Lisa White
Senior Resource Officials
Grants Management Officers
Nancy Duvall
Peggy Anthony
Francis Roth
William Hedling
John Nolan
Scott McMoran
Report No. 2002-P-00018
48
-------
Appendix C
Office of Inspector General
Inspector General (2410)
EPA Headquarters
Assistant Administrator for Air and Radiation (6101A)
Assistant Administrator for Enforcement and Compliance Assurance (2201A)
Assistant Administrator for International Activities (2610R)
Assistant Administrator for Prevention, Pesticides, and Toxic Substances (7101M)
Assistant Administrator for Research and Development (8101R)
Assistant Administrator for Solid Waste and Emergency Response (5101)
Assistant Administrator for Water (4101M)
Comptroller (2 731 A)
Agency FoUowup Official (2710A)
Agency Audit Followup Coordinator (2724A)
Associate Administrator for Congressional and Intergovernmental
Relations (1301 A)
Director, Office of Regional Operations (1301 A)
Associate Administrator for Office of Public Affairs (1701 A)
Director, Office of Grants andDebarment (3901R)
Director, Grants Administration Division (3903R)
Director, Office of Executive Support (1104A)
Regional Offices
Senior Resource Officials (Regions 1-10)
Regional Audit Follow-up Coordinators (Regions 1-10)
------- |