fv®;l OIG
X	OFFICE OF INSPECTOR GENERAL
Ca talyst for Improving the Environment
Audit Report
EPA Must Emphasize Importance
of Pre-Award Reviews
for Assistance Agreements
Report No. 2003-P-00007
March 31, 2003

-------
Report Contributors:
Kathy Finazzo
Stephanie Oglesby
Richard Howard
Matthew Simber
Abbreviations
EPA	Environmental Protection Agency
GAO	General Accounting Office
OAR	Office of Air and Radiation
OIG	Office of Inspector General
OGD	Office of Grants and Debarment
OW	Office of Water

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
MEMORANDUM
March 31, 2003
SUBJECT:
Report No. 2003-P-00007
EPA Must Emphasize Importance of
Pre-Award Reviews for Assistance Agreements
FROM:
Michael A. Rickey /s/ Michael A. Rickey
Director, Assistance Agreement Audits
TO:
Morris X. Winn
Assistant Administrator for Office of Administration and
Resources Management
This is a final report on the Office of Inspector General's (OIG's) audit of the Environmental Protection
Agency's (EPA's) pre-award review of assistance agreements.
Action Required
In responding to the draft report, the Agency provided action plans with milestone dates to address
each of the recommendations. Therefore, no further action is needed. The Agency should track the
implementation of the corrective actions in the Management Audit Tracking System. The Agency
indicated when responding to the draft report that it plans to provide further comments on examples in
the report. Should the Agency do so, we would welcome the comments and the opportunity to review
and discuss them.
We have no objection to the public release of this report. This final report contains findings that the
OIG has identified and corrective actions the OIG recommends. The audit report represents the
opinion of the OIG and the findings contained in this report do not necessarily represent the final EPA
position. Final determination on matters in this report will be made by EPA managers in accordance
with EPA audit resolution procedures.

-------
If you or your staff have any questions regarding this report, please contact me at (312) 886-3037 or
Kathy Finazzo, Project Manager, at (913) 551-7833.

-------
Executive Summary
Purpose
More than half of the Environmental Protection Agency's (EPA's) $7.8 billion fiscal
2001 budget was awarded to organizations outside the Agency through assistance
agreements. Therefore, it is imperative that EPA award these agreements for projects
that will contribute most effectively to achieving EPA's specific program objectives and
priorities.
We conducted this audit to determine whether the process for awarding assistance
agreements resulted in projects that supported EPA's mission. Specifically, we sought
to determine whether (1) project officer reviews are sufficient to ensure effective
workplans, and (2) project officer responsibilities are defined in position descriptions
and covered in performance agreements.
Results of Review
Project officers did not perform all necessary steps when conducting pre-award
reviews of assistance agreement applications. Specifically, we noted instances of the
following problems:
A link was missing between projects funded and Agency mission.
EPA did not assess probability of success prior to award.
EPA did not determine reasonableness of proposed project costs.
Outcomes were not negotiated.
Milestones and deliverables were not included in workplans.
EPA did not implement new workplan regulations designed to improve its fiscal
management and accountability.
Consequently, there was insufficient assurance that assistance agreement projects
would accomplish program objectives or desired environmental results. There was also
insufficient assurance that proposed costs were reasonable, and that recipients were
technically capable of performing the work. EPA may have lost the opportunity to fund
other projects that would have better achieved its mission.
Project officers are responsible for ensuring Federal funds are protected and prudently
awarded. However, Agency leadership had not always emphasized the importance of
project officer duties, nor held project officers accountable for conducting complete
pre-award reviews. It is crucial that management create an environment that considers
the management of assistance agreements and the project officer function vital to the
Agency's mission.

-------
Recommendations
We recommend that the Assistant Administrator for the Office of Administration and
Resources Management make improvements in project officer training course curricula
and develop a critical job element to be included in performance standards. We also
recommend that the Assistant Administrator request program and regional officials to
include this critical job element in project officers' performance standards and track
compliance with this request.
Agency Comments and Office of Inspector General (OIG)
Evaluation
EPA agreed with the recommendations in the report and indicated steps they have
taken or plan to take in the coming months. These steps have been incorporated into
the Agency's Grants Management Strategic Plan and Tactical Plan. EPA indicated that
they have revised the Project Officer Training Manual and training program to better
focus on the core competencies needed to manage assistance agreements. EPA has
also initiated a new pilot grants management training program for managers and
supervisors. The training program emphasizes the manager's role in providing guidance
and direction to project officers on how to effectively manage assistance agreements.
The Agency stated it is going to do a further evaluation of the cited examples in the
report and will provide comments on the examples when it responds to the final report.
EPA raised questions about the limited sample on which the information in the report is
based. We provided additional information to clarify our sample selection and reasons
for limiting specific types of assistance agreements in Appendix A, "Details on Scope
and Methodology". EPA's response to our draft report is included in Appendix B.

-------
Table of C
Executive Summary	 i
Introduction 	 1
Purpose 	 1
Background 	 1
Scope and Methodology	 2
EPA Must Emphasize Importance of Pre-Award Reviews	 3
Pre-Award Reviews Not Sufficiently Performed	 3
Link Missing Between Projects Funded and Agency Mission		4
EPA Did Not Assess Probability of Project Success Prior to Award 		5
EPA Did Not Determine Reasonableness of Proposed Costs 		7
Outcomes Not Negotiated 		8
Milestones or Deliverables Not in Workplans		9
EPA Did Not Implement New Workplan Regulations 		9
Management Needs to Further Emphasize
Importance of Project Officer Role 	 10
Performance Management	 10
Project Officer Skills and Training 	 12
Recommendations	 13
Agency Comments and OIG Evaluation	13
Appendix 	 15
A Details on Scope and Methodology	 15
B Agency Response to Draft Report	17
C Distribution 	21
iii

-------

-------
Introduction
Purpose
The Office of Inspector General (OIG) has reported for 6 years that one of the key
management challenges for the Environmental Protection Agency (EPA) is its use of
assistance agreements to accomplish its mission. EPA requires project officers to
appraise assistance agreement applications and recommend funding the ones that will
most effectively contribute to EPA program objectives and priorities.
We conducted this audit to determine whether the process for awarding assistance
agreements resulted in projects that supported EPA's mission. Our objectives were to
answer the following questions:
Are project officer reviews sufficient to ensure effective workplans?
Are project officer responsibilities defined in position descriptions and covered in
performance agreements?
Background
Assistance agreements are the primary vehicle through which EPA delivers
environmental and human health protection. In fiscal 2001, EPA awarded $4.0 billion
in assistance agreements, accounting for more than half of the Agency's $7.8 billion
budget.
EPA project officers should play a key role in assuring the proper expenditure of
assistance agreement funds. EPA's Managing Your Financial Assistance
Agreement Project Officer Responsibilities, Fourth Edition, dated 2001 (the Project
Officer Manual), issued by the Office of Administration and Resources Management,
identifies specific project officer responsibilities for assistance agreements. Project
officer responsibilities include pre-award reviews.
A pre-award review includes the project officer's programmatic and technical review
of the application package and results in a funding recommendation. The project
officer's review is an appraisal to select those applications that will contribute most
effectively to EPA program objectives and priorities. The applicant's workplan serves
as a performance commitment and should describe: what will be done, when it will be
accomplished, performance capability, and estimated costs. The workplan is the main
focus of the programmatic and technical review conducted by project officers. Critical
project officer responsibilities, as detailed in the Project Officer Manual, include:
1

-------
Determine the relevance of the proposal to EPA objectives.
Determine the programmatic, technical, and/or scientific merit of the proposal.
Review the budget to ensure the costs are necessary and reasonable.
Identify the relationship to other proposals and on-going work.
Identify gaps between program objectives and intended activities.
An effective pre-award review should result in a workplan that lists expected outputs
and outcomes, links those outputs and outcomes to funding, and identifies target dates
and milestones.
Scope and Methodology
Our review focused on EPA's pre-award process for assistance agreements.
We selected a stratified, random sample of 116 assistance agreements awarded
nationwide by the Office of Water, Office of Air and Radiation, and related regional
offices. These agreements totaled approximately $42 million and were awarded to
States, tribes, non-profit organizations, local government agencies, and universities (see
table below). We performed our audit in accordance with Government Auditing
Standards, issued by the Comptroller General of the United States. We conducted
our field work from July 2002 to November 2002. We interviewed project officers
and division directors in each of the 10 regions and at Headquarters. We also
reviewed project officer files, position descriptions, and performance agreements.
Further details, including a breakdown by region, are in Appendix A.
Number of Assistance Agreements and Dollar Amounts Sampled, by Entity
Entities
No. of
Water
Agreements
Water
Dollar
Value
No. of
Air
Agreements
Air
Dollar
Value
Total
Agreements
Selected
Total
Dollar
Value
States
20
$23,654,833
14
$4,844,808
34
$28,499,641
Tribes
17
$1,929,328
12
$1,725,253
29
$3,654,581
Non-profit
Organizations
12
$2,618,456
14
$3,746,211
26
$6,364,667
Local Government
Agencies
4
$408,583
12
$2,205,664
16
$2,614,247
Universities
7
$381,919
4
$186,226
11
$568,145
Totals
60
$28,993,119
56
$12,708,162
116
$41,701,281
2

-------
EPA Must Emphasize Importance of
Pre-Award Reviews
Project officers did not perform all necessary steps when conducting pre-award
reviews of assistance agreement applications. Project officers did not conduct
complete appraisals of applications, nor negotiate required components into assistance
agreement workplans. Further, EPA did not sufficiently define project officer
responsibilities in performance agreements and position descriptions. These
deficiencies occurred because EPA management had not always communicated that
project officer functions are vital to achieving the Agency's mission. The Agency relies
on project officers to recommend projects that will contribute to the accomplishment of
EPA program objectives. Therefore, since the project officers did not conduct
adequate pre-award reviews, there was insufficient assurance that: assistance
agreement projects would accomplish program objectives or achieve environmental
results, agreements were being awarded at a reasonable price, and recipients were
capable of performing the work. EPA may have lost the opportunity to fund other
projects that would better achieve its mission.
Pre-Award Reviews Not Sufficiently Performed
EPA established guidelines for the pre-award process, including requirements for
conducting programmatic and technical reviews. We found that project officers did not
always perform the following required steps.
Relevance:
A link was missing between projects funded and Agency mission.
Programmatic/
technical merit:
EPA did not assess probability of project success prior to award.
Cost reviews:
EPA did not determine reasonableness of proposed project costs.
Outcomes:
Outcomes were not negotiated.
Milestones and
deliverables:
Milestones and deliverables were not included in workplans.
Workplan
improvements:
EPA did not implement new workplan regulations designed to
improve its fiscal management and accountability.
Based on a statistical sample, the following chart illustrates the frequency these steps
were performed.
3

-------
Percentage of Key Steps Completed
During Pre-Award Process
100
90
80
70
60
50
40
30
20
10
0
81
76
69
21
II
&
~	Relevance
¦	Program/Technical
Merit
¦	Cost Review
~	Outcomes
~	Milestones and
Deliverables
~	Workplan
Improvements
EPA developed numerous guidance documents detailing the purpose and importance of
the pre-award review process, and also provided training. However, pre-award
reviews were not always complete because management had not emphasized the
importance of the pre-award review process and did not hold project officers
accountable for their performance. We believe EPA management had not fostered an
environment that considered the management of assistance agreements and the project
officer function important. Based on a random sample of 116 assistance agreements,
we found the following specific problem areas.
Link Missing Between Projects Funded and Agency Mission
Project officers did not identify the link between the projects EPA funded and the
achievement of EPA's mission. In 19 percent of the assistance agreements we
reviewed (22 out of 116), project officers did not determine the relevance of the
proposed workplans to EPA program objectives. By not verifying relevance, project
officers did not ensure assistance agreements addressed EPA program priorities and
were for the projects of greatest urgency.
When determining relevance of proposals, project officers are instructed to:
Be aware of program priorities,
Consider how well proposed projects would help achieve program priorities,
4

-------
Recommend funding the projects that will contribute most effectively to EPA
program objectives and priorities, and
Document the link between the proj ects funded and EPA's mission.
Project Officer Negotiations Link EPA Objectives to Environmental Improvement
EPA objectives are
accomplished and
subsequent
environmental
improvement
attained
EPA establishes
objectives to attain
environmental
improvement
Assistance
agreements are
awarded and
recipients perform
planned work
Project officers
negotiate work
assistance agreement
recipients will do via
workplans
Although EPA requires project officers to ensure that funded projects contribute to
program priorities, this requirement was not always met. Examples follow:
Examples of Links Not Being Determined
EPA awarded $700,000 without knowledge of the work the recipient was going to
perform. Theworkplan did not have clear objectives, milestones, deliverables, or
outcomes. The recipient stated in the workplan: "Because of the exploratory nature of
these activities and the need to bring together various market players, exact deliverables
and schedule will be determined based on what participants tell us they want from our
project."
EPA awarded $110,000 to develop an Arkansas coastal plain guidebook, host a
workshop on the guidebook, and continue development of a website. The project officer
did not determine how this project would contribute to Office of Water program priorities.
It was unclear from the project file why EPA funded this agreement.
EPA awarded $165,000 to fund personnel and contracted water quality management
projects. EPA awarded this agreement without knowing what projects would be
performed or how they would help accomplish EPA program objectives. Also, the
recipient's budget included two full time employees, but there was no indication what
work they would perform. The project officer should have identified the vagueness of the
application and absence of workplan commitments.
Project officers, by determining relevance, provide assurance that funded projects help
achieve environmental improvements. Based on our random sample, we estimate that
EPA's Office of Water, Office of Air and Radiation, and related regional offices
awarded at least $42 million in fiscal 2001 without determining relevance. By funding
projects that may not accomplish EPA objectives, the benefits received from Federal
funds may be limited.
EPA Did Not Assess Probability of Project Success Prior to Award
Project officers did not always determine the programmatic, technical, and/or
scientific merit of assistance agreement applications prior to award. For
5

-------
31 percent of assistance agreements reviewed (36 out of 116), project officers did
not evaluate the feasibility of assistance agreement proposals. By not properly
assessing proposals, there is limited assurance that EPA's environmental goals and
program objectives will be met.
Project officers are responsible for assessing applicant qualifications and the integrity
of projects to ensure program objectives will be met and Federal funds spent
appropriately. EPA's Project Officer Manual instructs project officers to determine
the programmatic, technical, and/or scientific merit of proposals, but does not provide
details on how to complete reviews. EPA's Assistance Administration Manual
defines technical reviews as an assessment of the applicant's qualifications and the
soundness of the concepts and techniques to be applied to the project.
Project officers did not always assess the probability of a project's success prior to
award. Examples follow:
Examples of Probability of Success Not Being Assessed
EPA awarded $125,000 to a recipient to fund water permitting, monitoring, and
assessment of a river. However, the project officer did not assess the applicant's ability
to complete the project, and the recipient subsequently was unable to perform all the
planned activities. Nonetheless, the budget was not reduced. Had the project officer
assessed the recipient's qualifications, this project may not have been awarded, or the
recipient's scope of work and budget might have been revised.
EPA awarded a recipient an agreement to process travel vouchers for State employees
to attend a training course. The project officer did not conduct a programmatic
assessment to determine the appropriateness of the project. The recipient received a
$10,000 fee to process 27 travel vouchers. Had the project officer evaluated the
proposal, EPA may have either more appropriately contracted for this service, or
processed invitational travel in-house.
EPA awarded $263,000 to a recipient to assess indoor radon levels in schools and
homes. The project officer did not evaluate the qualifications of the recipient prior to
award. We determined from documentation in the project officer files that the recipient
was behind schedule and had also not completed work on a previously funded
assistance agreement. Had the project officer evaluated this recipient's qualifications,
this project may not have been funded.
Project officers must assess the programmatic, technical, and/or scientific merit of
proposals to ensure recipients are capable of performing well thought out projects
that support EPA's mission. Based on our random sample, we estimate that EPA's
Office of Water, Office of Air and Radiation, and related regional offices awarded at
least $88 million in fiscal 2001 without assurance that recipients were able to perform
projects that would help accomplish program objectives.
6

-------
EPA Did Not Determine Reasonableness of Proposed Costs
Project officers did not document cost reviews of proposed budgets for 79 percent
of applicable assistance agreements reviewed. Of the 116 agreements in our sample,
only the 62 valued at greater than $100,000 were required to have cost reviews
documented. Of that amount, we found that 49 did not have cost reviews performed.
Therefore, EPA risked the possibility of reimbursing recipients for costs that were
unreasonable, unallowable, or unrelated to agreed upon activities.
EPA's Project Officer Manual states project officers are responsible for determining
whether costs are eligible and reasonable. EPA's Cost Review Guidance states that
a cost review will be conducted for every project selected for funding. Both of these
documents provide detailed information on how to conduct a cost review. Project
officers are responsible for documenting details of their review, regardless of the
complexity and depth of the review.
The level of funding did not appear to affect whether project officers conducted cost
reviews, and such reviews were not conducted when appropriate. Examples follow:
Examples of Cost Reviews Not Being Conducted
EPA awarded one recipient over $9 million to operate its nonpoint source program, and
another recipient $1.3 million to operate its air pollution control program. Neither project
officer determined the reasonableness of proposed costs in relation to the expected
effort and benefits for these projects.
EPA awarded a recipient $150,000 to operate its air pollution control program without a
cost review. We reviewed the workplan budget and identified duplicative costs for
supplies. Had the project officer performed the required review, the unnecessary costs
could have been deleted.
EPA awarded a recipient $20,000 to operate its indoor air program without a cost review.
The recipient agreed to assess the extent and amount of radon in both public and
residential buildings. Though training was identified in the workplan budget, the project
officer did not determine the reasonableness of the training. The recipient used its
training funds to attend sexual harassment, savings plan, retirement, and first aid
training courses. A project officer's cost review should have identified these
unnecessary costs.1
Project officers play a key role in ensuring the proper expenditure of Federal
funds. Based on our random sample, we estimate that project officers
recommended awarding at least $536 million in fiscal 2001 by EPA's Office of
Water, Office of Air and Radiation, and related regional offices, without
1 Although the grant is below the $100,000 threshold for requiring documentation of the cost review, a
review of the budget is still required. The grant budget clearly identified costs that were unnecessary.
7

-------
performing cost reviews. As a result, EPA risked the possibility that Federal funds
would not be spent in the most efficient or effective manner.
Outcomes Not Negotiated
EPA's pre-award process provided limited assurance that assistance agreements
would ultimately contribute to environmental improvements (outcomes). An
outcome represents the results of an activity (output) in relation to its intended
purpose. Project officers did not negotiate environmental outcomes in 42 percent
of the assistance agreements reviewed (49 out of 116). Without outcomes, it is
not clear how the work will ultimately benefit the public or contribute to EPA's
mission.
Environmental outcomes are a tool to provide accountability and help assistance
agreement recipients achieve environmental results. Outcomes were omitted from
the workplans of 16 continuing environmental program recipients. These 16
recipients received a total of $5 million and are funded annually. Without planned
outcomes, the benefits received from continuing environmental programs are
coincidental.
Continuing environmental program recipients were not alone in lacking outcomes
in their workplans. Project officers did not negotiate outcomes in other assistance
agreement workplans as well. Example follow:
Examples of Outcomes Not Being Negotiated
EPA awarded a recipient $200,000 to regulate costs charged by power companies.
However, there were no environmental outcomes identified in the workplan. In fact, the
workplan itself only provided possible activities, and stated specific projects would be
established later. The project officer wrote on the application, "why this, why now?"
yet still approved the workplan. Without outcomes, it was not clear how these Federal
funds would benefit the public or contribute to EPA's mission.
Another workplan not containing outcomes was a $19,500 award to develop priority
watersheds in the Baltic Sea area in Europe and establish a volunteer lake monitoring
program through the local school systems. The project officer did not ensure outcomes
were included in the workplan and there was no indication in the project file as to why
EPA funded this overseas project.
Project officers are responsible for ensuring assistance agreements help
accomplish EPA's mission. Project officer pre-award reviews provided limited
assurance that assistance agreement projects resulted in environmental
improvements. EPA should fund projects with outcomes so that the usefulness
and success of assistance agreements can be measured.
8

-------
Milestones or Deliverables Not in Workplans
Project officers did not ensure that milestones or deliverables were included in
24 percent of the assistance agreement workplans reviewed (28 out of 116).
Project officers were responsible for negotiating EPA's expectations of what
recipients would accomplish and when, and then monitoring project progress via
the milestones or deliverables in workplans. Without agreed upon expectations,
EPA's ability to monitor progress and measure achievements was impaired.
EPA was unable to determine whether assistance agreements were accomplishing
program objectives when milestones or deliverables were absent. For example:
Examples of Milestones or Deliverables Not in Workplans
EPA awarded $368,000 to a State to protect wetlands, and project delays occurred.
The agreement was awarded even though the recipient did not submit a workplan. The
project officer was not aware of milestones, deliverables, or State responsibilities under
the agreement. If EPA would have established milestones and the recipient's
responsibilities, subsequent project delays could have possibly been avoided.
EPA awarded a recipient $165,000 for its water quality monitoring and protection
program. Though the recipient submitted a workplan, it did not identify milestones,
deliverables, or expected results. By funding this vague workplan, EPA was unable to
monitor progress, and had no knowledge of the project's expected outcomes or outputs.
EPA awarded an agreement to "investigate operational issues at specific wetlands."
The project officer recommended funding the $25,700 project even though the workplan
lacked milestones or deliverables. EPA later granted the recipient an extension to
complete workplan objectives. Had milestones or deliverables been established and
EPA monitored progress, the delay may have been prevented.
When project officers do not negotiate milestones or deliverables, their efforts to
monitor projects are hindered. Inadequately monitored agreements often require
additional time for completion, thus delaying environmental benefits. Milestones or
deliverables would allow EPA to better monitor progress, measure achievements,
and hold grantees accountable for performing planned activities.
EPA Did Not Implement New Workplan Regulations
EPA did not implement new regulations designed to improve its fiscal management
and accountability for environmental protection. Project officers did not ensure
workplans included a performance evaluation process and reporting schedule, nor
did they define and include the roles and responsibilities of the recipient and EPA
as required by 40 Code of Federal Regulations, Part 35. This regulation also
requires that a process be established for jointly evaluating and reporting progress
and accomplishments under a workplan.
9

-------
Of the 116 assistance agreements in our sample, the new regulation only applied to
programs related to 53 agreements. Our review of the 53 disclosed that only 14
workplans contained information on how EPA and the recipient will jointly
evaluate performance, and only 11 documented roles and responsibilities of both
the recipient and EPA pertaining to the completion of the workplan commitments.
Only 2 workplans of the 53 agreements contained both elements. Therefore, for
96 percent of the applicable workplans (51 out of 53), project officers did not
ensure the required elements were included in negotiated workplans so that roles
and responsibilities were clear and adjustments could be made when project
expectations were not being met. By not implementing these required
improvements, EPA limited its ability to ensure sound fiscal management and
accountability for environmental performance.
Management Needs to Further Emphasize
Importance of Project Officer Role
Project officer pre-award reviews were insufficient because Agency leadership
had not effectively emphasized the importance of project officer duties. Project
officers were not held accountable for completing key pre-award steps prior to
recommending that assistance agreements be funded; there were also no
consequences when required duties were not performed. Though the Agency
provided training, it had not ensured project officers conducted complete
programmatic and technical reviews of assistance agreements. EPA did not
clearly define project officer responsibilities in performance agreements and
position descriptions. EPA also had not identified the skills and abilities needed
for a project officer. Since EPA awards nearly $4.5 billion in assistance
agreements annually, it is important that EPA stress the significance of the project
officer role and consider the function critical to the accomplishment of the
Agency's mission.
The fact that improvements are needed in the areas of performance management,
skills, and training is not unique to the project officer function. In its 2001 Annual
Report, EPA identified the implementation of its Human Capital Strategy as a
major management challenge.
Performance Management
EPA management uses performance agreements as a tool to communicate job
expectations and hold individuals accountable for performing required duties. We
reviewed performance agreements and position descriptions for the 92 project
officers in our sample. We found that 15 percent of performance agreements and
30 percent of position descriptions did not identify that the individual was a
project officer (the project officer function is often a collateral duty). Although
the project officer role was mentioned in many performance agreements, the
10

-------
following chart notes the percentage of performance agreements that included
specific duties.
Percentage of Performance Agreements
Officer Duties
~ Determine
Program/Technical
Merit
¦ Negotiate
Workplans
¦ Conduct Cost
Reviews
~ Prepare Decision
Package
In addition, there was no evidence of consequences when project officers did not
complete important steps in the pre-award process. We asked the Agency to
identify whether, in the last 2 years, there were any actions taken for poor
performance or improper conduct related to project officer responsibilities. The
Agency did not identify any actions.
To better emphasize the importance of recommending and awarding assistance
agreements, EPA needs to develop a uniform performance standard for its project
officers. The General Accounting Office (GAO), in its model for strategic capital
management, provides some insight into how EPA can link individual performance
to organizational goals. GAO, in explaining the model, states:
High-performing organizations design and implement
performance management systems that further
cascade accountability for results to managers and
front-line employees. These systems define individual
accountability by setting expectations so staff
understand how their daily activities contribute to
results-orientated programmatic goals.
GAO recognizes that shifting the orientation of individual performance
expectations and accountability systems from an adherence to processes and the
That Included Specific Project
y
.X 68

it
./
, r

P
40

15


	1
*
11

-------
completion of activities to a greater focus on results will require a cultural
transformation for most Federal agencies. To better link individual performance
with organizational goals, GAO suggests that agencies define and communicate to
employees the agency's mission, core values, strategic goals and objectives, and
business strategies. The agency's performance management and incentive
systems should then be designed and tested to make employees aware of their
roles and responsibilities in helping the agency achieve its performance goals.
EPA has taken steps to define and communicate to employees its mission,
strategies, and goals. However, EPA still needs to take the next step to design
and test performance agreements that show how project officer responsibilities
contribute to organizational goals. A uniform performance standard should stress
that a complete pre-award review is needed to ensure that selected projects
contribute most effectively to EPA goals.
Project Officer Skills and Training
EPA used training to ensure that staff had the skills needed to manage assistance
agreements. While all project officers attended the training, the training did not
always explain how to accomplish key project officer responsibilities or identify
skills and abilities project officers need to manage assistance agreements.
The Project Officer Manual, which is the basis of the project officer training
course and a primary reference document, states that project officers must
determine the programmatic, technical, and/or scientific merit of workplans.
However, there was no explanation on how to complete these reviews.
Additionally, as noted, the Project Officer Manual did not reflect the new Title
40 Code of Federal Regulations, Part 35 requirements regarding workplans.
EPA requires project officers to complete a refresher training course every
3 years. The refresher training did not identify that milestones, deliverables, and
the roles and responsibilities of the recipient and EPA are necessary components
of an approved workplan. Also, the refresher training did not provide instructions
on how project officers should determine the relevance of workplans to EPA
objectives, or the programmatic, technical, and/or scientific merit of workplans.
In the draft Grants Management Strategic Plan, dated November 14, 2002,
strategic goal 1 is to enhance the skills of EPA personnel involved in grants
management. The Strategic Plan states:
EPA must have a skilled work force of grants specialists
and project officers to manage its grants and ensure
that overall Agency goals are being addressed. Given
the emphasis on grants oversight, the Agency must
supplement the traditional skill set for grants specialists
12

-------
with a new competency centered on the business aspect of grants
management.
Though the Strategic Plan acknowledges the importance of having a skilled work
force of project officers, the objectives in the document only addressed the
enhancement and improvement of training; the Strategic Plan did not address the
skills and abilities needed by project officers to perform their duties. Without
defining these skills and abilities, EPA is unable to determine whether people
managing assistance agreements have the necessary skills.
EPA's Strategy for Human Capital emphasizes the need to link employee
development to mission needs. The strategy calls for EPA to (a) develop an
employee retraining plan to ensure that the current work force possesses the
necessary skills to meet current and future work challenges, and (b) strategically
plan for training to tailor development to the organization's strategic needs. EPA
needs to apply this strategy to the project officer function. EPA should identify the
skills project officers need to adequately manage assistance agreements, and then
design training to ensure that project officers have the necessary skills.
Recommendations
We recommend the Assistant Administrator for the Office of Administration and
Resources Management:
1.	Identify skills project officers need to perform pre-award reviews, and
evaluate and modify their training to ensure staff have the needed skills.
2.	Develop a uniform performance standard that clearly communicates key
project officer responsibilities critical to accomplishing EPA's mission.
3.	Update the Project Officer Manual and refresher training course to reflect all
requirements as well as instructions on how to implement them.
4.	Request national program and regional officials to include the uniform critical
job element in project officers' performance standards and track compliance
with this request.
Agency Comments and OIG Evaluation
EPA agreed with the recommendations and has taken or will take the following
actions as reflected in the Grants Management Strategic and Tactical Plan.
The Office of Grants and Debarment (OGD) will issue the 5th Edition of the
Project Officer Training Manual that will focus on core competencies needed
13

-------
to manage grants. The Manual will be enhanced to include additional materials on
pre-award review and application procedures, budget, cost and procurement
review, grants competition, environmental results in workplans, and use of the
Federal Audit Clearinghouse (March 30, 2003).
•	OGD, Regional Offices, and Headquarters Program Offices will conduct project
officer training with special emphasis on the core competency areas that were
enhanced in the 5th edition of the Training Manual (starting in May 2003).
•	OGD will issue a grants policy guidance document defining roles and
responsibilities of EPA Grants Management Officers, Program Office officials, and
project officers (2003).
OGD will be working with the Office of Human Resources and Organizational
Services and EPA=s Headquarters and Regional Offices to ensure that performance
standards of project officers adequately address their grants management
responsibilities. Compliance with this request will be determined through the new
comprehensive Grants Management Reviews of EPA Headquarters and Regional
Offices (2004).
As described in the Strategic Plan, EPA is taking several additional actions that
complement the recommendations in the draft report. For example, the OGD has
developed a pilot grants management training program that will emphasize the
managers role in providing guidance and direction to their project officers on how to
effectively manage assistance agreements. OGD will issue guidance to ensure that
grant workplans, decision memoranda, and/or terms and conditions of the grant
address environmental outcomes and how to measure them. Also, new policy
documents will require reporting on environmental outcomes as a criterion for
approval of interim and final reports and require that grantee success in reporting on
environmental outcomes becomes a factor in awarding new grants. Finally, the
Agency's new competition policy, which was issued in September 2002, requires
detailed pre-award review of competing proposals and applications.
The Agency planned actions, when implemented, should address the findings and
recommendations in the report.
14

-------
Appendix A
Details on Scope and Methodology
We reviewed EPA's Office of Water (OW), Office of Air and Radiation (OAR), and related regional
offices' pre-award process for assistance agreements. The decision to limit our review to two
program offices, air and water, and exclude air and water grants awarded by Office of Research and
Development, was made at the suggestion of the Director for the Grants Administration Division. The
Director was concerned that the results from sampling grants awarded by all program offices might not
be comparable. The Director believed that the results from reviewing grants awarded by two program
offices and the related regional offices would be sufficient to make any necessary systemic changes.
We obtained a universe of assistance agreements awarded in fiscal 2001 by OW, OAR, and related
regional offices, from the Grants Information Control System. From the assistance agreements
awarded by OW, and related regional offices, we excluded State Revolving Fund grants, construction
grants, and fellowship grants. The State Revolving Funds are unique because they are State-run
programs. Each State decides how it will operate its program and which projects to fund. In addition,
the State Revolving Fund capitalization grants are based on a formula specified in Section 205(c)(3) of
the Clean Water Act. We excluded construction grants because most are "earmarks" mandated by
Congress. Finally, we excluded fellowship grants because these funds were for individual education
rather than for the implementation of environmental programs. After our exclusions, the grant universe
used for this review was valued at approximately $1 billion, and consisted of $520 million awarded by
OW and $485 million awarded by OAR. Our universe is about 25 percent of the total $4 billion
awarded in fiscal 2001.
We stratified this universe by awarding office and dollar amount, separating small ($100,000 and less)
and large (greater than $100,000) assistance agreements. This provided us with four strata (OW
small, OW large, OAR small, and OAR large). We sampled 116 assistance agreements totaling $42
million. Ninety two project officers oversaw the 116 agreements. We projected our results using the
GAO Stats software program and a 95 percent confidence level. We reported the most conservative
estimate of errors by using the low dollar amount of the projections. A breakdown of the assistance
agreements reviewed by Region and at the Headquarters level follows:
Number and Type of Assistance Agreements Sampled, by Location
Sample
Type
Regio
n 1
Regio
n 2
Regio
n 3
Regio
n 4
Regio
n 5
Regio
n 6
Regio
n 7
Regio
n 8
Regio
n 9
Regio
n 10
HQ
Total
V\feter
1
2
4
7
10
5
5
4
12
5
5
60
Air
3
3
2
3
5
7
1
7
9
6
10
56
Totals
4
5
6
10
15
12
6
11
21
11
15
116
15

-------
We obtained project officer files for each sampled assistance agreement and verified whether the
programmatic, technical and cost reviews were conducted in accordance with Agency guidance. We
reviewed assistance agreement workplans and determined whether they contained elements required
by EPA guidance and/or Federal regulation. We obtained position descriptions and performance
agreements from the 92 project officers and determined whether project officer responsibilities were
identified. We also contacted training officers to obtain training certification dates for each project
officer and assessed compliance with training requirements. We judgmentally selected and interviewed
OW and OAR project officers and division directors in each region and at Headquarters. We
discussed project officer roles and responsibilities during the pre-award process, from the point of
application until the award was made.
We conducted our audit field work from July 2002 to November 2002. We performed this audit in
accordance with Government Auditing Standards, issued by the Comptroller General of the United
States, as they apply to program audits. We reviewed management controls and procedures
specifically related to our objectives. We did not review the internal controls associated with the input
and processing of information in EPA's Grants Information Control System.
Prior Audit Coverage of the Assistance Agreement Area
•	Additional Efforts Needed to Improve EPA's Oversight of Assistance Agreements,
EPA OIG Report No. 2002-P-00018, September 30, 2002.
•	Procurements Made by Assistance Agreement Recipients Should Be Competitive,
EPA OIG Report No. 2002-P-00009, March 28, 2002.
•	Surveys, Studies, Investigations, and Special Purpose Grants,
EPA OIG Report No. 2002-P-00005, March 21, 2002.
•	EPA '.s Competitive Practices for Assistance Awards,
EPA OIG Report No. 2001-P-00008, May 21, 2001.
16

-------
Appendix B
Agency Response to Draft Report
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
ADMINISTRATION
AND RESOURCES
MANAGEMENT
March 21, 2003
MEMORANDUM
SUBJECT:	Response to Office of Inspector General Draft Audit Report, AEPA Must Emphasize Importance of
Pre-Award Reviews for Assistance Agreements®
FROM:	Morris X. Winn /s/
Assistant Administrator
T O:	Michael A. Rickey
Director for Assistance Agreement Audits
Thank you for the opportunity to provide comments on the Office of Inspector General=s (OIG) draft audit
report (draft report) entitled AEPA Must Emphasize Importance of Pre-Award Reviews for Assistance Agreements,®
dated February 10, 2003. The draft report concludes that EPA project officers did not perform all necessary steps when
conducting pre-award reviews of assistance agreement applications. The draft report was shared with the relevant EPA
Headquarters and Regional Offices and this response reflects their comments and conclusions.
The draft audit reviewed a sample of 116 Air and Water assistance agreements totaling $42 million from a
universe of $1 billion of such agreements. The sample excludes several grant programs including State Revolving Fund
grants, fellowship grants and grants awarded by the Office of Research and Development totaling more than $2.2 billion,
which have aggressive pre- award and post-award review systems. It is unclear why these grant programs, which
represent almost one-half of the Agency=s assistance dollars, were excluded. As a result, the draft report leaves the
impression that the findings apply to the entire $4.5 billion EPA grant universe even though this is not the case.
In addition, the Agency was not able to determine the accuracy of the discussions related to the 14 examples
cited in the draft report. The Office of Administration and Resources Management has requested that the OIG provide
the grants numbers for the examples to make it easier to assess the accuracy of the information. In order not to delay
this response, the Agency will provide comments on the examples when it responds to the final audit.
Note: The original response was signed by Morris X. Winn.
17

-------
EPA has recognized the importance of the issues raised in the audit for some time. As a result, the
Agency has already begun to address the recommendations proposed in the audit. EPA has revised
its Project Officer Training Manual and training program to better focus them on the core
competencies needed to manage assistance agreements. The Manual and training have been
enhanced to include additional materials on pre-award review and application procedures, budget,
cost and procurement review, grants competition and environmental results. EPA has also initiated
a new pilot grants management training program for managers and supervisors. The training
program emphasizes the managers role in providing guidance and direction to their project officers
on how to effectively manage assistance agreements.
EPA agrees with the recommendations in the report and has indicated in the attachment the
steps we have taken or plan to take in the coming months to carry them out. These actions have
been incorporated into the Agency=s Grants Management Strategic Plan (Strategic Plan) and
Tactical Action Plan. The Agency believes that these efforts will ensure that assistance agreements
accomplish program objectives and enable the Agency to achieve its environmental missions.
Again, thank you for the opportunity to comment on the report. I have attached several
specific comments on the recommendations. If you have any questions please contact Howard
Corcoran, Director, Office of Grants and Debarment at (202) 564-1903.
Attachment
cc: Nikki Tinsley
Melissa Heist
Kathy Finazzo
Assistant Administrators
Regional Administrators
Dave 0=Connor
Howard Corcoran
Richard Kuhlman
Pat Patterson
Senior Resource Officials
Sherry Kaschak
Grants Management Officers
2
18

-------
Attachment
COMMENTS ON RECOMMENDATIONS
EPA Must Emphasize Importance of Pre-Award Reviews for Assistance Agreements
Recommendation 1: Identify skills project officers need to perform pre-award reviews, and
evaluate and modify their training to ensure staff have the needed skills.
Recommendation 3: Update the Project Officer Manual and refresher training course to reflect all
requirements as well as instructions on how to implement them.
Agency Response: The Agency agrees with these recommendations and has taken or will take
the following actions as reflected in the Grants Management Strategic and Tactical Plan to
implement them:
•	The Office of Grants and Debarment (OGD) will issue the 5th Edition of Project
Officer Training Manual that will focus on core competencies needed to manage
grants. The Manual will be enhanced to include additional materials on pre-award
review and application procedures, budget, cost and procurement review, grants
competition, environmental results in workplans, and use of the Federal Audit
Clearinghouse (March 30, 2003).
•	OGD, Regional Offices, and Headquarters Program Offices will conduct project
officer training with special emphasis on the core competency areas that were
enhanced in the 5th edition of the Training Manual (starting in May 2003).
Recommendation 2: Develop a uniform performance standard that clearly communicates key
project officer responsibilities critical to accomplishing EPA=s mission.
Recommendation 4: Request national program and regional officials to include the uniform critical
job element in project officers: performance standards and track compliance with this request.
Agency Response: The Agency agrees with these recommendations and is taking the following
actions:
1
19

-------
•	OGD will issue a grants policy guidance document defining roles and responsibilities
of EPA Grants Management Officers, Program Office officials, and project officers
(2003).
•	OGD will be working with the Office of Human Resources and Organizational
Services and EPA=s Headquarters and Regional Offices to ensure that Performance
Standards of project officers adequately address their grants management
responsibilities. Compliance with this request will be determined through the new
comprehensive Grants Management Reviews of EPA Headquarters and Regional
Offices (2004).
ADDITIONAL ACTIONS:
As described in the Strategic Plan, the Agency is taking several additional actions that
complement the recommendations in the draft audit. These action steps should be referenced in the
draft audit, as described below.
•	OGD has developed a pilot grants management training program for managers and
supervisors and has conducted training in 2002/2003. Scheduled classroom
training will begin in 2004 and online training in 2005. The training program will
emphasize the managers role in providing guidance and direction to their project
officers on how to effectively manage assistance agreements.
•	The Agency=s new competition policy, which was issued in September 2002,
requires detailed pre-award review of competing proposals and applications.
•	OGD will issue guidance to ensure that grant workplans, decision memoranda
and/or terms and conditions address environmental outcomes and how to measure
them (2003).
•	OGD will issue policy documents requiring reporting on environmental outcomes as
a criterion for approval of interim and final reports and requiring that grantee success
in reporting on environmental outcomes become a factor in awarding new grants
(2005).
2
20

-------
Appendix C
Distribution
EPA Headquarters
Assistant Administrator for Office of Administration and Resources Management
Assistant Administrator for Office of Air and Radiation
Assistant Administrator for Office of Water
Comptroller
Agency Followup Official
Agency Audit Followup Coordinator
Audit Followup Coordinator, Office of Administration and Resources
Management
Audit Followup Coordinator, Office of Air and Radiation
Audit Followup Coordinator, Office of Water
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Communications, Education, and Media Relations
Director, Office of Regional Operations
Director, Office of Grants and Debarment
Director, Grants Administration Division
Regional Offices
Regional Administrators, Regions 1 through 10
Audit Followup Coordinator, Region 1 through 10
Office of Inspector General
Inspector General
21

-------