¦ A v
I ^1V / OFFICE OF INSPECTOR GENERAL
\; "
-------
Report Contributors:
Erin Barnes-Weaver
Katie Butler
Ben Webster
Abbreviations
EPA Environmental Protection Agency
GAO Government Accountability Office
GPRA Government Performance and Results Act
OARM Office of Administration and Resources Management
OIG Office of Inspector General
-------
,^osr^
vV^
y*, ^
PRO^
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
2005-P-00016
June 2, 2005
Why We Did This Review
In 2005, EPA's Office of
Administration and Resources
Management introduced new
policies to improve EPA's
ability to demonstrate grant
results. We sought to
determine how these policies
compare with techniques used
by leading nongovernmental
organizations.
Background
EPA historically faced
challenges demonstrating
grant program impacts on
human health and the
environment. As recently as
2004, Congress, the
Government Accountability
Office, and the EPA Office of
Inspector General expressed
concerns about the Agency's
ability to demonstrate the
results of the $4 billion per
year it grants to States,
localities, tribes, nonprofits,
and other organizations.
For further information,
contact our Office of
Congressional and Public
Liaison at (202) 566-2391.
To view the full report,
click on the following link:
www.epa.aov/oia/reports/2005/
20050602-2005-P-00016.pdf
Catalyst for Improving the Environment
EPA's Efforts to Demonstrate Grant Results Mirror
Nongovernmental Organizations' Practices
What We Found
EPA recently took steps to improve its ability to demonstrate results from grants.
In 2005, EPA instituted Results and Pre-Award policies intended to (1) ensure
clear links between grant results and EPA goals, and (2) enhance oversight of
grantee qualifications and performance. We found the practices required by these
policies generally consistent with practices of leading nongovernmental
organizations that fund environmental projects and emphasize grantee
performance measurement. We identified nongovernmental organization
techniques that EPA could consider to augment its policies.
What We Suggest
We suggest that EPA:
¦ Track implementation of the Results and Pre-Award Policies to ensure that
EPA staff and grantees follow the policies and better demonstrate grant results.
¦ Adopt the technique of providing sample logic models that lead grant
applicants toward established environmental and human health improvement
goals.
¦ Consider providing an online resource for grantees that provides training,
examples, and a question and answer bulletin board, as recommended by the
Office of Water. This one-stop resource could provide grantees with extensive
information about how to meet results reporting requirements.
¦ In order to ensure that grant dollars fund projects with a high chance of
success, EPA could include grantees" past performance as a ranking criterion
when competing discretionary grants and selecting successful applicants.
¦ Conduct a retrospective evaluation of a sample of EPA grant results to provide
the Agency with valuable information about how grant dollars contributed in
the past to environmental and human health improvements.
EPA agreed with our suggestions, but recommended conducting a retrospective
evaluation in 3 or 4 years, once the new results policy has been implemented,
rather than engaging in a retrospective evaluation now.
-------
o* sp
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
I SaTlZ ? WASHINGTON, D.C. 20460
V 5*
PRO*^
OFFICE OF
INSPECTOR GENERAL
June 2, 2005
MEMORANDUM
SUBJECT: Evaluation Report: EPA's Efforts to Demonstrate Grant Results Mirror
Nongovernmental Organizations' Practices
Report No. 2005-P-00016
FROM: Jeffrey Harris /s/
Director, Cross Media Issues
TO: Luis Luna
Assistant Administrator
Office of Administration and Resources Management (OARM)
This memorandum transmits our final evaluation report titled EPA's Efforts to Demonstrate
Grant Results Mirror Nongovernmental Organizations' Practices conducted by the Office of
Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). EPA agreed with
the suggestions contained in the draft report, and the OIG did not substantially change the final
report. Therefore, this transmittal memorandum represents the final close-out for this evaluation.
We have no objection to the further release of this report to the public. For your convenience,
this report will be available at http://www.epa.gov/oig.
We appreciate the cooperation and assistance provided by your staff in completing this work. If
you or your staff has any questions regarding this report, please contact me at 202-566-0831 or
Erin Barnes-Weaver at 303-312-6871.
Attachment
cc: Sandy Womack, Audit Followup Coordinator, OARM
David O'Connor, Deputy Administrator, OARM
Howard Corcoran, Director, Office of Grants and Debarment, OARM
Mike Weckesser, Associate Director, Resource Management Staff, Office of Water
-------
Table of Contents
At a Glance
Introduction 1
Grant-Making at EPA 1
Congressional Testimony on EPA Grant Results 2
EPA Took Actions to Better Demonstrate Results 3
Purpose of This Evaluation 4
Scope and Methodology 4
Comparison of EPA and Nongovernmental
Organization Practices 5
Testimony Response 5
Additional Considerations 9
Conclusions 10
Summary of Suggestions and EPA Response 10
Appendices 12
A Process Flow for Competitive Grants 12
B Agency Response to Draft Report 13
C Distribution 16
-------
Introduction
The United States Environmental Protection Agency (EPA) historically faced challenges
demonstrating the impact its grant programs make on improving environmental and
human health. Guided by a series of recommendations from recent congressional
testimony, we compared EPA grant management practices to those used by a sample of
nongovernmental organizations. In many instances, EPA demonstrates practices
currently used by nongovernmental organizations. However, the nongovernmental
organizations we assessed use additional techniques that EPA could consider to better
measure grants effectiveness.
Grant-Making at EPA
EPA awards over one-half of its annual budget in grants to its State, local, tribal,
educational, and nonprofit partners. From fiscal years 1997 through 2003, EPA awarded
an average of $4,165,853,900 per year to governments, educational institutions, and
nonprofit organizations so that they could help EPA achieve its goals. As Figure 1
demonstrates, EPA awarded most of the funds (88 percent) to State, local, and tribal
governments. EPA awarded 11 percent of its grant funds to educational institutions and
nonprofit organizations.
Figure 1:
Distribution of Awarded Funds by Recipient Type
(Average Per Year for Fiscal Years 1997 - 2003)
Nonprofits
7%
Educational
Institutions
4%
Local Governments
9%
Other
State Governments
77%
Source: EPA
1
-------
EPA divides its grant management responsibilities between (1) program offices that
manage grant agreements and outcomes; and (2) the Office of Administration and
Resources Management (OARM), which manages administrative requirements associated
with EPA's grant programs. The table below depicts major responsibilities of program
offices and OARM with respect to grant management.
Grant Management Responsibilities at EPA
Program Offices
Oversees grant management and implementation, including:
¦ Writing requests for proposals;
¦ Agreeing with grantees on statements of work;
¦ Making determinations on programmatic special conditions;
¦ Reviewing work plans (and noting environmental results);
¦ Determining how closely EPA will work with recipients; and
¦ Reviewing grantees' interim and final technical reports.
OARM
Manages administrative requirements, including:
¦ Awarding grant funds to grantees chosen by program offices;
¦ Writing national policy related to grant administration; and
¦ Overseeing grantees and program offices to ensure grantees and
EPA staff follow orders, guidance, and procedures.
See Appendix A for a grant process flow chart describing how EPA issues competitive
grants.
Congressional Testimony on EPA Grant Results
Testimony before the United States House of Representatives on July 20, 2004,1
indicated that EPA faced challenges managing grants and, in particular, demonstrating
results from grants. During its testimony, the EPA Office of Inspector General (OIG)
testified that EPA project officers identified neither expected results nor a means of
measuring whether grant documents demonstrated results, and also did not include an
environmental performance measurement component when establishing new assistance
programs. The Government Accountability Office (GAO) testified that EPA did not
ensure that it linked awarded grants with outcomes in grant work plans. GAO reviewed
seven grant programs and found that an average of 69 percent of work plans did not
(1) Statement of John B. Stephenson, Director, Natural Resources and the Environment, Government Accountability Office, before the
Subcommittee on Water Resources and Environment, Committee on Transportation and Infrastructure, U.S. House of Representatives,
Grants Management: EPA Continues to Have Problems Linking Grants to Environmental Results.
(2) Statement of Nikki L. Tinsley, Inspector General, EPA, before the Subcommittee on Water Resources and Environment, Committee
on Transportation and Infrastructure, U.S. House of Representatives.
(3) Statement of David J. O'Connor, Acting Assistant Administrator for Administration and Resources Management, EPA, before the
Subcommittee on Water Resources and Environment, Committee on Transportation and Infrastructure, U.S. House of Representatives.
(4) Testimony of A. Alan Mohgissi, President, Institute of Regulatory Science, before the Subcommittee on Water Resources and
Environment, Committee on Transportation and Infrastructure, U.S. House of Representatives.
(5) Testimony of Peter Maggiore, Commission on Assessments and Reviews, before the Subcommittee on Water Resources and
Environment, Committee on Transportation and Infrastructure, U.S. House of Representatives.
2
-------
include outcomes. GAO also reported that the Office of Management and Budget
reviewed 10 grant programs and found that eight did not demonstrate results.
GAO, EPA OIG, and others testifying recommended the following additional steps that
OARM should take to enhance grant results management:
1. Develop outcome-based goals and measures for grant programs and demonstrate
the impacts of programs on human health and the environment.
2. Require grantees to submit work plans describing how grantees would achieve
measurable environmental results, including methods for measuring the
environmental results of funded projects or programs.
3. Develop a process for evaluating qualifications of investigators and institutions
seeking grants.
4. Train both EPA personnel and grantees on how to implement EPA's January 2005
grant results policy (described below) and how to better cooperate with each
other.
EPA Took Actions to Better Demonstrate Results
In its testimony, OARM described its plans to solve the problems raised during the
testimony. OARM said it would encourage strong senior leadership, effective
communication, and enforcement of new grants policies and procedures through internal
reviews and performance evaluations. OARM also noted a forthcoming policy that it
believed would help solve grant management problems:
EPA Order 5700.7: Environmental Results under EPA Assistance Agreements
(Results Policy), effective January 1, 2005: This established EPA's policy for
addressing environmental results under EPA assistance agreements, including grants
and cooperative agreements. The policy required program offices that administer
grants to (1) link proposed grants to EPA strategic goals and Government
Performance and Results Act (GPRA) plans; (2) ensure that grantee plans and reports
address outputs and outcomes; and (3) consider how results contribute to EPA goals
and objectives.
EPA subsequently issued a second policy designed to improve the way it evaluated
potential grantees:
EPA Order 5700.8: EPA Policy on Assessing Capabilities of Nonprofit
Applicants for Managing Assistance Awards (Pre-Award Policy), effective
March 31, 2005: This established internal controls for determining the capability of
nonprofit organizations applying for EPA assistance agreements to carry out planned
activities. This policy also enhanced post-award oversight by requiring EPA grants
officials to take appropriate remedies against nonprofit grant recipients who fail to
comply with the terms and conditions of the grants.
3
-------
Purpose of This Evaluation
We conducted this evaluation to address recurrent congressional concerns about how
effectively the results of EPA grants contribute to achieving the Agency's goals. We
specifically sought to determine: "how do EPA grant results policies compare with
techniques used in leading nongovernmental organizations?"
Scope and Methodology
To answer the question listed above, we interviewed EPA officials to determine how the
Agency responded to the issues raised during congressional testimony. We compared
EPA policies and practices with the techniques used by six leading nongovernmental
organizations. We chose nongovernmental organizations for assessment based on the
recommendations of other organizations.2 We based our sample on organizations'
familiarity with funding environmental projects, and their interest in measuring the
results of their funding activities. We limited the scope of this project to six
organizations, ranging from a small, nonprofit venture-philanthropy fund started in 2001
to a large, multi-billion dollar foundation started in 1930. We assessed how these
organizations demonstrated grant and investment results by searching printed materials
supplied by the organizations (e.g., annual reports), surveying information posted on
organizational Web sites, and conducting telephone interviews with individuals
responsible for monitoring grant results.
To determine the practicability of adapting these organizations' practices to EPA's grant
management process, we met with EPA officials and discussed the feasibility of
implementing suggested practices. We noted that EPA may be more limited in its ability
to control grant requirements than nongovernmental organizations, especially with the
grant funds given to State, local, and tribal governments each year (an average of 88
percent).
We performed our field work in February and March of 2005. We conducted our
evaluation in accordance with Government Auditing Standards issued by the Comptroller
General of the United States.
2
This method, "snowball sampling," relies on previously identified members of a group to identify other members of the population. As newly
identified members name others, the sample snowballs in size. (Fink, Arlene. 1995. How to Sample in Surveys. Thousand Oaks, Calif.: Sage
Publications. P. 70)
4
-------
Comparison of EPA and Nongovernmental
Organization Practices
Testimony Response
EPA took actions intended to solve the problems raised in the congressional testimony by
issuing the Results and Pre-Award Policies, and taking other actions described below.
We found that nongovernmental organizations used similar techniques to ensure that their
grantees contributed to organizational goals.
We summarized both the actions EPA took in response to the congressional testimony, as
well as practices used by a sample of nongovernmental organizations. We compared
EPA and nongovernmental organizations' practices and offered suggestions for EPA,
where applicable.
Testimony
Suggestion 1
Goals: Develop aoals and measures for all arant proarams that are outcome-
based and demonstrate the impacts of programs on human health and the
environment.
EPA's Actions
EPA outlines program goals in its annual GPRA plan. The Results Policy requires
program offices that administer grants to (1) link proposed grants to EPA strategic
goals and GPRA plans, (2) ensure that grantees address outputs and outcomes in
plans and reports, and (3) consider how results contribute to EPA goals and
objectives. The Results Policy requires each program office to determine
"significant results" to which grantees should contribute.
Nongovernmental
Organization
Practices
Each of the organizations we assessed described clear organizational goals. Three
organizations required that grant applications tier to organizational goals in order to
receive approval. One organization emphasized logic modeling in grant
applications to demonstrate that applicants considered connections between inputs,
planned program activities, anticipated outputs, and anticipated outcomes.
OIG Assessment
EPA and nongovernmental organizations both developed policies to address this
issue. In order to ensure that grantees clearly understand how to demonstrate
results, we suggest that EPA adopt the additional technique of providing sample
logic models that lead applicants toward established program goals.
5
-------
Testimony
Suggestion 2
Methods and Work Plans: Reauire arantees to submit work plans describina
how grantees would achieve measurable environmental results, including
methods for measuring the environmental results of the funded projects or
programs.
EPA's Actions
The Results Policy requires program offices that administer grants to ensure that
grantees addressed outputs and outcomes in grant plans. EPA provided
training/workshops in Headquarters and in Regions on implementing the Results
Policy. OARM ultimately expected program offices to provide their own program-
specific training on their respective environmental outcomes. If grantees do not
achieve their expected outcomes, the Results Policy required them to provide
reasons they did not succeed. OARM said that training associated with
implementing the Results Policy would focus on carefully negotiating grantee work
plans - including ensuring that grantees use appropriate methods to assess
outcomes.
Nongovernmental
Organization
Practices
The organizations we assessed required a range of levels of detail in work plans or
for agreed-upon projects. All organizations required recipients and/or applicants to
provide anticipated results. In some cases, organizations asked for basic
anticipated outcomes and performance measures. Other organizations required
more detail about desired results and performance measurement systems. They
offered specific directions and examples about how applicants should outline
potential results in grant applications, or worked with grantees to develop
appropriate results and measurement systems. One organization specified that its
recipients must engage in performance measurement throughout the project, or
face consideration as "in breach of the terms" of the grant or loan. This
organization developed a quarterly, forced-ranking system for evaluating investment
results on the basis of their current levels of success as defined through their
performance metrics.
OIG Assessment
Both EPA and nongovernmental organizations required grantees to demonstrate
their expected outcomes, and report on their success in work plans. In order to
ensure that EPA's actions address this issue, we suggest that EPA track
implementation of the Results Policy to ensure that its implementation corrects the
problems raised in the congressional testimony.
6
-------
Testimony
Suggestion 3
Grantee Qualifications: Develop a process for evaluatina Qualifications of
investigators and institutions seeking grants.
EPA's Actions
EPA required grantees to include grant applicant qualifications in work-plans for the
principal investigator and the project manager. EPA said that the Pre-Award Policy
required using the Grantee Compliance Assistance Initiative Database to conduct
pre-award audits because the database includes results of prior grantee reviews.
The database included results of programmatic and administrative reviews. EPA
staff validated data quarterly, and EPA relayed validation results to relevant national
program managers to indicate areas for improvement. Program offices would
evaluate grantee past performance using the results contained in the database.
Nongovernmental
Organization
Practices
One organization evaluated applicants' qualifications when deciding to issue a
grant. Before the organization made a grant request or program-related
investment, the organization advised applicants to produce letters of inquiry
describing the qualifications of those engaged in the project. After receiving the
letters, organization staff members asked grants seekers to submit formal proposals
that should include the names and curricula vitae of those engaged in the project.
They also asked applicants to recount accomplishments under current or previous
grants, and specifically asked for references to benchmarks provided in previous
proposals.
OIG Assessment
Both EPA and one nongovernmental organization required grant applicants to
report on the qualifications of its key grant staff. In order to ensure that EPA's
actions address this issue, we suggest that EPA track implementation of the Pre-
Award Policy to ensure that its implementation corrects the problems raised in the
congressional testimony.
7
-------
Testimony
Suggestion 4
Trainina: Train both EPA personnel and arantees on how to implement the
January 2005 policy and how to better cooperate with each other.
EPA's Actions
EPA provided training/workshops in Headquarters and in the Regions on
implementing the Results Policy. EPA anticipated that grant programs would
provide their own program-specific training on their respective outcomes. For
example, the Office of Water trained its project officers on measures grantees
needed to address in order to fulfill the Office's responsibilities under GPRA. The
Office of Water started preparing an online course that includes a section on
environmental results. EPA planned to make each program office aware of other
offices' activities so they could borrow/benefit from each other. EPA also planned a
manager certification course where managers must maintain their certification in
order to sign off on grant allocations. These courses would complement basic
training that provides skills such as developing and using logic models.
EPA provided online guidance for grant seekers.
Nongovernmental
Organization
Practices
The organizations we assessed used three types of techniques for training and
providing information to help grantees with results reporting:
In person: One organization regularly convened program officers and grantees
working on similar problems to compare strategies and advance learning, and
supported grantee travel to observe each other's work. Two organizations worked
with grantees to ensure that they chose appropriate goals and measures.
On paper: Two organizations provided paper guidance for applying for and
reporting on grants. One of these organizations included a list of "Answers to
Frequently Asked Questions" that addressed common concerns potential grantees
had about requirements.
On the Internet: Two organizations created, or planned to create, Web sites for the
exchange of ideas. One planned to provide an evaluation library with basic profiles
on all grants as well as information about how grantees used adaptive management
techniques. They hoped to provide message boards so that grantees could ask
and answer each other's questions about evaluating grant results.
OIG Assessment
Both EPA and nongovernmental organizations provided organizational staff and
grantees resources in person, on paper, and on the Internet. In order to provide
clear information about reporting results, we suggest that EPA consider providing
an additional online resource for grantees that provides training, examples, and a
question and answer bulletin board, as recommended by the Office of Water. This
one-stop resource could provide grantees with extensive information about how to
meet results reporting requirements. EPA could accomplish this through a central,
aovernment arants Web site such as www.arants.aov. This resource could provide
"tips" that users could search for according to grant type, dollar amount, grant goals,
and other characteristics.
8
-------
Additional Considerations
During our evaluation we noted two additional techniques used by nongovernmental
organizations that the congressional testimony did not specify. The first awards grants to
those recipients with the best chance of success. EPA has considered this technique in its
Pre-Award Policy. The second technique retrospectively evaluates previous grants.
While EPA does not currently undertake retroactive grant results evaluations, it might
consider doing so.
Considering Past Performance in Awarding Future Grants
Nongovernmental organizations looked at past performance to make decisions about
future grants. One organization asked applicants to recount accomplishments under
current or previous grants, and specifically asked for references to benchmarks provided
in previous proposals. Another organization surveyed its grant recipients to identify
areas for improvement in demonstrating outcome-oriented results from its grant
programs. One organization intended to use the results gleaned from a 30-year
retrospective evaluation to target future grants toward the most successful techniques or
subjects, as appropriate.
EPA created the Grantee Compliance Assistance Initiative Database, organized by
grantee, to house the results of EPA's monitoring. This database defined trends by type
of grantee based on findings or issues addressed in oversight reviews. EPA also used the
database to perform pre-award audits because the database included results from prior
grantee reviews. Database users checked a box to indicate whether they addressed
environmental results, and users provided attachments explaining their responses. Users
also completed corrective action plans for grants that did not adequately address database
components like environmental results. EPA planned to periodically analyze trends in
Grantee Compliance Assistance Initiative Database results, and EPA planned to relay
results to relevant national program managers to indicate areas for improvement. EPA's
Pre-Award Policy required grant program offices to verify that past grantees met
requirements before issuing them new grants.
EPA OIG previously reported on grant competition and strongly recommended using a
competitive process for awarding discretionary grants.3 Therefore, EPA could include a
grantee's past performance or ability to manage grant funds as a ranking criterion when
competing discretionary grants and selecting successful applicants. We urge EPA to
track implementation of the Pre-Award Policy to ensure that program managers award
grant dollars to the most capable sources.
3
See EPA OIG Audit Report: EPA Needs to Compete More Assistance Agreements, Report No. 2005-P-00014, March 31, 2005
("http://www.epa. gov/oig/reports/2005/2005033 l-2005-P-00014.pdf).
9
-------
Retrospective Evaluation
Two nongovernmental organizations we assessed hired independent contractors to
conduct retrospective evaluations of grant projects. One organization conducted a project
evaluation covering a number of long-term grants it issued over the past 30 years. They
intended to use the results gleaned from these evaluations to target future grants toward
the most successful techniques or subjects, as appropriate. Another organization
commissioned an independent group to survey its current grantees and unfunded grant
seekers.
EPA did not have plans to conduct a retrospective evaluation. An OARM official said
that because EPA just recently began focusing on outcomes, many previous grants might
not demonstrate outcome-based results. The official said that EPA would consider taking
a retrospective look at grant results.
We believe that a retrospective evaluation of a sample of EPA grant results would
provide the Agency with valuable information about how grant dollars contributed in the
past to environmental and human health improvements. EPA could apply methods and
results from retrospectively assessing completed grant projects to recently completed or
ongoing grants. EPA could report the results to the public.
Conclusions
EPA developed policies and procedures designed to improve the Agency's ability to
demonstrate grant results. We found the practices required by EPA's policies generally
consistent with practices of leading nongovernmental organizations that fund
environmental projects and emphasize grantee performance measurement. We also
found that nongovernmental organizations developed additional practices that EPA could
adopt when implementing corrective steps to improve the Agency's grants effectiveness.
Specifically, we suggest that EPA consider both providing sample logic models to grant
applicants that demonstrate how their activities connect with EPA goals, and also
developing an online grants resource Web site. We also believe that additional,
retrospective evaluation work could help EPA, Congress, and the public determine what
EPA has achieved with its grant dollars in the past. These steps could help EPA
surmount the challenge of demonstrating the environmental and human health impact of
the Agency's grant-making activities.
Summary of Suggestions and EPA Response
We suggest that EPA track implementation of the Results and Pre-Award Policies
to correct the problems raised in congressional testimony by establishing methods
to measure environmental results in grantee work plans, assessing grantee
qualifications, and awarding grant dollars to successful former grantees.
10
-------
To ensure that grantees clearly understand how to demonstrate results, we suggest
that EPA adopt the additional technique of providing sample logic models that
lead applicants toward established program goals.
To provide clear information about reporting results, we suggest that EPA
consider providing an additional online resource for grantees that provides
training, examples, and a question and answer bulletin board, as recommended by
the Office of Water. This one-stop resource could provide grantees with
extensive information about how to meet results reporting requirements. EPA
could accomplish this through a central, government grants Web site such as
www.grants.gov. This resource could provide "tips" that users could search for
according to grant type, dollar amount, grant goals, and other characteristics.
To ensure that grant dollars fund projects with a high chance of success, EPA
could include grantees' past performance as a ranking criterion when competing
discretionary grants and selecting successful applicants.
We believe that a retrospective evaluation of a sample of EPA grant results would
provide the Agency with valuable information about how its past grant dollars
contributed to environmental and human health improvements. EPA could report
this information to the public.
In its comments on the draft report, EPA agreed with the first four suggestions, but
suggested an alternate time frame for the fifth suggestion on conducting a
retrospective evaluation. EPA recommended that conducting the evaluation in 3 or 4
years, once the results policy has been implemented, would provide more information
about grant results than an evaluation conducted now. In the interim, EPA plans to
work with the Office of the Administrator to do a pilot review of work products
obtained from environmental education grants.
11
-------
Appendix A
Process Flow for Competitive Grants
~3 months
Pre-Award
Grantee
Selection
2-3 months
2 months
Grant Project
Implementation
2-3 years (in most cases)
3 months
Grant Close-out
Blue boxes
indicate other
steps
Grantee
completes
project
Grantee
submits final
technical and
other reports
Program
office reviews
final technical
report
Grantee
begins project
Project officer
(in program
office)trained
and certified
Applicant
submits
proposal
Red boxes indicate
oversight or
negotiation steps
Grantee provides
periodic progress
reports
Program office
submits funding
recommendation to
OARM
OARM conducts
administrative review
Program office
issues request
for proposals
on EPA and
fedgrants.gov
web sites
OARM
conducts legal
and
administrative
review, and
issues grant to
recipient
Program office
reviews grant
progress reports,
and monitors
technical
performance
OARM ensures
program office is
satisfied with final
technical report,
and ensures other
final reports were
submitted properly
Guidance on
submitting
proposals provided
through EPA and
government web
sites
OARM conducts
baseline
monitoring, and
on-site or desk
review of
recipients (results
entered into
database)
Program office
conducts"Pre-Award
Review" of technical
and programmatic
elements of proposal
(this may involve
negotiating grant
terms with applicants)
12
-------
Appendix B
Agency Response to Draft Report
May 23, 2005
MEMORANDUM
SUBJECT: Response to Draft Evaluation Report: "EPA's Efforts to Demonstrate Grant
Results Mirror Nongovernmental Organizations' Practices"
FROM: Luis A. Luna
Assistant Administrator
Office of Administration and Resources Management
TO: Jeffrey Harris
Director, Cross Media Issues
Office of the Inspector General
This provides the response of the Office of Administration and Resources Management
(OARM) to the Office of Inspector General (OIG) draft evaluation report (Report) entitled
"EPA's Efforts to Demonstrate Grant Results Mirror Nongovernmental Organizations'
Practices" (Report No. 2005-000847, dated April 14, 2005).
The purpose of the Report is to determine how EPA's grant results policies compare with
the techniques used in leading nongovernmental organizations. We are pleased with the Report's
conclusion that the practices required by EPA's new policies on Grant Environmental Results
(EPA Order 5700.7) and Pre-Award Reviews of non-profit organizations (EPA Order 5700.8)
are generally consistent with the practices of leading nongovernmental organizations that fund
environmental projects and emphasize grantee performance measurement. The Report also
contains suggestions on nongovernmental organization techniques that EPA could consider to
augment the new policies. OARM's response to these suggestions is as follows.
• Suggestion 1: Track Implementation of the Results and Pre-Award Policies to ensure that
EPA staff and grantees follow the policies and better demonstrate grant results.
OARM Response: We agree with this suggestion. The Results Order contemplates that
Office of Grants and Debarment (OGD) will assess compliance through its
Comprehensive Grants Management Reviews. In addition, we have convened a work
group to examine how to improve the Agency's data systems for collecting/reporting
grant results information.
13
-------
Suggestion 2: Adopt the technique of providing sample logic models that lead grant
applicants toward established environmental and human health improvement goals.
OARM Response: We agree with this suggestion and are in the process of putting on the
Internet a grantee tutorial on environmental results that will contain sample logic models.
Suggestion 3: Consider providing an online resource for grantees that provides training,
examples, and a question and answer bulletin board, as recommended by the Office of
Water. This one-stop resource could provide grantees with extensive information about
how to meet results reporting requirements.
OARM Response: We understand that the Office of Water recommended this approach
for training EPA project officers and not grantees. Consistent with this suggestion, we
will be providing training to grantees in three phases - the grantee tutorial noted above, a
nonprofit applicant forum in October, and online training in 2006.
Suggestion 4: Include grantees' past performance as a ranking criterion when competing
discretionary grants and selecting successful applicants.
OARM Response: We agree with this suggestion. Section 6c. of the Results Order
requires, effective January 1, 2006, that all competitive funding announcements include
ranking criteria for evaluating an applicant's past performance in reporting on outputs
and outcomes.
Suggestion 5: Conduct a retrospective evaluation of a sample of EPA grant results to
provide the Agency with valuable information about how grant dollars contributed in the
past to environmental and human health improvements.
OARM Response: OARM disagrees with this suggestion. Since many past grants have
focused primarily on outputs rather than outcomes, we believe that a retrospective
evaluation would produce limited useful information and not allow the Agency to draw
meaningful conclusions. A better approach would be to monitor compliance with the
Results Order, and provide training to project officers and grant applicants, with a view to
conducting a retrospective evaluation three or four years from now. In the interim, we
plan to work with the Office of the Administrator to do a pilot review of work products
obtained from environmental education grants.
14
-------
Thank you for the opportunity to comment on the Report. Please be assured that OARM
and EPA's program offices are committed to ensuring that the Results and Pre-Award policies
are efficiently and effectively implemented. If you have any questions about these comments,
please contact Howard Corcoran of OGD at (202) 564-1903.
cc: Dave O'Connor
Senior Resource Officials
Junior Resource Officials
Grants Management Officers
Peggy Anthony
Howard Corcoran
Richard Kuhlman
15
-------
Appendix C
Distribution
EPA Headquarters
Office of the Administrator
Assistant Administrator, Office of Administration and Resources Management
Director, Office of Grants and Debarment
Agency Followup Official (the CFO)
Agency Audit Followup Coordinator
General Counsel
Audit Followup Coordinator, Office of the Administrator
Audit Followup Coordinator, Office of Administration and Resources Management
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for Public Affairs
Inspector General
16
------- |