$
<
j&z
PRO^0
CD
T
o*
U.S. ENVIRONMENTAL PROTECTION AGENCY
OFFICE OF INSPECTOR GENERAL
EPA's Review of
Applications for a Water
Research Grant Did Not
Follow All Review
Procedures and Lacked
Transparency
Report No. 12-P-0864
September 25, 2012
Scan this mobile
code to learn more
about the EPA OIG.

-------
Report Contributors:
Rick Beusse
Rich Jones
Andrew Lavenburg
Rebecca Matichuk
Michael Young
Abbreviations
ASD
Applied Science Division
C.F.R
Code of Federal Regulations
EMS
Extramural Management Specialist
EPA
U.S. Environmental Protection Agency
FOIA
Freedom of Information Act
GCA
Grants Competition Advocate
NCER
National Center for Environmental Research
NRC
National Research Council of the National Academies
OGC
Office of General Counsel
OGD
Office of Grants and Debarment
OIG
Office of Inspector General
OMB
Office of Management and Budget
ORD
Office of Research and Development
PRD
Peer Review Division
RFA
Request for Applications
SRA
Science Review Administrator
STAR
Science to Achieve Results
Hotline
To report fraud, waste, or abuse, contact us through one of the following methods:
e-mail: OIG Hotline@epa.gov	write: EPA Inspector General Hotline
phone: 1-888-546-8740	1200 Pennsylvania Avenue NW
fax:	202-566-2599	Mailcode 2431T
online:
http://www.epa.gov/oiq/hotline.htm
Washington, DC 20460

-------
S74^v
*. U.S. Environmental Protection Agency	12-P-0864

|	\ Office of Inspector General	September 25, 2012
s
"V—'—J"
s v\|/v S
At a Glance
Why We Did This Review
Based on a request from the
U.S. Environmental Protection
Agency (EPA) Assistant
Administrator for the Office of
Research and Development
(ORD), we examined EPA's
review process for Science to
Achieve Results (STAR) grant
Request for Applications (RFA)
EPA-G2009-STAR-F1,
"Advancing Public Health
Protection through Water
Infrastructure Sustainability."
We sought to determine
whether EPA followed
applicable policies and
procedures, and communicated
with applicants appropriately.
ORD's National Center for
Environmental Research's
(NCER's) STAR grant program
funds research through a
competitive solicitation process
and independent peer review.
For the RFA reviewed, NCER
conducted two peer reviews.
NCER voided results of a
December 2009 peer review
panel due to concerns over
expertise and innovativeness,
and completed a second peer
review in June 2010.
This report addresses the
following EPA Goal or
Cross-Cutting Strategy:
• Advancing science, research,
and technological innovation
For further information, contact
our Office of Congressional and
Public Affairs at (202) 566-2391.
The full report is at:
www.epa.aov/oia/reports/2012/
20120925-12-P-0864.pdf
EPA's Review of Applications for a Water Research
Grant Did Not Follow All Review Procedures and
Lacked Transparency
What We Found
NCER did not follow all applicable policies and procedures in reviewing
applications submitted under RFA EPA-G2009-STAR-F1, and lacked procedures
for a key aspect of its STAR grant application peer review process. Specifically:
•	NCER did not follow the review process required by the Code of Federal
Regulations (C.F.R) under 40 C.F.R Part 40.150. EPA subsequently issued
a class exception from 40 C.F.R 40.150 that retroactively applied to the
process for this and other RFAs, but NCER did not make this known to the
public.
•	For more than half of the 72 applications reviewed during each peer review,
at least one of the three assigned peer reviewers did not provide written
comments addressing each evaluation criterion as required.
•	NCER did not have a clearly defined "firewall" policy for its peer review
process. The process used to select reviewers for the June 2010 review, in
our view, was inconsistent with descriptions of NCER's firewall practice
published in 2002 and 2003 National Academies reports.
NCER did not communicate with all applicants for the RFA in a transparent,
appropriate, accurate, and timely manner. For example, NCER was not
transparent in communicating its decision to conduct a second review, the
expected delays resulting from the second review, and whether the results sent
to applicants were based on the December 2009 or June 2010 review. NCER's
declination letters did not sufficiently explain why applicants were not selected or
inform them of the option to request a debriefing. NCER informally communicated
results to some applicants prior to final selection decisions.
The issues noted stemmed from a lack of program procedures and management
controls, resulting in delays and additional costs for NCER to review applications
for the RFA. In addition, the control weaknesses identified could also harm the
reputation of EPA's STAR program that has been characterized in the past as a
program with an independent, rigorous process that funds high-quality research.
Recommendations and Planned Agency Corrective Actions
We recommend that the Assistant Administrator for Research and Development
ensure that NCER makes the public aware of its class exception from 40 C.F.R
40.150, establishes and adheres to improved procedures and management
controls for administering the STAR grant program, and improves its guidance
and management controls for communicating with grant applicants. The Agency
agreed with our conclusions and agreed with the intent of our recommendations.
Planned corrective actions will be addressed in the Agency's 90-day response.

-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
THE INSPECTOR GENERAL
September 25, 2012
MEMORANDUM
SUBJECT: EPA's Review of Applications for a Water Research Grant Did Not Follow
All Review Procedures and Lacked Transparency
Report No. 12-P-0864
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
Action Required
In accordance with EPA Manual 2750, you are required to provide a written response to this
report within 90 calendar days. You should include a corrective actions plan for agreed-upon
actions, including milestone dates. Your response will be posted on the OIG's public website,
along with our memorandum commenting on your response. Your response should be provided
as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the
Rehabilitation Act of 1973, as amended. The final response should not contain data that you do
not want to be released to the public; if your response contains such data, you should identify the
data for redaction or removal. We have no objections to the further release of this report to the
public. We will post this report to our website at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact Carolyn Copper,
Assistant Inspector General for Program Evaluation, at (202) 566-0829 or
copper.carolyn@epa. gov; or Rick Beusse, Director for Air and Research Evaluations, at
(919) 541-5747 or beusse.rick@epa.gov.
FROM:
Arthur A. Elkins, Jr.
TO:
Lek Kadeli
Acting Assistant Administrator for Research and Development

-------
EPA's Review of Applications for a Water Research Grant
Did Not Follow All Review Procedures and Lacked Transparency
12-P-0864
Table of C
Chapters
1	Introduction	 1
Purpose	 1
Background	 1
Scope and Methodology	 5
2	NCER Needs Additional Procedures and Management Controls
to Ensure Independent Review Process	 6
NCER Did Not Comply With 40 C.F.R 40.150 for RFA
But Subsequently Received an Exception from OGD	 6
Peer Reviewer Comments Did Not Consistently Address
Evaluation Criteria Specified in RFA	 8
Lack of Clearly Defined "Firewall" Policy Presents Risk to Integrity
and Independence of STAR Grant Peer Review Process	 9
Impact of Project Officer Concerns on Peer Review Results
Points to Management Control Weakness	 13
Concerns Cited in Support of NCER Management Decision
to Void the December 2009 Peer Review Not Valid	 14
NCER's STAR Grant Review Process Needs Improved
Management Controls	 15
NCER's Decision to Void the December 2009 Peer Review
Resulted in Delays and Additional Costs	 16
Conclusions	 16
Recommendations	 16
Agency Comments and OIG Evaluation	 17
3	Improved Program Guidance, Management Controls Can
Aid NCER's Communications With Applicants	 19
NCER Not Transparent in Communicating Decision to Conduct
Second Review for RFA	 19
NCER's Formal Communication Wth All Grant Applicants
Did Not Adhere to Agency Policy, Guidance, and Guidelines	 21
NCER's Informal Communication Wth Applicants Presented
Risk of Providing Applicants Wth Inaccurate Information	 24
Inaccurate Contact Information Provided to Applicants Had Potential
to Impact Deadlines, But Did Not Impact Applications for RFA	 25
NCER Did Not Communicate Peer Review Results to Applicants in a
Timely Manner for RFA	 25
Process for Communicating Wth Grant Applicants Not
Well Defined by Agency Policy and Guidance	 26
Conclusions	 27
Recommendation	 27
Agency Comments and OIG Evaluation	 27
-continued-

-------
EPA's Review of Applications for a Water Research Grant
Did Not Follow All Review Procedures and Lacked Transparency
12-P-0864
Status of Recommendations and Potential Monetary Benefits	 29
Appendices
A Key Positions With Oversight Responsibilities and
Accountability for RFA EPA-G2009-STAR-F1		30
B Timeline of Events for Activities That Occurred During
RFA EPA-G2009-STAR-F1		31
C Detailed Scope and Methodology		33
D Evaluation Criteria and Peer Reviewer Written Comments		36
E Agency Comments to Draft Report and OIG Response		38
F Distribution		51

-------
Chapter 1
Introduction
Purpose
In a February 7, 2011, letter to the Inspector General, the U.S. Environmental
Protection Agency's (EPA's) Assistant Administrator for Research and
Development stated that EPA's Office of Research and Development (ORD) had
received a letter from a grant applicant expressing concerns with the peer review
process for a specific grant. The grant was a Science to Achieve Results (STAR)
grant with a Request for Applications (RFA) titled "Advancing Public Health
Protection through Water Infrastructure Sustainability" (RFA EPA-G2009-STAR-
Fl).1 The Assistant Administrator requested that the Office of Inspector General
(OIG) independently assess and determine the validity of the concerns raised by
the applicant and recommend any necessary actions. Based on the request, we
conducted an evaluation to determine whether ORD:
•	Followed applicable federal and EPA policies and procedures in managing
the technical peer review panel process for grant proposals submitted
under RFA EPA-G2009-STAR-F1
•	Communicated with grant applicants for RFA EPA-G2009-STAR-F1 in
an accurate, timely, appropriate, and transparent manner regarding the
status of their proposals
Background
The STAR program of ORD's National Center for Environmental Research
(NCER) funds research grants and graduate fellowships in numerous
environmental science and engineering disciplines through a competitive
solicitation process and independent peer review. The program engages scientists
and engineers in targeted research that complements EPA's intramural research
program and those of its partners in other federal agencies.
STAR research is funded through RFAs that are derived from the ORD Strategic
Plan and from research plans for specific topics developed by ORD. RFAs are
prepared in cooperation with other parts of the Agency and concentrate on areas
1 The RFA offered funding for regular as well as "early career" projects. Early career funding was intended for
applicants at the assistant professor level (or equivalent) who held a doctorate degree, were untenured at the closing
of the RFA, and employed in a tenure-track position by the award date. The funding opportunities were designated
as EPA-G2009-STAR-F1 for the regular funding opportunity and EPA-G2009-STAR-F2 for the early career
funding opportunity. Both funding opportunities were announced simultaneously, targeted the same research area
(advancing public health protection through water infrastructure sustainability), and were similarly managed by
NCER. As such, we evaluated NCER's management of these funding opportunities as one. References to RFA EPA-
G2009-STAR-F1 in this report also include the early career opportunity (i.e., RFA-EPA-G2009-STAR-F2).
12-P-0864
1

-------
of special significance to the EPA mission. Once applications are received by the
Agency in response to an RFA, NICER conducts a multi-stage review process to
determine which applications it will recommend for funding. According to EPA,
STAR research is currently focusing on the health effects of particulate matter,
drinking water, water quality, global change, ecosystem assessment and
restoration, human health risk assessment, endocrine disrupting chemicals,
pollution prevention and new technologies, children's health, and socio-economic
research.
Overview of STAR Grant Review Process
NCER's STAR grant application review process consists of multiple steps.2 After
the eligibility of each application is reviewed, the main review steps are:
•	An external peer review organized and managed by NCER's Peer
Review Division (PRD). The external peer review (hereinafter referred to
simply as "peer review") is designed to evaluate each application
according to its scientific merit and the evaluation criteria specified in the
RFA. All eligible3 grant applications are reviewed by a peer review panel
comprised of non-EPA experts (selected and managed by PRD) who are
accomplished in their respective disciplines and proficient in the technical
subjects they are reviewing. The peer review panel scores all applications.
•	A programmatic review organized and managed by the project officer
for the RFA. All applications receiving scores of "excellent" or "very
good" as a result of the peer review process receive a programmatic
review. This programmatic review is conducted by technical experts from
within EPA, including individuals from ORD and EPA program and
regional offices involved with the science or engineering proposed. The
programmatic review panel assesses the relevance of the proposed science
to EPA's research priorities and the proposed Lead Principal
Investigator's past performance and reporting history.
•	A decision meeting to formalize NCER's funding recommendations.
In the decision meeting, the NCER Director makes final funding decisions
based on the results of the external peer review, the internal programmatic
review, and other information as identified in the terms of the RFA (e.g.,
program balance and available funds). The decision meeting is supported
by a proposed decision memorandum drafted by the project officer and
2	According to NCER, the STAR grant award process consists of 10 steps that take about 17 months to complete.
These steps include preparation and certification of the RFA, eligibility screening of applications received, and the
review processes identified above. EPA's Grants and Interagency Agreement Management Division also reviews
final recommended awards.
3	RFA EPA-G2009-STAR-F1 defined "eligible applicants" as public and private nonprofit institutions/organizations
located in the United States, state and local governments, federally recognized Indian tribal governments, and U.S.
territories or possessions. Profit-making firms, federal agencies, and national laboratories funded by federal agencies
were not eligible applicants for this RFA.
12-P-0864
2

-------
circulated to applicable NCER personnel prior to the decision meeting.
Once final funding recommendations are made, the decision memorandum
is signed by the NCER Director. NCER staff who attend the decision
meeting include the Chief of Staff, Senior Science Advisor, the project
officer, the project officer's supervisor, and either the PRD Director or the
Science Review Administrator (SRA) for the RFA, according to the
NCER Director.
The National Research Council of the National Academies (NRC) evaluated
EPA's STAR grant program and published a report4 of its findings in 2003. NRC
concluded that the STAR program had established and maintained a high degree
of scientific excellence, providing the Agency access to independent information
by funding research through broadly advertised, competitive grants. NRC
concluded that the STAR grant award process compared favorably with, and in
some ways exceeded, the process used at other agencies that have extramural
research programs. In particular, NRC noted that EPA had established a rigorous
peer review process for its STAR program. A key factor cited by NRC in support
of this conclusion was a "firewall" practice used by NCER. A key component of
this firewall was its shielding of the peer review process from the influence of
EPA project officers and staff who oversee the individual investigator, fellowship,
and center awards. Thus, according to NRC, the Agency had taken effective steps
to ensure that the STAR grant application review process did not suffer from
conflicts of interest and was independent.
STAR Grant RFA EPA-G2009-STAR-F1
In May 2009, EPA opened RFA EPA-G2009-STAR-F1. EPA's announcement
requested research project proposals that focused on improving the effectiveness
of the water infrastructure for protecting public health. The RFA asked for
projects that demonstrated an integrated, multi-disciplinary approach that would
lead to advances in design, operation, and management of the water infrastructure,
and that directly tied those advances to public health protection in conjunction
with improving water efficiency and reducing energy requirements. The RFA also
identified and described the stages of the application review process, which are
outlined above (i.e., peer review, programmatic review, and funding decision).
According to the RFA, EPA had about $6 million to fund approximately eight
regular awards and four early career awards. Appendix A includes a summary of
the roles and responsibilities of key personnel at each phase of the review process
for this RFA.
For this particular RFA, NCER conducted a peer review of 72 applications in
December 2009; 11 of these applications received a passing score (i.e., were rated
either "excellent" or "very good"). NCER conducted an internal programmatic
4 The Measure of STAR: Review of the U. S. Environmental Protection Agency's Science to Achieve Results (STAR)
Research Grants Program. Committee to Review EPA's Research Grants Program, National Research Council.
National Academies Press. 2003.
12-P-0864
3

-------
review in January 2010 of the 11 applications that had passed the December 2009
peer review, and then had a decision meeting scheduled for February 2010 to
discuss 9 applications that were recommended for funding.
Around the time that the February 2010 decision meeting was scheduled to take
place, the NCER Director decided that a new peer review was needed based on
concerns expressed by the Applied Science Division (ASD) Director and the
project officer responsible for overseeing this RFA. NCER voided the results of
the December 2009 peer review panel and subjected all 72 applications to a
second peer review in June 2010. The June 2010 peer review resulted in 34
applications with a passing score; ultimately, 8 applications were selected for
funding in August 2010. The decision to change the review process was not
communicated to all applicants. Appendix B provides a timeline of events for
RFA EPA-G2009-STAR-F1.
Concerns Expressed by Applicant About NCER's Review Process for
RFA EPA-G2009-STAR-F1
On August 10, 2010, a grant applicant submitted a letter to EPA's Assistant
Administrator for Research and Development expressing concerns about the
review process for RFA EPA-G2009-STAR-F1. The applicant had received
notice from EPA in December 2009 that their proposal had passed the December
2009 peer review and that the application had been forwarded for programmatic
review. In May 2010, the applicant tried to contact the project officer to check on
the status of the review process, but was unsuccessful. In June 2010, the applicant
was informed by the project officer via telephone that the December 2009 peer
review results had been voided and that a second peer review had been conducted.
The applicant's proposal had not passed the second peer review. The applicant's
August 2010 letter to the Assistant Administrator for Research and Development
alleged that:
•	NCER's actions had the appearance that the first scientific and technical
review of proposals submitted under RFA EPA-G2009-STAR-F1 was
"... 'voided' ex post facto, because some parties to the decision did not
like the result."
•	NCER staff did not act in a timely and transparent manner in informing
applicants of the status of their proposals.
The concerns noted in the applicant's August 2010 letter prompted the Assistant
Administrator's request for OIG to examine NCER's review and communication
processes for RFA EPA-G2009-STAR-F1.
12-P-0864
4

-------
Scope and Methodology
We conducted our work at ORD, NCER, in the Washington, DC, area. As part of
our evaluation, we reviewed:
•	Applicable EPA, Office of Management and Budget (OMB), and other
federal policies, procedures, and guidance
•	STAR grant budget and expense data and other documentation specific to
RFA EPA-G2009-STAR-F1
•	Correspondence among ORD and STAR grant applicants for RFA
EPA-G2009-STAR-F1
Additionally, we interviewed EPA personnel, grant applicants, and peer review
panel members to obtain an understanding of the process used to review grant
applications and to communicate with applicants. We analyzed the data we
obtained against criteria established in the guidance, policy, regulation, and
external evaluations we determined to be applicable to NCER's grant review and
communication process.
We conducted our field work from March 2011 to June 2012. We conducted this
evaluation in accordance with generally accepted government auditing standards.
Those standards require that we obtain sufficient, appropriate evidence to provide
a reasonable basis for our findings and conclusions based on our evaluation
objectives. We believe that the evidence obtained provides a reasonable basis for
our findings and conclusions based on our objectives.
Appendix C provides a detailed description of our scope and methodology.
12-P-0864
5

-------
Chapter 2
NCER Needs Additional Procedures and Management
Controls to Ensure Independent Review Process
NCER did not follow all policies and procedures applicable to its STAR grant
review process, and lacked internal procedures and management controls
necessary to ensure an independent review process. The management control
weaknesses we identified during our review resulted in delays and additional
costs for NCER to review applications for RFA EPA-G2009-STAR-F1. These
weaknesses may also impact other RFAs and potentially harm the reputation of
EPA's STAR grant program as a program with an independent and rigorous
process that funds high-quality research. Regarding NCER's review of
applications for RFA EPA-G2009-STAR-F1:
•	NCER did not follow the review process required under the Code of
Federal Regulations (C.F.R) per 40 C.F.R Part 40.150. After the OIG
identified this discrepancy, EPA's Office of Grants and Debarment (OGD)
issued a class exception from 40 C.F.R 40.150 in November 2011 that
applied to all past and future use of NCER's review process, including the
process used for RFA EPA-G2009-STAR-F1. Although it was not
required, NCER did not make this exception known to the public.
•	For more than half of the 72 applications reviewed during each peer
review, at least one of the three peer reviewers assigned to each
application did not provide written comments addressing each RFA
evaluation criterion, as required by EPA guidance.
•	NCER did not have a clearly defined "firewall" practice for its peer review
process. NCER personnel had different interpretations of their roles in
vetting and selecting reviewers for a June 2010 peer review. The process
used to select reviewers for the June 2010 review, in our view, was
inconsistent with descriptions of NCER's firewall practice published in
2002 and 2003 National Academies reports.
NCER Did Not Comply With 40 C.F.R 40.150 for RFA But Subsequently
Received an Exception from OGD
In evaluating the application review process for RFA EPA-G2009-STAR-F1, as
well as the underlying authorities listed in the RFA, we determined that NCER's
review process did not comply with 40 C.F.R 40.150. After we brought this issue
to the attention of NCER management and EPA's Senior Associate Director for
12-P-0864
6

-------
Grants Competition,5 NCER requested, and was granted, a class exception from
the regulation. Although it was not required, this information was not made
known to the public through NCER's website.
40 C.F.R Part 40.150 Requirements and NCER's Review Process
The purpose of 40 C.F.R Part 40, listed in RFA EPA-G2009-STAR-F1 as an
applicable regulation, is to establish and codify mandatory policies and
procedures governing the award of research and demonstration grants by EPA.
According to the regulation, all EPA research and demonstration grants are
awarded subject to EPA interim general grant regulations and procedures and to
the applicable provisions of Part 40. 40 C.F.R Part 40.150 requires that:
•	Every grant application be reviewed by appropriate EPA staff for
relevancy to EPA program needs and priorities, and applicable criteria set
forth in 40 C.F.R 40.140.
•	For those applications considered relevant to EPA research, at least one
reviewer within EPA, and at least two reviewers outside of EPA, review
the applications' technical merit.
NCER's application review process for the RFA we evaluated included reviews
of both the relevancy of applications to program needs and priorities and the
technical merit of applications. However, the application review process did not
follow the specific requirements of 40 C.F.R 40.150. In particular, NCER's
review did not include a review of every application by appropriate EPA staff for
relevancy to EPA program needs and priorities, and applicable criteria set forth in
40 C.F.R 40.140; or steps or specific evaluation criteria to ensure that at least one
reviewer within EPA reviewed the applications' technical merit.
Class Exception from 40 C.F.R Part 40.150
The OIG met with EPA's Senior Associate Director for Grants Competition and
selected NCER staff to inform them that the STAR grant approval process did not
comply with 40 C.F.R 40.150. The Senior Associate Director for Grants
Competition acknowledged that NCER's process did not follow all of the
regulation's requirements, but both he and NCER's Extramural Management
Specialist (EMS) indicated that NCER's review process met the regulation's
intended purpose. EPA's Senior Associate Director for Grants Competition told
us that he believed the regulation was intended to ensure that EPA funds research
that is relevant to its needs and priorities.
Following our discussions with EPA, NCER's management requested a class
exception from 40 C.F.R 40.150. On November 18, 2011, EPA's OGD, with
concurrence from EPA's Office of General Counsel (OGC), issued a class
5 The Senior Associate Director for Grants Competition served as the Grants Competition Advocate (GCA) for RFA
EPA-G2009-STAR-F1.
12-P-0864
7

-------
exception from 40 C.F.R 40.150 that applied to NCER's STAR grant application
review process. This class exception was applied to both past and future reviews
of STAR grant applications. Therefore, it allowed NCER's STAR grant review
process to deviate from the regulation, and for NCER to continue its practice of
having STAR grants go through peer review first. EPA's OGC determined that
the class exception addressed a procedural matter and that publication of a Federal
Register Notice announcing the availability of the exception was not necessary.
In an April 2009 memo, the EPA Administrator reaffirmed EPA's commitment to
transparency in Agency operations, noting that, among other things, earning and
maintaining the public's trust in the Agency's decisionmaking depends in part on
EPA being transparent and inclusive. Under the category of Freedom of
Information Act (FOIA) policy, her memo noted that "offices should also take
steps to make information public on the Agency's Web site without waiting for a
request from the public to do so." In our view, EPA's decision to issue a class
exception from 40 C.F.R 40.150 for its STAR grants program should be made
known to the public through NCER's website.
Peer Reviewer Comments Did Not Consistently Address Evaluation
Criteria Specified in RFA
For this particular RFA, reviewers for both the December 2009 and June 2010
peer reviews did not consistently address the RFA evaluation criteria in their
written comments. Multiple EPA guidance documents stress the importance of
peer reviewers providing written comments addressing each of the stated
evaluation criteria in the RFA. For example, EPA Order 5700.5A1 requires "a
comprehensive, impartial, and objective examination of proposals/applications
based on the criteria contained in the announcement...." The policy further states
that "each reviewer must adequately document their evaluation of an applicant for
the evaluation factors and any subfactors that the applicants proposal/application
is evaluated against in order to demonstrate the reasonableness of the score or
rating that results from the evaluation." Additionally, guidance from the EPA
OGD states that reviewers should write the strengths and weaknesses of the
proposal as measured against each criterion stated in the announcement based on
how well the applicant's proposal is responsive to, and addresses, the stated
criteria.
For both the December 2009 and June 2010 peer reviews for RFA EPA-G2009-
STAR-F1, each of the 72 applications was assigned to and reviewed by 3 peer
reviewers. However, for more than half of the 72 applications reviewed during
each peer review, at least 1 of the 3 reviewers assigned to each application did not
provide written comments addressing each RFA criterion. We believe this
occurred because the evaluation criteria listed on NCER's "Individual
12-P-0864
8

-------
Evaluation" form6 did not align with the specific evaluation criteria listed in RFA
EPA-G2009-STAR-F1. Therefore, in essence, each peer reviewer for this RFA
received two different sets of evaluation criteria. Table D-l in appendix D shows
the differences in the RFA evaluation criteria and those listed on the Individual
Evaluation form. Tables D-2 and D-3 provide additional details on our analysis.
When we asked the SRA assigned to this RFA for the reason the Individual
Evaluation form did not align with the RFA evaluation criteria, he said the form is
a boilerplate template that is used by all SRAs in PRD. He said that PRD does not
create forms for specific RFAs. We believe the failure to align the Individual
Evaluation form with the criteria listed in the RFA led to many peer reviewers
focusing their written comments on the wrong set of evaluation criteria.
In addition to being required by EPA policy, written evaluation comments should
address each RFA evaluation criteria for multiple reasons. First, the Individual
Evaluation forms are used by peer reviewers and the SRA to facilitate discussions
of the application's merit at the face-to-face peer review meetings. For NCER to
fund research that aligns with the RFA, peer reviewers should focus their
discussions on the RFA evaluation criteria above all other factors. Second, written
evaluation comments provide NCER management and applicants with a means of
understanding how reviewers considered the application against the RFA criteria.
Such comments would be particularly useful to NCER management when
concerns are raised about a panel's ability to evaluate the most important research
qualities, as discussed in the RFA evaluation criteria.
In December 2011, after the OIG briefed NCER management on our tentative
findings relative to the peer review evaluation forms, NCER's EMS sent e-mail
guidance reminding NCER staff of the need to ensure Individual Evaluation
forms matched the criteria cited in the RFA.
Lack of Clearly Defined "Firewall" Policy Presents Risk to Integrity
and Independence of STAR Grant Peer Review Process
EPA Order 5700.5A1 requires an objective and impartial examination of
applications for Agency grants. According to the NRC's 2003 report, one of the
steps that NCER has taken to ensure the independence of its STAR grant review
process was establishing a firewall shielding its peer review process from the
influence of the project officers and staff who oversee the individual investigator,
fellowship, and center awards. Although the NRC report cited the firewall as a
key factor supporting its conclusion that NCER had an independent peer review
process, NCER has not defined its firewall in internal procedures or other written
policies. During our evaluation we received varying accounts of how the firewall
6 The Individual Evaluation form is distributed to peer reviewers electronically at the same time the applications are
mailed to the peer reviewers. Reviewers are asked to use this form to provide written comments for each application.
12-P-0864
9

-------
was defined and has been practiced within NCER. However, none of these
descriptions have been included in written EPA guidance, policies, or procedures.
We did not assess the independence of either peer review panel convened for this
RFA. However, the lack of a clearly defined firewall practice at NCER presented
a risk to the independence of the peer review process for RFA EPA-G2009-
STAR-F1. In the absence of clearly defined roles and responsibilities under
NCER's firewall, NCER utilized a process to select reviewers for its June 2010
peer review that, in our view, was inconsistent with descriptions of the firewall
included in the 2002 and 2003 National Academies reports.
NCER's Firewall is Not Clearly Defined
In 2003, NRC cited NCER's firewall as a key factor in concluding that the STAR
program had established a rigorous process to evaluate the quality of proposals.
However, the firewall concept is not described in NCER guidance, policies, or
procedures. During our evaluation, we found limited descriptions of NCER's
firewall in publications external to EPA. A 2002 Transportation Research Board
(TRB) of the National Academies report7 stated that for NCER's STAR grant
program "A firewall is established between staff who administer the independent
peer review and staff who write or select RFAs." In a 2003 report, the NRC
described NCER's firewall as follows:
The program's procedures provide for a firewall that shields the
peer-review process from any influence or potential conflicts of the
project officers and staff who oversee the individual investigator,
fellowship, and center awards. For instance, project officers can
provide the names of potential reviewers to the Science Review
Administrators (SRAs), also known as peer-review officers, but it
is the sole responsibility of the SRAs to select reviewers and to
make reviewer assignments. Project officers may attend peer-
review meetings as observers but may not provide any comments
that would affect peer review.8
We also obtained verbal descriptions of the firewall from NCER staff and
managers. While we found some agreement in the descriptions of the firewall
provided to us by NCER staff and managers, we also found discrepancies.
Further, NCER did not provide us with a definitive description of the firewall
outlining the roles and responsibilities of all personnel involved in the peer review
process. The Former ASD Director told us that people in NCER had differing
7 Surface Transportation Environmental Research: A Long-Term Strategy. Committee for the Surface
Transportation Environmental Cooperative Research Program Advisory Board, National Research Council. Special
Report 268. 2002.
g
According to the NRC report, the process for selecting proposals was described to the committee by the person
responsible for managing it and by several STAR project officers, and some committee members had participated in
the process previously.
12-P-0864
10

-------
interpretations on how the firewall should work, and how people should be
involved in the peer review process.
The limited description of the STAR grant program's firewall that was included in
2003 NRC report was central to our interpretation of how the firewall should be
practiced, as it was a key factor cited in support of NRC's conclusion that the
STAR grant program had established a rigorous and independent peer review
process.
NCER Personnel Had Different Interpretations of Roles in Vetting and
Selecting Reviewers for June 2010 Peer Review
For the December 2009 peer review, we found that the SRA followed protocol in
selecting the peer reviewer expertise needed to review applications for RFA EPA-
G2009-STAR Fl. However, an NCER manager's involvement in the reviewer
selection process for the June 2010 peer review, in our view, was inconsistent
with descriptions of the firewall in the 2002 and 2003 National Research Council
publications.
Typically, the SRA assigned to manage the peer review has flexibility in
identifying peer reviewers and often does so after reading abstracts or full
applications to independently determine the types of reviewers or expertise
needed for the panel. As noted in NCER's draft PRD procedures and policies, as
well as the 2003 NRC report, the SRA has sole responsibility for panel selection
decisions. The SRA, the PRD Director, the NCER Director, and the former ASD
Director all acknowledged that it is the SRA's, or PRD's, responsibility to make
such decisions.
For the June 2010 peer review, the SRA told us that he did not make the final
panel selection decisions. For this peer review, the PRD Director worked with the
NCER Director and NCER Senior Science Advisor to determine a peer review
methodology that included selecting reviewers with specific types of expertise.9
The SRA said he was told by the PRD Director to identify potential reviewers in
these specific areas of expertise and then submit the names to NCER's Senior
Science Advisor to obtain feedback. The SRA said this was not normal and that
NCER had never employed this type of selection process in the past.
The Senior Science Advisor told us that, among her other general responsibilities,
she reviews all draft RFA's before they are forwarded to EPA's grants office.
Therefore, according to the firewall description in the 2002 TRB report, we
considered the Senior Science Advisor to be an individual who should be
separated from the peer review process.
9 NCER determined that the June 2010 peer review panel should consist of the following types of reviewers:
(1) public health specialists; (2) water infrastructure specialists; and (3) "futurists," individuals who exhibited
characteristics of innovativeness.
12-P-0864
11

-------
The NCER Senior Science Advisor told us that she evaluated the resumes of
potential peer reviewers in consultation with managers in EPA's Office of Water
and ORD's Water Quality Research Program, and made recommendations to the
SRA about which reviewers might be good candidates for the SRA to contact for
the June 2010 peer review. She told us that she did not make final approvals for
reviewer selections. The Senior Science Advisor told us that she was asked by the
NCER Director to look at the qualifications of potential reviewers in terms of
whether or not they could provide an appropriate evaluation of applications for
the RFA. As an illustration of her role, ORD provided us with May 2010 e-mails
between the Senior Science Advisor and the SRA. In the e-mails, the SRA
provided names of potential peer reviewers, along with resumes, and asked the
Senior Science Advisor to "review and advise whether they are suitable..for the
June 2010 panel. NCER management told us that the process used in selecting
reviewers for the June 2010 peer review was appropriate, and that it was
acceptable for the Senior Science Advisor to review the suitability of potential
reviewers.
In contrast, the SRA and PRD Director told us that all peer reviewer selections for
the June 2010 peer review were reviewed and approved by NCER's Senior
Science Advisor. Internal ORD e-mails sent prior to the June 2010 peer review
supported what the SRA and PRD Director had told us. Although the Senior
Science Advisor was copied on some of the e-mails, they were not addressed to
her. These e-mails characterized the Senior Science Advisor as someone that was
approving peer reviewer selections. For example, on April 13, 2010, the PRD
Director sent an e-mail to the NCER Director stating that:
Based on recommendations from a number of people across the
Agency, PRD pulled together a list of 27 potential peer reviewers -
- nine reviewers in each of the three expertise categories (public
health, water infrastructure and futurists/innovation). We provided
that list to [name of Senior Science Advisor] along with the
available [resumes].
The e-mail further stated that the Senior Science Advisor had managers in EPA's
Office of Water and ORD's Water Quality Research Program review the list of
potential peer reviewers, and that these individuals "picked" peer reviewers out of
the list of 27 names. The PRD Director informed the NCER Director in the April
13, 2010, e-mail that:
PRD is putting together the paperwork for the potential peer
reviewers who have received a 'yes' nod from at least one of
[name of Senior Science Advisor's] reviewers.
A May 11, 2010, e-mail from a PRD Program Analyst to the PRD Director stated:
12-P-0864
12

-------
[name of SRA] has requested two additional people. One for
Group 3 [name of potential peer reviewer] and one for Group 2
[name of potential peer reviewer],
[name of Senior Science Advisor] approved the selection of [name
of potential peer reviewer the SRA identified for Group 3, above].
Her paperwork is in funding at the present time.
In the May 19, 2010, e-mail, the PRD Director stated that:
When people backed out last week, [name of SRA] moved quicly
[sic] to find replacements and get them 'OKed' by [name of Senior
Science Advisor],
Several sources indicated that NCER's firewall calls for PRD, specifically the
SRA, to make reviewer selections and assignments. As noted above, two
National Academies reports describe the firewall as having an element of
separation between the PRD and other NCER staff and management during the
peer review process. In our view, this separation was not maintained by NCER for
RFA EPA-G2009-STAR-F1, though NCER management told us that they
believed the Senior Science Advisor's role in this particular RFA was appropriate.
Without further clarification and written definition of the firewall practice, the
independence and integrity of the STAR grant review process are at risk.
Impact of Project Officer Concerns on Peer Review Results Points to
Management Control Weakness
The decision to convene a second peer review panel was due, in part, to non-PRD
staff influence over the peer review process that occurred during the December
2009 peer review. After observing the December 2009 peer review panel meeting,
the project officer expressed concerns over the lack of diversity in the expertise of
peer reviewers to NCER's ASD Director. Although the project officer stated that
she expressed her concerns to the ASD Director "shortly after the peer review,"
the ASD Director told us that he chose not to elevate the concerns about the
December 2009 peer review to the NCER Director, letting the process continue
on to programmatic review. In January 2010, after a programmatic review was
completed, the project officer drafted a decision memo, with consultation and
approval from the ASD Director, for the NCER Director. The decision memo
noted concerns with the December 2009 peer review, but also recommended nine
applications for funding. The memo noted that the review process produced "... a
suite of complementary and diverse research projects that tackle high-priority,
high-visibility nationally-relevant environmental issues." The memo further noted
that the projects would enable key stakeholders "to develop and implement
effective policies to reduce public health risks associated with exposure to
drinking water, safeguard the quality and availability of surface and underground
12-P-0864
13

-------
sources of drinking water, improve the water infrastructure, and establish health-
based measures of management effectiveness."
Although the process had produced a seemingly acceptable suite of research
projects that had passed both peer and programmatic review, the NCER Director
made a decision to void the December 2009 peer review results and subject the
applications to another peer review and programmatic review. When we asked the
NCER Director about his decision, he said that, based on the concerns stated in
the decision memo about the December 2009 peer review, he thought it was best
to go back and take a second look at the applications to ensure NCER was
funding the best research.
According to the NCER Deputy Director, the NCER Director's decision to void
the peer review was an appropriate use of discretion and responsibility. He stated
that the NCER Director has the authority and responsibility to void peer review
results, if necessary. The Senior Associate Director for Grants Competition
expressed a similar sentiment, stating that his advice would be to redo the
evaluation if there was merit to the concerns expressed. However, the NCER
Director did not provide evidence that he took steps to independently substantiate
staff concerns prior to making his decision, nor did he document his decision for
the record. We believe this undermined the purpose of the firewall, potentially
exposing the process to conflicts of interest. In this instance, the project officer
and the ASD Director—positions that, according to National Academies'
publications, are not supposed to have influence over the peer review process—
provided information to the NCER Director that led to changes in review process
results. The NCER Director did not substantiate the concerns raised by the project
officer and ASD Director; his decision to redo the application review process
resulted in two grant proposals not passing the June 2010 peer review that had
passed the December 2009 peer review. As evidenced by a grant applicant's letter
to EPA's Assistant Administrator for Research and Development, such actions
can call into question the integrity of the application review process.
Concerns Cited in Support of NCER Management Decision to Void the
December 2009 Peer Review Not Valid
In a February 2010 decision memo to the NCER Director, the project officer and
ASD Director noted two primary concerns with the December 2009 peer review:
the panel lacked public health expertise, and the panel did not value innovation.
According to the NCER Director, he made the decision to redo the external peer
review due to the concerns noted in the decision memo and staff concerns
expressed at the February 2010 decision meeting.10
10 The Project Officer told us that a February 2010 decision meeting never occurred, and other staff told us that they
could not remember if a meeting ever took place. This was due, in part, to inclement weather in Washington, DC,
around the time of the scheduled meeting.
12-P-0864
14

-------
We independently examined the concerns about the December 2009 peer review
panel. The panel was mainly comprised of individuals with engineering
backgrounds. However, based on our analysis of the peer reviewer biographies
provided by ORD, at least 8 of the 26 peer reviewers (31 percent) from the
December 2009 peer review, including some of the engineers, had public health
expertise. Therefore, the panel did not lack public health expertise, as stated in the
decision memo. We also determined that "innovation" was not an area of
expertise but rather a characteristic of the research proposals that, per the
evaluation criteria identified in the RFA, should have been evaluated by peer
reviewers with appropriate technical expertise. Based on our analysis, we found
no evidence suggesting that the SRA did not perform his duties with respect to
identifying and selecting appropriate panel members. Therefore, in our view, the
NCER Director's decision to void the December 2009 peer review results was
largely unfounded.
NCER's STAR Grant Review Process Needs Improved Management
Controls
In general, NCER lacked sufficient procedures and management controls to
ensure the independence of its STAR grants competition process. EPA Order
5700.5Al, the RFA, and the applicable regulations cited in the RFA were the only
formal policy and procedures documents provided to OIG by NCER that
governed its STAR grants award process. The other documents provided were
either informal guidance (provided to NCER personnel via e-mail, existed in draft
form, etc.) related to specific portions of the STAR grant review process (e.g.,
peer review) and not to the entire process, or did not include specific procedural
steps for NCER staff. None of the Agency's documents formalized NCER's
firewall practice, outlined procedures for substantiating concerns regarding peer
review results, identified a process for re-evaluating applications, or specified
how decisions to alter or change the typical review process should be
documented.
A lack of formalized procedures was particularly evident with respect to NCER's
decision to void the December 2009 peer review results. For example:
•	The NCER Director was not informed of staff concerns about the
December 2009 peer review until the application review process had
proceeded nearly to completion.
•	The NCER Director appeared to make a decision to redo the review
process without consulting key personnel, including NCER's EMS, the
ASD Director, the PRD Director, EPA's Senior Associate Director for
Grants Competition (or any other personnel in EPA's OGD), or Agency
ethics officials.
•	The NCER Director proceeded to void the December 2009 peer review
results without independently investigating or validating the concerns
12-P-0864
15

-------
noted about that peer review, or providing any documented justification
for his decision to void the results of the review.
According to the Senior Associate Director for Grants Competition, if issues
existed that affected the propriety, quality, objectivity, or results of the review,
it is advisable to redo the peer review. However, such issues should be
demonstrated, substantiated, and supported before a decision is made to void peer
review results.
NCER's Decision to Void the December 2009 Peer Review Resulted in
Delays and Additional Costs
NCER management's decision to conduct a second peer review in June 2010 cost
NCER over 250 hours of staff time, and over $9,250 in payments for peer
reviewers and application processing. The decision to utilize these additional
resources was made without demonstrating that a condition warranting such a use
of resources actually existed. In addition, the decision delayed funding decisions
by approximately 6 months. Four of the eight applicants interviewed by OIG
commented on the significant delays in NCER's review process. One applicant
noted that the length of the review process could have impacted the costs
associated with the research proposal because costs change over time. Another
applicant stated that he was upset and discouraged because his group missed an
opportunity to apply for another grant due to the delay in EPA's response.
Conclusions
Improved management controls are needed to ensure the independence and
effectiveness of EPA's STAR grants review process. We did not find evidence of
fraud or intentional manipulation of the application review process for RFA EPA-
G2009-STAR-F1. However, the management control weaknesses we identified
put the Agency at risk for fraud or other manipulations of its STAR grant
program. At a minimum, the management control weaknesses have the potential
to damage the reputation of the STAR grant program as an independent and
rigorous process that provides funding for high-quality research.
Recommendations
We recommend that the Assistant Administrator for Research and Development:
1.	Take steps to make EPA's decision to issue a class exception from
40 C.F.R 40.150 for its STAR grants program known to the public through
NCER's website.
2.	Direct NCER to develop and/or update written procedures so that the
review template is consistent with evaluation criteria published in the
STAR Grant RFA prior to releasing the review forms to panelists.
12-P-0864
16

-------
3.	Direct NCER to establish written procedures for administering the STAR
grant review process. Such procedures should include:
a.	A description of management controls needed to ensure applicable
regulations and policies are adhered to, and how such controls will
be implemented.
b.	Descriptions of roles and responsibilities of the various NCER
divisions and personnel involved in the STAR grant review
process. In particular, the roles and responsibilities of the SRA and
project officer should be clearly defined.
c.	A clearly defined policy for NCER's firewall, including how it will
be implemented and practiced.
4.	Direct NCER to develop and/or update written procedures that provide
guidance on voiding peer reviews or conducting re-reviews for STAR
grant competitions, including:
a.	How to appropriately evaluate and document the need for a
re-review.
b.	Descriptions of how and when to involve parties within and
outside of NCER.
c.	Appropriate timelines for each step.
Agency Comments and OIG Evaluation
The Agency generally agreed with our conclusion that improved management
controls are needed to ensure the independence and effectiveness of NCER's
STAR grant program. However, the Agency believed that we misinterpreted
statements attributed to EPA staff members, and mischaracterized NCER's
firewall. Further, the Agency stated that one chapter 2 section heading could be
taken out of context. We re-examined each of these areas and made changes to the
final report to state that NCER has not clearly defined its firewall practice. We
revised the heading in question to clarify that our findings are specific to NCER's
STAR grant review process. Further, we revised the report to state that NCER
personnel had different interpretations of their roles in vetting and selecting
reviewers for the June 2010 peer review. However, we continue to believe that
NCER's Senior Science Advisor's role in the peer review process was not
consistent with previously published descriptions of the firewall.
Several sources indicated that NCER's firewall calls for PRD, specifically the
SRA, to make reviewer selections and assignments. However, the SRA and PRD
Director told us that they did not make final reviewer selections for the June 2010
peer review, and that reviewers were approved by the Senior Science Advisor. We
have included language in the report to reflect NCER management's disagreement
with this characterization. Further, in support of the Senior Science Advisor's
12-P-0864
17

-------
assertion that she was not providing reviewer approvals, we have included
reference to an e-mail stating that she assessed the "suitability" of two prospective
peer reviewers submitted to her by the SRA for the June 2010 panel. However, we
have also included references to a number of e-mails characterizing the Senior
Science Advisor's role in the June 2010 peer review process as someone that was
involved in reviewing and approving peer reviewer selections. We also made it
clear that the Senior Science Advisor was copied on some of these e-mails, but
that they were not addressed to her.
Although the Agency's response contained alternative recommendations, the
Agency noted in the exit conference that it agreed with the intent of our
recommendations.11 The Agency generally agreed with Recommendation 1 as
written. For the other chapter 2 recommendations the Agency generally agreed to
develop the requisite internal controls but suggested alternative recommendations.
For recommendations 3a and 3b in the final report, NCER suggested we make an
alternative recommendation to develop overarching, NCER-wide operational
guidance for staff responsibilities, communication, and training. While we believe
this could be appropriate for NCER management to consider as it addresses our
recommendations, we believe that our recommendation to develop STAR grant-
specific procedures focusing on the issues we found is more appropriate given the
scope of our evaluation. We did not make changes to recommendations 3a and 3b.
The Agency also proposed alternative recommendations that met the intent of
some of our draft report recommendations. Therefore, we included the Agency's
alternative recommendations as recommendations 2 and 4 in the final report.
These recommendations replaced recommendations 2c, 2d, 2e, and 2f from the
draft report. The draft report recommendations are included in appendix E. The
only change we made to the Agency's suggested alternative recommendations
was to clarify that the recommendations are specific to the STAR grant program.
All recommendations in Chapter 2 are considered unresolved until such time as
the OIG and EPA reach agreement on required actions and planned completion
dates. In its 90-day response to the final report, EPA should provide a description
of the Agency's planned corrective actions, as well as the estimated or actual
milestone completion dates for each action. The Agency's written comments and
OIG's response are in appendix E.
11 During the exit conference, ORD said that their "alternative recommendations" were not alternatives to OIG's
original recommendations but proposed actions for addressing OIG's recommendations.
12-P-0864
18

-------
Chapter 3
Improved Program Guidance, Management Controls
Can Aid NCER's Communications With Applicants
NCER did not communicate with all applicants for RFA EPA-G2009-STAR-F1
in a transparent, appropriate, accurate, and timely manner. For example, NCER
was not transparent in communicating:
•	Its decision to conduct a second review
•	The expected delays resulting from the second review
•	Whether the results sent to applicants were based on the December 2009
or June 2010 peer review.
NCER's declination letters did not adhere to Agency policy because they did not
sufficiently explain why applicants were not selected, nor did they inform
applicants of their option to request a debriefing. NCER informally communicated
results to some applicants prior to final selection decisions. Further, some letters
NCER sent to applicants contained incorrect contact information, and some were
sent about a week later than the timelines specified by EPA policy. NCER lacks
sufficient guidance or procedures that formalize lines of accountability for
communicating with applicants during its review process. In particular, staff were
unsure as to who was responsible for organizing, preparing, signing, maintaining,
and sending notifications after the peer review and programmatic review. Also,
improved management controls are needed to ensure that NCER's communications
with grant applicants are transparent and accurate, and adhere to Agency policy and
guidance. Ineffective communications with grant applicants can potentially damage
the reputation of the STAR grant program as an independent and rigorous process
that provides funding for high quality science.
NCER Not Transparent in Communicating Decision to Conduct
Second Review for RFA
In an April 2009 memo, the EPA Administrator expressed the importance of
transparency and openness in conducting EPA operations. According to the
memo, this requires EPA to (1) remain open and accessible to those representing
all points of view, and (2) be responsible for decisions by soliciting the views of
those who will be affected by these decisions. The Administrator also stated that
she believes transparency will enhance the credibility of the Agency, boost public
trust, and improve the quality of EPA's decisions.
For this RFA, NCER was not transparent in communicating its decision to
conduct a second review and the expected delays this would cause were never
stated in any formal communications. Rather than being proactive in notifying all
12-P-0864
19

-------
applicants about the decisions and delays in the review process, NCER only
communicated the changes to its process to those applicants who made inquiries.
According to EPA's Senior Associate Director for Grants Competition, all
applicants should be notified if a program office conducting a competition decides
to redo the evaluation review of applications (i.e., conduct a new peer review).
However, in an e-mail from the NCER Director, the project officer for this RFA
and the ASD Director were directed to respond to the e-mail inquires with the
following statement:
there has been an unanticipated delay in our process, and we are
working to get back on track.
NCER staff responded to the e-mail inquires with the suggested text from the
NCER Director or other vague explanations as to why there were delays in the
review process. For instance, NCER told applicants that delays in the review
process were due to "re-processing" applications, "a bureaucratic process glitch,"
and "unusual setbacks" in processing the applications.
Further, NCER did not explain the changes that occurred during its review
process for this RFA in its declination letters. In particular, these letters did not
state that a second review was conducted or that the second peer review was a
mail review12 rather than a panel review. In addition, the peer review Individual
Evaluation forms included with the declination letters sent to applicants after the
June 2010 peer review were dated 2009, even though the review was completed in
2010. In our view, using a 2009 date on an Individual Evaluation form that was
generated in June 2010 may have been misleading to applicants.
As a result of the Agency's lack of transparency during the review of this RFA,
some applicants were unclear about the review process and others were confused
about why they had not heard from the Agency for an extended length of time or
by the anticipated timeline. For example:
•	Three applicants questioned why they received a notification from EPA,
stating that their application passed after receiving a declination letter.
•	One applicant filed a FOIA request to obtain additional information
because staff did not provide sufficient information to explain why the
review process for this RFA had to be redone.
•	Two applicants assumed the delays in the process or multiple notifications
were a result of an administrative or automated systems problem.
12 For this RFA, NCER held a face-to-face panel meeting for the December 2009 peer review and a mail review for
the June 2010 peer review. According to SRA for this RFA, the mail review is based on individual reviewers and
individual evaluations, and there is no group discussion of the results. According to NCER personnel, NCER does
not have guidance outlining its process for conducting mail reviews. NCER staff told us that they conducted a mail
review for the June 2010 peer review because they believed it would be faster than another face-to-face peer review.
12-P-0864
20

-------
•	One applicant believed that the declination letter she received after the
December 2009 peer review was the official result of NCER's review of
her application. She was not provided the results of the June 2010 peer
review, and was not aware that a second peer review had been conducted.
•	Two applicants incorrectly assumed that the peer review results attached
to the June 2010 declination letter were based on the first round of
reviews, and were unaware that a second peer review had been conducted.
•	Fifteen applicants contacted NCER to check on the status of the review.
•	Four of the 15 applicants who contacted NCER to check on the status of
the review did so on multiple occasions.
NCER's Formal Communication With All Grant Applicants Did Not
Adhere to Agency Policy, Guidance, and Guidelines
We reviewed documented communication between NCER and grant applicants
for RFA EPA-G2009-STAR-F1 and found that some of NCER's formal
communications with applicants did not follow applicable policy and guidance.
NCER sent formal13 notifications, in the form of a letter or e-mail, to applicants
regarding selection decisions after the December 2009 and June 2010 peer
reviews, and after the August 2010 final decision meeting. Examples of formal
notifications sent to applicants included:
•	Peer review declination letters that informed applicants that EPA was
unable to fund the proposed research
•	Review status letters that informed applicants that the proposed research
passed peer review and requested past performance and reporting history
information from the Lead Principal Investigator(s)
•	Notice of award letters that informed applicants that the proposed research
was being recommended for funding
•	Final declination letters that informed applicants that EPA was unable to
fund the proposed research based on scientific merit and relevancy to
EPA's programs
Formal Communication With Unsuccessful Applicants Did Not
Adhere to Agency Policy and Guidance
According to EPA Order 5700.5Al, EPA program offices conducting grant
competitions must provide unsuccessful applicants with written or e-mail
notification. These notifications must generally explain the reasons why the
application was not eligible for award consideration based on the threshold
eligibility review or score after an evaluation of its application against the
13 The OIG's use of "formal" communication included any official correspondence with applicants in the form of a
letter, e-mail, phone call, or personal visit. For this evaluation, information that was provided in an EPA notification
that was sent to all applicants as a result of a selection decision, or provided to all applicants during a conference call
or a professional meeting, was considered "formal" communication.
12-P-0864
21

-------
selection factors in the announcement. Based on the PRD draft procedures and
policies guidance document for the STAR grant peer review process, unsuccessful
applicants will be mailed declination letters, along with the relevant Individual
Evaluation form, after the peer review and final decision meeting.
EPA Order 5700.5A1 also states that these notifications must advise the applicant
that he or she may request a debriefing of the basis for the ineligibility
determination or selection decision. According to EPA Order 5700.5Al,
debriefings may be done orally or in writing, and will be limited to explaining
why the applicant was not considered for award. Prior to PRD sending
notifications to applicants after the December 2009 peer review, the NCER EMS
sent guidance stating that notifications sent to unsuccessful applicants must
provide an option to request a debriefing. The guidance noted that providing
applicants with a copy of the peer review results does not constitute a debriefing.
If applicants request a written debriefing, the guidance from the EMS also noted
that the letter serving as the written debriefing needs to state that it will serve as a
debriefing. After receiving a debriefing, EPA Order 5700.5Al states that an
unsuccessful applicant may file a written dispute with the Agency and must be
provided with reasonable access to Agency records relevant to the dispute in a
manner consistent with FOIA standards.
We found that the declination letters sent to applicants for RFA EPA-G2009-
STAR-F1 did not follow applicable policy and guidance. The declination letters
NCER sent to applicants after the December 2009 and June 2010 peer reviews did
not sufficiently explain to the applicants why their application was not selected
for an award. Further, these letters, as well as notifications sent after the final
selection meeting, did not inform the applicants of their option to request a
debriefing. As a result, applicants were not advised of an opportunity to obtain
additional information on the reasons why their application was not selected for
an award. For example, during interviews with eight applicants who submitted
applications under this RFA, we found that five of the applicants were not aware
of the option to request a debriefing. Not advising applicants of such an
opportunity potentially hinders the applicants' ability to improve their
applications for future EPA grant competitions.
PRD staff told us that one applicant requested a debriefing for this RFA, but the
debriefing conducted by PRD did not follow Agency policy and guidance. The
written debriefing did not:
•	State that it would serve as the applicant's debriefing
•	Explain why the applicant was not selected for award
•	Provide the applicant with information on the strengths and weaknesses of
his/her application in terms of the specific evaluation criteria used in the
competition
•	Explain the dispute process and dispute contact
12-P-0864
22

-------
Instead of informing the applicant of the opportunity and provisions for filing a
dispute, the written debriefing informed the applicant of the opportunity and
provisions for filing an FOIA request. When we asked the applicant about the
written debriefing provided to her by PRD, the applicant was unclear whether a
formal debriefing had actually occurred. She suggested that a discussion she had
with the NCER Director could have been the formal debriefing. However,
according to the NCER Director, the discussion with the applicant was to explain
what had happened during the review process for this RFA and not to explain the
applicant's results or serve as the applicant's debriefing.
Formal Communication With Successful Applicants Did Not Adhere
to Guidelines in RFA
The RFA states that applicants passing peer review will be sent a notification,
which we referred to as a review status letter, requesting past performance and
reporting history information from the lead Principal Investigator(s). NCER sent
review status letters to each of the 11 applicants who passed the December 2009
peer review. However, after the June 2010 peer review, 9 of the 34 applicants
(26 percent) who passed the peer review did not receive review status letters from
NCER as per the RFA guidelines. These nine applicants also passed the
December 2009 peer review. According to PRD staff, applicants who passed both
peer reviews were not sent review status letters after the June 2010 peer review
due to the format of the computer system. The computer system did not generate
another notification for these applicants because it had already sent notifications
after the December 2009 peer review. As a result, these applicants were not
informed that their earlier review status letter had been voided or that their
application had been subjected to and passed the peer review again.
Further, these applicants were not asked to provide updated past performance and
reporting history information even though about 6 months had passed. The
Agency's failure to request the applicants' updated information meant that NCER
was evaluating past performance and reporting history for some applicants that
was 6 months old, while others were being evaluated on information that was
current to June 2010. As noted by ORD, "An applicant's 'Past Performance and
Reporting History' provides information essential to evaluating the potential
performance success of an applicant." While we did not find that this condition
adversely impacted applicants for this RFA, all applicants should be treated
equally in terms of the information on which they are being evaluated.
12-P-0864
23

-------
NCER's Informal Communication With Applicants Presented Risk of
Providing Applicants With Inaccurate Information
Between December 2009 and October 2010, NCER informally14 communicated
with about half of the 72 applicants who were eligible under this RFA.15 NCER
informally communicated the status of the review process or whether applications
were still being considered for awards through phone conversations and e-mails.
While this informal communication is not explicitly prohibited by EPA policy, we
believe such communication increases the risk that applicants will be provided
erroneous or incomplete information. Further, informal communications with
applicants could create a perception of EPA favoring one applicant over others.
EPA Order 5700.5A1 states that, during the competition process, individuals can
communicate with applicants under the condition that the advice or information
will not give applicants a competitive advantage. After the submission of
applications, EPA Order 5700.5A1 also states that EPA personnel may have
limited communications with applicants for the purpose of clarifying certain
aspects of the application relating to threshold eligibility factors, for partial
funding purposes, or to resolve minor or clerical errors. According to EPA's
Senior Associate Director for Grants Competition, it is appropriate for NCER
staff, such as a project officer, to respond to inquiries from applicants seeking
updates on the status of the review process. However, he said that communication
with applicants should only indicate where the Agency is in the review process
and not provide any unofficial results related to specific applications.
Between June and September 2010, NCER informally communicated the results
of the review to 25 different applicants. For instance, after a June 2010
programmatic review had been completed, NCER sent five applicants an informal
e-mail notifying them that their proposals were still "in the running." However,
this occurred about 3 weeks before the NCER Director signed the memorandum
containing the final recommendations for funding (i.e., NCER's official funding
recommendations). Similarly, about a week before the official results, NCER sent
informal e-mails to 12 applicants notifying them that their proposals were "no
longer in the running." One applicant was notified by NCER after the June 2010
programmatic review that her application was "still very much in the running."
About 2 weeks later, but still prior to when official funding recommendations had
been made, NCER sent an e-mail to that same applicant stating that the proposal
was "very unlikely to be awarded."
By providing informal results to applicants in advance of those results becoming
official, NCER could have potentially led applicants to expect an award when, in
fact, their application was not selected at the final decision meeting. Similarly,
such communications could have led some applicants to believe their application
14	The OIG's use of "informal" communication included any unofficial correspondence with applicants in the form
of a letter, e-mail, phone call, or personal visit that does not classify as "formal" communication defined above.
15	This information is based on the records or documentation NCER staff provided to OIG.
12-P-0864
24

-------
was not going to receive an award when, in fact, it was still being considered for
funding.
Inaccurate Contact Information Provided to Applicants Had Potential
to Impact Deadlines, But Did Not Impact Applications for RFA
We found some inaccuracies in the formal NCER communications that we
reviewed for this RFA. Specifically, the review status letters NCER sent
applicants after the December 2009 peer review contained spelling errors,
including the misspelling of the project officer's last name in the signature and
e-mail address. However, we did not find that these inaccuracies had a significant
impact on the applicants for the RFA. Six applicants (about 55 percent of
applicants who passed the December 2009 peer review) indicated problems
sending the requested past performance and reporting history information to the
project officer due to the incorrect contact information in the review status letters.
One applicant's submission was delayed 11 days while the applicant sought
correct contact information for the project officer. We also noted that the SRA's
first name in the signature of the declination letters sent after the June 2010 peer
review was spelled incorrectly.
The inaccurate contact information that we found in the review status letters was
potentially important because these letters requested applicants to send past
performance and reporting history to the project officer within 3 weeks of the
receipt in order to be considered for the programmatic review. Even though some
applicants indicated problems sending the requested information to the project
officer, we did not find that the inaccurate contact information in these letters
prevented any applicants from providing required information to EPA within the
required timeframe or from being considered in NCER's programmatic review.
Further, we did not identify any impacts as a result of the SRA's first name being
spelled incorrectly. However, the inaccuracies we noted have the potential to
impact the ability of applicants to contact appropriate Agency staff and to submit
requested information to EPA prior to stated deadlines.
NCER Did Not Communicate Peer Review Results to Applicants in a
Timely Manner for RFA
EPA Order 5700.5A1 states that the program office conducting the competition
must send unsuccessful applicants a notification within 15 calendar days of the
Agency's selection decision.16 Further, according to NCER's EMS as well as a
summary document NCER provided to us describing its STAR grant award
16 NCER's STAR grant award process includes a programmatic review, which occurs after the peer review meeting
and before the decision meeting. The purpose of the programmatic review is to assure an integrated research
portfolio for the Agency and help determine which applications to recommend for award. As a result, we did not
consider the programmatic review to be a selection decision point. Therefore, the next selection decision point after
the peer review does not occur until the decision meeting, where decisions are made based on the results of the peer
and programmatic review.
12-P-0864
25

-------
process, NCER sends declination letters, along with notifications to those who
passed peer review, to applicants within 15 days of the peer review.
For this RFA, NCER sent declination and notice of award letters after the final
selection meeting within the time specified in the Agency's policy and guidance.
However, NCER did not communicate the results of the peer review in a timely
manner. NCER's letters communicating peer review results were not sent to
applicants within 15 days of either the December 2009 or June 2010 peer review
selection decisions. The letters sent after the December 2009 peer review were
5 days past the time requirement, and the letters sent after the June 2010 peer
review were 8 days past the time requirement. However, we did not identify any
impacts as a result of NCER not complying with Agency policy and guidance.
Process for Communicating with Grant Applicants Not Well Defined
by Agency Policy and Guidance
NCER lacks sufficient guidance or procedures to formalize lines of accountability
for communicating with applicants during its review process. As a result, staff
were unsure who was responsible for organizing, preparing, signing, maintaining,
and sending notifications after the peer review and programmatic review. For
instance, personnel were unsure what notifications were sent to applicants during
the review process for this RFA. Specifically, the PRD Director, the director of
the division responsible for sending notifications to applicants after peer review,
told the project officer that formal letters were not sent to applicants after the
December 2009 peer review when in fact they were. E-mails between NCER staff
also showed that staff were unsure about who was responsible for notifying
applicants and how applicants who passed the December 2009 peer review but did
not pass the June 2010 peer review should have been informed. Further, NCER
staff were unsure who was responsible for maintaining the records for those
applicants who did not pass the eligibility step per Section III of the RFA.
NCER does not have sufficient management controls to ensure that its
communications with grant applicants are transparent and accurate and adhere to
Agency policy and guidance. In particular, NCER lacks an effective process for
staff to track and monitor communications with grant applicants during the review
process. As a result, some of the formal notifications sent to grant applicants for
this RFA did not contain required information or were not sent in a timely
manner. Some of NCER's informal communications provided applicants with
incomplete information regarding the status and results of the review. Further,
NCER did not maintain records of the number of applications who did not pass
the eligibility review for this RFA and the notifications that NCER sent to these
applicants.
12-P-0864
26

-------
Conclusions
Improved guidance and management controls are needed to ensure that NCER
effectively and appropriately communicates with grant applicants during its
STAR grants competition process. The management control weaknesses we
identified regarding NCER's communications with applicants also have the
potential to damage the reputation of the STAR grant program. For example, the
lack of transparency that NCER exhibited in its communications for this RFA
prompted one grant applicant to call into question the integrity of the entire
review process. Effective and appropriate communications with applicants can
serve to avoid such perceptions while adding to the credibility of the STAR grant
program's process and its funding decisions.
Recommendation
We recommend that the Assistant Administrator for Research and Development:
5. Ensure that NCER establishes written procedures for communicating with
STAR grant applicants. Such procedures should include:
a.	A process for staff to effectively track and monitor
communications with applicants.
b.	Guidance for ensuring that appropriate information is included in
notifications to applicants.
c.	Descriptions of roles and responsibilities of the various NCER
divisions and personnel involved in communications with
applicants. In particular, the roles and responsibilities of the SRA
and project officer should be clearly defined.
d.	Annual reviews of the adequacy of internal controls over its
communications with applicants.
Agency Comments and OIG Evaluation
The Agency generally agreed with our conclusion that improved management
controls are needed to ensure that NCER effectively and appropriately
communicates with grant applicants during its STAR grants competition process.
However, the Agency believed that we had mischaracterized the "Past
Performance and Reporting History" it receives from applicants and the potential
effects of not having the most up-to-date history. The Agency also believed that
we had misinterpreted a statement from an EPA staff member and did not reflect
the individual's correct title. Further, the Agency stated that one chapter 3 section
heading could be taken out of context. We reviewed our characterization of the
past performance and reporting history and clarified the statements in this section
of the report to avoid confusion with the initial STAR grants eligibility
determinations. We do not agree that we have misinterpreted a statement from the
Acting Director of the Applied Science Division, but we have revised the report to
12-P-0864
27

-------
reflect his correct title. Finally, we revised the section heading in question to
clarify that our findings are specific to RFA EPA-G2009-STAR-F1.
The Agency generally agreed with recommendations 5b and 5d (numbered as 3b
and 3d in the draft report). For recommendation 5a (3a in the draft report), the
Agency provided a suggested alternative recommendation to provide staff training
to ensure staff are aware of operational procedures and their roles and
responsibilities. However, we believe that the OIG recommendation, as written, is
appropriate and necessary to address a basic management control weakness in
ORD's communications with grant applicants. Once the communications policy is
established, a training program may be appropriate for ensuring that the policy is
understood and followed. However, we do not believe that staff development and
training substitutes for the establishment of a process of tracking and monitoring
communications with grant applicants. The Agency believed that the
recommended actions in recommendation 5c (3c in the draft report) were no
longer necessary due to its suggested recommendation for recommendations 2a,
2b, and 5a. However, in light of our decision not to accept this suggested
alternative recommendation, we continue to believe that recommendation 5c is
appropriate.
All recommendations in chapter 3 are considered unresolved until such time as
the OIG and EPA reach agreement on required actions and planned completion
dates. In its 90-day response to the final report, EPA should provide a description
of the Agency's planned corrective actions, as well as the estimated or actual
milestone completion dates for each action. The Agency's written comments and
OIG's response are in appendix E.
12-P-0864
28

-------
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (In $000s)
Rec.
No.
Page
No.
Subject
Status1
Planned
Completion
Action Official	Date
Claimed
Amount
Agreed-To
Amount
16 Take steps to make EPA's decision to issue a class
exception from 40 C.F.R 40.150 for its STAR grants
program known to the public through NCER's website.
16	Direct NCER to develop and/or update written
procedures so that the review template is consistent
with evaluation criteria published in the STAR Grant
RFA prior to releasing the review forms to panelists.
17	Direct NCER to establish written procedures for
administering the STAR grant review process. Such
procedures should include:
a.	A description of management controls needed to
ensure applicable regulations and policies are
adhered to, and how such controls will be
implemented.
b.	Descriptions of roles and responsibilities of the
various NCER divisions and personnel involved
in the STAR grant review process. In particular,
the roles and responsibilities of the SRA and
project officer should be clearly defined.
c.	A clearly defined firewall policy, including how it
will be implemented and practiced.
17 Direct NCER to develop and/or update written
procedures that provide guidance on voiding peer
reviews or conducting re-reviews for STAR grant
competitions, including:
a.	How to appropriately evaluate and document
the need for a re-review.
b.	Descriptions of how and when to involve
parties within and outside of NCER.
c.	Appropriate timelines for each step.
27 Ensure that NCER establishes written procedures for
communicating with STAR grant applicants. Such
procedures should include:
a.	A process for staff to effectively track and
monitor communications with applicants.
b.	Guidance for ensuring that appropriate
information is included in notifications to
applicants.
c.	Descriptions of roles and responsibilities of the
various NCER divisions and personnel involved
in communications with applicants. In particular,
the roles and responsibilities of the SRA and
project officer should be clearly defined.
d.	Annual reviews of the adequacy of internal
controls over its communications with
applicants.
Assistant Administrator
for Research and
Development
Assistant Administrator
for Research and
Development
Assistant Administrator
for Research and
Development
Assistant Administrator
for Research and
Development
Assistant Administrator
for Research and
Development
0 = recommendation is open with agreed-to corrective actions pending
C = recommendation is closed with all agreed-to actions completed
U = recommendation is unresolved with resolution efforts in progress
12-P-0864	29

-------
Appendix A
Key Positions With Oversight Responsibilities and
Accountability for RFA EPA-G2009-STAR-F1
Stage of
review
process
Position
Summary of responsibilities for RFA EPA-G2009-STAR-F1
Eligibility
Review
Eligibility
Contact
Responds to eligibility questions related to Section III of the RFA from applicants while
the RFA is open. An eligibility screen is performed at the close of the RFA to determine
which applications are eligible for award consideration.

PRD
Director
The PRD Director and staff are in charge of the front-end processing of grant
applications, soliciting suggestions for and managing peer review panels, compiling and
forwarding peer review results to the project officer, and sending declination letters to
applicants that do not pass NCER's review process. The PRD Director also assigns a
SRA to each RFA.
External Peer
Review
SRA
Selects peer review panel members, ensures execution of contracts to obtain reviewer
services, monitors for potential conflicts of interest, assigns applications to reviewers,
works with PRD's Contractor Monitor to send all necessary materials to reviewers,
schedules the peer review meetings, leads and conducts peer review meetings, and
collects and finalizes scores from peer review meetings. Also responsible for providing
the results of the peer review to the PRD Director and project officers, working with
PRD's Contractor Monitor to send declination packages to applicants who did not pass
the peer review or programmatic review, and provide debriefings to applicants when
requested.

PRD
Contractor
Monitor
Oversees NCER contractors. The NCER contractors receive and record the applications
after the RFA closes, provide the application abstracts and applications to the SRA, and
prepare and send notifications (i.e., declination and review status letters) to applicants
regarding the results of the peer review, on behalf of the SRA or project officer.
Programmatic
ASD
Director
Sets the RFA budget and selects a project officer for each RFA. After the RFA is written,
the ASD Director is the first person to review and approve the solicitation. After the
programmatic review, the ASD Director reviews and approves the decision memo and
helps to identify whether selected projects relate to EPA's programmatic needs. The ASD
Director also attends a decision meeting with Division Directors, NCER Director,
appropriate project officer and SRA, and NCER's Senior Science Advisor.
Review
Project
Officer
Reviews and comments on the RFA prior to publication, responds to technical questions
from applicants while the RFA is open, observes the peer review panel, collects past
performance and reporting history information from Principal Investigators, organizes and
leads the programmatic review, writes a decision memorandum that identifies the
applications recommended for funding, communicates with applicants after the peer
review and while final funding decisions are being processed.
Final Funding
Decision
NCER
Director
Makes final funding decisions. Recommendations for funding are provided to the NCER
Director in a decision document and discussed with staff during a decision meeting. The
NCER Director makes final decisions shortly after the decision meeting.

Senior
Science
Advisor
Advises the NCER Director and the Division Directors on the strategic direction of the
extramural research at ORD. Reviews all of the RFA drafts before they are sent to the
Grants Office, and all funding recommendations after applications have gone through the
programmatic review.
Other
EMS
Reviews RFAs to ensure they follow applicable regulations and policies. Point of contact
for legal and administrative issues related to NCER grant competitions, developing
internal guidance, and serves as the liaison with the servicing officers, such as OGD.

GCA
Point of contact for implementing EPA Order 5700.5A1. The GCA must also review and
concur with the RFA before it can be published. Conflicts of interest should be addressed
in consultation with the GCA and/or OGC attorneys.
Source: OIG developed summary of roles and responsibilities from information provided by EPA managers and staff.
12-P-0864	30

-------
Appendix B
Timeline of Events for Activities That Occurred
During RFA EPA-G2009-STAR-F1
Date
Activity/event
May 18, 2009
RFA opening date
August 17, 2009
RFA closing date
December 1-2, 2009
First peer review meeting (December 2009 peer review)
•	Number of applications reviewed: 72
•	Results: 11 applications passed
December 9, 2010
PRD contractor requests past performance and reporting history from
applicants that passed the December 2009 peer review
December 22, 2009
PRD contractor sends declination letters to applicants who did not pass the
December 2009 peer review.
January 20, 2010
Date of programmatic review
•	Number of Applications Reviewed: 11
•	Results: 9 applications passed
February 8, 2010
Scheduled decision meeting to discuss decision memo. The meeting did not
happen due to the snow days.
February 16, 2010
Re-scheduled decision meeting—confirmation of this meeting cannot be
determined.
February 16, 2010
Director discussed the peer review issue with the ASD Director (based on
director's recollection of events; no corroborating documentation available).
February 16, 2010
Project officer informed that peer review will be redone or "supplemented"
with panelists of more varied expertise. Project officer asked to provide
detailed list of panelist nominations.
February 16, 2010
Director held discussions with the PRD Director and the EMS (based on
director's recollection of events; no corroborating documentation available).
February 17, 2010
Project officer provided list of peer reviewer suggestions to ASD Director.
February 18, 2010
EMS e-mailed PRD Director about conducting a second peer review and
expressed concerns about the planned approach that was explained to him
by ASD Director.
February 19, 2010
PRD Director responded to e-mail from EMS to clarify the approach
discussed with and approved by the NCER Director.
February 23,2010
Project officer provided list of peer reviewer suggestions to PRD.
March 4, 2010
Meeting scheduled with NCER Director, PRD Director, EMS, and SRA to
discuss possibility of another peer review—confirmation of this meeting
cannot be determined.
March 4, 2010
On March 4, or shortly after, the NCER Director decided to conduct second
peer review.
March 16, 2010
Project officer told to expect peer review results from second review on
May 19.
March 29, 2010
PRD Director informs SRA that she will forward prospective reviewers to the
Senior Science Advisor.
April 12, 2010
PRD Director tells project officer that PRD is working very hard to get final
scores by May 19.
12-P-0864
31

-------
Date
Activity/event
April 13, 2010
PRD Director requests approval from NCER Director that approach for
selecting mail reviewers is acceptable.
April 14-May 27, 2010
Peer reviewers for second peer review were selected and contacted.
May-June 2010
Second peer review conducted (June 2010 peer review).
June 9, 2010
Final peer review results from June 2010 peer review determined
•	Number of applications reviewed: 72
•	Results: 34 applications passed
June 15, 2010
PRD contractor requests past performance history information from
applicants that passed the June 2010 peer review.
June 28, 2010
Programmatic review meeting
•	Number of applications reviewed: 34
•	Results: 8 applications passed
June 30, 2010
PRD contractor sends declination letters to applicants that did not pass the
June 2010 peer review.
August 18, 2010
Final decision meeting
•	Results: 8 proposals recommended for funding
•	NCER Director concurred with recommendations
August 26, 2010
Date of supplemental memo containing final recommendations for funding.
August 27, 2010
NCER Director signed supplemental memo containing final recommendations
for funding.
August 30 and
September 1, 2010
Project officer sends official memos to applicants that were selected for
awards.
September 8, 2010
PRD contractor sends declination letters to applicants that did not pass the
second programmatic review.
January-August 2011
EPA disburses funds for all eight projects (total of $4,196,385 disbursed).
Source: OIG developed timeline from documentation, e-mails, and descriptions provided by EPA managers and staff.
12-P-0864
32

-------
Appendix C
Detailed Scope and Methodology
Based on a request from the EPA Assistant Administrator for Research and Development, we
sought to determine the factual basis of concerns raised in a grant applicant's August 2010 letter
about NCER's management of the review process for RFA EPA-G2009-STAR-F1. The concerns
were primarily related to the reasons behind NCER's decision to void the results of its technical
peer review of applications and a lack of timely, transparent communication on the part of
NCER. Therefore, our primary objectives were to determine whether ORD (1) followed
applicable federal and EPA policies and procedures in managing the technical peer review panel
process for proposals submitted, and (2) communicated with grant applicants in an accurate,
timely, appropriate, and transparent manner regarding the status of their proposals.
To independently confirm our understanding of the issues raised regarding ORD's review and
management of grant proposals under RFA EPA-G2009-STAR-F1, we:
•	Interviewed the concerned grant applicant as well as the other applicant who passed the
December 2009 peer review but not the June 2010 peer review
•	Reviewed documentation of all correspondence between the concerned grant applicant
and NCER
•	Interviewed key NCER personnel, including the NCER Director, NCER EMS, PRD
Director, SRA, and project officer
•	Developed a timeline of events relating to RFA EPA-G2009-STAR-F1 between May
2009 and September 2010
To determine whether ORD followed all applicable policies, procedures, and guidance during its
review of proposals under RFA EPA-G2009-STAR-F1 and its subsequent communications with
applicants, we obtained and reviewed data and information related to our two objectives, including:
•	Applicable EPA, OMB, and other federal policies, procedures, and guidance, including:
o 40 C.F.R Part 30 (Uniform Administrative Requirements for Grants and
Agreements with Institutions of Higher Education, Hospitals, and Other Non-
profit Organizations)
o 40 C.F.R Part 31 (Uniform Administrative Requirements for Grants and
Cooperative Agreements to State and Local Governments)
o 40 C.F.R Part 40 (Research and Demonstration Grants)
o OMB Circular A-21 (Cost Principles for Educational Institutions)
o OMB Circular A-l 10 (Uniform Administrative Requirements for Grants and
Other Agreements with Institutions of Higher Education, Hospitals and Other
Non-Profit Organizations)
o OMB Circular A-122 (Cost Principles for Non-Profit Organizations)
o EPA Order 5700.5A1, Policy for Competition of Assistance Agreements
o EPA Order 3120.5, Addressing Research Misconduct
o EPA NCER Peer Review Division Policies and Procedures
o RFA EPA-G2009-STAR-F1 (Advancing Public Health Protection through Water
Infrastructure Sustainability)
12-P-0864
33

-------
•	EPA websites and other sources of criteria, including:
o EPA grants website on the EPA Intranet
o EPA Principles of Scientific Integrity
o Memo from EPA Administrator on Scientific Integrity
o Memo from EPA Administrator on Transparency in EPA's Operations
o EPA Competition Guidance for Office of the Administrator Project Officers
•	STAR grant budget and expense data for RFA EPA-G2009-STAR-F1
•	Other STAR grant financial documents related to RF A EPA-G2009-STAR-F1
•	Other documentation specific to RFA EPA-G2009-STAR-F1
•	Correspondence between ORD and STAR grant applicants for RFA EPA-G2009-STAR-F1
We also interviewed:
•	Key NCER management personnel, including:
o NCER Director
o NCER Assistant Deputy Center Director
o NCER Senior Science Advisor
o NCER PRD Director
o Former NCER ASD Director
o NCER Extramural Management Specialist
•	Selected other NCER staff, including:
o Project Officer
o Science Review Administrator
o A Program Analyst within NCER's PRD
•	The EPA Senior Associate Director for Grants Competition.
To assess whether EPA complied with applicable federal and EPA policies and procedures in
managing the technical peer review panel process for proposals submitted (Objective #1), we:
•	Reviewed ORD internal documentation of processes followed before, during, and after
NCER's decision to void the December 2009 peer review
•	Examined internal ORD e-mails to assess whether any mismanagement, misconduct, or
fraudulent activity occurred
•	Reviewed peer reviewer biographies and written evaluation comments17
•	Interviewed four peer reviewers for RFA EPA-G2009-STAR-F1
•	Interviewed EPA and NCER personnel to corroborate and examine further NCER's
management of the review process for RFA EPA-G2009-STAR-F1 and its decision to
void the December 2009 peer review
To assess whether ORD communicated with grant applicants in an accurate, timely, appropriate,
and transparent manner regarding the status of their proposals (Objective #2), we:
17 We reviewed all peer reviewer written comments for a random sample of 41 Individual Evaluation forms to
determine whether the reviewers addressed each of the six RFA evaluation criteria. A random sample of
41 applications (out of 72 total applications) provided a 95% confidence level at +/-10 error. EPA guidance requires
that all peer reviewers' written comments address the RFA criteria. Therefore, our random sample allowed us to
determine whether the evaluation criteria was met.
12-P-0864
34

-------
•	Reviewed documentation of communications between ORD and grant applicants
•	Examined internal ORD e-mail exchanges during the review process
•	Interviewed a judgmental sample of grant applicants for RFA EPA-G2009-STAR-F1
•	Interviewed key NCER personnel responsible for various aspect of communicating with
grant applicants
Review of Management (Internal) Controls
Generally accepted government auditing standards require that auditors obtain an understanding
of internal controls significant to the audit objectives and consider whether specific internal
control procedures have been properly designed and placed in operation. We examined
management and internal controls as they related to our objectives. We reviewed the federal
regulations and guidance related to research and demonstration grants, and focused on EPA's
adherence to, and implementation of, those regulations and guidance. We tested whether EPA's
internal policies and procedures were applied properly during its review of grant applications
under RFA EPA-G2009-STAR-F1, and how they were applied. In general, we examined whether
management controls were effective through document review and analysis, corroborated with
testimonial evidence. In cases where documentation was lacking, we reviewed procedures to
determine whether design or implementation problems existed. The recommendations in our
report reflect alternative actions or additional steps that could be taken to improve the
management control weaknesses we found.
Prior Reports
Prior reports by EPA OIG, the U.S. Government Accountability Office, and NRC were
applicable to this evaluation. The reports listed below, in particular, provided the team with
additional insight into the NCER STAR grant program and important criteria to consider when
NCER's management of the review process for RFA EPA-G2009-STAR-F1:
•	EPA OIG Evaluation Report No. 2003-P-00019, Science to Achieve Results (STAR)
Fellowship Program Needs to Place Emphasis on Measuring Results, September 30, 2003
•	EPA OIG Evaluation Report No. 09-P-0147, EPA Can Improve Its Process for
Establishing Peer Review Panels, April 29, 2009
•	EPA OIG Audit Report No. 11 -P-03 86, Office of Research and Development Should
Increase Awareness of Scientific Integrity Policies, July 22, 201118
•	U.S. Government Accountability Office: Environmental Protection: Information on EPA
Project Grants and Use of Waiver Authority (2001)
•	NRC: The Measure of STAR: Review of the U. S. Environmental Protection Agency's
Science to Achieve Results (STAR) Research Grants Program (2003)
18 Our evaluation team met with members of the OIG Office of Audit Risk Assessment and Program Performance
team prior to their completion of this report. The Office of Audit team had interviewed ORD/NCER staff and
conducted a survey regarding scientific integrity and research misconduct at EPA. The information collected by
OIG's Office of Audit was relevant to our evaluation of NCER's management of RFA EPA-G2009-STAR-F1.
12-P-0864
35

-------
Appendix D
Evaluation Criteria and Peer Reviewer
Written Comments
Table D-l demonstrates the differences in the RFA evaluation criteria and those listed on the
Individual Evaluation form.
Table D-1: Two sets of evaluation criteria provided to peer reviewers
Six evaluation criteria specified in the RFA
Five evaluation criteria listed on the "Individual
Evaluation" form provided to peer reviewers
1) Identification of links between public
health protection and improved water
infrastructure sustainability
1) Research Plan and Quality Assurance
Statement
2) Definition of Success and Measurement
of Progress
2) Investigators
3) Investigators
3) Responsiveness
4) Responsiveness
4) Facilities/Equipment
5) Facilities and equipment
5) Budget
6) Budget

Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the
peer reviewer comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER.
We reviewed all written comments provided on a random sample of 41 Individual Evaluation
forms to determine whether the reviewers had still addressed the RFA criteria despite the form
and RFA criteria not aligning. For the December 2009 peer review, all three reviewers for about
32 percent (+/- 16 percent with a 95 percent confidence interval)19 of the applications addressed
all of the evaluation criteria identified in the RFA in their written comments. For the June 2010
peer review, all three reviewers for about 17 percent (+/- 16 percent with a 95 percent
confidence) of the applications addressed all of the evaluation criteria in the RFA.
The criteria that were most often left unaddressed by the reviewers in their written comments
were criteria 1 and 2 in the RFA (i.e., items number 1 and 2 listed in left column of table D-l).
For over 90 percent of the applications (for both peer reviews), all three reviewers for each
application provided written comments addressing criteria 3, 4, 5, and 6 in the RFA (i.e., items
number 3-6 listed in left column of table D-l). However, for criteria 1 and 2 in the RFA
(i.e., items number 1 and 2 listed in left column of table D-l), which are not identified with
corresponding headers on the Individual Evaluation form, from 37 to 71 percent of the
applications we sampled did not have all three reviewers provide written comments addressing
those evaluation criteria listed in the RFA. See tables D-2 and D-3 for additional details.
19 Our sample of 41 applications provided us with a 95% confidence level with an error of +/-10. To draw general
conclusions regarding how often all 3 reviewers addressed all 6 criteria (i.e., for all 72 applications), we used
interval estimation procedures for sample proportions. Based on our calculations using a Student t test, the final error
rate was +/- 16%. Therefore, the findings for the December 2009 and June 2010 peer reviews (32 and 17 percent,
respectively) are +/-16% error.
12-P-0864
36

-------
Table D-2: Number of reviewers addressing each RFA criteria for the December 2009 peer review
of applications submitted for RFA EPA-G2009-STAR-F1a
December 2009 Peer Review
No. of
RFA
RFA
RFA
RFA
RFA
RFA
reviewers
Criteria #1
Criteria #2
Criteria #3
Criteria #4
Criteria #5
Criteria #6
who






addressed






RFA criteria






Three of three
17 of 41
26 of 41
38 of 41
40 of 41
39 of 41
40 of 41
reviewers
(41.5%)
(63.4%)
(92.7%)
(97.6%)
(95.1%)
(97.6%)
Two of three
13 of 41
11 of 41
3 of 41
1 of 41
2 of 41
1 of 41
reviewers
(31.7%)
(26.8%)
(7.3%)
(2.4%)
(4.9%)
(2.4%)
One of three
10 of 41
3 of 41
0 of 41
Oof 41
Oof 41
0 of 41
reviewers
(24.4%)
(7.3%)
(0.0%)
(0.0%)
(0.0%)
(0.0%)
Zero of three
1 of 41
1 of 41
Oof 41
0 of 41
0 of 41
0 of 41
reviewers
(2.4%)
(2.4%)
(0.0%)
(0.0%)
(0.0%)
(0.0%)
a In the sample of 41 applications we analyzed, there were only 13 applications (31.7%) reviewed during the
December 2009 peer review where all 3 assigned reviewers addressed all 6 RFA evaluation criteria, and thus,
adhered to guidance.
Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the peer reviewer
comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER.
Table D-3: Number of reviewers addressing each RFA criteria for the June 2010 peer review of
applications submitted for RFA EPA-G2009-STAR-F1a
June 2010 Peer Review
Number of






reviewers






who






addressed
RFA
RFA
RFA
RFA
RFA
RFA
RFA criteria
Criteria #1
Criteria #2
Criteria #3
Criteria #4
Criteria #5
Criteria #6
Three of three
12 of 41
18 of 41
41 of 41
41 of 41
40 of 41
41 of 41
reviewers
(29.3%)
(43.9%)
(100%)
(100%)
(97.6%)
(100%)
Two of three
21 of 41
18 of 41
0 of 41
Oof 41
1 of 41
0 of 41
reviewers
(51.2%)
(43.9%)
(0.0%)
(0.0%)
(2.4%)
(0.0%)
One of three
5 of 41
3 of 41
0 of 41
Oof 41
0 of 41
0 of 41
reviewers
(12.2%)
(7.3%)
(0.0%)
(0.0%)
(0.0%)
(0.0%)
Zero of three
3 of 41
2 of 41
0 of 41
Oof 41
Oof 41
0 of 41
reviewers
(7.3%)
(4.9%)
(0.0%)
(0.0%)
(0.0%)
(0.0%)
a In the sample of 41 applications we analyzed, there were only 7 applications (17.1%) reviewed during the June 2010
peer review where all 3 assigned reviewers addressed all 6 RFA Evaluation Criteria, and thus, adhered to guidance.
Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the peer reviewer
comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER.
12-P-0864
37

-------
Appendix E
Agency Comments to Draft Report
and OIG Response
July 16, 2012
MEMORANDUM
SUBJECT: Office of Research and Development's (ORD) Response to the Office
of Inspector General's (OIG) Draft Report dated June 13, 2012, "EPA 's
Review of Applications for Water Infrastructure Sustainability Grant
Exhibited Lack of Management Controls and Transparency" (Report No.
OPE-FY11-008)
FROM: Lek G. Kadeli
Acting Assistant Administrator
Office of Research and Development
TO:	Arthur A. El kins, Jr.
Inspector General
Thank you for the opportunity to review and comment on OlG's draft report "EPA 's
Review of Applications for Water Infrastructure Sustainability Grant Exhibited Lack of
Management Controls and Transparency," (Report Number OPE-FY 1 1 -008). I
appreciate the OlG's acknowledgement that "no evidence of fraud or intentional
manipulation of the application review process" was found and the efforts in conducting
this review requested by the previous ORD Assistant Administrator. In response to the
draft report, ORD offers the following general and specific comments.
General Comments
ORD's National Center for Environmental Research (NCER) has successfully
administered a competitive grants program since its formation in 1995. NCER manages
a number of programs, only one of which is the Science To Achieve Results (STAR)
program, for which numerous research solicitations are managed annually. As noted in
the draft report, the 2003 review of the STAR program conducted by the National
Academy of Science acknowledged the rigorous peer review process NCER has
established, concluding that it "in many ways exceeds those in place at other
organizations that have extramural research programs. "
Since 1995, there have been two instances where a peer review was redone because
the panel was determined to have an inappropriate mix of expertise, the results
negated, and a new panel convened (i.e., a re-review). In the first instance, there-
12-P-0864
38

-------
review decision was made quickly and there-review managed efficiently. The
second instance resulted in the request for this OIG review. In both cases, ORD was
embarking on new science, thus, there were challenges in fully understanding the
diverse expertise needed for the panel to appropriately evaluate applications. In both
cases, the Director of NCER requested feedback from staff and based on that
feedback, the review was redone.
I believe that there is considerable agreement on what improvements are needed to ensure
accuracy and consistency. However, there are several areas where clarification is needed.
These areas are as follows:
Specific Comments
The draft report does not accurately describe NCER's STAR grant application review
and selection processes. For example, the draft report states (chapter 1):
"A decision meeting to formalize NCER's funding recommendations. In the decision
meeting, the NCER Director makes final funding decisions based on the results of
the external peer review and internal programmatic review. "
While the results of the external peer review and internal programmatic review are critical
sources of information the NCER Director uses when making final funding
recommendations, other information may be employed based on the terms of the RFA. The
RFA specifically states, "...the NCER Director may also consider program balance and
available funds." Program balance may take into account items of interest as identified in
the RFA such as geographic diversity, diversity of research methods, or diversity of eligible
awardee types.
OIG Response 1: Chapter 1 of the draft report was revised to state that the NCER Director may
also consider program balance and available funds in making final funding decisions.
The draft report also states that (chapter 3):
"The Agency's failure to request the applicants' updated information had the
potential to impact the applicants' status in the review process because any changes
in the applicants' history coirfd have resulted in an NCER riding of ineligibility for
the next stage of the review process. However, we did not identify such impacts for
this particular RFA."
An applicant's "Past Performance and Reporting History" provides information essential
to evaluating the potential performance success of an applicant. Performance success
includes the applicant's ability to complete the research and comply with reporting
requirements. However, NCER does not use "Past Performance and Reporting History" in
making eligibility determinations.
12-P-0864
39

-------
OIG Response 2: We agree that NCER does not use Past Performance and Reporting History in
making eligibility determinations.
What we intended to convey in the paragraph quoted above was that all applicants were not
afforded an opportunity to submit Past Performance and Reporting History that was current to the
point in time when NCER was evaluating applicants. This had the potential to impact how NCER
evaluated applicants during programmatic review. For those applicants who passed the December
2009 peer review, past performance and reporting history was collected in December 2009.
However, after the June 2010 peer review was conducted, past performance and reporting history
was collected only from applicants who had not passed the December 2009 peer review but
passed the June 2010 peer review. Therefore, NCER was not considering "information essential to
evaluating potential performance success of an applicant" that was current for all applicants. Any
relevant performance or reporting information that had occurred during the 6-month period between
December 2009 and June 2010 would not have been considered for those applicants who passed
both the December 2009 and June 2010 peer reviews. We believe that all applicants should be
treated equally in terms of the information they are being evaluated on, including past performance.
We revised the report to clarify our statements, and have removed reference to eligibility
considerations for this particular issue.
Additionally, the draft report misinterpreted several statements attributed to EPA staff.
First, the draft report states (chapter 2):
"According to EPA's Senior Associate Director for Grants Competition, the
regulation is intended to ensure that EPA funds research that is relevant to
its needs and priorities. "
We would like to clarify that EPA's Senior Associate Director for Grants Competition was
providing his own opinion of the intent of the regulation and not speaking to what the
writers of the regulation had intended.
OIG Response 3: We have revised the report to indicate that the Senior Associate Director for
Grants Competition was providing his own opinion of the intent of the regulation.
A second example states:
"In addition, the NCER Assistant Deputy Director advised the project officer to hold
off sending an e-mail explaining NCER 's decision to redo the peer review to an
applicant that passed the December 2009 peer review but not the June 2010 peer
review."
NCER does not have an Assistant Deputy Director. If your intention was to refer to NCER's
Assistant to the Deputy Director, we have already clarified that this individual did not
provide this guidance nor make a statement to the OIG that may have been misconstrued to
reflect this type of action on his part. If you have additional information on the source of
this comment, it would be helpful if you would share it with us.
12-P-0864
40

-------
OIG Response 4: The correct title for the individual we referred to in the draft report was the Acting
Director for the Applied Science Division of NCER.
We acknowledge that there may be differing interpretations of the guidance that the Acting Director
provided to the project officer regarding communicating the reasons for changes in the review
process with this applicant.
In a June 29, 2010, e-mail, the project officer proposed a draft e-mail, five paragraphs in length, to
one particular applicant whose application passed the December 2009 peer review but not the June
2010 peer review. The project officer's draft e-mail to the applicant included the following explanation
of what occurred with the review of applications for RFA EPA-G2009-STAR-F1:
The review of all 80 of the applications had to be reprocessed from the beginning
even though we were nearing the awards decision point. After reviewing all of the
documents, our senior management felt that the first round of Peer Review panelists
were lacking in multiple aspects of diversity.
The project officer then sent the draft e-mail to the Acting Director for the Applied Science Division,
with the following request in her transmittal:
How's this? I'd like to go ahead and send her the PR comments from the first round
at least.
The "PR comments from the first round" referred to the peer review comments on the application
from the December 2009 (voided) peer review, which the applicant had passed (this particular
applicant, however, had not passed the June 2010 peer review).
The Acting Director responded "Please hold off for now."
Subsequently, on July 2, 2010, the project officer sent an e-mail to the applicant that contained none
of the aforementioned detail about what happened with the first peer review.
In a July 15, 2010 e-mail, the SRA provided the applicant with the following explanation of the review
process:
After careful consideration of the panel make-up, it was determined that disciplines
necessary to adequately review the applications were not included in the December [2009]
panel. Therefore, the results of the December 2009 peer review were voided, and a second
peer review was conducted that included experts from additional fields. As the first peer
review was voided and not included in the decision-making process regarding these
applications, the results from that peer review were not provided.
During an interview with the project officer, we asked her if she was ever instructed not to be
forthright with applicants about the decision to re-review applications for this RFA. She told us that
she was generally under the impression that she was not to provide applicants with any information
about the peer review.
In a follow-up e-mail to OIG's draft report, the ORD audit follow-up coordinator stated that the Acting
Director's guidance to the project officer to "please hold off for now," was in reference to sending the
applicant peer review comments.
Because the applicant was ultimately provided with an explanation of changes to the review process,
we have revised the report to remove reference to the Acting Director for the Applied Science
Division's guidance to the project officer as an example of NCER's lack of transparency in
communicating with applicants.
12-P-0864
41

-------
Finally, in chapter 3 of the draft report:
"The NCER Senior Science Advisor told us that she evaluated the resumes of
potential peer reviewers in consultation with managers in EPA's Office of Water and
ORD 's Water Quality Research Program, and only made recommendations to the
SRA about which reviewers might be good candidates for the SRA to contact for the
June 2010 peer review. She told us that she did not make final approvals for reviewer
selections. In contrast, the SRA and PRD [Peer Review Division] Director stated
that all peer reviewer selections for the June 2010 peer review were reviewed and
approved by NCER's Senior Science Advisor. This was supported by e-mails from
NCER personnel that stated that the NCER Senior Science Advisor was approving
reviewer selections. Such involvement by NCER management in selecting and
approving peer review panel members does not adhere to its firewall practice. "
In documentation previously provided to your staff, we believe the evidence provided does
not support this statement included in your draft report. NCER's Senior Science Advisor did
not review and approve peer reviewers. The email documentation provided to the OIG is
between parties other than the Senior Science Advisor and is anecdotal regarding the role of
the Senior Science Advisor.
OIG Response 5: In light of NCER comments to the draft report and additional documentation
provided during and after the exit conference, we revised our Chapter 2 subheading that "NCER Did
Not Adhere to Its 'Firewall' Practice for RFA EPA-G2009-STAR-F1to reflect that NCER does not
have a clearly defined firewall policy or practice to adhere to. Additionally, we revised the report to
state that NCER personnel had different interpretations of their roles in vetting and selecting
reviewers for the June 2010 peer review. However, in our view the Senior Science Advisor's
involvement in the June 2010 peer review process was not consistent with previously published
descriptions of the firewall.
Several sources indicated that NCER's firewall calls for PRD, specifically the SRA, to make reviewer
selections and assignments. However, the SRA and PRD Director told us that they did not make final
reviewer selections for the June 2010 peer review, and that reviewers were approved by the Senior
Science Advisor. We have included language in the report to reflect NCER management's
disagreement with this characterization. Further, in support of the Senior Science Advisor's statement
that she did not approve reviewers, we have included reference to an e-mail stating that she
assessed the "suitability" of two prospective peer reviewers submitted to her by the SRA for the June
2010 panel. However, we have also included reference to a number of other e-mails characterizing
the Senior Science Advisor's role in the June 2010 peer review process as someone that was
involved in reviewing and approving peer reviewer selections. We also made it clear in the report that
the Senior Science Advisor was copied on some of these e-mails, but that they were not addressed
to her.
Another concern noted in the draft report is how NCER's firewall is characterized.
Chapter 2 states:
"[ORD's National Center for Environmental Research (NCER)] employs a
"firewall" to ensure an objective and impartial examination of applications, as
required by EPA Order 5700.5A1. If used properly, NCER's firewall shields the peer
review process from influences or potential conflicts with EPA personnel outside of
12-P-0864
42

-------
PRD and ensures the independence of the review process. However, for this
particular RFA, NCER did not adhere to its firewall practice, as numerous non-PRD
staff influenced the selection of peer reviewers for the June 2010 peer review. In
addition, non-PRD staff concerns about the December 2009 peer review influenced
the manner in which the review process was conducted, thus indirectly influencing
the results."
The purpose of NCER's firewall is to ensure peer reviewers are free of undo influences. The
firewall ensures that internal Agency and external influences cannot compromise the peer
review process. NCER peer review staff is encouraged to solicit input from many sources
including project officers and other Agency experts to identify the best qualified scientific
expertise to serve in the peer review function. The selection of panelists is made by the PRD
staff. As part of this process PRD may use, or entirely discount, information provided by
others. NCER staff outside of PRD expressing an opinion regarding potential panelist
expertise does not violate the firewall or invalidate the independence of the process. This
process is consistent with established peer review practices and has been reviewed and
approved by the National Academies of Science (see "The Measure of STAR", cited in the
draft report). However, the firewall is not intended to isolate the peer review staff from their
management or the Agencies scientific resources.
OIG Response 6: We agree that NCER staff outside of PRD may provide input to PRD staff or
managers regarding potential reviewers, and that such action does not compromise the
independence of the process. We have clarified this in the final report.
However, we would like to emphasize the point made in ORD's comment that "the selection of
panelists is made by the PRD staff." We believe this to be a key aspect of the firewall and in
ensuring the independence of the peer review process. This is highlighted by statements in
Chapter 1 of our report:
A key component of [the] firewall was its shielding of the peer review process
from the influence of EPA project officers and staff who oversee the individual
investigator, fellowship, and center awards.
And again in Chapter 2 of our report:
As noted in NCER's draft PRD procedures and policies, as well as the 2003 NRC report,
the SRA has sole responsibility for panel selection decisions. The SRA, the PRD Director,
the NCER Director, and the former ASD Director all acknowledged that it is the SRA's, or
PRD's, responsibility to make such decisions.
Chapter 2 concludes that NCER needs "improved management controls to ensure
independence and effectiveness of EPA's STAR grants review process."
"In general, NCER lacked sufficient procedures and management controls to ensure
the independence of its STAR grants competition process. EPA Order 5700.5Al, the
RFA, and the applicable regulations cited in the RFA were the only formal policy and
procedures documents provided to OIG by NCER that governed its STAR grants
award process. The other documents provided were either informal guidance
(provided to NCER personnel via e-mail, existed in draft form, etc.), related to
12-P-0864
43

-------
specific portions of the STAR grant review process (e.g., peer review) and not to the
entire process, or did not include specific procedural steps for NCER staff None of
the Agency's documents formalized NCER 's firewall practice, outlined procedures
for substantiating concerns regarding peer review results, identified a process for re-
evaluating applications, or specified how decisions to alter or change the typical
review process should be documented."
ORD agrees that the development of overarching guidance for staff responsibilities,
communications, and training that reference existing policies, procedures, and guidance will
assist in the improvement of management controls. The OIG review highlights the need for
NCER staff to develop a better understanding of each others' role in grants management
processes, and an overarching guidance document will prove beneficial in that effort. ORD
also agrees that written procedures on re-reviews are needed to improve the evaluation and
documentation of a need for are-review, involvement of parties within and outside NCER,
appropriate timelines for each step, and to ensure that appropriate communications with
applicants are known and followed.
The draft report includes numerous headings that could be taken out of context and reported
as applying to all NCER-managed RFAs. For example, a chapter 2 heading states, NCER's
Review Process Lacks Management Controls. While current controls are not specific as to
re-review processes, this heading implies that NCER lacks all management controls. The
OIG review request was limited to RFA EPA-G2009-STAR-F1 and findings are specific to
this RFA. In addition, a chapter 3 heading states, NCER Not Transparent in Communicating
its Review Process to Applicants. The RFA defines pertinent communications to applicants
including receipt of applications, outcome of the peer review, etc. The re-review
communications for this RFA were not as transparent; however, this heading implies that
NCER lacks transparency in all communications with applicants. Again, the OIG review
request was limited to RFA EPA-02009-STAR-F1 and findings are specific to this RFA.
OIG Response 7: We agree with ORD regarding the report headings cited in the comment above,
and have revised the report to more accurately convey the OIG findings for these particular
sections of the report.
I appreciate the analysis your office has provided in response to our request for this review.
Because of some of the concerns addressed above, I have attached ORD's response to the
draft report recommendations. Please consider ORD's comments as the final report is
prepared. As required by the EPA Order 2750, our written response to the final report will
address any recommendations that may be included at that time.
If you or your staff have any questions, please contact Deborah Heckman at (202) 564-
7274.
Attachment
cc: Ramona Trovato, Principal Deputy Assistant Administrator, ORD
Bob Kavlock, Deputy Assistant Administrator for Science, ORD
12-P-0864
44

-------
Amy Battaglia, Director, OP ARM, (3RD
Jim Johnson, Director, NICER, (3RD
Chris Zarba, Deputy Director, NICER, (3RD
Christiane Routt, Deputy Director, OP ARM, (3RD
12-P-0864

-------
Attachment I	OIG's Draft Report
EPA's Review of Applications for Water Infrastructure Sustainability Grant Exhibited Lack of Management Controls and Transparency
ORD Response to Recommendations
RecNo.
OIG Kecommendation
Lead Responsibility
OKD's Suggested Alternative Recommendation
1
Take steps to make EPA's decision to issue a class exception from 40 C.F.R
40.150 for its STAR grants program known to the public through NCER's
website.
Assistant Administrator for
Research and Development
ORD generally agrees with this current recommendation.
2a
Direct NCER to establish written procedures for administering the STAR
grant review process including a description of management controls needed
to ensure applicable regulations and policies are adhered to, and how such
controls will be implemented.
Assistant Administrator for
Research and Development
Direct NCER to develop operational procedures that provide
overarching NCER-wide guidance for staff responsibilities,
communications, and training which will also reference existing
policies, guidance, and standard operating procedures, as appropriate.
2b
Direct NCER to establish written procedures for administering the STAR
grant review process including descriptions of roles and responsibilities of
the various NCER divisions and personnel involved in the STAR grant
review process. In particular, the roles and responsibilities of the SRA and
project officer should be clearly defined.
Assistant Administrator for
Research and Development
2c
Direct NCER to establish written procedures for administering the STAR
grant review process including a process, including timelines, for receiving
and investigating internal and external complaints, as well as for making and
documenting subsequent management decisions.
Assistant Administrator for
Research and Development
Direct NCER to develop and/or update written procedures that provide
guidance on re-reviews including:
a)	how to appropriately evaluate and document the need for a re-review;
b)	the need for the involvement of parties within and outside of NCER;
c)	appropriate timelines for each step, and;
d)	appropriate communications with applicants are known and followed.
2d
Direct NCER to establish written procedures for administering the STAR
grant review process including provisions to consult with EPA personnel
who oversee EPA grant competitions in determining whether changes to the
review process are necessary.
Assistant Administrator for
Research and Development
2e
Direct NCER to establish written procedures for administering the STAR
grant review process including provisions to involve key NCER management
personnel in decision meetings that stem from staff or management concerns
about the review process.
Assistant Administrator for
Research and Development
2f
Direct NCER to establish written procedures for administering the STAR
grant review process including a process for aligning peer reviewer
evaluation comments with the RFA evaluation criteria.
Assistant Administrator for
Research and Development
Direct NCER to develop and/or update written procedures so that the
review template is consistent with what was published in the RFA prior
to releasing the review forms to panelists.
3a
Ensure that NCER establishes written procedures for communicating with
STAR grant applicants including a process for staff to effectively track and
monitor communications with applicants.
Assistant Administrator for
Research and Development
Direct NCER to develop training to ensure staff are aware of and
understand the operational procedures as well as their roles and
responsibilities in the review process.
3b
Ensure that NCER establishes written procedures for communicating with
STAR grant applicants including guidance for ensuring that appropriate
information is included in notifications to applicants.
Assistant Administrator for
Research and Development
ORD generally agrees with this current recommendation.
12-P-0864
46

-------
RecNo.
OIG Kecommendation
Lead Responsibility
OKD's Suggested Alternative Recommendation
3c
Ensure that NCER establishes written procedures for communicating with
STAR grant applicants including descriptions of roles and responsibilities of
the various NCER divisions and personnel involved in communications with
applicants. Ill particular, the roles and responsibilities of the SRA and
Project Officer should be clearly defined.
Assistant Administrator for
Research and Development
We believe that this would be no longer applicable based on the
suggested alternative recommendation for 2a, 2b, and 3 a.
3d
Ensure that NCER establishes written procedures for communicating with
STAR grant applicants including annual reviews of the adequacy of internal
controls over its communications with applicants.
Assistant Administrator for
Research and Development
ORD generally agrees with this current recommendation.
12-P-0864
47

-------
OIG Response 8: Our responses to ORD's alternative recommendations are included in the table below.
Draft Report
Recommendation
No.
1
2a
2b
Draft Report
Recommendation
No.
2c
2d
OIG Recommendation from
Draft Report
Take steps to make EPA's decision to
issue a class exception from 40 C.F.R
40.150 for its STAR grants program
known to the public through NCER's
website.
Direct NCER to establish written
procedures for administering the STAR
grant review process including a
description of management controls
needed to ensure applicable regulations
and policies are adhered to, and how
such controls will be implemented.
Direct NCER to establish written
procedures for administering the STAR
grant review process including
descriptions of roles and responsibilities
of the various NCER divisions and
personnel involved in the STAR grant
review process. In particular, the roles
and responsibilities of the SRA and
project officer should be clearly defined.
OIG Recommendation from
Draft Report
Direct NCER to establish written
procedures for administering the STAR
grant review process including a
process, including timelines, for
receiving and investigating internal and
external complaints, as well as for
making and documenting subsequent
management decisions.
Direct NCER to establish written
procedures for administering the STAR
grant review process including
provisions to consult with EPA
personnel who oversee EPA grant
competitions in determining whether
changes to the review process are
ORD Suggested
Alternative
Recommendation
ORD generally agrees
with this current
recommendation.
Direct NCER to develop
operational procedures
that provide overarching
NCER-wide guidance for
staff responsibilities,
communications, and
training which will also
reference existing
policies, guidance, and
standard operating
procedures, as
appropriate.
ORD Suggested
Alternative
Recommendation
Direct NCER to develop
and/or update written
procedures that provide
guidance on re-reviews
including:
a)	how to appropriately
evaluate and document
the need for a re-review;
b)	the need for the
involvement of parties
within and outside of
NCER; c) appropriate
timelines for each step,
OIG Evaluation of
ORD Response
In its 90-day response to the report, EPA needs to
provide a description of the planned corrective
actions and estimated or actual, milestone
completion dates. Recommendation 1 is
designated as unresolved in the final report.
We believe these particular OIG
recommendations are appropriate given the scope
of our evaluation. Our findings are specific to the
STAR Grant program. A recommendation to
develop overarching, NCER-wide operational
guidance does not ensure that the specific issues
we found regarding the STAR grant review
process and communication with STAR grant
applicants will be addressed. NCER may choose
to include the written procedures specified in
recommendations as part of a larger, NCER-wide
set of procedures. We will evaluate any such
actions included in ORD's corrective action plan.
However, we believe the OIG recommendations
to be appropriate. Recommendations 3a, 3b, and
3c in the final report are designated as
unresolved.
OIG Evaluation of
ORD Response
As stated above, our findings are specific to the
STAR Grant program, and we believe the
recommendations should be worded as such.
Otherwise, we agree that ORD's alternative
recommendation meets the intent of our original
recommendations, and have revised the report
accordingly. This information is now reflected in
recommendations 4a, 4b, and 4c in the final
report. In its 90-day response to the report, EPA
needs to provide estimated or actual milestone
completion dates for recommendations 4a, 4b,
and 4c. Recommendations 4a, 4b, and 4c are
designated as unresolved in the final report
Changes to Draft
Report
Recommendations
None
We have updated the
final report's
recommendation to
include a call for NCER
to develop a clearly
defined firewall policy.
The draft report
recommendations 2a and
2b appear as
recommendations 3a and
3b in the final report, with
recommendation 3c
added to recommend a
firewall policy.
Changes to Draft
Report
Recommendations
ORD's alternative
recommendation was
inserted into the final
report as
recommendations 4a, 4b,
and 4c (replacing
recommendations 2c, 2d,
and 2e of the draft
report) with the
clarification that the
alternative
recommendation is
specific to the STAR
grant program.
12-P-0864
48

-------
2e
2f
necessary.
Direct NCER to establish written
procedures for administering the STAR
grant review process including
provisions to involve key NCER
management personnel in decision
meetings that stem from staff or
management concerns about the
review process.
Direct NCER to establish written
procedures for administering the STAR
grant review process including a
process for aligning peer reviewer
evaluation comments with the RFA
evaluation criteria.
and;
d) appropriate
communications with
applicants are known
and followed.
Direct NCER to develop
and/or update written
procedures so that the
review template is
consistent with what was
published in the RFA
prior to releasing the
review forms to
panelists.
12-P-0864
As stated above, our findings are specific to the
STAR Grant program, and we believe the
recommendations should be worded as such.
Otherwise, we agree that ORD's alternative
recommendation meets the intent of our original
recommendations, and have revised the report
accordingly. This information is now reflected in
recommendation 2 in the final report. In its 90-day
response to the report, EPA needs to provide
estimated or actual milestone completion dates for
recommendation 2. Recommendation 2 in the
final report is designated as unresolved.
ORD's alternative
recommendation was
inserted into the final
report as
recommendation 2
(replacing
recommendation 2f of
the draft report), with the
clarification that it is
specific to the STAR
grant program.
49

-------
Draft Report
Recommendation
No.
3a
OIG Recommendation from
Draft Report
Ensure that NCER establishes written
procedures for communicating with
STAR grant applicants including a
process for staff to effectively track and
monitor communications with
applicants.
ORD Suggested
Alternative
Recommendation
Direct NCER to develop
training to ensure staff
are aware of and
understand the
operational procedures
as well as their roles and
responsibilities in the
review process.
3b
3c
3d
Ensure that NCER establishes written
procedures for communicating with
STAR grant applicants including
guidance for ensuring that appropriate
information is included in notifications to
applicants.
Ensure that NCER establishes written
procedures for communicating with
STAR grant applicants including
descriptions of roles and
responsibilities of the various NCER
divisions and personnel involved in
communications with applicants. In
particular, the roles and responsibilities
of the SRA and Project Officer should
be clearly defined.
Ensure that NCER establishes written
procedures for communicating with
STAR grant applicants including annual
reviews of the adequacy of internal
controls over its communications with
applicants.
ORD generally agrees
with this current
recommendation.
We believe that this
would be no longer
applicable based on the
suggested alternative
recommendation for 2a,
2b, and 3a.
ORD generally agrees
with this current
recommendation.
12-P-0864
OIG Evaluation of
ORD Response
We believe the OIG recommendation is
appropriate and necessary to address a basic
management control weakness in ORD's
communication with grant applicants. We do not
believe that staff development and training
substitutes for a process of tracking and
monitoring communications with grant applicants.
This recommendation has been re-numbered and
appears as recommendation 5a in the final report.
In its 90-day response to the report, EPA needs to
provide estimated or actual milestone completion
dates for recommendation 5a. Recommendation
5a is designated as unresolved in the final report.
In its 90-day response to the report, EPA needs to
provide a description of the planned corrective
actions and estimated or actual, milestone
completion dates. Recommendation 5b is
designated as unresolved in the final report.
Changes to Draft
Report
Recommendations
Recommendations 3a,
3b, 3c, and 3d ofthe
draft report appear in the
final report as
recommendations 5a, 5b,
5c, and 5d. No other
changes were made to
these recommendations.
See OIG evaluation of ORD response for 2a, 2b,
and 3a, above. Recommendation 5c is designated
as unresolved in the final report.
In its 90-day response to the report, EPA needs to
provide a description ofthe planned corrective
actions and estimated or actual, milestone
completion dates. Recommendation 5d is
designated as unresolved in the final report.
50

-------
Appendix F
Distribution
Office of the Administrator
Assistant Administrator for Research and Development
Deputy Assistant Administrator for Management, Office of Research and Development
Deputy Assistant Administrator for Science, Office of Research and Development
Agency Follow-Up Official (the CFO)
Agency Follow-Up Coordinator
General Counsel
Associate Administrator for Congressional and Intergovernmental Relations
Associate Administrator for External Affairs and Environmental Education
Audit Follow-Up Coordinator, Office of Research and Development
12-P-0864
51

-------