$ < j&z PRO^0 CD T o* U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL EPA's Review of Applications for a Water Research Grant Did Not Follow All Review Procedures and Lacked Transparency Report No. 12-P-0864 September 25, 2012 Scan this mobile code to learn more about the EPA OIG. ------- Report Contributors: Rick Beusse Rich Jones Andrew Lavenburg Rebecca Matichuk Michael Young Abbreviations ASD Applied Science Division C.F.R Code of Federal Regulations EMS Extramural Management Specialist EPA U.S. Environmental Protection Agency FOIA Freedom of Information Act GCA Grants Competition Advocate NCER National Center for Environmental Research NRC National Research Council of the National Academies OGC Office of General Counsel OGD Office of Grants and Debarment OIG Office of Inspector General OMB Office of Management and Budget ORD Office of Research and Development PRD Peer Review Division RFA Request for Applications SRA Science Review Administrator STAR Science to Achieve Results Hotline To report fraud, waste, or abuse, contact us through one of the following methods: e-mail: OIG Hotline@epa.gov write: EPA Inspector General Hotline phone: 1-888-546-8740 1200 Pennsylvania Avenue NW fax: 202-566-2599 Mailcode 2431T online: http://www.epa.gov/oiq/hotline.htm Washington, DC 20460 ------- S74^v *. U.S. Environmental Protection Agency 12-P-0864 | \ Office of Inspector General September 25, 2012 s "V'J" s v\|/v S At a Glance Why We Did This Review Based on a request from the U.S. Environmental Protection Agency (EPA) Assistant Administrator for the Office of Research and Development (ORD), we examined EPA's review process for Science to Achieve Results (STAR) grant Request for Applications (RFA) EPA-G2009-STAR-F1, "Advancing Public Health Protection through Water Infrastructure Sustainability." We sought to determine whether EPA followed applicable policies and procedures, and communicated with applicants appropriately. ORD's National Center for Environmental Research's (NCER's) STAR grant program funds research through a competitive solicitation process and independent peer review. For the RFA reviewed, NCER conducted two peer reviews. NCER voided results of a December 2009 peer review panel due to concerns over expertise and innovativeness, and completed a second peer review in June 2010. This report addresses the following EPA Goal or Cross-Cutting Strategy: Advancing science, research, and technological innovation For further information, contact our Office of Congressional and Public Affairs at (202) 566-2391. The full report is at: www.epa.aov/oia/reports/2012/ 20120925-12-P-0864.pdf EPA's Review of Applications for a Water Research Grant Did Not Follow All Review Procedures and Lacked Transparency What We Found NCER did not follow all applicable policies and procedures in reviewing applications submitted under RFA EPA-G2009-STAR-F1, and lacked procedures for a key aspect of its STAR grant application peer review process. Specifically: NCER did not follow the review process required by the Code of Federal Regulations (C.F.R) under 40 C.F.R Part 40.150. EPA subsequently issued a class exception from 40 C.F.R 40.150 that retroactively applied to the process for this and other RFAs, but NCER did not make this known to the public. For more than half of the 72 applications reviewed during each peer review, at least one of the three assigned peer reviewers did not provide written comments addressing each evaluation criterion as required. NCER did not have a clearly defined "firewall" policy for its peer review process. The process used to select reviewers for the June 2010 review, in our view, was inconsistent with descriptions of NCER's firewall practice published in 2002 and 2003 National Academies reports. NCER did not communicate with all applicants for the RFA in a transparent, appropriate, accurate, and timely manner. For example, NCER was not transparent in communicating its decision to conduct a second review, the expected delays resulting from the second review, and whether the results sent to applicants were based on the December 2009 or June 2010 review. NCER's declination letters did not sufficiently explain why applicants were not selected or inform them of the option to request a debriefing. NCER informally communicated results to some applicants prior to final selection decisions. The issues noted stemmed from a lack of program procedures and management controls, resulting in delays and additional costs for NCER to review applications for the RFA. In addition, the control weaknesses identified could also harm the reputation of EPA's STAR program that has been characterized in the past as a program with an independent, rigorous process that funds high-quality research. Recommendations and Planned Agency Corrective Actions We recommend that the Assistant Administrator for Research and Development ensure that NCER makes the public aware of its class exception from 40 C.F.R 40.150, establishes and adheres to improved procedures and management controls for administering the STAR grant program, and improves its guidance and management controls for communicating with grant applicants. The Agency agreed with our conclusions and agreed with the intent of our recommendations. Planned corrective actions will be addressed in the Agency's 90-day response. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 THE INSPECTOR GENERAL September 25, 2012 MEMORANDUM SUBJECT: EPA's Review of Applications for a Water Research Grant Did Not Follow All Review Procedures and Lacked Transparency Report No. 12-P-0864 This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. Action Required In accordance with EPA Manual 2750, you are required to provide a written response to this report within 90 calendar days. You should include a corrective actions plan for agreed-upon actions, including milestone dates. Your response will be posted on the OIG's public website, along with our memorandum commenting on your response. Your response should be provided as an Adobe PDF file that complies with the accessibility requirements of Section 508 of the Rehabilitation Act of 1973, as amended. The final response should not contain data that you do not want to be released to the public; if your response contains such data, you should identify the data for redaction or removal. We have no objections to the further release of this report to the public. We will post this report to our website at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact Carolyn Copper, Assistant Inspector General for Program Evaluation, at (202) 566-0829 or copper.carolyn@epa. gov; or Rick Beusse, Director for Air and Research Evaluations, at (919) 541-5747 or beusse.rick@epa.gov. FROM: Arthur A. Elkins, Jr. TO: Lek Kadeli Acting Assistant Administrator for Research and Development ------- EPA's Review of Applications for a Water Research Grant Did Not Follow All Review Procedures and Lacked Transparency 12-P-0864 Table of C Chapters 1 Introduction 1 Purpose 1 Background 1 Scope and Methodology 5 2 NCER Needs Additional Procedures and Management Controls to Ensure Independent Review Process 6 NCER Did Not Comply With 40 C.F.R 40.150 for RFA But Subsequently Received an Exception from OGD 6 Peer Reviewer Comments Did Not Consistently Address Evaluation Criteria Specified in RFA 8 Lack of Clearly Defined "Firewall" Policy Presents Risk to Integrity and Independence of STAR Grant Peer Review Process 9 Impact of Project Officer Concerns on Peer Review Results Points to Management Control Weakness 13 Concerns Cited in Support of NCER Management Decision to Void the December 2009 Peer Review Not Valid 14 NCER's STAR Grant Review Process Needs Improved Management Controls 15 NCER's Decision to Void the December 2009 Peer Review Resulted in Delays and Additional Costs 16 Conclusions 16 Recommendations 16 Agency Comments and OIG Evaluation 17 3 Improved Program Guidance, Management Controls Can Aid NCER's Communications With Applicants 19 NCER Not Transparent in Communicating Decision to Conduct Second Review for RFA 19 NCER's Formal Communication Wth All Grant Applicants Did Not Adhere to Agency Policy, Guidance, and Guidelines 21 NCER's Informal Communication Wth Applicants Presented Risk of Providing Applicants Wth Inaccurate Information 24 Inaccurate Contact Information Provided to Applicants Had Potential to Impact Deadlines, But Did Not Impact Applications for RFA 25 NCER Did Not Communicate Peer Review Results to Applicants in a Timely Manner for RFA 25 Process for Communicating Wth Grant Applicants Not Well Defined by Agency Policy and Guidance 26 Conclusions 27 Recommendation 27 Agency Comments and OIG Evaluation 27 -continued- ------- EPA's Review of Applications for a Water Research Grant Did Not Follow All Review Procedures and Lacked Transparency 12-P-0864 Status of Recommendations and Potential Monetary Benefits 29 Appendices A Key Positions With Oversight Responsibilities and Accountability for RFA EPA-G2009-STAR-F1 30 B Timeline of Events for Activities That Occurred During RFA EPA-G2009-STAR-F1 31 C Detailed Scope and Methodology 33 D Evaluation Criteria and Peer Reviewer Written Comments 36 E Agency Comments to Draft Report and OIG Response 38 F Distribution 51 ------- Chapter 1 Introduction Purpose In a February 7, 2011, letter to the Inspector General, the U.S. Environmental Protection Agency's (EPA's) Assistant Administrator for Research and Development stated that EPA's Office of Research and Development (ORD) had received a letter from a grant applicant expressing concerns with the peer review process for a specific grant. The grant was a Science to Achieve Results (STAR) grant with a Request for Applications (RFA) titled "Advancing Public Health Protection through Water Infrastructure Sustainability" (RFA EPA-G2009-STAR- Fl).1 The Assistant Administrator requested that the Office of Inspector General (OIG) independently assess and determine the validity of the concerns raised by the applicant and recommend any necessary actions. Based on the request, we conducted an evaluation to determine whether ORD: Followed applicable federal and EPA policies and procedures in managing the technical peer review panel process for grant proposals submitted under RFA EPA-G2009-STAR-F1 Communicated with grant applicants for RFA EPA-G2009-STAR-F1 in an accurate, timely, appropriate, and transparent manner regarding the status of their proposals Background The STAR program of ORD's National Center for Environmental Research (NCER) funds research grants and graduate fellowships in numerous environmental science and engineering disciplines through a competitive solicitation process and independent peer review. The program engages scientists and engineers in targeted research that complements EPA's intramural research program and those of its partners in other federal agencies. STAR research is funded through RFAs that are derived from the ORD Strategic Plan and from research plans for specific topics developed by ORD. RFAs are prepared in cooperation with other parts of the Agency and concentrate on areas 1 The RFA offered funding for regular as well as "early career" projects. Early career funding was intended for applicants at the assistant professor level (or equivalent) who held a doctorate degree, were untenured at the closing of the RFA, and employed in a tenure-track position by the award date. The funding opportunities were designated as EPA-G2009-STAR-F1 for the regular funding opportunity and EPA-G2009-STAR-F2 for the early career funding opportunity. Both funding opportunities were announced simultaneously, targeted the same research area (advancing public health protection through water infrastructure sustainability), and were similarly managed by NCER. As such, we evaluated NCER's management of these funding opportunities as one. References to RFA EPA- G2009-STAR-F1 in this report also include the early career opportunity (i.e., RFA-EPA-G2009-STAR-F2). 12-P-0864 1 ------- of special significance to the EPA mission. Once applications are received by the Agency in response to an RFA, NICER conducts a multi-stage review process to determine which applications it will recommend for funding. According to EPA, STAR research is currently focusing on the health effects of particulate matter, drinking water, water quality, global change, ecosystem assessment and restoration, human health risk assessment, endocrine disrupting chemicals, pollution prevention and new technologies, children's health, and socio-economic research. Overview of STAR Grant Review Process NCER's STAR grant application review process consists of multiple steps.2 After the eligibility of each application is reviewed, the main review steps are: An external peer review organized and managed by NCER's Peer Review Division (PRD). The external peer review (hereinafter referred to simply as "peer review") is designed to evaluate each application according to its scientific merit and the evaluation criteria specified in the RFA. All eligible3 grant applications are reviewed by a peer review panel comprised of non-EPA experts (selected and managed by PRD) who are accomplished in their respective disciplines and proficient in the technical subjects they are reviewing. The peer review panel scores all applications. A programmatic review organized and managed by the project officer for the RFA. All applications receiving scores of "excellent" or "very good" as a result of the peer review process receive a programmatic review. This programmatic review is conducted by technical experts from within EPA, including individuals from ORD and EPA program and regional offices involved with the science or engineering proposed. The programmatic review panel assesses the relevance of the proposed science to EPA's research priorities and the proposed Lead Principal Investigator's past performance and reporting history. A decision meeting to formalize NCER's funding recommendations. In the decision meeting, the NCER Director makes final funding decisions based on the results of the external peer review, the internal programmatic review, and other information as identified in the terms of the RFA (e.g., program balance and available funds). The decision meeting is supported by a proposed decision memorandum drafted by the project officer and 2 According to NCER, the STAR grant award process consists of 10 steps that take about 17 months to complete. These steps include preparation and certification of the RFA, eligibility screening of applications received, and the review processes identified above. EPA's Grants and Interagency Agreement Management Division also reviews final recommended awards. 3 RFA EPA-G2009-STAR-F1 defined "eligible applicants" as public and private nonprofit institutions/organizations located in the United States, state and local governments, federally recognized Indian tribal governments, and U.S. territories or possessions. Profit-making firms, federal agencies, and national laboratories funded by federal agencies were not eligible applicants for this RFA. 12-P-0864 2 ------- circulated to applicable NCER personnel prior to the decision meeting. Once final funding recommendations are made, the decision memorandum is signed by the NCER Director. NCER staff who attend the decision meeting include the Chief of Staff, Senior Science Advisor, the project officer, the project officer's supervisor, and either the PRD Director or the Science Review Administrator (SRA) for the RFA, according to the NCER Director. The National Research Council of the National Academies (NRC) evaluated EPA's STAR grant program and published a report4 of its findings in 2003. NRC concluded that the STAR program had established and maintained a high degree of scientific excellence, providing the Agency access to independent information by funding research through broadly advertised, competitive grants. NRC concluded that the STAR grant award process compared favorably with, and in some ways exceeded, the process used at other agencies that have extramural research programs. In particular, NRC noted that EPA had established a rigorous peer review process for its STAR program. A key factor cited by NRC in support of this conclusion was a "firewall" practice used by NCER. A key component of this firewall was its shielding of the peer review process from the influence of EPA project officers and staff who oversee the individual investigator, fellowship, and center awards. Thus, according to NRC, the Agency had taken effective steps to ensure that the STAR grant application review process did not suffer from conflicts of interest and was independent. STAR Grant RFA EPA-G2009-STAR-F1 In May 2009, EPA opened RFA EPA-G2009-STAR-F1. EPA's announcement requested research project proposals that focused on improving the effectiveness of the water infrastructure for protecting public health. The RFA asked for projects that demonstrated an integrated, multi-disciplinary approach that would lead to advances in design, operation, and management of the water infrastructure, and that directly tied those advances to public health protection in conjunction with improving water efficiency and reducing energy requirements. The RFA also identified and described the stages of the application review process, which are outlined above (i.e., peer review, programmatic review, and funding decision). According to the RFA, EPA had about $6 million to fund approximately eight regular awards and four early career awards. Appendix A includes a summary of the roles and responsibilities of key personnel at each phase of the review process for this RFA. For this particular RFA, NCER conducted a peer review of 72 applications in December 2009; 11 of these applications received a passing score (i.e., were rated either "excellent" or "very good"). NCER conducted an internal programmatic 4 The Measure of STAR: Review of the U. S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program. Committee to Review EPA's Research Grants Program, National Research Council. National Academies Press. 2003. 12-P-0864 3 ------- review in January 2010 of the 11 applications that had passed the December 2009 peer review, and then had a decision meeting scheduled for February 2010 to discuss 9 applications that were recommended for funding. Around the time that the February 2010 decision meeting was scheduled to take place, the NCER Director decided that a new peer review was needed based on concerns expressed by the Applied Science Division (ASD) Director and the project officer responsible for overseeing this RFA. NCER voided the results of the December 2009 peer review panel and subjected all 72 applications to a second peer review in June 2010. The June 2010 peer review resulted in 34 applications with a passing score; ultimately, 8 applications were selected for funding in August 2010. The decision to change the review process was not communicated to all applicants. Appendix B provides a timeline of events for RFA EPA-G2009-STAR-F1. Concerns Expressed by Applicant About NCER's Review Process for RFA EPA-G2009-STAR-F1 On August 10, 2010, a grant applicant submitted a letter to EPA's Assistant Administrator for Research and Development expressing concerns about the review process for RFA EPA-G2009-STAR-F1. The applicant had received notice from EPA in December 2009 that their proposal had passed the December 2009 peer review and that the application had been forwarded for programmatic review. In May 2010, the applicant tried to contact the project officer to check on the status of the review process, but was unsuccessful. In June 2010, the applicant was informed by the project officer via telephone that the December 2009 peer review results had been voided and that a second peer review had been conducted. The applicant's proposal had not passed the second peer review. The applicant's August 2010 letter to the Assistant Administrator for Research and Development alleged that: NCER's actions had the appearance that the first scientific and technical review of proposals submitted under RFA EPA-G2009-STAR-F1 was "... 'voided' ex post facto, because some parties to the decision did not like the result." NCER staff did not act in a timely and transparent manner in informing applicants of the status of their proposals. The concerns noted in the applicant's August 2010 letter prompted the Assistant Administrator's request for OIG to examine NCER's review and communication processes for RFA EPA-G2009-STAR-F1. 12-P-0864 4 ------- Scope and Methodology We conducted our work at ORD, NCER, in the Washington, DC, area. As part of our evaluation, we reviewed: Applicable EPA, Office of Management and Budget (OMB), and other federal policies, procedures, and guidance STAR grant budget and expense data and other documentation specific to RFA EPA-G2009-STAR-F1 Correspondence among ORD and STAR grant applicants for RFA EPA-G2009-STAR-F1 Additionally, we interviewed EPA personnel, grant applicants, and peer review panel members to obtain an understanding of the process used to review grant applications and to communicate with applicants. We analyzed the data we obtained against criteria established in the guidance, policy, regulation, and external evaluations we determined to be applicable to NCER's grant review and communication process. We conducted our field work from March 2011 to June 2012. We conducted this evaluation in accordance with generally accepted government auditing standards. Those standards require that we obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our evaluation objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our objectives. Appendix C provides a detailed description of our scope and methodology. 12-P-0864 5 ------- Chapter 2 NCER Needs Additional Procedures and Management Controls to Ensure Independent Review Process NCER did not follow all policies and procedures applicable to its STAR grant review process, and lacked internal procedures and management controls necessary to ensure an independent review process. The management control weaknesses we identified during our review resulted in delays and additional costs for NCER to review applications for RFA EPA-G2009-STAR-F1. These weaknesses may also impact other RFAs and potentially harm the reputation of EPA's STAR grant program as a program with an independent and rigorous process that funds high-quality research. Regarding NCER's review of applications for RFA EPA-G2009-STAR-F1: NCER did not follow the review process required under the Code of Federal Regulations (C.F.R) per 40 C.F.R Part 40.150. After the OIG identified this discrepancy, EPA's Office of Grants and Debarment (OGD) issued a class exception from 40 C.F.R 40.150 in November 2011 that applied to all past and future use of NCER's review process, including the process used for RFA EPA-G2009-STAR-F1. Although it was not required, NCER did not make this exception known to the public. For more than half of the 72 applications reviewed during each peer review, at least one of the three peer reviewers assigned to each application did not provide written comments addressing each RFA evaluation criterion, as required by EPA guidance. NCER did not have a clearly defined "firewall" practice for its peer review process. NCER personnel had different interpretations of their roles in vetting and selecting reviewers for a June 2010 peer review. The process used to select reviewers for the June 2010 review, in our view, was inconsistent with descriptions of NCER's firewall practice published in 2002 and 2003 National Academies reports. NCER Did Not Comply With 40 C.F.R 40.150 for RFA But Subsequently Received an Exception from OGD In evaluating the application review process for RFA EPA-G2009-STAR-F1, as well as the underlying authorities listed in the RFA, we determined that NCER's review process did not comply with 40 C.F.R 40.150. After we brought this issue to the attention of NCER management and EPA's Senior Associate Director for 12-P-0864 6 ------- Grants Competition,5 NCER requested, and was granted, a class exception from the regulation. Although it was not required, this information was not made known to the public through NCER's website. 40 C.F.R Part 40.150 Requirements and NCER's Review Process The purpose of 40 C.F.R Part 40, listed in RFA EPA-G2009-STAR-F1 as an applicable regulation, is to establish and codify mandatory policies and procedures governing the award of research and demonstration grants by EPA. According to the regulation, all EPA research and demonstration grants are awarded subject to EPA interim general grant regulations and procedures and to the applicable provisions of Part 40. 40 C.F.R Part 40.150 requires that: Every grant application be reviewed by appropriate EPA staff for relevancy to EPA program needs and priorities, and applicable criteria set forth in 40 C.F.R 40.140. For those applications considered relevant to EPA research, at least one reviewer within EPA, and at least two reviewers outside of EPA, review the applications' technical merit. NCER's application review process for the RFA we evaluated included reviews of both the relevancy of applications to program needs and priorities and the technical merit of applications. However, the application review process did not follow the specific requirements of 40 C.F.R 40.150. In particular, NCER's review did not include a review of every application by appropriate EPA staff for relevancy to EPA program needs and priorities, and applicable criteria set forth in 40 C.F.R 40.140; or steps or specific evaluation criteria to ensure that at least one reviewer within EPA reviewed the applications' technical merit. Class Exception from 40 C.F.R Part 40.150 The OIG met with EPA's Senior Associate Director for Grants Competition and selected NCER staff to inform them that the STAR grant approval process did not comply with 40 C.F.R 40.150. The Senior Associate Director for Grants Competition acknowledged that NCER's process did not follow all of the regulation's requirements, but both he and NCER's Extramural Management Specialist (EMS) indicated that NCER's review process met the regulation's intended purpose. EPA's Senior Associate Director for Grants Competition told us that he believed the regulation was intended to ensure that EPA funds research that is relevant to its needs and priorities. Following our discussions with EPA, NCER's management requested a class exception from 40 C.F.R 40.150. On November 18, 2011, EPA's OGD, with concurrence from EPA's Office of General Counsel (OGC), issued a class 5 The Senior Associate Director for Grants Competition served as the Grants Competition Advocate (GCA) for RFA EPA-G2009-STAR-F1. 12-P-0864 7 ------- exception from 40 C.F.R 40.150 that applied to NCER's STAR grant application review process. This class exception was applied to both past and future reviews of STAR grant applications. Therefore, it allowed NCER's STAR grant review process to deviate from the regulation, and for NCER to continue its practice of having STAR grants go through peer review first. EPA's OGC determined that the class exception addressed a procedural matter and that publication of a Federal Register Notice announcing the availability of the exception was not necessary. In an April 2009 memo, the EPA Administrator reaffirmed EPA's commitment to transparency in Agency operations, noting that, among other things, earning and maintaining the public's trust in the Agency's decisionmaking depends in part on EPA being transparent and inclusive. Under the category of Freedom of Information Act (FOIA) policy, her memo noted that "offices should also take steps to make information public on the Agency's Web site without waiting for a request from the public to do so." In our view, EPA's decision to issue a class exception from 40 C.F.R 40.150 for its STAR grants program should be made known to the public through NCER's website. Peer Reviewer Comments Did Not Consistently Address Evaluation Criteria Specified in RFA For this particular RFA, reviewers for both the December 2009 and June 2010 peer reviews did not consistently address the RFA evaluation criteria in their written comments. Multiple EPA guidance documents stress the importance of peer reviewers providing written comments addressing each of the stated evaluation criteria in the RFA. For example, EPA Order 5700.5A1 requires "a comprehensive, impartial, and objective examination of proposals/applications based on the criteria contained in the announcement...." The policy further states that "each reviewer must adequately document their evaluation of an applicant for the evaluation factors and any subfactors that the applicants proposal/application is evaluated against in order to demonstrate the reasonableness of the score or rating that results from the evaluation." Additionally, guidance from the EPA OGD states that reviewers should write the strengths and weaknesses of the proposal as measured against each criterion stated in the announcement based on how well the applicant's proposal is responsive to, and addresses, the stated criteria. For both the December 2009 and June 2010 peer reviews for RFA EPA-G2009- STAR-F1, each of the 72 applications was assigned to and reviewed by 3 peer reviewers. However, for more than half of the 72 applications reviewed during each peer review, at least 1 of the 3 reviewers assigned to each application did not provide written comments addressing each RFA criterion. We believe this occurred because the evaluation criteria listed on NCER's "Individual 12-P-0864 8 ------- Evaluation" form6 did not align with the specific evaluation criteria listed in RFA EPA-G2009-STAR-F1. Therefore, in essence, each peer reviewer for this RFA received two different sets of evaluation criteria. Table D-l in appendix D shows the differences in the RFA evaluation criteria and those listed on the Individual Evaluation form. Tables D-2 and D-3 provide additional details on our analysis. When we asked the SRA assigned to this RFA for the reason the Individual Evaluation form did not align with the RFA evaluation criteria, he said the form is a boilerplate template that is used by all SRAs in PRD. He said that PRD does not create forms for specific RFAs. We believe the failure to align the Individual Evaluation form with the criteria listed in the RFA led to many peer reviewers focusing their written comments on the wrong set of evaluation criteria. In addition to being required by EPA policy, written evaluation comments should address each RFA evaluation criteria for multiple reasons. First, the Individual Evaluation forms are used by peer reviewers and the SRA to facilitate discussions of the application's merit at the face-to-face peer review meetings. For NCER to fund research that aligns with the RFA, peer reviewers should focus their discussions on the RFA evaluation criteria above all other factors. Second, written evaluation comments provide NCER management and applicants with a means of understanding how reviewers considered the application against the RFA criteria. Such comments would be particularly useful to NCER management when concerns are raised about a panel's ability to evaluate the most important research qualities, as discussed in the RFA evaluation criteria. In December 2011, after the OIG briefed NCER management on our tentative findings relative to the peer review evaluation forms, NCER's EMS sent e-mail guidance reminding NCER staff of the need to ensure Individual Evaluation forms matched the criteria cited in the RFA. Lack of Clearly Defined "Firewall" Policy Presents Risk to Integrity and Independence of STAR Grant Peer Review Process EPA Order 5700.5A1 requires an objective and impartial examination of applications for Agency grants. According to the NRC's 2003 report, one of the steps that NCER has taken to ensure the independence of its STAR grant review process was establishing a firewall shielding its peer review process from the influence of the project officers and staff who oversee the individual investigator, fellowship, and center awards. Although the NRC report cited the firewall as a key factor supporting its conclusion that NCER had an independent peer review process, NCER has not defined its firewall in internal procedures or other written policies. During our evaluation we received varying accounts of how the firewall 6 The Individual Evaluation form is distributed to peer reviewers electronically at the same time the applications are mailed to the peer reviewers. Reviewers are asked to use this form to provide written comments for each application. 12-P-0864 9 ------- was defined and has been practiced within NCER. However, none of these descriptions have been included in written EPA guidance, policies, or procedures. We did not assess the independence of either peer review panel convened for this RFA. However, the lack of a clearly defined firewall practice at NCER presented a risk to the independence of the peer review process for RFA EPA-G2009- STAR-F1. In the absence of clearly defined roles and responsibilities under NCER's firewall, NCER utilized a process to select reviewers for its June 2010 peer review that, in our view, was inconsistent with descriptions of the firewall included in the 2002 and 2003 National Academies reports. NCER's Firewall is Not Clearly Defined In 2003, NRC cited NCER's firewall as a key factor in concluding that the STAR program had established a rigorous process to evaluate the quality of proposals. However, the firewall concept is not described in NCER guidance, policies, or procedures. During our evaluation, we found limited descriptions of NCER's firewall in publications external to EPA. A 2002 Transportation Research Board (TRB) of the National Academies report7 stated that for NCER's STAR grant program "A firewall is established between staff who administer the independent peer review and staff who write or select RFAs." In a 2003 report, the NRC described NCER's firewall as follows: The program's procedures provide for a firewall that shields the peer-review process from any influence or potential conflicts of the project officers and staff who oversee the individual investigator, fellowship, and center awards. For instance, project officers can provide the names of potential reviewers to the Science Review Administrators (SRAs), also known as peer-review officers, but it is the sole responsibility of the SRAs to select reviewers and to make reviewer assignments. Project officers may attend peer- review meetings as observers but may not provide any comments that would affect peer review.8 We also obtained verbal descriptions of the firewall from NCER staff and managers. While we found some agreement in the descriptions of the firewall provided to us by NCER staff and managers, we also found discrepancies. Further, NCER did not provide us with a definitive description of the firewall outlining the roles and responsibilities of all personnel involved in the peer review process. The Former ASD Director told us that people in NCER had differing 7 Surface Transportation Environmental Research: A Long-Term Strategy. Committee for the Surface Transportation Environmental Cooperative Research Program Advisory Board, National Research Council. Special Report 268. 2002. g According to the NRC report, the process for selecting proposals was described to the committee by the person responsible for managing it and by several STAR project officers, and some committee members had participated in the process previously. 12-P-0864 10 ------- interpretations on how the firewall should work, and how people should be involved in the peer review process. The limited description of the STAR grant program's firewall that was included in 2003 NRC report was central to our interpretation of how the firewall should be practiced, as it was a key factor cited in support of NRC's conclusion that the STAR grant program had established a rigorous and independent peer review process. NCER Personnel Had Different Interpretations of Roles in Vetting and Selecting Reviewers for June 2010 Peer Review For the December 2009 peer review, we found that the SRA followed protocol in selecting the peer reviewer expertise needed to review applications for RFA EPA- G2009-STAR Fl. However, an NCER manager's involvement in the reviewer selection process for the June 2010 peer review, in our view, was inconsistent with descriptions of the firewall in the 2002 and 2003 National Research Council publications. Typically, the SRA assigned to manage the peer review has flexibility in identifying peer reviewers and often does so after reading abstracts or full applications to independently determine the types of reviewers or expertise needed for the panel. As noted in NCER's draft PRD procedures and policies, as well as the 2003 NRC report, the SRA has sole responsibility for panel selection decisions. The SRA, the PRD Director, the NCER Director, and the former ASD Director all acknowledged that it is the SRA's, or PRD's, responsibility to make such decisions. For the June 2010 peer review, the SRA told us that he did not make the final panel selection decisions. For this peer review, the PRD Director worked with the NCER Director and NCER Senior Science Advisor to determine a peer review methodology that included selecting reviewers with specific types of expertise.9 The SRA said he was told by the PRD Director to identify potential reviewers in these specific areas of expertise and then submit the names to NCER's Senior Science Advisor to obtain feedback. The SRA said this was not normal and that NCER had never employed this type of selection process in the past. The Senior Science Advisor told us that, among her other general responsibilities, she reviews all draft RFA's before they are forwarded to EPA's grants office. Therefore, according to the firewall description in the 2002 TRB report, we considered the Senior Science Advisor to be an individual who should be separated from the peer review process. 9 NCER determined that the June 2010 peer review panel should consist of the following types of reviewers: (1) public health specialists; (2) water infrastructure specialists; and (3) "futurists," individuals who exhibited characteristics of innovativeness. 12-P-0864 11 ------- The NCER Senior Science Advisor told us that she evaluated the resumes of potential peer reviewers in consultation with managers in EPA's Office of Water and ORD's Water Quality Research Program, and made recommendations to the SRA about which reviewers might be good candidates for the SRA to contact for the June 2010 peer review. She told us that she did not make final approvals for reviewer selections. The Senior Science Advisor told us that she was asked by the NCER Director to look at the qualifications of potential reviewers in terms of whether or not they could provide an appropriate evaluation of applications for the RFA. As an illustration of her role, ORD provided us with May 2010 e-mails between the Senior Science Advisor and the SRA. In the e-mails, the SRA provided names of potential peer reviewers, along with resumes, and asked the Senior Science Advisor to "review and advise whether they are suitable..for the June 2010 panel. NCER management told us that the process used in selecting reviewers for the June 2010 peer review was appropriate, and that it was acceptable for the Senior Science Advisor to review the suitability of potential reviewers. In contrast, the SRA and PRD Director told us that all peer reviewer selections for the June 2010 peer review were reviewed and approved by NCER's Senior Science Advisor. Internal ORD e-mails sent prior to the June 2010 peer review supported what the SRA and PRD Director had told us. Although the Senior Science Advisor was copied on some of the e-mails, they were not addressed to her. These e-mails characterized the Senior Science Advisor as someone that was approving peer reviewer selections. For example, on April 13, 2010, the PRD Director sent an e-mail to the NCER Director stating that: Based on recommendations from a number of people across the Agency, PRD pulled together a list of 27 potential peer reviewers - - nine reviewers in each of the three expertise categories (public health, water infrastructure and futurists/innovation). We provided that list to [name of Senior Science Advisor] along with the available [resumes]. The e-mail further stated that the Senior Science Advisor had managers in EPA's Office of Water and ORD's Water Quality Research Program review the list of potential peer reviewers, and that these individuals "picked" peer reviewers out of the list of 27 names. The PRD Director informed the NCER Director in the April 13, 2010, e-mail that: PRD is putting together the paperwork for the potential peer reviewers who have received a 'yes' nod from at least one of [name of Senior Science Advisor's] reviewers. A May 11, 2010, e-mail from a PRD Program Analyst to the PRD Director stated: 12-P-0864 12 ------- [name of SRA] has requested two additional people. One for Group 3 [name of potential peer reviewer] and one for Group 2 [name of potential peer reviewer], [name of Senior Science Advisor] approved the selection of [name of potential peer reviewer the SRA identified for Group 3, above]. Her paperwork is in funding at the present time. In the May 19, 2010, e-mail, the PRD Director stated that: When people backed out last week, [name of SRA] moved quicly [sic] to find replacements and get them 'OKed' by [name of Senior Science Advisor], Several sources indicated that NCER's firewall calls for PRD, specifically the SRA, to make reviewer selections and assignments. As noted above, two National Academies reports describe the firewall as having an element of separation between the PRD and other NCER staff and management during the peer review process. In our view, this separation was not maintained by NCER for RFA EPA-G2009-STAR-F1, though NCER management told us that they believed the Senior Science Advisor's role in this particular RFA was appropriate. Without further clarification and written definition of the firewall practice, the independence and integrity of the STAR grant review process are at risk. Impact of Project Officer Concerns on Peer Review Results Points to Management Control Weakness The decision to convene a second peer review panel was due, in part, to non-PRD staff influence over the peer review process that occurred during the December 2009 peer review. After observing the December 2009 peer review panel meeting, the project officer expressed concerns over the lack of diversity in the expertise of peer reviewers to NCER's ASD Director. Although the project officer stated that she expressed her concerns to the ASD Director "shortly after the peer review," the ASD Director told us that he chose not to elevate the concerns about the December 2009 peer review to the NCER Director, letting the process continue on to programmatic review. In January 2010, after a programmatic review was completed, the project officer drafted a decision memo, with consultation and approval from the ASD Director, for the NCER Director. The decision memo noted concerns with the December 2009 peer review, but also recommended nine applications for funding. The memo noted that the review process produced "... a suite of complementary and diverse research projects that tackle high-priority, high-visibility nationally-relevant environmental issues." The memo further noted that the projects would enable key stakeholders "to develop and implement effective policies to reduce public health risks associated with exposure to drinking water, safeguard the quality and availability of surface and underground 12-P-0864 13 ------- sources of drinking water, improve the water infrastructure, and establish health- based measures of management effectiveness." Although the process had produced a seemingly acceptable suite of research projects that had passed both peer and programmatic review, the NCER Director made a decision to void the December 2009 peer review results and subject the applications to another peer review and programmatic review. When we asked the NCER Director about his decision, he said that, based on the concerns stated in the decision memo about the December 2009 peer review, he thought it was best to go back and take a second look at the applications to ensure NCER was funding the best research. According to the NCER Deputy Director, the NCER Director's decision to void the peer review was an appropriate use of discretion and responsibility. He stated that the NCER Director has the authority and responsibility to void peer review results, if necessary. The Senior Associate Director for Grants Competition expressed a similar sentiment, stating that his advice would be to redo the evaluation if there was merit to the concerns expressed. However, the NCER Director did not provide evidence that he took steps to independently substantiate staff concerns prior to making his decision, nor did he document his decision for the record. We believe this undermined the purpose of the firewall, potentially exposing the process to conflicts of interest. In this instance, the project officer and the ASD Directorpositions that, according to National Academies' publications, are not supposed to have influence over the peer review process provided information to the NCER Director that led to changes in review process results. The NCER Director did not substantiate the concerns raised by the project officer and ASD Director; his decision to redo the application review process resulted in two grant proposals not passing the June 2010 peer review that had passed the December 2009 peer review. As evidenced by a grant applicant's letter to EPA's Assistant Administrator for Research and Development, such actions can call into question the integrity of the application review process. Concerns Cited in Support of NCER Management Decision to Void the December 2009 Peer Review Not Valid In a February 2010 decision memo to the NCER Director, the project officer and ASD Director noted two primary concerns with the December 2009 peer review: the panel lacked public health expertise, and the panel did not value innovation. According to the NCER Director, he made the decision to redo the external peer review due to the concerns noted in the decision memo and staff concerns expressed at the February 2010 decision meeting.10 10 The Project Officer told us that a February 2010 decision meeting never occurred, and other staff told us that they could not remember if a meeting ever took place. This was due, in part, to inclement weather in Washington, DC, around the time of the scheduled meeting. 12-P-0864 14 ------- We independently examined the concerns about the December 2009 peer review panel. The panel was mainly comprised of individuals with engineering backgrounds. However, based on our analysis of the peer reviewer biographies provided by ORD, at least 8 of the 26 peer reviewers (31 percent) from the December 2009 peer review, including some of the engineers, had public health expertise. Therefore, the panel did not lack public health expertise, as stated in the decision memo. We also determined that "innovation" was not an area of expertise but rather a characteristic of the research proposals that, per the evaluation criteria identified in the RFA, should have been evaluated by peer reviewers with appropriate technical expertise. Based on our analysis, we found no evidence suggesting that the SRA did not perform his duties with respect to identifying and selecting appropriate panel members. Therefore, in our view, the NCER Director's decision to void the December 2009 peer review results was largely unfounded. NCER's STAR Grant Review Process Needs Improved Management Controls In general, NCER lacked sufficient procedures and management controls to ensure the independence of its STAR grants competition process. EPA Order 5700.5Al, the RFA, and the applicable regulations cited in the RFA were the only formal policy and procedures documents provided to OIG by NCER that governed its STAR grants award process. The other documents provided were either informal guidance (provided to NCER personnel via e-mail, existed in draft form, etc.) related to specific portions of the STAR grant review process (e.g., peer review) and not to the entire process, or did not include specific procedural steps for NCER staff. None of the Agency's documents formalized NCER's firewall practice, outlined procedures for substantiating concerns regarding peer review results, identified a process for re-evaluating applications, or specified how decisions to alter or change the typical review process should be documented. A lack of formalized procedures was particularly evident with respect to NCER's decision to void the December 2009 peer review results. For example: The NCER Director was not informed of staff concerns about the December 2009 peer review until the application review process had proceeded nearly to completion. The NCER Director appeared to make a decision to redo the review process without consulting key personnel, including NCER's EMS, the ASD Director, the PRD Director, EPA's Senior Associate Director for Grants Competition (or any other personnel in EPA's OGD), or Agency ethics officials. The NCER Director proceeded to void the December 2009 peer review results without independently investigating or validating the concerns 12-P-0864 15 ------- noted about that peer review, or providing any documented justification for his decision to void the results of the review. According to the Senior Associate Director for Grants Competition, if issues existed that affected the propriety, quality, objectivity, or results of the review, it is advisable to redo the peer review. However, such issues should be demonstrated, substantiated, and supported before a decision is made to void peer review results. NCER's Decision to Void the December 2009 Peer Review Resulted in Delays and Additional Costs NCER management's decision to conduct a second peer review in June 2010 cost NCER over 250 hours of staff time, and over $9,250 in payments for peer reviewers and application processing. The decision to utilize these additional resources was made without demonstrating that a condition warranting such a use of resources actually existed. In addition, the decision delayed funding decisions by approximately 6 months. Four of the eight applicants interviewed by OIG commented on the significant delays in NCER's review process. One applicant noted that the length of the review process could have impacted the costs associated with the research proposal because costs change over time. Another applicant stated that he was upset and discouraged because his group missed an opportunity to apply for another grant due to the delay in EPA's response. Conclusions Improved management controls are needed to ensure the independence and effectiveness of EPA's STAR grants review process. We did not find evidence of fraud or intentional manipulation of the application review process for RFA EPA- G2009-STAR-F1. However, the management control weaknesses we identified put the Agency at risk for fraud or other manipulations of its STAR grant program. At a minimum, the management control weaknesses have the potential to damage the reputation of the STAR grant program as an independent and rigorous process that provides funding for high-quality research. Recommendations We recommend that the Assistant Administrator for Research and Development: 1. Take steps to make EPA's decision to issue a class exception from 40 C.F.R 40.150 for its STAR grants program known to the public through NCER's website. 2. Direct NCER to develop and/or update written procedures so that the review template is consistent with evaluation criteria published in the STAR Grant RFA prior to releasing the review forms to panelists. 12-P-0864 16 ------- 3. Direct NCER to establish written procedures for administering the STAR grant review process. Such procedures should include: a. A description of management controls needed to ensure applicable regulations and policies are adhered to, and how such controls will be implemented. b. Descriptions of roles and responsibilities of the various NCER divisions and personnel involved in the STAR grant review process. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. c. A clearly defined policy for NCER's firewall, including how it will be implemented and practiced. 4. Direct NCER to develop and/or update written procedures that provide guidance on voiding peer reviews or conducting re-reviews for STAR grant competitions, including: a. How to appropriately evaluate and document the need for a re-review. b. Descriptions of how and when to involve parties within and outside of NCER. c. Appropriate timelines for each step. Agency Comments and OIG Evaluation The Agency generally agreed with our conclusion that improved management controls are needed to ensure the independence and effectiveness of NCER's STAR grant program. However, the Agency believed that we misinterpreted statements attributed to EPA staff members, and mischaracterized NCER's firewall. Further, the Agency stated that one chapter 2 section heading could be taken out of context. We re-examined each of these areas and made changes to the final report to state that NCER has not clearly defined its firewall practice. We revised the heading in question to clarify that our findings are specific to NCER's STAR grant review process. Further, we revised the report to state that NCER personnel had different interpretations of their roles in vetting and selecting reviewers for the June 2010 peer review. However, we continue to believe that NCER's Senior Science Advisor's role in the peer review process was not consistent with previously published descriptions of the firewall. Several sources indicated that NCER's firewall calls for PRD, specifically the SRA, to make reviewer selections and assignments. However, the SRA and PRD Director told us that they did not make final reviewer selections for the June 2010 peer review, and that reviewers were approved by the Senior Science Advisor. We have included language in the report to reflect NCER management's disagreement with this characterization. Further, in support of the Senior Science Advisor's 12-P-0864 17 ------- assertion that she was not providing reviewer approvals, we have included reference to an e-mail stating that she assessed the "suitability" of two prospective peer reviewers submitted to her by the SRA for the June 2010 panel. However, we have also included references to a number of e-mails characterizing the Senior Science Advisor's role in the June 2010 peer review process as someone that was involved in reviewing and approving peer reviewer selections. We also made it clear that the Senior Science Advisor was copied on some of these e-mails, but that they were not addressed to her. Although the Agency's response contained alternative recommendations, the Agency noted in the exit conference that it agreed with the intent of our recommendations.11 The Agency generally agreed with Recommendation 1 as written. For the other chapter 2 recommendations the Agency generally agreed to develop the requisite internal controls but suggested alternative recommendations. For recommendations 3a and 3b in the final report, NCER suggested we make an alternative recommendation to develop overarching, NCER-wide operational guidance for staff responsibilities, communication, and training. While we believe this could be appropriate for NCER management to consider as it addresses our recommendations, we believe that our recommendation to develop STAR grant- specific procedures focusing on the issues we found is more appropriate given the scope of our evaluation. We did not make changes to recommendations 3a and 3b. The Agency also proposed alternative recommendations that met the intent of some of our draft report recommendations. Therefore, we included the Agency's alternative recommendations as recommendations 2 and 4 in the final report. These recommendations replaced recommendations 2c, 2d, 2e, and 2f from the draft report. The draft report recommendations are included in appendix E. The only change we made to the Agency's suggested alternative recommendations was to clarify that the recommendations are specific to the STAR grant program. All recommendations in Chapter 2 are considered unresolved until such time as the OIG and EPA reach agreement on required actions and planned completion dates. In its 90-day response to the final report, EPA should provide a description of the Agency's planned corrective actions, as well as the estimated or actual milestone completion dates for each action. The Agency's written comments and OIG's response are in appendix E. 11 During the exit conference, ORD said that their "alternative recommendations" were not alternatives to OIG's original recommendations but proposed actions for addressing OIG's recommendations. 12-P-0864 18 ------- Chapter 3 Improved Program Guidance, Management Controls Can Aid NCER's Communications With Applicants NCER did not communicate with all applicants for RFA EPA-G2009-STAR-F1 in a transparent, appropriate, accurate, and timely manner. For example, NCER was not transparent in communicating: Its decision to conduct a second review The expected delays resulting from the second review Whether the results sent to applicants were based on the December 2009 or June 2010 peer review. NCER's declination letters did not adhere to Agency policy because they did not sufficiently explain why applicants were not selected, nor did they inform applicants of their option to request a debriefing. NCER informally communicated results to some applicants prior to final selection decisions. Further, some letters NCER sent to applicants contained incorrect contact information, and some were sent about a week later than the timelines specified by EPA policy. NCER lacks sufficient guidance or procedures that formalize lines of accountability for communicating with applicants during its review process. In particular, staff were unsure as to who was responsible for organizing, preparing, signing, maintaining, and sending notifications after the peer review and programmatic review. Also, improved management controls are needed to ensure that NCER's communications with grant applicants are transparent and accurate, and adhere to Agency policy and guidance. Ineffective communications with grant applicants can potentially damage the reputation of the STAR grant program as an independent and rigorous process that provides funding for high quality science. NCER Not Transparent in Communicating Decision to Conduct Second Review for RFA In an April 2009 memo, the EPA Administrator expressed the importance of transparency and openness in conducting EPA operations. According to the memo, this requires EPA to (1) remain open and accessible to those representing all points of view, and (2) be responsible for decisions by soliciting the views of those who will be affected by these decisions. The Administrator also stated that she believes transparency will enhance the credibility of the Agency, boost public trust, and improve the quality of EPA's decisions. For this RFA, NCER was not transparent in communicating its decision to conduct a second review and the expected delays this would cause were never stated in any formal communications. Rather than being proactive in notifying all 12-P-0864 19 ------- applicants about the decisions and delays in the review process, NCER only communicated the changes to its process to those applicants who made inquiries. According to EPA's Senior Associate Director for Grants Competition, all applicants should be notified if a program office conducting a competition decides to redo the evaluation review of applications (i.e., conduct a new peer review). However, in an e-mail from the NCER Director, the project officer for this RFA and the ASD Director were directed to respond to the e-mail inquires with the following statement: there has been an unanticipated delay in our process, and we are working to get back on track. NCER staff responded to the e-mail inquires with the suggested text from the NCER Director or other vague explanations as to why there were delays in the review process. For instance, NCER told applicants that delays in the review process were due to "re-processing" applications, "a bureaucratic process glitch," and "unusual setbacks" in processing the applications. Further, NCER did not explain the changes that occurred during its review process for this RFA in its declination letters. In particular, these letters did not state that a second review was conducted or that the second peer review was a mail review12 rather than a panel review. In addition, the peer review Individual Evaluation forms included with the declination letters sent to applicants after the June 2010 peer review were dated 2009, even though the review was completed in 2010. In our view, using a 2009 date on an Individual Evaluation form that was generated in June 2010 may have been misleading to applicants. As a result of the Agency's lack of transparency during the review of this RFA, some applicants were unclear about the review process and others were confused about why they had not heard from the Agency for an extended length of time or by the anticipated timeline. For example: Three applicants questioned why they received a notification from EPA, stating that their application passed after receiving a declination letter. One applicant filed a FOIA request to obtain additional information because staff did not provide sufficient information to explain why the review process for this RFA had to be redone. Two applicants assumed the delays in the process or multiple notifications were a result of an administrative or automated systems problem. 12 For this RFA, NCER held a face-to-face panel meeting for the December 2009 peer review and a mail review for the June 2010 peer review. According to SRA for this RFA, the mail review is based on individual reviewers and individual evaluations, and there is no group discussion of the results. According to NCER personnel, NCER does not have guidance outlining its process for conducting mail reviews. NCER staff told us that they conducted a mail review for the June 2010 peer review because they believed it would be faster than another face-to-face peer review. 12-P-0864 20 ------- One applicant believed that the declination letter she received after the December 2009 peer review was the official result of NCER's review of her application. She was not provided the results of the June 2010 peer review, and was not aware that a second peer review had been conducted. Two applicants incorrectly assumed that the peer review results attached to the June 2010 declination letter were based on the first round of reviews, and were unaware that a second peer review had been conducted. Fifteen applicants contacted NCER to check on the status of the review. Four of the 15 applicants who contacted NCER to check on the status of the review did so on multiple occasions. NCER's Formal Communication With All Grant Applicants Did Not Adhere to Agency Policy, Guidance, and Guidelines We reviewed documented communication between NCER and grant applicants for RFA EPA-G2009-STAR-F1 and found that some of NCER's formal communications with applicants did not follow applicable policy and guidance. NCER sent formal13 notifications, in the form of a letter or e-mail, to applicants regarding selection decisions after the December 2009 and June 2010 peer reviews, and after the August 2010 final decision meeting. Examples of formal notifications sent to applicants included: Peer review declination letters that informed applicants that EPA was unable to fund the proposed research Review status letters that informed applicants that the proposed research passed peer review and requested past performance and reporting history information from the Lead Principal Investigator(s) Notice of award letters that informed applicants that the proposed research was being recommended for funding Final declination letters that informed applicants that EPA was unable to fund the proposed research based on scientific merit and relevancy to EPA's programs Formal Communication With Unsuccessful Applicants Did Not Adhere to Agency Policy and Guidance According to EPA Order 5700.5Al, EPA program offices conducting grant competitions must provide unsuccessful applicants with written or e-mail notification. These notifications must generally explain the reasons why the application was not eligible for award consideration based on the threshold eligibility review or score after an evaluation of its application against the 13 The OIG's use of "formal" communication included any official correspondence with applicants in the form of a letter, e-mail, phone call, or personal visit. For this evaluation, information that was provided in an EPA notification that was sent to all applicants as a result of a selection decision, or provided to all applicants during a conference call or a professional meeting, was considered "formal" communication. 12-P-0864 21 ------- selection factors in the announcement. Based on the PRD draft procedures and policies guidance document for the STAR grant peer review process, unsuccessful applicants will be mailed declination letters, along with the relevant Individual Evaluation form, after the peer review and final decision meeting. EPA Order 5700.5A1 also states that these notifications must advise the applicant that he or she may request a debriefing of the basis for the ineligibility determination or selection decision. According to EPA Order 5700.5Al, debriefings may be done orally or in writing, and will be limited to explaining why the applicant was not considered for award. Prior to PRD sending notifications to applicants after the December 2009 peer review, the NCER EMS sent guidance stating that notifications sent to unsuccessful applicants must provide an option to request a debriefing. The guidance noted that providing applicants with a copy of the peer review results does not constitute a debriefing. If applicants request a written debriefing, the guidance from the EMS also noted that the letter serving as the written debriefing needs to state that it will serve as a debriefing. After receiving a debriefing, EPA Order 5700.5Al states that an unsuccessful applicant may file a written dispute with the Agency and must be provided with reasonable access to Agency records relevant to the dispute in a manner consistent with FOIA standards. We found that the declination letters sent to applicants for RFA EPA-G2009- STAR-F1 did not follow applicable policy and guidance. The declination letters NCER sent to applicants after the December 2009 and June 2010 peer reviews did not sufficiently explain to the applicants why their application was not selected for an award. Further, these letters, as well as notifications sent after the final selection meeting, did not inform the applicants of their option to request a debriefing. As a result, applicants were not advised of an opportunity to obtain additional information on the reasons why their application was not selected for an award. For example, during interviews with eight applicants who submitted applications under this RFA, we found that five of the applicants were not aware of the option to request a debriefing. Not advising applicants of such an opportunity potentially hinders the applicants' ability to improve their applications for future EPA grant competitions. PRD staff told us that one applicant requested a debriefing for this RFA, but the debriefing conducted by PRD did not follow Agency policy and guidance. The written debriefing did not: State that it would serve as the applicant's debriefing Explain why the applicant was not selected for award Provide the applicant with information on the strengths and weaknesses of his/her application in terms of the specific evaluation criteria used in the competition Explain the dispute process and dispute contact 12-P-0864 22 ------- Instead of informing the applicant of the opportunity and provisions for filing a dispute, the written debriefing informed the applicant of the opportunity and provisions for filing an FOIA request. When we asked the applicant about the written debriefing provided to her by PRD, the applicant was unclear whether a formal debriefing had actually occurred. She suggested that a discussion she had with the NCER Director could have been the formal debriefing. However, according to the NCER Director, the discussion with the applicant was to explain what had happened during the review process for this RFA and not to explain the applicant's results or serve as the applicant's debriefing. Formal Communication With Successful Applicants Did Not Adhere to Guidelines in RFA The RFA states that applicants passing peer review will be sent a notification, which we referred to as a review status letter, requesting past performance and reporting history information from the lead Principal Investigator(s). NCER sent review status letters to each of the 11 applicants who passed the December 2009 peer review. However, after the June 2010 peer review, 9 of the 34 applicants (26 percent) who passed the peer review did not receive review status letters from NCER as per the RFA guidelines. These nine applicants also passed the December 2009 peer review. According to PRD staff, applicants who passed both peer reviews were not sent review status letters after the June 2010 peer review due to the format of the computer system. The computer system did not generate another notification for these applicants because it had already sent notifications after the December 2009 peer review. As a result, these applicants were not informed that their earlier review status letter had been voided or that their application had been subjected to and passed the peer review again. Further, these applicants were not asked to provide updated past performance and reporting history information even though about 6 months had passed. The Agency's failure to request the applicants' updated information meant that NCER was evaluating past performance and reporting history for some applicants that was 6 months old, while others were being evaluated on information that was current to June 2010. As noted by ORD, "An applicant's 'Past Performance and Reporting History' provides information essential to evaluating the potential performance success of an applicant." While we did not find that this condition adversely impacted applicants for this RFA, all applicants should be treated equally in terms of the information on which they are being evaluated. 12-P-0864 23 ------- NCER's Informal Communication With Applicants Presented Risk of Providing Applicants With Inaccurate Information Between December 2009 and October 2010, NCER informally14 communicated with about half of the 72 applicants who were eligible under this RFA.15 NCER informally communicated the status of the review process or whether applications were still being considered for awards through phone conversations and e-mails. While this informal communication is not explicitly prohibited by EPA policy, we believe such communication increases the risk that applicants will be provided erroneous or incomplete information. Further, informal communications with applicants could create a perception of EPA favoring one applicant over others. EPA Order 5700.5A1 states that, during the competition process, individuals can communicate with applicants under the condition that the advice or information will not give applicants a competitive advantage. After the submission of applications, EPA Order 5700.5A1 also states that EPA personnel may have limited communications with applicants for the purpose of clarifying certain aspects of the application relating to threshold eligibility factors, for partial funding purposes, or to resolve minor or clerical errors. According to EPA's Senior Associate Director for Grants Competition, it is appropriate for NCER staff, such as a project officer, to respond to inquiries from applicants seeking updates on the status of the review process. However, he said that communication with applicants should only indicate where the Agency is in the review process and not provide any unofficial results related to specific applications. Between June and September 2010, NCER informally communicated the results of the review to 25 different applicants. For instance, after a June 2010 programmatic review had been completed, NCER sent five applicants an informal e-mail notifying them that their proposals were still "in the running." However, this occurred about 3 weeks before the NCER Director signed the memorandum containing the final recommendations for funding (i.e., NCER's official funding recommendations). Similarly, about a week before the official results, NCER sent informal e-mails to 12 applicants notifying them that their proposals were "no longer in the running." One applicant was notified by NCER after the June 2010 programmatic review that her application was "still very much in the running." About 2 weeks later, but still prior to when official funding recommendations had been made, NCER sent an e-mail to that same applicant stating that the proposal was "very unlikely to be awarded." By providing informal results to applicants in advance of those results becoming official, NCER could have potentially led applicants to expect an award when, in fact, their application was not selected at the final decision meeting. Similarly, such communications could have led some applicants to believe their application 14 The OIG's use of "informal" communication included any unofficial correspondence with applicants in the form of a letter, e-mail, phone call, or personal visit that does not classify as "formal" communication defined above. 15 This information is based on the records or documentation NCER staff provided to OIG. 12-P-0864 24 ------- was not going to receive an award when, in fact, it was still being considered for funding. Inaccurate Contact Information Provided to Applicants Had Potential to Impact Deadlines, But Did Not Impact Applications for RFA We found some inaccuracies in the formal NCER communications that we reviewed for this RFA. Specifically, the review status letters NCER sent applicants after the December 2009 peer review contained spelling errors, including the misspelling of the project officer's last name in the signature and e-mail address. However, we did not find that these inaccuracies had a significant impact on the applicants for the RFA. Six applicants (about 55 percent of applicants who passed the December 2009 peer review) indicated problems sending the requested past performance and reporting history information to the project officer due to the incorrect contact information in the review status letters. One applicant's submission was delayed 11 days while the applicant sought correct contact information for the project officer. We also noted that the SRA's first name in the signature of the declination letters sent after the June 2010 peer review was spelled incorrectly. The inaccurate contact information that we found in the review status letters was potentially important because these letters requested applicants to send past performance and reporting history to the project officer within 3 weeks of the receipt in order to be considered for the programmatic review. Even though some applicants indicated problems sending the requested information to the project officer, we did not find that the inaccurate contact information in these letters prevented any applicants from providing required information to EPA within the required timeframe or from being considered in NCER's programmatic review. Further, we did not identify any impacts as a result of the SRA's first name being spelled incorrectly. However, the inaccuracies we noted have the potential to impact the ability of applicants to contact appropriate Agency staff and to submit requested information to EPA prior to stated deadlines. NCER Did Not Communicate Peer Review Results to Applicants in a Timely Manner for RFA EPA Order 5700.5A1 states that the program office conducting the competition must send unsuccessful applicants a notification within 15 calendar days of the Agency's selection decision.16 Further, according to NCER's EMS as well as a summary document NCER provided to us describing its STAR grant award 16 NCER's STAR grant award process includes a programmatic review, which occurs after the peer review meeting and before the decision meeting. The purpose of the programmatic review is to assure an integrated research portfolio for the Agency and help determine which applications to recommend for award. As a result, we did not consider the programmatic review to be a selection decision point. Therefore, the next selection decision point after the peer review does not occur until the decision meeting, where decisions are made based on the results of the peer and programmatic review. 12-P-0864 25 ------- process, NCER sends declination letters, along with notifications to those who passed peer review, to applicants within 15 days of the peer review. For this RFA, NCER sent declination and notice of award letters after the final selection meeting within the time specified in the Agency's policy and guidance. However, NCER did not communicate the results of the peer review in a timely manner. NCER's letters communicating peer review results were not sent to applicants within 15 days of either the December 2009 or June 2010 peer review selection decisions. The letters sent after the December 2009 peer review were 5 days past the time requirement, and the letters sent after the June 2010 peer review were 8 days past the time requirement. However, we did not identify any impacts as a result of NCER not complying with Agency policy and guidance. Process for Communicating with Grant Applicants Not Well Defined by Agency Policy and Guidance NCER lacks sufficient guidance or procedures to formalize lines of accountability for communicating with applicants during its review process. As a result, staff were unsure who was responsible for organizing, preparing, signing, maintaining, and sending notifications after the peer review and programmatic review. For instance, personnel were unsure what notifications were sent to applicants during the review process for this RFA. Specifically, the PRD Director, the director of the division responsible for sending notifications to applicants after peer review, told the project officer that formal letters were not sent to applicants after the December 2009 peer review when in fact they were. E-mails between NCER staff also showed that staff were unsure about who was responsible for notifying applicants and how applicants who passed the December 2009 peer review but did not pass the June 2010 peer review should have been informed. Further, NCER staff were unsure who was responsible for maintaining the records for those applicants who did not pass the eligibility step per Section III of the RFA. NCER does not have sufficient management controls to ensure that its communications with grant applicants are transparent and accurate and adhere to Agency policy and guidance. In particular, NCER lacks an effective process for staff to track and monitor communications with grant applicants during the review process. As a result, some of the formal notifications sent to grant applicants for this RFA did not contain required information or were not sent in a timely manner. Some of NCER's informal communications provided applicants with incomplete information regarding the status and results of the review. Further, NCER did not maintain records of the number of applications who did not pass the eligibility review for this RFA and the notifications that NCER sent to these applicants. 12-P-0864 26 ------- Conclusions Improved guidance and management controls are needed to ensure that NCER effectively and appropriately communicates with grant applicants during its STAR grants competition process. The management control weaknesses we identified regarding NCER's communications with applicants also have the potential to damage the reputation of the STAR grant program. For example, the lack of transparency that NCER exhibited in its communications for this RFA prompted one grant applicant to call into question the integrity of the entire review process. Effective and appropriate communications with applicants can serve to avoid such perceptions while adding to the credibility of the STAR grant program's process and its funding decisions. Recommendation We recommend that the Assistant Administrator for Research and Development: 5. Ensure that NCER establishes written procedures for communicating with STAR grant applicants. Such procedures should include: a. A process for staff to effectively track and monitor communications with applicants. b. Guidance for ensuring that appropriate information is included in notifications to applicants. c. Descriptions of roles and responsibilities of the various NCER divisions and personnel involved in communications with applicants. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. d. Annual reviews of the adequacy of internal controls over its communications with applicants. Agency Comments and OIG Evaluation The Agency generally agreed with our conclusion that improved management controls are needed to ensure that NCER effectively and appropriately communicates with grant applicants during its STAR grants competition process. However, the Agency believed that we had mischaracterized the "Past Performance and Reporting History" it receives from applicants and the potential effects of not having the most up-to-date history. The Agency also believed that we had misinterpreted a statement from an EPA staff member and did not reflect the individual's correct title. Further, the Agency stated that one chapter 3 section heading could be taken out of context. We reviewed our characterization of the past performance and reporting history and clarified the statements in this section of the report to avoid confusion with the initial STAR grants eligibility determinations. We do not agree that we have misinterpreted a statement from the Acting Director of the Applied Science Division, but we have revised the report to 12-P-0864 27 ------- reflect his correct title. Finally, we revised the section heading in question to clarify that our findings are specific to RFA EPA-G2009-STAR-F1. The Agency generally agreed with recommendations 5b and 5d (numbered as 3b and 3d in the draft report). For recommendation 5a (3a in the draft report), the Agency provided a suggested alternative recommendation to provide staff training to ensure staff are aware of operational procedures and their roles and responsibilities. However, we believe that the OIG recommendation, as written, is appropriate and necessary to address a basic management control weakness in ORD's communications with grant applicants. Once the communications policy is established, a training program may be appropriate for ensuring that the policy is understood and followed. However, we do not believe that staff development and training substitutes for the establishment of a process of tracking and monitoring communications with grant applicants. The Agency believed that the recommended actions in recommendation 5c (3c in the draft report) were no longer necessary due to its suggested recommendation for recommendations 2a, 2b, and 5a. However, in light of our decision not to accept this suggested alternative recommendation, we continue to believe that recommendation 5c is appropriate. All recommendations in chapter 3 are considered unresolved until such time as the OIG and EPA reach agreement on required actions and planned completion dates. In its 90-day response to the final report, EPA should provide a description of the Agency's planned corrective actions, as well as the estimated or actual milestone completion dates for each action. The Agency's written comments and OIG's response are in appendix E. 12-P-0864 28 ------- Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFITS (In $000s) Rec. No. Page No. Subject Status1 Planned Completion Action Official Date Claimed Amount Agreed-To Amount 16 Take steps to make EPA's decision to issue a class exception from 40 C.F.R 40.150 for its STAR grants program known to the public through NCER's website. 16 Direct NCER to develop and/or update written procedures so that the review template is consistent with evaluation criteria published in the STAR Grant RFA prior to releasing the review forms to panelists. 17 Direct NCER to establish written procedures for administering the STAR grant review process. Such procedures should include: a. A description of management controls needed to ensure applicable regulations and policies are adhered to, and how such controls will be implemented. b. Descriptions of roles and responsibilities of the various NCER divisions and personnel involved in the STAR grant review process. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. c. A clearly defined firewall policy, including how it will be implemented and practiced. 17 Direct NCER to develop and/or update written procedures that provide guidance on voiding peer reviews or conducting re-reviews for STAR grant competitions, including: a. How to appropriately evaluate and document the need for a re-review. b. Descriptions of how and when to involve parties within and outside of NCER. c. Appropriate timelines for each step. 27 Ensure that NCER establishes written procedures for communicating with STAR grant applicants. Such procedures should include: a. A process for staff to effectively track and monitor communications with applicants. b. Guidance for ensuring that appropriate information is included in notifications to applicants. c. Descriptions of roles and responsibilities of the various NCER divisions and personnel involved in communications with applicants. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. d. Annual reviews of the adequacy of internal controls over its communications with applicants. Assistant Administrator for Research and Development Assistant Administrator for Research and Development Assistant Administrator for Research and Development Assistant Administrator for Research and Development Assistant Administrator for Research and Development 0 = recommendation is open with agreed-to corrective actions pending C = recommendation is closed with all agreed-to actions completed U = recommendation is unresolved with resolution efforts in progress 12-P-0864 29 ------- Appendix A Key Positions With Oversight Responsibilities and Accountability for RFA EPA-G2009-STAR-F1 Stage of review process Position Summary of responsibilities for RFA EPA-G2009-STAR-F1 Eligibility Review Eligibility Contact Responds to eligibility questions related to Section III of the RFA from applicants while the RFA is open. An eligibility screen is performed at the close of the RFA to determine which applications are eligible for award consideration. PRD Director The PRD Director and staff are in charge of the front-end processing of grant applications, soliciting suggestions for and managing peer review panels, compiling and forwarding peer review results to the project officer, and sending declination letters to applicants that do not pass NCER's review process. The PRD Director also assigns a SRA to each RFA. External Peer Review SRA Selects peer review panel members, ensures execution of contracts to obtain reviewer services, monitors for potential conflicts of interest, assigns applications to reviewers, works with PRD's Contractor Monitor to send all necessary materials to reviewers, schedules the peer review meetings, leads and conducts peer review meetings, and collects and finalizes scores from peer review meetings. Also responsible for providing the results of the peer review to the PRD Director and project officers, working with PRD's Contractor Monitor to send declination packages to applicants who did not pass the peer review or programmatic review, and provide debriefings to applicants when requested. PRD Contractor Monitor Oversees NCER contractors. The NCER contractors receive and record the applications after the RFA closes, provide the application abstracts and applications to the SRA, and prepare and send notifications (i.e., declination and review status letters) to applicants regarding the results of the peer review, on behalf of the SRA or project officer. Programmatic ASD Director Sets the RFA budget and selects a project officer for each RFA. After the RFA is written, the ASD Director is the first person to review and approve the solicitation. After the programmatic review, the ASD Director reviews and approves the decision memo and helps to identify whether selected projects relate to EPA's programmatic needs. The ASD Director also attends a decision meeting with Division Directors, NCER Director, appropriate project officer and SRA, and NCER's Senior Science Advisor. Review Project Officer Reviews and comments on the RFA prior to publication, responds to technical questions from applicants while the RFA is open, observes the peer review panel, collects past performance and reporting history information from Principal Investigators, organizes and leads the programmatic review, writes a decision memorandum that identifies the applications recommended for funding, communicates with applicants after the peer review and while final funding decisions are being processed. Final Funding Decision NCER Director Makes final funding decisions. Recommendations for funding are provided to the NCER Director in a decision document and discussed with staff during a decision meeting. The NCER Director makes final decisions shortly after the decision meeting. Senior Science Advisor Advises the NCER Director and the Division Directors on the strategic direction of the extramural research at ORD. Reviews all of the RFA drafts before they are sent to the Grants Office, and all funding recommendations after applications have gone through the programmatic review. Other EMS Reviews RFAs to ensure they follow applicable regulations and policies. Point of contact for legal and administrative issues related to NCER grant competitions, developing internal guidance, and serves as the liaison with the servicing officers, such as OGD. GCA Point of contact for implementing EPA Order 5700.5A1. The GCA must also review and concur with the RFA before it can be published. Conflicts of interest should be addressed in consultation with the GCA and/or OGC attorneys. Source: OIG developed summary of roles and responsibilities from information provided by EPA managers and staff. 12-P-0864 30 ------- Appendix B Timeline of Events for Activities That Occurred During RFA EPA-G2009-STAR-F1 Date Activity/event May 18, 2009 RFA opening date August 17, 2009 RFA closing date December 1-2, 2009 First peer review meeting (December 2009 peer review) Number of applications reviewed: 72 Results: 11 applications passed December 9, 2010 PRD contractor requests past performance and reporting history from applicants that passed the December 2009 peer review December 22, 2009 PRD contractor sends declination letters to applicants who did not pass the December 2009 peer review. January 20, 2010 Date of programmatic review Number of Applications Reviewed: 11 Results: 9 applications passed February 8, 2010 Scheduled decision meeting to discuss decision memo. The meeting did not happen due to the snow days. February 16, 2010 Re-scheduled decision meetingconfirmation of this meeting cannot be determined. February 16, 2010 Director discussed the peer review issue with the ASD Director (based on director's recollection of events; no corroborating documentation available). February 16, 2010 Project officer informed that peer review will be redone or "supplemented" with panelists of more varied expertise. Project officer asked to provide detailed list of panelist nominations. February 16, 2010 Director held discussions with the PRD Director and the EMS (based on director's recollection of events; no corroborating documentation available). February 17, 2010 Project officer provided list of peer reviewer suggestions to ASD Director. February 18, 2010 EMS e-mailed PRD Director about conducting a second peer review and expressed concerns about the planned approach that was explained to him by ASD Director. February 19, 2010 PRD Director responded to e-mail from EMS to clarify the approach discussed with and approved by the NCER Director. February 23,2010 Project officer provided list of peer reviewer suggestions to PRD. March 4, 2010 Meeting scheduled with NCER Director, PRD Director, EMS, and SRA to discuss possibility of another peer reviewconfirmation of this meeting cannot be determined. March 4, 2010 On March 4, or shortly after, the NCER Director decided to conduct second peer review. March 16, 2010 Project officer told to expect peer review results from second review on May 19. March 29, 2010 PRD Director informs SRA that she will forward prospective reviewers to the Senior Science Advisor. April 12, 2010 PRD Director tells project officer that PRD is working very hard to get final scores by May 19. 12-P-0864 31 ------- Date Activity/event April 13, 2010 PRD Director requests approval from NCER Director that approach for selecting mail reviewers is acceptable. April 14-May 27, 2010 Peer reviewers for second peer review were selected and contacted. May-June 2010 Second peer review conducted (June 2010 peer review). June 9, 2010 Final peer review results from June 2010 peer review determined Number of applications reviewed: 72 Results: 34 applications passed June 15, 2010 PRD contractor requests past performance history information from applicants that passed the June 2010 peer review. June 28, 2010 Programmatic review meeting Number of applications reviewed: 34 Results: 8 applications passed June 30, 2010 PRD contractor sends declination letters to applicants that did not pass the June 2010 peer review. August 18, 2010 Final decision meeting Results: 8 proposals recommended for funding NCER Director concurred with recommendations August 26, 2010 Date of supplemental memo containing final recommendations for funding. August 27, 2010 NCER Director signed supplemental memo containing final recommendations for funding. August 30 and September 1, 2010 Project officer sends official memos to applicants that were selected for awards. September 8, 2010 PRD contractor sends declination letters to applicants that did not pass the second programmatic review. January-August 2011 EPA disburses funds for all eight projects (total of $4,196,385 disbursed). Source: OIG developed timeline from documentation, e-mails, and descriptions provided by EPA managers and staff. 12-P-0864 32 ------- Appendix C Detailed Scope and Methodology Based on a request from the EPA Assistant Administrator for Research and Development, we sought to determine the factual basis of concerns raised in a grant applicant's August 2010 letter about NCER's management of the review process for RFA EPA-G2009-STAR-F1. The concerns were primarily related to the reasons behind NCER's decision to void the results of its technical peer review of applications and a lack of timely, transparent communication on the part of NCER. Therefore, our primary objectives were to determine whether ORD (1) followed applicable federal and EPA policies and procedures in managing the technical peer review panel process for proposals submitted, and (2) communicated with grant applicants in an accurate, timely, appropriate, and transparent manner regarding the status of their proposals. To independently confirm our understanding of the issues raised regarding ORD's review and management of grant proposals under RFA EPA-G2009-STAR-F1, we: Interviewed the concerned grant applicant as well as the other applicant who passed the December 2009 peer review but not the June 2010 peer review Reviewed documentation of all correspondence between the concerned grant applicant and NCER Interviewed key NCER personnel, including the NCER Director, NCER EMS, PRD Director, SRA, and project officer Developed a timeline of events relating to RFA EPA-G2009-STAR-F1 between May 2009 and September 2010 To determine whether ORD followed all applicable policies, procedures, and guidance during its review of proposals under RFA EPA-G2009-STAR-F1 and its subsequent communications with applicants, we obtained and reviewed data and information related to our two objectives, including: Applicable EPA, OMB, and other federal policies, procedures, and guidance, including: o 40 C.F.R Part 30 (Uniform Administrative Requirements for Grants and Agreements with Institutions of Higher Education, Hospitals, and Other Non- profit Organizations) o 40 C.F.R Part 31 (Uniform Administrative Requirements for Grants and Cooperative Agreements to State and Local Governments) o 40 C.F.R Part 40 (Research and Demonstration Grants) o OMB Circular A-21 (Cost Principles for Educational Institutions) o OMB Circular A-l 10 (Uniform Administrative Requirements for Grants and Other Agreements with Institutions of Higher Education, Hospitals and Other Non-Profit Organizations) o OMB Circular A-122 (Cost Principles for Non-Profit Organizations) o EPA Order 5700.5A1, Policy for Competition of Assistance Agreements o EPA Order 3120.5, Addressing Research Misconduct o EPA NCER Peer Review Division Policies and Procedures o RFA EPA-G2009-STAR-F1 (Advancing Public Health Protection through Water Infrastructure Sustainability) 12-P-0864 33 ------- EPA websites and other sources of criteria, including: o EPA grants website on the EPA Intranet o EPA Principles of Scientific Integrity o Memo from EPA Administrator on Scientific Integrity o Memo from EPA Administrator on Transparency in EPA's Operations o EPA Competition Guidance for Office of the Administrator Project Officers STAR grant budget and expense data for RFA EPA-G2009-STAR-F1 Other STAR grant financial documents related to RF A EPA-G2009-STAR-F1 Other documentation specific to RFA EPA-G2009-STAR-F1 Correspondence between ORD and STAR grant applicants for RFA EPA-G2009-STAR-F1 We also interviewed: Key NCER management personnel, including: o NCER Director o NCER Assistant Deputy Center Director o NCER Senior Science Advisor o NCER PRD Director o Former NCER ASD Director o NCER Extramural Management Specialist Selected other NCER staff, including: o Project Officer o Science Review Administrator o A Program Analyst within NCER's PRD The EPA Senior Associate Director for Grants Competition. To assess whether EPA complied with applicable federal and EPA policies and procedures in managing the technical peer review panel process for proposals submitted (Objective #1), we: Reviewed ORD internal documentation of processes followed before, during, and after NCER's decision to void the December 2009 peer review Examined internal ORD e-mails to assess whether any mismanagement, misconduct, or fraudulent activity occurred Reviewed peer reviewer biographies and written evaluation comments17 Interviewed four peer reviewers for RFA EPA-G2009-STAR-F1 Interviewed EPA and NCER personnel to corroborate and examine further NCER's management of the review process for RFA EPA-G2009-STAR-F1 and its decision to void the December 2009 peer review To assess whether ORD communicated with grant applicants in an accurate, timely, appropriate, and transparent manner regarding the status of their proposals (Objective #2), we: 17 We reviewed all peer reviewer written comments for a random sample of 41 Individual Evaluation forms to determine whether the reviewers addressed each of the six RFA evaluation criteria. A random sample of 41 applications (out of 72 total applications) provided a 95% confidence level at +/-10 error. EPA guidance requires that all peer reviewers' written comments address the RFA criteria. Therefore, our random sample allowed us to determine whether the evaluation criteria was met. 12-P-0864 34 ------- Reviewed documentation of communications between ORD and grant applicants Examined internal ORD e-mail exchanges during the review process Interviewed a judgmental sample of grant applicants for RFA EPA-G2009-STAR-F1 Interviewed key NCER personnel responsible for various aspect of communicating with grant applicants Review of Management (Internal) Controls Generally accepted government auditing standards require that auditors obtain an understanding of internal controls significant to the audit objectives and consider whether specific internal control procedures have been properly designed and placed in operation. We examined management and internal controls as they related to our objectives. We reviewed the federal regulations and guidance related to research and demonstration grants, and focused on EPA's adherence to, and implementation of, those regulations and guidance. We tested whether EPA's internal policies and procedures were applied properly during its review of grant applications under RFA EPA-G2009-STAR-F1, and how they were applied. In general, we examined whether management controls were effective through document review and analysis, corroborated with testimonial evidence. In cases where documentation was lacking, we reviewed procedures to determine whether design or implementation problems existed. The recommendations in our report reflect alternative actions or additional steps that could be taken to improve the management control weaknesses we found. Prior Reports Prior reports by EPA OIG, the U.S. Government Accountability Office, and NRC were applicable to this evaluation. The reports listed below, in particular, provided the team with additional insight into the NCER STAR grant program and important criteria to consider when NCER's management of the review process for RFA EPA-G2009-STAR-F1: EPA OIG Evaluation Report No. 2003-P-00019, Science to Achieve Results (STAR) Fellowship Program Needs to Place Emphasis on Measuring Results, September 30, 2003 EPA OIG Evaluation Report No. 09-P-0147, EPA Can Improve Its Process for Establishing Peer Review Panels, April 29, 2009 EPA OIG Audit Report No. 11 -P-03 86, Office of Research and Development Should Increase Awareness of Scientific Integrity Policies, July 22, 201118 U.S. Government Accountability Office: Environmental Protection: Information on EPA Project Grants and Use of Waiver Authority (2001) NRC: The Measure of STAR: Review of the U. S. Environmental Protection Agency's Science to Achieve Results (STAR) Research Grants Program (2003) 18 Our evaluation team met with members of the OIG Office of Audit Risk Assessment and Program Performance team prior to their completion of this report. The Office of Audit team had interviewed ORD/NCER staff and conducted a survey regarding scientific integrity and research misconduct at EPA. The information collected by OIG's Office of Audit was relevant to our evaluation of NCER's management of RFA EPA-G2009-STAR-F1. 12-P-0864 35 ------- Appendix D Evaluation Criteria and Peer Reviewer Written Comments Table D-l demonstrates the differences in the RFA evaluation criteria and those listed on the Individual Evaluation form. Table D-1: Two sets of evaluation criteria provided to peer reviewers Six evaluation criteria specified in the RFA Five evaluation criteria listed on the "Individual Evaluation" form provided to peer reviewers 1) Identification of links between public health protection and improved water infrastructure sustainability 1) Research Plan and Quality Assurance Statement 2) Definition of Success and Measurement of Progress 2) Investigators 3) Investigators 3) Responsiveness 4) Responsiveness 4) Facilities/Equipment 5) Facilities and equipment 5) Budget 6) Budget Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the peer reviewer comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER. We reviewed all written comments provided on a random sample of 41 Individual Evaluation forms to determine whether the reviewers had still addressed the RFA criteria despite the form and RFA criteria not aligning. For the December 2009 peer review, all three reviewers for about 32 percent (+/- 16 percent with a 95 percent confidence interval)19 of the applications addressed all of the evaluation criteria identified in the RFA in their written comments. For the June 2010 peer review, all three reviewers for about 17 percent (+/- 16 percent with a 95 percent confidence) of the applications addressed all of the evaluation criteria in the RFA. The criteria that were most often left unaddressed by the reviewers in their written comments were criteria 1 and 2 in the RFA (i.e., items number 1 and 2 listed in left column of table D-l). For over 90 percent of the applications (for both peer reviews), all three reviewers for each application provided written comments addressing criteria 3, 4, 5, and 6 in the RFA (i.e., items number 3-6 listed in left column of table D-l). However, for criteria 1 and 2 in the RFA (i.e., items number 1 and 2 listed in left column of table D-l), which are not identified with corresponding headers on the Individual Evaluation form, from 37 to 71 percent of the applications we sampled did not have all three reviewers provide written comments addressing those evaluation criteria listed in the RFA. See tables D-2 and D-3 for additional details. 19 Our sample of 41 applications provided us with a 95% confidence level with an error of +/-10. To draw general conclusions regarding how often all 3 reviewers addressed all 6 criteria (i.e., for all 72 applications), we used interval estimation procedures for sample proportions. Based on our calculations using a Student t test, the final error rate was +/- 16%. Therefore, the findings for the December 2009 and June 2010 peer reviews (32 and 17 percent, respectively) are +/-16% error. 12-P-0864 36 ------- Table D-2: Number of reviewers addressing each RFA criteria for the December 2009 peer review of applications submitted for RFA EPA-G2009-STAR-F1a December 2009 Peer Review No. of RFA RFA RFA RFA RFA RFA reviewers Criteria #1 Criteria #2 Criteria #3 Criteria #4 Criteria #5 Criteria #6 who addressed RFA criteria Three of three 17 of 41 26 of 41 38 of 41 40 of 41 39 of 41 40 of 41 reviewers (41.5%) (63.4%) (92.7%) (97.6%) (95.1%) (97.6%) Two of three 13 of 41 11 of 41 3 of 41 1 of 41 2 of 41 1 of 41 reviewers (31.7%) (26.8%) (7.3%) (2.4%) (4.9%) (2.4%) One of three 10 of 41 3 of 41 0 of 41 Oof 41 Oof 41 0 of 41 reviewers (24.4%) (7.3%) (0.0%) (0.0%) (0.0%) (0.0%) Zero of three 1 of 41 1 of 41 Oof 41 0 of 41 0 of 41 0 of 41 reviewers (2.4%) (2.4%) (0.0%) (0.0%) (0.0%) (0.0%) a In the sample of 41 applications we analyzed, there were only 13 applications (31.7%) reviewed during the December 2009 peer review where all 3 assigned reviewers addressed all 6 RFA evaluation criteria, and thus, adhered to guidance. Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the peer reviewer comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER. Table D-3: Number of reviewers addressing each RFA criteria for the June 2010 peer review of applications submitted for RFA EPA-G2009-STAR-F1a June 2010 Peer Review Number of reviewers who addressed RFA RFA RFA RFA RFA RFA RFA criteria Criteria #1 Criteria #2 Criteria #3 Criteria #4 Criteria #5 Criteria #6 Three of three 12 of 41 18 of 41 41 of 41 41 of 41 40 of 41 41 of 41 reviewers (29.3%) (43.9%) (100%) (100%) (97.6%) (100%) Two of three 21 of 41 18 of 41 0 of 41 Oof 41 1 of 41 0 of 41 reviewers (51.2%) (43.9%) (0.0%) (0.0%) (2.4%) (0.0%) One of three 5 of 41 3 of 41 0 of 41 Oof 41 0 of 41 0 of 41 reviewers (12.2%) (7.3%) (0.0%) (0.0%) (0.0%) (0.0%) Zero of three 3 of 41 2 of 41 0 of 41 Oof 41 Oof 41 0 of 41 reviewers (7.3%) (4.9%) (0.0%) (0.0%) (0.0%) (0.0%) a In the sample of 41 applications we analyzed, there were only 7 applications (17.1%) reviewed during the June 2010 peer review where all 3 assigned reviewers addressed all 6 RFA Evaluation Criteria, and thus, adhered to guidance. Source: OIG analysis of RFA evaluation criteria from Section V of RFA EPA-G2009-STAR-F1, and the peer reviewer comment forms for RFA EPA-G2009-STAR-F1 provided to OIG by NCER. 12-P-0864 37 ------- Appendix E Agency Comments to Draft Report and OIG Response July 16, 2012 MEMORANDUM SUBJECT: Office of Research and Development's (ORD) Response to the Office of Inspector General's (OIG) Draft Report dated June 13, 2012, "EPA 's Review of Applications for Water Infrastructure Sustainability Grant Exhibited Lack of Management Controls and Transparency" (Report No. OPE-FY11-008) FROM: Lek G. Kadeli Acting Assistant Administrator Office of Research and Development TO: Arthur A. El kins, Jr. Inspector General Thank you for the opportunity to review and comment on OlG's draft report "EPA 's Review of Applications for Water Infrastructure Sustainability Grant Exhibited Lack of Management Controls and Transparency," (Report Number OPE-FY 1 1 -008). I appreciate the OlG's acknowledgement that "no evidence of fraud or intentional manipulation of the application review process" was found and the efforts in conducting this review requested by the previous ORD Assistant Administrator. In response to the draft report, ORD offers the following general and specific comments. General Comments ORD's National Center for Environmental Research (NCER) has successfully administered a competitive grants program since its formation in 1995. NCER manages a number of programs, only one of which is the Science To Achieve Results (STAR) program, for which numerous research solicitations are managed annually. As noted in the draft report, the 2003 review of the STAR program conducted by the National Academy of Science acknowledged the rigorous peer review process NCER has established, concluding that it "in many ways exceeds those in place at other organizations that have extramural research programs. " Since 1995, there have been two instances where a peer review was redone because the panel was determined to have an inappropriate mix of expertise, the results negated, and a new panel convened (i.e., a re-review). In the first instance, there- 12-P-0864 38 ------- review decision was made quickly and there-review managed efficiently. The second instance resulted in the request for this OIG review. In both cases, ORD was embarking on new science, thus, there were challenges in fully understanding the diverse expertise needed for the panel to appropriately evaluate applications. In both cases, the Director of NCER requested feedback from staff and based on that feedback, the review was redone. I believe that there is considerable agreement on what improvements are needed to ensure accuracy and consistency. However, there are several areas where clarification is needed. These areas are as follows: Specific Comments The draft report does not accurately describe NCER's STAR grant application review and selection processes. For example, the draft report states (chapter 1): "A decision meeting to formalize NCER's funding recommendations. In the decision meeting, the NCER Director makes final funding decisions based on the results of the external peer review and internal programmatic review. " While the results of the external peer review and internal programmatic review are critical sources of information the NCER Director uses when making final funding recommendations, other information may be employed based on the terms of the RFA. The RFA specifically states, "...the NCER Director may also consider program balance and available funds." Program balance may take into account items of interest as identified in the RFA such as geographic diversity, diversity of research methods, or diversity of eligible awardee types. OIG Response 1: Chapter 1 of the draft report was revised to state that the NCER Director may also consider program balance and available funds in making final funding decisions. The draft report also states that (chapter 3): "The Agency's failure to request the applicants' updated information had the potential to impact the applicants' status in the review process because any changes in the applicants' history coirfd have resulted in an NCER riding of ineligibility for the next stage of the review process. However, we did not identify such impacts for this particular RFA." An applicant's "Past Performance and Reporting History" provides information essential to evaluating the potential performance success of an applicant. Performance success includes the applicant's ability to complete the research and comply with reporting requirements. However, NCER does not use "Past Performance and Reporting History" in making eligibility determinations. 12-P-0864 39 ------- OIG Response 2: We agree that NCER does not use Past Performance and Reporting History in making eligibility determinations. What we intended to convey in the paragraph quoted above was that all applicants were not afforded an opportunity to submit Past Performance and Reporting History that was current to the point in time when NCER was evaluating applicants. This had the potential to impact how NCER evaluated applicants during programmatic review. For those applicants who passed the December 2009 peer review, past performance and reporting history was collected in December 2009. However, after the June 2010 peer review was conducted, past performance and reporting history was collected only from applicants who had not passed the December 2009 peer review but passed the June 2010 peer review. Therefore, NCER was not considering "information essential to evaluating potential performance success of an applicant" that was current for all applicants. Any relevant performance or reporting information that had occurred during the 6-month period between December 2009 and June 2010 would not have been considered for those applicants who passed both the December 2009 and June 2010 peer reviews. We believe that all applicants should be treated equally in terms of the information they are being evaluated on, including past performance. We revised the report to clarify our statements, and have removed reference to eligibility considerations for this particular issue. Additionally, the draft report misinterpreted several statements attributed to EPA staff. First, the draft report states (chapter 2): "According to EPA's Senior Associate Director for Grants Competition, the regulation is intended to ensure that EPA funds research that is relevant to its needs and priorities. " We would like to clarify that EPA's Senior Associate Director for Grants Competition was providing his own opinion of the intent of the regulation and not speaking to what the writers of the regulation had intended. OIG Response 3: We have revised the report to indicate that the Senior Associate Director for Grants Competition was providing his own opinion of the intent of the regulation. A second example states: "In addition, the NCER Assistant Deputy Director advised the project officer to hold off sending an e-mail explaining NCER 's decision to redo the peer review to an applicant that passed the December 2009 peer review but not the June 2010 peer review." NCER does not have an Assistant Deputy Director. If your intention was to refer to NCER's Assistant to the Deputy Director, we have already clarified that this individual did not provide this guidance nor make a statement to the OIG that may have been misconstrued to reflect this type of action on his part. If you have additional information on the source of this comment, it would be helpful if you would share it with us. 12-P-0864 40 ------- OIG Response 4: The correct title for the individual we referred to in the draft report was the Acting Director for the Applied Science Division of NCER. We acknowledge that there may be differing interpretations of the guidance that the Acting Director provided to the project officer regarding communicating the reasons for changes in the review process with this applicant. In a June 29, 2010, e-mail, the project officer proposed a draft e-mail, five paragraphs in length, to one particular applicant whose application passed the December 2009 peer review but not the June 2010 peer review. The project officer's draft e-mail to the applicant included the following explanation of what occurred with the review of applications for RFA EPA-G2009-STAR-F1: The review of all 80 of the applications had to be reprocessed from the beginning even though we were nearing the awards decision point. After reviewing all of the documents, our senior management felt that the first round of Peer Review panelists were lacking in multiple aspects of diversity. The project officer then sent the draft e-mail to the Acting Director for the Applied Science Division, with the following request in her transmittal: How's this? I'd like to go ahead and send her the PR comments from the first round at least. The "PR comments from the first round" referred to the peer review comments on the application from the December 2009 (voided) peer review, which the applicant had passed (this particular applicant, however, had not passed the June 2010 peer review). The Acting Director responded "Please hold off for now." Subsequently, on July 2, 2010, the project officer sent an e-mail to the applicant that contained none of the aforementioned detail about what happened with the first peer review. In a July 15, 2010 e-mail, the SRA provided the applicant with the following explanation of the review process: After careful consideration of the panel make-up, it was determined that disciplines necessary to adequately review the applications were not included in the December [2009] panel. Therefore, the results of the December 2009 peer review were voided, and a second peer review was conducted that included experts from additional fields. As the first peer review was voided and not included in the decision-making process regarding these applications, the results from that peer review were not provided. During an interview with the project officer, we asked her if she was ever instructed not to be forthright with applicants about the decision to re-review applications for this RFA. She told us that she was generally under the impression that she was not to provide applicants with any information about the peer review. In a follow-up e-mail to OIG's draft report, the ORD audit follow-up coordinator stated that the Acting Director's guidance to the project officer to "please hold off for now," was in reference to sending the applicant peer review comments. Because the applicant was ultimately provided with an explanation of changes to the review process, we have revised the report to remove reference to the Acting Director for the Applied Science Division's guidance to the project officer as an example of NCER's lack of transparency in communicating with applicants. 12-P-0864 41 ------- Finally, in chapter 3 of the draft report: "The NCER Senior Science Advisor told us that she evaluated the resumes of potential peer reviewers in consultation with managers in EPA's Office of Water and ORD 's Water Quality Research Program, and only made recommendations to the SRA about which reviewers might be good candidates for the SRA to contact for the June 2010 peer review. She told us that she did not make final approvals for reviewer selections. In contrast, the SRA and PRD [Peer Review Division] Director stated that all peer reviewer selections for the June 2010 peer review were reviewed and approved by NCER's Senior Science Advisor. This was supported by e-mails from NCER personnel that stated that the NCER Senior Science Advisor was approving reviewer selections. Such involvement by NCER management in selecting and approving peer review panel members does not adhere to its firewall practice. " In documentation previously provided to your staff, we believe the evidence provided does not support this statement included in your draft report. NCER's Senior Science Advisor did not review and approve peer reviewers. The email documentation provided to the OIG is between parties other than the Senior Science Advisor and is anecdotal regarding the role of the Senior Science Advisor. OIG Response 5: In light of NCER comments to the draft report and additional documentation provided during and after the exit conference, we revised our Chapter 2 subheading that "NCER Did Not Adhere to Its 'Firewall' Practice for RFA EPA-G2009-STAR-F1to reflect that NCER does not have a clearly defined firewall policy or practice to adhere to. Additionally, we revised the report to state that NCER personnel had different interpretations of their roles in vetting and selecting reviewers for the June 2010 peer review. However, in our view the Senior Science Advisor's involvement in the June 2010 peer review process was not consistent with previously published descriptions of the firewall. Several sources indicated that NCER's firewall calls for PRD, specifically the SRA, to make reviewer selections and assignments. However, the SRA and PRD Director told us that they did not make final reviewer selections for the June 2010 peer review, and that reviewers were approved by the Senior Science Advisor. We have included language in the report to reflect NCER management's disagreement with this characterization. Further, in support of the Senior Science Advisor's statement that she did not approve reviewers, we have included reference to an e-mail stating that she assessed the "suitability" of two prospective peer reviewers submitted to her by the SRA for the June 2010 panel. However, we have also included reference to a number of other e-mails characterizing the Senior Science Advisor's role in the June 2010 peer review process as someone that was involved in reviewing and approving peer reviewer selections. We also made it clear in the report that the Senior Science Advisor was copied on some of these e-mails, but that they were not addressed to her. Another concern noted in the draft report is how NCER's firewall is characterized. Chapter 2 states: "[ORD's National Center for Environmental Research (NCER)] employs a "firewall" to ensure an objective and impartial examination of applications, as required by EPA Order 5700.5A1. If used properly, NCER's firewall shields the peer review process from influences or potential conflicts with EPA personnel outside of 12-P-0864 42 ------- PRD and ensures the independence of the review process. However, for this particular RFA, NCER did not adhere to its firewall practice, as numerous non-PRD staff influenced the selection of peer reviewers for the June 2010 peer review. In addition, non-PRD staff concerns about the December 2009 peer review influenced the manner in which the review process was conducted, thus indirectly influencing the results." The purpose of NCER's firewall is to ensure peer reviewers are free of undo influences. The firewall ensures that internal Agency and external influences cannot compromise the peer review process. NCER peer review staff is encouraged to solicit input from many sources including project officers and other Agency experts to identify the best qualified scientific expertise to serve in the peer review function. The selection of panelists is made by the PRD staff. As part of this process PRD may use, or entirely discount, information provided by others. NCER staff outside of PRD expressing an opinion regarding potential panelist expertise does not violate the firewall or invalidate the independence of the process. This process is consistent with established peer review practices and has been reviewed and approved by the National Academies of Science (see "The Measure of STAR", cited in the draft report). However, the firewall is not intended to isolate the peer review staff from their management or the Agencies scientific resources. OIG Response 6: We agree that NCER staff outside of PRD may provide input to PRD staff or managers regarding potential reviewers, and that such action does not compromise the independence of the process. We have clarified this in the final report. However, we would like to emphasize the point made in ORD's comment that "the selection of panelists is made by the PRD staff." We believe this to be a key aspect of the firewall and in ensuring the independence of the peer review process. This is highlighted by statements in Chapter 1 of our report: A key component of [the] firewall was its shielding of the peer review process from the influence of EPA project officers and staff who oversee the individual investigator, fellowship, and center awards. And again in Chapter 2 of our report: As noted in NCER's draft PRD procedures and policies, as well as the 2003 NRC report, the SRA has sole responsibility for panel selection decisions. The SRA, the PRD Director, the NCER Director, and the former ASD Director all acknowledged that it is the SRA's, or PRD's, responsibility to make such decisions. Chapter 2 concludes that NCER needs "improved management controls to ensure independence and effectiveness of EPA's STAR grants review process." "In general, NCER lacked sufficient procedures and management controls to ensure the independence of its STAR grants competition process. EPA Order 5700.5Al, the RFA, and the applicable regulations cited in the RFA were the only formal policy and procedures documents provided to OIG by NCER that governed its STAR grants award process. The other documents provided were either informal guidance (provided to NCER personnel via e-mail, existed in draft form, etc.), related to 12-P-0864 43 ------- specific portions of the STAR grant review process (e.g., peer review) and not to the entire process, or did not include specific procedural steps for NCER staff None of the Agency's documents formalized NCER 's firewall practice, outlined procedures for substantiating concerns regarding peer review results, identified a process for re- evaluating applications, or specified how decisions to alter or change the typical review process should be documented." ORD agrees that the development of overarching guidance for staff responsibilities, communications, and training that reference existing policies, procedures, and guidance will assist in the improvement of management controls. The OIG review highlights the need for NCER staff to develop a better understanding of each others' role in grants management processes, and an overarching guidance document will prove beneficial in that effort. ORD also agrees that written procedures on re-reviews are needed to improve the evaluation and documentation of a need for are-review, involvement of parties within and outside NCER, appropriate timelines for each step, and to ensure that appropriate communications with applicants are known and followed. The draft report includes numerous headings that could be taken out of context and reported as applying to all NCER-managed RFAs. For example, a chapter 2 heading states, NCER's Review Process Lacks Management Controls. While current controls are not specific as to re-review processes, this heading implies that NCER lacks all management controls. The OIG review request was limited to RFA EPA-G2009-STAR-F1 and findings are specific to this RFA. In addition, a chapter 3 heading states, NCER Not Transparent in Communicating its Review Process to Applicants. The RFA defines pertinent communications to applicants including receipt of applications, outcome of the peer review, etc. The re-review communications for this RFA were not as transparent; however, this heading implies that NCER lacks transparency in all communications with applicants. Again, the OIG review request was limited to RFA EPA-02009-STAR-F1 and findings are specific to this RFA. OIG Response 7: We agree with ORD regarding the report headings cited in the comment above, and have revised the report to more accurately convey the OIG findings for these particular sections of the report. I appreciate the analysis your office has provided in response to our request for this review. Because of some of the concerns addressed above, I have attached ORD's response to the draft report recommendations. Please consider ORD's comments as the final report is prepared. As required by the EPA Order 2750, our written response to the final report will address any recommendations that may be included at that time. If you or your staff have any questions, please contact Deborah Heckman at (202) 564- 7274. Attachment cc: Ramona Trovato, Principal Deputy Assistant Administrator, ORD Bob Kavlock, Deputy Assistant Administrator for Science, ORD 12-P-0864 44 ------- Amy Battaglia, Director, OP ARM, (3RD Jim Johnson, Director, NICER, (3RD Chris Zarba, Deputy Director, NICER, (3RD Christiane Routt, Deputy Director, OP ARM, (3RD 12-P-0864 ------- Attachment I OIG's Draft Report EPA's Review of Applications for Water Infrastructure Sustainability Grant Exhibited Lack of Management Controls and Transparency ORD Response to Recommendations RecNo. OIG Kecommendation Lead Responsibility OKD's Suggested Alternative Recommendation 1 Take steps to make EPA's decision to issue a class exception from 40 C.F.R 40.150 for its STAR grants program known to the public through NCER's website. Assistant Administrator for Research and Development ORD generally agrees with this current recommendation. 2a Direct NCER to establish written procedures for administering the STAR grant review process including a description of management controls needed to ensure applicable regulations and policies are adhered to, and how such controls will be implemented. Assistant Administrator for Research and Development Direct NCER to develop operational procedures that provide overarching NCER-wide guidance for staff responsibilities, communications, and training which will also reference existing policies, guidance, and standard operating procedures, as appropriate. 2b Direct NCER to establish written procedures for administering the STAR grant review process including descriptions of roles and responsibilities of the various NCER divisions and personnel involved in the STAR grant review process. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. Assistant Administrator for Research and Development 2c Direct NCER to establish written procedures for administering the STAR grant review process including a process, including timelines, for receiving and investigating internal and external complaints, as well as for making and documenting subsequent management decisions. Assistant Administrator for Research and Development Direct NCER to develop and/or update written procedures that provide guidance on re-reviews including: a) how to appropriately evaluate and document the need for a re-review; b) the need for the involvement of parties within and outside of NCER; c) appropriate timelines for each step, and; d) appropriate communications with applicants are known and followed. 2d Direct NCER to establish written procedures for administering the STAR grant review process including provisions to consult with EPA personnel who oversee EPA grant competitions in determining whether changes to the review process are necessary. Assistant Administrator for Research and Development 2e Direct NCER to establish written procedures for administering the STAR grant review process including provisions to involve key NCER management personnel in decision meetings that stem from staff or management concerns about the review process. Assistant Administrator for Research and Development 2f Direct NCER to establish written procedures for administering the STAR grant review process including a process for aligning peer reviewer evaluation comments with the RFA evaluation criteria. Assistant Administrator for Research and Development Direct NCER to develop and/or update written procedures so that the review template is consistent with what was published in the RFA prior to releasing the review forms to panelists. 3a Ensure that NCER establishes written procedures for communicating with STAR grant applicants including a process for staff to effectively track and monitor communications with applicants. Assistant Administrator for Research and Development Direct NCER to develop training to ensure staff are aware of and understand the operational procedures as well as their roles and responsibilities in the review process. 3b Ensure that NCER establishes written procedures for communicating with STAR grant applicants including guidance for ensuring that appropriate information is included in notifications to applicants. Assistant Administrator for Research and Development ORD generally agrees with this current recommendation. 12-P-0864 46 ------- RecNo. OIG Kecommendation Lead Responsibility OKD's Suggested Alternative Recommendation 3c Ensure that NCER establishes written procedures for communicating with STAR grant applicants including descriptions of roles and responsibilities of the various NCER divisions and personnel involved in communications with applicants. Ill particular, the roles and responsibilities of the SRA and Project Officer should be clearly defined. Assistant Administrator for Research and Development We believe that this would be no longer applicable based on the suggested alternative recommendation for 2a, 2b, and 3 a. 3d Ensure that NCER establishes written procedures for communicating with STAR grant applicants including annual reviews of the adequacy of internal controls over its communications with applicants. Assistant Administrator for Research and Development ORD generally agrees with this current recommendation. 12-P-0864 47 ------- OIG Response 8: Our responses to ORD's alternative recommendations are included in the table below. Draft Report Recommendation No. 1 2a 2b Draft Report Recommendation No. 2c 2d OIG Recommendation from Draft Report Take steps to make EPA's decision to issue a class exception from 40 C.F.R 40.150 for its STAR grants program known to the public through NCER's website. Direct NCER to establish written procedures for administering the STAR grant review process including a description of management controls needed to ensure applicable regulations and policies are adhered to, and how such controls will be implemented. Direct NCER to establish written procedures for administering the STAR grant review process including descriptions of roles and responsibilities of the various NCER divisions and personnel involved in the STAR grant review process. In particular, the roles and responsibilities of the SRA and project officer should be clearly defined. OIG Recommendation from Draft Report Direct NCER to establish written procedures for administering the STAR grant review process including a process, including timelines, for receiving and investigating internal and external complaints, as well as for making and documenting subsequent management decisions. Direct NCER to establish written procedures for administering the STAR grant review process including provisions to consult with EPA personnel who oversee EPA grant competitions in determining whether changes to the review process are ORD Suggested Alternative Recommendation ORD generally agrees with this current recommendation. Direct NCER to develop operational procedures that provide overarching NCER-wide guidance for staff responsibilities, communications, and training which will also reference existing policies, guidance, and standard operating procedures, as appropriate. ORD Suggested Alternative Recommendation Direct NCER to develop and/or update written procedures that provide guidance on re-reviews including: a) how to appropriately evaluate and document the need for a re-review; b) the need for the involvement of parties within and outside of NCER; c) appropriate timelines for each step, OIG Evaluation of ORD Response In its 90-day response to the report, EPA needs to provide a description of the planned corrective actions and estimated or actual, milestone completion dates. Recommendation 1 is designated as unresolved in the final report. We believe these particular OIG recommendations are appropriate given the scope of our evaluation. Our findings are specific to the STAR Grant program. A recommendation to develop overarching, NCER-wide operational guidance does not ensure that the specific issues we found regarding the STAR grant review process and communication with STAR grant applicants will be addressed. NCER may choose to include the written procedures specified in recommendations as part of a larger, NCER-wide set of procedures. We will evaluate any such actions included in ORD's corrective action plan. However, we believe the OIG recommendations to be appropriate. Recommendations 3a, 3b, and 3c in the final report are designated as unresolved. OIG Evaluation of ORD Response As stated above, our findings are specific to the STAR Grant program, and we believe the recommendations should be worded as such. Otherwise, we agree that ORD's alternative recommendation meets the intent of our original recommendations, and have revised the report accordingly. This information is now reflected in recommendations 4a, 4b, and 4c in the final report. In its 90-day response to the report, EPA needs to provide estimated or actual milestone completion dates for recommendations 4a, 4b, and 4c. Recommendations 4a, 4b, and 4c are designated as unresolved in the final report Changes to Draft Report Recommendations None We have updated the final report's recommendation to include a call for NCER to develop a clearly defined firewall policy. The draft report recommendations 2a and 2b appear as recommendations 3a and 3b in the final report, with recommendation 3c added to recommend a firewall policy. Changes to Draft Report Recommendations ORD's alternative recommendation was inserted into the final report as recommendations 4a, 4b, and 4c (replacing recommendations 2c, 2d, and 2e of the draft report) with the clarification that the alternative recommendation is specific to the STAR grant program. 12-P-0864 48 ------- 2e 2f necessary. Direct NCER to establish written procedures for administering the STAR grant review process including provisions to involve key NCER management personnel in decision meetings that stem from staff or management concerns about the review process. Direct NCER to establish written procedures for administering the STAR grant review process including a process for aligning peer reviewer evaluation comments with the RFA evaluation criteria. and; d) appropriate communications with applicants are known and followed. Direct NCER to develop and/or update written procedures so that the review template is consistent with what was published in the RFA prior to releasing the review forms to panelists. 12-P-0864 As stated above, our findings are specific to the STAR Grant program, and we believe the recommendations should be worded as such. Otherwise, we agree that ORD's alternative recommendation meets the intent of our original recommendations, and have revised the report accordingly. This information is now reflected in recommendation 2 in the final report. In its 90-day response to the report, EPA needs to provide estimated or actual milestone completion dates for recommendation 2. Recommendation 2 in the final report is designated as unresolved. ORD's alternative recommendation was inserted into the final report as recommendation 2 (replacing recommendation 2f of the draft report), with the clarification that it is specific to the STAR grant program. 49 ------- Draft Report Recommendation No. 3a OIG Recommendation from Draft Report Ensure that NCER establishes written procedures for communicating with STAR grant applicants including a process for staff to effectively track and monitor communications with applicants. ORD Suggested Alternative Recommendation Direct NCER to develop training to ensure staff are aware of and understand the operational procedures as well as their roles and responsibilities in the review process. 3b 3c 3d Ensure that NCER establishes written procedures for communicating with STAR grant applicants including guidance for ensuring that appropriate information is included in notifications to applicants. Ensure that NCER establishes written procedures for communicating with STAR grant applicants including descriptions of roles and responsibilities of the various NCER divisions and personnel involved in communications with applicants. In particular, the roles and responsibilities of the SRA and Project Officer should be clearly defined. Ensure that NCER establishes written procedures for communicating with STAR grant applicants including annual reviews of the adequacy of internal controls over its communications with applicants. ORD generally agrees with this current recommendation. We believe that this would be no longer applicable based on the suggested alternative recommendation for 2a, 2b, and 3a. ORD generally agrees with this current recommendation. 12-P-0864 OIG Evaluation of ORD Response We believe the OIG recommendation is appropriate and necessary to address a basic management control weakness in ORD's communication with grant applicants. We do not believe that staff development and training substitutes for a process of tracking and monitoring communications with grant applicants. This recommendation has been re-numbered and appears as recommendation 5a in the final report. In its 90-day response to the report, EPA needs to provide estimated or actual milestone completion dates for recommendation 5a. Recommendation 5a is designated as unresolved in the final report. In its 90-day response to the report, EPA needs to provide a description of the planned corrective actions and estimated or actual, milestone completion dates. Recommendation 5b is designated as unresolved in the final report. Changes to Draft Report Recommendations Recommendations 3a, 3b, 3c, and 3d ofthe draft report appear in the final report as recommendations 5a, 5b, 5c, and 5d. No other changes were made to these recommendations. See OIG evaluation of ORD response for 2a, 2b, and 3a, above. Recommendation 5c is designated as unresolved in the final report. In its 90-day response to the report, EPA needs to provide a description ofthe planned corrective actions and estimated or actual, milestone completion dates. Recommendation 5d is designated as unresolved in the final report. 50 ------- Appendix F Distribution Office of the Administrator Assistant Administrator for Research and Development Deputy Assistant Administrator for Management, Office of Research and Development Deputy Assistant Administrator for Science, Office of Research and Development Agency Follow-Up Official (the CFO) Agency Follow-Up Coordinator General Counsel Associate Administrator for Congressional and Intergovernmental Relations Associate Administrator for External Affairs and Environmental Education Audit Follow-Up Coordinator, Office of Research and Development 12-P-0864 51 ------- |