U.S. ENVIRONMENTAL PROTECTION AGENCY * OFFICE OF INSPECTOR GENERAL Catalyst for Improving the Environment Evaluation Report EPA Can Improve Its Process for Establishing Peer Review Panels Report No. 09-P-0147 April 29, 2009 ------- Report Contributors: Rick Beusse Hilda Canes Gardufio James Hatfield Geoffrey Pierce Abbreviations EPA U.S. Environmental Protection Agency FACA Federal Advisory Committee Act NAS National Academy of Sciences NCEA National Center for Environmental Assessment NIEHS National Institute of Environmental Health Sciences OIG Office of Inspector General OMB Office of Management and Budget ORISE Oak Ridge Institute for Science and Education SAB Science Advisory Board TERA Toxicology Excellence for Risk Assessment Cover photo: Cover of EPA's Peer Review Handbook, 3rd Edition, 2006 (EPA Number EPA/100/B-06/002). (EPA photo) ------- U.S. Environmental Protection Agency Office of Inspector General At a Glance 09-P-0147 April 29, 2009 Catalyst for Improving the Environment Why We Did This Review The Office of Inspector General (OIG) conducted this review in response to a request from the former U.S. Environmental Protection Agency (EPA) Deputy Administrator. We evaluated I whether (1) current laws, regulations, guidance, and other relevant requirements for EPA expert peer review panels are adequate to produce objective scientific reviews; and (2) the current system of populating and managing such panels could be improved. Background Peer review is a process for enhancing a scientific or technical work product so that the decision or position taken by the Agency, based on that product, has a sound, credible basis. EPA's National Center for Environmental Assessment produces highly influential scientific assessments such as human health risk assessments; thus, it is one of EPA's primary users of peer review services. For further information, contact our Office of Congressional, Public Affairs and Management at (202)566-2391. I To view the full report, click on the following link: www.epa.qov/oiq/reports/2009/ 20090429-09-P-0147.pdf EPA Can Improve Its Process for Establishing Peer Review Panels What We Found The laws, regulations, guidance, and other relevant requirements governing EPA's peer review process are adequate to produce objective scientific reviews, but certain areas of EPA operating guidance can be better defined. When we compared the EPA National Center for Environmental Assessment's (NCEA's) peer review panel selection process with the processes used by other major science-based organizations, we found that NCEA's process does not differ in many aspects from those other processes. However, NCEA's current system for populating and managing expert panels can be improved: • Although NCEA strives to select "impartial" panelists, this concept is vaguely defined and not explained in any NCEA-specific operating guidance. • NCEA does not have procedures for addressing conflicts of interest or potential biases that become known after a panel has completed deliberations. • There was no clear documentation of authority and responsibility for making final determinations regarding panel selection or how potential conflicts of interest were resolved. Following a prior OIG report, NCEA improved its peer review process by developing a questionnaire for EPA contractors to use in identifying potential conflicts of interests or biases of prospective panel members. Also, according to the NCEA Director, NCEA recently started to document its peer review process and is implementing a quality assurance checklist to ensure EPA contractors follow EPA's procedures. What We Recommended We recommended that the Assistant Administrator for Research and Development, which oversees NCEA, improve management controls by better defining the concept of "impartiality" and maintaining records of all management decisions pertaining to the selection of peer reviewers, particularly resolution of potential conflicts of interest. We also recommended that the Assistant Administrator develop guidance to address conflict of interest issues that arise after panel formulation and amend contracts for external peer review services to require that panelists re-certify their conflict of interest status prior to the panel convening. The Office of Research and Development agreed with our recommendations, and the Assistant Administrator's planned actions meet the intent of our recommendations. Additional information is needed regarding the timeframe for the Agency's implementation of one of our recommendations. ------- UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460 OFFICE OF INSPECTOR GENERAL April 29, 2009 MEMORANDUM SUBJECT: FROM: TO: EPA Can Improve Its Process for Establishing Peer Review Panels Report No. 09-P-0147 Wade T. Najjum Assistant Inspector General for Program Evaluation Lek G. Kadeli Acting Assistant Administrator for Research and Development This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe the problems the OIG has identified and corrective actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily represent the final EPA position. Final determinations on matters in this report will be made by EPA managers in accordance with established audit resolution procedures. The estimated cost of this report - calculated by multiplying the project's staff days by the applicable daily full cost billing rates in effect at the time - is $272,110. Action Required In accordance with EPA Manual 2750, EPA 's Audit Management Process, you should provide a written response within 90 calendar days. Your response should include a planned completion date for Recommendation 1. Since you submitted a corrective action plan that sufficiently addresses Recommendations 2 through 7, we are "closing" all recommendations, except for Recommendation 1, in our tracking system upon issuance of this report. These recommendations will be tracked to completion in the Agency's tracking system. No further response is required for Recommendations 2 through 7. As outlined in EPA Manual 2750, the Agency is responsible for tracking the implementation of these actions in its Management Audit Tracking System. We have no objections to the further release of this report to the public. This report will be available at http://www.epa.gov/oig. If you or your staff have any questions regarding this report, please contact me at (202) 566-0827 or najjum.wade@epa.gov: or Rick Beusse, Director for Program Evaluation, Air and Research Issues, at (919) 541-5747 or beusse.rick@epa.gov. ------- EPA Can Improve Its Process for 09-P-0147 Establishing Peer Review Panels Table of Contents Purpose 1 Background 1 Laws, Regulations, and Guidance Applicable to EPA Peer Reviews 2 NCEA's Peer Review Process 3 Noteworthy Achievements 4 Scope and Methodology 4 Results of Review 5 Key Steps in NCEA's Process Used to Determine Potential Conflicts of Interest and Impartiality 5 NCEA's Process is Similar to Other Peer Review Processes 5 NCEA Can Improve its Peer Review Process in Certain Areas 6 Conclusions 7 Recommendations 7 Agency Comments and OIG Evaluation 8 Status of Recommendations and Potential Monetary Benefits 10 Appendices A Peer Review Process Steps 12 B Agency Response to Draft Report 13 C Distribution 17 ------- 09-P-0147 Purpose In response to concerns about the U.S. Environmental Protection Agency's (EPA's) handling of allegations of impartiality on one of its peer review panels, former EPA Deputy Administrator Marcus Peacock requested that the Office of Inspector General (OIG) review EPA's peer review process. The objectives of our review1 were to determine whether: • current laws, regulations, guidance, and other relevant requirements for such panels are adequate to produce objective scientific reviews; and • the current system of populating and managing such expert panels could be improved. Background The foreword to EPA's Peer Review Handbook notes that strong, independent science is of paramount importance to EPA's environmental policies. The quality of the science that underlies EPA's regulations is vital to the credibility of EPA's decisions and, ultimately, the Agency's effectiveness in protecting human health and the environment. The peer review process enhances a scientific or technical work product so that the decision or position taken by the Agency, based on that product, has a sound, credible basis. It involves the review of a draft product by specialists in the field who were not involved in producing the draft. The peer reviewers then issue a report - an evaluation or critique - that is used by the authors of the draft to improve the product so that the final work product will reflect sound technical information and analyses. As described in the Handbook, peer reviewers typically evaluate the: • clarity of hypotheses, • validity of the research design, • quality of data collection procedures, • robustness of the methods employed, • appropriateness of the methods for the hypotheses being tested, • extent to which the conclusions follow from the analysis, and • strengths and limitations of the overall product. EPA's National Center for Environmental Assessment (NCEA), within EPA's Office of Research and Development, produces highly influential scientific assessments and thus is one of EPA's primary users of peer review services. EPA's NCEA uses extramural instruments, such as contracts and interagency agreements, to obtain peer review services to review highly influential scientific assessments, such as human health risk assessments. NCEA oversees the peer review process. 1 The Deputy Administrator also asked OIG to determine whether the actions taken by EPA or the panelists were done consistent with existing federal law, regulations, guidance, and other relevant requirements. This objective was addressed in a separate OIG report. ------- 09-P-0147 Laws, Regulations, and Guidance Applicable to EPA Peer Reviews The primary laws, regulations, and guidance governing peer review at EPA include the Office of Management and Budget's (OMB's) "Final Information Quality Bulletin for Peer Review," EPA's Peer Review Handbook, and the Federal Advisory Committee Act (FACA). OMB's Bulletin, issued in 2004, is the primary guidance for government agencies regarding peer reviews. The OMB Bulletin was based primarily on procedures used by the National Academy of Sciences (NAS). The OMB Bulletin established four key criteria to guide federal agencies in selecting a peer review panel. 1. Expertise is the most important factor when selecting a panelist. 2. Except for situations where it is unavoidable, panelists should be free of conflicts of interest. A "conflict of interest" is any financial or other interest that conflicts with the service of an individual on the review panel because it could impair the individual's objectivity or could create an unfair competitive advantage for a person or organization. 3. Panelists should be independent. The panelists should not have worked on the product being reviewed. 4. The panel should be balanced. Reviewers should be selected to represent a diversity of scientific perspectives. EPA's January 2006 Peer Review Policy memo establishes the policy for peer review of scientific and technical work products, including economic and social science products that are intended to inform Agency decisions. EPA's Peer Review Handbook is the Agency's primary guidance governing peer reviews. The Handbook was most recently revised in 2006 to be consistent with the provisions of the 2004 OMB Bulletin. Although the Handbook outlines EPA's preferred approach to ensuring the quality of peer i ,i • •,• , 11 ,i Figure 1. Relation between all EPA work products and reviews conducted or initiated by the those considered inf|uential scientific information or Agency, it is not a requirement. highly influential scientific assessments The principle underlying EPA's Peer Review Policy is that all influential scientific and technical work products used in decision making will be peer reviewed. Determining whether a scientific and/or technical work product is "influential" or "highly influential" is done on a case-by- case basis, taking into account various criteria and the circumstances surrounding the use of the work product. OMB defines highly influential scientific assessments as influential scientific information that the Agency considers (1) could have a potential impact of more than $500 million in any year; or (2) is novel, controversial, Influential Scientific Information Highly Influential Scientific Assessments Source: EPA Peer Review Handbook, 3 Edition (2006) ------- 09-P-0147 or precedent-setting, or has significant interagency interest. The OMB Bulletin calls for additional peer review procedures for highly influential scientific assessments. The FACA governs committees that advise the federal government on a variety of issues, including peer review of scientific research. Among other requirements, FACA committee membership must be balanced in terms of points of view represented and the functions to be performed by the committee. EPA's Science Advisory Board (SAB) is a FACA committee and often conducts peer reviews of EPA products. Consequently, the SAB must manage its peer review panels in accordance with FACA requirements. Some statutes, such as the Clean Air Act, mandate a peer review process for certain EPA decisions. These peer reviews are conducted by FACA Committees supported by SAB, and are subject to FACA requirements. NCEA's Peer Review Process There are no laws or regulations specifying requirements for the peer reviews conducted by NCEA. The peer review mechanism used by NCEA to conduct peer review in any particular case is within its discretion. The majority of NCEA's peer reviews are for assessments conducted for the Integrated Risk Information System program. Integrated Risk Information System documents describe the health effects of individual substances and contain descriptive and quantitative information on their cancer and noncancer effects. The system is described in EPA 's Integrated Risk Information System: Assessment Development Procedures. These procedures allow for OMB and interagency review and input on the external peer review charge questions developed by EPA. NCEA's process for peer review includes: • conducting all peer reviews for influential scientific information, including highly influential scientific assessments and work products, by peer review panels in accordance with the Agency's Peer Review Handbook; • providing members of each panel access to public comments received on the documents under review; and • using the specified Conflicts of Interest questionnaire (the questionnaire includes a series of yes/no questions and request for supporting documentation) when peer review services are obtained through contracts and interagency agreements. Specific requirements for conducting peer reviews are included in contract and interagency agreement statements of work. Depending upon the scientific product, NCEA may obtain peer review services from the NAS, the SAB, an EPA contract, or under an interagency agreement (at the time we conducted our review, NCEA had an interagency agreement with the Department of Energy's Oak Ridge Institute for Science and Education (ORISE)). According to NCEA officials, the assessments dealing with the most complex issues are given to the NAS to peer review; these occur about one every few years. Other assessments dealing with complex issues are given to SAB for peer review; these average from two to four each year. The majority of assessments are either reviewed under the peer review contract or an interagency agreement with another federal agency; in the past few years, NCEA has predominantly obtained peer review services for IRIS assessments through the ORISE interagency agreement. As of March 2009, NCEA does not plan to acquire peer review services under the ORISE interagency agreement since it expires in March 2009 and funds are no longer available to purchase such interagency services. ------- 09-P-0147 Noteworthy Achievements In response to a prior OIG evaluation,2 NCEA developed a questionnaire and received OMB approval for EPA contractors to use the questionnaire to help identify potential conflicts of interest or potential biases that may affect the selection of a potential panel member. The questionnaire asks potential panelists to address possible financial conflicts of interest of prospective panelists and their family members, as well as possible non-financial independence and impartiality issues. During the current evaluation, NCEA started taking actions to improve its peer review process. According to the NCEA Director, NCEA is developing a description of the peer review process used in the Integrated Risk Information System program that it intends to post to its Website as a reference for staff and others. Scope and Methodology Our review was limited to a design evaluation of EPA's external peer review process, with a primary focus on NCEA's peer review process. Accordingly, our report contains recommendations that apply to EPA's external peer review process in general, as well as recommendations that apply specifically to NCEA's process. NCEA oversees the peer review of EPA's health risk assessments, specifically the peer review panel process that prompted the former EPA Deputy Administrator's request to us. We reviewed the applicable laws, regulations, policies, and guidance related to the establishment of peer review panels for the independent peer review of NCEA research. This review included OMB's 2004 bulletin, "Final Information Quality Bulletin for Peer Review"; and EPA's Peer Review Handbook, 3rd Edition, June 2006. We also reviewed the FACA, as amended in 1997. We interviewed key officials within NCEA to understand the process NCEA uses to establish peer review panels. We reviewed NCEA's peer review services contract and the interagency agreement used to acquire peer review support. We also interviewed individuals from the NAS, EPA's SAB, the Toxicology Excellence for Risk Assessment (TERA) organization,3 and the National Institute of Environmental Health Sciences (NIEHS) to identify the practices used by other major organizations to identify and select peer review panelists. We also reviewed our prior report on the peer review process (see footnote 2) and confirmed that NCEA implemented the recommendations in that report. In addition, we reviewed two prior U.S. Government Accountability Office reports on the SAB's peer review process.4 EPA OIG. Review of Conflict of Interest Allegations Pertaining to the Peer Review of EPA 's Draft Report, "Exposure and Human Health Evaluation of Airborne Pollution from the World Trade Center Disaster. " Report No. 2005-S-00003. November 4, 2004. 3 TERA is a non-profit, 501(c)(3) corporation organized for scientific and educational purposes. TERA's mission is to protect public health by developing and communicating risk assessment information, sponsoring peer reviews and consultations, improving risk methods through research, and educating the public on risk assessment issues. 4 EPA 's Science Advisory Board Panels: Improved Policies and Procedures Needed to Ensure Independence and Balance. GAO-01-536. June 12, 2001; and Federal Advisory Committees: Additional Guidance Could Help Agencies Better Ensure Independence and Balance. GAO-04-328. April 16,2004. ------- 09-P-0147 We performed our evaluation between May 2008 and February 2009 in accordance with generally accepted government auditing standards issued by the Comptroller General of the United States. Those standards require that we plan and perform the evaluation to obtain sufficient and appropriate evidence to provide a reasonable basis for our findings and conclusions based on our objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our evaluation objectives. Results of Review We found that the laws, regulations, guidance, and other relevant requirements governing EPA's peer review process are adequate to produce objective scientific reviews, but EPA can better define certain areas of its operating guidance. NCEA's peer review process is similar to the process of other major peer review organizations we reviewed. However, NCEA can improve its system for populating and managing expert panels by better documenting conflict of interest decisions, establishing guidance for handling conflict of interest issues that arise after the panel has completed its deliberations, and providing more consistency between contractor and other third party procedures for selecting panels. Key Steps in NCEA's Process Used to Determine Potential Conflicts of Interest and Impartiality EPA's contractors and ORISE require potential panelists to complete conflict of interest questionnaires that address financial conflicts of interest of panelists and their family members, as well as non-financial independence and impartiality issues. Panelists must certify that their answers are correct. The questions solicit "yes" or "no" responses. If a potential conflict of interest is indicated, the contractor or ORISE seeks additional information from the potential panelist. If the contractor or ORISE have difficulty making a determination regarding conflict of interest or lack of impartiality, the matter is brought to the attention of EPA. If conflict of interest is identified, the contractor must notify EPA's Project Officer, who in turn brings the issue to the attention of NCEA's Director or Associate Director for Health for resolution. Prior to the panel convening, ORISE asks the selected panelists to confirm by return e-mail that there are no changes to their previous conflict of interest certification form. If their responses to any of the questions have changed, they must explain the changes to ORISE. ORISE changed this procedure beginning in June 2007. ORISE now sends a copy of the panelist's completed conflict of interest questionnaire back to the prospective peer reviewer so that they can review their original answers before confirming current status. NCEA's Process is Similar to Other Peer Review Processes We compared NCEA's peer review panel selection process with the process of four other science-based organizations.5 We found that NCEA's process does not differ in many aspects from the processes used by these other major science-based organizations. Appendix A provides the details on our comparison. ; The four organizations are the NAS, EPA's SAB, TERA, and NIEHS. ------- 09-P-0147 One noteworthy difference between NCEA's process and other organizations is panel selection and public input. FACA panels for NAS and the National Academy of Public Administration are required to obtain public comment on proposed panelists. While this requirement does not apply to EPA SAB panels, the SAB has adopted this practice. Unlike the NAS and the SAB, NCEA's consultants do not obtain public input or comment on proposed panel members. In addition, FACA panels such as those convened by SAB and NAS attempt to achieve consensus among its panelists, and concerns about the impartiality of panel members can be mitigated by balancing the panel. Peer review panels established through NCEA's extramural instruments do not seek consensus. Thus, NCEA does not mitigate the inclusion of impartial panel members through panel balance, but instead chooses to leave potentially partial (or biased) panelists off the panel. TERA was the only organization we contacted that provides the basis of its conflict of interest decisions to the public. TERA's peer review reports identify appearances of potential conflicts of interest that panelists may have and provide TERA's reasons for selecting these panelists. NIEHS uses peer reviews to evaluate studies completed for their National Toxicology Program. Rather than solely relying on completed questionnaires, NIEHS also conducts searches to identify possible impartialities or biases possessed by potential panel members. NCEA Can Improve its Peer Review Process in Certain Areas Although NCEA's external peer review process incorporates many of the procedures and controls used by other peer review organizations, certain areas of the process can be improved to provide more consistency and transparency to the process. These areas are described below. • Although NCEA strives to select "impartial" panelists, this concept is vaguely defined by OMB and EPA guidance and is not explained in any NCEA-specific operating guidance. Neither the 2004 OMB Bulletin nor the EPA Handbook defines what constitutes "impartiality." According to the Handbook, in general potential panelists who had a predominant influence on an organization's position or have taken a public position or "taken sides" should be avoided. • There was no clear documentation of authority and responsibility for making final determinations regarding panel selection or how potential conflicts of interest were resolved. • NCEA can improve staff and the public's understanding of its external peer review process by fully describing the process and making that description available to EPA staff and the public. For example, TERA provides a description of its peer review process and its procedures for panel selection on the Internet. NCEA currently does not have a comprehensive description of its external peer review process, although the NCEA Director said one is being developed and will be made available on its public Website. • NCEA does not have procedures for addressing conflicts of interest or potential biases, or allegations of such that become known or alleged after a panel has begun or completed its deliberations. NCEA does not have a policy or procedures regarding the circumstances under which a panelist's pay may be recouped or withheld when the panelist is dismissed or resigns before completion. ------- 09-P-0147 • Although NCEA's contractors conduct Internet searches to identify potential conflicts of interest and appearances of bias or partiality, ORISE - the current provider of peer review services under an interagency agreement - does not conduct Internet background searches. NCEA could improve its peer review process by establishing procedures for providers of peer review services to follow when conducting independent background searches on prospective panelists. • NCEA's contractors do not use similar procedures for identifying any changes in selected panelists' conflict of interest status. One of EPA's two contractors told us it asks panelists at the first meeting of the panel if there have been any changes in their conflict of interest status. Any changes should be brought to the attention of EPA officials. However, the EPA program manager for the contract could not provide documentation that the panelists' answers were placed in the peer review record. According to the EPA program manager for the second contractor, it does not ask panelists if there have been changes in their conflict of interest status. After our inquiry, the program manager told us it plans to incorporate procedures to identify whether any changes in status have occurred between the time panelists complete their conflict of interest questionnaire and begin panel deliberations. • NCEA can improve its oversight of peer reviews conducted by third parties to better ensure these peer reviews follow contractual guidelines. NCEA is working to develop an oversight tool to help ensure that significant steps in the peer review process are followed. Such a tool could also be useful in providing oversight of NCEA's peer review contracts and any future interagency agreements NCEA may use to obtain peer review services. Conclusions Certain areas of NCEA's current system for populating and managing expert panels can be improved. Although NCEA strives to select "impartial" panelists, this concept is vaguely defined and not explained in any NCEA-specific operating guidance. NCEA does not have procedures for addressing conflicts of interest or potential biases that become known after a panel has completed its deliberations. We also found that there was no clear documentation of authority and responsibility for making final determinations regarding panel selection or how potential conflicts of interest were resolved. We concluded that NCEA did not have adequate controls to establish accountability for suitability determinations and rationale for including or excluding each panelist. Recommendations We recommended that the Assistant Administrator for Research and Development: 1. Establish criteria, definitions, and/or example scenarios for the Peer Review Handbook term "appearance of a lack of impartiality," under which contractors and other external peer review services providers should operate. ------- 09-P-0147 2. Require and confirm that peer review records are maintained throughout the peer review process and that these records include any correspondence and decisions related to suitability, or potential conflicts of interest or biases of prospective panelists. In cases where panelists with potential conflicts or biases are accepted on the panel, the records should include a memorandum of decision explaining the suitability and rationale for including or excluding each panelist, which is signed off on by an EPA official. 3. Publish a description of the peer review process used in the Integrated Risk Information System program as a reference for staff and others. The process description should clearly define the roles and responsibilities of persons involved throughout the peer review process. 4. Establish procedures for addressing conflict of interest and lack of impartiality issues that arise after panel selection. These procedures should discuss under what circumstances peer review panelists' pay may be recouped or withheld. 5. Amend all extramural instruments to call for background Internet searches on potential panel members. 6. Modify NCEA's peer review contracts to require written recertification from panelists, before a peer review panel is convened, stating that their responses to the questionnaire have not changed. A copy of the questionnaire completed by the panelist should be included with the request for a written recertification. For both contracts and interagency agreements, EPA should require that reviewers self report any changes that may impact their conflict of interest status or lack of impartiality status at any point in the process. In cases where the Agency obtains the services of a reviewer through purchase orders not connected with contracts or interagency agreements, or without compensation, the terms should likewise require that reviewers self report changes that may impact their conflict of interest status or lack of impartiality status. 7. Develop an oversight tool to ensure that external peer review service providers follow all significant steps in the peer review process. Agency Comments and OIG Evaluation The Agency agreed with the report's conclusions and recommendations and proposed corrective actions that the Office of Research and Development plans to take in response to our recommendations. For six recommendations the planned corrective actions and planned timeframes for completion meet the intent of our recommendations. We are closing Recommendations 2 through 7 in our tracking system upon issuance of this report. These recommendations will be tracked to completion in the Agency's tracking system. The Agency's planned action in response to Recommendation 1 also meets the intent of our recommendation, but additional information is needed regarding the timeframe for the Agency's implementation of this recommendation. Accordingly, Recommendation 1 remains open pending our receipt of an estimated completion date for the Agency's proposed corrective action for this recommendation. ------- 09-P-0147 The Agency also provided several technical clarifications and comments to the report. We made changes to the final report based on these comments, as appropriate. The Agency's complete written response is in Appendix B. ------- 09-P-0147 Status of Recommendations and Potential Monetary Benefits RECOMMENDATIONS POTENTIAL MONETARY BENEFITS (in SOOOs) Rec. No. 1 Page No. 7 Subject Establish criteria, definitions, and/or example Status1 0 Action Official Assistant Administrator for Planned Completion Date Claimed Amount Agreed To Amount scenarios for the Peer Review Handbook term "appearance of a lack of impartiality," under which contractors and other external peer review services providers should operate. Require and confirm that peer review records are maintained throughout the peer review process and that these records include any correspondence and decisions related to suitability, or potential conflicts of interest or biases of prospective panelists. In cases where panelists with potential conflicts or biases are accepted on the panel, the records should include a memorandum of decision explaining the suitability and rationale for including or excluding each panelist, which is signed off on by an EPA official. Publish a description of the peer review process used in the Integrated Risk Information System program as a reference for staff and others. The process description should clearly define the roles and responsibilities of persons involved throughout the peer review process. Establish procedures for addressing conflict of interest and lack of impartiality issues that arise after panel selection. These procedures should discuss under what circumstances peer review panelists' pay may be recouped or withheld. Amend all extramural instruments to call for background Internet searches on potential panel members. Modify NCEA's peer review contracts to require written recertification from panelists, before a peer review panel is convened, stating that their responses to the questionnaire have not changed. A copy of the questionnaire completed by the panelist should be included with the request for a written recertification. For both contracts and interagency agreements, EPA should require that reviewers self report any changes that may impact their conflict of interest status or lack of impartiality status at any point in the process. In cases where the Agency obtains the services of a reviewer through purchase orders not connected with contracts or interagency agreements, or without compensation, the terms should likewise require that reviewers self report changes that may impact their conflict of interest status or lack of impartiality status. Research and Development Assistant Administrator for Research and Development 6/30/09 Assistant Administrator for 6/30/09 Research and Development Assistant Administrator for 6/30/09 Research and Development Assistant Administrator for 6/30/09 Research and Development Assistant Administrator for Research and Development 6/30/09 10 ------- 09-P-0147 POTENTIAL MONETARY RECOMMENDATIONS BENEFITS (in $OOOs) Planned Rec. Page Completion Claimed Agreed To No. No. Subject Status1 Action Official Date Amount Amount Develop an oversight tool to ensure that external peer C Assistant Administrator for 6/30/09 review service providers follow all significant steps in Research and Development the peer review process. 0 = recommendation is open with agreed-to corrective actions pending; C = recommendation is closed with all agreed-to actions completed; U = recommendation is undecided with resolution efforts in progress 11 ------- 09-P-0147 Appendix A Peer Review Process Steps Process Steps Prospective panelists identified from database of experts. Public is allowed the opportunity to recommend panel members. Prospective panelists complete detailed conflict of interest and an appearance of a lack of impartiality questionnaire. Prospective panelists provide information on past employment, research, etc. Independent research conducted to identify potential panelist biases or conflicts. Panelists are questioned about potential conflicts noted from the information provided. Names of prospective panelists published and public comment requested prior to final panel selection. Selected panelists verify no changes in conflict of interest status just prior to panel meeting. Agency decisions on potential conflict of interest documented in Peer Review report. NCEA Contracts Yes No Yes Yes Yes Yes No Yes (verbal) No NCEA-ORISE Yes No Yes Yes No Yes No Yes (e-mail) D No MAS Partially Yes Yes Yes Sometimes Yes Yes Yes (verbal) No EPA SAB Yes Yes Yes Yes Yes Yes Yes Yes (written) No TERA Yes Yes, if requested by sponsor Yes Yes Yes Yes No Yes (verbal) Yes NIEHS National Toxicology Program Yes Yes Partial a Yes Yes Yes No Yes (written) No Source: OIG-prepared table using information obtained from NCEA, NCEA contractors, ORISE, NAS, SAB, TERA, and NIEHS. 3 Instead of relying solely on information provided by panelists, NIEHS conducts independent research to determine whether potential biases or conflicts exist. NIEHS does not use a checklist. Potential panelists provide curricula vitae that address the needed information. b Procedure changed to require documented re-confirmation that no conflicts exist. 12 ------- 09-P-0147 Appendix B Agency Response to Draft Report April 24, 2009 OFFICE OF RESEARCH AND DEVELOPMENT MEMORANDUM SUBJECT: ORD Response to the Office of Inspector General Draft Evaluation Report: EPA Can Improve Its Process for Establishing Peer Review Panels, Assignment No. OPE-FY08-0007 FROM: LekG. Kadeli /s/ Acting Assistant Administrator TO: Wade T. Najjum Assistant Inspector General for Program Evaluation Office of Inspector General Attached please find: (1) A summary table of Office of Research and Development's (ORD) responses to the Inspector General's seven recommendations and (2) summary of comments on the Draft Evaluation Report regarding Peer Review. The comments represent collaboration across ORD because many of the draft report's recommendations could have impact to others in ORD. Any recommendations that go beyond ORD will be discussed with the Science Policy Council. Thank you for providing the report and giving consideration to our response. cc: Kevin Teichman Peter Preuss Fred Hauchman 13 ------- 09-P-0147 ORD's Response to OIG Draft Report "EPA Can Improve Its Process for Establishing Peer Review Panels" Assignment No. OPE-FY08-0007 This document is comprised of three sections: 1. Table of ORD's Response to IG's seven recommendations 2. Recommendations for editing the Report regarding the scope 3. Specific comments by page number 4. General comments regarding the recommendations 1. Table of ORD's Response to IG Recommendations Rec. No. 1 2 3 Subject Establish criteria, definitions, and/or example scenarios for the Peer Review Handbook term "appearance of a lack of impartiality," under which contractors and other external peer review services providers should operate. Require and confirm that peer review records are maintained throughout the peer review process and that these records include any correspondence and decisions related to suitability, or potential conflicts of interest or biases of prospective panelists. The records should include a Memorandum of Decision explaining the suitability and rationale for including or excluding each panelist, which is signed off on by an EPA official Publish a description of the peer review process used in the Integrated Risk Information System program as a reference for staff and others. The process description should clearly define the roles and responsibilities of persons involved throughout the peer review process. Lead Responsibility Office of Science Advisor ORD ORD ORD's Recommendation ORD agrees with this recommendation. The Office of the Science Advisor (OSA) will coordinate with the Science Policy Council (SPC) to consider any potential revisions to the Agency Peer Review Handbook regarding establishing criteria, definitions and/or example scenarios for the term "appearance of a lack of impartiality." ORD will provide OSA with scenarios for consideration. ORD agrees with this recommendation. For non-FACA reviews, ORD will develop documentation to specify the addition of a memo to ORD peer review files confirming records are maintained, and the inclusion of all correspondence and records of discussion during the peer review panel selection process. ORD will develop a Standard Operating Practice (SOP) and perform periodic audits of peer review files to ensure this new requirement is being met. ORD agrees with this recommendation. ORD is currently re-writing the IRIS peer review SOP description. The final IRIS peer review process SOP will be posted to the IRIS website for use by EPA staff, contractors, and the public. Planned Completion Date TBDin consultation with the SPC June 30, 2009 June 30, 2009 14 ------- 09-P-0147 Establish procedures for addressing conflict of interest and lack of impartiality issues that arise after panel selection. These procedures should discuss under what circumstances peer review panelists' pay may be recouped or withheld. ORD ORD agrees with this recommendation. ORD will develop procedures and document them in the IRIS peer review SOP. ORD will consult with the Office of General Counsel (OGC). June 30, 2009 Amend all extramural instruments to call for background Internet searches on potential panel members. ORD ORD agrees with this recommendation. Based on the EPA's Science Advisory Board protocol, ORD will add direction to conduct background Internet searches to all NCEA contract task orders. ORD will consult with OGC. June 30, 2009 Modify NCEA's peer review contracts to require written recertification from panelists, before a peer review panel is convened, stating that their responses to the questionnaire have not changed. A copy of the questionnaire completed by the panelist should be included with the request for a written recertification. For both contracts and interagency agreements, EPA should require that reviewers self report any changes that may impact their conflict of interest status or lack of impartiality status at any point in the process. ORD ORD agrees with this recommendation. ORD will develop additional language for contract task orders requiring initial written certification at the time of empanelment, written (email) recertification about two weeks prior to the panel meeting at the beginning of panel meetings, and self reporting of any changes that may impact their conflict status or lack of impartiality status at any point in the process. ORD will consult with OGC. June 30, 2009 Develop an oversight tool to ensure that external peer review service providers follow all significant steps in the peer review process. ORD ORD agrees with this recommendation. ORD will develop a QA checklist for contractors to use for all ORD peer reviews. June 30, 2009 2. Recommendations for editing the Report regarding the scope The scope of the Report is defined in the Purpose section on Page 1 ("... EPA Deputy Administrator Marcus Peacock requested that the Office of Inspector General (OIG) review EPA's peer review process.") and the Scope and Methodology section on Page 4 ("Our review was limited to a design evaluation of EPA's external peer review process, with a primary focus on NCEA's peer review process.") The Report provides recommendations for the ORD Assistant Administrator as the Action Official to address. Some of the recommendations are within the scope of NCEA to address in full (recommendation 3, 6) whereas others (recommendations 1, 2, 4, 5, 7), while actionable by NCEA, also affect ORD and EPA more broadly. We recommend that for each instance where the report makes reference to "EPA" and to "NCEA" that the appropriate level of the organization be reviewed to confirm that the scope of the organization being referenced is consistent with the intended scope of the text and recommendations. For example, the Results of Review section on pages 5-7 refers only to NCEA, yet the scope of many recommendations on pages 7-8 is not limited to NCEA. 15 ------- 09-P-0147 3. Specific comments: Page 1, Background: the text of the report indicates that NCEA uses grants to obtain peer review services. This is not correct. NCEA has used contracts and Interagency Agreements to obtain peer review services. Page 3, last paragraph, last sentence - replace "In the future" with "As of March 2009," NCEA does not plan to acquire peer review services under the ORISE ... Page 5, "Key steps...," reference is made to the Deputy Director for Health"; the correct reference here should be to the "Associate Director for Health." In addition, on the bottom of page 5 of the report, the statement is made that "FACA panels are required to obtain public comment on proposed panelists." This statement is factually incorrect. FACA does not require this, and neither does EPA. The SAB does obtain public comment on proposed panelists; however, this is not an agency-wide requirement, it is an SAB best practice. Page 6, the last bullet at bottom of this page. It is correct that ORISE did not conduct internet searches and it is correct that our other contractors, ERG and Versar, have conducted Internet background searches. ERG often does internet searches to identify candidates but does not do an independent internet search after receiving the self reporting questionnaire. Versar also does internet searches, particularly for controversial reviews. Page 7, last bullet before "Conclusions," it is stated that NCEA is working with ORISE to develop an oversight tool. The specific reference to ORISE should be deleted and be stated instead as "NCEA is working to develop an oversight too..." since the peer review work with ORISE is ending. 4. General comments: a) With regard to specific recommendations, one in particular (#2) would benefit from further elaboration. It states that peer review records should "include a memorandum of decision explaining the suitability and rationale for including or excluding each panelist." Two examples are given below to illustrate why additional explanation by the OIG would be helpful: • Lists of prospective panelists are often lengthy, and many worthy candidates may be excluded simply because they are unavailable. Would a memo of decision be required for this circumstance, or is the OIG referring to willful exclusion? • Conversely, when an individual is included on a panel, current practices for justifying their selection (background searches and documentation of expertise, work history, stature or prominence in the field, etc.) are in our view sufficient to explain the rationale, so long as there are no conflicts of interest. Perhaps a memo of decision is necessary only in those cases where a panelist is selected despite some objection or perceived bias b) Regarding recommendation #6, in cases where the Agency obtains the reviewer through purchase orders not connected with contracts or lAGs, or without compensation, the terms should likewise required that reviewers self report changes that may impact their conflict of interest status or lack of impartiality status. c) Finally, we have reviewed and are in support of the comments submitted by the Office of the Science Advisor. The comments will be submitted by OSA.6 6 OIG Note: The Office of Research and Development clarified that it had already incorporated the Office of the Science Advisor's comments into the above response. 16 ------- 09-P-0147 Appendix C Distribution Office of the Administrator Acting Assistant Administrator for Research and Development Director, National Center for Environmental Assessment Acting Science Advisor Agency Follow-up Official (the CFO) Agency Follow-up Coordinator Acting General Counsel Acting Associate Administrator for Congressional and Intergovernmental Relations Acting Associate Administrator for Public Affairs Audit Follow-up Coordinator, Office of Research and Development Acting Inspector General 17 ------- |