U.S. ENVIRONMENTAL PROTECTION AGENCY
* OFFICE OF INSPECTOR GENERAL
Catalyst for Improving the Environment
Evaluation Report
EPA Can Improve Its
Process for Establishing
Peer Review Panels
Report No. 09-P-0147
April 29, 2009
-------
Report Contributors: Rick Beusse
Hilda Canes Gardufio
James Hatfield
Geoffrey Pierce
Abbreviations
EPA U.S. Environmental Protection Agency
FACA Federal Advisory Committee Act
NAS National Academy of Sciences
NCEA National Center for Environmental Assessment
NIEHS National Institute of Environmental Health Sciences
OIG Office of Inspector General
OMB Office of Management and Budget
ORISE Oak Ridge Institute for Science and Education
SAB Science Advisory Board
TERA Toxicology Excellence for Risk Assessment
Cover photo: Cover of EPA's Peer Review Handbook, 3rd Edition, 2006 (EPA Number
EPA/100/B-06/002). (EPA photo)
-------
U.S. Environmental Protection Agency
Office of Inspector General
At a Glance
09-P-0147
April 29, 2009
Catalyst for Improving the Environment
Why We Did This Review
The Office of Inspector
General (OIG) conducted this
review in response to a request
from the former U.S.
Environmental Protection
Agency (EPA) Deputy
Administrator. We evaluated
I whether (1) current laws,
regulations, guidance, and other
relevant requirements for EPA
expert peer review panels are
adequate to produce objective
scientific reviews; and (2) the
current system of populating
and managing such panels
could be improved.
Background
Peer review is a process for
enhancing a scientific or
technical work product so that
the decision or position taken
by the Agency, based on that
product, has a sound, credible
basis. EPA's National Center
for Environmental Assessment
produces highly influential
scientific assessments such as
human health risk assessments;
thus, it is one of EPA's primary
users of peer review services.
For further information,
contact our Office of
Congressional, Public Affairs
and Management at
(202)566-2391.
I To view the full report,
click on the following link:
www.epa.qov/oiq/reports/2009/
20090429-09-P-0147.pdf
EPA Can Improve Its Process for Establishing
Peer Review Panels
What We Found
The laws, regulations, guidance, and other relevant requirements governing
EPA's peer review process are adequate to produce objective scientific reviews,
but certain areas of EPA operating guidance can be better defined.
When we compared the EPA National Center for Environmental Assessment's
(NCEA's) peer review panel selection process with the processes used by other
major science-based organizations, we found that NCEA's process does not differ
in many aspects from those other processes. However, NCEA's current system
for populating and managing expert panels can be improved:
• Although NCEA strives to select "impartial" panelists, this concept is
vaguely defined and not explained in any NCEA-specific operating guidance.
• NCEA does not have procedures for addressing conflicts of interest or
potential biases that become known after a panel has completed deliberations.
• There was no clear documentation of authority and responsibility for making
final determinations regarding panel selection or how potential conflicts of
interest were resolved.
Following a prior OIG report, NCEA improved its peer review process by
developing a questionnaire for EPA contractors to use in identifying potential
conflicts of interests or biases of prospective panel members. Also, according to
the NCEA Director, NCEA recently started to document its peer review process
and is implementing a quality assurance checklist to ensure EPA contractors
follow EPA's procedures.
What We Recommended
We recommended that the Assistant Administrator for Research and
Development, which oversees NCEA, improve management controls by better
defining the concept of "impartiality" and maintaining records of all management
decisions pertaining to the selection of peer reviewers, particularly resolution of
potential conflicts of interest. We also recommended that the Assistant
Administrator develop guidance to address conflict of interest issues that arise
after panel formulation and amend contracts for external peer review services to
require that panelists re-certify their conflict of interest status prior to the panel
convening. The Office of Research and Development agreed with our
recommendations, and the Assistant Administrator's planned actions meet the
intent of our recommendations. Additional information is needed regarding the
timeframe for the Agency's implementation of one of our recommendations.
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
INSPECTOR GENERAL
April 29, 2009
MEMORANDUM
SUBJECT:
FROM:
TO:
EPA Can Improve Its Process for Establishing Peer Review Panels
Report No. 09-P-0147
Wade T. Najjum
Assistant Inspector General for Program Evaluation
Lek G. Kadeli
Acting Assistant Administrator for Research and Development
This is our report on the subject evaluation conducted by the Office of Inspector General (OIG)
of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe
the problems the OIG has identified and corrective actions the OIG recommends. This report
represents the opinion of the OIG and does not necessarily represent the final EPA position.
Final determinations on matters in this report will be made by EPA managers in accordance with
established audit resolution procedures.
The estimated cost of this report - calculated by multiplying the project's staff days by the
applicable daily full cost billing rates in effect at the time - is $272,110.
Action Required
In accordance with EPA Manual 2750, EPA 's Audit Management Process, you should provide a
written response within 90 calendar days. Your response should include a planned completion
date for Recommendation 1. Since you submitted a corrective action plan that sufficiently
addresses Recommendations 2 through 7, we are "closing" all recommendations, except for
Recommendation 1, in our tracking system upon issuance of this report. These
recommendations will be tracked to completion in the Agency's tracking system. No further
response is required for Recommendations 2 through 7. As outlined in EPA Manual 2750, the
Agency is responsible for tracking the implementation of these actions in its Management Audit
Tracking System. We have no objections to the further release of this report to the public. This
report will be available at http://www.epa.gov/oig.
If you or your staff have any questions regarding this report, please contact me at (202) 566-0827
or najjum.wade@epa.gov: or Rick Beusse, Director for Program Evaluation, Air and Research
Issues, at (919) 541-5747 or beusse.rick@epa.gov.
-------
EPA Can Improve Its Process for 09-P-0147
Establishing Peer Review Panels
Table of Contents
Purpose 1
Background 1
Laws, Regulations, and Guidance Applicable to EPA Peer Reviews 2
NCEA's Peer Review Process 3
Noteworthy Achievements 4
Scope and Methodology 4
Results of Review 5
Key Steps in NCEA's Process Used to Determine
Potential Conflicts of Interest and Impartiality 5
NCEA's Process is Similar to Other Peer Review Processes 5
NCEA Can Improve its Peer Review Process in Certain Areas 6
Conclusions 7
Recommendations 7
Agency Comments and OIG Evaluation 8
Status of Recommendations and Potential Monetary Benefits 10
Appendices
A Peer Review Process Steps 12
B Agency Response to Draft Report 13
C Distribution 17
-------
09-P-0147
Purpose
In response to concerns about the U.S. Environmental Protection Agency's (EPA's) handling of
allegations of impartiality on one of its peer review panels, former EPA Deputy Administrator
Marcus Peacock requested that the Office of Inspector General (OIG) review EPA's peer review
process. The objectives of our review1 were to determine whether:
• current laws, regulations, guidance, and other relevant requirements for such panels
are adequate to produce objective scientific reviews; and
• the current system of populating and managing such expert panels could be improved.
Background
The foreword to EPA's Peer Review Handbook notes that strong, independent science is of
paramount importance to EPA's environmental policies. The quality of the science that
underlies EPA's regulations is vital to the credibility of EPA's decisions and, ultimately, the
Agency's effectiveness in protecting human health and the environment. The peer review
process enhances a scientific or technical work product so that the decision or position taken by
the Agency, based on that product, has a sound, credible basis. It involves the review of a draft
product by specialists in the field who were not involved in producing the draft. The peer
reviewers then issue a report - an evaluation or critique - that is used by the authors of the draft
to improve the product so that the final work product will reflect sound technical information and
analyses. As described in the Handbook, peer reviewers typically evaluate the:
• clarity of hypotheses,
• validity of the research design,
• quality of data collection procedures,
• robustness of the methods employed,
• appropriateness of the methods for the hypotheses being tested,
• extent to which the conclusions follow from the analysis, and
• strengths and limitations of the overall product.
EPA's National Center for Environmental Assessment (NCEA), within EPA's Office of
Research and Development, produces highly influential scientific assessments and thus is one of
EPA's primary users of peer review services. EPA's NCEA uses extramural instruments, such
as contracts and interagency agreements, to obtain peer review services to review highly
influential scientific assessments, such as human health risk assessments. NCEA oversees the
peer review process.
1 The Deputy Administrator also asked OIG to determine whether the actions taken by EPA or the panelists were
done consistent with existing federal law, regulations, guidance, and other relevant requirements. This objective
was addressed in a separate OIG report.
-------
09-P-0147
Laws, Regulations, and Guidance Applicable to EPA Peer Reviews
The primary laws, regulations, and guidance governing peer review at EPA include the Office of
Management and Budget's (OMB's) "Final Information Quality Bulletin for Peer Review,"
EPA's Peer Review Handbook, and the Federal Advisory Committee Act (FACA).
OMB's Bulletin, issued in 2004, is the primary guidance for government agencies regarding peer
reviews. The OMB Bulletin was based primarily on procedures used by the National Academy
of Sciences (NAS). The OMB Bulletin established four key criteria to guide federal agencies in
selecting a peer review panel.
1. Expertise is the most important factor when selecting a panelist.
2. Except for situations where it is unavoidable, panelists should be free of conflicts
of interest. A "conflict of interest" is any financial or other interest that conflicts
with the service of an individual on the review panel because it could impair the
individual's objectivity or could create an unfair competitive advantage for a
person or organization.
3. Panelists should be independent. The panelists should not have worked on the
product being reviewed.
4. The panel should be balanced. Reviewers should be selected to represent a
diversity of scientific perspectives.
EPA's January 2006 Peer Review Policy memo establishes the policy for peer review of scientific
and technical work products, including economic and social science products that are intended to
inform Agency decisions. EPA's Peer Review Handbook is the Agency's primary guidance
governing peer reviews. The Handbook was most recently revised in 2006 to be consistent with
the provisions of the 2004 OMB Bulletin. Although the Handbook outlines EPA's preferred
approach to ensuring the quality of peer
i ,i • •,• , 11 ,i Figure 1. Relation between all EPA work products and
reviews conducted or initiated by the those considered inf|uential scientific information or
Agency, it is not a requirement. highly influential scientific assessments
The principle underlying EPA's Peer
Review Policy is that all influential
scientific and technical work products used
in decision making will be peer reviewed.
Determining whether a scientific and/or
technical work product is "influential" or
"highly influential" is done on a case-by-
case basis, taking into account various
criteria and the circumstances surrounding
the use of the work product. OMB defines
highly influential scientific assessments as
influential scientific information that the
Agency considers (1) could have a
potential impact of more than $500 million
in any year; or (2) is novel, controversial,
Influential Scientific Information
Highly Influential Scientific
Assessments
Source: EPA Peer Review Handbook, 3 Edition (2006)
-------
09-P-0147
or precedent-setting, or has significant interagency interest. The OMB Bulletin calls for
additional peer review procedures for highly influential scientific assessments.
The FACA governs committees that advise the federal government on a variety of issues,
including peer review of scientific research. Among other requirements, FACA committee
membership must be balanced in terms of points of view represented and the functions to be
performed by the committee. EPA's Science Advisory Board (SAB) is a FACA committee and
often conducts peer reviews of EPA products. Consequently, the SAB must manage its peer
review panels in accordance with FACA requirements. Some statutes, such as the Clean Air Act,
mandate a peer review process for certain EPA decisions. These peer reviews are conducted by
FACA Committees supported by SAB, and are subject to FACA requirements.
NCEA's Peer Review Process
There are no laws or regulations specifying requirements for the peer reviews conducted by
NCEA. The peer review mechanism used by NCEA to conduct peer review in any particular
case is within its discretion. The majority of NCEA's peer reviews are for assessments
conducted for the Integrated Risk Information System program. Integrated Risk Information
System documents describe the health effects of individual substances and contain descriptive
and quantitative information on their cancer and noncancer effects. The system is described in
EPA 's Integrated Risk Information System: Assessment Development Procedures. These
procedures allow for OMB and interagency review and input on the external peer review charge
questions developed by EPA. NCEA's process for peer review includes:
• conducting all peer reviews for influential scientific information, including highly
influential scientific assessments and work products, by peer review panels in
accordance with the Agency's Peer Review Handbook;
• providing members of each panel access to public comments received on the
documents under review; and
• using the specified Conflicts of Interest questionnaire (the questionnaire includes a
series of yes/no questions and request for supporting documentation) when peer
review services are obtained through contracts and interagency agreements.
Specific requirements for conducting peer reviews are included in contract and interagency
agreement statements of work. Depending upon the scientific product, NCEA may obtain peer
review services from the NAS, the SAB, an EPA contract, or under an interagency agreement (at
the time we conducted our review, NCEA had an interagency agreement with the Department of
Energy's Oak Ridge Institute for Science and Education (ORISE)). According to NCEA officials,
the assessments dealing with the most complex issues are given to the NAS to peer review; these
occur about one every few years. Other assessments dealing with complex issues are given to
SAB for peer review; these average from two to four each year. The majority of assessments are
either reviewed under the peer review contract or an interagency agreement with another federal
agency; in the past few years, NCEA has predominantly obtained peer review services for IRIS
assessments through the ORISE interagency agreement. As of March 2009, NCEA does not plan
to acquire peer review services under the ORISE interagency agreement since it expires in March
2009 and funds are no longer available to purchase such interagency services.
-------
09-P-0147
Noteworthy Achievements
In response to a prior OIG evaluation,2 NCEA developed a questionnaire and received OMB
approval for EPA contractors to use the questionnaire to help identify potential conflicts of
interest or potential biases that may affect the selection of a potential panel member. The
questionnaire asks potential panelists to address possible financial conflicts of interest of
prospective panelists and their family members, as well as possible non-financial independence
and impartiality issues.
During the current evaluation, NCEA started taking actions to improve its peer review process.
According to the NCEA Director, NCEA is developing a description of the peer review process
used in the Integrated Risk Information System program that it intends to post to its Website as a
reference for staff and others.
Scope and Methodology
Our review was limited to a design evaluation of EPA's external peer review process, with a
primary focus on NCEA's peer review process. Accordingly, our report contains
recommendations that apply to EPA's external peer review process in general, as well as
recommendations that apply specifically to NCEA's process. NCEA oversees the peer review of
EPA's health risk assessments, specifically the peer review panel process that prompted the
former EPA Deputy Administrator's request to us. We reviewed the applicable laws,
regulations, policies, and guidance related to the establishment of peer review panels for the
independent peer review of NCEA research. This review included OMB's 2004 bulletin, "Final
Information Quality Bulletin for Peer Review"; and EPA's Peer Review Handbook, 3rd Edition,
June 2006. We also reviewed the FACA, as amended in 1997. We interviewed key officials
within NCEA to understand the process NCEA uses to establish peer review panels. We
reviewed NCEA's peer review services contract and the interagency agreement used to acquire
peer review support. We also interviewed individuals from the NAS, EPA's SAB, the
Toxicology Excellence for Risk Assessment (TERA) organization,3 and the National Institute of
Environmental Health Sciences (NIEHS) to identify the practices used by other major
organizations to identify and select peer review panelists.
We also reviewed our prior report on the peer review process (see footnote 2) and confirmed that
NCEA implemented the recommendations in that report. In addition, we reviewed two prior
U.S. Government Accountability Office reports on the SAB's peer review process.4
EPA OIG. Review of Conflict of Interest Allegations Pertaining to the Peer Review of EPA 's Draft Report,
"Exposure and Human Health Evaluation of Airborne Pollution from the World Trade Center Disaster. "
Report No. 2005-S-00003. November 4, 2004.
3 TERA is a non-profit, 501(c)(3) corporation organized for scientific and educational purposes. TERA's mission is
to protect public health by developing and communicating risk assessment information, sponsoring peer reviews and
consultations, improving risk methods through research, and educating the public on risk assessment issues.
4 EPA 's Science Advisory Board Panels: Improved Policies and Procedures Needed to Ensure Independence and
Balance. GAO-01-536. June 12, 2001; and Federal Advisory Committees: Additional Guidance Could Help
Agencies Better Ensure Independence and Balance. GAO-04-328. April 16,2004.
-------
09-P-0147
We performed our evaluation between May 2008 and February 2009 in accordance with
generally accepted government auditing standards issued by the Comptroller General of the
United States. Those standards require that we plan and perform the evaluation to obtain
sufficient and appropriate evidence to provide a reasonable basis for our findings and
conclusions based on our objectives. We believe that the evidence obtained provides a
reasonable basis for our findings and conclusions based on our evaluation objectives.
Results of Review
We found that the laws, regulations, guidance, and other relevant requirements governing EPA's
peer review process are adequate to produce objective scientific reviews, but EPA can better
define certain areas of its operating guidance. NCEA's peer review process is similar to the
process of other major peer review organizations we reviewed. However, NCEA can improve its
system for populating and managing expert panels by better documenting conflict of interest
decisions, establishing guidance for handling conflict of interest issues that arise after the panel
has completed its deliberations, and providing more consistency between contractor and other
third party procedures for selecting panels.
Key Steps in NCEA's Process Used to Determine Potential Conflicts of Interest
and Impartiality
EPA's contractors and ORISE require potential panelists to complete conflict of interest
questionnaires that address financial conflicts of interest of panelists and their family members, as
well as non-financial independence and impartiality issues. Panelists must certify that their
answers are correct. The questions solicit "yes" or "no" responses. If a potential conflict of
interest is indicated, the contractor or ORISE seeks additional information from the potential
panelist. If the contractor or ORISE have difficulty making a determination regarding conflict of
interest or lack of impartiality, the matter is brought to the attention of EPA. If conflict of interest
is identified, the contractor must notify EPA's Project Officer, who in turn brings the issue to the
attention of NCEA's Director or Associate Director for Health for resolution.
Prior to the panel convening, ORISE asks the selected panelists to confirm by return e-mail that
there are no changes to their previous conflict of interest certification form. If their responses to
any of the questions have changed, they must explain the changes to ORISE. ORISE changed
this procedure beginning in June 2007. ORISE now sends a copy of the panelist's completed
conflict of interest questionnaire back to the prospective peer reviewer so that they can review
their original answers before confirming current status.
NCEA's Process is Similar to Other Peer Review Processes
We compared NCEA's peer review panel selection process with the process of four other
science-based organizations.5 We found that NCEA's process does not differ in many aspects
from the processes used by these other major science-based organizations. Appendix A provides
the details on our comparison.
; The four organizations are the NAS, EPA's SAB, TERA, and NIEHS.
-------
09-P-0147
One noteworthy difference between NCEA's process and other organizations is panel selection
and public input. FACA panels for NAS and the National Academy of Public Administration are
required to obtain public comment on proposed panelists. While this requirement does not apply
to EPA SAB panels, the SAB has adopted this practice. Unlike the NAS and the SAB, NCEA's
consultants do not obtain public input or comment on proposed panel members. In addition,
FACA panels such as those convened by SAB and NAS attempt to achieve consensus among its
panelists, and concerns about the impartiality of panel members can be mitigated by balancing
the panel. Peer review panels established through NCEA's extramural instruments do not seek
consensus. Thus, NCEA does not mitigate the inclusion of impartial panel members through
panel balance, but instead chooses to leave potentially partial (or biased) panelists off the panel.
TERA was the only organization we contacted that provides the basis of its conflict of interest
decisions to the public. TERA's peer review reports identify appearances of potential conflicts
of interest that panelists may have and provide TERA's reasons for selecting these panelists.
NIEHS uses peer reviews to evaluate studies completed for their National Toxicology Program.
Rather than solely relying on completed questionnaires, NIEHS also conducts searches to
identify possible impartialities or biases possessed by potential panel members.
NCEA Can Improve its Peer Review Process in Certain Areas
Although NCEA's external peer review process incorporates many of the procedures and
controls used by other peer review organizations, certain areas of the process can be improved to
provide more consistency and transparency to the process. These areas are described below.
• Although NCEA strives to select "impartial" panelists, this concept is vaguely defined by
OMB and EPA guidance and is not explained in any NCEA-specific operating guidance.
Neither the 2004 OMB Bulletin nor the EPA Handbook defines what constitutes
"impartiality." According to the Handbook, in general potential panelists who had a
predominant influence on an organization's position or have taken a public position or "taken
sides" should be avoided.
• There was no clear documentation of authority and responsibility for making final
determinations regarding panel selection or how potential conflicts of interest were resolved.
• NCEA can improve staff and the public's understanding of its external peer review process
by fully describing the process and making that description available to EPA staff and the
public. For example, TERA provides a description of its peer review process and its
procedures for panel selection on the Internet. NCEA currently does not have a
comprehensive description of its external peer review process, although the NCEA Director
said one is being developed and will be made available on its public Website.
• NCEA does not have procedures for addressing conflicts of interest or potential biases, or
allegations of such that become known or alleged after a panel has begun or completed its
deliberations. NCEA does not have a policy or procedures regarding the circumstances
under which a panelist's pay may be recouped or withheld when the panelist is dismissed or
resigns before completion.
-------
09-P-0147
• Although NCEA's contractors conduct Internet searches to identify potential conflicts of
interest and appearances of bias or partiality, ORISE - the current provider of peer review
services under an interagency agreement - does not conduct Internet background searches.
NCEA could improve its peer review process by establishing procedures for providers of
peer review services to follow when conducting independent background searches on
prospective panelists.
• NCEA's contractors do not use similar procedures for identifying any changes in selected
panelists' conflict of interest status. One of EPA's two contractors told us it asks panelists at
the first meeting of the panel if there have been any changes in their conflict of interest
status. Any changes should be brought to the attention of EPA officials. However, the EPA
program manager for the contract could not provide documentation that the panelists'
answers were placed in the peer review record. According to the EPA program manager for
the second contractor, it does not ask panelists if there have been changes in their conflict of
interest status. After our inquiry, the program manager told us it plans to incorporate
procedures to identify whether any changes in status have occurred between the time
panelists complete their conflict of interest questionnaire and begin panel deliberations.
• NCEA can improve its oversight of peer reviews conducted by third parties to better ensure
these peer reviews follow contractual guidelines. NCEA is working to develop an oversight
tool to help ensure that significant steps in the peer review process are followed. Such a tool
could also be useful in providing oversight of NCEA's peer review contracts and any future
interagency agreements NCEA may use to obtain peer review services.
Conclusions
Certain areas of NCEA's current system for populating and managing expert panels can be
improved. Although NCEA strives to select "impartial" panelists, this concept is vaguely
defined and not explained in any NCEA-specific operating guidance. NCEA does not have
procedures for addressing conflicts of interest or potential biases that become known after a
panel has completed its deliberations. We also found that there was no clear documentation of
authority and responsibility for making final determinations regarding panel selection or how
potential conflicts of interest were resolved. We concluded that NCEA did not have adequate
controls to establish accountability for suitability determinations and rationale for including or
excluding each panelist.
Recommendations
We recommended that the Assistant Administrator for Research and Development:
1. Establish criteria, definitions, and/or example scenarios for the Peer Review Handbook
term "appearance of a lack of impartiality," under which contractors and other external
peer review services providers should operate.
-------
09-P-0147
2. Require and confirm that peer review records are maintained throughout the peer review
process and that these records include any correspondence and decisions related to
suitability, or potential conflicts of interest or biases of prospective panelists. In cases
where panelists with potential conflicts or biases are accepted on the panel, the records
should include a memorandum of decision explaining the suitability and rationale for
including or excluding each panelist, which is signed off on by an EPA official.
3. Publish a description of the peer review process used in the Integrated Risk Information
System program as a reference for staff and others. The process description should
clearly define the roles and responsibilities of persons involved throughout the peer
review process.
4. Establish procedures for addressing conflict of interest and lack of impartiality issues that
arise after panel selection. These procedures should discuss under what circumstances
peer review panelists' pay may be recouped or withheld.
5. Amend all extramural instruments to call for background Internet searches on potential
panel members.
6. Modify NCEA's peer review contracts to require written recertification from panelists,
before a peer review panel is convened, stating that their responses to the questionnaire
have not changed. A copy of the questionnaire completed by the panelist should be
included with the request for a written recertification. For both contracts and interagency
agreements, EPA should require that reviewers self report any changes that may impact
their conflict of interest status or lack of impartiality status at any point in the process. In
cases where the Agency obtains the services of a reviewer through purchase orders not
connected with contracts or interagency agreements, or without compensation, the terms
should likewise require that reviewers self report changes that may impact their conflict
of interest status or lack of impartiality status.
7. Develop an oversight tool to ensure that external peer review service providers follow all
significant steps in the peer review process.
Agency Comments and OIG Evaluation
The Agency agreed with the report's conclusions and recommendations and proposed corrective
actions that the Office of Research and Development plans to take in response to our
recommendations. For six recommendations the planned corrective actions and planned
timeframes for completion meet the intent of our recommendations. We are closing
Recommendations 2 through 7 in our tracking system upon issuance of this report. These
recommendations will be tracked to completion in the Agency's tracking system. The Agency's
planned action in response to Recommendation 1 also meets the intent of our recommendation,
but additional information is needed regarding the timeframe for the Agency's implementation of
this recommendation. Accordingly, Recommendation 1 remains open pending our receipt of an
estimated completion date for the Agency's proposed corrective action for this recommendation.
-------
09-P-0147
The Agency also provided several technical clarifications and comments to the report. We made
changes to the final report based on these comments, as appropriate. The Agency's complete
written response is in Appendix B.
-------
09-P-0147
Status of Recommendations and
Potential Monetary Benefits
RECOMMENDATIONS
POTENTIAL MONETARY
BENEFITS (in SOOOs)
Rec.
No.
1
Page
No.
7
Subject
Establish criteria, definitions, and/or example
Status1
0
Action Official
Assistant Administrator for
Planned
Completion
Date
Claimed
Amount
Agreed To
Amount
scenarios for the Peer Review Handbook term
"appearance of a lack of impartiality," under which
contractors and other external peer review services
providers should operate.
Require and confirm that peer review records are
maintained throughout the peer review process and
that these records include any correspondence and
decisions related to suitability, or potential conflicts of
interest or biases of prospective panelists. In cases
where panelists with potential conflicts or biases are
accepted on the panel, the records should include a
memorandum of decision explaining the suitability and
rationale for including or excluding each panelist,
which is signed off on by an EPA official.
Publish a description of the peer review process used
in the Integrated Risk Information System program as
a reference for staff and others. The process
description should clearly define the roles and
responsibilities of persons involved throughout the
peer review process.
Establish procedures for addressing conflict of interest
and lack of impartiality issues that arise after panel
selection. These procedures should discuss under
what circumstances peer review panelists' pay may be
recouped or withheld.
Amend all extramural instruments to call for
background Internet searches on potential panel
members.
Modify NCEA's peer review contracts to require
written recertification from panelists, before a peer
review panel is convened, stating that their responses
to the questionnaire have not changed. A copy of the
questionnaire completed by the panelist should be
included with the request for a written recertification.
For both contracts and interagency agreements, EPA
should require that reviewers self report any changes
that may impact their conflict of interest status or lack
of impartiality status at any point in the process. In
cases where the Agency obtains the services of a
reviewer through purchase orders not connected with
contracts or interagency agreements, or without
compensation, the terms should likewise require that
reviewers self report changes that may impact their
conflict of interest status or lack of impartiality status.
Research and Development
Assistant Administrator for
Research and Development
6/30/09
Assistant Administrator for 6/30/09
Research and Development
Assistant Administrator for 6/30/09
Research and Development
Assistant Administrator for 6/30/09
Research and Development
Assistant Administrator for
Research and Development
6/30/09
10
-------
09-P-0147
POTENTIAL MONETARY
RECOMMENDATIONS BENEFITS (in $OOOs)
Planned
Rec. Page Completion Claimed Agreed To
No. No. Subject Status1 Action Official Date Amount Amount
Develop an oversight tool to ensure that external peer C Assistant Administrator for 6/30/09
review service providers follow all significant steps in Research and Development
the peer review process.
0 = recommendation is open with agreed-to corrective actions pending;
C = recommendation is closed with all agreed-to actions completed;
U = recommendation is undecided with resolution efforts in progress
11
-------
09-P-0147
Appendix A
Peer Review Process Steps
Process Steps
Prospective panelists identified from
database of experts.
Public is allowed the opportunity to
recommend panel members.
Prospective panelists complete detailed
conflict of interest and an appearance
of a lack of impartiality questionnaire.
Prospective panelists provide
information on past employment,
research, etc.
Independent research conducted to
identify potential panelist biases or
conflicts.
Panelists are questioned about
potential conflicts noted from the
information provided.
Names of prospective panelists
published and public comment
requested prior to final panel selection.
Selected panelists verify no changes in
conflict of interest status just prior to
panel meeting.
Agency decisions on potential conflict
of interest documented in Peer Review
report.
NCEA
Contracts
Yes
No
Yes
Yes
Yes
Yes
No
Yes (verbal)
No
NCEA-ORISE
Yes
No
Yes
Yes
No
Yes
No
Yes (e-mail) D
No
MAS
Partially
Yes
Yes
Yes
Sometimes
Yes
Yes
Yes (verbal)
No
EPA SAB
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes (written)
No
TERA
Yes
Yes, if requested
by sponsor
Yes
Yes
Yes
Yes
No
Yes (verbal)
Yes
NIEHS
National
Toxicology
Program
Yes
Yes
Partial a
Yes
Yes
Yes
No
Yes (written)
No
Source: OIG-prepared table using information obtained from NCEA, NCEA contractors, ORISE, NAS, SAB, TERA, and NIEHS.
3 Instead of relying solely on information provided by panelists, NIEHS conducts independent research to determine whether potential biases or conflicts exist.
NIEHS does not use a checklist. Potential panelists provide curricula vitae that address the needed information.
b Procedure changed to require documented re-confirmation that no conflicts exist.
12
-------
09-P-0147
Appendix B
Agency Response to Draft Report
April 24, 2009
OFFICE OF
RESEARCH AND DEVELOPMENT
MEMORANDUM
SUBJECT: ORD Response to the Office of Inspector General Draft
Evaluation Report: EPA Can Improve Its Process for Establishing Peer
Review Panels, Assignment No. OPE-FY08-0007
FROM: LekG. Kadeli /s/
Acting Assistant Administrator
TO: Wade T. Najjum
Assistant Inspector General for Program Evaluation
Office of Inspector General
Attached please find: (1) A summary table of Office of Research and Development's (ORD)
responses to the Inspector General's seven recommendations and (2) summary of
comments on the Draft Evaluation Report regarding Peer Review. The comments
represent collaboration across ORD because many of the draft report's recommendations
could have impact to others in ORD. Any recommendations that go beyond ORD will be
discussed with the Science Policy Council.
Thank you for providing the report and giving consideration to our response.
cc: Kevin Teichman
Peter Preuss
Fred Hauchman
13
-------
09-P-0147
ORD's Response to OIG Draft Report
"EPA Can Improve Its Process for Establishing Peer Review Panels"
Assignment No. OPE-FY08-0007
This document is comprised of three sections:
1. Table of ORD's Response to IG's seven recommendations
2. Recommendations for editing the Report regarding the scope
3. Specific comments by page number
4. General comments regarding the recommendations
1. Table of ORD's Response to IG Recommendations
Rec.
No.
1
2
3
Subject
Establish criteria, definitions,
and/or example scenarios for the
Peer Review Handbook term
"appearance of a lack of
impartiality," under which
contractors and other external
peer review services providers
should operate.
Require and confirm that peer
review records are maintained
throughout the peer review
process and that these records
include any correspondence and
decisions related to suitability, or
potential conflicts of interest or
biases of prospective panelists.
The records should include a
Memorandum of Decision
explaining the suitability and
rationale for including or
excluding each panelist, which is
signed off on by an EPA official
Publish a description of the peer
review process used in the
Integrated Risk Information
System program as a reference
for staff and others. The process
description should clearly define
the roles and responsibilities of
persons involved throughout the
peer review process.
Lead
Responsibility
Office of
Science
Advisor
ORD
ORD
ORD's Recommendation
ORD agrees with this recommendation.
The Office of the Science Advisor (OSA) will
coordinate with the Science Policy Council
(SPC) to consider any potential revisions to
the Agency Peer Review Handbook
regarding establishing criteria, definitions
and/or example scenarios for the term
"appearance of a lack of impartiality." ORD
will provide OSA with scenarios for
consideration.
ORD agrees with this recommendation. For
non-FACA reviews, ORD will develop
documentation to specify the addition of a
memo to ORD peer review files confirming
records are maintained, and the inclusion of
all correspondence and records of
discussion during the peer review panel
selection process. ORD will develop a
Standard Operating Practice (SOP) and
perform periodic audits of peer review files
to ensure this new requirement is being
met.
ORD agrees with this recommendation.
ORD is currently re-writing the IRIS peer
review SOP description. The final IRIS peer
review process SOP will be posted to the
IRIS website for use by EPA staff,
contractors, and the public.
Planned
Completion
Date
TBDin
consultation
with the SPC
June 30,
2009
June 30,
2009
14
-------
09-P-0147
Establish procedures for
addressing conflict of interest
and lack of impartiality issues
that arise after panel selection.
These procedures should
discuss under what
circumstances peer review
panelists' pay may be recouped
or withheld.
ORD
ORD agrees with this recommendation.
ORD will develop procedures and document
them in the IRIS peer review SOP. ORD
will consult with the Office of General
Counsel (OGC).
June 30,
2009
Amend all extramural
instruments to call for
background Internet searches
on potential panel members.
ORD
ORD agrees with this recommendation.
Based on the EPA's Science Advisory
Board protocol, ORD will add direction to
conduct background Internet searches to all
NCEA contract task orders. ORD will
consult with OGC.
June 30,
2009
Modify NCEA's peer review
contracts to require written
recertification from panelists,
before a peer review panel is
convened, stating that their
responses to the questionnaire
have not changed. A copy of the
questionnaire completed by the
panelist should be included with
the request for a written
recertification. For both contracts
and interagency agreements,
EPA should require that
reviewers self report any
changes that may impact their
conflict of interest status or lack
of impartiality status at any point
in the process.
ORD
ORD agrees with this recommendation.
ORD will develop additional language for
contract task orders requiring initial written
certification at the time of empanelment,
written (email) recertification about two
weeks prior to the panel meeting at the
beginning of panel meetings, and self
reporting of any changes that may impact
their conflict status or lack of impartiality
status at any point in the process. ORD will
consult with OGC.
June 30,
2009
Develop an oversight tool to
ensure that external peer review
service providers follow all
significant steps in the peer
review process.
ORD
ORD agrees with this recommendation.
ORD will develop a QA checklist for
contractors to use for all ORD peer reviews.
June 30,
2009
2. Recommendations for editing the Report regarding the scope
The scope of the Report is defined in the Purpose section on Page 1 ("... EPA Deputy Administrator
Marcus Peacock requested that the Office of Inspector General (OIG) review EPA's peer review
process.") and the Scope and Methodology section on Page 4 ("Our review was limited to a design
evaluation of EPA's external peer review process, with a primary focus on NCEA's peer review process.")
The Report provides recommendations for the ORD Assistant Administrator as the Action Official to
address. Some of the recommendations are within the scope of NCEA to address in full (recommendation
3, 6) whereas others (recommendations 1, 2, 4, 5, 7), while actionable by NCEA, also affect ORD and
EPA more broadly. We recommend that for each instance where the report makes reference to
"EPA" and to "NCEA" that the appropriate level of the organization be reviewed to confirm that
the scope of the organization being referenced is consistent with the intended scope of the text
and recommendations. For example, the Results of Review section on pages 5-7 refers only to NCEA,
yet the scope of many recommendations on pages 7-8 is not limited to NCEA.
15
-------
09-P-0147
3. Specific comments:
Page 1, Background: the text of the report indicates that NCEA uses grants to obtain peer review
services. This is not correct. NCEA has used contracts and Interagency Agreements to obtain peer
review services.
Page 3, last paragraph, last sentence - replace "In the future" with "As of March 2009," NCEA does not
plan to acquire peer review services under the ORISE ...
Page 5, "Key steps...," reference is made to the Deputy Director for Health"; the correct reference here
should be to the "Associate Director for Health." In addition, on the bottom of page 5 of the report, the
statement is made that "FACA panels are required to obtain public comment on proposed panelists."
This statement is factually incorrect. FACA does not require this, and neither does EPA. The SAB does
obtain public comment on proposed panelists; however, this is not an agency-wide requirement, it is an
SAB best practice.
Page 6, the last bullet at bottom of this page. It is correct that ORISE did not conduct internet searches
and it is correct that our other contractors, ERG and Versar, have conducted Internet background
searches. ERG often does internet searches to identify candidates but does not do an independent
internet search after receiving the self reporting questionnaire. Versar also does internet searches,
particularly for controversial reviews.
Page 7, last bullet before "Conclusions," it is stated that NCEA is working with ORISE to develop an
oversight tool. The specific reference to ORISE should be deleted and be stated instead as "NCEA is
working to develop an oversight too..." since the peer review work with ORISE is ending.
4. General comments:
a) With regard to specific recommendations, one in particular (#2) would benefit from further elaboration.
It states that peer review records should "include a memorandum of decision explaining the suitability and
rationale for including or excluding each panelist." Two examples are given below to illustrate why
additional explanation by the OIG would be helpful:
• Lists of prospective panelists are often lengthy, and many worthy candidates may be
excluded simply because they are unavailable. Would a memo of decision be required for
this circumstance, or is the OIG referring to willful exclusion?
• Conversely, when an individual is included on a panel, current practices for justifying their
selection (background searches and documentation of expertise, work history, stature or
prominence in the field, etc.) are in our view sufficient to explain the rationale, so long as
there are no conflicts of interest. Perhaps a memo of decision is necessary only in those
cases where a panelist is selected despite some objection or perceived bias
b) Regarding recommendation #6, in cases where the Agency obtains the reviewer through purchase
orders not connected with contracts or lAGs, or without compensation, the terms should likewise required
that reviewers self report changes that may impact their conflict of interest status or lack of impartiality
status.
c) Finally, we have reviewed and are in support of the comments submitted by the Office of the Science
Advisor. The comments will be submitted by OSA.6
6 OIG Note: The Office of Research and Development clarified that it had already incorporated the Office of the
Science Advisor's comments into the above response.
16
-------
09-P-0147
Appendix C
Distribution
Office of the Administrator
Acting Assistant Administrator for Research and Development
Director, National Center for Environmental Assessment
Acting Science Advisor
Agency Follow-up Official (the CFO)
Agency Follow-up Coordinator
Acting General Counsel
Acting Associate Administrator for Congressional and Intergovernmental Relations
Acting Associate Administrator for Public Affairs
Audit Follow-up Coordinator, Office of Research and Development
Acting Inspector General
17
------- |