United States Science Advisory EPA-SAB-RSAC-00-002
Environmental Board (1400A) November 1999
Protection Agency Washington, DC ivww.epa.gov/sab
EPA AN SAB REPORT: REVIEW
OF THE PEER REVIEW
PROGRAM OF THE
ENVIRONMENTAL
PROTECTION AGENCY
A REVIEW BY THE RESEARCH
STRATEGIES ADVISORY
COMMITTEE (RSAC) OF THE
SCIENCE ADVISORY BOARD
-------
November 22, 1999
EPA-SAB-RSAC-00-002
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: An SAB Report: Review of the Peer Review Program of the
Environmental Protection Agency
Dear Ms. Browner:
The review conducted by the Research Strategies Advisory Committee (RSAC) on
September 23-24, 1999 focused on whether or not key components of sound peer review process
are in place at the EPA, whether appropriate tools and training are available, and whether
management commitment exists to carry out EPA directives for peer review. The Committee was
pleased to see the Agency's diligence with respect to Peer Review. From the materials presented
to RSAC, EPA's Peer Review Process is well articulated and appears to be fundamentally sound
and, with a few exceptions, working as intended. Future RSAC reviews are planned to evaluate
the extent, adequacy and timeliness to which peer review is carried out by various program
offices, Regions and EPA laboratories.
The RSAC found that peer review processes appear to be well established at the EPA. In
addition, these processes are continuing to improve through a mechanism of continued internal
examination, led by the Office of Research and Development, and process changes carried out by
Decision-makers at the direction of the Science Policy Council. The driver for the Peer Review
Program has been established by EPA management leadership. This high level management
commitment to peer review major EPA Technical and Scientific products was manifested in such
actions as the 1994 Peer Review Policy statement, the establishment of the Science Policy
Council, the Peer Review Handbook, and most recently the establishment of a policy that all rules
planning and analytic blueprints must specifically address and include consideration of peer
review. The continued success of the Peer Review process at EPA will only be ensured with
continual committment on the part of Senior managers to the Peer Review process in all
appropriate aspects of the Agency's endeavors. At the same time, RSAC recommends that EPA
take the necessary steps to keep the process from becoming inappropriately bureaucratic.
The RSAC has made several suggestions to strengthen the peer review process. A key
-------
suggestion is expanding the scope to the evaluation of interagency and international products
considered important to environmental decision-making. Peer review should also be extended to
the up-front review of scientific and technical planning products such as strategic plans, analytic
blueprints, research plans, and environmental goals documents. In addition to major science and
technology products, social science research products can and should be subjected to the peer
review process in a manner similar to natural science products. Products that are policy analytic,
in that they are not purely science-based but involve the application of policy and values, should
also be peer reviewed to ensure that appropriate methods and procedures have been used,
including an explicit treatment of assumptions and value judgments, adequate sensitivity analysis,
and adequate treatment of uncertainty. Thus, the explicit need to review major social science
products (in addition to economic products), and policy analytic products should be added to the
Peer Review Handbook. Finally there should be a requirement for completion of training before a
person can be designated as a Peer Review Leader.
We appreciate the opportunity to review and provide advice on the Agency's Peer Review
process. The Research Strategies Advisory Committee would be pleased to expand on any of the
findings in the attached report, and we look forward to your response.
Sincerely,
Dr. Joan M. Daisey, Chair Dr. William Randall Seeker, Chair
Science Advisory Board Research Strategies Advisory Committee
Science Advisory Board
-------
NOTICE
This report has been written as part of the activities of the Science Advisory Board, a
public advisory group providing extramural scientific information and advice to the Administrator
and other officials of the Environmental Protection Agency. The Board is structured to provide
balanced, expert assessment of scientific matters related to problems facing the Agency. This
report has not been reviewed for approval by the Agency and, hence, the contents of this report
do not necessarily represent the views and policies of the Environmental Protection Agency, nor
of other agencies in the Executive Branch of the Federal government, nor does mention of trade
names or commercial products constitute a recommendation for use.
Distribution and Availability This Science Advisory Board report is provided to the EPA
Administrator, senior Agency management, appropriate program staff, interested members of the
public, and is posted on the SAB website (www.epa.gov/sab). Information on its availability is
also provided in the SAB's monthly newsletter (Happenings at the Science Advisory Board).
Additional copies and further information are available from the SAB Staff.
-------
ABSTRACT
On September 23 and 24, 1999 the RSAC met to review the Agency's peer review
program. The Committee responded to five charge questions. The RSAC found that the Peer
Review processes appear to be well established at the EPA. The Science Policy Council Peer
Review Handbook is an excellent guidance document and provides definitive criteria for deciding
what to peer review. Some concerns still remain regarding the process leading to the decision to
peer review or not because of the funding available and/or because of pressure to complete the
work product before peer review due to timing constraints. Consistent with the intent of the
policy, the practice of peer review should not be limited to final products but should also be
extended to the up-front review of significant scientific and technical planning products such as
strategic plans, analytic blueprints, research plans, and goals documents. In general, the
Handbook contains good guidance on issues related to conflict of interest for the peer reviewers.
However, there is concern with respect to conflict of interest of the Peer Review Leader. The
Handbook states that the Peer Review Leader could be the Decision-maker, but Decision-makers
often have a professional interest in the outcome of the review. The RSAC suggests that EPA
include interagency and international products that are used in support of environmental decision-
making in the US under the peer review policy. The evaluation of EPA's peer review process is
expected to be conducted over two to three years. Therefore, RSAC could not fully address the
charge questions "Are the reviews and resulting advice timely" and "Do the peer reviews make a
difference?"during this first review. Data need to be collected and case studies developed to help
the RSAC address these questions.
Keywords: Peer review
-------
US ENVIRONMENTAL PROTECTION AGENCY
SCIENCE ADVISORY BOARD
RESEARCH STRATEGIES ADVISORY COMMITTEE (RSAC)
CHAIR
Dr. W. Randall Seeker, Senior Vice President, General Electric Energy & Environmental
Research Corp., Irvine, CA
MEMBERS
Dr. William Adams, Director of Environmental Science, Kennecott Utah Copper Corp., Magna,
UT
Dr. Stephen L. Brown, Director, R2C2 Risks of Radiation and Chemical Compounds, Oakland,
CA
Dr. Theo Colborn, Director, Wildlife and Contaminants Program, World Wildlife Fund,
Washington, DC
Dr. Philip Hopke, R. A. Plane Professor of Chemistry, Clarkson University, Department of
Chemistry, Pottsdam, NY
Dr. Paulette Middleton, Deputy Director, RAND Environmental Science & Policy Center,
Boulder, CO
Dr. Maria Morandi, Assistant Professor, University of Texas Health Science Center at Houston,
School of Public Health, Houston, TX
Dr. Ishwar P. Murarka, Chief Scientist and President, Ish Inc., Cupertino, CA
Dr. William Smith, Professor of Forest Biology, School of Forestry and Environmental Studies,
Yale University, New Haven, CT
SCIENCE ADVISORY BOARD STAFF
Dr. John "Jack" R. Fowle III, Designated Federal Officer, Environmental Protection Agency,
Science Advisory Board (1400A), 401 M Street, SW, Washington, DC 20460
Ms. Betty Fortune, Senior Environmental Employee, US Environmental Protection Agency,
Science Advisory Board (1400A), Washington, DC 20460
in
-------
TABLE OF CONTENTS
1.0 EXECUTIVE SUMMARY 1
2.0 INTRODUCTION 4
3.0 OVERVIEW OBSERVATIONS AND COMMENTS 6
4.0 RESPONSE TO THE CHARGE 8
4.1 Is EPA peer reviewing the right products? 8
4.1.1 Are there sufficient criteria for deciding what to peer review? 8
4.1.2 Is the information collected about projects sufficient to apply the
criteria and to judge whether and where the peer review should
take place? 8
4.1.3 What information should be gathered on individual work products
for managers of the peer review process [to] use to determine how
best to conduct the peer review? 8
4.1.4 Are the right products being peer reviewed? 9
4.1.5 Do the products go to the right place for peer review? 10
4.2 Are the peer reviews conducted appropriately? 10
4.2.1 Does the process take the right steps to minimize conflicts of
interest? 10
4.2.2 Does the process give proper attention to controversial and polar
positions? 11
4.2.3 Are the reviews set up appropriately to address the needs of the
Agency? 12
4.3 Does EPA peer review all the science it uses (e.g., data submitted from
parties outside the Agency)? How well does EPA capture which scientific
components were peer reviewed and which were not? 12
4.4 Does the RSAC have additional comments/guidance for EPA? 13
5.0 NEXT STEPS 16
5.1 Are the reviews and resulting advice timely? 16
5.2 Do the peer reviews make a difference? 16
5.3 To what extent are the review comments responded to and acted on by the
Program Office/Region? 17
REFERENCES R-l
IV
-------
1.0 EXECUTIVE SUMMARY
On September 23 and 24, 1999, the Research Strategies Advisory Committee (RSAC) of
the Science Advisory Board (SAB) reviewed the Agency's peer review program. RSAC was
pleased to see the attention given to the RSAC review by senior EPA personnel.
As part of the review process, the RSAC responded to five charge questions:
a) Is EPA peer reviewing the right products?
b) Are the peer reviews conducted appropriately?
c) Do the peer reviews make a difference?
d) Does EPA peer review all the science it uses (e.g., data submitted from parties
outside the Agency)?
e) Does the RSAC have additional comments/guidance for EPA?
The credibility of science used by EPA to support its decisions has been called into
question over the last several years as more controversial issues and more subtle health and
ecological effects have been investigated. The Committee thinks that independent peer review of
scientific products makes it much more difficult to question the quality of the Agency's science
base. Independent peer review, as defined in section 1.4.8 of the Peer Review handbook, is one
of the central elements that ensure the quality and improves the credibility of the Agency's
scientific and technical products. Thus, a comprehensive peer review mechanism is a necessary
component for the Agency's decision-making process and must be rigorously imposed by Agency
managers and scientists.
Peer Review processes appear to be well established at the EPA. In addition, these
processes are continuing to improve through a mechanism of continued internal examination led
by the Office of Research and Development and process changes carried out by Decision-makers
at the direction of the Science Policy Council. A key to success in implementing the peer review
process has been the involvement of the Office of Research and Development in the oversight
role. ORD scientists have an understanding of the importance of peer review in developing good
scientific and technical products. ORD has provided effective support to the establishment of the
Peer Review process within EPA through development and implementation of training, collection
of data on products and their review status, and also by bench marking EPA's peer review efforts
against reviews at other organizations.
The Science Policy Council Peer Review Handbook is an excellent guidance document
and provides definitive criteria for deciding what to peer review. While the guidance is clear,
some concern still remains regarding present mechanisms leading to the decision to peer review or
not based on funding available and/or pressure to complete the work product before peer review
-------
due to timing constraints. There will always be some difficult decisions, but these are areas where
the Agency could focus its attention on in the future. Every effort should be made to ensure that
all major scientific and technical products are appropriately peer reviewed.
The focus of the "Peer Review Handbook" developed by the Agency's Science Policy
Council is the peer review of major scientific and technical products. The review of major
products is obviously important to ensure that they are of acceptable quality. The Handbook
clearly defines and articulates the nature of the products that should be peer reviewed, and it is an
excellent guide to help managers select both the products for peer review and the proper
mechanisms for performing the review.
However, the focus of the peer review should not be limited to the end products of the
scientific endeavor. Peer review should also be extended to the up-front review of scientific and
technical planning products such as major strategic plans, analytic blueprints, research plans, and
goals documents. The Office of Research and Development follows this practice by peer
reviewing its strategic plans and research strategies. While the Handbook does discuss peer input
it does not explicitly address the importance of peer review in the planning of products and
documents.
In general, Section 3.4 of the Handbook contains good guidance on issues related to
conflict of interest for the peer reviewers. However, there is concern with respect to conflict of
interest of the Peer Review Leader. The Handbook states that the Peer Review Leader could be
the Decision-maker (Section 1.4.4). The Decision-maker will often have a professional interest in
the outcome of the review. An essential part of peer review as defined by the EPA is
independence of the review from the person (or persons) who has developed the product or
overseen the development of the product. In addition, the role of sign-off upon completion of the
review by the peer review leader is not defined. Comparing peer review to quality assurance
(QA), one of the important aspects of the QA officer's responsibilities is to stop activity if there is
a serious problem with implementation of the QA/QC program. Currently only the Decision-
maker signs off when the peer review is complete. It should be the Peer Review Leader's
responsibility to determine if changes to the product have been satisfactorily made and to stop the
product from moving forward until that determination can be made.
The RSAC has made several other suggestions to strengthen the peer review process. A
key suggestion is expanding the scope to the evaluation of interagency and international products
considered important to environmental decision-making. Peer review should also be extended to
the up-front review of scientific and technical planning products such as strategic plans, analytic
blueprints, research plans, and environmental goals documents. Products that are policy analytic,
in that they are not purely science-based but involve the application of policy and values, should
also be peer reviewed to ensure that appropriate methods and procedures have been used,
including an explicit treatment of assumptions and value judgments, adequate sensitivity analysis,
and adequate treatment of uncertainty. For example, flawed decision-making on the
implementation of environmental control technologies (which may include considerations and
approaches that are nested in the social science framework) can result in a poor program even if
-------
the technological options are intrinsically effective. Thus, in addition to major science and
technology products, social science research products, and policy analytic products, can and
should be subjected to the peer review process. The explicit need to review major social science
products (in addition to economic products) should be added to the Peer Review Handbook.
Finally, there should be a requirement for completion of training before a person can be
designated as a Peer Review Leader.
The RSAC review of EPA's peer review process is expected to be conducted by the SAB
over two to three years. This report is the product of the first meeting in a multi-year process. It
was not possible to fully address the charge questions concerning the timeliness of peer reviews
and resulting advice (i.e., "Are the reviews and resulting advice timely?") and the effectiveness of
peer review in improving Agency decision-making (i.e., "Do the peer reviews make a
difference?"). Data need to be collected and case studies developed to help the RSAC evaluate
these issues.
-------
2.0 INTRODUCTION
On June 7, 1994, the EPA Administrator issued a Peer Review Policy requiring that all
major scientifically and technically based work products related to Agency decisions should be
peer reviewed. This Peer Review Policy was re-emphasized in the subsequent memos issued by
the senior management in the Agency and by the release of the Peer Review Handbook in 1998.
The Deputy Administrator also initiated an audit by the National Center for Environmental
Research and Quality Assurance within ORD to evaluate the extent to which the Agency offices
and regions were complying with the peer review procedures. The General Accounting Office has
completed at least two independent evaluations of the Peer Review Program at EPA. Recently,
the National Research Council issued its Blue Ribbon panel report on Evaluating Federal
Research Programs (NAS, 1999).
The Research Strategies Advisory Committee (RSAC) of the Science Advisory Board was
requested to conduct a review of the overall process and the Agency's initial efforts to develop
and implement the peer review policy and guidance. RSAC decided to conduct this review in two
phases. The first phase, which is the subject of this report, focuses on EPA's Peer Review
Processes and Policies. The second phase will focus on the implementation of the processes and
policies, and the impact of the Peer Review Policy on the Agency's decision-making.
EPA's charge to the RSAC was:
a) Is EPA peer reviewing the right products?
(i) Are there sufficient criteria for deciding what to peer review?
(ii) Is the information collected about projects sufficient to apply the criteria
and to judge whether and where the peer review should take place?
(iii) What information should be gathered on individual work products for
managers of the peer review process to determine how best to conduct the
peer review?
(iv) Are the right products being peer reviewed?
(v) Do the products go to the right place for peer review?
b) Are the peer reviews conducted appropriately?
(i) Does the process take the right steps to minimize conflicts of interest?
(ii) Does the process give proper attention to controversial and polar
positions?
(iii) Are the reviews set up appropriately to address the needs of the Agency?
(iv) Are the reviews and the resulting advice timely?
c) Do the peer reviews make a difference?
(i) To what extent are the review comments responded to and acted on by the
Program Office/Region?
-------
d) Does EPA peer review all the science it uses (e.g., data submitted from parties
outside the Agency)? How well does EPA capture which scientific components
were peer reviewed and which were not?
e) Does the RSAC have additional comments/guidance for EPA?
On September 23 and 24, 1999, the RSAC met to conduct the review and prepare its
report to the Administrator. This RSAC review was limited to an overall evaluation of the peer-
review process gleaned from the Peer Review Handbook (1998), several GAO reports, and a
number of letters and memos from senior management in the Agency. This RSAC review was
further aided by presentations from and interactions with Agency staff during the September 23-
24 1999 public meeting. RSAC has planned to conduct a subsequent in-depth analysis to fully
examine trends in the use of peer review in EPA, the impacts of the peer reviews, and additional
opportunities for enhancing the benefits from peer review in the form of quality, credibility,
relevance and Agency leadership position. The following subset of the charge questions, from the
list shown above, will be addressed in the subsequent RSAC review.
a) Are the reviews and resulting advice timely?
b) Do the peer reviews make a difference?
c) To what extent are the review comments responded to and acted on by the
Program Offices and Regions?
The RSAC based its review on briefings by Agency staff during the meeting and the
reference documents provided by the Agency. These documents are listed in the reference section
at the end of this report.
-------
3.0 OVERVIEW OBSERVATIONS AND COMMENTS
The Office of Research and Development has had a peer review policy in place since 1982.
However, the process was not standardized and documented except for those reviews carried out
through the SAB. Thus, in general, there was no way to determine the effectiveness of the ORD-
organized peer reviews in improving the science work products that were reviewed. Thus, the
implementation of a formal process with central collection of information regarding the review is
an important step in providing greater confidence in the scientific quality of the Agency's major
science work products.
On September 23 and 24, 1999 the Research Strategies Advisory Committee (RSAC) of
the Science Advisory Board (SAB) met to review the Peer Review Program of the US
Environmental Protection Agency (EPA). The review meeting was conducted in public session
under the provisions of the Federal Advisory Committee Act (FACA). The Committee was
provided with background documents supplied by the Agency, supplemented by briefings from
Agency senior managers during the meeting.
The Committee was pleased to see the Agency's diligence with respect to Peer Review.
From the materials presented to RSAC, EPA's Peer Review Process is well articulated and
appears to be fundamentally sound and, with a few exceptions, working as intended. The
materials provided before the meeting were very comprehensive and useful in preparing for the
review. Overall, it seems that EPA has been responsive to the SAB, GAO and other
organizations' recommendations regarding peer review. The process currently in place covers
many facets of peer review requirements.
The RSAC found that peer review processes appear to be well established at the EPA. In
addition these processes are continuing to improve through a mechanism of continued internal
examination led by the Office of Research and Development and process changes carried out by
Decision-makers at the direction of the Science Policy Council. The driver for the Peer Review
Program has been established by EPA management leadership. This high level management
commitment to Peer Review of major EPA Technical and Scientific products was manifested in
such actions as the 1994 Peer Review Policy statement, the establishment of the Science Policy
Council, the release of the Peer Review Handbook, and, most recently, in the policy requiring that
all rules, planning, and analytic blueprints must specifically address and include consideration of
peer review (Robertson, 1999). The continued success of the Peer Review process at EPA will
only be ensured with continual committment on the part of Senior managers to the Peer Review
process in all appropriate aspects of the Agency's endeavors.
Another key to success in implementing the peer review process has been the involvement
of the Office of Research and Development in the oversight role. ORD scientists have a good
understanding of the importance of peer review in developing good scientific and technical
products. ORD has provided effective support to the emergence of the Peer Review process
within EPA through development and implementation of training, collection of data on products
-------
and their review status, and evaluations of the appropriate level of reviews at other organizations.
The development of the Peer Review Data Base can become a powerful tool to allow the
continual assessment of whether all major EPA generated Science and Technology (S&T) and
economics products are being reviewed by the proper mechanisms. In addition, it will be critically
important for the Agency to continue audits of different offices to ensure that the peer reviews are
being properly done on the variety of products generated by the Agency.
-------
4.0 RESPONSE TO THE CHARGE
4.1 Is EPA peer reviewing the right products?
4.1.1 Are there sufficient criteria for deciding what to peer review?
Yes. The Science Policy Council Peer Review Handbook is an excellent guidance
document and provides definitive criteria for deciding what to peer review. The rule of thumb is
that all major scientific and technical work products which affect major Agency decisions must be
peer reviewed, and when there is doubt the work product should be listed for peer review.
Interpretative guidance is provided on what is and what is not a major product. Adequate
descriptive criteria for each category are provided.
While the guidance is clear, some concern still remains regarding the process leading to
the decision to peer review or not based on funding available and/or pressure to complete the
work product due to timing constraints. There will always be some difficult decisions, but these
are areas where the Agency must focus its attention in the future so that no major products escape
an appropriate peer review. Every effort should be made to ensure that all major scientific and
technical products are appropriately peer reviewed.
4.1.2 Is the information collected about projects sufficient to apply the criteria and
to judge whether and where the peer review should take place?
To answer this question in the next phase of our review, RS AC would like to see
representative examples of the forms or other procedures used to describe work products being
considered for peer review, from more than one organization if they differ from organization to
organization. RS AC would also like to review a sample of completed forms along with the
justification for placing the candidate product on list B (products to be peer reviewed) or list C
(products deemed minor or peer review is infeasible). RSAC notes that the criteria are sufficiently
clear and simple so that every EPA organization should be able to apply them without an
elaborate information-gathering process.
4.1.3 What information should be gathered on individual work products for
managers of the peer review process [to] use to determine how best to
conduct the peer review?
The most important tasks for the peer review managers are to write an appropriate charge
and to select peer reviewers with proper expertise and independence (as described in section 1.4.8
of the Peer Review Handbook), either individually or through another agent such as the SAB or a
peer review contractor. To do so, the managers must understand the scientific content of the
product and what areas of the science are most likely to be weak or controversial. Such
determinations will be easier the closer the product is to completion as a draft work product, but
in most cases the peer review should be planned early in the process of work product
-------
development. Because the peer review managers are located within the organizations developing
the work products, they should already possess a relatively good understanding of the scientific,
technical or economic issues addressed by the work product and the decisions they are intended
to inform. To be sufficiently independent to manage the peer review effectively, however, they
may not be (at least initially) fully familiar with the content of the work and may need to interview
the product team to enhance their understanding. It may also be prudent to seek advice from
outside the performing organization on who should be leading the effort and on who might be
peer review panel candidates for the more important work products.
Specific types of information that might be valuable for organizing the peer review
include:
a) information on purpose of the project;
b) information on the potential for regulatory use or rule making;
c) information on the potential for the project to be used as a foundation for
additional research or to be directly utilized;
d) information on key scientific and technical findings.
4.1.4 Are the right products being peer reviewed?
The focus of the Peer Review Handbook developed by the Agency's Science Policy
Council is to provide detailed guidance on conducting peer review of major scientific and
technical products. The review of products is obviously important to ensure that the products are
of acceptable quality. The handbook clearly defines and articulates the types of products that
should be peer reviewed, and is an excellent guide to help managers select completed products for
peer review and the appropriate peer review mechanisms.
However, the focus should not be limited to just end products of the scientific endeavor.
Peer review should also be extended to the up-front review of scientific and technical planning
products such as strategic plans, analytic blueprints, research plans, and goals documents. This is
a practice followed by the Office of Research and Development as seen in the peer review of
strategic plans and research strategies. While the Handbook does discuss peer input, it does not
explicitly address the importance of peer review in the planning products and documents. This is
an important omission in the handbook. An up-front peer review of these planning products will
allow peer input prior to the conduct of the research and scientific undertaking and, therefore, can
have a significant impact on the direction of the effort and the quality of the final product. It is in
the planning process where changes can more easily be made as opposed to the peer review of the
final products where there is more resistance to change, as well as more pressure to use the
available information for timely decision making. RSAC recommends that future supplements and
updates of the Handbook include the importance of conducting peer review of planning products
and documents.
-------
The following documents were provided by the Agency upon request from the
Committee:
a) Appendix C- June 1996 List. Products Peer Reviewed since 1991;
b) Appendix D- June 1996 List. Candidate Products for Future Peer Review;
c) Appendix C- June 1997 List. Products Peer Reviewed from 6/96 Through 5/97;
d) Appendix D-l- June 1997 List. Candidate for Future Peer Review;
e) Products Considered and Not Selected for Peer Review. Appendix D2 - Historical
Data to be converted to List C.
They contain information on the product; region; program office; peer review leader; date
of product completion; review mechanism; dates of initial entry, initiation and completion of the
peer review, and receipt of comments; and comments relevant to the outcome of the process,
including the rationale for proceeding or not with the peer review. The latter should contain the
information needed to address this question of the charge by the RSAC. However, inclusion of
the rationale for the eventual decision about peer review is not consistent.
Based on the titles of products, and the rationale (when included), it appears that the right
products are being peer reviewed. However the determination of those products deemed minor
or where peer review is infeasible (list C) is harder to judge. The information provided in these
documents needs to be more complete in order for the Committee to fully answer this question. It
is also important to the Agency that the process of decision about peer review be transparent both
internally and externally. It is recommended that the Agency develop a list of categories based on
those presented in Chapter 2 of the Peer Review Handbook (expanded, if needed), so that the
rationale is clearly and systematically presented across all products. A code could be attached to
each category. This would allow for direct referral to the Handbook, and can also facilitate and
expedite the peer review decision process.
While all products listed as peer reviewed were appropriately designated, it is not clear if
some of the products classified as "not for peer review" should have been peer reviewed. The
RSAC cannot evaluate this issue at this time, but will revisit it during the next round of this
review. Specific but representative case studies should be selected and evaluated in more detail as
discussed in section 5 relative to the next phase of this review.
4.1.5 Do the products go to the right place for peer review?
Based on the documentation provided (see charge question 4.1.4) it appears that the
products are directed to the right regions/offices/programs and to the appropriate type of peer
review. Further clarification is needed about which specific entity at EPA takes the lead for peer
review about products that cross multiple offices and programs.
10
-------
4.2 Are the peer reviews conducted appropriately?
4.2.1 Does the process take the right steps to minimize conflicts of interest?
In general, Section 3.4 of the Handbook contains good guidance on issues related to
conflict of interest for the peer reviewers. As indicated in that section of the Handbook,
eliminating all conflicts of interest may be practically impossible and, further, may not be in the
best interest of the Agency for obtaining a peer review of the best possible quality. RSAC notes
that direct economic interest in the actions that might be taken on the basis of the product is not
the only type of interest that could cause conflicts; most qualified candidates for peer review
responsibilities will have taken some prior position that could be perceived to be a conflict. The
RSAC therefore agrees with the Handbook that the important goal is achieving balance in the
spectrum of biases represented in the review group. As with any other such endeavor, balance
will remain to some extent in the eye of the beholder, and the peer review managers must strive to
eliminate their own individual or organizational opinions from unduly influencing the judgment.
Although the RSAC is not planning to investigate the overall balance in the selection of reviewers,
it has no reason to suspect that EPA is not pursuing a prudent course.
There is concern with respect to conflict of interest with the Peer Review Leader. The
Handbook states that the Peer Review Leader could be the Decision-maker (Section 1.4.4). The
Decision-maker will often have a professional interest in the outcome of the review. An essential
part of peer review as defined by the EPA is independence of the review from the person who has
developed the product or overseen the development of the product. For example, journal
publications are given great weight because the editorial review process is perceived to be
independent. The person who is to use the product or someone who is subordinate to the
Decision-maker is not clearly independent and has an intellectual stake in the product. Thus,
although they may do an appropriate job of managing the peer review, there is the perception of
possible conflict of interest that could potentially undermine the credibility of the process.
The role of sign-off upon completion of the review by the Peer Review Leader is not
defined. Comparing peer review to quality assurance, one of the important aspects of the QA
officer's responsibilities is to stop activity if there is a serious problem with implementation of the
QA/QC program. Currently only the Decision-maker signs off at the end to indicate that the peer
review is complete. It should be the Peer Review Leader's responsibility to determine if changes
to the product have been satisfactorily addressed and to stop the product from moving forward if
necessary. An independent review process has not been completed if there is not this possible
outcome. It may be that the Decision-maker has to utilize the product without the Peer Review
Leader sign-off, but then it should be acknowledged that the product has not been peer reviewed.
4.2.2 Does the process give proper attention to controversial and polar positions?
The Handbook adopts the general philosophy that people who have publically expressed
specific views on an issue should be avoided. Depending on the issue, it may be better to utilize
the approach generally used by the National Research Council and SAB to obtain balanced views.
11
-------
The objective of a peer review is not necessarily to achieve consensus, but to determine if the
scientific content in the product is credible. Obtaining disparate views may provide better insights
into the criticisms that the product will face once it becomes public. The Handbook may be too
conservative in its recommendations of avoiding controversy in the reviews, but such diversity
may be useful in assessing if the product is completely addressing the relevant issues. Agreement
on the scientific content provides more weight to the subsequent decisions based on the product,
but if there are differences in understanding or interpretation of the scientific data, it is best to
acknowledge them and provide a clear rationale for making particular choices. The product does
not have to agree with all of the peer review comments. Peer reviewers can be wrong. The task
of the Peer Review Leader is to make a professional judgement on what changes do or do not
need to be made to the product. A diversity of views can be helpful in making such judgments.
4.2.3 Are the reviews set up appropriately to address the needs of the Agency?
The most recent changes in the Agency's peer review process were instituted because of
persistent criticisms (e.g., in the NAS and GAO reports) that EPA's science was not sufficiently
peer reviewed and therefore of questionable quality. The new process appears to be well suited to
improve both EPA's scientific activities and the external world's perception of those activities. It
should help in making the scientific products and the decisions based on them more
credible and truly effective. Although the Agency cannot expect criticisms of its science to
completely disappear, it probably will be better prepared to address such criticisms.
In a different sense, the flexibility of the guidelines in the Handbook is appropriate to
address the need of the Agency to trade off multiple considerations (budgets, statutory or judicial
deadlines, quality and credibility of the science) in deciding how much peer review to undertake
and by whom.
The extent to which an individual product meets the Agency's needs can be made more
explicit in the review process by including a charge question which addresses the product's ability
to meet a well-defined Agency need.
4.3 Does EPA peer review all the science it uses (e.g., data submitted from parties
outside the Agency)? How well does EPA capture which scientific components were
peer reviewed and which were not?
RSAC has not yet been provided with documentation that would allow an "audit" of the
products on lists A (peer review complete), B (candidate products to be peer reviewed), and C
(products to not be peer reviewed) against all the EPA activities that might be appropriate for
peer review. Section 2.2 of the Peer Review Handbook is rather inclusive in its definitions of
what products should be considered for peer review; if it is being followed, few scientific products
should escape peer review. From anecdotal evidence, some of the program offices may not look
very closely at the peer review status of science done for them under contracts and grants or
submitted by interested parties (either regulated entities or the environmental community). It is
12
-------
also not clear to what extent peer review is being applied to non-economic social science (e.g.,
risk communication studies) products.
The EPA should use peer-reviewed science as the basis for all actions and decisions.
However, all non-EPA science cannot possibly be peer reviewed by the Agency. EPA must
systematically determine if the science used by the Agency has been peer-reviewed. Science
published in peer-reviewed journals is reviewed by the journal's reviewers. Science provided by
other agencies/laboratories may have been subjected to peer review by that agency/laboratory.
Section 2.4.5 (page 35) of the Peer Review Handbook requires EPA to conduct its own peer
review even if the material being used has been previously peer reviewed. This is a very
conservative requirement most appropriate in cases where EPA's application is novel or marginal
to the purpose of the research product. In the instance of confident recognition of high-quality,
non-Agency peer review, direct relevance of peer-reviewed science to EPA's needs, and/or non-
controversial or relatively low significance decisions, the requirement to conduct additional EPA
peer review should be placed on the Peer Review Manager. EPA should strive to determine if the
science they are using has been peer reviewed. If it has not, the importance and significance of
the science and/or the issue may be used to determine whether or not EPA should conduct its
own peer review.
In addition to major science and technology products, social science research products can
and should be subjected to the peer review process in a manner similar to natural science
products. In addition, some products are policy analytic, in that they are not purely natural and
social science-based but involve the application of policy and values. Peer review ensures that
appropriate methods and procedures have been used, including an explicit treatment of
assumptions and value judgments, adequate sensitivity analysis, and adequate treatment of
uncertainty. For example, flawed decision-making on the implementation of environmental
control technologies (which may include considerations and approaches that are nested in the
social science framework) can result in a poor program even if the technological options are
intrinsically effective. Thus, the explicit need to review major social science products (in addition
to economic products), and policy analytic products should be added to the Peer Review
Handbook.
4.4 Does the RSAC have additional comments/guidance for EPA?
The RSAC has several suggestions for strengthening the peer review process in the future.
These involve expanding the scope of peer review to further enhance the Agency's scientific
credibility and effectiveness at developing policies based on sound science.
The peer review process currently focuses on review of individual EPA products. RSAC
encourages EPA to include evaluation of interagency and international products that are
considered in support of environmental policy making in the US. Such work can be very valuable
to EPA policy making, provided that it has been reviewed in the context of it's use in support of
EPA's decision-making.
13
-------
Although the guidance provided (specifically, the Science Policy Council Handbook on
Peer Review) emphasizes flexibility, it is highly structured and has some potential for becoming a
bureaucratic hindrance to scientific activities at the Agency. Care must be taken to preserve
flexibility and to avoid reviews that consume resources but do not necessarily contribute to the
overall credibility of EPA policy making.
One way to help insure that peer review is as effective as possible is through periodic
evaluation of how well activities under review are helping EPA achieve its goals and how well
EPA is responding to peer review recommendations. This level of continued regular review of
the overall peer review process and its impact could provide the opportunity to modify the review
process to better meet Agency goals.
Credibility also will be enhanced by peer evaluation of the Agency as a leader in
international as well as national scientific research and its application to policy making. This
evaluation can assess current status of the Agency in this regard and the extent to which EPA is
building leadership through programs that encourage high quality science and increased
cooperation between EPA and scientists in other Agencies and around the world on important
global issues.
Peer Review works best when both the reviewers and the reviewees are convinced that the
goal and outcome of peer review activities will be better products and decisions. Unless the
reviewees feel that their work will be well received and more effective, they will have little
incentive to make the most of a peer review. Unless the peer reviewers feel that their advice will
be considered and used as much as possible, they will be reluctant to spend the time and effort
needed for a constructive review.
The peer review process described in the Peer Review Handbook is non-adversarial.
RSAC agrees with this approach; the peer review process should not be viewed as punitive by the
peer reviewers. If they perceive that peer review is a largely bureaucratic process that will
consume their energies and delay the completion of their work without much real benefit, it will
not likely be very effective. For the most part, the Handbook is consistent with a positive
approach by both parties. However, RSAC sees some seeds of bureaucracy that could germinate
and grow. Having the process managed by the Office of Research and Development could be
perceived as an application of power rather than a collegial association. Emphasis on the
documentation of the peer review process could grow into a form-driven bureaucratic and
inefficient process. Section 3.6 of the Handbook seems to focus excessively on rules for the
acquisition of peer review services, not on ways to ease the acquisition of those services. RSAC
recommends that EPA take the necessary steps to keep the process from becoming
inappropriately bureaucratic.
The RSAC also recommends that charges for peer reviews not become overly elaborate.
The WTI charge seen in Appendix B, pages B-2 to B-10, appears to fall in that category. We
have also seen charges that ask only very selected questions, sometimes dodging the crucial
scientific issues. Every charge should have a final open-ended question similar to this one (#5).
14
-------
The Handbook provides guidance requiring the reviewer to agree not to disclose draft
work products to the public. While RSAC understands the evils that can come from release of a
draft that might well change substantially, we note that any review by a FACA committee such as
the SAB means that the draft being reviewed is accessible to the public. Some resolution of this
conflict, or at least a discussion thereof, is needed.
There should be a requirement of completion of training before a person can be designated
as a Peer Review Leader. Under current practice, EPA personnel cannot be project monitors
without training. Peer review of the major science products is an important function within the
Agency and should be recognized as such by requiring appropriate training before taking on such
tasks.
RSAC concluded that all Agency documents should contain an information statement
indicating whether or not the science in the document has been reviewed along with some
description of the peer review processes used. The Peer Review Handbook does require
documentation of the peer review processes and responses but there is not an explicit requirement
for the information to be placed in the actual document that presents the results of the study. It is
equally important to allow the reader of the document know if the information in report was not
peer reviewed as well as the rationale for the decision not to peer review. The Agency should
explore the implications for having a policy requiring all documents to include an information
statement on the status of the peer review process for the product.
Past reviews of the EPA peer review process by outside groups (GAO, NAS) have
indicated a few reoccurring problems: lack of understanding of what peer review is; uneven
application of the guidance policy across the Agency in conducting peer reviews; inconsistent
oversight mechanisms including the role of QA review; criteria used to opt out of peer review;
tracking of all peer review projects; peer review of directed research; inconsistency in who selects
and approves reviewers. The Science Policy Council Peer Review Handbook and associated EPA
policies are aimed at correcting all of these deficiencies. Implementation is still a work-in-
progress.
The Peer Review Data Base can become a very useful tool for assisting in the evaluation
of the status and trends of the Agency's programs. As EPA develops additional data bases on
those same programs, it should consider adding linkages to the Peer Review Database that would
permit cross-referencing among multiple databases.
Looking ahead some additional emphasis could be given to the following issues;
a) Training programs could further emphasize the need to consider peer review a part
of the over-all process of conducting and completing research and science and
technology products;
b) Further consideration should be given to consistency in training across the
Agency;
15
-------
c) Feedback on projects which were not peer reviewed could be enhanced.
16
-------
5.0 NEXT STEPS
The review of EPA's peer review process is expected to be conducted over two to three
years. This report is the product of the first meeting that launched the process. Thus, it was not
possible to fully address all of the charge questions at this time. This section contains the charge
questions for which further examination and review will be required. It is anticipated that a series
of case studies will be developed laying out how selected Agency programs and products were
developed; what the planning was for peer review; how the peer review was conducted; and how
the Agency used the peer review advice.
5.1 Are the reviews and resulting advice timely?
RSAC was not provided with sufficient information to evaluate this question in depth
during this review. Clearly, peer review will always add to the time needed to complete any work
product, and the Agency has expressed concerns about the timeliness of certain SAB or NAS
reviews in the past. Especially for external reviews, the timeliness of the review will always be, to
some extent, in the hands of the reviewers themselves. RSAC does applaud the Handbook's call
for an explicit review schedule that reflects a compromise between the Agency's need for closure
and the reviewers' need for adequate time to deliberate.
In order to evaluate this charge question, data should be collected on the time required for
the various peer review steps. In addition, the EPA groups affected by the product and its peer
review need to provide an assessment of timeliness. Such a formal evaluation and reporting of
timeliness provides data to help evaluate the overall value of the review process in the future.
5.2 Do the peer reviews make a difference?
The Peer Review Policy established on June 7, 1994 requires scientifically- and
technically-based work products related to EPA decisions to be peer reviewed. For those work
products intended to support the Agency's most important decisions, peer review is the procedure
of choice. Therefore, to evaluate the impact of the peer review process, RSAC requests
information on:
a) The scientific and technical character of the work products;
b) The scope and depth of the peer review to be conducted;
c) The peer review results/recommendations generated;
d) The use of peer review results in Agency decisions (e.g., how did peer review
result in better decisions?).
17
-------
RSAC needs to carefully examine several completed peer reviewed products, interview
Agency Decision-makers and establish what types of products and peer reviews improved Agency
decisions, which ones did not and why.
The Committee also needs information on completeness and comprehensiveness of the
peer review. For example, if there were to be a risk assessment and EPA decision on the need for
regulations of a solid waste stream, the questions to ask would be:
a) Was peer review conducted for the quality, adequacy and completeness of the data
developed and obtained from the literature for the risk assessments and Agency
decisions?
b) When was the peer review conducted?
c) Did the peer reviewers have adequate time to conduct an in-depth review?
d) How were the peer review results utilized to improve EPA 's decision?
In principle, peer reviews should be making a difference in the sense that more sound
science is being and will be used in policy making as a result of the peer review process.
However, in order to be able to answer this question, the EPA needs a mechanism for monitoring
the effectiveness of its peer review program.
5.3 To what extent are the review comments responded to and acted on by the Program
Office/Region?
The SAB always requests feedback on how its findings and recommendations were
incorporated into the work product, and many of the Program Offices are quite conscientious
about providing such feedback. Sometimes, however, SAB is still left without much sense of how
its review made a difference. This limitation is likely to extend to other types of peer review.
Whether the reviews are being ignored or being heeded without sufficient documentation is not
certain. Case studies will be used in the next phase of this review so that the impacts of the
Agency peer review processes can be evaluated. The focus will be on assessing the response and
actions of the program offices/regions to the peer review process. Information on the case studies
will be needed on what peer review comments were made and how the Agency responded to
these comments.
18
-------
REFERENCES
EPA. 1996. Agency Peer Review memo from Assistant Administrator for Research and
Development Robert Huggett, November 1996
EPA. 1996. Evaluation of Peer Review Implementation memo from Assistant Administrator for
Research and Development Robert Huggett, November 1996.
EPA. 1997. Enhancement of Agency Peer Review and Evaluation of Its Implementation memo
from Assistant Administrator for Research and Development Robert Huggett, January
1997
EPA. 1998. Report on the Evaluation of Peer Review Implementation memo from Acting
Assistant Administrator for Research and Development Henry Longest, January 1998
EPA. 1998. Science Policy Council Peer Review Handbook. EPA 100-B-98-001. January 1998.
EPA. 1998. Implementation of the Peer Review Policy Report to Congress from Deborah
Dietrich, Director Office of Resources Management and Administration, Office of
Research and Development April 1998.
EPA. 1996. Appendix C- June 1996 List. Products Peer Reviewed since 1991. June 1996.
EPA. 1996. Appendix D- June 1996 List. Candidate Products for Future Peer Review. June 1996
EPA. 1997. Appendix C- June 1997 List. Products Peer Reviewed from 6/96 Through 5/97. June
1997.
EPA. 1997. Appendix D-l- June 1997 List. Candidate for Future Peer Review. June 1997.
EPA. 1997. Products Considered and Not Selected for Peer Review. Appendix D2 - Historical
Data to be converted to List C, June 1997.
EPA. 1998. Peer Review Handbook memo from Administrator Carol Browner and Deputy
Administrator Fred Hansen, February 1998.
EPA. 1999. Annual Update of Peer Review Information and Call for Science Advisory Board
Review Nominations memo from Deputy Administrator Peter Robertson, June 1999.
EPA. 1999. Sound Science and Peer Review in Rulemaking memo from Deputy Administrator
Peter Robertson, June 1999.
R-l
-------
EPA. 1999. Robertson, Peter, 1999. Annual Update of Peer Review Information and Call for
Science Advisory Board Review Nominations. June 15, 1999. Washington, DC
EPA. 1999. Annual Peer Review Reporting Requirements and Instructions for Using the Peer
Review Product Tracking Database memo from Assistant Administrator for Research and
Development Norine Noonan, July 1999.
GAO. 1994. General Accounting Office Report: Peer Review - EPA Needs Implementation
Procedures and Additional Controls, February 1994.
GAO. 1995. General Accounting Office Report: Environmental Protection - EPA's Problems with
Collection and Management of Scientific Data and Its Efforts to Address Them, May 1995
GAO. 1996. General Accounting Office Report: Peer Review - EPA's Implementation Remains
Uneven, September 1996
GAO. 1997. General Accounting Office Report: Peer Review - EPA's Implementation Remains
Uneven Supplement, March 1997
GAO. 1999. General Accounting Office Report: Federal Research - Peer Review Practices at
Federal Agencies Vary, March 1999
NAS. 1999. Evaluating Federal Research Programs: Research and the Government Performance
and Results Act. National Academy Press, Washington, DC. 1999.
R-2
-------
------- |