United States
Environmental Protection
Agency
Office of Science Policy
Office of Research and Development
Washington, DC 20460
EPA100-B-98-001
January 1998
www.epa.gov
Science Policy Council
HANDBOOK
EE
-------
THIS PAGE LEFT INTENTIONALLY BLANK
-------
EPA 100-B-98-001
January 1998
U.S. Environmental Protection Agency
PEER REVIEW HANDBOOK
Prepared for the U.S. Environmental Protection Agency
by members of the Peer Review Advisory Group,
a group of EPA's Science Policy Council
Principal Authors
Kerry L. Dearfield, Ph.D. A. Robert Flaak
Office of Science Policy Science Advisory Board
Office of Research and Development Office of the Administrator
Major Contributors
Jean C. Schumann Nancy W. Wentworth
Office of Solid Waste Office of Research and
and Emergency Response Development
Roland B. Hemmett, Ph.D. Arnold M. Kuzmack, Ph.D.
Region 2 Office of Water
Science Policy Council
U.S. Environmental Protection Agency
Washington, DC 20460
-------
Page ii Peer Review Handbook
DISCLAIMER
This document has been reviewed in accordance with U.S. Environmental Protection
Agency policy and approved for publication and distribution to the Agency. Mention of trade
names or commercial products does not constitute endorsement or recommendation for use.
-------
Peer Review Handbook Page iii
TABLE OF CONTENTS
FOREWORD Page ix
SUMMARY OF THE PEER REVIEW PROCESS Page 1
Figure 1 - Flowchart for Planning a Peer Review Page 2
Figure 2 - Flowchart for Conducting a Peer Review Page 3
Figure 3 - Flowchart for Completing a Peer Review Page 4
Managers Planning Checklist for Peer Review Page 5
PEER REVIEW GUIDANCE Page 7
1. THE NEED FOR PEER REVIEW Page 9
1.1 Overview Statement Page 9
1.2 Understanding Peer Review Page 9
1.2.1 Why use Peer Review? Page 9
1.2.2 What is Peer Involvement? Page 10
1.2.3 What is Peer Review? Page 10
1.2.4 What is Peer Input? Page 11
1.2.5 How is Peer Review Different from Peer Input? Page 11
1.2.6 Can Someone Who Provided Peer Input Become an Independent
Peer Reviewer for the Same Work Product Later in the Process? Page 12
1.2.7 How is Peer Review Different from Public Comment? Page 12
1.2.8 How is Peer Review Different from Stakeholder Involvement? .. Page 12
1.2.9 What Role does Peer Review have in the Regulatory
Development Process? Page 13
1.2.10 What Role does Peer Review have in Regulatory Negotiations? . Page 14
1.3 Annual Agency Reporting Requirements Page 14
1.3.1 What are the Annual Reporting Requirements? Page 14
1.3.2 What Listings are Required for the Annual Reporting? Page 14
1.3.3 Is There a Coding System for Work Products on the Annual
Reporting Lists? Page 16
1.3.4 When will the Handbook Itself be Revised? Page 17
1.4 The Roles of People and Organizations in Peer Review Page 17
1.4.1 Who is Ultimately Accountable for Peer Review? Page 17
1.4.2 Who are the Agency Staff involved in Peer Review? Page 17
-------
Page iv Peer Review Handbook
1.4.3 Who are the Decision-Makers & What are Their
Responsibilities? Page 17
1.4.4 Who are the Peer Review Leaders & What are
Their Responsibilities? Page 18
1.4.5 Who are the Peer Review Coordinators & What are Their
Responsibilities? Page 19
1.4.6 Who are the Peer Reviewers? Page 21
1.4.7 What are the Responsibilities of Peer Reviewers? Page 21
1.4.8 What is an Independent Peer Reviewer? Page 21
1.4.9 When does an Agency Internal Peer Reviewer Qualify
as Independent? Page 21
1.4.10 What is a Peer Review Panel? Page 22
1.4.11 What is a Subject Matter Expert? Page 22
1.4.12 What is the Role of the Science Policy Council (SPC)? Page 22
1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)? . Page 22
1.4.14 What is the Role of the Office of Research and
Development (ORD)? Page 23
2. PLANNING A PEER REVIEW Page 25
2.1 Overview Statement Page 25
2.2 Determining Which Work Products to Peer Review Page 25
2.2.1 What are Scientific and Technical Work Products? Page 25
2.2.2 What Scientific and Technical Work Products Need
Peer Review? Page 25
2.2.3 How Does One Determine Whether a Scientific and/or
Technical Work Product is "Major"? Page 26
2.2.4 What Economic Work Products Need Peer Review? Page 27
2.2.5 How Should Peer Review be Handled for Products
Developed under an Interagency Agreement? Page 28
2.2.6 Should Products from Grants, Contracts and Cooperative
Agreements Receive Peer Review? Page 28
2.2.7 Should Site Specific Decisions be Subject to Peer Review? Page 28
2.2.8 Should NEPA Products (e.g., EISs) be Subject to Peer Review? . Page 29
2.2.9 Should Environmental Regulatory Models be Peer Reviewed? . . Page 29
2.2.10 Is Peer Review Needed for Other Organization's Work Products
that Have been Submitted to EPA for Use in Decision Making? . Page 29
2.2.11 Can Work Products That are Not Determined to be
Major Still be Peer Reviewed? Page 30
-------
Peer Review Handbook Page v
2.3 Determining Which Work Products Do Not Receive Peer Review Page 30
2.3.1 Are There Circumstances When a Major Work Product is
Not Peer Reviewed? Page 30
2.3.2 What Products Normally Do Not Need Peer Review? Page 31
2.4 Choosing a Peer Review Mechanism Page 32
2.4.1 How Do You Determine the Appropriate Peer Review
Mechanism? Page 32
2.4.2 What are Examples of Internal Peer Review? Page 33
2.4.3 What are Examples of External Peer Review? Page 34
2.4.4 What is the Role of Peer Review by a Refereed
Scientific Journal? Page 35
2.4.5 Is Peer Review Necessary when Journal Articles are
Used in an Agency Work Product? Page 35
2.4.6 When and How Often Should Peer Review Occur? Page 36
2.4.7 What Factors are Considered in Setting the Time Frame
for Peer Review? Page 36
2.4.8 Which Office/Region or Other Agency is Responsible for
Conducting the Peer Review? Page 37
2.5 Creating the Peer Review Record Page 37
2.5.1 What is the Peer Review Record? Page 37
2.5.2 How Can the Peer Review Record Improve the
Peer Review Process? Page 37
2.5.3 What Should Be in the Peer Review Record? Page 37
2.5.4 What Should I Do with a Peer Review Record That
Pertains to a Rulemaking Action? Page 38
2.5.5 When Should the Peer Review Record Building Process Begin? . Page 38
2.5.6 What are the Differences in Record Keeping for a Review
by an Individual Compared to a Panel? Page 39
2.5.7 Where Should the Peer Review Record be Kept and
For How Long? Page 39
2.6 Budget Planning Page 40
2.6.1 What Budgetary Factors Should I Consider in a Peer Review? . . Page 40
2.6.2 What Input is Needed for the Annual Budget Formulation
and Budget Execution Process? Page 40
2.7 Legal Considerations Page 40
2.7.1 Are There Legal Ramifications From the Peer Review Policy? . . Page 40
2.7.2 Is Legal Advice Needed? Page 41
2.7.3 Is Peer Review Subject to the Federal Advisory
Committee Act (FACA)? Page 41
-------
Page vi Peer Review Handbook
3. CONDUCTING A PEER REVIEW Page 43
3.1 Overview Statement Page 43
3.2 Charge to the Peer Reviewers Page 43
3.2.1 What is a Charge? Page 43
3.2.2 What are the Essential Elements of a Charge? Page 44
3.2.3 Where Can I Get an Example of a Charge? Page 44
3.3 Time Line Page 44
3.3.1 What are the Factors in Scheduling a Peer Review? Page 44
3.4 Selection of Peer Reviewers Page 45
3.4.1 What are Considerations for Selecting Peer Reviewers? Page 45
3.4.2 Where Do I Find Peer Reviewers? Page 45
3.4.3 Are External or Internal Peer Reviewers Preferred? Page 46
3.4.4 What is Important in the Mix of a Peer Review Panel? Page 46
3.4.5 What is a Conflict of Interest? Page 47
3.4.6 What Techniques Help Ensure Disclosure and Appropriate
Resolution of Conflicts of Interest? Page 47
3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews? . Page 48
3.4.8 Are There Constraints to Selecting Peer Reviewers? Page 49
3.5 Materials for Peer Reviewers Page 49
3.5.1 What Instructions Do You Give Peer Reviewers? Page 49
3.5.2 What Materials Should be Sent to Peer Reviewers? Page 50
3.5.3 How Closely can EPA Interact with Peer Reviewers
During the Review? Page 51
3.6 Peer Review Services Page 52
3.6.1 What are Gratuitous Services for Peer Review? Page 52
3.6.2 Can I Use a Contract to Obtain Peer Review Services? Page 52
3.6.3 How Do I Write a Statement of Work for Contracts? Page 53
3.6.4 What are Advisory and Assistance Services (AAS) or
Sensitive Activities? Page 53
3.6.5 What are Some Management Controls for Contracts? Page 54
3.6.6 Can the Agency Identify and/or Select Peer Reviewers
Through a Contract? Page 56
3.6.7 Can I Use Simplified Acquisition Procedures to Obtain
Peer Reviewers? Page 57
3.6.8 How is Travel Handled with Contracts or Purchase Orders? .... Page 58
3.6.9 How is Travel Handled with Special Government Employees? . . Page 59
-------
Peer Review Handbook Page vii
4. COMPLETING A PEER REVIEW Page 61
4.1 Overview Page 61
4.2 Final Work Product Page 61
4.2.1 How Do I Incorporate Peer Review Comments into the
Final Work Product? Page 61
4.2.2 What Actions are Potentially Forthcoming from Peer Review? . . Page 62
4.2.3 Can the Identity of Peer Reviewers be Kept Anonymous? Page 62
4.3 Completing the Peer Review Record Page 63
4.3.1 How Do I Complete the Peer Review Record? Page 63
4.3.2 Where Should the Peer Review Records be Kept, and for
How Long? Page 63
SUBJECT INDEX Page 65
COMMONLY USED ACRONYMS Page 67
APPENDIX A - Peer Review Policy Page A-l
Peer Review Program Page A-2
Peer Review and Peer Involvement at the U.S. Environmental
Protection Agency Page A-5
APPENDIX B - Examples of Charges Page B-l
CHARGE EXAMPLE 1 - Charge to Reviewers for the WTI Draft
Final Risk Assessment Page B-2
CHARGE EXAMPLE 2 - IRIS Pilot Program Page B-l 1
CHARGE EXAMPLE 3 - Science Advisory Board Review of the
Agency's National Risk Management Research Laboratory's
(NRMRL) Program Page B-14
CHARGE EXAMPLE 4 - Science Advisory Board (SAB) Review of the
Technical Aspects of the Multi-Agency Radiation Survey and Site
Investigation Manual (MARSSIM) Page B-15
CHARGE EXAMPLE 5 - Science Advisory Board (SAB) Review of the
Statistical Performance of the Agency's Protozoan Oocyst
Monitoring Methods Page B-16
CHARGE EXAMPLE 6 - Science Advisory Board (SAB) Review of
the Environment Monitoring and Assessment Program (EMAP)
Research Strategy and Research Plan Page B-17
-------
Page viii Peer Review Handbook
APPENDIX C - Guidance on Requesting a Review by the Science Advisory
Board (SAB) Page C-l
APPENDIX D - Example Statements of Work for Contracts Page D-l
STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical
Review Contractor for Panel Review of Assistance Agreement or
Fellowship Applications Page D-2
STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization
Tool Report Page D-5
STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of
Protozoa Method Development Criteria Document PageD-12
APPENDIX E - References Concerning Peer Review Page E-l
-------
Peer Review Handbook Page ix
FOREWORD
EPA's Science Policy Council (SPC) has organized this Peer Review Handbook as
guidance to EPA staff and managers on the organization and conduct of peer review pursuant to
the Administrator's June 7, 1994 Peer Review Policy statement. The Handbook is based in part
on the central themes set forth in the Policy statement (see Appendix A for the full policy):
Major scientifically and technically based work products related to Agency decisions
normally should be peer reviewed. Agency managers within Headquarters, Regions,
laboratories, and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope, and
timing. These decisions are made in conformance with program goals and priorities,
resource constraints, and statutory or court-ordered deadlines. For those work products
that are intended to support the most important decisions or that have special importance
in their own right, external peer review is the procedure of choice. Peer review is not
restricted to the penultimate version of work products; in fact, peer review at the
planning stage can often be extremely beneficial.
In addition, the Handbook augments these themes by stating and explaining widely-
accepted principles and practices that have long guided peer review in the universities, in private
research organizations, and at the EPA and other government agencies.
The goal of the Peer Review Policy and this
Handbook is to enhance the quality and credibility of The Policy requires peer review
. j • • u ^u A4.u • 4.-C j of the basis of the decision n.e.,
Agency decisions by ensuring that the scientific and , , , . . . . ~
. , • , , 11 . j i • .u j • • the underlying major scientific
technical work products underlying these decisions ,. , • , , , ,
. , i r. • u-j j^ and/or technical work products),
receive appropriate levels of peer review by independent ... , . . .. 1f
t fioi the decision itself.
scientific and technical experts. To serve this goal, the ^^^^^^^^^-^^J^^^^^^^
Handbook provides information and outlines procedures
in several different areas:
basic principles and definitions, including distinctions between peer review and
peer input, public comment, and stakeholder involvement;
preparing for peer review, including identifying work products, identifying
appropriate peer review mechanisms, and identifying qualified experts; and,
conducting and completing peer reviews, including materials required for peer
review, creating a peer review record, and utilizing peer review comments.
-------
Pagex Peer Review Handbook
This Handbook has three parts. The first contains flow charts that outline the key steps in
conducting a peer review, along with a managers' checklist for planning peer reviews. The
second part contains peer review guidance detailing the procedures outlined in the flow charts in
a question and answer format. The third part contains Appendices including the 1994 Peer
Review Policy and examples to help perform quality peer reviews. Some procedures outlined in
the 1994 Policy have been completed while others are continuing to change in line with Agency
experience. These changes have been incorporated into the current Handbook.
Dorothy E. Patton, Ph.D.
Executive Director, Science Policy Council
Science Policy Council
Fred Hansen, OA, Chair Lynn Goldman, OPPTS, Vice-Chair
Joseph Alexander, ORD Donald Barnes, OA
Tudor Davies, OW Jeanne Fox, Region 2
Sylvia Lowrance, OECA Ramona Trovato, OA
Michael Shapiro, OSWER Lawrence Weinstock, OAR
Robert Wolcott, OPPE
Science Policy Council Steering Committee
Dorothy Patton, ORD, Chair Donald Barnes, OA
Patricia Cirone, Region 10 William Farland, ORD
Michael Feldman, OCFO Penelope Fenner-Crisp, OPPTS
Michael Firestone, OPPTS Jerri-Anne Garl, Region 5
Roland Hemmett, Region 2 Carl Mazza, OAR
James Nelson, OGC Jennifer Orme-Zavaleta, ORD
Peter Preuss, ORD Larry Reed, OSWER
Joseph Reinert, OPPE Rosemarie Russo, Region 4
Vanessa Vu, OPPTS Mary Ellen Weber, OPPTS
Jeanette Wiltse, OW William Wood, ORD
Science Policy Council Staff
Edward Bender Kerry Dearfield
James Rowe Mary McCarthy-O'Reilly
Dorothy Patton, Executive Director, Science Policy Council
-------
Peer Review Handbook Page 1
U.S. Environmental Protection Agency
SUMMARY OF THE PEER REVIEW PROCESS
This section of the Peer Review Handbook contains flowcharts and
descriptions of the major steps in conducting a peer review. Cross references to the
appropriate section on Peer Review Guidance are shown in parenthesis and bolded.
The Managers Planning Checklist for Peer Review (on page 5) is designed to
give Managers/Decision-Makers a simple tool to help plan for a successful peer
review. It asks questions that should be considered by a manager or Decision-
Maker during the peer review process to insure that necessary actions are taking
place. This checklist is also intended to be used by staff (especially Peer Review
Leaders and Coordinators) to inform managers and Decision-Makers on some of
the key steps and considerations that are necessary in carrying out a successful peer
review.
-------
Page 2
Peer Review Handbook
Figure 1 - Flowchart for Planning a Peer Review
Determination of "major scientific and
technical work product":
~ It is a scientific, engineering,
economic, or statistical document
(§ 2.2.1)
~ Determine if the work product is
major (§ 2.2.3)
~ Major products meet certain criteria
(§ 2.2.3)
Work product is a candidate for peer
review :
~ Major work products are subject to
peer review (§ 2.2.2)
~ Some non-major work products have
to be evaluated to determine if peer
review would still benefit the product
(§ 2.2.11)
Work product is not a candidate for peer
review :
~ Most non-major work products are
typically not candidates for peer
review (§ 2.3.2)
~ Major work product consists only of
science previously peer reviewed and
adequate under the Agency's Policy
(§ 2.3.1)
~ Place work product on List C
(§ L3.2c))
If a work product is subject to peer review:
~ Identify basis for charge (§ 3.2.1)
- Identify key staff (§ 1.4)
~ Create a peer review record (§ 2.5)
~ Ensure source of funding for the peer
review (§ 2.6)
~ Estimate completion date for peer
review (§ 3.3.1)
- Place the work product on List B (§ 1.3.2b))
Prepare
Work Product
Work Product is Not
Subject to Peer Review
Policy
Work Product is a Candidate
for Peer Review
• Identify Basis for Charge
-Identify Key Staff
- Create Peer Review Record
• Ensure Availability of Funds
• Determine Overall Time Frame
Place Work Product on
List B
Continue with Conducting
a Peer Review
-------
Peer Review Handbook
Page 3
Figure 2 - Flowchart for Conducting a Peer Review
6.
Develop the charge (§ 3.2.2):
~ Determine which key issues to address
~ Include in peer review record (§ 2.5.3)
Select a peer review mechanism (§ 2.4)
- Internal (§ 2.4.2)
- External (§ 2.4.3)
- Mail review (§ 2.4.3)
~ Face to face meeting (§ 2.4.3)
~ One time or multiple meetings (§ 2.4.6)
~ Include logistical information in peer
review record (§ 2.5.3)
Determine the specific time line (§ 3.3):
~ When will the review be started
~ What are the intermediate check points
~ What is the deadline for completion
Select peer reviewers (§ 3.4):
~ Determine sources of peer reviewers (§ 3.4.2)
~ Determine expertise required (§ 3.4.4)
~ Consider balance/address (§ 3.4.4)
- Consider conflicts of interest (§ 3.4.5 & 3.4.6)
~ Include documentation in peer
review record (§ 2.5.3)
Materials for the peer review (§ 3.5):
~ Obtain materials from Program for review
~ Prepare instructions for peer reviews (§ 3.5.1)
~ Forward materials to peer reviewers (§ 3.5.2)
~ Include copy of materials in peer
review record (§ 2.5.3)
Conduct the peer review
~ Obtain written comments from reviewers
~ Include in peer review record (§ 2.5.3)
Note: Some of these steps may occur concurrently.
Add to Peer
Review Record
Continue with Completing
a Peer Review
-------
Page 4
Peer Review Handbook
Figure 3 - Flowchart for Completing a Peer Review
Receive Comments from
Peer Reviewers
Evaluate comments from peer reviewers (§ 4.2.1)
~ Responding and reacting to comments
~ Obtaining clarification, if needed
Comments that are considered, but not used (§ 4.3.1)
~ Determine why not used and document
~ Include comments in peer review record (§ 2.5.3)
Comments that are useful (§ 4.3.1)
~ Revise the work product by
incorporating comments
~ Send revised work product back to
peer reviewers, if necessary
~ Include comments in peer review record (§ 2.5.3)
Finalize work product (§ 4.3.1)
~ Include in peer review record (§ 2.5.3)
~ Move work product from List B to List A
Are Peer
Reviewers
Comments
used in Work
Product?
Move the Work Product
from List B to List A
-------
Peer Review Handbook Page 5
Managers Planning Checklist for Peer Review
1) Title of Work Product:
2) What Decision/Rule/Regulation/Action Does this Work Product Support:
3) Determination of Major Scientific and Technical Work Products
D Is the work product scientific or technical yes no?
D Is the work product major or non-major?
4) Determining What Peer Review is Needed
D If major, peer review is needed
D If not major, is peer review still needed
D When does the review need to be done?
D How much time will be needed to conduct/complete the review?
D Are there court ordered deadlines or other constraints?
D Has senior management (AA/RA/others) been informed of progress/problems?
D What would constitute success for this review?
5) Determining the Resources for Peer Review
D What is the priority of this project relative to other projects in the same office?
D What resources are needed to conduct the review?
D What are the impacts of the review on personnel?
D Who will lead the peer review?
D Who will conduct the peer review?
D Who will maintain the peer review record?
D Where will the peer review record be kept?
D What mechanism will be used for the peer review?
D Has the charge been developed?
D Has internal and external coordination been initiated/completed?
D Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes
to the final work product) been made?
D How will results of the review be presented and addressed in the final work product (e.g., in a
preamble, in an accompanying appendix — as well as changes in the work product itself)?
D Has the work product been entered onto List B or C, as appropriate?
6) Comments:
-------
Page 6 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page 7
U.S. Environmental Protection Agency
PEER REVIEW GUIDANCE
-------
Page 8 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page 9
1. THE NEED FOR PEER REVIEW
1.1 Overview Statement
Peer review at the U.S. Environmental Protection Agency (EPA) takes many different
forms depending on the nature of the work product, relevant statutory requirements, and office-
specific policies and practices. In January 1993, responding to recommendations in the report
Safeguarding the Future: Credible Science, Credible Decisions, former Administrator William
Reilly issued an Agency-wide policy for peer review. Administrator Carol Browner reaffirmed
the central role of peer review in the Agency on June 7, 1994 and instituted an Agency-wide
implementation program (see Appendix A). Following Agency-wide implementation, office and
region-specific standard operating procedures (SOPs) were written and have been used since
1994. Based on the experiences of those managing peer review throughout the Agency since
then, as reported to the Science Policy Council (SPC) and its Peer Review Advisory Group
(PRAG), we found that a single Agency-wide document on peer review was needed. Therefore,
this Peer Review Handbook is created as a single, centralized form of implementation guidance
for Agency staff and managers. While the Handbook supersedes the original SOPs, Agency
offices and regions can still prepare brief tailored guidance that meets their individual needs to
supplement the information in this Handbook.
1.2 Understanding Peer Review
1.2.1 Why use Peer Review?
Peer review is intended to uncover any technical problems or unresolved issues in a
preliminary (or draft) work product through the use of independent experts. This information is
then used to revise that draft product so that the final work
product will reflect sound technical information and analyses.
Peer review is a process for enhancing a scientific or technical
work product so that the decision or position taken by the
Agency, based on that product, has a sound, credible basis. To
be most effective, peer review of a major scientific and/or
technical work product needs to be incorporated into the
up-front planning of any action based on the work product ~ this includes obtaining the
proper resource commitments (people and money) and establishing realistic schedules.
Peer review is not free;
however, not doing peer
review can be costly.
Peer review of major scientific and technical work products should not be looked upon as
another "hurdle" in the Agency decision making processes. While peer review requires that time
-------
Page 10 Peer Review Handbook
and resources be planned into the decision making process, the benefits justifies the added cost.
Peer review enhances the credibility and acceptance of the decision based on the work product.
By ensuring a sound basis for decisions, greater cost savings are realized since decisions will not
be challenged as often and extra effort will not be required to go back and redo the work product.
So while peer review is not free, the cost of not doing peer review is usually much more
expensive. Furthermore, not conducting a peer review can potentially place the Agency in the
position of attempting to defend a scientifically invalid position — which can be very costly in
terms of both resources, and more importantly, credibility.
1.2.2 What is Peer Involvement?
As defined in the Peer Review Policy, peer involvement is the process whereby Agency
staff involve subject-matter experts from outside their program in one or more aspects of the
development of work products. Peer involvement, therefore, constitutes active outreach to and
participation by the broad scientific, engineering, and economics communities beyond the
Agency (external) as well as within the Agency (internal). Typically, peer involvement takes two
general forms: peer input (ongoing discussions during the development of the work product) and
peer review (an evaluation of a workplan, preliminary draft or the like, or most often, the critical,
final objective expert evaluation of the work product).
1.2.3 What is Peer Review?
Peer review is a documented critical review of a specific Agency major scientific and/or
technical work product. The peer review is conducted by qualified individuals (or organizations)
who are independent of those who performed the work, but who are collectively equivalent in
technical expertise (i.e., peers) to those who performed the original work. The peer review is
conducted to ensure that activities are technically adequate, competently performed, properly
documented, and satisfy established quality requirements. The peer review is an in-depth
assessment of the assumptions, calculations, extrapolations, alternate interpretations,
methodology, acceptance criteria, and conclusions pertaining to the specific major scientific
and/or technical work product and of the documentation that supports them. Peer review may
provide an evaluation of a subject where quantitative methods of analysis or measures of success
are unavailable or undefined; such as research and development. Peer review is usually
characterized by a one-time interaction or a limited number of interactions by independent peer
reviewers. Peer review can occur during the early stages of the project or methods selection, or
as typically used, as part of the culmination of the work product, ensuring that the final product is
technically sound.
-------
Peer Review Handbook Page 11
1.2.4 What is Peer Input?
Many Agency work products are developed with the input of various scientific and
technical experts inside and outside the Agency. Like the contribution made by peer reviewers,
peer input is valuable and enhances the scientific or technical basis of the products. Peer input,
sometimes referred to as peer consultation, generally connotes an interaction during the
development of an evolving Agency work product, providing an open exchange of data, insights,
and ideas. Peer input may be characterized by a continued and iterative interaction with
scientific experts during work product development. A common example of peer input is the
input received from workgroup members during development of a product. Many Agency
products are developed through the efforts of a workgroup, which may include external experts,
such as State and Tribal representatives. These workgroup members have an active, ongoing
participation in developing the work product. Another example of obtaining peer input is of an
Agency office sending a draft work product to a list of stakeholder representatives for general
comments (stakeholder representatives often include experts who could be considered "peers").
1.2.5 How is Peer Review Different from Peer Input?
The key distinctions between peer input as described above and formal peer review are
the independence of the peer reviewers and their level of involvement. The goal of peer review
is to obtain an independent, third-party review of the product from experts who haven't
substantially contributed to its development. When experts have a material stake in the outcome
of the peer review (such as a regulated party) or have participated substantially in the
development of the product (such as a workgroup member), those experts' reviews may not
qualify as unbiased, independent peer review and may be better characterized as peer input.
Peer Input is not a
substitute for Peer Review
It is clear that peer input provides valuable
contributions to the development of the work product.
However, peer input does not substitute for peer review. Once
a work product is considered major, it is a candidate for peer
review and entered on List B (Candidate Products for Future
Peer Review -- see section 1.3.2b)) — even though the work product may already have a
substantial amount of peer input. In other words, one cannot argue that a peer review is not
necessary if a major work product has received "enough" peer input. If the work product is not
considered major and has had peer input, it is entered on List C (Products for Which a Decision
has been made not to Peer Review — see section 1.3.2c)) with comments about the extent of the
peer input.
-------
Page 12 Peer Review Handbook
1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer
Reviewer for the Same Work Product Later in the Process?
Generally, the answer is no as that expert is no longer independent, but rather a
contributor to the work product. There may be special circumstances where the expertise is so
narrow that another peer reviewer isn't available. The Peer Review Leader (see section 1.4.4)
will normally be responsible for making this determination.
1.2.7 How is Peer Review Different from Public Comment?
Peer review and public comment are mutually exclusive. Public comment solicited from
the general public through the Federal Register or by other means is often required by the
Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited
for policy purposes. The Agency takes public comment on some strictly scientific products and
almost all regulatory decisions. Public commenters usually include a broad array of people with
an interest in the technical analysis or the regulatory
decision; some are scientific experts (which may
provide some peer input), some are experts in other
areas, and some are interested non-experts. The
critical distinction is that public comment doesn't
necessarily draw the kind of independent, expert
information and in-depth analyses expected from the peer review process. Public comment is
open to all issues, whereas the peer review process is limited to consideration of technical issues.
While it may be an important component of the review process, public comment does not
substitute for peer review.
Public comment does not substitute
for peer review.
1.2.8 How is Peer Review Different from Stakeholder Involvement?
Stakeholder involvement occurs when the Agency works with external interest groups
that have some stake in or concerns over the outcome of the technical work product or regulatory
position. This is an interactive process, working with
other agencies, industry groups, regulated-community
Stakeholder involvement is not
a peer review mechanism.
experts, environmental groups, other interest groups that
represent a broad spectrum of the regulated community,
etc., and usually strives for a consensus approach. The
goal of peer review, on the other hand, is to obtain an
independent, third-party review. Stakeholder involvement is not a peer review mechanism — as
with peer input, once a decision is made that peer review is needed, stakeholder involvement
does not substitute for peer review even though it adds value to the work product.
-------
Peer Review Handbook Page 13
1.2.9 What Role does Peer Review have in the Regulatory Development Process?
The peer review of scientific and technical work products that support rulemaking actions
is an important, fundamental step in the policy setting process and which affirms the credibility
of the Agency. Because new rules, and the work products supporting them, must often withstand
intense scrutiny by the general public and the stakeholders involved in the action, the peer review
process selected for such work products needs to be well planned and documented. The rule or
regulation itself is not subject to the Peer Review Policy. However, if the rule or regulation is
supported by a major scientific and/or technical work product, that work product should be peer
reviewed prior to its use in the rule (see section 2.2 for determination of major work products).
The decision to peer review or not peer review any scientific and/or technical work product will
be documented through the Agency's annual peer review reporting process (see section 1.3).
Remember, public comment and stakeholder involvement do not constitute peer review.
Tier 1 and Tier 2 rulemakings, are by definition important, major Agency rulemakings
within the Agency. Therefore, work products supporting Tier 1 and Tier 2 rules in particular
(including rules that are determined to be "significant" by OMB under Executive Order 12866
because they have an economic impact of $100 million or more) should be closely scrutinized to
determine whether they meet the criteria for major (see section 2.2.3). Work products supporting
Tier 3 rulemakings may also be considered major and thus candidates for peer review. External
peer review is the procedure of first choice for a work product that is intended to support a Tier 1
or Tier 2 rulemaking. Although acceptable in certain circumstances, any decision to use an
internal peer review mechanism for such work products would be the exception rather than the
rule. For work products supporting a Tier 3 rule, internal or external peer review may be
appropriate depending on the nature of the product and other factors (see section 2.4.1). For Tier
1 and Tier 2 rulemakings, the Final Agency Review/closure memo needs to indicate that the Peer
Review Policy was followed. For Tier 3 rulemakings, the action memo needs to indicate that the
Peer Review Policy was followed.
Analytic blueprints are required for Tier 1 and Tier 2 rulemakings, and are encouraged for
Tier 3 rulemakings; some individual EPA offices require it for Tier 3. For peer review purposes,
the analytic blueprint is the process whereby the project manager identifies the supporting
scientific and technical work products and identifies needed peer review. It also shows the
schedule of the peer review in the context of the schedule for the overall rulemaking. In general,
peer review should be completed as early in the process as practicable. Where possible, peer
review of work products should be completed prior to issuance of the draft regulation. In some
cases, support work products for final regulations may require an additional peer review if those
scientific and technical work products change significantly after the public comment period.
-------
Page 14 Peer Review Handbook
1.2.10 What Role does Peer Review have in Regulatory Negotiations?
Regulatory negotiations are not candidates for peer review; however, to ensure final
decisions are based on sound and credible science, the major scientific and technical work
products that support the negotiation need peer review before the negotiation takes place.
1.3 Annual Agency Reporting Requirements
1.3.1 What are the Annual Reporting Requirements?
The Peer Review Coordinator (for each AA/RA; see section 1.4.5) will organize an
annual review of all peer review activities and submit this information to the Office of Research
and Development (ORD). ORD will staff this function at the direction of the Deputy
Administrator. In the Deputy Administrator's annual call for submissions, guidance on format
and submission of this information will be provided. ORD will review the submissions for
completeness, i.e. all information is provided and products are accounted for each year. ORD
will then provide a review of the completeness of the information in the submissions through
consultation with the appropriate persons in each organization (see section 1.4). ORD will then
consolidate the information and findings for the SPC and the Deputy Administrator. Any
conflicts arising from the review will be resolved by the Deputy Administrator. The due date for
the annual reporting will be announced each year in the annual call letter; however, for planning
purposes, it is normally due in the early summer.
1.3.2 What Listings are Required for the Annual Reporting?
Three listings of products are required for the annual reporting:
a) List A - Products Peer Reviewed Since 1991
1) List A is a cumulative list of peer reviewed products from 1991 to the
present.
2) For each new work product entered onto List A, a short report
summarizing the peer review must also be provided and attached to List A.
This summary report is signed by the appropriate Decision-Maker (see
section 1.4.3). Signatures by Decision-Makers should be affixed as
originals in the peer review record and copies submitted with List A.
However, for reports submitted by electronic means, the submitted report
should identify the name of the signer and the date signed.
-------
Peer Review Handbook Page 15
3) The format and content of the summary report will be provided in the
annual call letter (see section 1.3.1).
b) List B - Candidate Products for Future Peer Review
1) List B is a listing of products that are expected to be peer reviewed in the
near future.
2) This List contains major scientific and technical work products and any
non-major scientific and technical work products for which peer review
has been deemed necessary or appropriate.
3) Work products placed on List B remain on List B until they are either peer
reviewed (after which they are moved to List A) or a decision is made not
to peer review that work product (at which point it is moved to List C).
4) The peer review summary report that eventually accompanies each List A
peer review product should be initiated at the time a work product is
decided to be peer reviewed and filled in as the peer review proceeds.
This is easier than waiting until the peer review of the work product is
completed while the details are still fresh.
c) List C - Products for Which a Decision has Been Made Not to Peer Review
1) List C is a cumulative listing of all scientific and technical work products
that do not receive peer review (see section 2.3).
2) List C includes: (a) any major scientific and technical work products for
which a decision was made not to peer review; (b) those work products
which were originally placed on List B, but for which it was decided that
peer review was not necessary (e.g., the product was not used in decision
making; the project was canceled); and (c) all non-major scientific and
technical work products.
3) List C also includes several categories of work products that would not
normally receive peer review. It is not necessary to list these work
products individually on List C, however, the total number prepared by the
organization must be included on List C. The organization may need to
identify the individual products if requested (e.g., due to litigation, FOIA,
-------
Page 16 Peer Review Handbook
etc.). Such categories can include, but may not be limited to: chemical
action reports, RCRA permits, scientific analyses for Premanufacturing
Notices (PMNs) that are conducted on a routine basis and that do not
deviate from established practice, and NPDES permits.
4) List C also includes any scientific papers (articles) that were published in a
credible peer reviewed journal (i.e. for purposes of listing journal articles,
only published articles, rather than projected articles, is intended). It is not
necessary to list these papers individually on List C, however, the total
number prepared by the organization and peer reviewed by journals must
be included on List C. The organization may need to identify the
individual papers and where published, if requested (e.g., due to litigation,
FOIA, etc.). (See sections 2.4.4 and 2.4.5).
5) Each work product on List C needs a brief description in the "comments"
column of the reason(s) it is not being peer reviewed. In addition, each
Decision-Maker must provide signed documentation that identifies the
work products that they decided not to peer review (other than work
products that are included in categories in 3) and 4), above). ORD will
provide a standardized format for this documentation. Signatures by
Decision-Makers should be affixed as originals in the peer review record
and copies submitted with List C. However, for reports submitted by
electronic means, the submitted document should identify the name of the
signer and the date signed.
1.3.3 Is There a Coding System for Work Products on the Annual Reporting
Lists?
The work products that are placed on these lists should be assigned a discrete code
number. A coding system will uniquely identify each work product when it is first listed on any
of these lists. This will assist in tracking each work product from year to year and when it moves
from one list to another. The code will be: YEAR (2 digits) - OFFICE/REGION (usual initials) -
RUNNING NUMBER (4 digits and in sequential numbering for that year the work product was
first placed on any list); for example, 97-OAR-0001, 98-OW-0010, 99-R05-0012, 00-OPPTS-
0054, or 01-Rl0-0002.
The code number needs to be assigned at the time the annual report is developed
(although the code number can be assigned anytime during the year that the peer review is
-------
Peer Review Handbook Page 17
planned). The code number is assigned by the Peer Review Coordinator to provide a single,
consistent numbering sequence.
1.3.4 When will the Handbook Itself be Revised?
During each annual reporting cycle, suggestions for revisions to the Handbook should be
submitted. A decision will be made by the Science Policy Council (SPC) on whether to revise
the Handbook, in part or total, or not based on the suggestions. The SPC will then direct the Peer
Review Advisory Group (PRAG) to draft the revision(s) for Agency comment and SPC approval.
1.4 The Roles of People and Organizations in Peer Review
1.4.1 Who is Ultimately Accountable for Peer Review?
Under the June 7, 1994 Peer Review Policy, the Administrator has designated the
Assistant Administrators and Regional Administrators (AAs and RAs) to be accountable for
implementing the Policy in their respective organizations. The Deputy Administrator is
ultimately responsible for peer review across the Agency and is the final arbitrator of conflicts
and concerns about peer review.
1.4.2 Who are the Agency Staff involved in Peer Review?
The principal Agency staff involved are Decision-Makers (and their line managers), Peer
Review Leaders and Peer Review Coordinators. In addition, staff in ORD have been designated
by the Deputy Administrator for ensuring the Agency's Peer Review Policy requirements are
met.
1.4.3 Who are the Decision-Makers & What are Their Responsibilities?
The AA/RA is the ultimate Decision-Maker for their organization and is accountable for
the decisions regarding the identification of major scientific and technical work products and the
mechanism(s) of peer review utilized for each of the products. The AA/RA may designate the
Office Directors and Division Directors (or other appropriate level line-managers) as the front-
line Decision-Makers.
Generally, the Decision-Makers [usually line-managers] decide whether a work product is
major and needs peer review or not, and what peer review mechanism to use. Furthermore, the
Decision-Makers commit the resources needed to ensure a proper peer review. Decision-Makers
are responsible for ensuring that the peer reviews are properly performed and documented.
-------
Page 18 Peer Review Handbook
Specific responsibilities of the Decision-Maker(s) are the following:
a) Determine which work products in their organization require peer review
b) Designate (in conjunction with the Project Manager) a Peer Review Leader to
organize the peer review
c) Provide advice, guidance, and support to the Peer Review Leader in the
preparation, conduct, and completion of the peer review
d) Ensure that sufficient funds are designated in the office's budget request to
conduct the peer review; also ensure that adequate resources and/or extramural
management support are available for the peer review
e) Establish a realistic peer review schedule
f) Designate the stage(s) of product development where peer review is appropriate
g) Ensure that the results of peer review are carried forward in the work product
h) By signature, document the decisions made that are reported in the annual
reporting to the SPC and Deputy Administrator; this includes signing the peer
review summary report for each completed peer review
i) Certify any decision NOT to peer review a product by signature on a List C
submission
1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities?
The Peer Review Leader organizes and oversees the peer review for a specific individual
work product. This person(s) can be the Decision-Maker(s), but will usually be someone who is
authorized by the Decision-Maker to organize, conduct, and complete the peer review. This
individual may also be the Project Manager for the work product. The Peer Review Leader will
obtain the assistance and support of the Peer Review Coordinator (see below) as well as any
others within the Agency to help perform the peer review. The Peer Review Leader will be
chosen on a case by case basis depending on the work product needing peer review.
-------
Peer Review Handbook Page 19
Specific responsibilities of the Peer Review Leader are these:
a) Keep the Decision-Maker informed of the status of a given project; provide Peer
Review Coordinator with data for the annual report
b) Organize, conduct, and complete the peer review following Agency procedures
c) Establish and maintain the peer review record for the specific individual peer
review currently being performed (see section 2.5); this includes initiating the peer
review summary report for the Decision-Maker to sign when the peer review is
completed
d) Select the peer reviewers in consultation with others involved with the peer
review (e.g., Decision-Maker or responsible line manager)
e) Advise peer reviewers of their responsibilities
f) Brief management and obtain management approval on the approach to
responding to peer reviewer comments
g) Provide information to Decision-Maker on the charge, profile of peer reviewers,
the peer review comments, and how those comments are to be used
h) Notify the Peer Review Coordinator that the peer review is completed for the
annual report
i) Archive the peer review record in a manner consistent with their organization's
archiving procedures
1.4.5 Who are the Peer Review Coordinators & What are Their Responsibilities?
The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor peer
review activities in their respective organization or organizational unit. This person must be of
sufficient stature and judgment to have the access to and confidence of all levels of office or
regional management when needed. The Peer Review Coordinator is the main contact for their
organization; they can also direct interested parties to other persons/contacts in the office on
specific work products (e.g., Peer Review Leader).
-------
Page 20 Peer Review Handbook
Specific responsibilities of the Peer Review Coordinator are these:
a) General oversight responsibility for the Office's or Region's peer review process
b) Report peer review activities to the AA/RA
c) Help mediate difficult issues between their organization and others; if they can't
resolve issue, then bring the issue to the attention of the appropriate level
Decision-Makers in each organization for resolution.
d) Function as the liaison with ORD and the Science Policy Council (SPC):
1) Represent office/region before the SPC
2) Advise ORD of any changes in the list of work products and peer review
mechanisms during the annual reporting, and when necessary, at other
times
3) Participate in Agency peer review training, workshops, etc., as requested
and disseminate this information to the organization; coordinate and/or
present training within their organization
e) Submit information on organization's peer review candidates for each year as
requested (this is the annual reporting, see section 1.3)
1) Generate and update Lists A, B, and C
2) Assure the proper approval signature on the completed submission with
the accompanying explanation for any departures from the Policy
f) Establish procedures to assure that the required work product peer review
documentation (i.e., peer review record) is filed and maintained in an appropriate
manner (see section 2.5)
g) Provide advice, guidance, and support to the various Peer Review Leaders for the
performance of the peer reviews
h) Distribute Agency-wide peer review guidance and materials to appropriate
office/region personnel, as requested
-------
Peer Review Handbook Page 21
1.4.6 Who are the Peer Reviewers?
Peer reviewers are individuals who have technical expertise in the subject matter of the
work product undergoing peer review. Peer reviewers can come from EPA, another Federal
agency, or from outside of the Federal government.
1.4.7 What are the Responsibilities of Peer Reviewers?
Peer reviewers should maintain the confidentiality of the product, perform the review in a
timely manner, and be unbiased and objective. Peer reviewers need to be willing participants in
the peer review process — they should agree to read all materials, attend all sessions, and protect
confidential information that arises.
1.4.8 What is an Independent Peer Reviewer?
An independent peer reviewer is an expert who wasn't associated with the generation of
the specific work product either directly by substantial contribution to its development or
indirectly by significant consultation during the development of the specific product. The
independent peer reviewer, thus, is expected to be objective.
The quality of the peer review is
dependent on the competence and
independence of the reviewers.
Independence is freedom from institutional,
ideological, or technical bias regarding the issues
under review and is necessary for objective, fair, and
responsible evaluation of the work product. If a
selected reviewer has a particular scientific or
technical perspective, it may be desirable to balance
the review with peer reviewers of other perspectives. Ideally, peer reviewers should be free of
real or perceived conflicts-of-interest or there should be a balancing of interests among peer
reviewers. If there are potential conflicts of interest (real or perceived), they should be fully
identified to ensure a credible peer review. (See section 3.4 for further information).
1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent?
An Agency independent peer reviewer is one who comes from a different organizational
unit than the one where the review question or document originates. A different organizational
unit usually denotes, at minimum, a different office (i.e., above a division level) within the
organization. In particular, a reviewer shouldn't come from within the chain of command, either
upward or downward.
-------
Page 22 Peer Review Handbook
1.4.10 What is a Peer Review Panel?
A peer review panel can range from a few individuals to ten or more, depending on the
issue being investigated, the time available and any limitations on resources. Individuals who
serve as peer reviewers must have appropriate scientific and technical expertise that covers the
broad spectrum of expertise required to treat the issues/questions presented in the charge.
1.4.11 What is a Subject Matter Expert?
A subject matter expert is one who has specific scientific and technical expertise in the
matter under review. The importance of scientific and technical expertise in the subject matter is
obvious, however, knowledge or just "knowing" about the subject area isn't equivalent to
expertise in the subject matter. For Agency decisions, a multi-disciplinary group of experts
corresponding to the disciplines that contribute to complex Agency decisions is often necessary
for a full and complete peer review. For example, a risk assessment that relies on both animal
and human data usually requires experts in both areas for a complete review. For economic
analyses, experts from the corresponding economic disciplines are necessary.
1.4.12 What is the Role of the Science Policy Council (SPC)?
According to the 1994 Peer Review Policy statement: "The Science Policy Council is
responsible for overseeing Agency-wide implementation. Its responsibilities include promoting
consistent interpretation, assessing Agency-wide progress, and developing recommendations for
revisions of the policy as necessary." The SPC meets its responsibilities through coordination
with the Peer Review Coordinators, the Peer Review Advisory Group (PRAG) and the Office of
Research and Development (ORD).
The SPC, PRAG and ORD are not responsible for identifying specific products for peer
review or determining the level of review or mechanisim for that review; those functions are the
responsibility of management within each Office or Region.
1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)?
The Science Policy Council has created the Peer Review Advisory Group (PRAG) to
assist in the implementation of the Agency's Peer Review Policy. The primary role of the PRAG
is to provide interpretation of the policy and to assist the SPC and Agency Offices and Regions in
the annual update of the Peer Review Handbook.
-------
Peer Review Handbook Page 23
1.4.14 What is the Role of the Office of Research and Development (ORD)?
The Deputy Administrator has designated the Office of Research and Development
(ORD) to assist the Program Offices and Regions in the collection and review of information that
is contained in the annual submission of Lists A, B and C (for detailed information, see sections
1.3.1, 1.3.2 and 1.3.3).
-------
Page 24 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook
Page 25
2. PLANNING A PEER REVIEW
2.1 Overview Statement
Planning a peer review is a critical first step to ensure a successful peer review of a work
product. The initial step is to determine whether your work product requires peer review. Once
you have determined that a peer review will be conducted, the Decision-Makers and Peer Review
Leaders need to plan an appropriate review. This includes identification of resources (budget and
personnel), the schedule for the completion of the peer review, the mechanism for peer review,
the choice of peer reviewers, and the development of the peer review record.
2.2 Determining Which Work Products to Peer Review
2.2.1 What are Scientific and Technical Work Products?
The first step in determining which work products require peer review, is to identify
products that are scientific and/or technical in nature. Scientific and technical work products are
documents or positions that are used to support a research agenda, regulatory program, policy
position or other Agency position or action. Categories include the following: risk assessments,
technical studies and guidance, analytical methods, scientific database designs, technical models,
technical protocols, statistical survey/studies, technical background materials, technical guidance,
and research plans and strategies.
Products that wouldn't be considered scientific and technical work products can include
those: that address procedural matters (e.g., planning, reporting, coordination, notification); that
are primarily policy statements (e.g., relocation policy); that are conference proceedings (unless
the proceedings are used as the scientific basis for an Agency action or decision); and that are
decision documents (e.g., Record of Decision (ROD) — the decision document itself is not
subject to the Peer Review Policy, but the underlying scientific and/or technical support work
product is a candidate for peer review).
2.2.2 What Scientific and Technical Work Products Need Peer Review?
The principle underlying the Peer
Review Policy is that all major scientific and
technical work products used in decision
making will be peer reviewed. The process
for identifying which of these products is
When in doubt about whether to peer
review a work product or not, always
decide to make it a candidate for peer
review.
-------
Page 26 Peer Review Handbook
"major" (and thus a candidate for peer review) and then determining the mechanism of review
will take into account various criteria and the circumstances surrounding the use of that work
product. To maintain flexibility, the Decision-Maker(s) for peer review should consider the full
field of possible work products that could benefit from peer review and the full spectrum of peer
review mechanisms for each product. Once a decision is made to perform peer review, the
product is listed in the annual submission of Candidate Products for Future Peer Review (List B
- see section 1.3.2b)).
2.2.3 How Does One Determine Whether a Scientific and/or Technical Work
Product is "Major"?
Determinations of a scientific and/or technical work product as "Major" will largely be
case-by-case. The continuum of work products covers the range from the obviously major,
which clearly need peer review, to those products which are not major and clearly don't need
peer review. The rest of the work products fall in-between those two distinctions. This "middle-
ground" probably represents the majority of work products, each of which needs to be evaluated
closely and be compared to certain criteria (see below). The Decision-Maker needs to make a
judgment as to whether a work product meets the criteria for major or not. There is no easy
single yes/no test of major covering the whole continuum of work products. A rule of thumb to
remember — if there is any doubt about whether a work product needs peer review, then go ahead
and consider it a candidate for peer review (and place it on List B - Candidate Products for
Future Peer Review).
Scientific and technical work products that are used to support a regulatory program or
policy position and that meet one or more of the following criteria are candidates for peer review:
a) Establishes a significant precedent, model, or methodology
b) Addresses significant controversial issues
c) Focuses on significant emerging issues
d) Has significant cross-Agency/inter-agency implications
e) Involves a significant investment of Agency resources
f) Considers an innovative approach for a previously defined
problem/process/methodology
-------
Peer Review Handbook Page 27
g) Satisfies a statutory or other legal mandate for peer review
Usually, a major scientific and/or technical work product supports a regulatory decision
or policy/guidance of major impact. Major impact can mean that it will have applicability to a
broad spectrum of regulated entities and other stakeholders, or that it will have narrower
applicability, but with significant consequences on a smaller geographic or practical scale. The
scientific and/or technical work that underlies many of the Agency's major rulemakings and
policy and guidance documents of general applicability would be designated "major" under this
scope of impact criterion because of their far-reaching or significant impacts.
The novelty or controversy associated with the work product helps determine whether it
is major or not. A major work product may be novel or innovative, precedential, controversial,
or emerging ("cutting edge"). An application of an existing, adequately peer reviewed
methodology or model to a situation that departs significantly from the situation it was originally
designed to address is a candidate for peer review. Similarly, a modification of an existing,
adequately peer reviewed methodology or model that departs significantly from its original
approach is a candidate for peer review. Determination of "significant departure" as used in this
section is the responsibility of the Decision-Maker.
In summary, a major scientific or technical work product has a major impact, involves
precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory
obligation to conduct a peer review.
2.2.4 What Economic Work Products Need Peer Review?
The following three types of major economic work products require peer review:
a) internal Agency guidance for conducting economic analysis
b) new economic methodologies or unique or novel applications of existing
economic methodologies, particularly those that are pathbreaking
c) broad-scale economic assessments of regulatory programs, such as the
Congress!onally-mandated study of the costs and benefits of the Clean Air Act
For all these types of economic work products, we will pursue an independent and expert
external peer review, such as by the recently reconstituted Environmental Economics Advisory
Committee (a subcommittee of the SAB), or other appropriate experts. The straight-forward
application of accepted, previously peer-reviewed economic methods or analyses in regulatory
-------
Page 28 Peer Review Handbook
impact analyses supporting rulemakings or policy development, however, will typically not be
subject to formal peer review. Of course, if the particular facts and circumstances of any piece of
economic analysis warrant peer review beyond that described above, the Agency will
accommodate those on a case-by-case basis.
2.2.5 How Should Peer Review be Handled for Products Developed under an
Interagency Agreement?
When funds are passed to another agency, it will likely be placed in a contract,
cooperative agreement, or grant. In many instances, the receiving agency's guidance for peer
review will not be the same as ours. (See sections 2.2.6, 2.2.8, 2.2.10 and 3.6 for further details).
2.2.6 Should Products from Grants, Contracts and Cooperative Agreements
Receive Peer Review?
If there is a scientific and/or technical work product resulting from a grant, contract, or
cooperative agreement and it is considered major and will likely be used in Agency decision-
making, the work product needs peer review. Since it would probably result in a perceived, if not
real, conflict of interest, a group that is generating the work product usually cannot conduct or
perform the peer review of its own work product. Exceptions may be made in certain instances
for organizations that have adequate and well established recognized procedures for peer review,
such as the National Academy of Sciences. In practice, the Agency may need to peer review the
product on its own, or arrange with an independent third group (e.g., via another extramural
vehicle) to arrange for the peer review. The Agency should not use the major scientific and
technical work products from grants, contracts, or cooperative agreements to support decision
making unless the work products are peer reviewed for both scientific and technical rigor and
applicability to the specific use to be made of the product.
2.2.7 Should Site Specific Decisions be Subject to Peer Review?
The site specific decision itself is not subject to peer review and doesn't need peer review
based solely on the Peer Review Policy. However, if a site specific decision is supported by a
major scientific and/or technical work product, that work product needs peer review. While the
same considerations for major apply here, several of the criteria above (see section 2.2.3;
specifically criteria b, c, d, and g) are considered more useful for regional consideration than
other criteria. So generally speaking, a close examination of how the underlying major scientific
and/or technical work product is adapted to the site specific circumstances is required.
-------
Peer Review Handbook Page 29
2.2.8 Should NEPA Products (e.g., EISs) be Subject to Peer Review?
Not everything requires peer review, and in the case of an Environmental Impact
Statement (EIS) prepared under the requirements of the National Environmental Policy Act
(NEPA), the document already has received extensive review through the "scoping" and
interagency review processes that are part of NEPA.
The rule of thumb is that if the underlying scientific and/or technical work product is
major, then the work product needs peer review. In general, the Agency's role in the NEPA
document would suggest what sort of review the document gets. If EPA is developing the
document as part of an EPA action/decision (EPA is the Lead agency under NEPA), and it meets
the definition of "major," then it needs independent peer review. If it is not a major work product
(little impact, non-controversial, etc.), then peer input/continuing involvement might well be
appropriate.
On the other hand, if EPA is reviewing an EIS from another Agency (EPA is not the Lead
agency under NEPA), it is likely that we are reviewing for conflicts with EPA policy and general
environmental concerns. However, EPA must ask if the underlying major scientific and/or
technical work product that supports the EIS has been peer reviewed. If not, this would raise
concern about the full credibility and soundness of the EIS based on the science and technical
support. EPA should work with the other organization/agency to ensure that the major scientific
and/or technical work product receives adequate peer review.
2.2.9 Should Environmental Regulatory Models be Peer Reviewed?
Generally, yes. Specific guidelines for the peer review of environmental regulatory
models have been published by the Agency. These can be found on the EPA web site under the
Science Policy Council home page (http://www.epa.gov/ORD/spc).
2.2.10 Is Peer Review Needed for Other Organization's Work Products that Have
been Submitted to EPA for Use in Decision Making?
Yes. Any scientific and/or technical work product that is used in Agency decision
making and is considered major needs peer review regardless if the work product is produced by
the Agency or another organization. It is hoped that the other organization outside the Agency
has had the work product independently peer reviewed and the peer review meets the intent of
the Agency's Peer Review Policy and EPA's proposed use of the product. Agency staff should
examine closely the particulars of the peer review to ensure independence and a conscious effort
to incorporate the peer reviewers' comments into the final work product. If there are perceived,
-------
Page 30 Peer Review Handbook
or real, conflicts of interest, this may preclude the use of that peer review and, in those instances,
another peer review would be needed. See section 2.4 for further details on peer review
mechanisms.
2.2.11 Can Work Products That are Not Determined to be Major Still be Peer
Reviewed?
Yes, they could be. Scientific and technical work products that do not come under the
"major" distinction discussed above may nonetheless be candidates for peer review. For
example, a project manager may decide to use peer review because of particular program needs
and goals. Peer review may also be warranted because it adds substantial value to the work
product.
2.3 Determining Which Work Products Do Not Receive Peer Review
2.3.1 Are There Circumstances When a Major Work Product is Not Peer
Reviewed?
There may be circumstances where a work product is considered major, but a decision for
no peer review can then be justified. For example:
a) Additional peer review is not required with work that has been previously
reviewed by recognized experts or an expert body. For example, a cancer risk
assessment methodology or an exposure modeling technique that was the subject
of earlier peer review would not require additional peer review, even if the
product supported a significant Agency decision.
b) Additional peer review is not required if an application of an adequately peer
reviewed work product does not depart significantly from its scientific or
technical approach (see section 2.2.3).
c) Additional peer review is not required when the scientific and/or technical
methodologies or information being used are commonly accepted in the field of
expertise (this would need the appropriate documentation to support the
commonly held view).
d) Most often, a major work product would not receive peer review when the
regulatory activity or action which the work product supports is terminated or
canceled — no further action, including peer review, is necessary.
-------
Peer Review Handbook Page 31
e) In a few instances, statutory and court ordered deadlines and other time
constraints may limit or preclude peer review of products that would otherwise be
considered major. However, it is up to the Decision-Maker(s) to make every
attempt possible to assure that peer review of major work products occurs taking
into account these deadlines.
f) Very rarely, resource limitations may also restrict peer review. Programs or
Regions will evaluate these circumstances on a case by case basis; decisions will
be based on consultations involving line management, the Project Manager, the
Peer Review Leader, and the Peer Review Coordinator.
If peer review of a major scientific and/or technical work product is not conducted, a
written justification must be placed in the "comments" section of List C (Products for Which a
Decision has Been Made Not to Peer Review). The justification is signed-off by the appropriate
Decision-Maker (see section 1.3.2c)).
2.3.2 What Products Normally Do Not Need Peer Review?
Products that are not major scientific and technical work products normally do not require
peer review under the intent of the Peer Review Policy. Most of these scientific and technical
work products are then placed on List C with a written comment of why it was determined to be
not major (see section 1.3.2c)). This justification is signed-off by the appropriate Decision-
Maker to assure that all scientific and technical work products received consideration for peer
review.
Some scientific and technical work products are not considered major and generally do
not need to be placed on List C (Products for Which a Decision has Been Made Not to Peer
Review). These types of work products typically include: derivative products (i.e., a product that
only summarizes an already peer reviewed product or products), compendiums of existing
models, methods and/or technologies; or preliminary or incidental analyses prepared separately
from the work product ultimately used to support an Agency action or decision (e.g., during the
course of developing a rule, managers may direct staff to prepare various "what if analyses;
those that aren't used in the work product do not need to be listed).
-------
Page 32 Peer Review Handbook
2.4 Choosing a Peer Review Mechanism
2.4.1 How Do You Determine the Appropriate Peer Review Mechanism?
During the planning of a peer review, the Decision-Maker and the Peer Review Leader
may consider several mechanisms for the peer review of major scientific and technical work
products. These options range from consultations with EPA colleagues not involved in
developing the product to a large and formal panel of
outside subject matter experts. The peer review effort I
might be a focused one-time evaluation, or could The ^hanism of the peer
encompass several examinations over the course of a review ma c es e
project. In principle, peer review provides the greatest importance and complexity of
,.,.,., c • , , , , .. • , ,, the major work product.
credibility tor major work products when it involves well- _
qualified external reviewers, is intensive in its
examination, and operates through a more or less formal
process. As a practical matter, however, time and resource considerations in many cases impose
limitations on what can be reasonably achieved. Arranging for the most appropriate and feasible
peer review will involve good judgment and a willingness to consider substance, time, and
resource tradeoffs. Developing a peer review plan that provides for appropriate depth, timing,
and content is an important matter for early consideration by the Decision-Maker and Peer
Review Leader. Note that use of peer input, public or stakeholder involvement does not qualify
as peer review.
The approach best suited to a specific work product will depend on the nature of the topic
and the intended final product. Generally, the more novel or complex the science or technology,
the greater the cost implications of the impending decision, and the more controversial the issue,
then the stronger the indication for a more extensive and involved peer review and for external
peer review in particular. Certain work products will clearly lend themselves to extensive
external peer review; generally these will be products with large impacts (e.g., those that support
Tier 1 and Tier 2 rulemakings). Other major work products may not need a large scale external
peer review and may utilize a less involved, less resource intensive review. The peer review of
some products may be better served with some form of internal peer review or a combination of
internal and external peer review.
The choice of peer review mechanism will depend upon the experience and assessment of
the Decision-Maker(s) dealing with peer review issues. It is important to make this choice at the
time that the work is planned (for products supporting rule makings, at the analytic blueprint
stage) so that peer review costs and time can be budgeted into the work plan. Essentially, the
level of peer review matches the impact and complexity of the major work product. For
-------
Peer Review Handbook Page 33
example, a rule under development carries considerable weight and deserves careful handling
and attention; therefore, the supporting work product deserves similar care and attention for its
peer review. Both internal and external peer review mechanisms are available, have been used in
the past, and have served to address the needs and challenges of a particular peer review
situation. Nevertheless, no single peer review mechanism is likely to work best in all situations.
Some useful rules-of-thumb include:
a) Major work products intended to support the most important decisions, or that
have special importance in their own right, ordinarily should be the subject of
external peer review. Generally, the more complex, novel and/or controversial the
product, the more the Decision-Maker should consider implementing a large-scale
peer review involving external experts.
b) Major work products that are less complex, novel, or controversial may not need
such a large-scale and external peer review. These products might be subject to
one of the less extensive, less resource-intensive review processes.
c) Group discussion with peer reviewers can be very helpful at some point in the
peer review process. On the other hand, simply soliciting individual comments is
easier, faster, and less expensive. Individual review is probably more appropriate
for peer review at the early stages of a product's development or for products with
less impact and complexity.
d) Strict time constraints, such as a court-ordered deadline, can make a less involved
or formal peer review mechanism imperative. But Decision-Makers and Peer
Review Leaders must make maximum efforts to assure that such a process is
perceived as systematic and objective.
2.4.2 What are Examples of Internal Peer Review?
a) Independent experts from within the Agency (e.g., ORD experts on non-cancer
effects of lead reviews a draft article on benchmark dose)
b) An ad hoc panel of independent experts from within the Agency (e.g., an
independent internal workgroup convened to examine the case for the
classification of a chemical as a carcinogen)
c) Technical merit review by scientists in an Agency laboratory (e.g., an initial
review of the risk assessment for a regional incinerator by Agency scientists)
-------
Page 34 Peer Review Handbook
2.4.3 What are Examples of External Peer Review?
a) Independent experts from outside the Agency (e.g., a letter review by outside
scientists)
b) An ad hoc panel of independent experts outside the Agency (e.g., a group is
convened to develop a consensus on the carcinogenicity of a particular industrial
chemical)
c) Agency-sponsored peer review workshops (e.g., a review of potential indicators of
ecosystem damage)
d) Review by an established Federal advisory committee such as the Science
Advisory Board (SAB), FIFRA Scientific Advisory Panel (SAP), ORD's Board of
Scientific Counselors, or the Clean Air Scientific Advisory Committee (e.g., a
review of a criteria document for a particular chemical risk)
e) Agency-based federal advisory committee (other than those established and
discussed in d above)
f) Agency appointed special board or commission (e.g., a review of the risk
assessment methodology prepared by the Clean Air Act Commission on Risk
Assessment) Note: The Office of General Counsel should be consulted regarding
EPA's authority to establish and finance the activities of a commission or board.
g) Interagency committee (e.g., a review of prospective research plans by the
Committee on the Environment and Natural Resources coordinated by the White
House)
h) A committee convened by another federal agency or government organization
(e.g., a review of the Dioxin Reassessment by the FtHS Committee to Coordinate
Environmentally Related Programs)
i) Review by non-governmental groups (e.g., a Society of Risk Analysis review of
cancer guidelines)
j) Review by the National Academy of Sciences (e.g., a review of the state of current
knowledge about children's health risks from pesticide exposures)
-------
Peer Review Handbook Page 35
2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?
Peer review performed under the auspices of a peer reviewed, refereed, published journal
contributes to the scientific and technical credibility of the reviewed product. Peer review of an
EPA produced scientific and technical article by a recognized refereed journal is a satisfactory
form of peer review for the purposes of publication in that journal. Peer review by that journal is
also considered adequate for reviewing the scientific credibility and validity of the findings (or
data) in that article. Products so reviewed are included on List C (Products for Which a Decision
has Been Made Not to Peer Review) (see section 1.3.2c)). No other action is needed under the
Peer Review Policy unless the conditions outlined in section 2.4.5 apply.
Prior to submitting an article to a journal for peer review, EPA employees are encouraged
to have the article internally peer reviewed (see section 1.4.9); such internal peer review is
already common procedure in certain parts of EPA. Articles may also need examination in
accordance with any organizational clearance procedures, especially when the author is
presenting him or herself as an EPA employee. For EPA employees, conflict of interest
regulations will also apply.
2.4.5 Is Peer Review Necessary when Journal Articles are Used in an Agency
Work Product?
Peer review for publication in a journal is usually performed for specific reasons for that
journal. The use of articles that have been peer reviewed by a credible journal strengthens the
scientific and technical credibility of any work product in which the article appears, but does not
eliminate the need to have the work product itself peer reviewed. For instance, journal peer
review may not cover issues and concerns that the Agency would want peer reviewed to support
an Agency action. Under these circumstances, the major scientific and/or technical work product
in which the article appears becomes a candidate for peer review. A journal article authored by
EPA employees would be used in the same manner as an article published by anyone else in a
credible, well recognized journal.
If an Agency work product is based solely on a single article that has received peer review
by a credible journal (e.g., where a model is suggested for a singular use that fits a specific
Agency need), journal peer review of the article is not a substitute for peer review of the Agency
work product. Generally, when a major work product is supported by more than one journal
article, that work product becomes a candidate for peer review. These decisions need to be
documented in the peer review record.
-------
Page 36 Peer Review Handbook
One important factor to remember with regard to the use of articles that have received
journal peer review deals with the availability of documentation from that peer review. Ideally,
EPA needs to maintain a clear, easily accessible record of the peer review to assure the credibility
and validity of the peer review (see section 2.5 for details on peer review record). Much of the
information required for the Agency peer review record may not be readily accessible from a
journal publisher or editorial board (e.g., names of the peer reviewers, the charge to them, and
their specific comments). However, in most cases this documentation from the journal peer
review is not needed to complete the Agency's peer review record for a given action — in
extreme cases, such as where litigation is involved, such documentation may be needed.
2.4.6 When and How Often Should Peer Review Occur?
The Decision-Maker and Peer Review Leader have significant discretion in deciding the
timing and the frequency of peer review. Options abound, each with merits depending on the
context and specified peer review objectives.
A single peer review event, beginning when the final draft work product becomes
available, is the approach usually taken in many situations. On the other hand, a peer review far
earlier in a project cycle could be a superior approach in some circumstances. For example, early
review might be beneficial at the stage of research design or data collection planning where the
product involves extensive primary data collection. Or, there may be substantial incremental
benefit to conducting several peer reviews along the way, particularly where a project involves
complex tasks, has decision branching points, or could be expected to produce controversial
findings. (See also section 3.4 on the selection of peer reviewers).
2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review?
Several factors impact how quickly a peer review may be needed. These include
deadlines for completion of a project, research program, or rulemaking, funding availability,
availability of quality peer reviewers, and statutory and/or court-ordered deadlines.
Peer review sometimes leads to new information and analyses. Reviewers may make
recommendations for new research that would alter the work product and thus modify the
scientific/technical basis for the action or rule it supports. For this reason, a completed peer
review is desirable before issuing any proposal for public comment. If that is not logistically
possible because of court or statutory deadlines, or other appropriate reasons, the Decision-Maker
should make every effort to complete the peer review before the close of the comment period.
Because peer review comments on such work products could be of sufficient magnitude to
-------
Peer Review Handbook Page 37
warrant a revision to the proposed action or rule, Decision-Makers should exercise diligence in
completing the peer review prior to the proposal stage whenever possible.
2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the
Peer Review?
The organization of the Decision-Maker is normally responsible for conducting the peer
review. Responsibility for conducting a peer review can be negotiable when more than one
Agency office or region or other agencies are involved. Usually, the degree of involvement by
any of the organizations and agencies and their ability to fund peer review will often determine
who has the lead for the peer review.
2.5 Creating the Peer Review Record
2.5.1 What is the Peer Review Record?
It is the formal record (file) of decision on the conduct of the peer review, the type of peer
review performed, and a summary of the outcome of that peer review. It includes sufficient
documentation for an uninvolved person to understand what actually happened and why. The
Peer Review Leader (with the program manager if there is one) creates a separate, clearly marked
peer review file section within the overall file for development of the work. Once the peer
review is completed, it is the responsibility of the Peer Review Leader to ensure that the peer
review record is filed and maintained in accordance with the organization's procedures.
2.5.2 How Can the Peer Review Record Improve the Peer Review Process?
A good peer review record allows future reference to what happened during the peer
review, and helps Decision-Makers make appropriate use of peer reviewer input. In addition, a
good record helps ensure that EPA's Peer Review Policy is followed. The Peer Review Leader is
responsible for ensuring that the peer review record for individual work products is collected and
maintained until completion of the peer review effort.
2.5.3 What Should Be in the Peer Review Record?
The peer review record should include all materials considered by the individual peer
reviewers of the peer review panel, as well as their written comments and other input. Such
materials include, at a minimum (see also section 4.3.1):
a) The draft work product submitted for peer review
-------
Page 38 Peer Review Handbook
b) Materials and information (including the charge) given to the peer reviewers
c) Written comments, information, and materials received from the peer reviewers
d) Information about the peer reviewers (such as reviewers' names, affiliations, and
identified potential conflicts and their resolution)
e) Logistical information about conduct of the peer review (such as times and
locations of meetings)
f) A memorandum or other written record responding to the peer review comments
specifying acceptance or, where thought appropriate, rebuttal and non-acceptance
g) The final work product
h) Peer review summary report (see sections 1.3.1 and 1.3.2)
When deciding if particular materials should be included in the record, the Peer Review
Leader should consider whether the materials would help reconstruct the peer review process and
results at a later time. If the materials may be helpful, they should be part of the peer review
record.
In addition to hard copies of materials, Peer Review Leaders need to maintain electronic
copies of the materials (e.g., charge) that are necessary for the annual reports compiled by the
Peer Review Coordinators. Peer Review Leaders should discuss what are those materials and
requirements for the annual report with their coordinators.
2.5.4 What Should I Do with a Peer Review Record That Pertains to a Rulemaking
Action?
The Peer Review Leader should coordinate with their program's docket office to see that
proper docketing requirements are satisfied for a peer review of a work product supporting a new
rule. The Peer Review Leader is also responsible for notifying the workgroup chair as well as the
Peer Review Coordinator (for the annual report) that a peer review is completed.
2.5.5 When Should the Peer Review Record Building Process Begin?
An early start at developing and maintaining a peer review record will help ensure the
record is complete and helpful. Ideally, the record begins when the decision to peer review a
-------
Peer Review Handbook Page 39
work product is made (and includes the written decision). The Peer Review Leader needs to
construct the peer review record from this point on — this will avoid potentially time-consuming
reconstruction at a later point. Note that the peer review record is not complete until it contains
a copy of the final work product which addresses the peer review comments.
2.5.6 What are the Differences in Record Keeping for a Review by an Individual
Compared to a Panel?
Strictly speaking, a true peer review requires more than a single reviewer. A review
conducted by one individual will rarely provide the depth of commentary required to improve the
work product. In addition, you will not receive the range of views and richness necessary to
ensure improvement.
In the case of a review panel, there will often be conflicting comments that must be
resolved. This resolution should be in the record.
2.5.7 Where Should the Peer Review Record be Kept and For How Long?
During the active conduct of the peer review, the Peer Review Leaders maintain the peer
review record with themselves until the peer review is totally completed. Minimally, the file
should be maintained until one year after the completed peer review is reported in the next
annual reporting. After that, the peer review record should be maintained for a "reasonable
period of time." Establishment and maintenance of the archive where the peer review records
ultimately reside are an organization's responsibility (i.e., not that of an individual program
manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions
will decide the appropriate level of organizational responsibility and how they will meet any
"routinely available" requirements. The peer review record may be kept with other records
relating to the overall project, as long as it is easily and separately identifiable.
There are also specific requirements regarding the use of dockets for record-keeping;
however, these are not covered in this Peer Review Handbook. The documents contained in the
peer review record should be maintained in accordance with the Agency's record keeping
retention schedule for such records. One long-term archiving mechanism may be the formal
archiving at the Federal Records Center in Suitland, MD.
-------
Page 40 Peer Review Handbook
2.6 Budget Planning
2.6.1 What Budgetary Factors Should I Consider in a Peer Review?
Resources needed to implement the Peer Review Policy need to be requested through the
usual Agency budgetary processes. The budget formulation process within the Executive Branch
is followed, after appropriation bills are passed by
Congress, by budget execution. These two processes „ . . f.,
°. J . . „ ^ . . . Peer review is part of the
provide opportunities to secure resources tor activities , , /, .
^ ^ normal cost oj doing
business.
carried out by Headquarters and Regional offices, including
peer review. The major work products for which decisions
for peer review have been made (List B candidates) need to
have adequate funding for peer review in budget requests
for the coming fiscal year. Similarly, adequate funding needs to appear in the actual approved
operating budget to ensure their conduct. For purposes of budget planning, the costs of peer
review would include the FTE cost of staff, the contract or other costs associated with the use of
outside peer reviewers, and the administrative costs of conducting a review (copying, travel
expenses, etc).
2.6.2 What Input is Needed for the Annual Budget Formulation and Budget
Execution Process?
Senior Management in Office and Regions (including Decision-Makers and budget
officers) need to be sure that budget requests include anticipated resources for peer review. Peer
review needs to be considered as a normal part of doing business. Peer review resource
considerations should also be addressed in the analytic blueprint for Agency rule-making actions.
2.7 Legal Considerations
2.7.1 Are There Legal Ramifications From the Peer Review Policy?
The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. Except where
provided otherwise by law, peer review is not a formal part of or substitute for notice and
comment rulemaking or adjudicative procedures. EPA's decision to conduct peer review in any
particular case is wholly within the Agency's discretion. Similarly, nothing in the Policy creates
a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA
-------
Peer Review Handbook Page 41
decisions rely on scientific and technical work products that have been subjected to peer review,
the remarks of peer reviewers should be included in the record for that decision.
2.7.2 Is Legal Advice Needed?
AA/RA staff and management should work regularly with individual OGC/Regional
Counsel (RC) staff assigned to Agency activities. Peer Review Leaders should initially consult
with their customary OGC/RC advisors for legal advice or referral. Headquarters attorneys have
specialties in specific areas and can be consulted as needed (e.g., FACA considerations (see
below); contractual responsibilities; ethics and potential conflicts of interest).
2.7.3 Is Peer Review Subject to the Federal Advisory Committee Act (FACA)?
It depends on how the peer review is carried out. The Federal Advisory Committee Act,
5 U.S.C. App. 2, imposes certain open meeting, balanced membership, and chartering
requirements (with OMB and GSA approval) before the Agency "establishes or utilizes" an
"advisory committee" for advice or recommendations.
Peer review carried out by formal and established (chartered) Federal advisory
committees (such as the Science Advisory Board or the FIFRA Science Advisory Panel) is
always subject to FACA requirements. However, not all peer review is carried out by
established Federal advisory committees. For example, if EPA conducts a peer review meeting
for the purpose of obtaining advice from the individual attendees and not for the purpose of
obtaining a peer review product from the group as a whole (consensus), the meeting would not be
subject to FACA.
Committees established by an outside organization (such as by a Federal contractor) to
provide the outside party with advice and recommendations are probably exempt from FACA.
You should be aware, however, that the outside party's committee could be subject to FACA if
EPA manages or controls the committee (e.g., EPA selects the members of the panel, runs the
meeting, etc.). Also remember, a contractor cannot arrange to have peer review performed on a
work product the contractor itself prepared.
EPA officials conducting peer review may wish to mitigate the potential for a FACA
challenge by seeking balanced participation at peer review meetings and allowing interested
members of the public to attend Questions concerning the applicability of FACA to peer
review meetings should be addressed to the FACA experts in the Cross-Cutting Issues
Division of OGC (Mail Code 2322 at Headquarters).
-------
Page 42 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page 43
3. CONDUCTING A PEER REVIEW
3.1 Overview Statement
The success and usefulness of any peer review depends on the quality of the draft work
product submitted for peer review, the care given to the statement of the issues or "charge," the
match between the peer review draft product and the form of peer review, the match between the
peer review draft product and the scientific/technical expertise of the reviewers, and Agency use
of peer review comments in the final product. It is not simply enough to conduct a peer review;
each of the foregoing elements requires serious attention.
3.2 Charge to the Peer Reviewers
3.2.1 What is a Charge?
As part of each peer review, the Peer Review Leader must formulate a clear, focused
charge that identifies recognized issues and invites comments or assistance. This request signals
the Agency's awareness of potential issues and its receptivity to expert recommendations. The
charge to peer reviewers usually makes two general requests. First, it focuses the review by
presenting specific questions and concerns that the
Agency expects the reviewers to address. Secondly, it
invites general comments on the entire work product.
The specific and general comments should focus
mostly on the scientific and technical merits of the
work product and, where germane, whether the
scientific/technical studies have been applied in a
sound manner. Remember, the peer review is not for the decision or action itself, but for the
underlying scientific and/or technical work product. Focused questions greatly simplify the task
of collating, analyzing and synthesizing peer review comments on a topical basis. The questions
should be specific enough to get helpful comments, but not so specific (unless very specific
points are needed to be addressed) that they preclude creative responses. Moreover, the written
responses to these questions by peer reviewers help the Agency create a peer review record. As a
general rule, the time drafting a good charge letter is well-spent and is necessary for an effective
peer review.
The time spent preparing a good
charge is well spent, and is crucial
for an effective peer review.
-------
Page 44 Peer Review Handbook
3.2.2 What are the Essential Elements of a Charge?
a) Brief overview or introduction (describe what the work product is, how it was
developed, how it will be used),
b) As needed, a brief description or listing of any background materials provided to
the peer reviewers, and
c) The issues or questions to be addressed by the peer reviewer(s).
In addition, the following should also be included as important administrative
components of a charge: a) the due date of reviewer comments and format of reviewer responses,
and b) the point of contact in case peer reviewers have questions.
3.2.3 Where Can I Get an Example of a Charge?
Appendix B contains examples of successful charges that cover a variety of issues. These
include: WTI Risk Assessment, Region 10 review of the Eastern Columbia Plateau sole source
aquifer, letter reviews for IRIS (Integrated Risk Information System) entries, and several
examples of charges for SAB reviews. Appendix C provides guidance for obtaining Science
Advisory Board (SAB) services.
3.3 Time Line
3.3.1 What are the Factors in Scheduling a Peer Review?
The peer review schedule is a critical feature of the process. The schedule must take into
account the availability of a quality draft work product, availability of appropriate experts, time
available for using peer review comments, deadlines for the final work product, and logistical
aspects of the peer review (e.g., contracting procedures).
The schedule for peer review should take into account the overall rulemaking (or other
decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of
the peer review should be included in the development of the analytic blueprint. Peer review
sometimes leads to new information and analyses, or recommendations for new research that
would alter the work product and thus modify the scientific/technical basis for the action. For
this reason, it is usually advisable to complete the peer review before taking public comment, or
at least before the close of the public comment period.
-------
Peer Review Handbook Page 45
3.4 Selection of Peer Reviewers
3.4.1 What are Considerations for Selecting Peer Reviewers?
Selection of independent peer reviewers is not a trivial task, and it is crucial to an
effective peer review. It is important that peer reviewers be selected for independence and
scientific/technical expertise. However, the very need to have experienced individuals on a peer
review, along with the desire to have appropriate technical balance and representation, can mean
that the selection of potential peer reviewers often comes from those who are considered as
having a potential bias. To reduce the concern that a potential panel may have unnecessary bias,
it may be useful to obtain an informal review of the expertise and balance of potential peer
reviewers from others in your organization, from OGC or even from outside groups. Sometimes
selecting individuals who have served in a variety of organizations rather than a single one for an
extended period, provides expertise with diverse perspective. The emphasis on independence
and expertise applies equally to government experts and experts from the larger scientific
community.
Some peer reviews can be conducted with two or three reviewers; others involve panels
of peer reviewers. In either case, each peer reviewer should have recognized technical expertise
that bears on the subject matter under discussion. In instances where there is more than one peer
reviewer, the peer reviewers of a work product should represent a balanced range of technically
legitimate points of view. In addition, cultural diversity and "address" (e.g., industrial, academic,
or environmental community) are other factors that can play a role in selecting peer reviewers.
3.4.2 Where Do I Find Peer Reviewers?
Recommendations for potential peer reviewers can be identified from a number of
organizations. These include external groups such as the affected party(ies), special interest
groups, public interest groups, environmental groups, professional societies, trade or business
associations, state organizations or agencies, Native American Tribes, colleges and universities,
the National Research Council, and other Federal agencies with an involvement in or familiarity
with the issue. Agency associated groups include the staff of the Science Advisory Board (SAB)
or the Scientific Advisory Panel (SAP), and relevant scientific and technical experts from
Program or Regional offices.
In certain circumstances, existing peer review organizations such as the SAB or SAP may
be used to conduct a peer review. These groups establish their own criteria for accepting work
and coordination must be made directly with them (see Appendix C for SAB procedures). Both
SAB and SAP conduct formal, public, external peer reviews.
-------
Page 46 Peer Review Handbook
Occasionally, a member of the scientific community will offer his/her services for peer
review during an ongoing peer review. These offers may be at no cost or based on an expectation
that reimbursement will be made. Disposition of these unsolicited offers will be handled on a
case by case basis by the Peer Review Leader, and as necessary, in consultation with the Peer
Review Coordinator, the Office of General Counsel (OGC), and appropriate Decision-Makers.
If you use a contract mechanism to conduct a peer review, the contractor may have its
own pool of scientific and technical experts for peer review. With contractors, EPA can provide
information on potential sources of peer reviewers for conducting a peer review if such a listing
were prepared in alphabetical order, and contained an Agency disclaimer that EPA doesn't
recommend any particular individual or firm on the list. Furthermore, when utilizing a contract
mechanism to conduct peer review, EPA is not permitted to direct the prime contractor to a
specific sub-contractor or peer reviewer nor is EPA permitted to directly interact with the peer
reviewer (sub-contractor). All interaction must be with the prime contractor's designated
representative and not the contractor's staff.
3.4.3 Are External or Internal Peer Reviewers Preferred?
External peer reviewers are generally preferred, particularly for most final work products.
For some work products, like those reviewed at interim steps, either external or internal peer
review may be appropriate. Selection of internal peer reviewers should be based upon technical
expertise, available time and "address" — that is, they should not come from the immediate office
or group producing the product or have any other connection with the product or document being
peer reviewed. External peer reviewers should be selected based upon technical expertise as
well, however, care must be taken not to use individuals who have been involved in the
development of the work product. (See also sections 1.4.6 to 1.4.9).
3.4.4 What is Important in the Mix of a Peer Review Panel?
A peer review panel or group can number from just a few individuals to ten or more,
depending on the issue, the time and resources available, and the broad spectrum of expertise
required to treat the range of issues/questions in the charge. Naturally, experts whose
understanding of the specific technical area(s) being evaluated are critical; nevertheless, it is also
important to include a broad enough spectrum of other related experts to completely evaluate the
relevant impacts on other less obvious concerns (i.e., to comment not only if the job is being
done right, but also whether the right job is being done). For example, for health related peer
reviews, experts in such fields as ecology and economics may provide very useful insights.
Although persons who are familiar and have a substantial reputation in the field are often called
-------
Peer Review Handbook Page 47
upon repeatedly to be reviewers, it is important to keep a balance with new people who bring
fresh perspectives to the review of a work product.
There is usually a continuum of views on any issue. To the extent possible or practicable,
selected experts should have technically legitimate points of views that fall to either side of the
central position along the continuum, but not too far to either extreme. This will help maintain a
balanced viewpoint, while allowing all views to be expressed and discussed. A balanced panel
will allow consensus building (if consensus is the object of a particular peer review; if not, it
allows a spectrum of reviews for the Agency to evaluate). As a general rule, experts who have
made public pronouncements on an issue (e.g., those who have clearly "taken sides") may have
difficulty in reaching consensus positions and should be avoided.
3.4.5 What is a Conflict of Interest?
Conflict of interest is a situation in which a person is unable or potentially unable to
render impartial assistance or advice to the Agency, or the person's objectivity in performing the
work is or might be otherwise impaired, or a person has an unfair competitive advantage.
Generally, a conflict of interest or a perceived conflict of interest arises when the person is
affected by his/her private interests, or when he/she or his/her associates would derive benefit
from incorporation of their point of view in an Agency product. Whenever there are questions
about conflicts of interest, you should contact the appropriate official in OGC for clarification.
3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of
Conflicts of Interest?
Before finalizing the selection of reviewers, the Peer Review Leader should ask potential
reviewers if they have any real or perceived conflicts of interest. One way of identifying
conflicts is to ask potential reviewers about current and prior work, and prior clients that might
create conflicts or the appearance thereof. This information becomes part of the peer review
record.
Care must be taken to reduce the possibility for real or apparent conflicts of interest
between the reviewers and the work product under review. Various tools are available to identify
and limit conflicts of interest (e.g., attention to the employment, financial, and professional
affiliations of the participants; filing Confidential Financial Disclosure Forms (SF-450) in the
case of members of Federal Advisory Committees; exploring directly the issue with each of the
participants before the review process takes place; and disclosing publicly at the beginning of
meetings any previous involvement with the issue). Established peer review groups such as the
Science Advisory Board provide useful models for addressing balance and conflict-of-interest
-------
Page 48 Peer Review Handbook
issues. Assistance in determining legal conflicts of interest and in providing an appropriate
response can be obtained through the Office of the General Counsel. In addition, additional
advice can be obtained from the Designated Agency Ethics Officials (DAEO).
Of course, conflicts do not necessarily arise merely because a peer reviewer knows
something about the subject matter. In fact, experts with a stake in the outcome — and therefore
a potential conflict — may be some of the most knowledgeable and up-to-date experts because
they have concrete reasons to maintain their expertise. Such experts could be used provided the
potential conflicts of interest are disclosed and the peer review panel or group being used as a
whole is balanced. In some cases, however, the conflict may be so direct and substantial as to
rule out a particular expert, for instance, a potential peer reviewer who may have a client or
employer with a direct financial stake in the matter under review (e.g., a manufacturer of a
chemical under review). However, review of a general methodology that applies to numerous
chemicals would not necessarily raise such a concern. (Note: language that reflects these
concerns should be made part of contracts/statements of work (SOW) or purchase orders (PO) —
see section 3.6 and Appendix D).
A Peer Review Leader may want to adopt measures that will prevent creation of conflicts
as the peer review is underway. Such measures might include clauses in a contract or purchase
order that requires reviewers to receive advance approval on future work, or places limits on such
work, while they are performing the current peer review. The intent behind such measures is to
avoid having the peer reviewer gain an unfair future advantage as a result of their participation on
a peer review process. Note that at some level these types of measures will discourage experts
from serving as peer reviewers. (See section 3.6.5 for further information dealing with contracts
and suggestions for appropriate management controls).
3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews?
There may be instances where parties external to EPA will want to conduct and/or pay for
a peer review on a particular work product (presumably their own work product or one they are
closely interested in, or they wouldn't be interested in expending resources). This may look
benign at first blush, but is a very complex and sensitive situation that can raise significant
concerns for perceived and/or actual conflicts of interest for interested parties "paying" for a peer
review of their own work product. While the Agency cannot prevent external parties from
conducting and paying for a peer review, it is desireable that any such peer review meets the
intent of the Agency's Peer Review Policy and adheres to the principles and guidance in this
Handbook. If the external party submits their work product and accompanying peer review, the
materials will be treated by the Agency as anything else submitted for the Agency's evaluation.
-------
Peer Review Handbook Page 49
We will evaluate the work product and the peer review for scientific credibility and validity
before making any decisions based on the materials.
3.4.8 Are There Constraints to Selecting Peer Reviewers?
Sometimes the need for a peer review is accelerated due to a court-ordered deadline or
other time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain
external peer reviewers in time to conduct a full external peer review. It may even be impossible
to conduct a small scale internal peer review using just a few individuals. Mechanisms for
identifying and using a small number of peer reviewers should be developed so that quick,
effective peer review can be included for even the most rapidly moving products.
Another possible constraint involves confidential business information (CBI). To
evaluate certain Agency-generated studies properly, some peer reviewers may need access to
CBI. However, unless the reviewers are Federal employees with CBI clearance, the Agency
doesn't have the independent authority to disclose CBI to them. Therefore, whenever
contemplating the use of outside peer reviewers, Agency staff should determine whether the
reviewers will need access to CBI. If they don't have CBI clearance, the Office of the General
Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to
disclose the information to peer reviewers.
Offices need to be aware of the requirements of the Federal Advisory Committee Act
(FACA) when establishing peer review mechanisms (see section 2.7.3). Federal advisory
committees that are subject to chartering by the General Services Administration must hold
meetings that are open to the public, and have balanced membership requirements. The Office of
the General Counsel should be consulted regarding the applicability of FACA to peer review
panels.
3.5 Materials for Peer Reviewers
3.5.1 What Instructions Do You Give Peer Reviewers?
The Peer Review Leader is responsible for ensuring that peer reviewers understand and
comply with these responsibilities (see section 3.6 if a contract is involved):
a) Advise the Agency of any real or perceived conflicts-of-interest
b) Provide written comments in specified format by the specified deadline that are
responsive to the charge
-------
Page 50 Peer Review Handbook
c) Comply with the request for not disclosing draft work products to the public
3.5.2 What Materials Should be Sent to Peer Reviewers?
For a peer review to be successful, peer reviewers should receive several documents at
the beginning of the process. Typically, the most important among these documents are the
charge letter and the current work product. The charge letter describes what the peer reviewers
are being asked to do, and should serve to focus and structure the review. The work product is,
of course, the material being subject to peer review.
Remember, no documents should be provided directly to a potential peer reviewer when
that reviewer is going to be working under a contract or purchase order. In the case of a contract,
the Agency provides the work product with associated background material to be peer reviewed
to the prime contractor who in turn distributes these documents to the peer reviewers. In the case
of a purchase order, the "charge or statement of work" must be part of the PO (purchase order)
and the provision of any documents needs to be coordinated with the contracting officer handling
the order.
a) Essential documentation for each peer reviewer includes:
1) A current copy of the work product to be peer reviewed with associated
background material. The work product needs to be of the best possible
scientific/technical quality to ensure an adequate and useful peer review.
2) A clear charge or statement of work seeking informed comment on
identified issues to properly focus the efforts of the peer reviewers and
ensure that their individual efforts can be compared or contrasted.
3) Some information concerning the process that you use for the peer review,
including the due date of reviewer comments, the format of those
responses, and a point of contact in case the peer reviewer has questions.
Responses should be written and submitted to the Peer Review Leader by
an agreed upon deadline. In certain rare cases, oral commentary may be
sufficient. However, in such cases, a follow-up written response for the
record is required.
4) In some cases, Agency materials being peer reviewed will be available to
the public, even if they are marked as drafts. For example, all materials
reviewed by the SAB are available. Agency managers may also decide
-------
Peer Review Handbook Page 51
that a broad accessibility has benefits for the Agency. In other cases,
confidentiality needs to be maintained. In these cases, each peer reviewer
must be informed of the need for confidentiality with regard to the release
of Agency products that are stamped as "DRAFT" or "DRAFT - Do Not
Cite, Quote, or Release." Premature release of draft Agency products,
views, or positions is inappropriate and can be damaging to the credibility
of the Agency or the peer reviewer. While not necessarily having legal
consequences, such language will be included in the charge to the peer
reviewers. Other mechanisms to use in discouraging premature release
include a disclaimer that appears in a separate section at the front of the
document and creating the document with watermarks clearly delineating
DRAFT status (or a header or footer that states DRAFT status) on every
page. In addition, in any solicitation for peer reviewers, the necessity for
confidentiality and the non-release of materials shall be emphasized.
b) Useful, but not critical materials that may be sent to peer reviewers include:
1) The name, address, and phone and fax numbers, and/or Internet address of
each peer reviewer working on the specific review
2) A bibliography and/or any particularly relevant scientific articles from the
literature
3) A work product that has line numbering added in the margin for ease in
providing and referencing comments
c) Peer Reviewers should be given what is needed to complete their task — they
should not be overburdened with excess material.
3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?
The Peer Review Leader normally has administrative contacts with the reviewers during
the development and conduct of the peer review. In some cases (e.g., SAB peer review), peer
reviewers may also receive a briefing on the product to be peer reviewed. Otherwise, the Peer
Review Leader and other EPA staff should not contact the reviewers during the course of the
review. Such contact can lead to perceived conflicts or inappropriate direction that could
compromise the independence of the review.
-------
Page 52 Peer Review Handbook
If peer review is conducted under a contract mechanism, EPA must limit direct contact to
the prime contractor's designated representative and not have general contact and direction to the
contractor's staffer peer reviewers (sub-contractors). Note, when a peer review is conducted
under a contract, there are constraints where EPA staff are prohibited from contacting peer
reviewers to avoid personal services arrangements. Personal services contracts exist when the
nature of the relationship between the contractor and the EPA can be characterized as an
employer - employee relationship. Any communications with peer reviewers must be only from
the Peer Review Leader or contracting officer.
3.6 Peer Review Services
A range of peer review services are available to the Agency including internal, external
(gratuitous services, contracts, purchase order), and Special Government Employee (SGE)
mechanisms. The mechanism selected is generally based on the nature of the scientific or
technical work product.
3.6.1 What are Gratuitous Services for Peer Review?
The provision of peer review products or services to EPA without compensation are
provided as so-called "gratuitous" services. If a person or organization wishes to perform peer
review services for EPA without compensation, the program must ask them to sign an agreement
whereby the person or organization agrees to provide the prescribed peer review services as
gratuitous services, with no expectation of receiving compensation for these services from EPA.
An agreement must be executed because the Antideficiency Act (31 U.S.C. §1342) prohibits the
Agency's acceptance of "voluntary" services. "Voluntary" services are provided to EPA without
an agreement in advance that such services are provided at no cost to EPA. Note that persons
cannot waive compensation (i.e., agree to provide gratuitous services) for which there is a
statutory right to payment, unless a law permits the waiver.
3.6.2 Can I Use a Contract to Obtain Peer Review Services?
The Agency may contract for peer review services. The contract may be written solely
for peer reviews or be included as one of several specifically described interrelated tasks in a
contract that requires the contractor to provide more than just peer review services.
For assistance in preparing the necessary pre-award documents, program officials should
consult The Cookbook: How to Get Contracts Awarded in EPA and Chapter 2 of the Contracts
Management Manual (CMM). The Cookbook and Chapter 2 are available on EPA's Intranet
-------
Peer Review Handbook Page 53
(http://www.intranet.epa.gov/oamintra) and on the Agency LAN Services (Administration/
Management/OAM Procurement Policy Information).
3.6.3 How Do I Write a Statement of Work for Contracts?
The statement of work (SOW) must clearly specify that the contractor is responsible for
preparing peer review evaluations and set forth guidelines for the peer review of scientific or
technical documents. The contractor may perform the peer review with in-house staff,
subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness
and defensibility must be developed by the program office and made part of the contract. The
contractor would then ensure that the peer reviews adhered to the guidelines.
The SOW cannot simply define the role of the prime contractor as arranging for the
services of others to perform peer reviews and logistics for meetings. Unless the prime
contractor is clearly tasked with responsibility for performing peer reviews, individual peer
reviewers' fees and associated travel expenses are not payable under the contract.
The EPA may pay for the reviewer's comments or evaluation, and also for attendance at a
meeting with the Agency and other reviewers to discuss the results of the peer review. If the
SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the
Agency and other peer reviewers to discuss the results of the peer review, payment is appropriate.
The peer reviewer's attendance at the meeting would then be part of contract performance.
Example statements of work are found in Appendix D.
3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive Activities?
Contracts that provide services that support or improve Agency decision-making or
policy development are subject to special management controls. These services include services
acquired from non-Governmental sources by contract to support or improve Agency policy
development, decision-making, management, and administration, or research and development
activities. See Federal Acquisition Regulation (FAR) 37.201 for a more specific definition of
AAS. Such services may take the form of information, advice, opinions, alternatives,
conclusions, recommendations, training, and direct assistance. For additional information on
advisory and assistance services and sensitive activities, program officials should review Chapter
2oftheCMM.
New contracts for these services require management approvals prior to issuance of the
solicitation. For the thresholds that have been established for approval of these justifications, see
-------
Page 54 Peer Review Handbook
Figure 5, Item B Management Approvals set forth after Chapter 2 of the CMM (for current
approval levels).
3.6.5 What are Some Management Controls for Contracts?
Contracting for peer review services is permitted. However, because of the potential for
improper use of these contracts, special management controls are required.
a) Inherently Governmental Functions (IGFs) -- OFPP Policy Letter 92-1, dated
September 23, 1992, describes (1) functions that are inherently governmental and
must be performed only by Government employees and (2) functions that may be
contracted, but so closely support Government employees in their performance of
IGFs that the contract terms and performance require close scrutiny by Federal
officials. Federal Acquisition Regulation (FAR) coverage of inherently
governmental functions is at FAR Subpart 7.5.
Peer reviews represent only a contractor's recommendations, advice or analysis of
a document. Agency officials must make the official Agency decision regarding
acceptability and/or quality of the document. To ensure that Agency officials are
not improperly influenced by recommendations in the peer review, management
controls must be included in the contract. One possible control would be to
require the peer reviewers to submit with their evaluations or comments a
description of the procedures used to arrive at their recommendations; a summary
of their findings; a list of sources relied upon; and make clear and substantiate the
methods and considerations upon which their recommendations are based. To the
extent possible, the contract should set forth any guidelines or criteria for
performance of the peer review. Agency officials should document their
evaluations of the quality and validity of the peer review.
b) Conflict of Interest (COI) — Another important factor is that the objectivity of
the peer review should not be improperly influenced or undermined by the
contractor performing the review. To identify and avoid or mitigate actual or
potential COI, the contract should include controls. Such controls might require
the contractor to report on prior and current work, and prior clients that might
create COI. Other controls might include Agency review and placing limits or
advance approval on future work. There should also be procedures implemented
to assure that the contractor does not gain an unfair advantage in future
requirements as a result of their performance of peer reviews. Program officials
should consult the Contracting Officer (CO) for special contract clauses.
-------
Peer Review Handbook Page 55
FAR coverage of conflicts of interest is at FAR Subpart 9.5. The EPA
Acquisition Regulations (EPAAR) at 48 CFR Subpart 1509.5 generally mandates
conflicts of interest solicitation provisions and contract clauses, but makes them
optional for procurements accomplished through the use of simplified acquisition
procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507-l(b)(3) and 48
CFR 1552.209-70, -71 & -72 as additional resources.
Contract for peer review services: An EPA contracting officer will include
conflicts of interest solicitation provisions and contract clauses as a matter of
course without involvement by the EPA project officer, if the peer review services
are not obtained pursuant to the simplified acquisition procedures in FAR Part 13.
If the peer review services are subcontracted pursuant to a prime contract, then the
prime contractor is ordinarily required to include a conflicts of interest clause
substantially similar to the conflicts of interest clause in the primary contract in its
subcontract to the peer reviewer.
Although the EPA contracting officer and/or prime contractor has the primary
responsibility to include the required conflicts of interest provisions/clauses, the
EPA project officer may nevertheless wish to:
1) Highlight the conflict of interest requirements in the Scope of Work for the
procurement of the peer review services
2) Develop a specific conflict of interest clause regarding the peer review at
issue as a substitute or in addition to the standard conflicts of interest
clause
3) Review the solicitation/contract to make sure that the required conflicts of
interest clause has been included
Simplified acquisitions of peer review services: Although conflict of interests
requirements are optional for simplified acquisitions, they are nevertheless a good
idea. Accordingly, an EPA project officer obtaining peer review services with
simplified acquisition procedures should request the purchasing agent/contracting
officer to include a conflict of interest solicitation provision and contract clause in
the purchase order.
-------
Page 56 Peer Review Handbook
c) Confidential Business Information (CBI)/Privacy Act Protected Information
and Other Sensitive Information — When peer reviewers are not employees of
the United States Government, it is unlikely that the EPA will have authority to
give reviewers access to confidential business information in the absence of
consent for such disclosure by the CBI submitter. Therefore, all documents
provided to non-Federal reviewers must be screened for information claimed as
CBI. Even where business information has not been explicitly claimed as CBI, if
it is of a kind where the submitter might be expected to object to its release, prior
to release the submitter must be asked whether it wishes to assert a claim, unless
the submitter has previously been informed that failure to assert a CBI claim may
result in disclosure without notice. Language is included in the contract to clearly
identify any required procedures or processes prior to release of any protected
information, including any requirements for confidentiality agreements, as well as
limits on use and disclosure of the data by contractor personnel.
d) Personal services — Under contracts, the EPA may not engage the peer reviewers
in any improper personal services relationships, i.e., an arrangement under which
contractor personnel are subject to relatively continuous supervision and direct
control by an Agency official or employee. These relationships are characterized
as one where the contractor employee interacts with the Agency in a manner
similar to that of a Federal employee.
To avoid these improper relationships, program officials should write well-
defined SOWs. The SOWs should set forth the requirements in detail for work to
be performed independently, including the manner in which it will be evaluated.
The SOW must set forth what work is to be performed not how the work is to be
performed. Technical direction may be used to clarify ambiguous technical
requirements to ensure efficient and effective contractor performance, and is not
considered supervision or assignment of tasks. For additional information,
program officials should consult EPA Order 1901.1 A. Use of Contractor Services
to Avoid Improper Contracting Relationships and FAR Subpart 37.1
3.6.6 Can the Agency Identify and/or Select Peer Reviewers Through a Contract?
Program officials cannot interfere in a contractor's authority and responsibility to perform
work by "selecting" who will perform the peer review (doing so may invoke FACA — see section
2.7.3). The Federal Acquisition Regulation (FAR) governs the Contracting Officer's (CO's) and
program officials' relationship with the contractor.
-------
Peer Review Handbook Page 57
However, the CO does have the ability to review and consent to subcontractors and
consultants. The contract can also specify which individuals are key personnel and include peer
reviewers. The contract will set forth the procedures governing the use and replacement of key
personnel. Further, the contract can require workplans for approval by the Agency, wherein the
contractor will propose the peer reviewers it is considering for selection.
Program officials should identify the qualifications required to perform the review work
and the criteria for technical acceptability. The EPA may identify a pool of qualified
subcontractors and consultants to the prime contractor (listed in alphabetical order), but cannot
direct the use of any particular subcontractor or consultant.
3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers?
In some instances, peer reviewers can be obtained via simplified acquisition procedures.
The acquisition of supplies or non-personal services from the open market and on a sole source
basis when the aggregate amount involved in any one transaction does not exceed $100,000
constitutes a simplified acquisition (FAR Part 13). The same considerations in the section 3.6.5
discussion on IGFs, COI, access to CBI, and personal services apply to simplified acquisitions.
Normally, the Government issues a purchase order directly to the individual peer reviewer,
instead of to a prime contractor who may subcontract for performance of the peer review.
a) Approvals — All small purchases for peer reviews are considered AAS. See
Chapter 2 of the Contract Management Manual (CMM) for the appropriate
approval levels for AAS.
b) Competition — The Federal Acquisition Regulation (FAR) requires competition
for purchases in excess of $2,500. Purchases for more than $2,500 and not more
than $100,000 are to be made only from small businesses unless the Contracting
Officer is not able to obtain two or more offers from small businesses that are
competitive in terms of market price, quality, and delivery. Only one source need
be solicited if the Contracting Officer determines that only one source is
reasonably available. Contracting Officers are encouraged to use best value.
c) Procurement Requests — Program Officers should include the following in all
PRs for the purchase of peer reviews:
1) A fixed-price amount at or below the simplified acquisition threshold
-------
Page 58 Peer Review Handbook
2) A detailed description of the requested services, inclusive of:
(a) Total quantity per line item
(b) Estimated unit price per line item
(c) Total cost per line item
(d) Specific deliverables for each line item
(e) Total cost of the purchase request
3) Reference FAR Subpart 3.6 and Environmental Protection Agency
Acquisition Regulation (EPAAR) Subpart 1503.601 regarding sources
from Government employees or organizations owned and controlled by
them.
4) Provide sources from small businesses, if available.
5) If the request is a sole source purchase, justification must be provided in
accordance with the EPAAR Subpart 1513.170-1.
3.6.8 How is Travel Handled with Contracts or Purchase Orders?
Funds obligated on a contract or purchase order are available to pay for the costs of
producing the peer review including the travel costs and fee of the peer reviewer.
The EPA acquires peer reviews through simplified acquisitions issued directly to peer
reviewers or through contracts with companies, which provide the peer review services. By
issuing a purchase order or awarding a contract for peer review services, the EPA may pay not
only for the peer review services/comments, but also for attendance at a meeting with
the Agency and other reviewers to discuss comments. The scope of work of the contract must
require the contractor or individual peer reviewer as appropriate to provide peer review services
and indicate whether the contractor or peer reviewer will be required to discuss a specific peer
review work product with the Agency and/or with other peer reviewers. Attendance at a meeting
to discuss a peer review work product would then be part of the contract's performance. Thus,
the contract may serve as the mechanism to pay for peer review services and associated travel
expenses to provide comments to the EPA.
-------
Peer Review Handbook Page 59
3.6.9 How is Travel Handled with Special Government Employees?
The term Special Government Employee (SGE) is defined in 18 U.S.C. 202(a) as an
officer or employee of an agency who performs temporary duties, with or without compensation,
for not more than 130 days in a period of 365 days, either on a full-time or intermittent basis.
Travel and per diem expenses of experts hired as SGEs for peer review may only be paid
through the issuance of invitational travel orders (5 U.S.C. §5703). These invitational travel and
per diem expenses should be charged to an appropriate EPA travel account.
Members of the SAB, SAP, and other FACA advisory committees are often brought on
board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory
committee members or other SGEs through a contract.
-------
Page 60 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page 61
4. COMPLETING A PEER REVIEW
4.1 Overview
Performance of the formal peer review is not the final stage in the development of the
work product. Rather, it is an important stage in developing the work product, with the final
work product representing the true end of the peer review. As a result, the peer review process
closes with three major activities: evaluating comments and recommendations, utilizing peer
review comments for completing the final work product, and organizing and maintaining a
record of the peer review.
Careful attention to all of these elements, singly and together, assures a credible peer
review process. Conversely, inattention can nullify the peer review attempt. A well-planned
peer review applied to a reasonable quality starting work product, followed by responsible,
visible utilization of peer review suggestions in
the final product assures a credible product for
use in Agency decision-making.
The peer review is not completed until
the peer review comments are
incorporated into the final work
product
The peer review is not completed until the
peer review comments are incorporated into the
final work product, or reasons are stated why such
comments are not to be incorporated. Once the
peer review is completed, the Peer Review Coordinator moves the peer reviewed scientific
and/or technical work product from List B (or List C in some instances) to List A in their office's
annual report.
4.2 Final Work Product
4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product?
The Peer Review Leader must carefully evaluate and analyze all peer review comments
and recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers
simplifies organizing and analyzing comments. Also, any other issues that are raised need to be
identified and evaluated.
The validity and objectivity of the comments need to be evaluated. Analyses may include
consultation with other experts/staff within the Office and/or Agency. Adequate documentation
is needed to show that comments are accepted or rejected — the documentation can be brief, but
-------
Page 62 Peer Review Handbook
must address the legitimate, valid comments. It is the responsibility of the Peer Review Leader
to obtain management approval of the approach to addressing the peer review comments.
Comments that have significant impact on time, budgetary, and/or resource requirements
are particularly important and need to be evaluated in consultation with management. These
comments may lead to allocation of additional resources and a revised schedule for the
completion of the work product.
4.2.2 What Actions are Potentially Forthcoming from Peer Review?
Peer review comments and recommendations may entail significant impacts on the
planned project schedule, budget, or other resource requirements. Management decisions related
to revisions in one or more of these areas may be appropriate.
The substantive issues or concerns expressed by peer reviewers may suggest that wider
scientific and technical consultation is needed to ensure the adequacy of the work product.
The peer review comments and recommendations on a final product may provide a basis
for bringing the associated project to closure.
4.2.3 Can the Identity of Peer Reviewers be Kept Anonymous?
If the matter has gone to litigation, the litigating parties can discover the names of anyone
who contributed to a Federal product, including peer reviewers. Therefore, it is not possible to
totally shield peer reviewers. In addition, it may be difficult to shield the names of the peer
reviewers when the Agency is responding to a Freedom of Information request. However, in the
ordinary course of events, you can often offer comments received without attributing the
comments to a specific reviewer.
If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review
Leader needs to inform the peer reviewer of the above possible eventualities. The Agency will in
the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and
their comments for public consumption, but it is recognized in many instances, for example open
public meetings and the above circumstances, this can't be assured. Remember, the Agency is
committed to working "as if in a fishbowl" and most of its activities are transparent to the public
(except where confidential business information is concerned). It is recognized that this may be a
deterrent to possible peer reviewers, but this is a reality that has to be understood.
-------
Peer Review Handbook Page 63
4.3 Completing the Peer Review Record
4.3.1 How Do I Complete the Peer Review Record?
Once the Peer Review Leader has completed the peer review, the peer review record is
brought up to date and then archived according to that organization's procedure (see section
2.5.7). The peer review record must be indexed and maintained in an organization's archive
(repository). The location of the peer review record needs to be readily identifiable so interested
parties can locate and obtain materials easily and quickly. The peer review record should be
placed in any associated established public docket, if required, in addition to the organizational
archive. As a courtesy, a copy of the revised work product may be sent to the peer reviewers for
information.
The Peer Review Leader will collect the following materials for the peer review record
and submit for archiving; including at least (see also section 2.5.3):
a) The draft work product submitted for peer review
b) Materials and information (including the charge) given to the peer reviewers
c) Written comments, information, and materials received from the peer reviewers
d) Information about the peer reviewers (such as reviewers' names, affiliations, and
identified potential conflicts and their resolution)
e) Logistical information about conduct of the peer review (such as times and
locations of meetings)
f) A memorandum or other written record responding to the peer review comments
specifying acceptance or, where thought appropriate, rebuttal and non-acceptance
g) The final work product
h) Peer review summary report (see sections 1.3.1 and 1.3.2)
4.3.2 Where Should the Peer Review Records be Kept, and for How Long?
During the active conduct of the peer review, the Peer Review Leaders maintain the peer
review record with themselves until the peer review is totally completed. Minimally, the file
-------
Page 64 Peer Review Handbook
should be maintained until one year after the completed peer review is reported in the next
annual reporting. After that, the peer review record should be maintained for a "reasonable
period of time." Establishment and maintenance of the archive where the peer review records
ultimately reside are an organization's responsibility (i.e., not that of an individual program
manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions
will decide the appropriate level of organizational responsibility and how they will meet the
"routinely available" requirement. The peer review record may be kept with other records
relating to the overall project, as long as it is easily and separately identifiable. The peer review
record should be maintained in accordance with the Agency's record keeping schedule for such
records. One long term archiving mechanism may be the formal archiving at the Federal Records
Center in Suitland, MD. (Note: This is the same question as section 2.5.7, but applies in this
chapter as well).
-------
Peer Review Handbook
Page 65
SUBJECT INDEX
This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers
where they are found.
-A-
-B-
-C-
-D-
Administrative Procedures Act (12)
Administrator (9, 17)
Advisory and Assistance Services (53)
Analytic blueprints (13, 32, 40)
Annual call letter (15)
Annual reporting requirements (14, 18)
Anonymity of peer reviewers (62)
Archiving (39, 63)
Assistant Administrators (17)
Balancing of peer reviewers (21)
Board of Scientific Counselors (BOSC) (34)
Budget (18, 32, 40)
Categories of work products (15)
Charge (19, 37, 43)
Chemical action reports (16)
Clean Air Scientific Advisory Committee
(34)
Coding system (16)
Compensation (52)
Competition (57)
Completing a peer review (4, 61)
Conducting a peer review (3, 43)
Conference proceedings (25)
Confidential business information (CBI) (49,
56)
Confidential Financial Disclosure (47)
Confidentiality (51)
Conflicts of interest (21, 28, 47, 54)
Contracting officer (50, 52, 54, 56)
Contracts (28, 55)
Contracts Management Manual (CMM) (52,
53)
Controversy (27, 32)
Cooperative agreements (28)
Court ordered deadlines (31, 36, 49)
Decision-Maker (14, 16, 17, 26, 32)
Deputy Administrator (14, 17, 18)
-E-
-F-
-G-
-I-
-J-
-L-
-M-
Designated Agency Ethics Official (48)
Division Directors (17)
Docket (38, 39, 63)
Economic work products (27)
Electronic records (14, 16, 38)
Environmental Impact Statement (EIS) (29)
Environmental regulatory models (29)
External peer review (33, 34, 46)
Federal Acquisition Regulation (FAR) (53)
Federal Advisory Committee Act (FACA)
(41, 49)
Final work product (61)
Grants (28)
Gratuitous services (52)
Independent peer reviewer (10, 12, 21, 45)
Inherently governmental functions (IGFs)
(54)
Interacting with peer reviewers (51)
Interagency agreement (28)
Internal peer review (33)
Internal peer reviewer (21)
Journal article (16, 35)
Legal considerations (40)
Line-managers (17)
List A (14)
List B (15, 26)
List C (15)
Litigation (15, 62)
Major impact (27, 32)
Major scientific and technical work product
(5, 9, 15, 25, 26, 30)
Managers planning checklist for peer review
(5)
-------
Page 66
Peer Review Handbook
-N-
-O-
-P-
-R-
-S-
Materials for peer reviewers (49)
Mechanisms for peer review (32)
National Academy of Sciences (34)
NEPA Products (29)
Non-major scientific and technical work
products (15, 30, 31)
Novelty (27, 32)
NPDES permits (16)
Office Directors (17)
Office of Research and Development (ORD)
(14, 16, 20, 23)
Open meeting (41)
Peer consultation (11)
Peer input (11)
Peer involvement (10)
Peer review (9, 10)
Peer Review Advisory Group (PRAG) (17)
Peer review comments (19, 38, 61)
Peer Review Coordinator (14, 17, 19, 22)
Peer Review Leader (18, 18, 32, 39, 48, 49,
61,63)
Peer review mechanism (5, 32)
Peer review panel (22, 45, 46)
Peer review record (5, 20, 36, 37, 63)
Peer review schedule (18)
Peer review services (52)
Peer review summary report (14, 15, 18, 38)
Peer reviewers (21, 21, 38, 49, 56)
Personal services (56)
Planning a peer review (2, 25)
Premanufacturing notices (PMNs) (16)
Procurement requests (57)
Project manager (13, 18, 30)
Public comment (12)
RCRA permits (16)
Record of Decision (ROD) (25)
Regional Administrators (17)
Regulatory negotiations (14)
Rulemaking (13, 27, 38)
Science Advisory Board (SAB) (34, 45)
Science Policy Council (SPC) (14, 17, 20,
22)
Scientific Advisory Panel (SAP) (34, 45)
-T-
-V-
Scientific and technical work product (25)
Selection of peer reviewers (45)
Sensitive activities (53)
Signatures (14, 16)
Simplified acquisitions of peer review
services (55, 57)
Site specific decisions (28)
Special Government Employee (SGE) (59)
Stakeholder involvement (12)
Statement of work for contracts (53)
Subject matter expert (22)
Tier land Tier 2 (13, 44)
Tier 3 (13)
Time frame for peer review (36, 44)
Travel (58)
Voluntary services (52)
-------
Peer Review Handbook
Page 67
COMMONLY USED ACRONYMS
AA Assistant Administrator
AAS Advisory and Assistance Services
BOSC Board of Scientific Counselors
CBI Confidential Business Information
CFR Code of Federal Regulations
CMM Contracts Management Manual
CO Contract Officer
COI Conflict of Interest
EIS Environmental Impact Statement
EPA Environmental Protection Agency
EPAAR EPA Acquisition Regulations
FACA Federal Advisory Committee Act
FAR Federal Acquisition Regulations
FOIA Freedom of Information Act
FTE Full Time Equivalent
GSA General Services Administration
HHS Health and Human Services
IGF Inherently Governmental Function
IRIS Integrated Risk Information System
LAN Local Area Network
NEPA National Environmental Policy Act
NPDES National Pollutant Discharge Elimination System
OAM Office of Acquisition Management
OGC Office of General Counsel
OMB Office of Management and Budget
ORD Office of Research and Development
PMNs Premanufacture Notice
PRAG Peer Review Advisory Group
RA Regional Administrator
RC Regional Counsel
RCRA Resource Conservation and Recovery Act
RIAs Regulatory Impact Analyses
ROD Record of Decision
SAB Science Advisory Board
SAP Scientific Advisory Panel
SGE Special Government Employee
SOPs Standard Operating Procedures
SOW Statement of Work
SPC Science Policy Council
WTI Waste Technologies Industries
-------
Page 68 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page A-l
APPENDIX A
U.S. Environmental Protection Agency
Peer Review Policy
June 7,1994
-------
Page A-2 Peer Review Handbook
June 7, 1994
MEMORANDUM
SUBJECT: Peer Review Program
TO: Assistant Administrators
General Counsel
Inspector General
Associate Administrators
Regional Administrators
Staff Office Directors
Today, I am reaffirming the central role of peer review in our efforts to ensure that EPA
policy decisions rest on sound, credible science and data (see attached policy statement).
Toward that end, as its first major task, EPA's Science Policy Council (SPC) is instituting a
program to expand and improve peer review in all EPA offices. This memorandum gives an
overview of current practices and outlines the new program.
Peer Review Practices and Policy
Peer review at EPA takes several different forms, ranging from informal consultations
with Agency colleagues who were not involved in developing the product to the formal, public
processes of the Science Advisory Board (SAB) and the FIFRA Scientific Advisory Panel
(SAP). In any form, peer review assists the Agency's work by bringing independent expert
experience and judgment to bear on issues before the Agency to the benefit of the final product.
EPA's Peer Review Policy, which responds in part to recommendations in the "Credible
Science, Credible Decisions" report, outlines general principles for peer review at EPA.
Different EPA offices have undertaken various implementing activities, including an Agency-
wide information and planning workshop, internal guideline development, and numerous
specific peer reviews. Even with these activities, however, I am concerned that EPA does not
yet have a comprehensive Agency-wide program for implementing its Peer Review Policy. I
therefore welcome the SPC initiative toward effective, efficient implementation of the policy in
all the program areas to which it applies.
-------
Peer Review Handbook Page A-3
Expanding and Improving Peer Review
The Science Policy Council and its Steering Committee have outlined a dual-track
implementation program of planning and assistance for all Agency offices. The first track has
three major milestones.
First, during the next few weeks, Steering Committee members will consult with senior
management in each office to exchange information on current peer review activities, assistance
needed, possible obstacles to implementation, and implementation planning.
Second, using information and materials developed during the first stage, peer review
task groups in each office will develop standard operating procedures (SOPs) for use in each
office, based in part on generic guidance to be issued by the SPC and in part on peer review
needs and capabilities specific to each office. The resulting SOPs will delineate as appropriate
the scope of application of peer review with respect to various types of scientific and technical
work products such as reports of original research, risk assessments, and analytical methods of
economic analysis. OARM and OGC staff will assist each office as needed on legal, budget and
administrative matters. Each AA and RA will submit draft SOPs for Steering Committee
review by July 15.
Third, the SPC review group will work with each office to complete each plan by
September 15.
In parallel with the above, consistent with the Peer Review Policy, the Science Policy
Council will work with each AA and RA to identify "major scientific and technical work
products" as peer review candidates for the coming year. This process will consider existing
and new plans for internal reviews and for SAB, FIFRA SAP, and other external reviews. The
two-fold objective is to plan reviews for technical products covered by the Peer Review Policy
and to gain experience with options and obstacles. We will use this experience to review and
revise the SOPs as needed. Also, to establish a baseline for comparison, each AA and RA will
identify the "major technical products" completed within his/her program during the past 12
months.
The Science Policy Council has sent additional information to each office offering
guidance on the procedures that you are asked to develop and the schedule for these activities.
Please note, however, that because the policy is effective immediately, current peer review
planning should continue on present schedules in parallel with developing the formal SOPs.
-------
Page A-4 Peer Review Handbook
To begin this process, I have asked each Assistant Administrator and Regional
Administrator to designate a Peer Review Coordinator to work with the Steering Committee on
implementation activities specific to each office. I am very pleased that the Science Policy
Council is taking this important step. A comprehensive peer review program is essential to
maintaining and improving the quality of the analyses that underlie Agency actions. I look
forward to working with you and your staff on this important activity.
/s/
Carol M. Browner
Attachment
cc: Science Policy Council
Science Policy Council Steering Committee
-------
Peer Review Handbook Page A-5
PEER REVIEW AND PEER INVOLVEMENT
AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY
This document establishes the policy of the United States Environmental Protection
Agency (EPA) for peer review of scientifically and technically based work products that are
intended to support Agency decisions. Peer review is presented in the context of the broader
concept, peer involvement.
BACKGROUND
The report "Safeguarding the Future: Credible Science, Credible Decisions"1 focused on
the state of science at EPA. The panel of experts who prepared the report emphasized the
importance of peer review, especially external peer review, and the need for broader and more
systematic use of it at EPA to evaluate scientific and technical work products. Their specific
recommendation regarding peer review reads as follows:
"Quality assurance and peer review should be applied to the planning and results
of all scientific and technical efforts to obtain data used for guidance and
decisions at EPA, including such efforts in the program and regional offices.
Such a requirement is essential if EPA is to be perceived as a credible, unbiased
source of environmental and health information, both in the United States and
throughout the world."
In response to this recommendation, then-Administrator Reilly directed staff to develop an EPA-
wide policy statement, which he issued in January, 1993. The paragraphs below preserve the
core of that earlier statement while updating it to specify the role of the Science Policy Council in
guiding further implementation of the policy. Effective use of peer review is indispensable for
fulfilling the EPA mission and therefore deserves high-priority attention from program managers
and scientists within all pertinent Headquarters and Regional Offices.
1 EPA/600/9-91/050, March 1992.
-------
Page A-6 Peer Review Handbook
PEER INVOLVEMENT AND PEER REVIEW
EPA strives to ensure that the scientific and technical underpinnings of its decisions meet
two important criteria: they should be based upon the best current knowledge from science,
engineering, and other domains of technical expertise; and they should be judged credible by
those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement —
that is, they augment their capabilities by inviting relevant subject-matter experts from outside
the program to become involved in one or more aspects of the development of the work products
that support policies and actions.
One particularly important type of peer involvement occurs when scientifically and
technically based work products undergo peer review — that is, when they are evaluated by
relevant experts from outside the program who are peers of the program staff, consultants, and/or
contractor personnel who prepared the product. Properly applied, peer review not only enriches
the quality of work products but also adds a degree of credibility that cannot be achieved in any
other way. Further, peer review early in the development of work products in some cases may
conserve future resources by steering the development along the most efficacious course.
Peer review generally takes one of two forms. The review team may consist primarily of
relevant experts from within EPA, albeit individuals who have no other involvement with respect
to the work product that is to be evaluated (internal peer review). Or the review team may
consist primarily of independent experts from outside EPA (external peer review).
POLICY STATEMENT
Major scientifically and technically based work products related to Agency decisions
normally should be peer-reviewed. Agency managers within Headquarters, Regions,
laboratories, and field components determine and are accountable for the decision whether to
employ peer review in particular instances and, if so, its character, scope, and timing. These
decisions are made in conformance with program goals and priorities, resource constraints, and
statutory or court-ordered deadlines. For those work products that are intended to support the
most important decisions or that have special importance in their own right, external peer review
is the procedure of choice. Peer review is not restricted to the penultimate version of work
products; in fact, peer review at the planning stage can often be extremely beneficial.
SCOPE
Agency managers routinely make regulatory and other decisions that necessarily involve
many different considerations. This policy applies to major work products that are primarily
scientific and technical in nature and may contribute to the basis for policy or regulatory
decisions. By contrast, this policy does not apply to non-major or nontechnical matters that
-------
Peer Review Handbook Page A-7
Agency managers consider as they make decisions. Similarly, this policy does not apply to these
ultimate decisions.
This policy applies where appropriate, as determined by the National and Regional
Program Managers, to major scientifically and technically based work products initiated
subsequent to the date of issuance. Peer review should be employed to the extent reasonable to
relevant work products that currently are under development. This policy does not apply to the
bases for past decisions, unless and until the relevant scientific and technical issues are
considered anew in the Agency's decision-making processes.
Except where it is required by law, formal peer review (as distinguished from the
Agency's normal internal review procedures) should be conducted in a manner that will not
cause EPA to miss or need extension of a statutory or court-ordered deadline. Agency managers
still may undertake peer review if it can be conducted concurrently with necessary rulemaking
steps.
LEGAL EFFECT
This policy statement does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. On a continuing
basis, Agency management is expected to evaluate the policy as well as the results of its
application throughout the Agency and undertake revisions as necessary. Therefore, the policy
does not stand alone; nor does it establish a binding norm that is finally determinative of the
issues addressed. Minor variations in its application from one instance to another are appropriate
and expected; they thus are not a legitimate basis for delaying or complicating action on
otherwise satisfactory scientific, technical, and regulatory products.
Except where provided otherwise by law, peer review is not a formal part of or substitute
for notice and comment rulemaking or adjudicative procedures. EPA's decision whether to
conduct peer review in any particular case is wholly within the Agency's discretion. Similarly,
nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However,
to the extent that EPA decisions rely on scientific and technical work products that have been
subjected to peer review, the remarks of peer reviewers should be included in the record for that
decision.
IMPLEMENTATION
The Science Policy Council is responsible for overseeing Agency-wide implementation.
Its responsibilities include promoting consistent interpretation, assessing Agency-wide progress,
and developing recommendations for revisions of the policy as necessary.
-------
Page A-8 Peer Review Handbook
The Science Policy Council will oversee a peer-review work group, which will include
representatives from program units throughout EPA to effect a consistent, workable
implementation of the policy. The work group will assist the programs in (1) formulating and, as
necessary, revising standard operating procedures (SOPs) for peer review consistent with this
policy; (2) identifying work products that are subject to review; and (3) for each major work
product, selecting an appropriate level and timing of peer review.
In assisting the programs, the work group will take into account statutory and court
deadlines, resource implications, and availability of disinterested peer reviewers. The group will
work closely with Headquarters offices and the Regional Offices toward ensuring effective,
efficient uses of peer review in supporting their mission objectives. However, the Assistant
Administrators and Regional Administrators remain ultimately responsible for developing SOPs,
identifying work products subject to peer review, determining the type and timing of such
review, documenting the process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.
Because peer review can be time-consuming and expensive, Agency managers within
Headquarters, Regions, laboratories, and field components are expected to plan carefully with
respect to its use — taking account of program priorities, resource considerations, and any other
relevant constraints as well as the policy goal of achieving high-quality, credible underpinnings
for decisions. External peer reviewers should be chosen carefully to ensure an independent and
objective evaluation. The affiliations of peer reviewers should be identified on the public record,
so as to avoid undercutting the credibility of the peer-review process by conflicts of interest.
The policy is effective immediately. The peer-review work group mentioned above will
identify the focal point to whom comments and questions should be addressed and, from time to
time, will provide further information about implementation activities.
/s/
APPROVED: DATE: JUN 7 1994
CAROL M. BROWNER, ADMINISTRATOR
-------
Peer Review Handbook Page B-l
APPENDIX B
Examples of Charges
Please note ~ certain questions that are posed in charges can be responded to
with a yes or no answer. Clearly, this is not the type of response we generally want,
therefore, it is important to phrase charge questions carefully to ensure that you
receive a fully satisfactory and thoughtful response. Where a yes or no answer
might be expected, be sure to ask for a full explanation supporting the yes or no
answer.
-------
Page B-2
Peer Review Handbook
CHARGE EXAMPLE 1 - CHARGE TO REVIEWERS FOR THE WTI DRAFT FINAL
RISK ASSESSMENT
The draft final WTI risk assessment is divided into several volumes covering the
scientific disciplines of toxicology, environmental fate and transport, combustion engineering,
atmospheric modeling, exposure assessment, ecological risk assessment, and accident analysis.
As a reviewer of the WTI draft final risk assessment, you should use your best technical
knowledge and professional judgment to comment on the technical accuracy, completeness and
scientific soundness of the assessment. Each reviewer is asked to focus on several specific issues
in his or her area of expertise with comments on other areas invited but optional. Your
comments will be considered in finalizing the risk assessment.
For the peer review workshop reviewers will be organized into 5 work groups:
Combustion Engineering, Air Dispersion and Deposition Modeling and Accident Analysis,
Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be
familiar with the Executive Summary (Volume I) and the Facility Background (Volume II)
sections of the draft risk assessment. In addition, each work group should focus on specific
Volumes as specified below:
Workgroup
Combustion Engineering
Air Dispersion and
Deposition Modeling and
Accident Analysis
Toxicology
Exposure Assessment
Ecological Risk Assessment
Risk Assessment Volumes
Volume III - Facility
Emissions
Volume IV - Atmospheric
Dispersion and Deposition
Modeling
Volume V - Human Health
Risk Assessment
Volume V - Human Health
Risk Assessment
Volume VI - Screening
Ecological Risk Assessment
Volume VII - Accident
Analysis
Volume VII - Accident
Analysis
Volume VII - Accident
Analysis
-------
Peer Review Handbook Page B-3
While reviewing these sections of the document, please address the following general
issues.
1) Comment on the organization of the risk assessment document. Does the layout
follow a logical format? Is the presentation of information in the document clear,
concise and easy to follow?
2) Does the executive summary accurately reflect the data and methodologies used
and the conclusions derived in the risk assessment?
3) Were the major recommendations of the 1993 peer review workshop for the risk
assessment plan addressed?
4) As with any risk assessment, there are always additional data and method
development efforts that could be undertaken to reduce the level of uncertainty.
However, are there any major data or methodological gaps that would preclude
the use of this risk assessment for decision making? If so, how should they be
addressed?
5) What long-term research would you recommend that could improve risk
assessments of this type in the future?
In addition, the following workgroup specific issues should be addressed.
Emissions Characterization
Emissions characterization includes identification of substances of concern and the
development of emission rates for these contaminants. Emission rates were developed through a
combination of site specific stack test data and models. Please comment on the following issues
with respect to this aspect of the draft risk assessment.
1) To characterize the nature of the emissions, waste stream profiles were developed
and entered into a database. Several refinements and adjustments (e.g., the
Subtraction Correction Factor for chlorinated compounds) were applied to the
profiles before substances of concern were identified. Please comment on
whether or not these adjustments are appropriate. What is the anticipated effect
on the risk assessment?
2) Comment on the selected chemicals of concern. Have important chemicals been
missed due to the selection technique?
-------
Page B-4 Peer Review Handbook
3) Comment on the approaches used to estimate stack emission rates (e.g., use of the
95% UCL of the arithmetic mean or the maximum detected value, whichever is
smaller, for high end emission rates). Are the approaches appropriate? Are their
effects on the risk assessment adequately characterized? Comment on the
adjustment made to PCDD/PCDF emission rates to account for brominated
dioxin-like compounds. Also, comment on the approach to characterizing
emission rates from fugitive sources (e.g., use of the TANKS 2 model for the
Carbon Adsorption Bed).
4) Comment on the identified sources of fugitive emissions. Was the approach used
to select these sources appropriate? Have important sources been missed? Have
emissions from process upsets been given appropriate consideration?
5) There have been a number of controlled burns at the WTI facility. Please
comment on the adequacy of these data in estimating potential exposure. Please
comment on the assumptions made from the tests in regard to composition of
wastes received at WTI and emissions when the plant operates in the future.
6) Comment on the use of emission factors from coal burning to estimate the
emission rate of fly ash from WTI. Are the factors used to adjust the coal
emission rate appropriate? Are the uncertainties introduced from this approach
adequately characterized?
7) Overall, is the identification of the key assumptions used in characterizing the
nature and magnitude of emissions thorough? Are the magnitude and direction of
effect of these assumptions on the overall risk assessment accurately
characterized? Is the uncertainty and variability inherent in this analysis
adequately discussed? Does the sensitivity analysis cover the major parameters
expected to have an effect on the risk assessment?
Dispersion and Deposition Modeling
To develop this risk assessment, computer models have been used with site specific data
on emission rates and meteorological conditions to simulate the air concentrations and deposition
rates for contaminants potentially emitted from the WTI facility. The models used include the
Industrial Source Complex - Complex Terrain Deposition (ISC-COMPDEP), the CALPUFF, and
the INPUFF models. In your review, please address the following issues.
1) Since the 1993 peer review of the risk assessment plan, a number of efforts have
been completed to reduce the uncertainty associated with the air dispersion and
deposition modeling. These efforts include the collection of site-specific data for
-------
Peer Review Handbook Page B-5
emission rates and meteorological conditions. Also, a wind tunnel study was
conducted to evaluate the effects of the complex terrain surrounding the WTI
facility. Does the risk assessment document adequately summarize these
activities? Is the link between these data collection efforts, the air dispersion
models, and the risk assessment clearly established?
2) The results of 12 sets of sensitivity tests indicate that geophysical variables (e.g.,
terrain) are more likely to affect dispersion and deposition than emission variables
(e.g., stack temperature). Were these sensitivity analyses adequate? Comment on
the conclusions reached. To further examine the effect of geophysical variables,
wind tunnel testing was conducted to model the terrain induced flow effects
expected near WTI. It was concluded that changes in peak concentrations
attributed to these effects are relatively minor and that the ISC-COMPDEP model
is sufficiently conservative. Comment on this conclusion. Have these analyses
helped to characterize and/or reduce the uncertainty in the air dispersion modeling
associated with the complex terrain surrounding WTI.
3) The ISC-COMPDEP model does not allow for non-steady state conditions such as
calm winds and strong temperature inversions. Therefore, CALPUFF was used to
estimate air dispersion and deposition under these conditions. However,
CALPUFF gave similar peak, 24 hour, and annual average concentrations as ISC-
COMPDEP. Comment on the adequacy of this analysis. Comment on the
conclusions reached. Has this analysis helped to characterize and/or reduce the
uncertainty in the air dispersion modeling associated with non-steady state
meteorological conditions?
4) Atmospheric dispersion modeling was used to estimate air concentrations of
hazardous chemicals for the accident analysis. The SLAB model was used for
vapor releases from spills and the mixing of incompatible wastes. ISC-
COMPDEP was used for releases associated with fires. Comment on the
selection of the models and inputs. Are they appropriate selections?
5) Overall, have adequate sensitivity tests been conducted to demonstrate the
magnitude of variation in concentrations and deposition estimates with model
inputs? Please explain fully.
Human Health Risks
Human Health Risk Assessment includes hazard identification, dose-response evaluation,
exposure assessment, and risk characterization. To develop the risk assessment, potentially
exposed populations have been identified and the magnitude, frequency, and duration of their
-------
Page B-6 Peer Review Handbook
exposure quantified. This information was then integrated with the hazard identification and
dose response evaluation for the risk characterization. For this risk assessment, both
carcinogenic and non-carcinogenic health effects have been evaluated. In your review, please
comment on the following issues.
Exposure
1) EPA's Exposure Assessment Guidelines identify certain exposure descriptors that
should be used to characterize exposure estimates. The Guidelines define high
end exposure estimates as those representing individuals above the 90th percentile
on the exposure distribution but not higher than the individual in the population
who has the highest exposure. Bounding exposure estimates are those that are
higher than the exposure incurred by the person in the population with the highest
exposure. Central tendency exposure estimates are defined as the best
representation of the center of the exposure distribution (e.g., arithmetic mean for
normal distributions). Comment on whether or not the WTI exposure assessment
properly characterizes each of the exposure estimates in terms of these
descriptors.
2) The factors that go into estimating a central tendency or high end exposure, once
the population has been defined, include the environmental media concentration,
the intake rate, and the duration and/or frequency of exposure. Comment on
whether or not the WTI exposure assessment does an adequate job of describing
the logical procedure of combining these factors to develop central tendency, high
end, and/or bounding estimates of exposure for each of the exposed
subpopulations.
3) An important factor in an exposure assessment is identifying all of the important
exposure sources. Please comment on the adequacy of the WTI assessment in
identifying the important sources and pathways of exposure.
4) Have the key assumptions for estimation of chemical concentration and for
estimation of exposure been identified? Are the magnitude and direction of effect
correct for the assumptions that have been identified?
5) Supposedly, conservative assumptions have been applied in this assessment to
account for uncertainty. Are the conservative assumptions appropriately factored
into the ultimate characterization of what descriptor best applies to each exposure
estimate? Please comment on whether the uncertainties were confronted in an
adequate manner. If they were not, please state what should be done differently.
-------
Peer Review Handbook Page B-7
Hazard Identification/Dose Response and Risk Characterization
1) To select surrogate compounds for quantitative risk assessment, a two step
process was used in which chemicals were ranked on the basis of emission rate,
toxicity (both cancer and non-cancer), and bioaccumulation potential. Please
comment on this selection process. Are the ranking factors appropriate? Could
important compounds have been omitted from the analysis based on the ranking
procedure?
2) For the majority of the chemicals of concern, traditional approaches to dose
response evaluation were employed (e.g., use of a slope factor for cancer and use
of a RfD/RfC for non-cancer). However for certain chemicals or groups of
compounds a different methodology was used. Specifically, dioxins, furans,
PAHs, lead, mercury, nickel, chromium, acid gases, and particulate matter were
given special consideration. Please comment on the methodology used for these
compounds. Was it appropriate? Have the uncertainties associated with the
methodology been adequately characterized? Comment on the assumptions used
due to a lack of chemical specific data.
3) Please comment on the selection of the overall population and the various
subpopulations at risk. Were site specific data, such as the informal home
gardening survey, properly utilized to identify these subpopulations?
4) It is stated in the risk assessment that average risk estimates are based on average
emission rates, average air dispersion/deposition within a subarea, and typical
exposure factors. Further, maximum risks are based on average emission rates,
typical exposure factors, and the maximum air concentration within a subarea.
Please comment on this use of the terms average and maximum risks. Are these
descriptive terms appropriate given the parameters used to derive each? Please
explain fully.
5) Comment on whether or not the non-cancer risks of chemicals of concern have
been adequately addressed by the risk assessment? For example, has an adequate
discussion of endocrine disrupters been provided which either characterizes their
risks or clearly explains why their risks cannot be characterized? Further, have
non-cancer chronic toxicities of dioxins and furans been adequately addressed in
the risk assessment?
6) Please comment on whether or not the uncertainties associated with the additivity
and/or synergy of risks from pollutants emitted together from the WTI facility are
adequately discussed in the risk assessment.
-------
Page B-8 Peer Review Handbook
7) Have the key assumptions for estimation of dose and risk been identified? Are the
magnitude and direction of effect correct for the assumptions that have been
identified? Please comment on whether the uncertainties were confronted in an
adequate manner. If they were not, please state what should be done differently.
8) Please comment on the overall adequacy of the risk characterization. Does the
risk characterization include a statement of confidence in the risk assessment
including a discussion of the major uncertainties. Are the hazard identification,
dose-response assessment, and exposure assessment clearly presented? Have
sufficient risk descriptors which include important subgroups been presented and
discussed?
Screening Ecological Risk Assessment
As with the human health risk assessment, the ecological risk assessment pulls together
elements of exposure analysis and dose-response evaluations to develop a risk characterization.
For the Screening level Ecological Risk Assessment (SERA), Ecological Chemicals of Concern
(ECOC) and indicator species have been identified to provide conservative estimates of risk.
Please address the following issues in your review.
1) Are there any components of the SERA which you feel undermine the scientific
validity of the assessment? If so, what are they and can you provide suggestions
to strengthen the identified components?
2) Is the organization of the document clear and does it present the material in a clear
and concise manner consistent with the Framework for Ecological Risk
Assessment (EPA, 1992)? Please explain fully.
3) Uncertainties are discussed in numerous sections of the SERA and compose
Section VIII of the SERA. In each case, do these discussions cover all relevant
and important aspects of the uncertainties which you think should be addressed in
the SERA?
4) In your opinion, what is the weakest and what is the strongest aspect of the
SERA? Can you make any suggestions on how the weakest parts can be
strengthened by the Agency?
5) In Section II, are the stressors, ecological effects, and both the assessment and
measurement endpoints adequately characterized? Are the five emission
scenarios adequate to characterize the exposures for the WTI facility? Are there
other emission scenarios which you think should be included in the SERA?
-------
Peer Review Handbook Page B-9
6) In Section III, is the site characterization adequate to support the SERA? Why or
why not?
7) In Section IV, is the tiered process used to identify the ecological chemicals of
concern (ECOC) from the initial list of potential chemicals considered
scientifically defensible? Does application of this tiered approach support the
statement made in the SERA "by focusing on the potential risk from the selected
ECOCs, the SERA provides a thorough screening-level evaluation for the WTI
facility?"
8) In Sections V and VI, are the exposure and ecological effects adequately
characterized? Are the most appropriate estimation techniques available used?
Are the assumptions clearly stated? Please explain fully.
9) In Section VIII, are there any major elements missing from the risk
characterization which you think need to be included or which would strengthen
the risk characterization? Does the risk characterization support the summary and
conclusions presented in Section IX?
10) In Section IX, given the assumptions made and the processes used to select and
evaluate chemicals, receptors, and exposure pathways, do you think the SERA
adequately met its objective of not inadvertently underestimating risk?
Accident Analysis
The Accident Analysis for the WTI incinerator involves evaluating the probability of an
emergency incident occurring which results in the release of hazardous waste. The consequences
of this release are also evaluated using exposure and human health effects information. Unlike
the human health risk assessment which has a primary goal of quantifying risks, the accident
analysis typically provides information that can be used to reduce the likelihood, extent and
impact of possible accidents. Please comment on the following issues in your review of this
aspect of the risk assessment.
1) The WTI accident assessment selected five scenarios for quantitative evaluation
that were considered to be of primary concern. The scenarios are an on-site spill,
an on-site fire, an on-site mixing of incompatible waste, an off-site spill, and an
off-site spill and fire. Please comment on the selection of these scenarios. Were
any significant scenarios missed?
2) Specific chemicals were selected to evaluate each scenario. Please comment on
the selections. Would other chemicals have been more appropriate?
-------
Page B-10 Peer Review Handbook
3) Chemical specific release rates are calculated for each scenario. Please comment
on the procedures used to estimate the release rates. Was an appropriate approach
used?
4) Atmospheric dispersion modeling was used to estimate air concentrations of
hazardous chemicals. Specifically, the SLAB model was used for vapor releases
from spills and the mixing of incompatible wastes. ISC-COMPDEP was used for
releases associated with fires. Comment on the selection of the models and in-
puts. Are they appropriate selections? Should other models or inputs been used?
5) Please comment on the assessment's conclusions on the severity of consequences
and probability of occurrence. Has the report correctly categorized the severity of
the consequences of the different accident scenarios? Has the assessment
adequately justified the reported probability of occurrence of each of the accident
events?
6) Key assumptions were made in the identification of accident scenarios and the
description of the conservative and typical events. Included were a description of
the magnitude of the effect of the assumptions and direction of the effect. Please
comment on the assumptions. Are they justified? Are the descriptions of the
magnitude and directions of the effects correct? Has the accident assessment
adequately confronted the uncertainties involved in doing this type of analysis? If
not, what else should be done?
7) Comment on the appropriateness of using IDLH values for characterizing the
severity of consequences in the accident analysis. Comment on the
appropriateness of using 10 X LOG for chemicals for which IDLH values have not
been established.
8) In the accident analysis, IDLH (or 10 X LOG) values were used to determine the
downwind distances over which adverse human health effects might occur. To
evaluate the uncertainty introduced by using the IDLH, a sensitivity analysis was
conducted where these distances were recalculated using the LOG (a more
stringent health criteria). Other sources of uncertainty that are identified in the
accident analysis include concentration averaging times, chemical concentrations,
emission rates, and meteorological conditions. For most of these parameters it is
stated that conservative assumptions were used to avoid underestimating risks.
Have the uncertainties inherent in the accident analysis been adequately
characterized? For those parameters where sensitivity analyses were not
conducted, is the conclusion that conservative assumptions have avoided
underestimation valid?
-------
Peer Review Handbook Page B-ll
CHARGE EXAMPLE 2 - IRIS Pilot Program
Instructions to Peer Reviewers for Reviewing IRIS Summaries and Supporting
Documentation
The U.S. EPA is conducting a peer review of the scientific basis supporting the health
hazard and dose response assessments for the subject chemical that will appear on the Agency's
online database, the Integrated Risk Information System (IRIS). Materials to be reviewed
include the summary information that will appear on IRIS (the inhalation reference concentration
[RfC], oral reference dose [RfD], and cancer assessment) and the supporting document, the
Toxicological Review, which will also be made available to the public.
A listing of Agency Guidelines and Methodologies that were used in the development of
these hazard and dose-response assessments included the following: The Risk Assessment
Guidelines (1986), the (new) Proposed Guidelines for Carcinogen Risk Assessment ( 1996),
Guidelines for Developmental Toxicity Risk Assessment, (proposed) Interim Policy for Particle
Size and Limit Concentration Issues in Inhalation Toxicity, (proposed) Guidelines for
Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations
and Application of Inhalation Dosimetry, Recommendations for and Documentation of
Biological Values for Use in Risk Assessment and Use of the Benchmark Dose Approach in
Health Risk Assessment. Copies of these documents (and/or their relevant sections) will be
made to the reviewer upon request.
Peer review is meant to ensure that science is used credibly and appropriately in
derivation of these dose-response assessments. You have been chosen as an expert on the
chemical under consideration, on a scientific discipline related to at least one of the assessments,
or in the field of risk assessment. At least three peer reviewers per chemical are being chosen to
review the scientific basis of these draft dose-response assessments before they are forwarded on
to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and
dose-response assessments will then appear on IRIS and become available as Agency consensus
health effect information.
The primary function of the peer reviewer should be to judge whether the choice, use,
and interpretation of data employed in the derivation of the assessments is appropriate and
scientifically sound. This review is not of the recommended Agency risk assessment guidelines
or methodologies used to derive cancer or RfD/C assessments as these have been reviewed by
external scientific peers, the public, and EPA Science Advisory Boards. The reviewer's
comments on the application of these guidelines/methodologies within the individual
assessments is, however, welcomed and encouraged. For example, the reviewer may ascertain
whether or not there is data sufficient to support use of other than default assumptions for areas
such as sensitive subpopulations or linear cancer extrapolation. The reviewer may also have
-------
Page B-12 Peer Review Handbook
opinions on other areas of uncertainty such as subchronic to chronic duration (when only a
subchronic study is available) or an incomplete data base but should focus on the specific area of
uncertainty rather than on the magnitude of the overall estimate.
Below are two groups of questions regarding this review. The first is a set of general
questions that are meant to guide you through your review. It is not imperative that you
specifically answer each question of this group. The second group of questions, however, are
specific for the chemical assessments and deal with areas of scientific controversy or uncertainty
in which the Agency may have to make a scientific judgment. Your input to this set of questions
is considered vital to the review process.
Questions for IRIS Peer Reviewers -General
1) Are you aware of any other data/studies that are relevant (i.e., useful for the
hazard identification or dose-response assessment) for the assessment of the
adverse health effects, both cancer and noncancer, of this chemical? Please
explain fully.
2) For the RfD and RfC, has the most appropriate critical effect been chosen (i.e.,
that adverse effect appearing first in a dose-response continuum)? For the cancer
assessment, are the tumors observed biologically significant? relevant to human
health? Points relevant to this determination include whether or not the choice
follows from the dose-response assessment, whether the effect is considered
adverse, and if the effect (including tumors observed in the cancer assessment)
and the species in which it is observed is a valid model for humans.
3) Have the noncancer and cancer assessments been based on the most appropriate
studies? These studies should present the critical effect/cancer (tumors or
appropriate precursor) in the clearest dose-response relationship. If not, what
other study (or studies) should be chosen and why?
4) Studies included in the RfD and RfC under the heading "Supporting/Additional
studies" are meant to lend scientific justification for the designation of critical
effect by including any relevant pathogenesis in humans, any applicable
mechanistic information, any evidence corroborative of the critical effect, or to
establish the comprehensiveness of the data base with respect to various endpoints
(such as reproductive/developmental toxicity studies). Should other studies be
included under the "Supporting/Additional" category? Should some studies be
removed?
-------
Peer Review Handbook Page B-13
5) For the noncancer assessments, are there other data that should be considered in
developing the uncertainty factors or the modifying factor? Do you consider that
the data support use of different (default) values than those proposed?
6) Do the Confidence statements and weight-of-evidence statements present a clear
rationale and accurately reflect the utility of the studies chosen, the relevancy of
the effects (cancer and noncancer) to humans, and the comprehensiveness of the
data base? Do these statements make sufficiently apparent all the underlying
assumptions and limitations of these assessments? If not, what needs to be
added?
Questions for IRIS Peer Reviewers - Chemical Specific
[example: cumene]
1) Based on the information noted in the Principal study currently designated
(Cushman et al., 1995) is the discounting of the renal effects in males justified? Is
sufficient rationale given to let stand the organ weight changes in female rats as a
critical effect?
2) Is the information in the Toxicological Review sufficient to consider cumene as
having a low potential for causing reproductive effects? Please explain fully.
RECOMMENDATIONS
Based on your reading and analysis of the information provided, please identify your
overall recommendation for the IRIS materials you have reviewed as
1) Acceptable as is
2) Acceptable with minor revision (as indicated)
3) Acceptable with major revision (as outlined)
4) Not acceptable
-------
Page B-14 Peer Review Handbook
CHARGE EXAMPLE 3 - Science Advisory Board Review of the Agency's National Risk
Management Research Laboratory's (NRMRL) Program
The Office of Research and Development (ORD) requests that the Science Advisory
Board review the Agency's National Risk Management Research Laboratory's (NRMRL)
program.
In the "Strategic Plan for the Office of Research and Development" (EPA, 1996a), ORD
described the relationship of risk assessment to the risk management process, and emphasized the
need for scientific and engineering research to enable sound risk management decisions and
actions. Within the framework of that strategic plan, NRMRL's mission is to conduct research to
reduce uncertainties and costs associated with making and implementing environmental risk
management decisions. NRMRL has therefore developed a research agenda to reduce risk
uncertainty that also focuses on those important, relevant issues where it can make a difference.
The charge to the SAB is to:
1) Examine and critique the research programmatic directions such as whether
NRMRL is pursuing the most appropriate research problem areas;
2) Comment on strategic directions, e.g. use of its core technical competencies,
transition from primarily extramural to an intramural R&D organization,
leveraging with other agencies and organization;
3) Review and comment on the effectiveness of NRMRL's approach to science
management, e.g. measures of success and science quality, soundness of peer
review process;
4) Examine and critique the relationship of NRMRL's risk management research and
its intended role in the risk assessment/risk management paradigm; and
5) Review and comment on the strategic balance for the next decade among
pollution prevention, technology development, remediation, and risk management
assessment activities.
-------
Peer Review Handbook Page B-15
CHARGE EXAMPLE 4 - Science Advisory Board (SAB) Review of the Technical Aspects
of the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM)
The EPA Science Advisory Board (SAB) is asked to review the technical aspects of the
Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM). The review
document was developed collaboratively by four Federal agencies, departments and commissions
having authority for control of radioactive materials: Department of Defense, Department of
Energy, Environmental Protection Agency, and Nuclear Regulatory Commission. MARSSIM
addresses the need for a nationally consistent approach to conducting radiation surveys of
potentially radioactively contaminated sites that are being considered for release to the public. A
condition of release is a demonstration that residual radioactivity levels do not exceed a specified
risk or dose level, also known as a release criterion. MARSSIM provides guidance to users
performing and assessing the results of such a demonstration for surface soils and building
surfaces.
The SAB is asked by the Agency's Office of Radiation and Indoor Air (ORIA) to respond
to the following charge in its review:
1) Is the overall approach to the planning, data acquisition, data assessment, and data
interpretation as described in the MARSSIM technically acceptable? Please
explain fully.
2) Are the methods and assumptions for demonstrating compliance with a dose- or
risk-based regulation technically acceptable? Please explain fully.
3) Are the hypotheses and statistical tests and their method of application
appropriate? Please explain fully.
-------
Page B-16 Peer Review Handbook
CHARGE EXAMPLE 5 - Science Advisory Board (SAB) Review of the Statistical
Performance of the Agency's Protozoan Oocyst Monitoring Methods
The Science Advisory Board (SAB) is asked to review a report describing the statistical
performance of the Agency's protozoan oocyst monitoring methods. Agency staff recognized
that the protozoan analysis methodology that had been formally adopted for the Information
Collection Rule (ICR) was crude and had very low and highly variable recoveries of added
oocysts. The statistical methodology was considered by Agency staff to be necessary to
determine whether the Agency can take advantage of the large monitoring program agreed to
under the negotiated rulemaking process.
The Agency charge asks that the SAB evaluate the report and address the following
concerns:
1) Evaluate the factual and conceptual soundness of the approach and methods used,
and the soundness of the results and conclusions of the report.
2) Evaluate the viability of the assumptions and conditions tested in the report.
3) Evaluate the suitability of the report as a basis for making a decision on the use of
protozoan monitoring data for a national impact assessment.
4) Evaluate whether the degree of accuracy and precision of the protozoan method is
acceptable for an impact analysis.
-------
Peer Review Handbook Page B-17
CHARGE EXAMPLE 6 - Science Advisory Board (SAB) Review of the Environment
Monitoring and Assessment Program (EMAP) Research Strategy and Research Plan
The Science Advisory Board (SAB) is asked to review the Environment Monitoring and
Assessment Program (EMAP) Research Strategy and Research Plan. The review is requested by
the Office of Research and Development (ORD) with the following specific charge issues:
1) Previous peer reviews recommended that EMAP develop a close working relation
with EPA Program Offices and other federal monitoring efforts. Does the EMAP
strategy support the [Office of Science and Technology Policy's Committee on
Environment and Natural Resources] CENR National Monitoring Framework and
EPA Program Offices?
2) Previous peer reviews recommended that EMAP initiate a focused research
program on indicator development. Does the intramural EMAP program on
ecological indicator development, coordinated with the [ORD Science To Achieve
Results] STAR solicitations, respond to this research need?
3) Previous peer reviews recommended that the EMAP design be modified to
include a set of nonrandomly selected sentinel sites with intensive data collection.
Does the development of Index Sites as outdoor laboratories in the national parks
([National Park Service] NFS and [US Geological Survey] USGS) and selected
estuaries ([National Oceanic and Atmospheric Administration] NOAA) add this
dimension to the EMAP?
4) Previous peer reviews recommended that EMAP combine effects-oriented and
stressor-oriented monitoring approaches. Do the focused geographic
demonstration pilot studies (initially in the Mid-Atlantic region) combine these
elements?
-------
Page B-18 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page C-l
APPENDIX C
GUIDANCE ON REQUESTING A REVIEW
BY THE SCIENCE ADVISORY BOARD (SAB)
-------
Page C-2 Peer Review Handbook
Guidance on Requesting a Review
By the Science Advisory Board (SAB)
Summary
The Science Advisory Board (SAB) annually solicits proposals for review projects every
spring for the following fiscal year. This appendix provides guidance to Programs and Regions
to help them submit requests for SAB reviews. Requests should be submitted to the Science
Advisory Board in both hard copy and electronic versions usually by mid-June for the following
fiscal year. The requests may be part of the annual submissions that respond to the peer review
activities of the Agency, or they may be submitted directly to the Board. Although providing
requests at one defined time in the spring helps with SAB planning, we recognize that projects
also come to light during other parts of the year. Please contact the SAB staff for details on
making submissions during the remainder of the year (see end of this document for contacts).
Background
A key priority for Administrator Browner is to base Agency actions on sound scientific
data, analyses, and interpretations. She issued the Agency's Peer Review Policy to increase the
quality of the technical foundations upon which EPA's regulatory structures are built. The SAB
is a key scientific peer review mechanism available to Programs and Regions in implementing
the Peer Review Policy. However, because the Board has finite resources it cannot conduct all
reviews. This document is designed to help Programs and Regions determine which projects to
submit to the SAB. Note particularly that the SAB focuses on the technical underpinnings of
Agency positions; i.e., risk assessment issues, in contrast to risk management issues.
The topics that are best suited for the Board's agenda are those that satisfy several of the
following criteria:
1) Integrate science into Agency actions in new ways.
2) Influence long-term technological developments.
3) Impact overall environmental protection.
4) Address novel scientific problems or principles.
5) Address problems that transcend federal-agency or other organizational
boundaries.
6) Strengthen the Agency's basic capabilities.
7) Serve Congressional or other leadership interests.
8) Deal with controversial issues.
In suggesting issues for SAB involvement, Programs and Regions should note the breadth
of SAB activities:
-------
Peer Review Handbook Page C-3
Historically, most of the outputs of the Board are in the form of "full" reports.
They present the findings of peer reviews of Agency document(s) and contain
considerable detail about the findings and recommendations of the Board. They also
address the specific questions posed by the Charge to the Board. "Letter" reports fulfill
the same function as reports, but are simply shorter in length. Due to the need to be more
responsive with advice, the Board has recently begun to produce more short letter reports
than full reports, as they can be produced and finalized in less time.
The SAB has also introduced the "Consultation" as a means of conferring—in
public session—with the Agency on a technical matter before the Agency has begun
substantive work on that issue. The goal is to leaven EPA's thinking on an issue by
brainstorming a variety of approaches to the problem very early in the development
process. There is no attempt or intent to express an SAB consensus or to generate an
SAB report. The Board, via a brief letter simply notifies the Administrator that a
Consultation has taken place.
More recently, the Board introduced a new vehicle for communicating with its
clients — the "Advisory" - which provides, via a formal SAB consensus report, critical
input on technical issues that arise during the Agency's issue development process. The
Advisory generally involves a review of a multi-year Agency project. The intent is to
provide some mid-course assessment to see if the Agency is heading in a scientifically
credible direction. In order to maintain an objective, arms-length relation with the
Agency and its projects, the SAB review of the final product at some point in the future
will include experts who did not participate in producing the Advisory.
The Agenda Setting Process
Each Assistant Administrator and Regional Administrator is normally asked to submit a
list of candidate topics for SAB action/review. A "project sheet" (see attached example) is used
to define each topic that is nominated for SAB review. The project sheet is prepared by the
requesting office and contains the following information:
1) Project title/subject (Descriptive short title of project)
2) Requesting Organization/Office (Primary office requesting review, AA/RA level)
3) Requesting Official (Name and position of senior official requesting review,
usually office or division level - this is the person who may receive a summary
briefing from the Chair following the review)
-------
Page C-4 Peer Review Handbook
4) Program Contact (Name/phone number/mail code - this is the principal contact for
SAB Staff to interact with during development of the SAB review)
5) Background (brief history of the project and why it is important)
6) Tentative Charge (what the SAB is being asked to comment on - usually a set of
questions)
7) Tentative Schedule and Committee (when the review is expected to be conducted,
e.g., Winter 1998; and which SAB committee is appropriate for the review — final
choice as to the review committee is at the discretion of the Board)
8) Budget Estimate (Rough estimates of Agency funding for the subject over the past
5 years (if applicable) and for the next 2 years (if applicable). The SAB Executive
Committee has asked for this information to help it better appreciate the level of
Agency involvement in and commitment to the issue)
9) Preparer (name, phone, office of preparer of Project Sheet and the date prepared)
The proposed topics will be examined and discussed in a number of forums:
1) The individual SAB Committees - Throughout the late Spring and Summer, the
SAB Committees will be examining options for the following fiscal year,
including all suggestions made by the Agency.
2) The Science Policy Council-Steering Committee (SPC-SO - The SPC-SC usually
meets in early summer to examine the proposals for each fiscal year. The goal is
to provide cross-office critique/integration of the proposals. The SPC-SC will be
used as a forum for continuing discussion throughout the process and throughout
the year as new topics emerge.
3) The Deputy's perspective - In the summer, the Deputy Administrator will review
the requests and provide insights on priorities.
4) The SAB Executive Committee - During its summer meeting, the SAB's EC will
examine the nominated topics, adding its own perspective on an appropriate
agenda, using its selection criteria.
5) The Administrator - In September, the list of proposed topics will be delivered to
Administrator Browner for information and added insights.
-------
Peer Review Handbook Page C-5
The completed project sheets should be submitted electronically to the SAB Deputy Staff
Director (fowle.jack@epamail.epa.gov) and in signed hard copy (mail code 1400).
SAB Staff Contacts
Dr. Donald Barnes (202-260-4126) - Staff Director; Designated Federal Officer (DFO) for the
Executive Committee:
Dr. JackFowle (202-260-8325) - Deputy Staff Director.
Mr. Robert Flaak (202-260-5133) - Team Leader, Committee Operations Staff; DFO for the
Clean Air Scientific Advisory Committee (CASAC); DFO for the Research Strategies Advisory
Committee (RSAC).
Ms. Kathleen Conway (202-260-2558) - DFO for the Environmental Engineering Committee
(EEC).
Dr. Jack Kooyoomjian (202-260-2560) - DFO for the Radiation Advisory Committee (RAC);
DFO for the Advisory Council on Clean Air Compliance Analysis (Council).
Mr. Tom Miller (202-260-5886) - DFO for the Drinking Water Committee (DWC); DFO for the
Environmental Economics Advisory Committee (EEAC).
Mr. Sam Rondberg (202-260-2559) and LCDR Roslyn Edson (PHS) - DFOs for the
Environmental Health Committee (EHC); DFOs for the Integrated Human Exposure Committee
(fflEC)
Ms. Stephanie Sanzone (202-260-6557) - DFO for the Ecological Processes and Effects
Committee (EPEC).
All SAB staff can be contacted via Agency Email.
-------
Page C-6 Peer Review Handbook
Science Advisory Board
Proposed Project
Project title/subject: Proposed Amendments to the Risk Assessment Guidelines for
Carcinogens
Requesting Organization/Office: Office of Research and Development (ORD)
Requesting Official: Name, Title, Office/Organization
Program Contact: Name, Title, Office/Organization, 202-260-xxxx
Background: EPA's Health Risk Assessment Guidelines provide generic science and science
policy guidance on risk assessment issues for use in all Agency offices. EPA has currently issued
or proposed nine guidelines (or amendments) in this series, all of which have been submitted to
the Science Advisory Board for review.
The current guidelines for carcinogen risk assessment were reviewed by the SAB and issued as
final guidance in 1986. In 1988, the Forum initiated a public process for considering
amendments to these guidelines. A Risk Assessment Technical Panel considered submissions
from the public as well as information developed by experts at two public workshops in revising
these guidelines.
Tentative Charge: Review the amended and expanded guidance, with special emphasis on (a)
weight-of-evidence issues, (b) a new classification system, (c) dose response modeling, and (d)
the use of pharmacokinetic and metabolic data. A more detailed charge will be negotiated with
SAB at a later date.
Tentative Schedule and Committee: Winter, 1998, Environmental Health Committee
Budget: FY 1995 - $xxx and yy FTE
FY 1996 - $xxx and yy FTE
FY 1997 - $xxx and yy FTE
FY 1998 - $xxx and yy FTE
FY 1999 - FY2000 - estimated costs of $xxx and yy FTE each year
Preparer: Name, Title, Office/Organization, 202-260-xxxx
Date: June 1, 1997
-------
Peer Review Handbook Page D-l
APPENDIX D
Example Statements of Work for Contracts
-------
Page D-2 Peer Review Handbook
STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical Review
Contractor for Panel Review of Assistance Agreement or Fellowship Applications
1) Purpose
The purpose of this contract is to purchase peer review services of a contractor with
expertise in Exploratory Research; Environmental Chemistry. The services are for peer
reviewing applications received in response to the Office of Research and Development's
(ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the
evaluation sheets shall be prepared prior to the reviewer's participation in a 3-dav panel
discussion to be held in Washington, D.C. on May 5 -7,1997.
2) Statement of Work
ORD's National Center for Environmental Research and Quality Assurance (NCERQA)
is responsible for overseeing the recently expanded research grants and fellowships programs.
Each year NCERQA (alone or in conjunction with other organizations) solicits applications in
each of these programs. The applications to be reviewed under this contract were submitted in
response to the solicitation for the 1997 Science to Achieve Results program. As part of the
selection process, NCERQA must conduct a peer review that is designed to evaluate the
scientific quality of each application; this is accomplished through the ad hoc use of technical
experts.
The peer review services required by this contract necessitate the independent review of a
maximum of 10 applications and the preparation of a typed evaluation summary and an overall
rating for each of these applications. Each evaluation summary shall support and be consistent
with the overall rating that is assigned; it also shall be completed prior to the contractor's
participation in the panel discussions. After the panel discussions for the applications assigned to
the contractor, the contractor shall submit all completed evaluation summaries to the designated
Science Review Administrator (SRA).
The contractor also shall serve as the panel's rapporteur for approximately 6-8 of the
applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed
evaluation summary (on-site typing support will be provided by NCERQA) that reflects the
panel's discussion of the respective application as well as the panel's overall rating (the criteria
for the panel's overall rating are the same as those for each peer reviewer's overall rating). As
rapporteur, the contractor shall submit these panel evaluation summaries to the designated SRA
prior to leaving the panel meeting.
-------
Peer Review Handbook Page D-3
Before the contractor shall be allowed to participate in the review process, the contractor
shall have disclosed any actual or potential conflicts of interest and shall have signed and
submitted to NCERQA a Conflict of Interest/Confidentiality Form. The contractor is directed to
assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular
reviewer. Upon receipt of an approved Purchase Order (PO), NCERQA will send the following
items to the contractor:
a) A copy of the Purchase Order or the Purchase Order number
b) The applications assigned to the contractor
c) For grants, a set of abstracts for all the applications being reviewed by the panel
d) For persons reviewing grant applications, a copy of the pertinent section(s) of the
solicitation package to provide background information; for persons reviewing
fellowship applications, information on how to access the solicitation package on
the Internet
e) A sample evaluation form to help the contractor prepare an acceptable evaluation
form for each assigned application
f) A blank evaluation form for each assigned application and the criteria for
completing the form and determining the overall rating
g) A blank and sample invoice as well as instructions for completing and submitting
the invoice to EPA
h) Information on the points of contact for additional information (e.g., NCERQA's
SRA)
i) Logistics information on the location and time of the panel discussions
NCERQA will transmit the above items under a cover letter. In this cover letter,
NCERQA will provide additional details about each item, including (as needed) more specific
instructions for the set of applications assigned to the reviewer.
Each contractor shall be responsible for making his/her own travel reservations for hotel
and transportation.
3) Reviewer Tasks
a) Review the assigned applications using the guidance provided with NCERQA's
evaluation form.
b) Submit completed evaluation forms to the SRA designated in the cover letter
immediately following the panel discussions for the applications assigned to the contractor.
THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY
FOR EACH APPLICATION MUST SUPPORT AND BE CONSISTENT WITH THE
OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONS
-------
Page D-4 Peer Review Handbook
WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE
CONTRACTOR TO REDO THE FORM.
c) For those applications for which the contractor is serving as the panel's rapporteur,
submit a completed panel evaluation summary to the SRA designated in the cover letter prior to
leaving the panel meeting THE PANEL'S EVALUATION SUMMARY MUST BE TYPED
(ON-SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA) AND BE
CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS WHERE
THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE CONTRACTOR
TO REDO THE FORM.
d) Make own airline and hotel accommodations for participation in the panel review
meeting. Round-trip air fare must be a commercial REFUNDABLE ticket.
-------
Peer Review Handbook Page D-5
STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization Tool Report
Work Assignment No.:
Title: Peer Review of Prioritization Tool Report
Work Assignment Manager (WAM):
Name: John Q. Government Employee
Address: Office of Solid Waste
Phone No.: (202) 260-XXXX
Background:
The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the
process of implementing the Waste Minimization National Plan, announced by the Agency on
November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction
over waste management, in keeping with the policy stated in the 1984 amendments to the
Resource, Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act
(PPA). The Plan outlines major goals, objectives, and action items to achieve national reductions
in the generation of hazardous wastes.
One of the objectives of the Plan is to: "develop a framework for setting national
priorities; develop and distribute a flexible screening tool for identifying priorities at individual
facilities; [and] identify constituents of concern." This objective is a key building block in
implementing subsequent objectives of the Plan.
In September 1995, WMB formed the Waste Minimization Prioritization Team, which
includes representatives from EPA regions and states, to implement this objective. The Team
has worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools
that are currently available. The Team plans to summarize this work, along with its
recommendations, in a report (referred to herein as the Prioritization Tool report) that would be
available in draft form in July 1996.
WMB and the Team wish to obtain independent peer review of the Prioritization Tool
report prior to briefing EPA management. The report is being prepared with the support of ICF,
Inc.; therefore, for the peer review to be considered independent, it must be performed by another
contractor.
-------
Page D-6 Peer Review Handbook
Purpose and Scope of Work:
The purpose of this work assignment is to provide support to WMB and the Team in
finalizing the Prioritization Tool report by conducting an independent peer review of the report.
Work Statement:
Task 1 -Management work plan and budget
Within 15 days of CO approval of this work assignment, the contractor shall deliver a
management work plan including a proposed level of effort, schedule, and budget for all tasks.
Task 2 -Provide independent peer review of Prioritization Tool report
The contractor shall provide support to WMB and the Team in preparing the
Prioritization Tool report by performing an independent peer review of the report. The
contractor shall establish a panel of peer reviewers including three senior-level persons who
collectively have extensive expertise in particular areas to be identified by the WAM upon
approval of the work assignment.
Within three weeks of work assignment approval, receipt from the WAM of the necessary
qualifications of peer reviewers (in a TD), and receipt from the WAM of the peer review
"charge" (in a TD), whichever comes latest, the contractor shall identify the three peer reviewers
and prepare a memo that lists the names of the peer reviewers and their affiliations and includes
the peer reviewers' bio's. Within five weeks of WAM approval of the of the peer reviewers (via
a TD) and receipt of the draft Prioritization Tool report from the WAM (via a TD), whichever
comes later, the contractor shall conduct the peer review, assemble the peer review comments
and recommendations in a peer review report organized by charge question, prepare an
introduction to the peer review report with a clear and concise overview of the comments, and
attach to the peer review report any marginal comments the peer reviewers had on the
Prioritization Tool report.
It is not necessary that the peer reviewers jointly reach consensus on their findings and
recommendations, since there may be limited overlap in the peer reviewers' areas of expertise
and in the charge questions that they focus on. The contractor shall assume, for the purpose of
estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200
pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and
writing comments. EPA plans to provide the report to the contractor in mid-July.
-------
Peer Review Handbook
Page D-7
Deliverables and Schedule:
Task
1
2
3
Deliverable
Work plan and budget
Memo identifying peer reviewers
Peer review report
Schedule
Within 1 5 days of CO approval of work
assignment
Within 3 weeks of work assignment approval,
receipt of peer reviewer qualifications from
WAM, and receipt of charge from WAM,
whichever comes latest
Within five weeks of WAM approval of peer
reviewers and receipt of draft Prior itization
Tool report from WAM, whichever comes
later
Other Requirements:
CONTRACTOR COMMUNICATIONS
Upon approval of the Work Plan, the contractor shall maintain at least weekly
communications with the Work Assignment Manager regarding the status of work on the Work
Assignment.
CONFLICT OF INTEREST
The contractor must adhere to the following requirements:
1) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior
to commencement of any work, notify both the CO and PO of any actual or potential
organizational or personal conflicts of interest.
2) Provide a written certification, within 20 days of receipt of a Work Assignment,
QRT, or similar tasking document, that:
a) Either all actual or potential organizational conflicts of interest have been
reported to the CO or that no actual or potential organizational conflicts of
interest exist. The contractor is directed to assure that none of the
conflicts disclosed are so direct and substantial as to rule out a particular
reviewer.
b) All personnel who perform work under this Work Assignment or relating
to this Work Assignment have been informed of their obligation to report
personal and organizational conflicts of interest to the CO.
-------
Page D-8 Peer Review Handbook
c) The Contractor recognizes its continuing obligation to identify and report
any actual or potential conflicts of interest arising during performance of
this Work Assignment.
3) If an actual or potential organizational conflict of interest is identified during
performance under this Work Assignment, the Contractor shall immediately make a full
disclosure in writing to the CO. The disclosure shall include a description of action
which the Contractor has taken or proposes to take, after consultation with the CO, to
avoid, mitigate, or neutralize the actual or potential conflict of interest.
EXPENDITURE OF FUNDS/HOURS
In addition to the requirements of the contract, the contractor shall notify both the Project
Officer and the Work Assignment Manager when 75% of funds or hours for this Work
Assignment have been expended.
INFORMATION COLLECTION
Any other provision of this Work Assignment notwithstanding, the contractor shall not
proceed with any information collection where the same or similar information will be collected
from ten or more public respondents until written approval is received from the Contracting
Officer. This approval will cite an approval number from the Office of Management and Budget
as required by the Paperwork Reduction Act (PRA).
Only Federal agencies and their employees are exempt from the PRA definition of
"public respondent." State agencies and their employees are classified as "public respondents."
Soliciting similar information applies to any collection method, i.e., written, oral,
electronic, etc., and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc.
The PRA applies equally to "willing participants" and participation that is mandated by law.
Any question of applicability of the PRA shall be resolved by submitting a complete
description of the circumstances in a written request to the Contracting Officer. No collection
shall be undertaken until the Contracting Officer provides written notice to the Contractor as to
the applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not
initiate any collection until the requisite approval is received.
The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the
Interagency Reports Management Program as derived from 44 U.S.C. Chapters 29 and 31. All
work performed under this Work Assignment involving federal interagency reporting must be
done in full compliance with these GSA procedures.
-------
Peer Review Handbook Page D-9
CONFIDENTIAL BUSINESS INFORMATION
If this Work Assignment requires use of RCRA Confidential Business Information (CBI),
the contract must specifically authorize the contractor to have access to RCRA CBI and the
contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA CBI
Security Manual and in the contract. The contractor shall identify in the Work Plan budget all
estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as
it is no longer needed under this Work Assignment or before the expiration of the Work
Assignment, whichever occurs first.
PRINTING AND DUPLICATION
The contractor is prohibited from performing any printing under the Government Printing
and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000
impressions of a single-page document or 25,000 impressions of a multiple-page stand-alone
document, is limited to one color (black) copies, and does not exceed the maximum image size
of 10 3/4 by 14 1/4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA
WAM will determine in advance if the work can be performed more cost effectively and under
the job or time constraints at the EPA Print Shop. If the total number of photocopies for this
Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the
photocopying costs by task and deliverable.
WORK ASSIGNMENT/WORK PLAN BUDGETS
The contractor shall not exceed either the dollar or PL hour budget contained in the
approved Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not
exceed the PL hour budget of the QRT.
TECHNICAL DIRECTION
The Designated Work Assignment Manager (WAM) on this Work Assignment is
authorized to provide technical direction to the extent allowed under EPAAR (1552.237-71)
(APR 1984) (DEVIATION). Other than the Designated WAM, only the Project Officer and the
Contracting Officer are authorized to provide technical direction.
Technical direction includes:
(1) Direction to the contractor which assists the contractor in accomplishing the
Statement of Work
(2) Comments on and approval/acceptance of reports or other deliverables
-------
Page D-10 Peer Review Handbook
Technical direction must be within the contract and the Work Assignment statement of
work. The Project Officer and the WAM do not have the authority to issue technical direction
which (1) institutes additional work outside the scope of either the contract or this Work
Assignment; (2) constitutes a change as defined in the "Changes" clause; (3) causes an increase
or decrease in the estimated cost of the contract or Work Assignment; (4) alters the period of
performance or deliverable due dates; or (5) changes any of the other express terms or conditions
of the contract or Work Assignment.
Technical direction will be issued in writing or confirmed in writing within five (5)
calendar days after verbal issuance. The technical direction memorandum will be provided to the
contractor and copies will be forwarded to the Contracting Officer and the Project Officer. If the
contractor has not received written confirmation within five (5) calendar days of a oral issuance,
the contractor must so notify the Project Officer.
INHERENTLY GOVERNMENTAL FUNCTIONS
The contractor shall not perform any inherently governmental functions (IGF) under this
Work Assignment. If during the course of developing the plan of work, through receipt of
technical direction, or in carrying out the assignment any portion of the effort is considered to
possibly be an inherently governmental function, the contractor must immediately notify the
Project Officer and the Contracting Officer.
OCCUPATIONAL HEALTH AND SAFETY
Facility site visits conducted under a Work Assignment that include on-site inspections or
sampling must be conducted in full compliance with the Department of Labor, Occupational
Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440
(Occupational Health and Safety Manual).
TRAVEL COSTS
The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal
regulations in incurring allowable travel costs under this Work Assignment, and correspondingly
must at all times seek and obtain Government rates whenever available and observe current
subsistence ceilings.
QUICK RESPONSE TASKS
Each Quick Response Task (QRT) shall be confirmed in writing and approved by the
Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and the
-------
Peer Review Handbook Page D-ll
CO within two working days, giving a brief description of the plan of work, including best
estimate of hours (by P-level) and a break-out of costs to accomplish the task.
No task shall exceed a duration of 30 calendar days from start date to completion date.
The level of effort for each task shall be limited to a maximum of 250 labor hours.
Quick Response Task Requests do not change the dollar or professional labor hour
budgets of a Work Assignment.
-------
Page D-12 Peer Review Handbook
STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of Protozoa Method
Development Criteria Document
Period of Performance: Work Plan Approval to August 1, 1997
Work Assignment Manager: Sally Q. Government Employee
Office of Water
U.S. Environmental Protection Agency
LOE: 196 hours
SOW: 2.4
BACKGROUND INFORMATION:
The United States Environmental Protection Agency (EPA), Office of Water is charged with
protecting public health and the environment from adverse exposure to chemicals and microbials
in water media, such as ambient and drinking waters, wastewater/sewage sludge and sediments.
In support of this mission OW's Office of Science and Technology (OST) develops health
standards, health criteria, health advisories, and technical guidance documents for water and
water-related media. Under this work assignment, documents prepared by OST are to undergo
peer review.
Peer review is an important component of the scientific process. It provides a focused, objective
evaluation of a research proposal, publication, risk assessment, health advisory, guidance or other
document submitted for review. The criticism, suggestions and new ideas provided by the peer
reviewers stimulate creative thought, strengthens the reviewed document and confer credibility
on the product. Comprehensive, objective peer reviews leads to good science and product
acceptance within the scientific community.
Under this work assignment, the contractor will receive one document (Protozoa Method
Development Criteria Document) for peer review which is related to human health and
ecological effects.
STATEMENT OF WORK:
Task 1. The contractor shall develop a work plan to address all tasks in this work
assignment. The work plan shall describe the steps that will be taken by the
contractor to provide for peer review, including selection of peer reviewer
candidates with appropriate expertise, determining absence of conflict of interest,
document and reference distribution, establishing schedules, preparing the peer
review report, and submittal of the peer review package. Curriculum vitae for all
-------
Peer Review Handbook Page D-13
persons assigned to complete this work assignment shall be provided. All P
levels, hours and total costs for each task will be provided and costs greater than
$100.00 shall be itemized in detail.
Task 2. The contractor shall select a group of peer reviewers and determine their
availability for the task and absence of conflict of interest, and establish a
schedule for the peer review. The contractor is directed to assure that none of the
conflicts disclosed are so direct and substantial as to rule out a particular reviewer.
Three peer reviewers shall participate in the review. No single peer reviewer may
charge more than 40 hours to this task. It is fully acceptable for peer reviewers to
commit to less than 40 hours. The peer review will be conducted for the Protozoa
Method Development Criteria Document. Reviewers selected by and working for
the contractor shall be approved by the EPA Project Officer in writing prior to
their beginning work. Minimally, all peer reviewers shall be accomplished in
protozoan methods for sample recovery and analysis from water. Approval
submissions shall include the reviewers' names and curriculum vitae.
Task 3. The contractor shall arrange for the selected peer reviewers to review the EPA
document. Prepare the charge to the peer reviewers based on technical direction
received from the EPA WAM. Provide the peer reviewers with copies of the
candidate report and all relevant references and instruct the selected peer
reviewers to undertake the review. The WAM will provide the contractor with the
final version of the document to be reviewed.
Task 4. The contractor shall monitor peer reviewers' progress to assure timely completion.
The contractor shall collate peer review comments, and organize the comments in
the peer review "for comments" document. Provide the peer review document
and all materials submitted by the peer reviewers to the EPA WAM.
SCHEDULE AND DELIVERABLES:
Task 1. (Work Plan) 15 days after receipt of work assignment
Task 2. 1 week after work plan approval
Task 3. 1 week after selection of peer reviewers
Task 4. 1 week after receiving comments from the peer reviewers
TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable
to this work assignment must be submitted and approved by the project officer.
-------
Page D-14 Peer Review Handbook
THIS PAGE LEFT INTENTIONALLY BLANK
-------
Peer Review Handbook Page E-l
APPENDIX E
References Concerning Peer Review
American Chemical Society and the Conservation Foundation (1985) Issues in Peer Review of
the Scientific Basis for Regulatory Decisions., Washington, DC, November 1985.
Browner, C. (1994) Peer Review Program, Washington, DC, Memorandum issued June 7, 1994.
(NOTE: Attached as Appendix A to this Handbook)
Chubin, D. (1994) Grants peer review in theory and practice, Evaluation Review 18: 20-30.
Chubin, D. and E. Hackett (1990) Peerless Science: Peer Review and Science Policy, Albany,
NY: State University of New York Press.
Jasanoff, S. (1990) The Fifth Branch: Science Advisors as Policymakers, Cambridge, MA:
Harvard University Press.
Kostoff, R. (1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual
Meeting, Baltimore, MD, February 9, 1996.
National Environmental Policy Institute (1996) Enhancing the Integrity and Transparency of
Science in the Regulatory Process, Washington, DC: National Environmental Policy
Institute, Fall 1996.
National Research Council (1995) Interim Report of the Committee on Research and Peer
Review in EPA, Washington, DC: National Academy Press, March 1995.
Reilly, W. (1993) Peer-review Policy, Washington, DC, Memorandum issued January 19, 1993.
Science Advisory Board (1992) Safeguarding the Future: Credible Science, Credible Decisions,
Washington, DC, SAB Report issued March 1992.
Spitzer, H. (1995) Peer Review Practices in the Federal Government, Bethesda, MD:
Environmental Network, report prepared for the American Industrial Health Council,
April 26, 1995)
-------
Page E-2 Peer Review Handbook
U.S. General Accounting Office (1994) Peer Review: EPA Needs Implementation Procedures
and Additional Controls, GAO/RCED-94-89, Washington, DC: U.S. Government
Printing Office, February 1994.
U.S. General Accounting Office (1996) Peer Review: EPA 's Implementation Remains Uneven,
GAO/RCED-96-236, Washington, DC: U.S. Government Printing Office, September
1996.
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
------- |