United States
Environmental Protection
Agency
Office of Science Policy
Office of Research and Development
Washington, DC 20460
EPA100-B-00-001
December 2000
www.epa.gov
&EPA Science Policy Council
HANDBOOK
EE
2nd Edition
-------
EPA 100-B-OO-OOl
December 2000
U.S. Environmental Protection Agency
PEER REVIEW HANDBOOK
2nd Edition
Prepared for the U.S. Environmental Protection Agency
by members of the Peer Review Advisory Group,
a group of EPA's Science Policy Council
Principal Authors
Kerry L. Dearfield, Ph.D. A. Robert Flaak
Office of Science Policy EPA Science Advisory Board
Office of Research and Development Office of the Administrator
Major Contributors
Jean C. Schumann Nancy W. Wentworth
Office of Solid Waste Office of Environmental
and Emergency Response Information
Roland B. Hemmett, Ph.D. Arnold M. Kuzmack, Ph.D.
Region 2 Office of Water
Brett Snyder
Office of Policy, Economics,
and Innovation
Science Policy Council
U.S. Environmental Protection Agency
Washington, DC 20460
-------
Page ii Peer Review Handbook
DISCLAIMER
This document has been reviewed in accordance with U.S. Environmental Protection
Agency policy and approved for publication and distribution to the Agency. Mention of trade
names or commercial products does not constitute endorsement or recommendation for use.
-------
Peer Review Handbook Page iii
TABLE OF CONTENTS
FOREWORD Page xiii
ACKNOWLEDGMENTS Page xvii
SUMMARY OF THE PEERREVIEW PROCESS Page 1
Figure 1 - Flowchart for Planning a Peer Review Page 2
Figure 2 - Flowchart for Conducting a Peer Review Page 3
Figure 3 - Flowchart for Completing a Peer Review Page 4
Manager's Planning Checklist for Peer Review Page 5
PEER REVIEW GUIDANCE Page 7
1. THE NEED FOR PEER REVIEW Page 9
1.1 Overview Statement Page 9
1.2 Understanding Peer Review Page 9
1.2.1 Why use Peer Review? Page 9
1.2.2 What is Peer Involvement? Page 10
1.2.3 What is Peer Review? Page 10
1.2.4 What is Peer Input? Page 10
1.2.5 How is Peer Review Different from Peer Input? Page 11
1.2.6 Can Someone Who Provided Peer Input Become an
Independent Peer Reviewer for the Same Work Product Later
in the Process? Page 11
1.2.7 Can the Same Peer Reviewer be Used More than Once if a
Product will Be Peer Reviewed More than Once, and can the
Same Peer Reviewer be Used Again and Again for Different
Products? Page 12
1.2.8 How is Peer Review Different from Public Comment? Page 12
1.2.9 How is Peer Review Different from Stakeholder Involvement? .. Page 12
1.2.10 What Role does Peer Review have in the Regulatory
Development Process? Page 13
1.2.11 What Role does Peer Review have in Regulatory Negotiations? . Page 14
1.3 Annual Agency Reporting Requirements Page 14
1.3.1 What are the Annual Reporting Requirements? Page 14
1.3.2 What Listings are Required for the Annual Reporting? Page 14
1.3.3 When will the Handbook Itself be Revised? Page 17
-------
Page iv Peer Review Handbook
1.4 The Roles of People and Organizations in Peer Review Page 17
1.4.1 Who is Ultimately Accountable for Peer Review? Page 17
1.4.2 Who are the Agency Staff involved in Peer Review? Page 17
1.4.3 Who are the Decision Makers & What are Their
Responsibilities? Page 17
1.4.4 Who are the Peer Review Leaders & What are Their
Responsibilities? Page 19
1.4.5 Who are the Peer Review Coordinators & What are Their
Responsibilities? Page 20
1.4.6 Who are the Peer Reviewers? Page 21
1.4.7 What are the Responsibilities of Peer Reviewers? Page 21
1.4.8 What is an Independent Peer Reviewer? Page 21
1.4.9 When does an Agency Internal Peer Reviewer Qualify as
Independent? Page 22
1.4.10 What is a Peer Review Panel? Page 22
1.4.11 What is a Subject Matter Expert? Page 22
1.4.12 What is the Role of the Science Policy Council (SPC)? Page 23
1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)? . Page 23
1.4.14 What is the Role of the Office of Research and
Development (ORD)? Page 23
2. PLANNING A PEER REVIEW Page 25
2.1 Overview Statement Page 25
2.2 Determining Which Work Products to Peer Review Page 25
2.2.1 What are Scientific and Technical Work Products? Page 25
2.2.2 What Scientific and Technical Work Products Need Peer
Review? Page 26
2.2.3 How Does One Determine Whether a Scientific and/or
Technical Work Product is "Major"? Page 26
2.2.4 What Economic Work Products Need Peer Review? Page 27
2.2.5 Should Economic Work Products Prepared in Support of
Regulations that are Classified as "Major" or "Economically
Significant" be Peer Reviewed? Page 28
2.2.6 What Other Economic Work Products Might Benefit from
Peer Review? Page 29
2.2.7 What Other Social Science Work Products Need Peer Review? . . Page 30
2.2.8 How Should Peer Review be Handled for Products Developed
under an Interagency Agreement (IAG)? Page 31
2.2.9 Should Products from Contracts, Grants, and Cooperative
Agreements Receive Peer Review? Page 31
-------
Peer Review Handbook Page v
2.2.10 How Does Peer Review Apply to Products Generated Through
EPA Grants or Cooperative Agreements? Page 31
2.2.11 Can the Recipient of a Grant or Cooperative Agreement Use
Agreement Funds to Pay Peer Reviewers of their Work Products? Page 34
2.2.12 Do Products Generated under EPA Grants or Cooperative
Agreements Need to be Reported in the Peer Review Product
Tracking (PRPT) Database? Page 34
2.2.13 Should Site Specific Decisions be Subject to Peer Review? Page 35
2.2.14 Should NEPA Products (e.g., EISs) be Subject to Peer Review? . Page 35
2.2.15 Should Environmental Regulatory Models be Peer Reviewed? .. Page 36
2.2.16 Is Peer Review Needed for Other Organization's Work Products
that Have been Submitted to EPA for Use in Decision Making? . Page 36
2.2.17 Can Work Products That are Not Determined to be Major Still
be Peer Reviewed? Page 37
2.3 Determining Which Work Products Do Not Receive Peer Review Page 37
2.3.1 Are There Circumstances When a Major Work Product is Not
Peer Reviewed? Page 37
2.3.2 What Products Normally Do Not Need Peer Review? Page 38
2.3.3 Do Voluntary Consensus Standards Require Peer Review? Page 39
2.4 Choosing a Peer Review Mechanism Page 39
2.4.1 How Do You Determine the Appropriate Peer Review
Mechanism? Page 39
2.4.2 What are Examples of Internal Peer Review? Page 41
2.4.3 What are Examples of External Peer Review? Page 41
2.4.4 What is the Role of Peer Review by a Refereed Scientific
Journal? Page 42
2.4.5 Do Agency Work Products Become Candidates for Peer
Review when Peer Reviewed Journal Articles are Used in
Support of that Work Product? Page 43
2.4.6 When and How Often Should Peer Review Occur? Page 43
2.4.7 What Factors are Considered in Setting the Time Frame for
Peer Review? Page 44
2.4.8 Which Office/Region or Other Agency is Responsible for
Conducting the Peer Review? Page 45
2.5 Creating the Peer Review Record Page 45
2.5.1 What is the Peer Review Record? Page 45
2.5.2 How Can the Peer Review Record Improve the Peer
Review Process? Page 45
2.5.3 What Should Be in the Peer Review Record? Page 45
2.5.4 What Should I Do with a Peer Review Record That Pertains to a
Rulemaking Action? Page 46
-------
Page vi Peer Review Handbook
2.5.5 When Should the Peer Review Record Building Process Begin? . Page 46
2.5.6 What are the Differences in Record Keeping for a Review by an
Individual Compared to a Panel? Page 47
2.5.7 Where Should the Peer Review Record be Kept and For
How Long? Page 47
2.5.8 Are Internal Peer Review Comments Included in the Peer
Review Record? Page 47
2.6 Budget Planning Page 48
2.6.1 What Budgetary Factors Should I Consider in a Peer Review? . . . Page 48
2.6.2 What Input is Needed for the Annual Budget Formulation and
Budget Execution Process? Page 48
2.7 Legal Considerations Page 49
2.7.1 Are There Legal Ramifications From the Peer Review Policy? .. Page 49
2.7.2 Is Legal Advice Needed? Page 49
2.8 Federal Advisory Committee Act (FACA) Considerations Page 49
2.8.1 When Do FACA Requirements Apply to EPA-Run
Peer Reviews? Page 50
2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? Page 50
2.8.3 How Do I Ensure that a Contractor-Run Peer Review Does
Not Become Subject to FACA? Page 51
3. CONDUCTING A PEER REVIEW Page 53
3.1 Overview Statement Page 53
3.2 Charge to the Peer Reviewers Page 53
3.2.1 What is a Charge? Page 53
3.2.2 What are the Essential Elements of a Charge? Page 53
3.2.3 Where Can I Get an Example of a Charge? Page 54
3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer
Reviewers? Page 54
3.2.5 Who Writes the Charge When I Hire a Contractor to Conduct
the Peer Review? Page 54
3.2.6 Is it Okay For Me to Ask a Contractor to Develop the Charge
to the Peer Reviewers? Page 55
3.3 Time Line Page 55
3.3.1 What are the Factors in Scheduling a Peer Review? Page 55
3.4 Selection of Peer Reviewers Page 55
3.4.1 What are Considerations for Selecting Peer Reviewers? Page 55
3.4.2 Where Do I Find Peer Reviewers? Page 56
3.4.3 Are External or Internal Peer Reviewers Preferred? Page 57
3.4.4 What is Important in the Mix of a Peer Review Panel? Page 57
3.4.5 What is a Conflict of Interest? Page 58
-------
Peer Review Handbook Page vii
3.4.6 What Techniques Help Ensure Disclosure and Appropriate
Resolution of Conflicts of Interest? Page 59
3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews? . Page 61
3.4.8 Are There Constraints to Selecting Peer Reviewers? Page 61
3.4.9 If State Employees are Used as Peer Reviewers, Can EPA
Pay Them for this Service? Page 62
3.5 Materials for Peer Reviewers Page 62
3.5.1 What Instructions Do You Give Peer Reviewers? Page 62
3.5.2 What Materials Should be Sent to Peer Reviewers? Page 63
3.5.3 How Closely can EPA Interact with Peer Reviewers During
the Review? Page 64
3.6 Peer Review Services Page 65
3.6.1 What are Gratuitous Services for Peer Review? Page 65
3.6.2 Can I Use a Contract to Obtain Peer Review Services? Page 66
3.6.3 How Do I Write a Statement of Work for Contracts? Page 66
3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive
Activities? Page 67
3.6.5 What are Some Management Controls for Contracts? Page 67
3.6.6 Can I Identify and/or Select Peer Reviewers When Using
a Contract? Page 70
3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer
Reviewers? Page 70
3.6.8 How is Travel Handled with Contracts or Purchase Orders? .... Page 72
3.6.9 How is Travel Handled with Special Government Employees? . . Page 72
4. COMPLETING A PEER REVIEW Page 73
4.1 Overview Page 73
4.2 Final Work Product Page 73
4.2.1 How Do I Incorporate Peer Review Comments into the
Final Work Product? Page 73
4.2.2 What Actions are Potentially Forthcoming from Peer Review? . . Page 74
4.2.3 What Should the Final Work Product Say About the Peer
Review Process? Page 74
4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous? Page 75
4.3 Completing the Peer Review Record Page 75
4.3.1 How Do I Complete the Peer Review Record? Page 75
4.3.2 Where Should the Peer Review Records be Kept, and for
How Long? Page 76
4.3.3 Is Information Regarding a Peer Review Subject to Release
under FOIA? Page 77
-------
Page viii Peer Review Handbook
SUBJECT INDEX Page 79
COMMONLY USED ACRONYMS Page 81
APPENDIX A - U.S. ENVIRONMENTAL PROTECTION AGENCY PEER
REVIEW POLICY Page A-l
APPENDIX B - SOUND SCIENCE AND PEER REVIEW IN RULEMAKING Page B-l
APPENDIX C - EXAMPLES OF CHARGES Page C-l
CHARGE EXAMPLE 1 - Framework for Developing a Living Resources
Research and Monitoring Plan for the Peconic Estuary Page C-2
CHARGE EXAMPLE 2 -EPA Science Advisory Board (SAB) Review of the
Environment Monitoring and Assessment Program (EMAP)
Research Strategy and Research Plan Page C-5
CHARGE EXAMPLE 3 - EPA Science Advisory Board Review of the
Agency's National Risk Management Research Laboratory's
(NRMRL) Program Page C-6
CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the
Technical Aspects of the Multi-Agency Radiation Survey and Site
Investigation Manual (MARSSM) Page C-7
CHARGE EXAMPLE 5 - Economics - Benefits Transfer from Adults
to Children Page C-8
CHARGE EXAMPLE 6 - Economics - Study of Municipal Government
Costs and Financial Impacts from Environmental Regulations Page C-9
CHARGE EXAMPLE 7 - Economics - Valuation of Fatal Cancer Risks Page C-l 1
CHARGE EXAMPLE 8 - Hudson River PCBs Site Reassessment RI/FS
Modeling Approach Charge: Peer Review 1 Page C-l5
CHARGE EXAMPLE 9 - IRIS Pilot Program - Instructions to Peer
Reviewers for Reviewing IRIS Summaries and Supporting
Documentation Page C-l8
CHARGE EXAMPLE 10 - Charge to Reviewers for the WTI Draft Final
Risk Assessment Page C-21
APPENDIX D - GUIDANCE ON REQUESTING A REVIEW BY THE US EPA
SCIENCE ADVISORY BOARD (SAB) Page D-l
APPENDIX E - EXAMPLE STATEMENTS OF WORK FOR CONTRACTS Page E-l
STATEMENT OF WORK -EXAMPLE 1 - Statement of Work: Technical
Review Contractor for Panel Review of Assistance Agreement or
Fellowship Applications Page E-2
-------
Peer Review Handbook Page ix
STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization
Tool Report Page E-5
STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of
Protozoa Method Development Criteria Document Page E-12
APPENDIX F - USEFUL FORMS Page F-l
PEER REVIEW CONFLICT OF INTEREST INQUIRY Page F-2
PEER REVIEW CHECKLIST FOR DETERMINING
WHETHER A WORK PRODUCT NEEDS PEER REVIEW Page F-3
PEER REVIEW CHECKLIST FOR CONDUCTING A PEER REVIEW Page F-7
VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT . . . Page F-l4
APPENDIX G Page G-l
REFERENCES CONCERNING PEER REVIEW Page G-l
-------
-------
Peer Review Handbook Page xi
FOREWORD (2nd Edition)
It has been three years since EPA's Science Policy Council (SPC) issued the 1st Edition
of the Peer Review Handbook (January 1998). The original Handbook has been used extensively
by personnel throughout the Agency for the peer review of Agency work products. At the time it
was issued, we planned a two to three-year period during which Agency staff could use the
Handbook and provide feedback on its utility, and at the same time, identify questions or issues
not originally addressed.
To date, we have received numerous comments and suggestions from regional and
headquarters personnel concerning the Handbook. These comments have been uniformly
positive and encouraging. The comments indicate that the Handbook contributed greatly to the
Agency goal of sound science and substantially improving EPA's peer review process.
We have collected the comments and additional questions, discussed them with the SPC's
Peer Review Advisory Group and Agency Peer Review Coordinators, and developed revised and
new answers to the questions received. In addition, we also incorporated many revisions in
response to recommendations from the EPA Science Advisory Board (SAB; from "An SAB
Report: Review of the Peer Review Program of the Environmental Protection Agency" a review
by the Research Strategies Advisory Committee (RSAC) of the SAB; report #EPA^AB-RSAC-
00-002 dated November 1999), the EPA Office of the Inspector General (OIG; from "EPA's
Selection of Peer Reviewers"; report #1999-P-217 dated September 29, 1999), and the National
Research Council (NRC; from "Strengthening Science at the U.S. Environmental Protection
Agency: Research Management and Peer Review Practices"; report issued 2000). After revising
the Handbook , the revisions were circulated extensively throughout the Agency for final
comment before the SPC approved the 2nd Edition of the Peer Review Handbook at their
October 17, 2000 meeting.
While we strived to make the Peer Review Handbook a definitive source of information
about peer review and its processes, we need to sound one precautionary note. During the
revision process, we heard many requests to put as much detail as possible into the Handbook to
make it a one stop source for all things related to peer review. We have attempted to do this as
much as possible; however, much of the detail in several instances delves into areas for which
there is specialized expertise elsewhere in the Agency, particularly contracting and legal issues.
If we provide all the detail requested in many of these areas, the Handbook would turn into a
contract manual or legal book, something we did not want to do. Since many of these processes
(like contracting) are updated frequently, we did not want to outdate the Handbook
unnecessarily. Also, we felt if the Handbook became much bigger than it is now, its utility and
-------
Page xii
Peer Review Handbook
ease of use would decrease. We have provided pointers and contacts where necessary in the
Handbook to help Agency people deal with these specific issues in relation to peer review.
We are proud to bring the 2nd Edition to you for your use in peer review at EPA. The
Handbook represents a large Agency collective effort to improve its science to the highest levels
possible. Peer review done correctly significantly adds to our goal of sound and credible science
that underlays all Agency decisions and actions.
Kerry Dearfield and Robert Flaak
Co-Chairs, Peer Review Advisory Group (PRAG)
Science Policy Council
Science Policy Council
W. Michael McCabe, OA, Chair
Donald Barnes, OA
Tudor Davies, OW
Sylvia Lowrance, OECA
William Muszynski, Region 2
Michael Shapiro, OSWER
Ramona Trovato, OCHP
Norine Noonan, ORD, Vice-Chair
Robert Brenner, OAR
Steven Galson, OPPTS
Albert McGartland, OPEI
Michael Ryan, OCFO
Elaine Stanley, OEI
Science Policy Council Steering Committee
Donald Barnes, OA
Reginald Cheatham, OEI
William Farland, ORD
Penelope Fenner-Crisp, OPPTS
Jerri-Anne Garl, Region 5
Roland Hemmett, Region 2
Carl Mazza, OAR
Jennifer Orme-Zavaleta, ORD
Larry Reed, OSWER
Rosemarie Russo, Region 4
Mary Ellen Weber, OPPTS
William Wood, ORD
Michael Brody, OCFO
Patricia Cirone, Region 10
Michael Feldman, OCFO
Michael Firestone, OCHP
Peter Grevatt, OSWER
Kate Mahaffey, OPPTS
James Nelson, OGC
Peter Preuss, ORD
Joseph Reinert, OPEI
Vanessa Vu, ORD
Jeanette Wiltse, OW
Science Policy Council Staff
Edward Bender
Kerry Dearfield
James Rowe
-------
Peer Review Handbook Page xiii
FOREWORD (From 1st Edition)
EPA's Science Policy Council (SPC) has organized this Peer Review Handbook as
guidance to EPA staff and managers on the organization and conduct of peer review pursuant to
the Administrator's June 7, 1994 Peer Review Policy statement. The Handbook is based in part
on the central themes set forth in the Policy statement (see Appendix A for the full policy):
Major scientifically and technically based work products related to Agency decisions
normally should be peer reviewed. Agency managers within Headquarters, Regions,
laboratories, and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope, and
timing. These decisions are made in conformance with program goals and priorities,
resource constraints, and statutory or court-ordered deadlines. For those work products
that are intended to support the most important decisions or that have special importance
in their own right, external peer review is the procedure of choice. Peer review is not
restricted to the penultimate version of work products; in fact, peer review at the
planning stage can often be extremely beneficial.
In addition, the Handbook augments these themes by stating and explaining widely-
accepted principles and practices that have long guided peer review in the universities, in private
research organizations, and at the EPA and other government agencies.
The goal of the Peer Review Policy and this
Handbook is to enhance the quality and credibility of
Agency decisions by ensuring that the scientific and
technical work products underlying these decisions
receive appropriate levels of peer review by independent
scientific and technical experts. To serve this goal, the
Handbook provides information and outlines procedures
in several different areas:
The Policy requires peer review
of the basis of the decision (Le.,
the underlying major scientific
and/or technical work products),
not the decision itself.
basic principles and definitions, including distinctions between peer review and
peer input, public comment, and stakeholder involvement;
preparing for peer review, including identifying work products, identifying
appropriate peer review mechanisms, and identifying qualified experts; and,
conducting and completing peer reviews, including materials required for peer
review, creating a peer review record, and utilizing peer review comments.
-------
Page xiv Peer Review Handbook
This Handbook has three parts. The first contains flow charts that outline the key steps in
conducting a peer review, along with a managers' checklist for planning peer reviews. The
second part contains peer review guidance detailing the procedures outlined in the flow charts in
a question and answer format. The third part contains Appendices including the 1994 Peer
Review Policy and examples to help perform quality peer reviews. Some procedures outlined in
the 1994 Policy have been completed while others are continuing to change in line with Agency
experience. These changes have been incorporated into the current Handbook.
Dorothy E. Patton, Ph.D.
Executive Director, Science Policy Council
-------
Peer Review Handbook Page xv
ACKNOWLEDGMENTS
Compiling a comprehensive handbook that incorporates the most relevant, as well as
some of the obscure policies and guidance, regarding peer review and its application throughout
EPA has been a herculean task involving not just the members of the Peer Review Advisory
Group, but many others as well.
We want to acknowledge the contributions of the headquarters and regional Peer
Review Coordinators and their respective office and division coordinators, the Science Policy
Council (SPC) and its Steering Committee, the Office of Environmental Information's (OETs)
Quality Assurance Staff, Office of Science Policy (OSP) Program Support Staff, and the many
other individuals who provided comments on the revisions to the current edition.
In particular, we want to acknowledge the following individuals who provided critical
expertise and timely revisions (often several times) to various sections of the Peer Review
Handbook: Hale Hawbecker (Office of General Counsel), Theresa Trainor (Office of Water),
and Bruce Bakaysa (Office of Acquisition Management).
Finally, we want to acknowledge the tireless efforts and mentorship of the recently
retired Executive Director of the Science Policy Council, Dr. Dorothy Patton. She is one of the
pillars of the Agency's efforts to improve the quality of science and peer review and we are
indebted to her.
-------
-------
Peer Review Handbook Page 1
U.S. Environmental Protection Agency
SUMMARY OF THE PEER REVIEW PROCESS
This Section of the Peer Review Handbook contains flowcharts and
descriptions of the major steps in conducting a peer review. Cross references to
the appropriate section on Peer Review Guidance are shown in parenthesis and
bolded.
The Managers Planning Checklist for Peer Review (on page 5) is designed
to give Managers/Decision Makers a simple tool to help plan for a successful peer
review. It asks questions that should be considered by a manager or Decision
Maker during the peer review process to insure that necessary actions are taking
place. This checklist is also intended to be used by staff (especially Peer Review
Leaders and Coordinators) to inform managers and Decision Makers on some of
the key steps and considerations that are necessary in carrying out a successful
peer review. Expanded Check Lists which can be adapted for your use are also
included in Appendix F - Useful Forms (courtesy of Region V).
-------
Page 2
Peer Review Handbook
Figure 1 - Flowchart for Planning a Peer Review
Determination of "major scientific and
technical work product":
-- It is a scientific, engineering,
economic, or statistical document
(§ 2.2.1)
-- Determine if the work product is
major (§2.2.3, 2.2.4, 22.7)
— Major products meet certain criteria
(§ 2-2-3)
Work product is a candidate for peer
review :
-- Major work products are subject to
peer review (§ 2.2.2)
-- Some non-major work products have
to be evaluated to determine if peer
review is still warranted
(§2.2.17)
Work product is not a candidate for peer
review :
-- Most non-major work products are
typically not candidates for peer
review (§ 2.3.2)
-- Major work product consists only of
science previously peer reviewed and
adequate under the Agency's Policy
(§2.3.1)
-- Place work product on List C
(§ 1.3.2c))
If a work pro duct is subject to peer review:
-- Identify basis for charge (§ 3.2.1)
- Identify key staff (§ 1.4)
-- Create a peer review record (§ 2.5)
-- Ensure source of funding for the peer
review (§ 2.6)
-- Determine overall time frame for
peer review (§ 3.3.1)
-- Place the work product on
List B (§ 1.3.2b))
Worn Product 11 Not
Subject to Peer Re view
Policy
li It a Scientific
and Technical
A''ort Product?
Wort Product 11
Not Placed on any Lilt
Would tiie Wort
Product Still
Warrant Peer
Review?
li tiie Wort
Product Major?
Doe i 'A'0rt
Product Con il it
onlyoT science
Pne vlou ily Peer
Reviewed?
Wort Product 11 a Candidate
tor Peer Review
• Identify ES il i Tor Cflarge
• Wentify Key Staff
• Create Peer Re view Record
• Biiure Availability or Fundi
• Determine Overall Time Frame
-------
Peer Review Handbook
Page 3
Figure 2 - Flowchart for Conducting a Peer Review
Develop the charge (§ 3.2):
-- Determine which key issues to address
-- Include in peer review record (§ 2.5.3)
Select a peer review mechanism (§ 2.4)
- Internal (§ 2.4.2)
-External (§2.4.3)
-- Mail (e.g., Letter) review (§ 2.4.3)
-- Face to face meeting (§ 2.4.3)
-- One time or multiple meetings (§ 2.4.6)
-- Include logistical information in peer
review record (§ 2.5.3)
Determine the specific time line (§ 3.3):
-- When will the review be started
— What are the intermediate check points
-- What is the deadline for completion
Select peer reviewers (§ 3.4):
-- Determine sources of peer reviewers (§ 3.4.2)
-- Determine expertise required (§ 3.4.4)
-- Consider balance/address (§ 3.4.4)
— Consider conflicts of interest (§ 3.4.5 & 3.4.6)
-- Include documentation in peer
review record (§ 2.5.3)
Materials for the peer review (§ 3.5):
-- Obtain materials from Program for review
-- Prepare instructions for peer reviews (§ 3.5.1)
-- Forward materials to peer reviewers (§ 3.5.2)
-- Include copy of materials in peer
reviewrecord (§ 2.5.3)
Conduct the peer review
-- Obtain written comments from reviewers
-- Include in peer review record (§ 2.5.3)
Note: Some of these steps may occur concurrently.
Work Product
(Ready for Peer Review J
i
r
Develop the
Charge
i
r
Select the Peer
Review
Mechanism
i
r
Set the
Timelines,
including
Deadlines
i
r
Select Peer
Reviewers
i
r
Send Materials
to Peer
Reviewers &
Conduct
Review
i
r
1
Add to Peer
Review Record
r
Complete Peer Review
and obtain Comments
of Peer Reviewers
Continue with Completing
a Peer Review
-------
Page 4
Peer Review Handbook
Figure 3 - Flowchart for Completing a Peer Review
Evaluate comments from peer reviewers (§ 4.2.1)
-- Respond and react to comments
-- Obtain clarification, if needed
Brief your Decision Maker; obtain written
management approval of response to
comments (§ 4.2.1)
Comments that are considered, but not used (§ 4.3.1)
-- Determine why not used and document
-- Include comments in peer review record (§ 2.5.3)
Comments that are useful (§ 4.3.1)
— Revise the work product by
incorporating comments
-- Send revised work product back to
peer reviewers, if necessary
-- Include comments in peer review record (§ 2.5.3)
Finalize work product (§ 4.3.1)
-- Include in peer review record (§ 2.5.3)
-- Move work product from List B to List A
Receive Comments frorfi
Peer Reviewers
Are Pee
Reviewers
Comments
used in Wor
Product?
Finalize the Work
Product
Move the Work Produi t
from List B to List A
-------
Peer Review Handbook Page 5
Manager's Planning Checklist for Peer Review
1) Title of Work Pro duct:
2) What Decision/Rule/Regulation/Action Does this Work Product Support:
3) Determination of Major Scientific and Technical Work Products
D Is the work product scientific or technical yes no?
D Is the work product major or non-major?
4) Determining What Peer Review is Needed
D If major, peer review is needed?
D If not major, is peer review still needed?
D What peer review mechanism is needed (internal and/or external)?
D When does the review need to be done?
D How much time will be needed to conduct/complete the review?
D Are there court ordered deadlines or other constraints?
D Has senior management (AA/RA/others) been informed of progress/problems?
D What would constitute success for this review?
5) Determining the Resources for Peer Review
D What is the priority of this project relative to other projects in the same office?
D What resources are needed to conduct the review?
D What are the impacts of the review on personnel?
D Who will lead the peer review?
D Who will conduct the peer review?
D Who will maintain the peer review record?
D Where will the peer review record be kept?
D What mechanism will be used for the peer review?
D Has the charge been developed?
D Has internal and external coordination been initiated/completed?
D Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes
to the final work product) been made?
D How will results of the review be presented and addressed in the final work product (e.g., in a
preamble, in an accompanying appendix -- as well as changes in the work product itself)?
D Has the work product been entered onto List B or C, as appropriate?
6) Comments:
See Appendix F (Useful Forms) for expanded checklists
-------
-------
Peer Review Handbook Page 7
U.S. Environmental Protection Agency
PEER REVIEW GUIDANCE
-------
-------
Peer Review Handbook Page 9
1. THE NEED FOR PEER REVIEW
1.1 Overview Statement
Peer review at the U.S. Environmental Protection Agency (EPA) takes many different
forms depending on the nature of the work product, relevant statutory requirements, and office-
specific policies and practices. In January 1993, responding to recommendations in the report
Safeguarding the Future: Credible Science. Credible Decisions. Administrator William Reilly
issued an Agency-wide policy for peer review. Administrator Carol Browner reaffirmed the
central role of peer review in the Agency on June 7, 1994 and instituted an Agency-wide
implementation program (see Appendix A - USEPA Peer Review Policy). Following Agency-
wide implementation, office and region-specific standard operating procedures (SOPs) were
written and used from 1994 to 1998. In 1998, this Peer Review Handbook was created as a
single, centralized form of implementation guidance for Agency staff and managers. The Peer
Review Handbook was revised throughout calendar year 2000 and was reissued as Edition 2 (the
current version) in December 2000. Agency offices and regions, however, can still prepare brief,
tailored guidance that meets their individual needs to supplement the information in this
Handbook.
1.2 Understanding Peer Review
1.2.1 Why use Peer Review?
Peer review is intended to uncover any technical problems or unresolved issues in a
preliminary (or draft) work product through the use of independent experts. This information is
then used to revise that draft product so that the final work
product will reflect sound technical information and analyses.
Peer review is a process for enhancing a scientific or technical
work product so that the decision or position taken by the
Agency, based on that product, has a sound, credible basis. To
be most effective, peer review of a major scientific and/or
technical work product needs to be incorporated into the up-
front planning of any action based on the work product - this includes obtaining the proper
resource commitments (veovle and money) and establishing realistic schedules.
Peer review is not free;
however, not doing peer
review can be costly.
Peer review of major scientific and technical work products should not be looked upon as
another "hurdle" in the Agency decision making processes. While peer review requires that time
and resources be planned into the decision making process, the benefits justify the added cost.
Peer review enhances the credibility and acceptance of the decision based on the work product.
-------
Page 10 Peer Review Handbook
By ensuring a sound basis for decisions, greater cost savings are realized since decisions will not
be challenged as often and extra effort will not be required to go back and redo the work product.
So while peer review is not free, the cost of not doing peer review is usually much more
expensive. Furthermore, not conducting a peer review can potentially place the Agency in the
position of attempting to defend a scientifically invalid position — which can be very costly in
terms of both resources, and more importantly, credibility.
1.2.2 What is Peer Involvement?
As defined in the Peer Review Policy, peer involvement is the process whereby Agency
staff involve subject-matter experts from outside their program in one or more aspects of the
development of work products. Peer involvement, therefore, constitutes active outreach to and
participation by the broad scientific, engineering, and economics and social science communities
beyond the Agency (external) as well as within the Agency (internal). Typically, peer
involvement takes two general forms: peer input (ongoing discussions during the development of
the work product) and peer review (an evaluation of a workplan, preliminary draft or the like, or
most often, the critical, final objective expert evaluation of the work product).
1.2.3 What is Peer Review?
Peer review is a documented critical review of a specific Agency major scientific and/or
technical work product. The peer review is conducted by qualified individuals (or organizations)
who are independent of those who performed the work, but who are collectively equivalent in
technical expertise (i.e., peers) to those who performed the original work. The peer review is
conducted to ensure that activities are technically adequate, competently performed, properly
documented, and satisfy established quality requirements. The peer review is an in-depth
assessment of the assumptions, calculations, extrapolations, alternate interpretations,
methodology, acceptance criteria, and conclusions pertaining to the specific major scientific
and/or technical work product and of the documentation that supports them. Peer review may
provide an evaluation of a subject where quantitative methods of analysis or measures of success
are unavailable or undefined; such as research and development. Peer review is usually
characterized by a one-time interaction or a limited number of interactions by independent peer
reviewers. Peer review can occur during the early stages of the project or methods selection, or
as typically used, as part of the culmination of the work product, ensuring that the final product is
technically sound.
1.2.4 What is Peer Input?
Many Agency work products are developed with the input of various scientific and
technical experts inside and outside the Agency. Like the contribution made by peer reviewers,
peer input is valuable and enhances the scientific or technical basis of the products. Peer input,
-------
Peer Review Handbook Page 11
sometimes referred to as peer consultation, generally connotes an interaction during the
development of an evolving Agency work product, providing an open exchange of data, insights,
and ideas. Peer input may be characterized by a continued and iterative interaction with
scientific experts during work product development. A common example of peer input is the
input received from workgroup members during development of a product. Many Agency
products are developed through the efforts of a workgroup, which may include external experts,
such as State and Tribal representatives. These workgroup members have an active, ongoing
participation in developing the work product. Another example of obtaining peer input is of an
Agency office sending a draft work product to a list of stakeholder representatives for general
comments (stakeholder representatives often include experts who could be considered "peers").
1.2.5 How is Peer Review Different from Peer Input?
The key distinctions between peer input as described above and formal peer review are
the independence of the peer reviewers and their level of involvement. The goal of peer review
is to obtain an independent, third-party review of the product from experts who haven't
substantially contributed to its development. When experts have a material stake in the outcome
of the peer review (such as a regulated party) or have participated substantially in the
development of the product (such as a workgroup member), those experts' reviews may not
qualify as unbiased, independent peer review and may be better characterized as peer input.
Peer Input is not a
substitute for Peer Review
It is clear that peer input provides valuable
contributions to the development of the work product.
However, peer input does not substitute for peer review. Once
a work product is considered major, it is a candidate for peer
review and entered on List B (Candidate Products for Future
Peer Review -- see Section 1.3.2b)) — even though the work product may already have a
substantial amount of peer input. In other words, one cannot argue that a peer review is not
necessary if a major work product has received "enough" peer input. If the work product is not
considered major and has had peer input, it is entered on List C (Products for Which a Decision
has been made not to Peer Review -- see Section 1.3.2c)) with comments about the extent of the
peer input.
1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer
Reviewer for the Same Work Product Later in the Process?
Generally, the answer is no as that expert is no longer independent, but rather a
contributor to the work product. There may be special circumstances where the expertise is so
narrow that another peer reviewer isn't available. The Peer Review Leader (see Section 1.4.4)
will normally be responsible for making this determination and documenting the decision in the
peer review record.
-------
Page 12 Peer Review Handbook
1.2.7 Can the Same Peer Reviewer be Used More than Once if a Product will Be
Peer Reviewed More than Once, and can the Same Peer Reviewer be Used
Again and Again for Different Products?
There is no prohibition on using the same peer reviewer more than once on the same
product or for multiple products of the same office. However, it is a good idea to use some
different people each time where possible to provide a broader perspective. When using a
contractor to provide peer review services, you should recognize that contractors may have a
"pool" of reviewers that they use regularly. If the same peer reviewers are used repeatedly, they
may lose their independence (or the appearance of independence) from the work product(s). If a
peer reviewer is asked to participate in multiple reviews of the same product it should be noted
in the peer review record.
1.2.8 How is Peer Review Different from Public Comment?
Peer review and public comment are mutually exclusive. Public comment solicited from
the general public through the Federal Register or by other means is often required by the
Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited
for policy purposes. The Agency takes public comment on some strictly scientific products and
almost all regulatory decisions. Public commenters usually include a broad array of people with
an interest in the technical analysis or the regulatory
decision; some are scientific experts (which may
provide some peer input), some are experts in other
areas, and some are interested non-experts. The
critical distinction is that public comment doesn't
necessarily draw the kind of independent, expert
information and in-depth analyses expected from the peer review process. Public comment is
open to all issues, whereas the peer review process is limited to consideration of technical issues.
While it may be an important component of the review process, public comment does not
substitute for peer review.
Public comment does not substitute
for peer review.
1.2.9 How is Peer Review Different from Stakeholder Involvement?
Stakeholder involvement occurs when the Agency works with external interest groups
that have some stake in or concerns over the outcome of the technical work product or regulatory
position. This is an interactive process, working with
other agencies, industry groups, regulated-community
experts, environmental groups, other interest groups that
represent abroad spectrum of the regulated community,
etc., and usually strives for a consensus approach. The
goal of peer review, on the other hand, is to obtain an
Stakeholder involvement is not
a peer review mechanism.
-------
Peer Review Handbook Page 13
independent, third-party review for ensuring scientific integrity and technical credibility of the
work product that supports a policy or decision. Stakeholder involvement is not a peer review
mechanism and it does not substitute for peer review even though it adds value to the work
product.
1.2.10 What Role does Peer Review have in the Regulatory Development Process?
The peer review of scientific and technical work products that support rulemaking actions
is an important, fundamental step in the policy setting process and which affirms the credibility
of the Agency. Because new rules, and the work products supporting them, must often withstand
intense scrutiny by the general public and the stakeholders involved in the action, the peer review
process selected for such work products needs to be well planned and documented. The rule or
regulation itself is not subject to the Peer Review Policy. However, if the rule or regulation is
supported by a major scientific and/or technical work product, that work product should be peer
reviewed prior to its use in the rule (see Section 2.2 for determination of major work products).
The decision to peer review or not peer review any scientific and/or technical work product will
be documented through the Agency's annual peer review reporting process (see Section 1.3).
Remember, public comment and stakeholder involvement do not constitute peer review.
Tier 1 and Tier 2 rulemakings, are by definition important, major Agency rulemakings
within the Agency. Therefore, work products supporting Tier 1 and Tier 2 rules in particular
(including rules that are determined to be "significant" by OMB under Executive Order 12866
because they have an economic impact of $100 million or more) should be closely scrutinized to
determine whether they meet the criteria for major (see Section 2.2.3). Work products
supporting Tier 3 rulemakings may also be considered major and thus candidates for peer
review. External peer review is the procedure of first choice for a work product that is intended
to support a Tier 1 or Tier 2 rulemaking. Although acceptable in certain circumstances, any
decision to use an internal peer review mechanism for such work products would be the
exception rather than the rule. For work products supporting a Tier 3 rule, internal or external
peer review may be appropriate depending on the nature of the product and other factors (see
Section 2.4.1). For Tier 1 and Tier 2 rulemakings, the Final Agency Review/closure memo
needs to indicate that the Peer Review Policy was followed. For Tier 3 rulemakings, the action
memo needs to indicate that the Peer Review Policy was followed.
Analytic blueprints are required for Tier 1 and Tier 2 rulemakings, and are encouraged
for Tier 3 rulemakings; some individual EPA offices require it for Tier 3. For peer review
purposes, development of the analytic blueprint is the process whereby the project manager
identifies the supporting scientific and technical work products and identifies needed peer
review. In the directive from the Deputy Administrator (memo from Acting Deputy
Administrator Robertson, June 15, 1999: Sound Science and Peer Review in Rulemaking; see
-------
Page 14 Peer Review Handbook
Appendix B - Sound Science and Peer Review in Rulemaking), planning for peer review must be
specifically addressed in each new analytic blueprint. You are required to integrate peer review
into the rulemaking process and to include specific language addressing peer review in the action
memorandum. The analytic blueprint shows the schedule of the peer review in the context of the
schedule for the overall rulemaking. In general, peer review should be completed as early in the
process as practicable. Where possible, peer review of work products should be completed prior
to issuance of the proposed regulation. In some cases, support work products for final
regulations may require an additional peer review if those scientific and technical work products
change significantly after the public comment period.
1.2.11 What Role does Peer Review have in Regulatory Negotiations?
Regulatory negotiations are not candidates for peer review; however, to ensure final
decisions are based on sound and credible science, the major scientific and technical work
products that support the negotiation need peer review before the negotiation takes place.
1.3 Annual Agency Reporting Requirements
1.3.1 What are the Annual Reporting Requirements?
The Peer Review Coordinator (for each AA/RA; see Section 1.4.5) will organize an
annual review of all peer review activities and submit this information to the Office of Research
and Development (ORD). ORD will staff this function at the direction of the Deputy
Administrator. In the Deputy Administrator's annual call for submissions, guidance on format
and submission of this information will be provided. ORD will review the submissions for
completeness, i.e., all information is provided and products are accounted for each year. ORD
will then provide a review of the completeness of the information in the submissions through
consultation with the appropriate persons in each organization (see Section 1.4). ORD will then
consolidate the information and findings for the SPC and the Deputy Administrator. Any
conflicts arising from the review will be resolved by the Deputy Administrator. The due date for
the annual reporting will be announced each year in the annual call letter; however, for planning
purposes, it is normally due in the early summer. The Office of Environmental Information
(OEI) assists ORD in these annual reporting functions.
1.3.2 What Listings are Required for the Annual Reporting?
The Peer Review Product Tracking (PRPT) database is a Lotus Notes shared-database
developed to track and report peer review activities across the Agency. The database is the
single repository for product-specific peer review reporting and tracking and uses a common
reporting form for all reporting. Work products for which peer review is completed are reported
as List A (Peer Review Completed); candidate work products for future peer review are reported
-------
Peer Review Handbook Page 15
as List B (Peer Review Needed); and work products that do not meet peer review criteria are
reported as List C (Peer Review Not Needed). List D (Scientific Articles/Reports Peer Reviewed
Outside of EPA) was recently added largely to track the articles that EPA staff produce.
Four listings of products are required for the annual reporting:
a) List A - Products Peer Reviewed Since 1991
1) List A is a cumulative list of peer reviewed products from 1991 to the
present.
2) Each work product entered onto List A must include information
summarizing the peer review. The database will specify the needed
information. The entry must be electronically signed (verified) by the
Peer Review Leader, the Peer Review Coordinator, and the Decision
Maker involved (see Section 1.4.3 to help determine who the Decision
Makers are).
b) List B - Candidate Products for Future Peer Review
1) List B is a list of products that are expected to be peer reviewed in the near
future.
2) This List contains major scientific and technical work products and any
non-major scientific and technical work products for which peer review
has been deemed necessary or appropriate. All entries on List B must be
electronically signed (verified) by the Peer Review Leader, the Peer
Review Coordinator, and the Decision Maker involved.
3) Work products placed on List B remain on List B until they are either peer
reviewed (after which they are moved to List A) or a decision is made not
to peer review that work product (at which point it is moved to List C
including the rationale for not conducting peer review). If Ihe product is
terminated, it is moved to List C.
c) List C - Products for Which a Decision has Been Made Not to Peer Review
1) List C is a cumulative list of all scientific and technical work products that
do not receive peer review (see Section 2.3).
-------
Page 16 Peer Review Handbook
2) List C includes: (a) any major scientific and technical work products for
which a decision was made not to peer review; (b) those work products
which were originally placed on List B, but for which it was decided that
peer review was not necessary (e.g., the product was not used in decision
making; the project was canceled); and (c) all non-major scientific and
technical work products.
3) List C also includes several categories of work products that would not
normally receive peer review. It is not necessary to list these work
products individually on List C, however, the total number prepared by the
organization must be included on List C. The organization may need to
identify the individual products if requested (e.g., due to litigation, FOIA,
etc.). Such categories can include, but may not be limited to: chemical
action reports, RCRA permits, scientific analyses for Premanufacturing
Notices (PMNs) that are conducted on a routine basis and that do not
deviate from established practice, and NPDES permits.
4) Each work product on List C needs a brief description of the reason(s) it is
not being peer reviewed. All entries on List C must be electronically
signed (verified) by the Peer Review Leader, the Peer Review
Coordinator, and the Decision Maker involved.
d) List D - Scientific Articles and Reports That Are Peer Reviewed by Organizations
Outside of EPA
1) List D includes EPA authored scientific papers (articles) that are peer
reviewed by a credible refereed scientific journal (See Sections 2.4.4 and
2.4.5).
2) Listing such documents in this manner shows that these items have
already been peer reviewed, but differentiates them from Agency work
products. In addition, it gives EPA an opportunity to highlight the
extensive work it produces in the scientific literature.
3) It is not necessary to list these papers individually on List D, although
some organizations may choose to do so. However, at a minimum, the
total number prepared by the organization and peer reviewed by journals
must be included on List D. The organization may ultimately need to
identify the individual papers and where published, if requested (e.g., due
to litigation, FOIA, etc.).
-------
Peer Review Handbook Page 17
4) Once any of these articles or reports are used in an Agency work product,
that work product becomes a candidate for peer review and must be listed
on List B (or List C if a decision is made not to peer review the work
product).
5) Entries on List D do not require an electronic signature.
1.3.3 When will the Handbook Itself be Revised?
During each annual reporting cycle, suggestions for revisions to the Handbook should be
submitted. Periodically, a decision will be made by the Science Policy Council (SPC) on
whether to revise the Handbook, in part or total, or not based on the suggestions. The SPC will
then direct the Peer Review Advisory Group (PRAG) to draft the revision(s) for Agency
comment and SPC approval.
1.4 The Roles of People and Organizations in Peer Review
1.4.1 Who is Ultimately Accountable for Peer Review?
Under the June 7, 1994 Peer Review Policy (Appendix A - USEPA Peer Review Policy),
the Administrator has designated the Assistant Administrators and Regional Administrators
(AAs and RAs) to be accountable for implementing the Policy in their respective organizations.
The Deputy Administrator is ultimately responsible for peer review across the Agency and is the
final arbitrator of conflicts and concerns about peer review.
1.4.2 Who are the Agency Staff involved in Peer Review?
The principal Agency staff involved are Decision Makers (and their line managers), Peer
Review Leaders and Peer Review Coordinators. In addition, there are many other critical staff in
each office and region who have responsibility for peer review activities (e.g., office and division
peer review coordinators, technical information managers, and, of course, any Agency staff that
serve as internal peer reviewers). Finally, ORD has oversight, as designated by the Deputy
Administrator, for ensuring the Agency's Peer Review Policy requirements are met.
1.4.3 Who are the Decision Makers & What are Their Responsibilities?
The AA/RA is the ultimate Decision Maker for their organization and is accountable for
the decisions regarding the identification of major scientific and technical work products and the
mechanism(s) of peer review utilized for each of the products. The AA/RA may designate
Office Directors and/or Division Directors as the front-line Decision Makers.
-------
Page 18 Peer Review Handbook
Generally, the Decision Makers decide whether a work product is major and needs peer
review or not, and what peer review mechanism to use. Furthermore, the Decision Makers
commit the resources needed to ensure a proper peer review. Decision Makers are responsible
for ensuring that the peer reviews are properly performed and documented.
In order to ensure greater independence of peer reviews, it is necessary to strictly separate
management of work products from the actual peer review of those work products. Therefore,
the Decision Maker and the Peer Review Leader (see Section 1.4.4) for a work product should
never be the same person.
The Decision Maker needs appropriate training on how to manage the peer review
process. The Peer Review Coordinator for the office can advise the Decision Maker on how to
obtain the appropriate training; the training approach varies among the different EPA offices.
Specific responsibilities of the Decision Maker(s) are the following:
a) Determine which work products in their organization require peer review
b) Designate (in conjunction with the Project Manager) a Peer Review Leader to
organize the peer review
c) Provide advice, guidance, and support to the Peer Review Leader in the
preparation, conduct, and completion of the peer review
d) Ensure that sufficient funds are designated in the office's budget request to
conduct the peer review; also ensure that adequate resources and/or extramural
management support are available for the peer review
e) Establish a realistic peer review schedule
f) Designate the stage(s) of product development where peer review is appropriate
g) Ensure all relevant issues and comments raised by the peer reviewer(s) are
adequately addressed and documented for the record, and where appropriate,
incorporated into the work product that is used as basis for decision making
h) By signature, document the decisions made that are reported in the annual
reporting to the SPC and Deputy Administrator
i) Certify any decision NOT to peer review a product by signature on a List C
(Products for Which a Decision has been made not to Peer Review) submission
-------
Peer Review Handbook Page 19
1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities?
The Peer Review Leader is assigned by the Decision Maker to organize, conduct and
complete the peer review for a specific individual work product. The Peer Review Leader will
obtain the assistance and support of the Peer Review Coordinator (see below) as well as any
others within the Agency to help perform the peer review. The Peer Review Leader will be
chosen on a case by case basis depending on the work product needing peer review. The Peer
Review Leader cannot be the Decision Maker. The Peer Review Leader could be the Project
Manager for the work product.
The Peer Review Leader must have appropriate training on how to conduct a peer review
before conducting the peer review. The Peer Review Coordinator for the office can advise the
Peer Review Leader on how to obtain the appropriate training; the training approach varies
among the different EPA offices however, uniform training modules are available.
Specific responsibilities of the Peer Review Leader are these:
a) Keep the Decision Maker informed of the status of a given project; provide Peer
Review Coordinator with data for the annual report
b) Organize, conduct, and complete the peer review following Agency procedures
c) Establish and maintain the peer review record for the specific individual peer
review currently being performed (see Section 2.5); this includes providing the
peer review summary information in the Peer Review Product Tracking (PRPT)
Database for the Decision Maker to sign when the peer review is completed
d) Select the peer reviewers in consultation with others involved with the peer
review (e.g., Decision Maker) and ensure that conflict of interest issues are
addressed and documented in the peer review record
e) Advise peer reviewers of their responsibilities
f) Provide information to the Decision Maker (including all appropriate managers in
the Peer Review Leader's chain of command) on the charge, profile of peer
reviewers, the peer review comments, and a proposal on how to address the
comments. Obtain Decision Maker approval on the approach to responding to
peer reviewer comments. Clearly identify for the Decision Maker any peer
review comments that will not be addressed in the agreed upon approach.
-------
Page 20 Peer Review Handbook
g) Notify the Peer Review Coordinator that the peer review is completed for the
annual report
h) Archive the peer review record in a manner consistent with their organization's
archiving procedures
When a contractor is used to conduct a peer review, some of the above responsibilities
are assumed by the contractor (see Section 3.6).
1.4.5 Who are the Peer Review Coordinators & What are Their Responsibilities?
The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor
peer review activities in their respective organization or organizational unit. This person must be
of sufficient stature and judgment to have the access to and confidence of all levels of office or
regional management when needed. The Peer Review Coordinator is the main contact for their
organization; they can also direct interested parties to other persons/contacts in the office on
specific work products (e.g., Peer Review Leader).
Specific responsibilities of the Peer Review Coordinator are these:
a) General oversight responsibility for the Office's or Region's peer review process
b) Report peer review activities to the AA/RA
c) Help mediate difficult issues between their organization and others; if they can't
resolve issue, then bring the issue to the attention of the appropriate level
Decision Makers in each organization for resolution.
d) Function as the liaison with ORD and the Science Policy Council (SPC):
1) Represent office/region before the SPC
2) Advise ORD of any changes in the list of work products and peer review
mechanisms during the annual reporting, and when necessary, at other
times
3) Participate in Agency peer review training, workshops, etc., as requested
and disseminate this information to the organization; coordinate and/or
present training within their organization
-------
Peer Review Handbook Page 21
e) Submit information on organization's peer review candidates for each year as
requested (this is the annual reporting, see Section 1.3)
1) Generate and update Lists A, B, C, and D (See Section 1.3.2)
2) Assure the proper approval signature on the completed submission with
the accompanying explanation for any departures from the Policy
f) Establish procedures to assure that the required work product peer review
documentation (i.e., peer review record) is filed and maintained in an appropriate
manner (see Section 2.5)
g) Provide advice, guidance, and support to the various Peer Review Leaders for the
performance of the peer reviews
h) Distribute Agency-wide peer review guidance and materials to appropriate
office/region personnel, as requested
1.4.6 Who are the Peer Reviewers?
Peer reviewers are individuals who have technical expertise in the subject matter of the
work product undergoing peer review. Peer reviewers can come from EPA, another Federal
agency, or from outside of the Federal government.
1.4.7 What are the Responsibilities of Peer Reviewers?
Peer reviewers need to be willing participants in the peer review process — they should
agree to read all materials, participate fully, and protect confidential information that arises. Peer
reviewers should maintain the confidentiality of the product, perform the review in a timely
manner, and be unbiased and objective.
1.4.8 What is an Independent Peer Reviewer?
An independent peer reviewer is an expert who wasn't associated with the generation of
the specific work product either directly by substantial contribution to its development or
indirectly by significant consultation during the development of the specific product. The
independent peer reviewer, thus, is expected to be objective (See Sections 1.2.6 & 1.2.7 for
further information)
-------
Page 22 Peer Review Handbook
The quality of the peer review is
dependent on the competence and
independence of the reviewers.
Independence is freedom from institutional,
ideological, or technical bias regarding the issues
under review and is necessary for objective, fair, and
responsible evaluation of the work product. If a
selected reviewer has a particular scientific or
technical perspective, it may be desirable to balance
the review with peer reviewers with other perspectives. Ideally, peer reviewers should be free of
real or perceived confiicts-of-interest or there should be a balancing of interests among peer
reviewers. If there are potential conflicts of interest (real or perceived), they should be fully
identified to ensure a credible peer review. (See Sections 3.4.5 & 3.4.6 for further information).
1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent?
An Agency independent peer reviewer is one who comes from a different organizational
unit than the one where the review question or document originates. A different organizational
unit usually denotes, at minimum, a different office (i.e., above division level in programs; above
branch level in regions) within the organization. In particular, a reviewer shouldn't come from
within the chain of command, either upward or downward.
1.4.10 What is a Peer Review Panel?
A peer review panel can range from a few individuals to ten or more, depending on the
issue being investigated, the time available and any limitations on resources. Individuals who
serve as peer reviewers must have appropriate scientific and technical expertise such that the
review panel covers the broad spectrum of expertise required to treat the issues/questions
presented in the charge.
1.4.11 What is a Subject Matter Expert?
A subject matter expert is one who has specific scientific and technical expertise in the
matter under review. The importance of scientific and technical expertise in the subject matter is
obvious, however, knowledge or just "knowing" about the subject area isn't equivalent to
expertise in the subject matter. For Agency decisions, a multi-disciplinary group of experts
corresponding to the disciplines that contribute to complex Agency decisions is often necessary
for a full and complete peer review. For example, a risk assessment that relies on both animal
and human data usually requires experts in both areas for a complete review. For economic
analyses, experts from the corresponding economic disciplines are necessary.
-------
Peer Review Handbook Page 23
1.4.12 What is the Role of the Science Policy Council (SPC)?
According to the 1994 Peer Review Policy statement: "The Science Policy Council is
responsible for overseeing Agency-wide implementation. Its responsibilities include promoting
consistent interpretation, assessing Agency-wide progress, and developing recommendations for
revisions of the policy as necessary." The SPC meets its responsibilities through coordination
with the Peer Review Coordinators, the Peer Review Advisory Group (PRAG) and the Office of
Research and Development (ORD).
The SPC, PRAG and ORD are not responsible for identifying specific products for peer
review or determining the level of review or mechanism for that review; those functions are the
responsibility of management within each Office or Region.
1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)?
The Science Policy Council has created the Peer Review Advisory Group (PRAG) to
assist in the implementation of the Agency's Peer Review Policy. The primary role of the PRAG
is to provide interpretation of the policy and to assist the SPC and Agency Offices and Regions
in the annual update of the Peer Review Handbook.
1.4.14 What is the Role of the Office of Research and Development (ORD)?
The Deputy Administrator has designated the Office of Research and Development
(ORD) to provide oversight to the Program Offices and Regions in the collection and review of
information that is contained in the annual submission of Lists A, B, C, and D (for detailed
information, see Section 1.3). The Office of Environmental Information (OEI) assists ORD in its
oversight role.
-------
-------
Peer Review Handbook Page 25
2. PLANNING A PEER REVIEW
2.1 Overview Statement
Planning a peer review is a critical first step to ensure a successful peer review of a work
product. The initial step is to determine whether your work product requires peer review. Once
you have determined that a peer review will be conducted, the Decision Makers and Peer Review
Leaders need to plan an appropriate review. This includes identification of resources (budget
and personnel), the schedule for the completion of the peer review, the mechanism for peer
review, the choice of peer reviewers, and the development of the peer review record.
2.2 Determining Which Work Products to Peer Review
2.2.1 What are Scientific and Technical Work Products?
The first step in determining which work products require peer review, is to identify
products that are scientific and/or technical in nature. Scientific and technical work products are
used to support a research agenda, regulatory program, policy position or other Agency position
or action. Scientific and technical work products include economic and social science work
products. Categories of work products include, for example: risk assessments, technical studies
and guidance, analytical methods, scientific database designs, technical models, technical
protocols, statistical survey/studies, technical background materials, technical guidance (except
for guidance providing policy judgments), research plans, and research strategies.
Products that wouldn't be considered scientific and technical work products can include
those: that address procedural matters (e.g., planning, reporting, coordination, notification); that
are primarily policy statements (e.g., relocation policy); that are conference proceedings (unless
the proceedings are used as the scientific basis for an Agency action or decision); and that are
decision documents (e.g., Record of Decision (ROD) — the decision document itself is not
subject to the Peer Review Policy, but the underlying scientific and/or technical support work
product is a candidate for peer review). In addition, the following Agency documents are not
considered scientific and/or technical work products under the Peer Review Policy strategic
plans, analytic blueprints, and goals documents.
Making final determinations concerning which work products are scientific and technical
is a responsibility of the Decision Maker (See Section 1.4.3)
-------
Page 26 Peer Review Handbook
2.2.2 What Scientific and Technical Work Products Need Peer Review?
The principle underlying the Peer Review Policy is that all major scientific and technical
work products used in decision making will
be peer reviewed. The process for identifying „„ . , ,. , . , ..
i- i c i , . u • „ / . / When in doubt about whether to peer
which of these products is maior (and thus . , , . . .
... ,, • \ j i review a work product or not, always
a candidate for peer review) and then » •» ^ , •.. »•» „ f
, . . . . • c • -11 decide to make it a candidate for peer
determining the mechanism of review will
* 1 • 4. 4- • -4- • A 41, review,
take into account vanous cnteria and the
circumstances surrounding the use of that
work product. To maintain flexibility, the
Decision Maker(s) for peer review should consider the full field of possible work products to
identify those additional products that might still warrant peer review as well as the full spectrum
of peer review mechanisms for each product. Once a decision is made to perform peer review,
the product is listed in the annual submission of Candidate Products for Future Peer Review
(List B - see Section 1.3.2b)). If a decision is made not to peer review a scientific and technical
work product, the product is listed in List C - Products for Which a Decision has Been Made Not
to Peer Review (see Section 1.3.2c)).
2.2.3 How Does One Determine Whether a Scientific and/or Technical Work
Product is "Major"?
Determinations of a scientific and/or technical work product as "Major" will largely be
case-by-case. The continuum of work products covers the range from the obviously major,
which clearly need peer review, to those products which are not major and clearly don't need
peer review. The rest of the work products fall in-between those two distinctions. This "middle-
ground" probably represents the majority of work products, each of which needs to be evaluated
closely and be compared to certain criteria (see below). The Decision Maker needs to make a
judgment as to whether a work product meets the criteria for major or not. There is no easy
single yes/no test of major covering the whole continuum of work products. A rule of thumb to
remember — if there is any doubt about whether a work product needs peer review, then go ahead
and consider it a candidate for peer review (and place it on List B - Candidate Products for
Future Peer Review).
Scientific and technical work products that are used to support a regulatory program or
policy position and that meet one or more of the following criteria are candidates for peer
review:
a) Establishes a significant precedent, model, or methodology
b) Addresses significant controversial issues
-------
Peer Review Handbook Page 27
c) Focuses on significant emerging issues
d) Has significant cross-Agency/inter-agency implications
e) Involves a significant investment of Agency resources
f) Considers an innovative approach for a previously defined
problem/process/methodology
g) Satisfies a statutory or other legal mandate for peer review
Usually, a major scientific and/or technical work product supports a regulatory decision
or policy/guidance of major impact. Major impact can mean that it will have applicability to a
broad spectrum of regulated entities and other stakeholders, or that it will have narrower
applicability, but with significant consequences on a smaller geographic or practical scale. The
scientific and/or technical work that underlies many of the Agency's major rulemakings and
policy and guidance documents of general applicability would be designated "major" under this
scope of impact criterion because of their far-reaching or significant impacts.
The novelty or controversy associated with the work product helps determine whether it
is major or not. A major work product may be novel or innovative, precedential, controversial,
or emerging ("cutting edge"). An application of an existing, adequately peer reviewed
methodology or model to a situation that departs significantly from the situation it was originally
designed to address is a candidate for peer review. Similarly, a modification of an existing,
adequately peer reviewed methodology or model that departs significantly from its original
approach is a candidate for peer review. Determination of "significant departure" as used in this
Section is the responsibility of the Decision Maker.
In summary, a major scientific or technical work product has a major impact, involves
precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory
obligation to conduct a peer review.
2.2.4 What Economic Work Products Need Peer Review?
Economic analyses and reports are considered scientific and technical work products, and
as such they are candidates for peer review. The following economic work products will
normally be classified as major and will require peer review:
a) internal Agency guidance for conducting economic and financial analysis
-------
Page 28 Peer Review Handbook
b) new economic and financial methodologies that will serve as a principal method
or protocol used to conduct economic analyses within a program
c) unique or novel applications of existing economic and financial methodologies,
particularly those that are recognized to be outside of mainstream economic
practices
d) broad-scale economic assessments of regulatory programs, such as those required
by Congressional-mandates (e.g., the Clean Air Act reports to Congress on
benefits and costs)
e) new stated preference (e.g., contingent valuation) and revealed preference surveys
(e.g., recreational travel cost surveys) developed to assist in the economic analysis
of a regulation or program
f) new national surveys of costs and expenditures for environmental protection (e.g.,
financial needs surveys, pollution abatement expenditures surveys)
g) economic research plans developed to assess and advance the state-of-science in
economic theory, methodologies or modeling (in particular, the technical
feasibility of the plan's components)
h) new meta-analyses that re-analyze existing published literature and supporting
data on the measurement of economic benefits, costs and impacts
Generally, because of the nature of these types of economic work products, you should
conduct an external peer review. External peer reviews can be provided by the Science Advisory
Board's Environmental Economics Advisory Committee, or other appropriate outside
organizations or individuals that have expertise in the technical economic issues raised in the
economic work product.
2.2.5 Should Economic Work Products Prepared in Support of Regulations that
are Classified as "Major" or "Economically Significant" be Peer Reviewed?
Normally they would not require peer review, if the economic work product applies
accepted, previously peer reviewed methods in a straightforward manner. Economic studies
prepared to support "major" or "economically significant" regulations ("major" as defined below
in this section) typically do not utilize innovative or untried economic methods. It is
unnecessary to conduct peer reviews of straightforward applications or transfers of accepted,
previously peer reviewed economic methods or analyses. Economic assessments prepared to
support the regulatory development process routinely make use of previously published peer
-------
Peer Review Handbook Page 29
reviewed literature and adopt tools that allow for the transfer or adaptation of these techniques
and information. The procedures used to transfer or adapt this work will generally be
established by separate economic guidance documents which have been peer reviewed.
Therefore, economic documents that are developed using these procedures will not normally
require additional peer review, even those prepared in support of "major" and "economically
significant" rules.
Even where peer review is not needed, additional peer input can be beneficial in the
development of economic work products for "major" and "economically significant" rules. At
present, some peer input of these analyses is already likely to be included as part of the
regulatory development process, including input received from other EPA offices represented on
the workgroup for the rule, the Agency's Regulatory Steering Committee, and from the public as
part of the public comment process for the rule. But there may be added benefit to employing
additional peer input procedures, such as actively soliciting input from economists in other
Agency offices on the quality and completeness of the economic analysis.
"Economically significant" rules under Executive Order 12866 are defined as rules that
may have an annual effect on the economy of $100 million or more or adversely affect in a
material way the economy, a sector of the economy, productivity, competition, jobs, the
environment, public health or safety, or State, local, or tribal governments or communities. The
term "major," as used in this Q&A, does not mean the same thing as the definition of "major"
for purposes of deciding whether a work product requires peer review. Here, we are using the
term as defined in the Congressional Review Act, which defines a "major rule" as one that has
resulted in or is likely to result in: an annual effect on the economy of $100 million or more; a
major increase in costs or prices for consumers, individual industries, Federal, State, or local
government agencies, or geographic regions; or significant adverse effects on competition,
employment, investment, productivity, innovation, or on the ability of U.S.-based enterprises to
compete with foreign-based enterprises in domestic and export markets.
2.2.6 What Other Economic Work Products Might Benefit from Peer Review?
There may be other economic work products not covered in the preceding sections for
which peer review might be useful. Examples of such work products are presented below;
however, we do not intend to establish a presumption of peer review for these work products.
a) analyses measuring the economic impacts and effectiveness of adopting market-
based or economic incentives as regulatory management instruments
b) analyses of economic policies established under other government organizations
(e.g., economic models used to study transportation, economic development, and
international trade policies)
-------
Page 30 Peer Review Handbook
Most of these types of economic work products do not exhibit the degree of complexity,
or establish an innovative or untried approach, that would warrant a peer review. However,
other factors, such as the potential significance of the analysis for cross-Agency or inter-agency
practices, or the significance of the issues addressed, may make peer review desirable.
2.2.7 What Other Social Science Work Products Need Peer Review?
Typically, a social science work product is one that includes empirical, logic-based
approaches to answer technical questions about human motivation, human behavior, social
interactions, and social processes, which are relevant to the environmental issues being
addressed. The term "behavior" includes overt actions; underlying psychological processes such
as cognition, emotion, temperament and motivation; and biobehavioral interactions. The term
"social" includes sociocultural, socioeconomic, and sociodemographic status; biosocial
interactions; and the various levels of social context from small groups to complex cultural
systems. Examples of social science work products include analyses and/or evaluations related
to such topics as pollution prevention, risk communication, environmental information,
environmental justice, quality of life, decision-making, and public participation.
The following social science work products will normally be classified as major and will
require peer review:
a) internal Agency guidance for conducting social impact assessments and other
community cultural assessments related to different environmental protection
approaches such as community-based watershed protection (heretofore referred to
as social assessments).
b) new social science methodologies that will serve as a principal method or
protocol used to conduct social assessments.
c) unique or novel applications of existing social science methods such as surveys,
focus groups, interviews, network analyses, comparative analyses, and content
analyses.
d) new national surveys of values, perceptions and preferences related to
environmental protection.
e) innovative research or analyses that address the human dimensions of
environmental protection or environmental change in terms of social trends,
future predictions and/or behavioral generalizations.
-------
Peer Review Handbook Page 31
f) social science research plans developed to assess and advance the state-of-science
in social science theory, methodologies or modeling (in particular, the technical
feasibility of the plan's components)
2.2.8 How Should Peer Review be Handled for Products Developed under an
Interagency Agreement (IAG)?
Under an interagency agreement (IAG), EPA provides funds to another agency for that
agency to use for a specific purpose. The receiving agency's guidance for peer review will most
likely be different from the EPA peer review policy. Therefore, if EPA plans to use any work
products from that agreement, EPA must decide whether that document needs review under the
EPA peer review policy (see Section 2.2.16).
2.2.9 Should Products from Contracts, Grants, and Cooperative Agreements
Receive Peer Review?
If there is a scientific and/or technical work product resulting from a grant, contract, or
cooperative agreement and it is considered major and will likely be used in Agency decision-
making, the work product needs peer review. Since it would probably result in a perceived, if
not real, conflict of interest, a group that is generating the work product usually cannot conduct
or perform the peer review of its own work product. Exceptions maybe made in certain
instances for organizations that have adequate and well established recognized procedures for
peer review, such as the National Academy of Sciences (NAS). In practice, the Agency may
need to peer review the product on its own, or arrange with an independent third group (e.g., via
another extramural vehicle) to conduct the peer review. The Agency should not use the major
scientific and technical work products from contracts, grants, or cooperative agreements to
support decision making unless the work products are peer reviewed for both scientific and
technical rigor and applicability to the specific use to be made of the product.
Be aware that contracts are very different from grants and cooperative agreements.
Please note that there are important legal restrictions (discussed in 2.2.10 below) on the direct
use of work products developed under grants and cooperative agreements in the Agency's
decision-making process.
2.2.10 How Does Peer Review Apply to Products Generated Through EPA Grants
or Cooperative Agreements?
Major scientific and technical work products that are generated through EPA grants or
cooperative agreements are candidates for peer review, but special considerations apply.
-------
Page 32 Peer Review Handbook
First, you need to understand some background information on the proper use of
assistance agreements (grants and cooperative agreements) versus contracts. Under the Federal
Grant and Cooperative Agreement Act, grants or cooperative agreements may only be used
where the principal purpose of the agreement is to accomplish a public purpose that is authorized
by statute. EPA may derive some incidental use or benefit from the award as long as the
principle purpose of the project is public support. However, if the principal purpose of the
agreement is to obtain a product or service for the direct benefit or use of EPA, a contract must
be used rather than a grant or cooperative agreement.
EPA Order 5700.1, "Policy for Distinguishing Between Assistance and Acquisition,"
includes the following examples of projects that cannot be performed using grants or cooperative
agreements:
a) Research and studies which gather specific information desired by EPA for its
own use.
b) Research which provides technical or analytical advice for EPA's direct benefit or
use, such as information used to set guidelines.
c) Projects that produce specific information that will be directly incorporated by
EPA into technical, policy, or regulatory decisions.
Note that under the Order, EPA may legally provide financial assistance for research that
is intended to stimulate or support development of scientific knowledge that is not primarily for
EPA's direct use or benefit. The resulting work products would be widely disseminated either
through publication in scientific journals or through other means as opposed to a report tailored
to EPA's specific needs and requirements. EPA can consider these work products just as it can
review other published scientific works when formulating its programs and policies. Further,
EPA retains a royalty free, nonexclusive and irrevocable right to use the work products for
Federal purposes, even if the recipient has copyrighted the material. [40 CFR 30.36(a).]
Consult the Order and OGC for more information about making the proper choice
between an assistance agreement and a contract.
Provided that EPA's use of a grant or cooperative agreement recipient's work product is
incidental to the agreement's principal purpose, EPA may still determine that the recipient's
work product is a "major scientific or technical work product" under EPA's peer review policy
because: 1) it will be used to support an EPA program or policy position (assuming this use is
incidental to the principal purpose of the agreement), and 2) it meets one or more of the seven
criteria outlined in Section 2.2.3. In this situation, the work product requires peer review.
-------
Peer Review Handbook Page 33
The following are options for peer reviewing the product:
a) EPA can have the product peer reviewed with the participation of the assistance
agreement recipient/author. In this case, EPA could arrange for an independent
peer review of the product itself, or may contract with a third party to conduct the
review. EPA would also enter into a contract with the author (formerly the
recipient), which would task the author to prepare a response to the peer
reviewers' comments, and to revise or prepare an addendum to the product in
response to peer reviewer comments as determined appropriate by EPA.
A caveat to this approach is that it may be difficult to get the recipient/author to
agree to allow EPA to determine how to revise the product in response to the peer
review comments.
b) EPA can have the product peer reviewed without the participation of the
recipient/author. EPA could arrange for the peer review itself, or could contract
with a third party to conduct the review. In this case, however, the work product
would not be revised to incorporate the peer review comments. Instead, EPA
would receive the comments and prepare a statement that documents its own
response to the comments. The EPA Decision Maker(s) who is using the work
product to support an EPA program or policy decision must be provided
information on both the conclusions of the recipient's work product and EPA's
own conclusions from the peer review.
c) Recipients can get their products peer reviewed on their own Recipients may
determine on their own that peer review would benefit the credibility of their
product. Provided EPA agrees that a peer review would further the pub lie
purpose of the assistance agreement, EPA may include funds for the peer review
in the agreement. (See Section 2.2.11 for additional information.) Alternatively,
the recipient may make arrangements for, and fund, an independent peer review
of their product. In either case, EPA would need to evaluate whether the peer
review process undertaken by the recipient was acceptable for the purposes for
which EPA was planning to use the work product. EPA may accept the peer
review if it determines that it fulfilled the requirements of EPA's Peer Review
Handbook and that EPA could defend the peer review as if it were conducted by
EPA itself.
Under options a) or b), issues may arise over obtaining access for peer reviewers to the
raw data used by the recipient to generate the work product. Under 40 CFR 30.36(c)(2), EPA
has a right to obtain raw data produced by a non-profit organization or university under an
assistance agreement, even where the agreement doesn't specifically provide for this access.
-------
Page 34 Peer Review Handbook
Nevertheless, it may be prudent to include a specific term in the assistance agreement clarifying
this point to avoid misunderstandings. EPA must pay for obtaining access to the data if its
transmittal imposes additional costs on the recipient. Assuming our use of the data is incidental
to the principal purpose of the agreement, we also have a specific right to authorize peer
reviewers to use the data for Federal purposes under 40 CFR 30.36(c)(2).
The only time EPA cannot obtain access to the raw data is where EPA specifically
bargains away this right in the assistance agreement.
Again, consult OGC for help in drafting appropriate language for your assistance
agreement.
2.2.11 Can the Recipient of a Grant or Cooperative Agreement Use Agreement
Funds to Pay Peer Reviewers of their Work Products?
As noted in Section 2.2.10 above, provided EPA agrees that a peer review would further
the public purpose of the assistance agreement, EPA may include funds for the peer review in
the agreement. A payment to peer reviewers in exchange for their review of a
scientific/technical work product is allowable as a fee for professional services under assistance
agreements. (To accurately characterize this cost, however, it is important that the payment be
referred to as a fee, rather than an honorarium.) See OMB Circular A-21, Section J, item 32,
Professional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, Item
39, Professional Services Costs (Non-Profit Organizations), and OMB Circular A-87,
Attachment B, Item 33, Professional Services Costs (State, Local and Indian Tribal
Governments). (See http://www.whitehouse.gov/OMB/circulars/ for further details.)
2.2.12 Do Products Generated under EPA Grants or Cooperative Agreements Need
to be Reported in the Peer Review Product Tracking (PRPT) Database?
If a grant or cooperative agreement product is determined to be a major scientific and/or
technical work product and is used in Agency decision making (assuming this use is incidental to
the principal purpose of the agreement), it must generally be considered a candidate for peer
review. Therefore:
a) if it is determined to be a major work product and will be peer reviewed by EPA,
the work product is entered onto List B (Candidate Products for Future Peer
Review) as a "future peer review" product, then moved to List A (Products Peer
Reviewed Since 1991) when the peer review is completed.
b) if it is determined to be a major work product, but will not be peer reviewed (see
the possible reasons for not peer reviewing a major product in Section 2.3.1), it is
-------
Peer Review Handbook Page 35
entered onto List C (Products for Which a Decision has been made not to Peer
Review), with a justification for not peer reviewing it.
c) if it is determined to not be a major work product but EPA decides that it still
warrants peer review (see Section 2.2.17), it is entered onto List B (Candidate
Products for Future Peer Review) as a "future peer review" product, then moved
to List A (Products Peer Reviewed Since 1991) when the peer review is
completed.
d) if EPA decides to use the work product (due to its importance in EPA
environmental decision making), but considers it not major and not a candidate
for peer review, it is entered onto List C (Products for Which a Decision has been
made not to Peer Review) with the reasons for it being non-major.
e) if the grant or cooperative agreement product will not be used in Agency decision
making, it is not generally considered a candidate for peer review. Such products
do not have to be listed.
2.2.13 Should Site Specific Decisions be Subject to Peer Review?
The site specific decision itself is not subject to peer review and doesn't need peer review
based solely on the Peer Review Policy. However, if a site specific decision is supported by a
major scientific and/or technical work product, that work product needs peer review. While the
same considerations for major apply here, several of the criteria above (see Section 2.2.3;
specifically criteria b, c, d, and g) are considered more useful for regional consideration than
other criteria. So generally speaking, a close examination of how the underlying major scientific
and/or technical work product is adapted to the site specific circumstances is required.
2.2.14 Should NEPAProducts (e.g., EISs) be Subject to Peer Review?
Not everything requires peer review, and in the case of an Environmental Impact
Statement (EIS) prepared under the requirements of the National Environmental Policy Act
(NEPA), the document already has received extensive review (although not necessarily peer
review) through the "scoping" and interagency review processes that are part of NEPA.
The rule of thumb is that if the underlying scientific and/or technical work product is
major, then the work product needs peer review. In general, the Agency's role in the NEPA
document would suggest what sort of review the document gets. If EPA is developing the
document as part of an EPA action/decision (EPA is the Lead agency under NEPA), and it meets
the definition of "major," then it needs independent peer review. If it is not a major work
-------
Page 36 Peer Review Handbook
product (little impact, non-controversial, etc.), then peer input/continuing involvement might
well be appropriate.
On the other hand, if EPA is reviewing an EIS from another Agency (EPA is not the
Lead agency under NEPA), it is likely that we are reviewing for conflicts with EPA policy and
general environmental concerns. However, EPA must ask if the underlying major scientific
and/or technical work product that supports the EIS has been peer reviewed. If not, this would
raise concern about the full credibility and soundness of the EIS based on the science and
technical support. EPA should work with the other organization/agency to ensure that the major
scientific and/or technical work product receives peer review adequate for EPA purposes.
2.2.15 Should Environmental Regulatory Models be Peer Reviewed?
Generally, yes. Specific guidelines for the peer review of environmental regulatory
models have been published by the Agency. These can be found on the EPA web site under the
Science Policy Council home page (http://www.epa.gov/ORD/spc).
2.2.16 Is Peer Review Needed for Other Organization's Work Products that Have
been Submitted to EPA for Use in Decision Making?
Any scientific and/or technical work product that is used in Agency decision making and
is considered major becomes a candidate for peer review regardless of whether the work product
is produced by the Agency or another organization. Therefore, all major work products
important to EPA environmental decision making that are independently generated by other
organizations (e.g., other Federal agencies, interagency groups, State and Tribal bodies,
environmental groups, industry, educational institutions, international bodies) need to be
considered as candidates for peer review just as major work products generated by EPA are
considered (these would then be included on List B (Candidate Products for Future Peer
Review) if EPA is conducting or arranging for the peer review).
If possible, when EPA knows that a work product is being generated by another
organization and is of interest to EPA for future use, the appropriate EPA office(s) should work
with that organization, and others, as appropriate (e.g., the states), to promote the use of peer
review. For example, the Office of International Activities (OIA) as well as the impacted
program or regional office(s) should be included when international products are being
considered for EPA use.
It is hoped that if the other organization has the work product independently peer
reviewed, the peer review will meet the intent of the Agency's Peer Review Policy and EPA's
proposed use of the product (i.e., the peer review is basically equivalent to what EPA would do).
Agency staff from the appropriate office(s) should examine closely the particulars of the peer
-------
Peer Review Handbook Page 37
review to ensure independence and a conscious effort to incorporate the peer reviewers'
comments into the final work product. If there are perceived, or real, conflicts of interest, this
may preclude the use of that peer review and, in those instances, another peer review would be
needed. See Section 3.4.7 for considerations for when an outside party conducts and/or funds
peer review of their own work product and submits it to the Agency.
If the outside organization does not have the major work product peer reviewed and EPA
decides it needs peer review, the appropriate EPA office(s) will have to ensure peer review of
that work product occurs prior to the Agency's use of the work product in decision making. Peer
review can be accomplished by asking the outside organization to do so, or if they refuse, EPA
may conduct or arrange for the peer review. If EPA is conducting or arranging the peer review,
the product should be entered on to List B (Candidate Products for Future Peer Review).
2.2.17 Can Work Products That are Not Determined to be Major Still be Peer
Reviewed?
Yes, they could be. Scientific and technical work products that do not come under the
"major" distinction discussed above may nonetheless be candidates for peer review. For
example, a project manager may decide to use peer review because of particular program needs
and goals. Peer review may also be warranted because it adds substantial value to the work
product. In these cases, the product should be entered onto List B (Candidate Products for
Future Peer Review) as a candidate for future peer review.
2.3 Determining Which Work Products Do Not Receive Peer Review
2.3.1 Are There Circumstances When a Major Work Product is Not Peer
Reviewed?
There may be circumstances where a work product is considered major, but a decision
for no peer review can then be justified. For example:
a) Additional peer review is not required with work that has been previously
reviewed by recognized experts or an expert body. For example, a cancer risk
assessment methodology or an exposure modeling technique that was the subject
of earlier peer review would not require additional peer review, even if the
product supported a significant Agency decision.
b) Additional peer review is not required if an application of an adequately peer
reviewed work product does not depart significantly from its scientific or
technical approach.
-------
Page 38 Peer Review Handbook
c) Additional peer review is not required when the scientific and/or technical
methodologies or information being used are commonly accepted in the field of
expertise (e.g., Control Techniques Guidelines (CTGs) or other such
compilations). This would need the appropriate documentation to support the
commonly held view.
d) Most often, a major work product would not receive peer review when the
regulatory activity or action which the work product supports is terminated or
canceled — no further action, including peer review, is necessary.
e) In a few instances, statutory or court ordered deadlines or other time constraints
may limit or preclude peer review of product that would otherwise be considered
major. However, it is up to the Decision Maker(s) to make every attempt possible
to assure that peer review of major products occurs taking into account these
deadlines. With proper up-front planning for peer reviews of major products and
the ability to tailor the method of review to the product and circumstances, it
should rarely be the case that major products don't receive some type of peer
review due to time constraints.
f) Very rarely, resource limitations may also restrict peer review. Programs or
Regions will evaluate these circumstances on a case by case basis; decisions will
be based on consultations involving line management, the Project Manager, the
Peer Review Leader, and the Peer Review Coordinator.
If peer review of a major scientific and/or technical work product is not conducted, a
written justification must be placed in the "Justification for Non-Peer Review Product" Section
of the data base entry for that product on List C (Products for Which a Decision has been made
not to Peer Review). The justification is signed-off by the appropriate Decision Maker (see
Section 1.3.2c)).
2.3.2 What Products Normally Do Not Need Peer Review?
Products that are not major scientific and technical work products normally do not
require peer review under the intent of the Peer Review Policy. Most of these scientific and
technical work products are then placed on ListC (Products for Which a Decision has Been
Made Not to Peer Review) with a written explanation of why it was determined to be not major
(see Section 1.32c)). This justification is signed-off by the appropriate Decision Maker to
assure that all scientific and technical work products received consideration for peer review.
Some scientific and technical work products are not considered major and generally do
not need to be placed on List C (Products for Which a Decision has Been Made Not to Peer
-------
Peer Review Handbook Page 39
Review). These types of work products typically include: derivative products (i.e., a product that
only summarizes an already peer reviewed product or products; fact sheets), compendiums of
existing models, methods and/or technologies; minor intermediate products (e.g., a technical
memorandum from a contractor) describing methods or results incorporated in a larger product
which will be peer reviewed; or preliminary or incidental analyses prepared separately from the
work product ultimately used to support an Agency action or decision (e.g., during the course of
developing a rule, managers may direct staff to prepare various "what if analyses; those that
aren't used in the work product do not need to be listed).
2.3.3 Do Voluntary Consensus Standards Require Peer Review?
Generally, no. The National Technology Transfer and Advancement Act of 1995
(NTTAA) directs EPA to use available voluntary consensus standards in its regulatory activities
unless to do so would be inconsistent with applicable law or otherwise impractical. For
purposes of the NTTAA, voluntary consensus standards are defined as technical standards (e.g.,
materials specifications, test methods, sampling procedures, and business practices) that are
developed or adopted by voluntary consensus bodies (such as ISO, ASTM). The general
purpose of the NTTAA is to reduce private and governmental costs by avoiding having the
government "reinvent the wheel." Voluntary consensus standards would normally not require
peer review because the underlying process used by issuing organizations to develop and
approve these standards is generally considered adequate for purposes of the Agency's peer
review policy. EPA reserves the right to conduct a peer review if it determines that the standard
is not a voluntary consensus standard under the NTTAA.
2.4 Choosing a Peer Review Mechanism
2.4.1 How Do You Determine the Appropriate Peer Review Mechanism?
During the planning of a peer review, the Decision Maker and the Peer Review Leader
may consider several mechanisms for the peer review of major scientific and technical work
products. These options range from consultations with EPA colleagues not involved in
developing the product to a large and formal panel of
outside subject matter experts. The peer review effort
The mechanism of the peer
review matches the
importance and complexity of
the major work product.
might be a focused one-time evaluation, or could
encompass several examinations over the course of a
project. In principle, peer review provides the greatest
credibility for major work products when it involves well-
qualified external reviewers, is intensive in its
examination, and operates through a more or less formal
process. As a practical matter, however, time and resource considerations in many cases impose
limitations on what can be reasonably achieved. Arranging for the most appropriate and feasible
peer review will involve good judgment and a willingness to consider substance, time, and
resource tradeoffs. Developing a peer review plan that provides for appropriate depth, timing,
-------
Page 40 Peer Review Handbook
and content is an important matter for early consideration by the Decision Maker and Peer
Review Leader. Note that use of peer input, public or stakeholder involvement does not qualify
as peer review.
The approach best suited to a specific work product will depend on the nature of the topic
and the intended final product. Generally, the more novel or complex the science or technology,
the greater the cost implications of the impending decision, and the more controversial the issue,
then the stronger the indication for a more extensive and involved peer review and for external
peer review in particular. Certain work products will clearly lend themselves to extensive
external peer review; generally these will be products with large impacts (e.g., those that support
Tier 1 and Tier 2 rulemakings). Other major work products may not need a large scale external
peer review and may utilize a less involved, less resource intensive review. The peer review of
some products may be better served with some form of internal peer review or a combination of
internal and external peer review.
It is important to make the choice of peer review mechanism at the time that the work is
planned (for products supporting rule makings, at the analytic blueprint stage) so that peer
review costs and time can be budgeted into the work plan. Essentially, the level of peer review
matches the impact and complexity of the major work product. For example, a rule under
development carries considerable weight and deserves careful handling and attention; therefore,
the supporting work product deserves similar care and attention for its peer review. Both
internal and external peer review mechanisms are available, have been used in the past, and have
served to address the needs and challenges of a particular peer review situation. Nevertheless,
no single peer review mechanism is likely to work best in all situations. Some useful guidance
includes:
a) Major work products intended to support the most important decisions, or that
have special importance in their own right, ordinarily should be the subject of
external peer review. Generally, the more complex, novel and/or controversial
the product, or the higher impact it has, the more the Decision Maker should
consider implementing a large-scale peer review involving external experts.
b) Major work products that are less complex, novel, or controversial, or have a
lower impact may not need such a large-scale and external peer review. These
products might be subject to one of the less extensive, less resource-intensive
review processes.
c) Group discussion with peer reviewers can be very helpful at some point in the
peer review process. On the other hand, simply soliciting individual comments is
easier, faster, and less expensive. Individual review is probably more appropriate
-------
Peer Review Handbook Page 41
for peer review at the early stages of a product's development or for products with
less impact and complexity.
d) Strict time constraints, such as a court-ordered deadline, can make a less involved
or formal peer review mechanism imperative. But Decision Makers and Peer
Review Leaders must make maximum efforts to assure that such a process is
perceived as systematic and objective.
2.4.2 What are Examples of Internal Peer Review?
a) Independent experts from within the Agency (e.g., ORD experts on non-cancer
effects of lead reviews a draft article on benchmark dose)
b) An ad hoc panel of independent experts from within the Agency (e.g., an
independent internal workgroup convened to examine the case for the
classification of a chemical as a carcinogen)
c) Technical merit review by scientists in an Agency laboratory (e.g., an initial
review of the risk assessment for a regional incinerator by Agency scientists)
2.4.3 What are Examples of External Peer Review?
a) Independent experts from outside the Agency (e.g., a letter review by outside
scientists)
b) An ad hoc panel of independent experts outside the Agency (e.g., a group is
convened to develop a consensus on the carcinogenicity of a particular industrial
chemical)
c) Agency-sponsored peer review workshops (e.g., a review of potential indicators
of ecosystem damage)
d) Review by an established Federal advisory committee such as the Science
Advisory Board (SAB), FIFRA Scientific Advisory Panel (SAP), ORD's Board
of Scientific Counselors, or the Clean Air Scientific Advisory Committee (e.g., a
review of a criteria document for a particular chemical risk)
e) Agency-based federal advisory committee (other than those established and
discussed in d above)
-------
Page 42 Peer Review Handbook
f) Agency appointed special board or commission (e.g., a review of the risk
assessment methodology prepared by the Clean Air Act Commission on Risk
Assessment) Note: The Office of General Counsel should be consulted
regarding EPA's authority to establish and finance the activities of a commission
or board.
g) Interagency committee (e.g., a review of prospective research plans by the
Committee on the Environment and Natural Resources coordinated by the White
House)
h) A committee convened by another federal agency or government organization
(e.g., a review of the Dioxin Reassessment by the Health and Human Services
(HHS) Committee to Coordinate Environmentally Related Programs)
i) Review by non-governmental groups (e.g., a Society of Risk Analysis review of
cancer guidelines)
j) Review by the National Academy of Sciences (e.g., a review of the state of
current knowledge about children's health risks from pesticide exposures)
2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?
Peer review of journal articles (written by EPA or non-EPA authors) performed by a
credible, refereed scientific journal contributes to the scientific and technical credibility of the
reviewed product. EPA considers peer review by such journals as adequate for reviewing the
scientific credibility and validity of the findings (or data) in that article, and is therefore a
satisfactory form of peer review. However, in some cases, peer review of an Agency work
product that uses these articles may be required (see Section 2.4.5).
EPA authored journal articles, whether used in an Agency work product or not, are
tracked using List D (Scientific Articles and Reports That Are Peer Reviewed by Organizations
Outside of EPA -- see Section 1.3.2d) for details). This shows that these items have been peer
reviewed, and differentiates them from other Agency work products. In addition, it gives EPA
an opportunity to highlight the extensive work it produces in the scientific literature.
It is not necessary to list these papers individually on List D (Scientific Articles and
Reports That Are Peer Reviewed by Organizations Outside of EPA), although some
organizations may choose to do so. However, at a minimum, the total number prepared by the
organization and peer reviewed by journals must be included on List D. The organization may
ultimately need to identify the individual papers and where published, if requested (e.g., due to
litigation, FOIA, etc.).
-------
Peer Review Handbook Page 43
Prior to submitting an article to a journal for peer review, EPA employees are encouraged
to have the article internally peer reviewed (see Section 1.4.9); such internal peer review is
already common procedure in certain parts of EPA. Articles may also need examination in
accordance with any organizational clearance procedures, especially when the author is
presenting him or herself as an EPA employee. For EPA employees, conflict of interest
regulations will also apply.
2.4.5 Do Agency Work Products Become Candidates for Peer Review when Peer
Reviewed Journal Articles are Used in Support of that Work Product?
In most instances, major Agency work products are candidates for peer review even
when supported by peer reviewed journal article(s). Although the use of articles that have been
peer reviewed by a credible journal strengthens the scientific and technical credibility of any
work product in which the article(s) appears or is referenced, it does not automatically eliminate
the need to have the work product itself peer reviewed. In most cases, journal peer review may
not cover issues and concerns that the Agency would want peer reviewed to support an Agency
action. Under these circumstances, the major scientific and/or technical work product in which
the article(s) appears or is referenced becomes a candidate for peer review. A journal article
authored by EPA employees would be used in the same manner as an article published by
anyone else in a credible, well recognized journal.
If an Agency work product is based solely on a single article that has received peer
review by a credible journal (e.g., where a model is suggested for a singular use that fits a
specific Agency need), peer review of the Agency work product may or may not be necessary
depending on how closely you apply the findings from the article. If an Agency work product is
based on two or more articles that have received peer review by a credible journal(s), the Agency
work product generally becomes a candidate for peer review. Decisions to make (or not make) a
work product a candidate for peer review needs to be documented in the peer review record.
One important factor to remember with regard to the use of articles that have received
journal peer review deals with the availability of documentation from that peer review. Ideally,
EPA needs to maintain a clear, easily accessible record of the peer review to assure the
credibility and validity of the peer review (see Section 2.5 for details on the peer review record).
However, the documentation from a journal peer review would not normally be available to the
Agency, so such documentation is not expected in the peer review record.
2.4.6 When and How Often Should Peer Review Occur?
The Decision Maker and Peer Review Leader have significant discretion in deciding on
the timing and the frequency of peer review. Options abound, each with merits depending on the
context and specified peer review objectives. In many situations, a single peer review event,
-------
Page 44 Peer Review Handbook
beginning when the final draft work product becomes available, is the approach taken.
However, it is increasingly apparent that peer review performed earlier in the work product
development stages is a superior approach for some work products. There maybe substantial
incremental benefit to conducting more than one peer review during the whole process of work
product development, particularly where it involves complex tasks, has decision branching
points, or could be expected to produce controversial findings. In addition, early review would
be beneficial at the stage of research design or data collection planning where the product
involves extensive primary data collection. The Decision Maker and Peer Review Leader need
to determine when the peer review(s) should occur, considering the type of work product under
development and at what point in its development process a peer review would be most
beneficial.
Other types of work products that would benefit from early, up-front peer review in their
development are scientific and technical planning products. Examples of such products are
research proposals, plans, and strategies. Also, while not products per se, ongoing research
programs can be peer reviewed.
Remember though, that while more than one peer review can be beneficial, the
distinction between peer input and peer review needs to be kept in mind. Experts providing
input during the development or planning stages of the work product generally do not become
peer reviewers of that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction).
2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review?
Several factors impact how quickly a peer review maybe needed. These include
deadlines for completion of a project, research program, or rulemaking, funding availability,
availability of quality peer reviewers, and statutory and/or court-ordered deadlines.
Peer review sometimes leads to new information and analyses. Reviewers may make
recommendations for new research that would alter the work product and thus modify the
scientific/technical basis for the action or rule it supports. For this reason, a completed peer
review is desirable before issuing any proposal for public comment. If that is not logistically
possible because of court or statutory deadlines, or other appropriate reasons, the Decision
Maker should make every effort to complete the peer review before the close of the comment
period. Because peer review comments on such work products could be of sufficient magnitude
to warrant a revision to the proposed action or rule, Decision Makers should exercise diligence
in completing the peer review prior to the proposal stage whenever possible.
-------
Peer Review Handbook Page 45
2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the
Peer Review?
The organization of the Decision Maker is normally responsible for conducting the peer
review. Responsibility for conducting a peer review can be negotiable when more than one
Agency office or region or other agencies are involved. Usually, the degree of involvement by
any of the organizations and agencies and their ability to fund peer review will often determine
who has the lead for the peer review.
2.5 Creating the Peer Review Record
2.5.1 What is the Peer Review Record?
It is the formal record (file) of decision on the conduct of the peer review, the type of
peer review performed, and an explanation of how the peer review comments were addressed. It
includes sufficient documentation for an uninvolved person to understand what actually
happened and why. The Peer Review Leader (with the program manager if there is one) creates
a separate, clearly marked peer review file Section within the overall file for development of the
work. Once the peer review is completed, it is the responsibility of the Peer Review Leader to
ensure that the peer review record is filed and maintained in accordance with the organization's
procedures.
The Peer Review Record is separate from the entry in the Peer Review Product Tracking
(PRPT) Database. While some information from the peer review record appears in the database,
the peer review record is the official record of the peer review.
2.5.2 How Can the Peer Review Record Improve the Peer Review Process?
A good peer review record allows future reference to what happened during the peer
review, and helps Decision Makers make appropriate use of peer reviewer input. In addition, a
good record helps ensure that EPA's Peer Review Policy is followed. The Peer Review Leader
is responsible for ensuring that the peer review record for individual work products is collected
and maintained until completion of the peer review effort.
2.5.3 What Should Be in the Peer Review Record?
The peer review record should include all materials considered by the individual peer
reviewers of the peer review panel, as well as their written comments and other input. Such
materials include, at a minimum (see also Section 4.3.1):
a) The draft work product submitted for peer review
-------
Page 46 Peer Review Handbook
b) Materials and information (including the charge) given to the peer reviewers
c) Written comments, information, and materials received from the peer reviewers
d) Information about the peer reviewers (such as reviewers' names, affiliations, and
a statement concerning potential conflicts and their resolution)
e) Logistical information about conduct of the peer review (such as times and
locations of meetings)
f) A memorandum, or other written record, approved by the Decision Maker,
responding to the peer review comments specifying acceptance or, where thought
appropriate, rebuttal and non-acceptance
g) The final work product
When deciding if particular materials should be included in the record, the Peer Review
Leader should consider whether the materials would help reconstruct the peer review process
and results at a later time. If the materials may be helpful, they should be part of the peer review
record.
In addition to hard copies of materials, Peer Review Leaders need to maintain electronic
copies of the materials (e.g., charge) that are necessary for the annual reports to the Peer Review
Tracking Database. Peer Review Leaders consult with their Peer Review Coordinators to
identify those materials.
2.5.4 What Should I Do with a Peer Review Record That Pertains to a
Rulemaking Action?
The Peer Review Leader should coordinate with their program's docket office to see that
proper docketing requirements are satisfied for a peer review of a work product supporting a new
rule. The Peer Review Leader is also responsible for notifying the workgroup chair as well as
the Peer Review Coordinator (for the annual report) that a peer review is completed.
2.5.5 When Should the Peer Review Record Building Process Begin?
An early start at developing and maintaining a peer review record will help ensure the
record is complete and helpful. Ideally, the record begins when the decision to peer review a
work product is made. The Peer Review Leader needs to construct the peer review record from
this point on — this will avoid potentially time-consuming reconstruction at a later point. Note
-------
Peer Review Handbook Page 47
that the peer review record is not complete until it contains a copy of the final work product
which addresses thepeer review comments.
2.5.6 What are the Differences in Record Keeping for a Review by an Individual
Compared to a Panel?
Strictly speaking, a true peer review requires more than a single reviewer. A review
conducted by one individual will rarely provide the depth of commentary required to improve
the work product. In addition, you will not receive the range of views and richness necessary to
ensure improvement.
In the case of a review panel, there will often be conflicting comments that must be
resolved. This resolution should be in the record.
2.5.7 Where Should the Peer Review Record be Kept and For How Long?
During the active conduct of the peer review, the Peer Review Leaders maintain the peer
review record with themselves until the peer review is totally completed. Minimally, the file
should be maintained until one year after the completed peer review is reported in the next
annual reporting. After that, the peer review record should be maintained for a "reasonable
period of time." Establishment and maintenance of the archive where the peer review records
ultimately reside are an organization's responsibility (i.e., not that of an individual program
manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions
will decide the appropriate level of organizational responsibility and how they will meet any
"routinely available" requirements. The peer review record may be kept with other records
relating to the overall project, as long as it is easily and separately identifiable.
There are also specific requirements regarding the use of dockets for record-keeping;
however, these are not covered in this Peer Review Handbook. The documents contained in the
peer review record should be maintained in accordance with the Agency's record keeping
retention schedule for such records (for details, see EPA's National Records Management
Program; http://www.epa.gov/recordsA. One long-term archiving mechanism may be the formal
archiving at the Federal Records Center in Suitland, MD.
2.5.8 Are Internal Peer Review Comments Included in the Peer Review Record?
An internal EPA peer review must be a formal process to be considered a legitimate peer
review. This process adheres to the guidance found in this Handbook for planning, conducting,
and completing a peer review. When you follow this formal process to obtain peer review from
EPA peers (see Section 1.4.9), then the whole record of that internal peer review is included in
the peer review record. This includes all the materials detailed in Section 2.5.3 (also see Section
-------
Page 48 Peer Review Handbook
4.3.1). Conducting a formal internal peer review is not the same thing as informal input from
your EPA colleagues (i.e., "colleagues down the hall"), nor peer input from Agency personnel
helping to develop the work product, nor organizational review and clearance processes. Such
inputs from these informal processes should not be placed in the peer review record. The peer
review record should contain only the information obtained when you conduct a formal internal
peer review.
In some cases, an internal EPA peer review maybe followed by a separate external peer
review. In this case, the formal record of the internal peer review should be included in the peer
review record, however, the external peer review will generally stand as the peer review of
greater significance since it is viewed more independent in nature, has broader fields of available
expertise which can be brought to bear on the issues and often includes greater depth for specific
disciplines. Peer Review Leaders should ensure that any applicable EPA record keeping
regulations are followed (for details, see EPA's National Records Management Program;
http://www.epa.gov/records/).
2.6 Budget Planning
2.6.1 What Budgetary Factors Should I Consider in a Peer Review?
Resources needed to implement the Peer Review Policy need to be requested through the
usual Agency budgetary processes. The budget formulation process within the Executive Branch
is followed, after appropriation bills are passed by
Congress, by budget execution. These two processes ~~ ' ' ~~
.
t IS UttVt f)T trt
provide opportunities to secure resources for activities *. . J
carried out by Headquarters and Regional offices, normal cost oj doing
including peer review. The major work products for which '
decisions for peer review have been made (List B
(Candidate Products for Future Peer Review) candidates)
need to have adequate funding for peer review in budget requests for the coming fiscal year.
Similarly, adequate funding needs to appear in the actual approved operating budget to ensure
their conduct. For purposes of budget planning, the costs of peer review would include the FTE
cost of staff, the contract or other costs associated with the use of outside peer reviewers, and the
administrative costs of conducting a review (copying, travel expenses, etc).
2.6.2 What Input is Needed for the Annual Budget Formulation and Budget
Execution Process?
Senior Management in Office and Regions (including Decision Makers and budget
officers) need to be sure that budget requests include anticipated resources for peer review. Peer
review needs to be considered as a normal part of doing business. Peer review resource
-------
Peer Review Handbook Page 49
considerations should also be addressed in the analytic blueprint for Agency rule-making
actions.
2.7 Legal Considerations
2.7.1 Are There Legal Ramifications From the Peer Review Policy?
The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. Except where
provided otherwise by law, peer review is not a formal part of or substitute for notice and
comment rulemaking or adjudicative procedures. EPA's decision to conduct peer review in any
particular case is wholly within the Agency's discretion. Similarly, nothing in the Policy creates
a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA
decisions rely on scientific and technical work products that have been subjected to peer review,
the remarks of peer reviewers should be included in the record for that decision.
2.7.2 Is Legal Advice Needed?
AA/RA staff and management should work regularly with individual OGC/Regional
Counsel (RC) staff assigned to Agency activities. Peer Review Leaders should initially consult
with their customary OGC/RC advisors for legal advice or referral. Headquarters attorneys have
specialties in specific areas and can be consulted as needed (e.g., FACA considerations (see
below); contractual responsibilities; ethics and potential conflicts of interest).
2.8 Federal Advisory Committee Act (FACA) Considerations
The Federal Advisory Committee Act, 5 U.S.C. App. 2, imposes certain open meeting
(public announcement in the Federal Register), balanced membership, and chartering
requirements (with the approval of the General Services Administration (GSA)) before the
Agency establishes, controls or manages an "advisory committee" for advice or
recommendations. Peer review carried out by formal and established (chartered) Federal
advisory committees, such as the Science Advisory Board (SAB) or the FIFRA Scientific
Advisory Panel (SAP), is always subject to FACA requirements. However, FACA does not
apply to every EPA and contractor-run peer reviews.
In this section you will find information on the applicability of FACA to EPA- or
contractor-run peer reviews.
-------
Page 50 Peer Review Handbook
2.8.1 When Do FACA Requirements Apply to EPA-Run Peer Reviews?
In most cases, Federal Advisory Committee Act (FACA) requirements apply to EPA-run
peer reviews that are conducted by formal and established (chartered) Federal advisory
committees, such as the Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel
(SAP). These groups have the following characteristics:
a) the group is established, controlled, or managed by EPA;
b) the group has a fixed membership, established purpose, and a set agenda; and
c) the group strives to produce collegial, rather than, individual advice to EPA
EPA run peer reviews that are not originally intended to be subject to FACA, but which
exhibit the above characteristics, may unintentionally become subject to FACA. Questions
concerning the applicability of FACA to peer review meetings should be addressed to the FACA
experts in the Cross-Cutting Issues Law Office ofOGC (Mail Code 2322A at Headquarters), or
the appropriate Office of Regional Counsel.
2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA?
If EPA conducts a peer review with the purpose of obtaining advice from the individual
peer reviewers and not for the purpose of obtaining a peer review product from the group (as a
collective or consensus body), the peer review would, in most cases, not be subject to FACA.
Peer review participants provide only their own views or recommendations and do not vote nor
do they provide collective or consensus recommendations to EPA. When referring to the
recommendations of the individual reviewers, EPA should not characterize these
recommendations using terms such as "collective" or "consensus." As a general matter,
workshops and "letter reviews" that seek individual views or comments are usually not subject
to the requirements of FACA.
In addition to ensuring that peer reviewers only provide comments as individuals, EPA
officials may wish to lessen the potential for a challenge under the Federal Advisory Committee
Act (FACA) by seeking balanced participation at peer review meetings, and allowing interested
members of the public to attend, and ensuring that they have access to appropriate materials.
Non-FACA meetings may be announced in the Federal Register (providing that it is clear
in the notice that such meetings are not subject to FACA) as it provides the public with useful
information and a point of contact concerning the peer review. In addition, non-FACA (as well
as FACA) meetings should also be advertised via other avenues (e.g., the Web, local
newspapers, and mailing lists).
-------
Peer Review Handbook Page 51
2.8.3 How Do I Ensure that a Contractor-Run Peer Review Does Not Become
Subject to FACA?
Committees (or other peer review groups) established, controlled or managed by an
outside organization (such as by an EPA contractor) to provide that outside organization with
advice and recommendations (that will be submitted eventually to EPA as a contractor report)
are not subject to FACA. Although FACA should not apply to contractor-run peer reviews,
there are things that you (i.e., EPA) can inadvertently do that may invoke FACA.
The following are considerations that you should be aware of when a contractor runs a
peer review (e.g., letter review, panel, workshop, etc.) for EPA:
a) the outside party's peer review could be subject to FACA if EPA establishes,
manages or controls the peer review group (e.g., EPA selects the members of the
panel, runs the meeting, etc.).
b) EPA should not provide contractors with a draft agenda or suggested format for
peer review meetings. EPA contractors should manage and control the process,
including running any meetings.
c) At the request of the EPA contractor, EPA can provide a briefing to the peer
reviewers (e.g., in a conference call with the contractor on the line) on the history
or background of the development of the peer review document. EPA only
should provide technical or background information and not use the call to take
over the contractor's peer review group. Not only should the contractor be "on
the line" but it should be very clear to all participants that the contractor is in
charge of the call. The contractor, not EPA, should invite people to participate,
make all administrative arrangements, conduct the meeting, and control the
agenda.
d) EPA employees may attend the peer review panel meetings or workshops.
However, they may not take over the control of the meeting. The contractor
should call on them to speak when appropriate, but they should limit their
participation to providing technical and/or background information, and not
attempt to, or appear to, take over the contractor's meeting.
e) Since FACA does not apply when a contractor establishes, controls, or manages a
peer review, the contractor need not avoid terms such as "collective" or
"consensus" when reporting agreement among its peer reviewers
(subcontractors).
-------
Page 52 Peer Review Handbook
f) EPA may provide comments to the contractor on the contractor's peer review
only to the extent that the Agency is verifying that the contractor has completed
satisfactorily the report in accordance with the work assignment. EPA should not
attempt to make changes in the contractor's conclusions; this would compromise
the independence of the peer review conducted by the contractor.
-------
Peer Review Handbook Page 53
3. CONDUCTING A PEER REVIEW
3.1 Overview Statement
The success and usefulness of any peer review depends on the quality of the draft work
product submitted for peer review, the care given to the statement of the issues or "charge," the
match between the peer review draft product and the form of peer review, the match between the
peer review draft product and the scientific/technical expertise of the reviewers, and Agency use
of peer review comments in the final product. It is not simply enough to conduct a peer review;
each of the foregoing elements requires serious attention.
3.2 Charge to the Peer Reviewers
3.2.1 What is a Charge?
As part of each peer review, the Peer Review Leader must formulate a clear, focused
charge that identifies recognized issues and invites comments or assistance. This request signals
the Agency's awareness of potential issues and its receptivity to expert recommendations. The
charge to peer reviewers usually makes two general requests. First, it focuses the review by
presenting specific questions and concerns that the
Agency expects the reviewers to address. Secondly,
it invites general comments on the entire work
product. The specific and general comments should
focus mostly on the scientific and technical merits of
the work product and, where germane, whether the
scientific/technical studies have been applied in a
sound manner. Remember, the peer review is not for
the decision or action itself, but for the underlying scientific and/or technical work product.
Focused questions greatly simplify the task of collating, analyzing and synthesizing peer review
comments on a topical basis. The questions should be specific enough to get helpful comments,
but not so specific (unless very specific points are needed to be addressed) that they preclude
creative responses. Moreover, the written responses to these questions by peer reviewers help
the Agency create a peer review record. As a general rule, the time drafting a good charge letter
is well-spent and is necessary for an effective peer review.
The time spent preparing a good
charge is well spent, and is crucial
for an effective peer review.
3.2.2 What are the Essential Elements of a Charge?
a) Brief overview or introduction (describe what the work product is, how it was
developed, how it will be used)
-------
Page 54 Peer Review Handbook
b) As needed, a brief description or listing of any background materials provided to
the peer reviewers
c) The issues or questions to be addressed by the peer reviewer(s)
1) The due date of reviewers' comments
2) The format of reviewer responses
3) The point of contact in case peer reviewers have questions
3.2.3 Where Can I Get an Example of a Charge?
Appendix C (Examples of Charges) contains examples of successful charges that cover a
variety of issues. Appendix D (Guidance on Requesting a Review by the Science Advisory
Board) provides guidance for obtaining Science Advisory Board (SAB) services.
3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?
Yes. EPA may decide to obtain stakeholder input on the charge to the peer reviewers,
but EPA must make the final determination on what elements to include in the charge to ensure
that it meets the needs of the EPA peer review.
a) If you obtain stakeholder input, you must include any and all interested parties to
the extent feasible based upon statutory, regulatory, budgetary and/or time
constraints. Do not limit input to one stakeholder only (e.g., a responsible party
or environmental group).
b) If EPA has hired a contractor to perform the peer review, it should still be EPA
personnel who obtain stakeholder input and provide the list of charge questions to
the contractor.
c) If you form a committee of stakeholders to help develop the charge, be aware that
your committee may become subject to the requirements of FACA (see Section
2.8).
3.2.5 Who Writes the Charge When I Hire a Contractor to Conduct the Peer
Review?
In general, if EPA hires a contractor to perform the peer review, EPA must allow the
contractor independence in conducting the review. However, with regard to the charge, EPA can
-------
Peer Review Handbook Page 55
and should provide the contractor with the substantive list of questions that EPA wants included
in the charge letter to ensure that the peer review meets EPA's needs. The list of charge
questions can be incorporated into the Statement of Work if the EPA project manager has the list
ready at that time. Based on this list, the contractor would then prepare and submit the actual
charge letter to the peer reviewers. Prior to submitting the charge letter to the peer reviewers, the
contractor should give EPA an opportunity to review the charge letter to ensure that it meets
EPA's needs. EPA cannot submit the charge directly to the peer reviewers.
3.2.6 Is it Okay For Me to Ask a Contractor to Develop the Charge to the Peer
Reviewers?
No. EPA will provide the charge questions to the contractor as discussed above. The
contractor, though, may provide assistance and advice in the development of the charge.
3.3 Time Line
3.3.1 What are the Factors in Scheduling a Peer Review?
The peer review schedule is a critical feature of the process. The schedule must take into
account the availability of a quality draft work product, availability of appropriate experts, time
available for using peer review comments, deadlines for the final work product, and logistical
aspects of the peer review (e.g., contracting procedures).
The schedule for peer review should take into account the overall rulemaking (or other
decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of
the peer review should be included in the development of the analytic blueprint. Peer review
sometimes leads to new information and analyses, or recommendations for new research that
would alter the work product and thus modify the scientific/technical basis for the action. For
this reason, it is usually advisable to complete the peer review before taking public comment, or
at least before the close of the public comment period.
3.4 Selection of Peer Reviewers
3.4.1 What are Considerations for Selecting Peer Reviewers?
Selection of independent peer reviewers is not a trivial task, and it is crucial to an
effective peer review. It is important that peer reviewers be selected for independence and
scientific/technical expertise. Therefore, EPA should always make every effort to use peer
reviewers who do not have any real or perceived bias or conflict of interest and who are
completely independent. However, the very need to have experienced individuals on a peer
review, along with the desire to have appropriate technical balance and representation, can mean
-------
Page 56 Peer Review Handbook
that the selection of potential peer reviewers often comes from those who are considered as
having a potential bias. To reduce the concern that a potential panel may have unnecessary bias,
it may be useful to obtain an informal review of the expertise and balance of potential peer
reviewers from others in your organization, from OGC or even from outside groups. Sometimes
selecting individuals who have served in a variety of organizations rather than a single one for an
extended period, provides expertise with diverse perspective. The emphasis on independence
and expertise applies equally to government experts and experts from the larger scientific
community.
Some peer reviews can be conducted with two or three reviewers; others involve panels
of peer reviewers. In either case, each peer reviewer should have recognized technical expertise
that bears on the subject matter under discussion. The peer reviewers of a work product should
represent a balanced range of technically legitimate points of view. In addition, cultural
diversity and "address" (e.g., industrial, academic, or environmental community) are other
factors that can play a role in selecting peer reviewers.
3.4.2 Where Do I Find Peer Reviewers?
Recommendations for potential peer reviewers can be identified from a number of
organizations. These include external groups such as the affected party(ies), special interest
groups, public interest groups, environmental groups, professional societies, trade or business
associations, state organizations or agencies, Native American Tribes, colleges and universities,
the National Research Council, and other Federal agencies with an involvement in or familiarity
with the issue. Agency associated groups include the staff of the Science Advisory Board (SAB)
or the Scientific Advisory Panel (SAP), and relevant scientific and technical experts from
Program or Regional offices.
In certain circumstances, existing peer review organizations such as the SAB or SAP may
be used to conduct a peer review. These groups establish their own criteria for accepting work
and coordination must be made directly with them (see Appendix D - Guidance on Requesting a
Review by the Science Advisory Board) for SAB procedures. Both SAB and SAP conduct
formal, public, external peer reviews.
Occasionally, a member of the scientific community will offer his/her services for peer
review during an ongoing peer review. These offers may be at no cost or based on an
expectation that reimbursement will be made. Disposition of these unsolicited offers will be
handled on a case by case basis by the Peer Review Leader, and as necessary, in consultation
with the Peer Review Coordinator, the Office of General Counsel (OGC), and appropriate
Decision Makers.
-------
Peer Review Handbook Page 57
If you use a contract mechanism to conduct a peer review, the contractor will have its
own pool of scientific and technical experts for peer review. With contractors, EPA can provide
information on potential sources of peer reviewers for conducting a peer review if such a listing
is prepared in alphabetical order. In no way should EPA insist that the contractor select from
EPA's list, or require that the contractor receive EPA approval before selecting any peer
reviewer (sometimes known as a "subcontractor"). However, EPA should review the list of
reviewers for conformance to work assignment specifications and adherence to conflict of
interest concerns. Furthermore, when utilizing a contract mechanism to conduct peer review,
EPA is not permitted to direct the prime contractor to a specific subcontractor (or peer reviewer)
nor is EPA permitted to direct the peer reviewer (subcontractor). All interactions with the peer
reviewers must be coordinated through the prime contractor. (See Section 3.6 for further
information)
Keep in mind that for contracting purposes, contractors are required to obtain
Contracting Officer (CO) approval of subcontractors and the CO generally seeks the input of the
work assignment manager (WAM - the WAM may be the Peer Review Leader in many cases)
before approving the use of subcontractors. In this case, as noted above, the Peer Review Leader
should review the list of potential reviewers for conformance to the work assignment
specifications and adherence to conflict of interest concerns.
3.4.3 Are External or Internal Peer Reviewers Preferred?
External peer reviewers are generally preferred, particularly for most final work products.
For some work products, like those reviewed at interim steps, either external or internal peer
review may be appropriate. Selection of internal peer reviewers should be based upon technical
expertise, available time and "address" — that is, they should not come from the immediate office
or group producing the product or have any other connection with the product or document being
peer reviewed. External peer reviewers should be selected based upon technical expertise as
well, however, care must be taken not to use individuals who have been involved in the
development of the work product. (See Section 1.2.6; see also Sections 1.4.6 to 1.4.9).
3.4.4 What is Important in the Mix of a Peer Review Panel?
A peer review panel or group can number from just a few individuals to ten or more,
depending on the issue, the time and resources available, and the broad spectrum of expertise
required to treat the range of issues/questions in the charge. Objective technical expertise and
lack of a conflict of interest are critical in selecting peer reviewers. Naturally, experts whose
understanding of the specific technical area(s) being evaluated are necessary; nevertheless, it is
also important to include a broad enough spectrum of other related experts to completely
evaluate the relevant impacts on other less obvious concerns (i.e., to comment not only if the job
is being done right, but also whether the right job is being done). For example, for health related
-------
Page 58 Peer Review Handbook
peer reviews, experts in such fields as ecology and economics may provide very useful insights.
Although persons who are familiar and have a substantial reputation in the field are often called
upon repeatedly to be reviewers, it is important to keep a balance with new people who bring
fresh perspectives to the review of a work product.
There is usually a continuum of views on any issue. To the extent possible or
practicable, selected experts should have technically legitimate points of views that fall along the
continuum. A review panel should include experts that are considered "mainstream" (nearer the
center of the continuum) as well as those further along the continuum (while generally avoiding
the extremes). This will help maintain a balanced review panel, while allowing all views to be
expressed and discussed. A balanced panel will allow consensus building (if consensus is the
object of a particular peer review; if not, it allows a spectrum of (re)views for the Agency to
evaluate). As a general rule, experts who have made public pronouncements on an issue (e.g.,
those who have clearly "taken sides") may have difficulty in being objective and should be
avoided.
3.4.5 What is a Conflict of Interest?
The matter of obtaining a fair and credible peer review, as well as maintaining the
credibility of the Agency and the Agency's scientific products, is of paramount importance. Peer
review leaders are strongly encouraged to obtain peer reviewers who do not have a legal or
perceived conflict of interest (i.e., creates the appearance that the peer reviewer lacks impartiality
or objectivity). In reality, we recognize that a totally independent peer reviewer is rare and very
difficult to identify (see Section 3.4.6 for steps to take to ensure a credible peer review).
Conflict of interest is a situation in which, because of other activities or relationships
with other persons, an individual is unable or potentially unable to render impartial assistance or
advice to the Agency, or the person's objectivity in performing the work is or might be otherwise
impaired, or a person has an unfair competitive advantage. Generally, a conflict of interest of
interest arises when the person is affected by his/her private interests, when he/she or his/her
associates would derive benefit from incorporation of their point of view in an Agency product,
or when their professional standing and status or the significance of their principal area of work
might be affected by the outcome of the peer review. Clearly, peer reviewers should not be
placed in the position of reviewing their own research and analyses that form the basis of the
work product under review as this might impair their objectivity. Whenever there are questions
about conflicts of interest, you should contact the appropriate official in OGC for clarification.
Be aware that COI can be used in a generic fashion in speaking of common, everyday
experience that also involves elements of ethics. In most cases, that is how it is treated in the
Peer Review Handbook. However, COI is a very specific issue when used in a governmental
-------
Peer Review Handbook Page 59
contracting sense. In the contracting sense, the Federal Acquisition Regulations (FAR), the
Environmental Protection Agency Acquisition Regulations (EPAAR), and other internal Agency
documents define COI and describe the Agency's COI policies and procedures. The Handbook
provides guidance in some of these aspects related to peer review (see Section 3.6 and Appendix
E - Example Statements of Work for Contracts), however, specific questions should be
addressed to the Office of Acquisition Management.
3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of
Conflicts of Interest?
Before finalizing the selection of reviewers, the Peer Review Leader should ascertain
whether each potential peer reviewer's involvement in certain activities could pose a conflict of
interest (COI) or create the appearance that the peer reviewer lacks impartiality. One way of
identifying conflicts is to ask potential reviewers about current and prior work, and prior clients
that might create conflicts or the appearance of a lack of impartiality in carrying out peer review
activities. This information obtained by the Peer Review Leader becomes part of the peer review
record. When the peer review process is being conducted by a contractor, the requirement for
addressing peer reviewers' possible conflicts of interest should be highlighted in the Statement
of Work of the work ordering instrument (e.g., Work Assignment, Delivery Order, Task Order,
etc.) and is a matter that is bound by contractual clauses with the Contracting Officer as the final
Decision Maker in contracting matters.
Care must be taken to reduce actual or potential organizational or personal conflicts of
interest between the reviewers and the work product under review. Remember, each potential
conflict situation is unique and must be treated on a case-by-case basis. The following are
considerations that should be addressed in evaluating COI:
a) attention to the employment, financial, and professional affiliations of the
participants;
b) exploring directly the issue with each of the participants before the review
process takes place;
c) disclosing publicly at the beginning of meetings any previous involvement with
the issue;
d) in the cases of regular government employees and Special Government
Employees (SGEs), filing annual (or updated, as appropriate) Confidential
Financial Disclosure Forms (OGE Form 450), discussing any potential conflicts
of interest with their designated ethics official (DEO), and advising the Peer
Review Leader of any relevant concerns;
-------
Page 60 Peer Review Handbook
e) in the case of non-Federal peer reviewers, provide them a copy of the peer review
COI inquiry (see Appendix F - Useful Forms). This form is sent to each
prospective peer reviewer by the Peer Review Leader (or contractor, in the case of
contractor-run reviews) to advise them of the need to address COI issues prior to
the actual review taking place. A follow-up contact with the Peer Review Leader
(or contractor, in the case of contractor-run reviews) is then made to discuss any
relevant issues. The Peer Review Leader then documents this effort in the peer
review record; this includes a summary provided by the contractor documenting
their inquiries and efforts.
The Peer Review Leader needs to ensure that the peer review COI inquiry (see Appendix
F - Useful Forms) took place and this appears in the peer review record. You should ensure that
any documentation in the peer review record does not contain information subject to privacy
laws (for example, see The Privacy Act of 1974. 5 U.S.C. § 552a - Records maintained on
individuals; a copy can be found at: http://www.usdoj.gov/04foia/privstat.htm). Peer Review
Leaders should ensure that any applicable EPA record keeping regulations are followed (for
details, see EPA's National Records Management Program; http://www.epa.gov/records/). For
information on Freedom of Information Act (FOIA) requests, please see EPA's Ethics Advisory
88-12 (http://intranet.epa.gov/ogcmioO l/ethics/88-12.htm). If you have any questions, be sure to
ask your appropriate Office of General Counsel and/or office contract official(s).
Established peer review groups such as the Science Advisory Board provide useful
models for addressing balance and COI issues. Assistance in identifying conflicts of interest and
in providing an appropriate response can be obtained from the Office of the General Counsel.
Additional advice can be obtained from one of the Agency's designated ethics officials (DEOs).
Assistance in evaluating conflicts of interest in a contractual situation require the involvement of
the Contracting Officer and the resources available within the Office of Acquisition
Management.
Of course, conflicts do not necessarily arise merely because a peer reviewer knows
something about the subject matter. In fact, experts with a stake in the outcome — and therefore
a potential conflict — may be some of the most knowledgeable and up-to-date experts because
they have concrete reasons to maintain their expertise. Such experts could be used provided the
potential conflicts of interest are disclosed and the peer review panel or group being used as a
whole is balanced. In some cases, however, the conflict maybe so direct and substantial as to
rule out a particular expert, for instance, a potential peer reviewer who may have a client or
employer with a direct financial stake in the particular specific party matter under review (e.g.,
i) a Federal grant or contract to the potential peer reviewer or his/her employer that relates to the
matter under review; ii) the potential peer reviewer's or their company's work on a specific
chemical under review). However, review of a general methodology that applies to numerous
-------
Peer Review Handbook Page 61
chemicals would not necessarily raise such a concern. (Note: COI language should be made part
of contracts/statements of work (SOW) or purchase orders (PO) — see Section 3.6.)
A Peer Review Leader may also want to adopt measures that will prevent creation of
conflicts as the peer review is underway. Any measures contemplated that involve a contractual
action must be coordinated with the cognizant Contracting Officer. Some measures might
include clauses in a contract or purchase order that require reviewers to receive advance approval
on future work, or place limits on such work, while they are performing the current peer review.
Note that at some level these types of measures will discourage experts from serving as peer
reviewers. (See Section 3.6.5 for further information dealing with contracts and suggestions for
appropriate management controls.)
3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews?
There may be instances where parties external to EPA will want to conduct and/or pay
for a peer review on a particular work product (presumably their own work product or one they
are closely interested in, or they wouldn't be interested in expending resources). This may look
benign at first blush, but is a very complex and sensitive situation that can raise significant
concerns for perceived and/or actual conflicts of interest for interested parties "paying" for a peer
review of their own work product. While the Agency cannot prevent external parties from
conducting and paying for a peer review, it is desirable that any such peer review meets the
intent of the Agency's Peer Review Policy and adheres to the principles and guidance in this
Handbook. If the external party submits their work product and accompanying peer review, the
materials will be treated by the Agency as anything else submitted for the Agency's evaluation.
We will evaluate the work product and the peer review for scientific credibility and
validity before making any decisions based on the materials.
[This issue is under active Agency consideration at this time; therefore, users of this Handbook
should be aware that the language in Section 3.4.7 will likely change in the future.]
3.4.8 Are There Constraints to Selecting Peer Reviewers?
Sometimes the need for a peer review is accelerated due to a court-ordered deadline or
other time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain
external peer reviewers in time to conduct a full external peer review. It may even be impossible
to conduct a small scale internal peer review using just a few individuals. Mechanisms for
identifying and using a small number of peer reviewers should be developed so that quick,
effective peer review can be included for even the most rapidly moving products.
-------
Page 62 Peer Review Handbook
Another possible constraint involves confidential business information (CBI). To
evaluate certain Agency-generated studies properly, some peer reviewers may need access to
CBI. However, unless the reviewers are Federal employees with CBI clearance, the Agency
doesn't have the independent authority to disclose CBI to them. Therefore, whenever
contemplating the use of outside peer reviewers, Agency staff should determine whether the
reviewers will need access to CBI. If they don't have CBI clearance, the Office of the General
Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to
disclose the information to peer reviewers.
Offices need to be aware of the requirements of the Federal Advisory Committee Act
(FACA) when establishing peer review mechanisms (see Section 2.8). Federal advisory
committees that are subject to chartering by the General Services Administration must hold
meetings that are open to the public, and have balanced membership requirements. The Office
of the General Counsel should be consulted regarding the applicability of FACA to peer review
panels.
3.4.9 If State Employees are Used as Peer Reviewers, Can EPA Pay Them for this
Service?
Maybe. First, the Peer Review Leader needs to determine: a) if the State agency has a
policy on whether their employees can perform this type of work for EPA, and b) if the person
will be acting as an individual or as an employee of the State. If the State allows the person to
be a peer reviewer for EPA and the person is acting as an individual, we can pay them for their
peer review services. If the State allows the person to be a peer reviewer and the person will be
acting as a State employee, we can only pay them if State policy allows us to. The most efficient
vehicle for paying the State employee will likely be a contract or purchase order (see Sections
3.6.6 & 3.6.7), but you should consult with OGC and the procurement office for advice. In most
instances, EPA can also pay travel expenses (consult with your administrative staff for details).
If we don't pay a State person for their peer review services, they must sign an agreement
stating that they don't expect payment (See Section 3.6.1 - gratuitous services).
3.5 Materials for Peer Reviewers
3.5.1 What Instructions Do You Give Peer Reviewers?
The Peer Review Leader is responsible for ensuring that peer reviewers understand their
responsibilities (see Section 3.6 if a contract is involved):
-------
Peer Review Handbook Page 63
a) Advise the Agency of actual or potential organizational or personal conflicts of
interest or other matters that would create the appearance of a lack of impartiality
(e.g., see Section 3.4.6)
b) Provide written comments in specified format by the specified deadline that are
responsive to the charge
c) Comply with any requests for not disclosing draft work products to the public
3.5.2 What Materials Should be Sent to Peer Reviewers?
For a peer review to be successful, peer reviewers should receive several documents at
the beginning of the process. Typically, the most important among these documents are the
charge letter and the current work product. The charge letter describes what the peer reviewers
are being asked to do, and should serve to focus and structure the review. The work product is,
of course, the material being subject to peer review.
Remember, no documents should be provided directly to a potential peer reviewer when
that reviewer is going to be working under a contract or purchase order. In the case of a contract,
the Agency provides the work product with associated background material to be peer reviewed
to the prime contractor who in turn distributes these documents to the peer reviewers. In the
case of a purchase order, the "charge or statement of work" must be part of the PO (purchase
order) and the provision of any documents needs to be coordinated with the purchasing agent
handling the order.
a) Essential documentation for each peer reviewer includes:
1) A current copy of the work product to be peer reviewed with associated
background material. The work product needs to be of the best possible
scientific/technical quality to ensure an adequate and useful peer review.
2) A clear charge or statement of work seeking informed comment on
identified issues to properly focus the efforts of the peer reviewers and
ensure that their individual efforts can be compared or contrasted.
3) Some information concerning the process that you use for the peer review,
including the due date of reviewer comments, the format of those
responses, and a point of contact in case the peer reviewer has questions.
Responses should be written and submitted to the Peer Review Leader by
an agreed upon deadline. In certain rare cases, oral commentary maybe
-------
Page 64 Peer Review Handbook
sufficient. However, in such cases, a follow-up written response for the
record is required.
4) In some cases, Agency materials being peer reviewed will be available to
the public, even if they are marked as drafts. For example, all materials
reviewed by the SAB are available. Agency managers may also decide
that a broad accessibility has benefits for the Agency. In other cases,
confidentiality needs to be maintained. In these cases, each peer reviewer
must be informed of the need for confidentiality with regard to the release
of Agency products that are stamped as "DRAFT" or "DRAFT - Do Not
Cite, Quote, or Release." Premature release of draft Agency products,
views, or positions is inappropriate and can be damaging to the credibility
of the Agency or the peer reviewer. While not necessarily having legal
consequences, such language will be included in the charge to the peer
reviewers. Other mechanisms to use in discouraging premature release
include a disclaimer that appears in a separate section at the front of the
document and creating the document with watermarks clearly delineating
DRAFT status (or a header or footer that states DRAFT status) on every
page. In addition, in any solicitation for peer reviewers, the necessity for
confidentiality and the non-release of materials shall be emphasized.
b) Useful, but not critical materials that may be sent to peer reviewers include:
1) The name, address, and phone and fax numbers, and/or Internet address of
each peer reviewer working on the specific review
2) A bibliography and/or any particularly relevant scientific articles from the
literature
3) A work product that has line numbering added in the margin for ease in
providing and referencing comments
c) Peer Reviewers should be given what is needed to complete their task — they
should not be overburdened with excess material.
3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?
a) When EPA Conducts the Peer Review - The Peer Review Leader normally has
administrative contacts with the reviewers during the development and conduct of
the peer review. In some cases (e.g., SAB peer review), peer reviewers may also
receive a briefing on the product to be peer reviewed. Otherwise, the Peer
-------
Peer Review Handbook Page 65
Review Leader and other EPA staff should not contact the reviewers during the
course of the review. Such contact can lead to perceived conflicts or
inappropriate direction that could compromise the independence of the review.
b) When the Contractor Conducts the Peer Review - If peer review is conducted
under a contract mechanism, EPA must limit direct contact to the prime
contractor's designated representative and not have general contact and direction
to the contractor's staffer peer reviewers (sub-contractors). Note, when a peer
review is conducted under a contract, there are constraints where EPA staff are
prohibited from contacting peer reviewers to avoid personal services
arrangements. Personal services contracts exist when the nature of the
relationship between the contractor and the EPA can be characterized as an
employer - employee relationship. Any communications with peer reviewers
must be coordinated through the prime contractor.
3.6 Peer Review Services
A range of peer review services are available to the Agency including internal, external
(gratuitous services, contracts, purchase order), and Special Government Employee (SGE)
mechanisms. The mechanism selected is generally based on the nature of the scientific or
technical work product.
3.6.1 What are Gratuitous Services for Peer Review?
The provision of peer review products or services to EPA without compensation are
provided as so-called "gratuitous" services. If a person wishes to perform peer review services
for EPA without compensation, the EPA must ask them to sign an agreement whereby the
person agrees to provide the prescribed peer review services as gratuitous services, with no
expectation of receiving compensation for these services from EPA (see Appendix F for EPA
Form 3100-14 which is used in situations where a gratuitous services contract is not used). An
agreement (such as Form 3100-14) must be executed because the AntideficiencyAct (31 U.S.C.
§1342) prohibits the Agency's acceptance of'Voluntary" services ("voluntary services" are
services provided to EPA without an agreement in advance that such services are provided at no
cost to EPA). Note that persons cannot waive compensation (i.e., agree to provide gratuitous
services) for which there is a statutory right to payment, unless a law permits the waiver. (For
situations concerning State employees, see Section 3.4.10. Laws that permit services without
compensation are 5 U.S.C. §3109 for experts or consultants and 5 U.S.C. §3111 for student
volunteers.)
-------
Page 66 Peer Review Handbook
3.6.2 Can I Use a Contract to Obtain Peer Review Services?
The Agency may contract for peer review services. The contract maybe written solely
for peer reviews or be included as one of several specifically described interrelated tasks in a
contract that requires the contractor to provide more than just peer review services.
For assistance in preparing the necessary pre-award documents, program officials should
consult Chapter 2 of the Contracts Management Manual (CMM). Chapter 2 is available on
EPA's Intranet (http://epawww.epa.gov/oamintra/policy/cmm.pdf) and on the Agency LAN
Services (Administration/ Management/OAM Procurement Policy Information).
3.6.3 How Do I Write a Statement of Work for Contracts?
The Statement of Work (SOW) must clearly specify that the contractor is responsible for
preparing peer review evaluations and set forth guidelines for the peer review of scientific or
technical documents. The contractor may perform the peer review with in-house staff,
subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness
and defensibility must be developed by the program office and made part of the contract. The
contractor would then ensure that the peer reviews adhere to the guidelines.
The SOW should include the list of questions that EPA wants the contractor to include in
the charge to the peer reviewers. While the contractor will be the one that will prepare and send
the formal charge to the peer reviewers, EPA should provide the list of questions to the
contractor (see Section 3.2 for general discussion on charge to peer reviewers).
Within the SOW, the contractor should be directed to inquire whether prospective peer
reviewers have any actual or potential organizational or personal conflicts of interest or other
matters that would create the appearance of a lack of impartiality, including whether they have
had or presently have a financial relationship with EPA. Further, the SOW needs to specify that
contractors are required to provide EPA with a summary of the procedures and efforts made to
identify, disclose, and ensure that no independence or conflict of interest concerns arise during
the performance of the contract.
The SOW cannot simply define the role of the prime contractor as arranging for the
services of others to perform peer reviews and logistics for meetings. Unless the prime
contractor is clearly tasked with responsibility for performing peer reviews, individual peer
reviewers' fees and associated travel expenses are not payable under the contract.
The EPA may pay for the reviewer's comments or evaluation, and also for attendance at a
meeting with the Agency and other reviewers to discuss the results of the peer review. If the
SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the
-------
Peer Review Handbook Page 67
Agency and other peer reviewers to discuss the results of the peer review, payment is
appropriate. The peer reviewer's attendance at the meeting would then be part of contract
performance.
Example statements of work are found in Appendix E (Example Statements of Work for
Contracts).
3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive Activities?
Contracts that provide services that support or improve Agency decision-making or
policy development are subject to special management controls. These services include services
acquired from non-Governmental sources by contract to support or improve Agency policy
development, decision-making, management, and administration, or research and development
activities. See Federal Acquisition Regulation (FAR) 37.201 for a more specific definition of
AAS. Such services may take the form of information, advice, opinions, alternatives,
conclusions, recommendations, training, and direct assistance. For additional information on
advisory and assistance services and sensitive activities, program officials should review Chapter
2 of the CMM.
New contracts for these services require management approvals prior to issuance of the
solicitation. For the thresholds that have been established for approval of these justifications,
see Figure 5, Item B Management Approvals set forth after Chapter 2 of the CMM (for current
approval levels).
3.6.5 What are Some Management Controls for Contracts?
Contracting for peer review services is permitted. However, because of the potential for
improper use of these contracts, special management controls are required.
a) Inherently Governmental Functions (IGFs) - OFPP Policy Letter 92-1, dated
September 23, 1992, describes (1) functions that are inherently governmental and
must be performed only by Government employees and (2) functions that may be
contracted, but so closely support Government employees in their performance of
IGFs that the contract terms and performance require close scrutiny by Federal
officials. Federal Acquisition Regulation (FAR) coverage of inherently
governmental functions is at FAR Subpart 7.5.
Peer reviews represent only a contractor's recommendations, advice or analysis of
a document. Agency officials must make the official Agency decision regarding
acceptability and/or quality of the document. To ensure that Agency officials are
not improperly influenced by recommendations in the peer review, management
-------
Page 68 Peer Review Handbook
controls must be included in the contract. One possible control would be to
require the peer reviewers to submit with their evaluations or comments a
description of the procedures used to arrive at their recommendations; a summary
of their findings; a list of sources relied upon; and make clear and substantiate the
methods and considerations upon which their recommendations are based. To the
extent possible, the contract should set forth any guidelines or criteria for
performance of the peer review. Agency officials should document their
evaluations of the quality and validity of the peer review.
b) Conflict of Interest (COI) — Another important factor is that the objectivity of
the peer review should not be improperly influenced or undermined by the
contractor performing the review. To identify and avoid or mitigate actual or
potential COI, the contract should include controls. Such controls might require
the contractor to report on prior and current work, and prior clients that might
create COI. Other controls might include Agency review and placing limits or
advance approval on future work. There should also be procedures implemented
to assure that the contractor does not gain an unfair advantage in future
requirements as a result of their performance of peer reviews. Program officials
should consult the Contracting Officer (CO) for special contract clauses.
FAR coverage of conflicts of interest is at FAR Subpart 9.5. The EPA
Acquisition Regulations (EPAAR) at 48 CFR Subpart 1509.5 generally mandates
conflicts of interest solicitation provisions and contract clauses, but makes them
optional for procurements accomplished through the use of simplified acquisition
procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507-l(b)(3) and
48 CFR 1552.209-70, -71 & -72 as additional resources.
Contract for peer review services: An EPA contracting officer will include
conflicts of interest solicitation provisions and contract clauses as a matter of
course without involvement by the EPA project officer, if the peer review
services are not obtained pursuant to the simplified acquisition procedures in
FAR Part 13. If the peer review services are subcontracted pursuant to a prime
contract, then the prime contractor is ordinarily required to include a conflicts of
interest clause substantially similar to the conflicts of interest clause in the
primary contract in its subcontract to the peer reviewer.
Although the EPA contracting officer and/or prime contractor has the primary
responsibility to include the required conflicts of interest provisions/clauses, the
EPA project officer may nevertheless wish to:
-------
Peer Review Handbook Page 69
1) Highlight the conflict of interest requirements in the Statement of Work
(SOW) for the procurement of the peer review services
2) Develop a specific conflict of interest clause regarding the peer review at
issue as a substitute or in addition to the standard conflicts of interest
clause
3) Review the solicitation/contract to make sure that the required conflicts of
interest clause has been included
4) Ensure that the SOW includes a requirement for the contractor to provide
EPA a summary of the work performed to identify and resolve conflict of
interest/independence concerns with peer review candidates
Simplified acquisitions of peer review services: Although conflict of interests
requirements are optional for simplified acquisitions, they are nevertheless a good
idea. Accordingly, an EPA project officer obtaining peer review services with
simplified acquisition procedures (see Section 3.6.7) should request the
purchasing agent/contracting officer to include a conflict of interest solicitation
provision and contract clause in the purchase order.
c) Confidential Business Information (CBI)/Privacy Act Protected Information
and Other Sensitive Information — When peer reviewers are not employees of
the United States Government, it is unlikely that the EPA will have authority to
give reviewers access to confidential business information in the absence of
consent for such disclosure by the CBI submitter. Therefore, all documents
provided to non-Federal reviewers must be screened for information claimed as
CBI. Even where business information has not been explicitly claimed as CBI, if
it is of a kind where the submitter might be expected to object to its release, prior
to release the submitter must be asked whether it wishes to assert a claim, unless
the submitter has previously been informed that failure to assert a CBI claim may
result in disclosure without notice. Language is included in the contract to clearly
identify any required procedures or processes prior to release of any protected
information, including any requirements for confidentiality agreements, as well as
limits on use and disclosure of the data by contractor personnel.
d) Personal services — Under contracts, the EPA may not engage the peer reviewers
in any improper personal services relationships, i.e., an arrangement under which
contractor personnel are subject to relatively continuous supervision and direct
control by an Agency official or employee. These relationships are characterized
-------
Page 70 Peer Review Handbook
as one where the contractor employee interacts with the Agency in a manner
similar to that of a Federal employee.
To avoid these improper relationships, program officials should write well-
defined SOWs. The SOWs should set forth the requirements in detail for work to
be performed independently, including the manner in which it will be evaluated.
The SOW must set forth what work is to be performed not how the work is to be
performed. Technical direction may be used to clarify ambiguous technical
requirements to ensure efficient and effective contractor performance, and is not
considered supervision or assignment of tasks. For additional information,
program officials should consult EPA Order 1901.1 A. Use of Contractor Services
to Avoid Improper Contracting Relationships and FAR Subpart 37.1
3.6.6 Can I Identify and/or Select Peer Reviewers When Using a Contract?
No. Program officials cannot interfere in a contractor's authority and responsibility to
perform work by "selecting" who will perform the peer review (doing so may invoke FACA —
see Section 2.8). The Federal Acquisition Regulation (FAR) governs the Contracting Officer's
(CO's) and program officials' relationship with the contractor. Keep in mind that for contracting
purposes, contractors are required to obtain Contracting Officer (CO) approval of subcontractors
and the CO generally seeks the input of the work assignment manager (WAM — the WAM may
be the Peer Review Leader in many cases) before approving the use of subcontractors.
EPA can establish criteria for the sort of individuals that might participate on a peer
review panel. However, the Agency must not be involved in the selection of individual peer
reviewers, and should avoid commenting on the contractor's selection of peer reviewers other
than to determine whether the panel, once selected, meets the criteria established. EPA may
identify a pool of qualified subcontractors and consultants to the prime contractor (listed in
alphabetical order), but cannot direct the use of any particular subcontractor or consultant.
3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers?
In some instances, peer reviewers can be obtained via simplified acquisition procedures.
The acquisition of supplies or non-personal services from the open market and on a sole source
basis when the aggregate amount involved in anyone transaction does not exceed $100,000
constitutes a simplified acquisition (FAR Part 13). The same considerations in the Section 3.6.5
discussion on IGFs, COI, access to CBI, and personal services apply to simplified acquisitions.
Normally, the Government issues a purchase order directly to the individual peer reviewer,
instead of to a prime contractor who may subcontract for performance of the peer review.
-------
Peer Review Handbook Page 71
a) Approvals — All small purchases for peer reviews are considered Advisory and
Assistance Services (AAS). See Chapter 2 of the Contract Management Manual
(CMM) for the appropriate approval levels for AAS.
b) Competition — The Federal Acquisition Regulation (FAR) requires competition
for purchases in excess of $2,500. Purchases for more than $2,500 and not more
than $100,000 are to be made only from small businesses unless the Contracting
Officer is not able to obtain two or more offers from small businesses that are
competitive in terms of market price, quality, and delivery. Only one source need
be solicited if the Contracting Officer determines that only one source is
reasonably available. Contracting Officers are encouraged to use best value.
c) Procurement Requests — Program Officers should include the following in all
PRs for the purchase of peer reviews:
1) A fixed-price amount at or below the simplified acquisition threshold
2) A detailed description of the requested services, inclusive of:
(a) Total quantity per line item
(b) Estimated unit price per line item
(c) Total cost per line item
(d) Specific deliverables for each line item
(e) Total cost of the purchase request
3) Reference FAR Subpart 3.6 and Environmental Protection Agency
Acquisition Regulation (EPAAR) Subpart 1503.601 regarding sources
from Government employees or organizations owned and controlled by
them.
4) Provide sources from small businesses, if available.
5) If the request is a sole source purchase, justification must be provided in
accordance with the EPAAR Subpart 1513.170-1.
-------
Page 72 Peer Review Handbook
3.6.8 How is Travel Handled with Contracts or Purchase Orders?
Funds obligated on a contract or purchase order are available to pay for the costs of
producing the peer review including the travel costs and fee of the peer reviewer.
The EPA acquires peer reviews through simplified acquisitions issued directly to peer
reviewers or through contracts with companies, which provide the peer review services. By
issuing a purchase order or awarding a contract for peer review services, the EPA may pay not
only for the peer review services/comments, but also for participation in a meeting with the
Agency and other reviewers to discuss comments. The scope of work of the contract must
require the contractor or individual peer reviewer as appropriate to provide peer review services
and indicate whether the contractor or peer reviewer will be required to discuss a specific peer
review work product with the Agency and/or with other peer reviewers. Participation in a
meeting to discuss a peer review work product would then be part of the contract's performance.
Thus, the contract may serve as the mechanism to pay for peer review services and associated
travel expenses to provide comments to the EPA.
3.6.9 How is Travel Handled with Special Government Employees?
The term Special Government Employee (SGE) is defined in 18 U.S.C. 202(a) as an
officer or employee of an agency who performs temporary duties, with or without compensation,
for not more than 130 days in a period of 365 days, either on a full-time or intermittent basis.
Travel and per diem expenses of experts hired as SGEs for peer review may only be paid
through the issuance of invitational travel orders (5 U.S.C. §5703). These invitational travel and
per diem expenses should be charged to an appropriate EPA travel account. The Federal Travel
Regulations govern the invited travelers reimbursement.
Members of the SAB, SAP, and other FACA advisory committees are often brought on
board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory
committee members (SGEs) through a contract.
-------
Peer Review Handbook Page 73
4. COMPLETING A PEER REVIEW
4.1 Overview
Performance of the formal peer review is not the final stage in the development of the
work product. Rather, it is an important stage in developing the work product, with the final
work product representing the true end of the peer review. As a result, the peer review process
closes with three major activities: evaluating comments and recommendations, utilizing peer
review comments for completing the final work product, and organizing and maintaining a
record of the peer review.
Careful attention to all of these elements, singly and together, assures a credible peer
review process. Conversely, inattention can nullify the peer review attempt. A well-planned
peer review applied to a reasonable quality starting work product, followed by responsible,
visible utilization of peer review suggestions in the final product assures a credible product for
use in Agency decision-making.
_, . , ., , The peer review is not complete until
The peer review is not complete until the ,, . ,
r . r , . , the peer review comments are
peer review comments are incorporated into the . , ,. , ., ,, , ,
7. , t , . 1 incorporated into the final work
final work product, or reasons are stated why , .
, t i4. t_ • 4. j product.
such comments are not to be incorporated.
However, for the purposes of the annual report to
the Agency's Peer Review Product Tracking
Database only, the work product can be moved from List B (Candidate Products for Future Peer
Review) (or List C - Products for Which a Decision has been made not to Peer Review) in some
instances) to List A (Products Peer Reviewed Since 1991) when the Decision Maker decides on
how the peer review comments will be addressed and this decision is documented in the peer
review record.
The peer review record is completed only when it contains a copy of the final work
product (when there is one) that addresses the peer review comments.
4.2 Final Work Product
4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product?
The Peer Review Leader must carefully evaluate and analyze all peer review comments
and recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers
simplifies organizing and analyzing comments. Also, any other issues that are raised need to be
identified and evaluated.
-------
Page 74 Peer Review Handbook
The validity and objectivity of the comments need to be evaluated. Analyses may
include consultation with other experts/staff within the Office and/or Agency. Adequate
documentation is needed to show that comments are accepted or rejected - the documentation
can be brief, but must address the legitimate, valid comments. The peer review record must
contain a document describing the Agency's response to the peer review comments.
The Peer Review Leader should brief the Decision Maker (including all appropriate
managers in the Peer Review Leader's chain of command) on the charge, profile of peer
reviewers, the peer review comments, and provide a proposal on how to address the peer review
comments. It is the responsibility of the Peer Review Leader to obtain Decision Maker approval
of the approach to addressing the peer review comments. The Peer Review Leader should
clearly identify for the Decision Maker any major peer review comments that will not be
accepted and why.
Comments that have significant impact on time, budgetary, and/or resource requirements
are particularly important and need to be evaluated in consultation with management. These
comments may lead to allocation of additional resources and a revised schedule for the
completion of the work product.
4.2.2 What Actions are Potentially Forthcoming from Peer Review?
Peer review comments and recommendations may entail significant impacts on the
planned project schedule, budget, or other resource requirements. Management decisions related
to revisions in one or more of these areas may be appropriate.
The substantive issues or concerns expressed by peer reviewers may suggest that wider
scientific and technical consultation is needed to ensure the adequacy of the work product.
The peer review comments and recommendations on a final product may pro vide a basis
for bringing the associated project to closure.
4.2.3 What Should the Final Work Product Say About the Peer Review Process?
If the product has been peer reviewed, you should describe the peer review in the
document. Frequently, this will be part of a description of the process of developing the product.
It can be brief and does not need to describe the process or discuss the peer-review comments in
great detail. The description can be included in an introduction, preamble, or appendix.
When there were significant peer-review comments, and particularly if they are not being
accepted, the document will generally discuss the issue and describe the reasons for the
Agency's choices in the appropriate sections of the document. The level of detail that is needed
-------
Peer Review Handbook Page 75
is a matter of judgment and will depend on the importance and degree of controversy of the
issue.
If a major scientific or technical product has not been peer-reviewed, this fact should be
noted in the document, perhaps in an introduction or description of its scope. This section
should briefly indicate the reasons that peer review was not conducted.
Derivative products of major scientific or technical products (such as fact sheets, press
releases, and brochures) do not need to discuss whether the underlying products were peer
reviewed.
4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous?
Yes, up to a point. In the ordinary course of events, you can often discuss comments
received without attributing the comments to a specific reviewer. However, if the matter has
gone to litigation, the litigating parties can discover the names of anyone who contributed to a
Federal product, including peer reviewers. Therefore, it is not possible to totally shield peer
reviewers. In addition, it may be difficult to shield the names of the peer reviewers when the
Agency is responding to a Freedom of Information Act (FOIA) request.
If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review
Leader needs to inform the peer reviewer of the above possible eventualities. The Agency will
in the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and
their comments from public disclosure, but it is recognized in many instances, for example open
public meetings and the above circumstances, this can't be assured. Remember, the Agency is
committed to working "as if in a fishbowl" and most of its activities are transparent to the public
(except where confidential business information is concerned). It is recognized that this maybe
a deterrent to possible peer reviewers, but this is a reality that has to be understood.
4.3 Completing the Peer Review Record
4.3.1 How Do I Complete the Peer Review Record?
Once the Peer Review Leader has completed the peer review and the final work product
(where one is prepared), the peer review record is brought up to date and then archived
according to that organization's procedure (see Section 4.3.2). The peer review record must be
indexed and maintained in an organization's archive (repository). The location of the peer
review record needs to be readily identifiable so interested parties can locate and obtain materials
easily and quickly. The peer review record should be placed in any associated established public
docket, if required, in addition to the organizational archive. As a courtesy, a copy of the
revised work product may be sent to the peer reviewers for information.
-------
Page 76 Peer Review Handbook
The Peer Review Leader will collect the following materials for the peer review record
and submit for archiving; including at least (see also Section 2.5.3):
a) The draft work product submitted for peer review
b) Materials and information (including the charge) given to the peer reviewers
c) Written comments, information, and materials received from the peer reviewers
d) Information about the peer reviewers (such as reviewers' names, affiliations, and
a summary of efforts to identify potential conflicts and their resolution)
e) Logistical information about conduct of the peer review (such as times and
locations of meetings)
f) A memorandum, or other written record, approved by the Decision Maker,
responding to the peer review comments specifying acceptance or, where thought
appropriate, rebuttal and non-acceptance
g) The final work product
4.3.2 Where Should the Peer Review Records be Kept, and for How Long?
During the active conduct of the peer review, the Peer Review Leaders maintain the peer
review record with themselves until the peer review is totally completed. Minimally, the file
should be maintained until one year after the completed peer review is reported in the next
annual reporting. After that, the peer review record should be maintained for a "reasonable
period of time." Establishment and maintenance of the archive where the peer review records
ultimately reside are an organization's responsibility (i.e., not that of an individual program
manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions
will decide the appropriate level of organizational responsibility and how they will meet the
"routinely available" requirement. The peer review record may be kept with other records
relating to the overall project, as long as it is easily and separately identifiable. The peer review
record should be maintained in accordance with the Agency's record keeping schedule for such
records. One long term archiving mechanism maybe the formal archiving at the Federal
Records Center in Suitland, MD. (Note: This is the same question as Section 2.5.7, but applies
in this chapter as well).
-------
Peer Review Handbook Page 77
4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA?
Yes, it is subject to release if EPA receives a Freedom of Information Act (FOIA)
request, unless the peer review information meets the criteria for an exemption under the FOIA.
(http://www.epic.org/open_gov/foia/us_foia_act.html).
-------
-------
Peer Review Handbook
Page 79
SUBJECT INDEX
This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers
where they are found.
- A -
- B -
-C -
- D -
Administrative Procedures Act (12)
Administrator (9, 17)
Advisory and Assistance Services (66)
Analytic blueprints (13,25, 40, 49)
Annual reporting requirements (14, 18)
Anonymity of peer reviewers (75)
Archiving (47, 75)
Assistant Administrators (17)
Balancing of peer reviewers (22)
Board of Scientific Counselors (BOSC) (41)
Budget (18, 40,48)
Candidate for peer review (26)
Categories of work products (16,25)
Charge (46, 53, 76)
Chemical action reports (16)
Clean Air Scientific Advisory Committee
(41)
Compensation (65)
Competition (70)
Completing a peer review (4, 73)
Conducting a peer review (3, 53)
Conference proceedings (25)
Confidential business information (CBI)
(61, 69)
Confidential Financial Disclosure (59)
Confidentiality (63)
Conflicts of interest (22, 31, 58,67)
Contracting officer (57, 63, 68, 69)
Contracts (31, 64, 65,68)
Contracts Management Manual (CMM) (65,
67)
Controversy (27, 40)
Cooperative agreements (31)
Court ordered deadlines (44, 61)
Decision-Maker (17, 26, 39)
Deputy Administrator (14, 17, 18)
Designated Ethics Official (DEO) (59)
Division Directors (17)
- E -
- F -
- G -
-I-
- J-
-L -
- M -
Docket (46, 47,75)
Economic work products (27)
Electronic records (46)
Environmental Impact Statement (EIS) (35)
Environmental regulatory models (36)
Executive Order 12866 (13, 29)
External peer review (40, 41, 57)
Federal Acquisition Regulation (FAR) (67)
Federal Advisory Committee Act (FACA)
(49, 62)
Federal Register (12, 50)
Final work product (73)
Freedom of Information Act (FOIA) (16, 42,
60, 77)
Frequency ofpeer review (43)
Grants (31)
Gratuitous services (65)
Independent peer reviewer (10, 11,21, 55)
Inherently governmental functions (IGFs)
(67)
Interacting with peer reviewers (64)
Interagency agreement (31)
Internal peer review (41)
Internal peer reviewer (22)
Journal article (16, 42)
Legal considerations (49)
Letter reviews (41, 50)
List A (15)
ListB (15)
List C (15, 16)
List D (16,42)
Litigation (16,75)
Major impact (27, 40)
Major scientific and technical work product
(5, 9, 15,26, 37)
-------
Page 80
Peer Review Handbook
-N -
- O -
-P -
- R-
Managers planning checklist for peer review
(5)
Materials for peer reviewers (62)
Mechanisms for peer review (39)
National Academy of Sciences (31, 42)
NEPA Products (35)
Non-major scientific and technical work
products (16, 37, 38)
Novelty (27, 40)
NPDES permits (16)
Office Directors (17)
Office of Research and Development (ORD)
(14, 20, 23)
Open meeting (49)
Other organization's work products (36)
Peer consultation (11)
Peer input (10)
Peer involvement (10)
Peer review (9, 10)
Peer Review Advisory Group (PRAG) (17,
23)
Peer review comments (46, 47, 73, 76)
Peer Review Coordinator (14, 17, 20, 23)
Peer Review Leader (17-19, 39, 47, 61, 62,
73, 75)
Peer review materials (62)
Peer review mechanism (5, 39)
Peer review panel (22, 56, 57)
Peer Review Product Tracking (PRPT)
database (14, 19, 34, 45)
Peer review record (5, 21, 45, 75)
Peer review schedule (18)
Peer review services (65)
Peer reviewers (21,46, 55, 56, 61,70, 76)
Personal services (69)
Planning a peer review (2, 25)
Premanufacturing notices (PMNs) (16)
Privacy Act (60)
Procurement requests (71)
Project manager (18, 19,37)
Public comment (12)
RCRA permits (16)
Record of Decision (ROD) (25)
Records management (60)
Regional Administrators (17)
- S -
- T -
-V -
- w-
Regulatory development process (13)
Regulatory negotiations (14)
Rulemaking(13, 27, 46)
Science Advisory Board (SAB) (28, 41, 50,
56)
Science Policy Council (SPC) (14, 17, 20,
23)
Scientific Advisory Panel (SAP) (41, 50,
Scientific and technical work
product (41,50, 56)
Selection of peer reviewers (25)
Sensitive activities (55)
Simplified acquisitions of peer review
services (66)
Site specific decisions (69, 70)
Social Science work product (35)
Special Government Employee (SGE) (30)
Stakeholder involvement (59, 72)
State employees (12,13, 40, 54)
Statement of work for contracts (62)
Strategic plans (59, 66)
Subcontractor (25)
Subject matter expert (51, 57)
Tier 1 and Tier 2 (22)
TierS (13, 55)
Time frame for peer review (13)
Training (44, 55)
Travel (18, 19)
Voluntary consensus standards (71)
Voluntary services (39)
Work Assignment Manager (WAM) (57)
-------
Peer Review Handbook
Page 81
COMMONLY USED ACRONYMS
AA Assistant Administrator
AAS Advisory and Assistance Services
ASTM American Society of Testing and Materials
BOSC Board of Scientific Counselors
CBI Confidential Business Information
CFR Code of Federal Regulations
CMM Contracts Management Manual
CO Contract(ing) Officer
COI Conflict of Interest
DEO Designated Ethics Official
EIS Environmental Impact Statement
EPA Environmental Protection Agency
EPAAR EPA Acquisition Regulations
FACA Federal Advisory Committee Act
FAR Federal Acquisition Regulations
FIFRA Federal Insecticide, Fungicide and Rodenticide Act
FOIA Freedom of Information Act
FTE Full Time Equivalent
GSA General Services Administration
HHS Health and Human Services
IAG Interagency Agreement
IGF Inherently Governmental Function
IRIS Integrated Risk Information System
ISO International Organization for Standardization
LAN Local Area Network
NAS National Academy of Sciences
NEPA National Environmental Policy Act
NPDES National Pollutant Discharge Elimination System
NTTAA National Technology Transfer and Advancement Act of 1995
OAM Office of Acquisition Management
OEI Office of Environmental Information
OGC Office of General Counsel
OMB Office of Management and Budget
ORD Office of Research and Development
PMNs Premanufacture Notice
PO Purchase Order
PRAG Peer Review Advisory Group
PRPT Peer Review Product Tracking Database
RA Regional Administrator
RC Regional Counsel
RCRA Resource Conservation and Recovery Act
RIAs Regulatory Impact Analyses
-------
Page 82 Peer Review Handbook
ROD Record of Decision
SAB Science Advisory Board
SAP Scientific Advisory Panel
SGE Special Government Employee
SOPs Standard Operating Procedures
SOW Statement of Work
SPC Science Policy Council
USC United States Code
WTI Waste Technologies Industries
WAM Work Assignment Manager
-------
Peer Review Handbook Page A-l
APPENDIX A - U.S. ENVIRONMENTAL
PROTECTION AGENCY PEER REVIEW POLICY
June 7,1994
-------
Page A-2 Peer Review Handbook
June 7, 1994
MEMORANDUM
SUBJECT: Peer Review Program
TO: Assistant Administrators
General Counsel
Inspector General
Associate Administrators
Regional Administrators
Staff Office Directors
Today, I am reaffirming the central role of peer review in our efforts to ensure that EPA
policy decisions rest on sound, credible science and data (see attached policy statement).
Toward that end, as its first major task, EPA's Science Policy Council (SPC) is instituting a
program to expand and improve peer review in all EPA offices. This memorandum gives an
overview of current practices and outlines the new program.
Peer Review Practices and Policy
Peer review at EPA takes several different forms, ranging from informal consultations
with Agency colleagues who were not involved in developing the product to the formal, public
processes of the Science Advisory Board (SAB) and the FIFRA Scientific Advisory Panel
(SAP). In any form, peer review assists the Agency's work by bringing independent expert
experience and judgment to bear on issues before the Agency to the benefit of the final product.
EPA's Peer Review Policy, which responds in part to recommendations in the "Credible
Science, Credible Decisions" report, outlines general principles for peer review at EPA.
Different EPA offices have undertaken various implementing activities, including an Agency-
wide information and planning workshop, internal guideline development, and numerous
specific peer reviews. Even with these activities, however, I am concerned that EPA does not
yet have a comprehensive Agency-wide program for implementing its Peer Review Policy. I
therefore welcome the SPC initiative toward effective, efficient implementation of the policy in
all the program areas to which it applies.
-------
Peer Review Handbook Page A-3
Expanding and Improving Peer Review
The Science Policy Council and its Steering Committee have outlined a dual-track
implementation program of planning and assistance for all Agency offices. The first track has
three major milestones.
First, during the next few weeks, Steering Committee members will consult with senior
management in each office to exchange information on current peer review activities, assistance
needed, possible obstacles to implementation, and implementation planning.
Second, using information and materials developed during the first stage, peer review
task groups in each office will develop standard operating procedures (SOPs) for use in each
office, based in part on generic guidance to be issued by the SPC and in part on peer review
needs and capabilities specific to each office. The resulting SOPs will delineate as appropriate
the scope of application of peer review with respect to various types of scientific and technical
work products such as reports of original research, risk assessments, and analytical methods of
economic analysis. OARM and OGC staff will assist each office as needed on legal, budget
and administrative matters. Each AA and RA will submit draft SOPs for Steering Committee
review by July 15.
Third, the SPC review group will work with each office to complete each plan by
September 15.
In parallel with the above, consistent with the Peer Review Policy, the Science Policy
Council will work with each AA and RA to identify "major scientific and technical work
products" as peer review candidates for the coming year. This process will consider existing
and new plans for internal reviews and for Science Advisory Board (SAB), FFRA Scientific
Advisory Panel (SAP), and other external reviews. The two-fold objective is to plan reviews
for technical products covered by the Peer Review Policy and to gain experience with options
and obstacles. We will use this experience to review and revise the SOPs as needed. Also, to
establish a baseline for comparison, each AA and RA will identify the "major technical
products" completed within his/her program during the past 12 months.
The Science Policy Council has sent additional information to each office offering
guidance on the procedures that you are asked to develop and the schedule for these activities.
Please note, however, that because the policy is effective immediately, current peer review
planning should continue on present schedules in parallel with developing the formal SOPs.
-------
Page A-4 Peer Review Handbook
To begin this process, I have asked each Assistant Administrator and Regional
Administrator to designate a Peer Review Coordinator to work with the Steering Committee on
implementation activities specific to each office. I am very pleased that the Science Policy
Council is taking this important step. A comprehensive peer review program is essential to
maintaining and improving the quality of the analyses that underlie Agency actions. I look
forward to working with you and your staff on this important activity.
/s/
Carol M. Browner
Attachment
cc: Science Policy Council
Science Policy Council Steering Committee
-------
Peer Review Handbook Page A-5
PEER REVIEW AND PEER INVOLVEMENT
AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY
This document establishes the policy of the United States Environmental Protection
Agency (EPA) for peer review of scientifically and technically based work products that are
intended to support Agency decisions. Peer review is presented in the context of the broader
concept, peer involvement.
BACKGROUND
The report "Safeguarding the Future: Credible Science, Credible Decisions"1 focused on
the state of science at EPA. The panel of experts who prepared the report emphasized the
importance of peer review, especially external peer review, and the need for broader and more
systematic use of it at EPA to evaluate scientific and technical work products. Their specific
recommendation regarding peer review reads as follows:
"Quality assurance and peer review should be applied to the planning and results
of all scientific and technical efforts to obtain data used for guidance and
decisions at EPA, including such efforts in the program and regional offices.
Such a requirement is essential if EPA is to be perceived as a credible, unbiased
source of environmental and health information, both in the United States and
throughout the world."
In response to this recommendation, then-Administrator Reilly directed staff to develop an EPA-
wide policy statement, which he issued in January, 1993. The paragraphs below preserve the
core of that earlier statement while updating it to specify the role of the Science Policy Council
in guiding further implementation of the policy. Effective use of peer review is indispensable
for fulfilling the EPA mission and therefore deserves high-priority attention from program
managers and scientists within all pertinent Headquarters and Regional Offices.
EPA/600/9-91/050, March 1992.
-------
Page A-6 Peer Review Handbook
PEER INVOLVEMENT AND PEER REVIEW
EPA strives to ensure that the scientific and technical underpinnings of its decisions meet
two important criteria: they should be based upon the best current knowledge from science,
engineering, and other domains of technical expertise; and they should be judged credible by
those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement —
that is, they augment their capabilities by inviting relevant subject-matter experts from outside
the program to become involved in one or more aspects of the development of the work products
that support policies and actions.
One particularly important type of peer involvement occurs when scientifically and
technically based work products undergo peer review — that is, when they are evaluated by
relevant experts from outside the program who are peers of the program staff, consultants, and/or
contractor personnel who prepared the product. Properly applied, peer review not only enriches
the quality of work products but also adds a degree of credibility that cannot be achieved in any
other way. Further, peer review early in the development of work products in some cases may
conserve future resources by steering the development along the most efficacious course.
Peer review generally takes one of two forms. The review team may consist primarily of
relevant experts from within EPA, albeit individuals who have no other involvement with
respect to the work product that is to be evaluated (internal peer review). Or the review team
may consist primarily of independent experts from outside EPA (external peer review).
POLICY STATEMENT
Major scientifically and technically based work products related to Agency decisions
normally should be peer-reviewed. Agency managers within Headquarters, Regions,
laboratories, and field components determine and are accountable for the decision whether to
employ peer review in particular instances and, if so, its character, scope, and timing. These
decisions are made in conformance with program goals and priorities, resource constraints, and
statutory or court-ordered deadlines. For those work products that are intended to support the
most important decisions or that have special importance in their own right, external peer review
is the procedure of choice. Peer review is not restricted to the penultimate version of work
products; in fact, peer review at the planning stage can often be extremely beneficial.
SCOPE
Agency managers routinely make regulatory and other decisions that necessarily involve
many different considerations. This policy applies to major work products that are primarily
scientific and technical in nature and may contribute to the basis for policy or regulatory
decisions. By contrast, this policy does not apply to non-major or nontechnical matters that
Agency managers consider as they make decisions. Similarly, this policy does not apply to these
ultimate decisions.
-------
Peer Review Handbook Page A-7
This policy applies where appropriate, as determined by the National and Regional
Program Managers, to major scientifically and technically based work products initiated
subsequent to the date of issuance. Peer review should be employed to the extent reasonable to
relevant work products that currently are under development. This policy does not apply to the
bases for past decisions, unless and until the relevant scientific and technical issues are
considered anew in the Agency's decision-making processes.
Except where it is required by law, formal peer review (as distinguished from the
Agency's normal internal review procedures) should be conducted in a manner that will not
cause EPA to miss or need extension of a statutory or court-ordered deadline. Agency managers
still may undertake peer review if it can be conducted concurrently with necessary rulemaking
steps.
LEGAL EFFECT
This policy statement does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. On a continuing
basis, Agency management is expected to evaluate the policy as well as the results of its
application throughout the Agency and undertake revisions as necessary. Therefore, the policy
does not stand alone; nor does it establish a binding norm that is finally determinative of the
issues addressed. Minor variations in its application from one instance to another are
appropriate and expected; they thus are not a legitimate basis for delaying or complicating action
on otherwise satisfactory scientific, technical, and regulatory products.
Except where provided otherwise bylaw, peer review is not a formal part of or substitute
for notice and comment rulemaking or adjudicative procedures. EPA's decision whether to
conduct peer review in any particular case is wholly within the Agency's discretion. Similarly,
nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However,
to the extent that EPA decisions rely on scientific and technical work products that have been
subjected to peer review, the remarks of peer reviewers should be included in the record for that
decision.
IMPLEMENTATION
The Science Policy Council is responsible for overseeing Agency-wide implementation.
Its responsibilities include promoting consistent interpretation, assessing Agency-wide progress,
and developing recommendations for revisions of the policy as necessary.
The Science Policy Council will oversee a peer-review work group, which will include
representatives from program units throughout EPA to effect a consistent, workable
implementation of the policy. The work group will assist the programs in (1) formulating and,
as necessary, revising standard operating procedures (SOPs) for peer review consistent with this
-------
Page A-8 Peer Review Handbook
policy; (2) identifying work products that are subject to review; and (3) for each major work
product, selecting an appropriate level and timing of peer review.
In assisting the programs, the work group will take into account statutory and court
deadlines, resource implications, and availability of disinterested peer reviewers. The group will
work closely with Headquarters offices and the Regional Offices toward ensuring effective,
efficient uses of peer review in supporting their mission objectives. However, the Assistant
Administrators and Regional Administrators remain ultimately responsible for developing SOPs,
identifying work products subject to peer review, determining the type and timing of such
review, documenting the process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.
Because peer review can be time-consuming and expensive, Agency managers within
Headquarters, Regions, laboratories, and field components are expected to plan carefully with
respect to its use — taking account of program priorities, resource considerations, and any other
relevant constraints as well as the policy goal of achieving high-quality, credible underpinnings
for decisions. External peer reviewers should be chosen carefully to ensure an independent and
objective evaluation. The affiliations of peer reviewers should be identified on the public record,
so as to avoid undercutting the credibility of the peer-review process by conflicts of interest.
The policy is effective immediately. The peer-review work group mentioned above will
identify the focal point to whom comments and questions should be addressed and, from time to
time, will provide further information about implementation activities.
/s/
APPROVED: DATE: JUN 7 1994
CAROL M. BROWNER, ADMINBTRATOR
-------
Peer Review Handbook Page B-l
APPENDIX B - SOUND SCIENCE AND PEER
REVIEW IN RULEMAKING
Using EPA's regulatory agenda as the cornerstone of the Agency's mission,
the Deputy Administrator's memo implements an additional requirement as part of
the ongoing effort to use peer review to enhance the quality and credibility of
Agency decisions.
-------
Page B-2 Peer Review Handbook
June 15, 1999
MEMORANDUM
SUBJECT: Sound Science and Peer Review in Rulemaking
TO: The Regulatory Policy Council
Over a year ago, the Agency enhanced its peer review policy and procedures with the
finalization and release of the Peer Review Handbook. The Handbook was developed to serve
as a guide for Agency scientists and managers on the organization and conduct of peer review.
With our regulatory agenda as the cornerstone of the Agency's mission, I would now like to
implement an additional requirement as part of the ongoing effort to use peer review to enhance
the quality and credibility of Agency decisions.
Beginning July 12, 1999, the action memorandum accompanying each rule submitted for
signature must explicitly address the use of peer review in that rulemaking. The Administrator
and I want to ensure that the Agency takes complete advantage of the benefits that thorough peer
reviews can provide in our deliberations on the many challenging decisions that must be made.
A key step in producing credible regulatory decisions supported by well-documented scientific
data is ensuring that our work is based on broadly accepted scientific studies. Peer review is an
essential tool for accomplishing this goal.
In addition, I ask that, beginning today, planning for peer review be specifically
addressed in each new Analytic Blueprint. The general guidance on integrating peer reviews
into the rulemaking process is included as Attachment A. Detailed guidance on action
memoranda, updated to address peer review, is provided in Attachment B. Guidance regarding
which scientific or technical studies must be peer reviewed is contained in the fact sheet on
"Peer Review in the Rulemaking Process" in Attachment C and in the Peer Review Handbook
available at http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm. I am also taking this
opportunity to bring to your attention a memorandum from the Office of General Counsel that
discusses the implications of a recent decision in a challenge to a peer review brought against the
Agency under the Federal Advisory Committee Act (Attachment D) [note added: this
attachment is not attached to the Peer Review Handbook].
I appreciate your support in this effort. For additional information about peer review in
the rulemaking process, please contact your Steering Committee representative, your peer review
coordinator (Attachment E) [note added: this attachment is not attached to the Peer Review
Handbook}, or Phil Schwartz in the Office of Policy at xxx-xxxx.
[signed 6/15/99, AX 9807793]
Peter D. Robertson
Acting Deputy Administrator
-------
Peer Review Handbook
Page B-3
ATTACHMENT A
Regulatory Management Guidance PEER REVIEW REQUIREMENTS
Office of Policy FOR EPA RULEMAKING
June 1999
Reference: RMD 5
BACKGROUND
On February 25, 1998, the Agency reaffirmed its commitment to quality decision-making by
issuing the comprehensive Peer Review Handbook. The Peer Review Handbook is available at:
http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm, which amplified and clarified key Agency
peer review policy issues. The Handbook states that, in general, all major scientific and technical
work products related to Agency decisions should normally be peer reviewed. Peer review is a
documented, critical review of a work product, performed by experts who are independent of
those who developed the product. Work products that are developed to support regulations, such
as risk assessments, should be evaluated to determine whether they meet the criteria for "major
scientific and/or technical work products" and thus require peer review (See Attachment C).
Regulations themselves are not peer reviewed. The straight-forward application of accepted,
previously peer-reviewed economic methods or analyses in Regulatory Impact Analyses (RIA's),
are not typically subject to formal peer review. If, however, the particular facts and
circumstances of any piece of economic analysis in an RIA warrant peer review, the Agency will
accommodate those needs on a case-by-case basis.
PLANNING FOR PEER REVIEW
Remember that:
• Analytical Blueprints should discuss the plan for peer review of major scientific
and technical products. This should include a schedule for the review and should
identify the resources that will be needed.
• When negotiating rule deadlines with courts or with litigants, be sure that your
proposals allow adequate time for any needed peer reviews.
PEER REVIEW REQUIREMENTS
Program offices have the primary responsibility for complying with the Agency peer review
requirements. Senior managers must confirm in their Action Memoranda that their offices have
complied with the peer review requirements; detailed explanations are not necessary. If no major
scientific and technical product as defined by the Peer Review Handbook was used to support
the action, include the following statement in the action memorandum:
-------
Page B-4 Peer Review Handbook
Peer Review
There were no major scientific or technical products supporting this action as defined by
the Agency's Peer Review Handbook We did not, therefore, submit any support
documents for peer review.
If a major scientific or technical product was used to support the action, include the following
statement in the Action Memorandum:
Peer Review
[Insert Name ofAAship] has complied with the Agency's Peer Review requirements with
respect to the underlying major scientific and technical products supporting this action.
You may add any details you think important, but you may not modify this compliance
statement. If you utilized peer-reviewed products, but you could not fully comply with the peer
review requirements, explain in the Action Memorandum why the deviation was necessary.
FINAL AGENCY REVIEW (FAR)
For rules that go through Final Agency Review, OP will document compliance in the FAR
memorandum. This documentation will be based on the peer review information included in the
draft Action Memorandum.
These requirements will be effective on July 12,1999. After that date, OP's Office of
Regulatory Management and Information (ORMI) will only forward for signature those actions
that include one of the following:
• a statement that no major scientific or technical documents were utilized to
support the rulemaking;
• a statement of compliance with the peer review requirements for major scientific
and technical documents;
• an explanation for deviation from the peer review requirements.
Questions? Contact your Regulatory Steering Committee representative or Phil Schwartz in the
Regulatory Management Division at (202) 260-5493.
-------
Peer Review Handbook
Page B-5
Regulatory Management Guidance
Office of Policy
Revised June 1999
Reference RMD-6
ATTACHMENT B
FRAMEWORK FOR ACTION MEMORANDA
OVERVIEW
The overview provides background information which briefly describes the rule,
characterizes the environmental issue(s) or public health problem(s) being addressed, and
summarizes the history of the regulatory action. It should explain why EPA is taking this
action, and where appropriate, cover the following points:
• Define the court or statutory deadline;
• Identify whether the action amends the CFR, and if so, explain what kind of
amendment (technical, procedural, etc.);
• Identify other regulatory actions underway that will affect this particular program
or sector;
• Describe the specific environmental issue(s) or public health problem(s) being
addressed, and the goal intended by taking this action;
• Describe what the action does, and specifically, how the regulated community is
affected (performance standards, specific requirements); and
• Describe how flexible the implementation of the action will be for states and
regulated entities.
REGULATORYIMPACTS
Summarize the costs and benefits of the regulatory action (include costs to State and
local governments, and tribal communities) and the results of the economic and risk
analyses. Discuss the economic impacts on all stakeholders including, where appropriate,
the results of the regulatory flexibility analysis and the specific effects of the action on
small entities. Explain the roles of cost-benefit and cost-effectiveness analyses in shaping
the regulatory approach.
Describe the reporting and Record keeping burden and what the Agency has done to
reduce it. Indicate possible impacts on other Federal agencies.
-------
Page B-6 Peer Review Handbook
ADMINISTRATOR'S PRIORITIES
If appropriate, discuss how the regulatory action relates to the Administrator's priorities.
Explain how the regulatory action relates to a specific priority only where the relationship
is significant, clear and distinct.
STAKEHOLDER INVOLVEMENT
Briefly discuss the role of both government entities and private sector stakeholders in the
development of the action. Summarize the concerns they have raised and what the
Agency has done to address them.
INTERNAL REVIEW
Identify whether the regulatory action was developed under Tier 1,2, or 3. Describe any
outstanding issues from workgroup closure or other internal review. Identify program
offices or regions with outstanding issues and indicate why they cannot be resolved or
accommodated. Also, provide the basis for any decision made to not address an
identified cross-media impact. Where there is not a formal workgroup, identify those
offices that have reviewed the package.
PEER REVIEW
If you did not utilize major scientific and technical products as defined by the Peer
Review Handbook to support the action, include the following statement:
There were no major scientific or technical products supporting this action as
defined by the Agency's Peer Review Handbook. We did not, therefore, submit
any support documents for peer review.
If a major scientific or technical product has been used to support the action, include the
following statement:
[Insert Name ofAAship] has followed the Agency's Peer Review requirements
with respect to the underlying, major scientific and technical products
supporting this action.
You may add any details you think important, but you may not modify this compliance
statement. If you utilized peer-reviewed products, but you could not fully comply with
the peer review requirements, explain why the deviation was necessary.
-------
Peer Review Handbook Page B-7
PLAIN LANGUAGE
Either in the action memorandum or in a separate note briefly describe which elements of
plain language have been addressed.
OMB TRANSACTION
Highlight significant issues resulting from OMB's review under Executive Order 12866.
Explain any changes made to the regulatory action as a result of that review. If OMB
review was waived, please indicate this.
ANTICIPATED PUBLIC REACTION
Describe the type of public response anticipated and identify both the involved
stakeholders and the nature of their expected response. Characterize the likely reaction
to the action by all interested parties including industry; environmental groups; Congress;
state, local, and tribal governments; and OMB. Explain why the Agency should take the
action despite any controversial response anticipated from the public.
If the regulatory action will not be issued for public comment (i.e., direct final rule,
administrative stay, etc.), explain the basis for the Agency's decision not to solicit
comment.
RECOMMENDATION
Identify the action the administrator is expected to take.
NOTE: An Action Memo should generally not exceed 5 pages.
-------
Page B-8 Peer Review Handbook
ATTACHMENT C
Fact Sheet
Peer Review in the Rulemaking Process
What Role does Peer Review have in the Regulatory Development Process?
Peer review of scientific and technical work products affirms the scientific credibility of
the Agency's rulemakings. Work products need to be well planned and documented because new
rules, and the work products supporting them, may have substantial cost impacts and must often
withstand intense public scrutiny.
What Work Products Need Peer Review?
While rules themselves are not subject to peer review, any major scientific and/or
technical work products that support rules should be peer reviewed. (See Section 2.2 of the Peer
Review Handbook http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm to determine which
work products are considered major.) Designations of a scientific and/or technical work product
as "major" will largely be case-by-case. Those that are used to support a regulatory program or
policy position and that meet one or more of the following criteria are candidates for peer
review:
a) Establishes a significant precedent, model, or methodology
b) Addresses significant controversial issues
c) Focuses on significant emerging issues
d) Has significant cross-Agency/inter-agency implications
e) Involves a significant investment of Agency resources
f) Considers an innovative approach for a previously defined
problem/p rocess/methodology
The straight-forward application of accepted, previously peer-reviewed economic
methods or analyses in Regulatory Impact Analyses (RIA's), are not typically subject to formal
peer review. If, however, the particular facts and circumstances of any piece of economic
analysis in an RIA warrant peer review, the Agency will accommodate those needs on a
case-by-case basis. The decision whether to peer review any work product will be documented
through the Agency's annual peer review reporting process (see Section 1.3 of the Handbook).
When should determinations about the need for peer review be made?
The need for, and scheduling of, peer reviews should be addressed in the Analytic
Blueprint that is required for all Tier 1 and 2 regulatory actions. Work products supporting Tier
3 rules may also be considered major and Analytic Blueprints are encouraged for Tier 3 rules. In
any event, the need for, and scheduling of, peer reviews should be addressed early in the
development of Tier 3 actions.
-------
Peer Review Handbook Page C-l
APPENDIX C - EXAMPLES OF CHARGES
Please note — certain questions that are posed in charges can be responded
to with a yes or no answer. Clearly, this is not the type of response we generally
want, therefore, it is important to phrase charge questions carefully to ensure that
you receive a fully satisfactory and thoughtful response. Where a yes or no answer
might be expected, be sure to ask for a full explanation supporting the yes or no
answer.
Charges can run the gamut from rather simplistic to highly complex. The
examples shown here cover a variety of types. Examples 1 through 6 have less
complex questions and are looking for the overall quality of the reports. Examples
7 through 11 have numerous technical questions that need to be addressed and are
therefore more complex in their nature.
-------
Page C-2 Peer Review Handbook
CHARGE EXAMPLE 1 - Framework for Developing a Living Resources Research and
Monitoring Plan for the Peconic Estuary
Background
The Peconic Estuary is located on the eastern end of Long Island, New York. Under the
Federal Clean Water Act, the Peconic Estuary was named an "Estuary of National Significance"
in 1992. Because of its high concentration of rare, threatened and endangered species and
habitats, The Nature Conservancy named the Peconic Ecosystem as one of the "Last Great Places
in the Western Hemisphere."
The Peconic Estuary Program (PEP) released its draft Comprehensive Conservation and
Management Plan (CCMP) in September 2, 1999. The draft Plan addresses the environmental
management issues facing the estuary and its watershed, including brown tide (a nuisance algal
bloom), nutrients, habitats and living resources, pathogens, and toxics, as well post-CCMP
management structure, public education and outreach, and financing. A final Plan is now under
preparation. The preparation of the Framework document is called for in the draft CCMP.
With population increasing in the watershed, the Peconic Estuary is being threatened by
over-development and misuse of its resources. To fully realize the impacts of people and their
activities on this system, there must be a better understanding of how the Peconic Estuary
functions ecologically. This knowledge can only be achieved through comprehensive research
and monitoring of the entire ecosystem.
The long-term goal of this Framework for Developing a Living Resources Research and
Monitoring Plan is to develop a strategy of coordinated research and monitoring to fill
significant information gaps and assist in the planning, conservation, and management of the
Peconic Estuary. A key component of this strategy is to develop an applied, multi-scale,
integrated approach to gain a better understanding of the estuary. To achieve, this goal, the
Framework for Developing a Living Resources Research and Monitoring Plan sets broad
priorities, provides the context for specific studies, guidelines for a detailed living resources
research and monitoring plan, and stimulus for funding agencies and organizations and
researchers. The short-term goal of this Framework is to provide a basis for securing funding for
the priority initiatives described in the Framework document.
In order of importance, the objectives of this Framework document are to:
1) understand threats so as to improve resource protection through management and
conservation;
2) examine the biology and ecology of particular organisms identified in the CCMP
as important either due to their commercial or recreational value or their role in
the food web and ecosystem; and,
-------
Peer Review Handbook Page C-3
3) undertake system-wide studies of the Peconic Estuary to understand the
significance of key habitats and ecosystem productivity.
Recommendations included in this framework are based on information gaps identified
in the Characterization Report of the Living Resources of the Peconic Estuary (Bortman and
Niedowski 1998), the Habitat Module of the draft CCMP, PEP Natural Resource Subcommittee
meetings, and recommendations made at a PEP-sponsored Living Resources Research and
Monitoring workshop held in 1998. Copies of the draft CCMP and Characterization Report are
included with this package.
Tasks
The peer review charge is to assess the adequacy of this document to provide a
framework for integrated, system-wide ecological research and monitoring to understand the
dynamic, multi-scale ecological patterns and processes that sustain biota and their supporting
natural systems in the Peconic Estuary.
Specific Questions:
Peer Reviewers shall answer/comment on the following:
1) Is the organization of the document appropriate and does it present the material in
a clear and concise manner? Please explain fully.
2) In your opinion, what are the weakest and the strongest aspects of the framework?
Please make suggestions on how the weakest parts can be strengthened.
3) Are there any elements missing from the framework which you think need to be
included or which would strengthen the document? Please explain fully.
4) Are you aware of any other significant data/studies that are relevant and should be
included or referenced in this document? Please explain fully.
5) Is the stated goal realistic? Are the stated objectives adequately met? Please
explain fully.
6) Examine and critique the interrelationship among the specified assessment,
research, and monitoring priorities.
7) Does this framework present a holistic approach to preserve, protect and restore
the estuary? Does the approach cover all the relevant and important areas that
warrant investigation? If not, please state what should be done differently.
-------
Page C-4 Peer Review Handbook
8) Are the initial set of assessment, research, and monitoring priorities adequate to
provide the information needed to successfully protect and manage this system?
Please comment on the selection of priorities. Were any significant priorities
missed? What other priorities should be specified and why?
-------
Peer Review Handbook Page C-5
CHARGE EXAMPLE 2 -EPA Science Advisory Board (SAB) Review of the Environment
Monitoring and Assessment Program (EMAP) Research Strategy and Research Plan
The Science Advisory Board (SAB) is asked to review the Environment Monitoring and
Assessment Program (EMAP) Research Strategy and Research Plan. The review is requested by
the Office of Research and Development (ORD) with the following specific charge issues:
1) Previous peer reviews recommended that EMAP develop a close working relation
with EPA Program Offices and other federal monitoring efforts. Does the EMAP
strategy support the [Office of Science and Technology Policy's Committee on
Environment and Natural Resources] CENR National Monitoring Framework and
EPA Program Offices?
2) Previous peer reviews recommended that EMAP initiate a focused research
program on indicator development. Does the intramural EMAP program on
ecological indicator development, coordinated with the [ORD Science To
Achieve Results] STAR solicitations, respond to this research need?
3) Previous peer reviews recommended that the EMAP design be modified to
include a set of nonrandomly selected sentinel sites with intensive data collection.
Does the development of Index Sites as outdoor laboratories in the national parks
([National Park Service] NPS and [US Geological Survey] USGS) and selected
estuaries ([National Oceanic and Atmospheric Administration] NOAA) add this
dimension to the EMAP?
4) Previous peer reviews recommended that EMAP combine effects-oriented and
stressor-oriented monitoring approaches. Do the focused geographic
demonstration pilot studies (initially in the Mid-Atlantic region) combine these
elements?
-------
Page C-6 Peer Review Handbook
CHARGE EXAMPLE 3 - EPA Science Advisory Board Review of the Agency's National
Risk Management Research Laboratory's (NRMRL) Program
The Office of Research and Development (ORD) requests that the Science Advisory
Board review the Agency's National Risk Management Research Laboratory's (NRMRL)
program.
In the "Strategic Plan for the Office of Research and Development" (EPA, 1996a), ORD
described the relationship of risk assessment to the risk management process, and emphasized
the need for scientific and engineering research to enable sound risk management decisions and
actions. Within the framework of that strategic plan, NRMRL's mission is to conduct research
to reduce uncertainties and costs associated with making and implementing environmental risk
management decisions. NRMRL has therefore developed a research agenda to reduce risk
uncertainty that also focuses on those important, relevant issues where it can make a difference.
The charge to the SAB is to:
1) Examine and critique the research programmatic directions such as whether
NRMRL is pursuing the most appropriate research problem areas;
2) Comment on strategic directions, e.g., use of its core technical competencies,
transition from primarily extramural to an intramural R&D organization,
leveraging with other agencies and organization;
3) Review and comment on the effectiveness of NRMRL's approach to science
management, e.g., measures of success and science quality, soundness of peer
review process;
4) Examine and critique the relationship of NRMRL's risk management research
and its intended role in the risk assessment/risk management paradigm; and
5) Review and comment on the strategic balance for the next decade among
pollution prevention, technology development, remediation, and risk management
assessment activities.
-------
Peer Review Handbook Page C-7
CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the Technical
Aspects of the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM)
The EPA Science Advisory Board (SAB) is asked to review the technical aspects of the
Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM). The review
document was developed collaboratively by four Federal agencies, departments and
commissions having authority for control of radioactive materials: Department of Defense,
Department of Energy, Environmental Protection Agency, and Nuclear Regulatory Commission.
MARSSIM addresses the need for a nationally consistent approach to conducting radiation
surveys of potentially radioactively contaminated sites that are being considered for release to
the public. A condition of release is a demonstration that residual radioactivity levels do not
exceed a specified risk or dose level, also known as a release criterion. MARSSIM provides
guidance to users performing and assessing the results of such a demonstration for surface soils
and building surfaces.
The SAB is asked by the Agency's Office of Radiation and Indoor Air (ORIA) to
respond to the following charge in its review:
1) Is the overall approach to the planning, data acquisition, data assessment, and
data interpretation as described in the MARSSIM technically acceptable? Please
explain fully.
2) Are the methods and assumptions for demonstrating compliance with a dose- or
risk-based regulation technically acceptable? Please explain fully.
3) Are the hypotheses and statistical tests and their method of application
appropriate? Please explain fully.
-------
Page C-8 Peer Review Handbook
CHARGE EXAMPLE 5 - Economics - Benefits Transfer from Adults to Children
Background
EPA established the Office of Children's Health Protection (OCHP) in 1997 to support
its efforts to increase the protection of children's health throughout its programs. One of the
many difficult issues the Office is addressing is the appropriate treatment of children's health
effects in the economic analyses performed by the Agency. Policy analysis efforts at the Agency
often rely on the benefits transfer technique, and very few of the Agency's benefit transfers have
explicitly addressed children's health issues. In addition, no accepted systematic process for
conducting benefits transfer currently exists. To assist the Agency in its efforts, this paper
discusses the benefits transfer technique as it applies to estimating values for children's health.
The first section provides some general background on the technique, and its application to
estimate health-related values. The second section raises important general issues to consider
when conducting a benefits transfer for children's health values. The last section discusses the
implications of using the benefits transfer method to estimate values for children's health. The
scarcity and state of existing child-oriented health valuation literature suggests that it may be
necessary to transfer adult-oriented values to estimate child-related values (Neumann and
Greenwood 1999). However, the results of this paper suggest that transfer of these value
estimates to children at best provides estimates for a scoping analysis. In cases where these
scoping exercises indicate that children's health values may be a crucial component in the policy
analysis, primary research should be undertaken to estimate child-related values.
1) Assess the appropriateness of transferring health benefit values estimated for
adult populations to children. Describe the specific issues that arise in these
transfers. Which variables or situations improve or decrease the appropriateness
of transferring benefit values from adults to children?
2) Does the analysis support the proposition that the value of children's health
effects should be estimated differently than adult health effects? Why or why
not?
3) Identify issues for further research that would improve our ability to estimate
values for children's health effects.
-------
Peer Review Handbook Page C-9
CHARGE EXAMPLE 6 - Economics - Study of Municipal Government Costs and
Financial Impacts from Environmental Regulations
Background
Municipalities play a major role in supplying environmental services. Local
governments have taken responsibility for providing drinking water, sewage treatment, and
waste disposal in a majority of communities. Over the past fifteen to twenty years, most of the
mandates found in the federal environmental legislation enacted in the early 1970s have been
met. The increase in the number of people served and improvements in the quality of local
environmental services have been considerable, as has the investment in public infrastructure to
meet these laws.
Recent revisions to the environmental legislation have established a broader and more
stringent set of standards to be met by suppliers of environmental services. As a result, many
local governments are now faced with having to maintain all or some part of their public services
at a higher level of performance. To meet these new standards will require additional
investments in capital, and increases in rates charged to customers for environmental services.
Improvements in environmental services are but one of several demands being made of
local public infrastructure. Studies prepared on public infrastructure needs and the availability
of funds to meet these needs indicate that there will be an excess demand for money to rebuild
and improve upon the existing stock of public infrastructure. Therefore, it is important to
recognize that additional environmental requirements will have to compete with other
infrastructure needs (e.g., highways, bridges), as well as other public services (e.g., police,
education, health and welfare programs) provided at the local level.
Given the increasing demand for public services, this study examines what additional
investments the new environmental legislation will require local governments to undertake, and
the likelihood that they will face difficulties raising the necessary funds through capital markets
and revenues from customers. The economic impacts of individual EPA actions are considered
during the regulatory process in those situations permitted by environmental statutes. The
unique feature of this study is its attempt to estimate the cumulative costs and impacts of
meeting a combined set of EPA requirements, and to determine whether they will place a
significant burden on the fiscal conditions of local governments, and require them to
significantly increase existing charges for improved environmental services.
Please find attached a copy of the draft study and appendices for your review. This
version of the report reflects Agency comments received on an earlier draft. We expect that this
version of the report, with some additional minor modifications, will be the final version. Your
comments will be useful in preparing the final version of the report and discussing the findings
of the reports with the public.
-------
Page C-10 Peer Review Handbook
To assist in your review of the report, we ask that you pay particular attention to the
following questions:
1) Do state and local governments and financial markets consider household costs
(measured as a percent of household income) and selected municipal financial
information (debt service to general revenues or taxable property values) when
evaluating the ability of enterprise systems and municipalities to issue bonds or
obtain loans?
2) Having selected a series of financial indicators, are the criteria used in the
analysis acceptable?
3) Do the results support our conclusions? What additional conclusions can be
reached from the analysis?
4) What modifications would you suggest be made to the recommendation section?
-------
Peer Review Handbook Page C-ll
CHARGE EXAMPLE 7 - Economics - Valuation of Fatal Cancer Risks
Background
The Science Advisory Board-Environmental Economics Advisory Committee (SAB-
EEAC) review of the draft Guidelines for Preparing Economic Analyses (Guidelines) helped the
Agency to identify valid and sound economic procedures to use when conducting benefit-cost
analyses of environmental policies. Both the Guidelines and the SAB-EEAC review (EPA-
SAB-EEAC-99-020, dated September 1999) recognize that economic theory will evolve and
new empirical literature will be forthcoming that may necessitate revisiting the analytic
procedures contained in the Guidelines. There may also be situations where the Guidelines do
not provide sufficient detail to respond to analytic questions whose answers have broad
implications for the conduct of economic analysis at the EPA. Both circumstances have recently
arisen, making it necessary for the Agency to submit for SAB-EEAC review a document that
examines the "benefit transfer" issues that arise when using the value of statistical life (VSL)
literature that is based on accidental risks to estimate the economic benefits of environmental
policies that reduce fatal cancer risks.
The Guidelines provide information and guidance on the valuation of reduced mortality
risks (Chapter 7, pp. 37-43). The Agency Guidelines conclude - and we understand the SAB-
EEAC to have concurred in their review on this subject - that one practical and well-supported
means to value changes in mortality risks is to use the Value of a Statistical Life (VSL)
approach. Further, in response to the SAB-EEAC review, the Guidelines describe a number of
important factors to consider in applying benefit transfer approaches using VSL estimates from
the empirical literature on wage-risk tradeoffs. Recognizing that this is an important benefit
category, the Agency Guidelines stated that the EPA would "continue to conduct annual reviews
of the risk valuation literature" and "reconsider and revise the recommendations in these
guidelines accordingly." Furthermore, the EPA would "seek advice from the Science Advisory
Board as guidance recommendations are revised."
The Agency needs to return to the SAB-EEAC and obtain additional counsel on this
subject. Some economists within the government have suggested some particular approaches to
dealing with the benefit-transfer issues. Since the Guidelines were drafted, a few relevant
articles have been published that examine benefit transfer issues surrounding the use of VSL
estimates when there is a passage of time (or latency period) between the pollution exposure and
harm, or when fatal cancer risks are involved. The importance of these issues was articulated in
a recently proposed regulation to reduce human health risks from radon in drinking water. The
proposed rule estimated the number of reduced fatal cancers resulting from different regulatory
options. The Agency presented information on the economic values for the reductions in fatal
cancer risks, along with other quantified benefits. A brief discussion of some of the benefit
transfer issues involved in this estimation was published in the preamble to the proposed rule for
setting standards for exposure to radon from drinking water sources (Federal Register,
-------
Page C-12 Peer Review Handbook
November 2, 1999 volume 64, Number 211, pages 59245-59378). Quoting from the Federal
Register notice requesting SAB review:
Latency is one of a number of adjustments or factors that are related to an evaluation of
potential benefits associated with this rule, how those benefits are calculated, and when those
economic benefits occur. Other factors which may influence the estimate of economic benefits
associated with avoided cancer fatalities include (1) a possible "cancer premium" (i.e., the
additional value or sum that people may be willing to pay to avoid the experiences of dread, pain
and suffering, and diminished quality of life associated with cancer-related illness and ultimate
fatality); (2) the willingness of people to pay more over time to avoid mortality risk as their
income rises; (3) a possible premium for accepting involuntary risks as opposed to voluntary
assumed risks; (4) the greater risk aversion of the general population compared to the workers in
the wage-risk valuation studies; (5) "altruism" or the willingness of people to pay more to reduce
risk in other sectors of the population; and (6) a consideration of health status and life years
remaining at the time of premature mortality. Use of certain of these factors may significantly
increase the present value estimate. EPA therefore believes that adjustments should be
considered simultaneously. The Agency also believes that there is currently neither a clear
consensus among economists about how to simultaneously analyze each of these adjustments
nor is there adequate empirical data to support definitive quantitative estimates for all potentially
significant adjustment factors. As a result, the primary estimates of economic benefits presented
in the analysis of this rule rely on the unadjusted $5.8 million estimate. However, EPA solicits
comment on whether and how to conduct these potential adjustments to economic benefits
estimates together with any rationale or supporting data commenters wish to offer. Because of
the complexity of these issues, EPA will ask the Science Advisory Board (SAB) to conduct a
review of these benefits transfer issues associated with economic valuation of adjustments in
mortality risks. In its analysis of the final rule, EPA will attempt to develop and present an
analysis and estimate of the latency structure and associated benefits transfer issues outlined
previously consistent with the recommendations of the SAB and subject to resolution of any
technical limitations of the data and models." (page 59326)
In the process of responding to reviews prepared during deliberations on the proposed
radon rule, the Agency found that the Guidelines lack sufficient detail on how to fully evaluate
and characterize the different risk attributes that are central to a complete understanding of the
benefit-cost implications of this rule. For example, time can pass between the point of initial
exposure to a carcinogen, the biological manifestation or onset of cancer in the body, the medical
diagnosis of cancer, and death caused by the cancer. During development of policies affecting
cancer risks, suggestions have been made to discount the VSL estimate (i.e., $5.8 million
recommended in the Guidelines) to account for latencies, or the delay in time between reduced
exposure and when the cancer death would have occurred absent the exposure reduction.
Others argued that a suitable approach for valuing benefits from reduced cancer risks
must consider simultaneously all of the benefit transfer factors related to valuing cancer risks to
ensure a careful and full treatment of benefits. There is evidence in the economics literature
-------
Peer Review Handbook Page C-13
regarding many such factors (e.g., potential premiums ascribed to cancer risk reductions due to a
higher willingness to pay to avoid the dread, pain and suffering, morbidity effects, and other
features of cancer endpoints) that may suggest introducing upward adjustments factors which
offset any potential downward adjustments caused by accounting for cancer latency. In addition,
proponents argue that adjustments for the age of population at risk, income, altruism and other
risk characteristics (e.g., controllability, voluntariness) can all have some potential influence on
the value of a statistical cancer fatality (VSCF) and therefore need to be reflected in the
quantitative benefit assessment.
While developing the primary benefit estimates for reduced fatal cancer risks in the
proposed radon rule, questions arose regarding the implementation of adjustments for some
factors, but not others. For example, would it ever be appropriate to adjust only for latency
periods, and not other factors, in the valuation of reduced cancer deaths? To help answer this
and related questions regarding the valuation of cancer risks, the Agency seeks the SAB-EEAC's
counsel. We further ask that your guidance reflect the typical uncertainties facing EPA
economists, including those surrounding the underlying risk assessments, the prediction (or lack
thereof) of latency periods for cancers, and the risk characteristics associated with the VSL
approach.
Therefore, the Agency proposes to seek review of a "white paper" and list of charge
questions by the SAB-EEAC on the valuation and benefit transfer practices arising in the
calculation of the economic benefits of reduced fatal cancer risks. The Agency seeks SAB-
EEAC review of the treatment and presentation of quantitative and qualitative information for
these types of benefits. Numeric case studies are included in the white paper, to both identify
and present prospective approaches to address these issues. The results of the SAB-EEAC
review of this document, and responses to the specific charge questions, will be considered by
the Agency during future revisions to the Guidelines, consistent with the Agency's commitment
to credible and consistent economic analysis in support of the policy making process.
Charge Questions:
As the Committee considers the charge questions, it is asked to keep in mind the
differing situations relating to differing degrees of data availability or uncertainty in key
parameters.
1) Does the white paper accurately describe the empirical economic literature
relevant to the benefit transfer issues that ensue when using the VSL literature to
estimate the VSCF in a benefit-cost analysis?
2) Does the white paper present the important risk and demographic factors that can
affect benefit transfer approaches that use VSL estimates for VSCF?
-------
Page C-14 Peer Review Handbook
3) Does the white paper accurately describe attempts in the economic literature to
measure VSCF directly?
4) There are two numeric case studies of environmental cancer risks developed for
the white paper. Each presents risk assessment information that forms the basis
for quantifying the number of statistical cancer fatalities that will be reduced as a
consequence of a hypothetical proposed environmental policy. The case studies
are then used to illustrate the outcome of using direct measures of the VSCF and
benefit transfer adjustments to VSL estimates in order to calculate the VSCF.
a) Which of the valuation approaches applied to the case study designated as
ALPHA are valid to use? Does this case study omit any credible
alternative protocols for valuing reductions in fatal cancer risks for
benefit-cost analyses of environmental programs?
b) Which of the valuation approaches applied to the case study designated as
OMEGA are valid to use? Does this case study omit any credible
alternative protocols for valuing reductions in fatal cancer risks for
benefit-cost analyses of environmental programs?
5) Which economic methods illustrated with the case studies, or additional methods
identified by the Committee undercharge questions 4.a and 4.b, serve as credible
protocols for the Agency to use in representing quantitative data, qualitative
information, and sensitivity analyses for the economic value of reduced fatal
cancer risks reported in benefit-cost analyses?
-------
Peer Review Handbook Page C-15
CHARGE EXAMPLE 8 - Hudson River PCBs Site Reassessment RI/FS
Modeling Approach Charge: Peer Review 1
Members of this peer review will be tasked to determine whether the models being used
to support the decision-making process for the Reassessment, and the assumptions therein, are
appropriate. The peer reviewers will base their assessment on the review the Preliminary Model
Calibration Report (PMCR), an updated Technical Scope of Work for the Baseline Modeling
Report (Appendix B of the PMCR) and the responses to selected comments received from
stakeholders during the public comment period on the PMCR.
In October 1996, EPA released the Preliminary Model Calibration Report (PMCR),
which described the models, datasets and assumptions being used as part of the Hudson River
PCB Reassessment RI/FS. The PMCR represents the status of the preliminary PCB modeling
effort as of Fall 1995. Datasets, database corrections and other pertinent information which
became available after October 1995 were not incorporated within the fate and transport
modeling presented in the PMCR. The PMCR was an interim document prepared to describe
work in progress and was not intended to be a conclusive report. In particular the HUDTOX
model presented in the PMCR was not intended to be used as a predictive tool to assess remedial
action scenarios. In addition, while time-varying mechanistic models of bioaccumulation will be
used along with other models to predict fish body burdens, these models are not described in the
PMCR.
The PMCR was not formally peer reviewed at the time of publication, but was
distributed to interested parties who were invited to submit comments and questions. Written
responses were made to all of these comments and questions. In addition, the work plan
contained in Appendix B of the PMCR has been revised to reflect the ongoing work being
conducted as part of the Baseline Modeling effort. Results from this effort will be presented in a
Baseline Modeling Report that will be formally peer reviewed.
The peer reviewers are requested to determine whether the models being used to support
the decision-making process for the Reassessment RI/FS, and the assumptions therein, are
appropriate. The peer reviewers are not being asked whether they would conduct the work in
the same manner, only whether the work being conducted will yield scientifically credible
conclusions.
It is suggested that the reviewer first read the PMCR. The Responses to Comments
provides information on the context of the PMCR within the overall modeling effort and
additional details beyond the PMCR results. The current work plan as revised in June 1998
reflects the ongoing Baseline Modeling effort and revisions to some of the original modeling
tasks proposed in Appendix B of the PMCR. In addition, the USEPA/TAMS Phase 2 database
has been considerably revised. New datasets have been added and some earlier datasets have
been extensively revised.
-------
Page C-16 Peer Review Handbook
The peer reviewers are asked to comment on the following:
1) Is EPA using appropriate models, datasets and assumptions on which to base a
scientifically credible decision?
2) Will the models, with the associated datasets and assumptions, be able to answer
the following principal study questions as stated in the PMCR?
a) When will PCB levels in the fish population recover to levels meeting
human health and ecological risk criteria under No Action?
b) Can remedies other than No Action significantly shorten the time required
to achieve acceptable risk levels?
c) Are there contaminated sediments now buried and effectively sequestered
from the food chain which are likely to become "reactivated" following a
major flood, resulting in an increase in contamination of the fish
population?
3) Specific questions:
a) Are the modeling approaches suitable for developing quantitative
relationships between external forcing functions (e.g., hydraulic flows,
solids and PCB loads, sediment initial conditions, etc.) and PCB
concentrations in the water column, sediments and fish? Are the models
adequate for discriminating between water-related and sediment-related
sources of PCBs?
b) Are the spatial and temporal scales of the modeling approaches adequate
to answer the principal study questions? If not, what levels of spatial and
temporal resolution are required to answer these questions? What
supporting data are required for calibration/ validation of these spatial and
temporal scales?
c) It is contemplated that PCB concentrations in fish will be estimated using
several modeling approaches: an empirical probabilistic model derived
from Hudson River data, a steady state model that takes into account
mechanisms of bioaccumulation body burdens, and a time-varying
mechanistic model (not included in the PMCR). A bi-variate statistical
model may also be used to provide insight into accumulations. This multi-
model approach is being contemplated because of the uncertainties
associated with any individual model. Is this a reasonable approach or
should predictions be made using a single "best" model?
-------
Peer Review Handbook Page C-17
d) Is the level of process resolution in the models adequate to answer the
principal study questions? If not, what processes and what levels of
resolution are required to answer these questions? What supporting data
(such as data to support specifications of a mixed depth layer, solids and
scour dynamics, groundwater inflow, etc.) are required for these processes
and levels of resolution?
e) The results of the modeling effort will be used, in part, to support human
and ecological risk assessments. In your judgment, will the models
provide estimates adequate for this purpose?
4) Are there any changes to the work effort outlined in the revised work plan that
would significantly improve the outcome?
5) In terms of evaluating the overall and specific effects and behavior of PCBs in the
Hudson River, are there any serious flaws in the modeling approach (theory,
structure, physical parameters, etc.) that would limit or invalidate any conclusions
or further work based upon the results of these models?
Recommendations
Based on your reading and analysis of the information provided, please identify and
submit an explanation of your overall recommendation for the modeling effort for the Hudson
River PCB Reassessment RI/FS:
1) Acceptable as is
2) Acceptable with minor revision (as indicated)
3) Acceptable with major revision (as outlined)
4) Not acceptable (under any circumstance)
-------
Page C-18 Peer Review Handbook
CHARGE EXAMPLE 9 - IRIS Pilot Program - Instructions to Peer Reviewers for
Reviewing IRIS Summaries and Supporting Documentation
The U.S. EPA is conducting a peer review of the scientific basis supporting the health
hazard and dose response assessments for the subject chemical that will appear on the Agency's
online database, the Integrated Risk Information System (IRIS). Materials to be reviewed
include the summary information that will appear on IRIS (the inhalation reference concentration
[RfC], oral reference dose [RfD], and cancer assessment) and the supporting document, the
Toxicological Review, which will also be made available to the public.
A listing of Agency Guidelines and Methodologies that were used in the development of
these hazard and dose-response assessments included the following: The Risk Assessment
Guidelines (1986), the (new) Proposed Guidelines for Carcinogen Risk Assessment (1996),
Guidelines for Developmental Toxicity Risk Assessment, (proposed) Interim Policy for Particle
Size and Limit Concentration Issues in Inhalation Toxicity, (proposed) Guidelines for
Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations
and Application of Inhalation Dosimetry, Recommendations for and Documentation of
Biological Values for Use in Risk Assessment and Use of the Benchmark Dose Approach in
Health Risk Assessment. Copies of these documents (and/or their relevant Sections) will be
made to the reviewer upon request.
Peer review is meant to ensure that science is used credibly and appropriately in
derivation of these dose-response assessments. You have been chosen as an expert on the
chemical under consideration, on a scientific discipline related to at least one of the assessments,
or in the field of risk assessment. At least three peer reviewers per chemical are being chosen to
review the scientific basis of these draft dose-response assessments before they are forwarded on
to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and
dose-response assessments will then appear on IRIS and become available as Agency consensus
health effect information.
The primary function of the peer reviewer should be to judge whether the choice, use,
and interpretation of data employed in the derivation of the assessments is appropriate and
scientifically sound. This review is not of the recommended Agency risk assessment guidelines
or methodologies used to derive cancer or RfD/C assessments as these have been reviewed by
external scientific peers, the public, and EPA Science Advisory Boards. The reviewer's
comments on the application of these guidelines/methodologies within the individual
assessments is, however, welcomed and encouraged. For example, the reviewer may ascertain
whether or not there is data sufficient to support use of other than default assumptions for areas
such as sensitive subpopulations or linear cancer extrapolation. The reviewer may also have
opinions on other areas of uncertainty such as subchronic to chronic duration (when only a
subchronic study is available) or an incomplete data base but should focus on the specific area of
uncertainty rather than on the magnitude of the overall estimate.
-------
Peer Review Handbook Page C-19
Below are two groups of questions regarding this review. The first is a set of general
questions that are meant to guide you through your review. It is not imperative that you
specifically answer each question of this group. The second group of questions, however, are
specific for the chemical assessments and deal with areas of scientific controversy or uncertainty
in which the Agency may have to make a scientific judgment. Your input to this set of questions
is considered vital to the review process.
Questions for IRIS Peer Reviewers -General
1) Are you aware of any other data/studies that are relevant (i.e., useful for the
hazard identification or dose-response assessment) for the assessment of the
adverse health effects, both cancer and noncancer, of this chemical? Please
explain fully.
2) For the RfD and RfC, has the most appropriate critical effect been chosen (i.e.,
that adverse effect appearing first in a dose-response continuum)? For the cancer
assessment, are the tumors observed biologically significant? relevant to human
health? Points relevant to this determination include whether or not the choice
follows from the dose-response assessment, whether the effect is considered
adverse, and if the effect (including tumors observed in the cancer assessment)
and the species in which it is observed is a valid model for humans.
3) Have the noncancer and cancer assessments been based on the most appropriate
studies? These studies should present the critical effect/cancer (tumors or
appropriate precursor) in the clearest dose-response relationship. If not, what
other study (or studies) should be chosen and why?
4) Studies included in the RfD and RfC under the heading "Supporting/Additional
studies" are meant to lend scientific justification for the designation of critical
effect by including any relevant pathogenesis in humans, any applicable
mechanistic information, any evidence corroborative of the critical effect, or to
establish the comprehensiveness of the data base with respect to various
endpoints (such as reproductive/developmental toxicity studies). Should other
studies be included under the "Supporting/Additional" category? Should some
studies be removed?
5) For the noncancer assessments, are there other data that should be considered in
developing the uncertainty factors or the modifying factor? Do you consider that
the data support use of different (default) values than those proposed?
6) Do the Confidence statements and weight-of-evidence statements present a clear
rationale and accurately reflect the utility of the studies chosen, the relevancy of
the effects (cancer and noncancer) to humans, and the comprehensiveness of the
-------
Page C-20 Peer Review Handbook
data base? Do these statements make sufficiently apparent all the underlying
assumptions and limitations of these assessments? If not, what needs to be
added?
Questions for IRIS Peer Reviewers - Chemical Specific
[example: cumene]
1) Based on the information noted in the Principal study currently designated
(Cushman et al., 1995) is the discounting of the renal effects in males justified?
Is sufficient rationale given to let stand the organ weight changes in female rats as
a critical effect?
2) Is the information in the Toxicological Review sufficient to consider cumene as
having a low potential for causing reproductive effects? Please explain fully.
RECOMMENDATIONS
Based on your reading and analysis of the information provided, please identify your
overall recommendation for the IRIS materials you have reviewed as
1) Acceptable as is
2) Acceptable with minor revision (as indicated)
3) Acceptable with major revision (as outlined)
4) Not acceptable
-------
Peer Review Handbook
Page C-21
CHARGE EXAMPLE 10 - Charge to Reviewers for the WTI Draft Final Risk Assessment
The draft final WTI risk assessment is divided into several volumes covering the
scientific disciplines of toxicology, environmental fate and transport, combustion engineering,
atmospheric modeling, exposure assessment, ecological risk assessment, and accident analysis.
As a reviewer of the WTI draft final risk assessment, you should use your best technical
knowledge and professional judgment to comment on the technical accuracy, completeness and
scientific soundness of the assessment. Each reviewer is asked to focus on several specific
issues in his or her area of expertise with comments on other areas invited but optional. Your
comments will be considered in finalizing the risk assessment.
For the peer review workshop reviewers will be organized into 5 work groups:
Combustion Engineering, Air Dispersion and Deposition Modeling and Accident Analysis,
Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be
familiar with the Executive Summary (Volume I) and the Facility Background (Volume II)
Sections of the draft risk assessment In addition, each work group should focus on specific
Volumes as specified below:
Workgroup
Combustion Engineering
Air Dispersion and
Deposition Modeling and
Accident Analysis
Toxicology
Exposure Assessment
Ecological Risk Assessment
Risk Assessment Volumes
Volume in - Facility
Emissions
Volume IV - Atmospheric
Dispersion and Deposition
Modeling
Volume V - Human Health
Risk Assessment
Volume V - Human Health
Risk Assessment
Volume VI - Screening
Volume VII - Accident
Analysis
Volume VII - Accident
Analysis
Volume VII - Accident
Analysis
Ecological Risk Assessment
While reviewing these Sections of the document, please address the following general
issues.
-------
Page C-22 Peer Review Handbook
1) Comment on the organization of the risk assessment document. Does the layout
follow a logical format? Is the presentation of information in the document clear,
concise and easy to follow?
2) Does the executive summary accurately reflect the data and methodologies used
and the conclusions derived in the risk assessment?
3) Were the major recommendations of the 1993 peer review workshop for the risk
assessment plan addressed?
4) As with any risk assessment, there are always additional data and method
development efforts that could be undertaken to reduce the level of uncertainty.
However, are there any major data or methodological gaps that would preclude
the use of this risk assessment for decision making? If so, how should they be
addressed?
5) What long-term research would you recommend that could improve risk
assessments of this type in the future?
In addition, the following workgroup specific issues should be addressed.
Emissions Characterization
Emissions characterization includes identification of substances of concern and the
development of emission rates for these contaminants. Emission rates were developed through a
combination of site specific stack test data and models. Please comment on the following issues
with respect to this aspect of the draft risk assessment.
1) To characterize the nature of the emissions, waste stream profiles were developed
and entered into a database. Several refinements and adjustments (e.g., the
Subtraction Correction Factor for chlorinated compounds) were applied to the
profiles before substances of concern were identified. Please comment on
whether or not these adjustments are appropriate. What is the anticipated effect
on the risk assessment?
2) Comment on the selected chemicals of concern. Have important chemicals been
missed due to the selection technique?
3) Comment on the approaches used to estimate stack emission rates (e.g., use of the
95% UCL of the arithmetic mean or the maximum detected value, whichever is
smaller, for high end emission rates). Are the approaches appropriate? Are their
effects on the risk assessment adequately characterized? Comment on the
-------
Peer Review Handbook Page C-23
adjustment made to PCDD/PCDF emission rates to account for brominated
dioxin-like compounds. Also, comment on the approach to characterizing
emission rates from fugitive sources (e.g., use of the TANKS 2 model for the
Carbon Adsorption Bed).
4) Comment on the identified sources of fugitive emissions. Was the approach used
to select these sources appropriate? Have important sources been missed? Have
emissions from process upsets been given appropriate consideration?
5) There have been a number of controlled burns at the WTI facility. Please
comment on the adequacy of these data in estimating potential exposure. Please
comment on the assumptions made from the tests in regard to composition of
wastes received at WTI and emissions when the plant operates in the future.
6) Comment on the use of emission factors from coal burning to estimate the
emission rate of fly ash from WTI. Are the factors used to adjust the coal
emission rate appropriate? Are the uncertainties introduced from this approach
adequately characterized?
7) Overall, is the identification of the key assumptions used in characterizing the
nature and magnitude of emissions thorough? Are the magnitude and direction of
effect of these assumptions on the overall risk assessment accurately
characterized? Is the uncertainty and variability inherent in this analysis
adequately discussed? Does the sensitivity analysis cover the major parameters
expected to have an effect on the risk assessment?
Dispersion and Deposition Modeling
To develop this risk assessment, computer models have been used with site specific data
on emission rates and meteorological conditions to simulate the air concentrations and
deposition rates for contaminants potentially emitted from the WTI facility. The models used
include the Industrial Source Complex - Complex Terrain Deposition (ISC-COMPDEP), the
CALPUFF, and the INPUFF models. In your review, please address the following issues.
1) Since the 1993 peer review of the risk assessment plan, a number of efforts have
been completed to reduce the uncertainty associated with the air dispersion and
deposition modeling. These efforts include the collection of site-specific data for
emission rates and meteorological conditions. Also, a wind tunnel study was
conducted to evaluate the effects of the complex terrain surrounding the WTI
facility. Does the risk assessment document adequately summarize these
activities? Is the link between these data collection efforts, the air dispersion
models, and the risk assessment clearly established?
-------
Page C-24 Peer Review Handbook
2) The results of 12 sets of sensitivity tests indicate that geophysical variables (e.g.,
terrain) are more likely to affect dispersion and deposition than emission
variables (e.g., stack temperature). Were these sensitivity analyses adequate?
Comment on the conclusions reached. To further examine the effect of
geophysical variables, wind tunnel testing was conducted to model the terrain
induced flow effects expected near WTI. It was concluded that changes in peak
concentrations attributed to these effects are relatively minor and that the ISC-
COMPDEP model is sufficiently conservative. Comment on this conclusion.
Have these analyses helped to characterize and/or reduce the uncertainty in the air
dispersion modeling associated with the complex terrain surrounding WTI.
3) The ISC-COMPDEP model does not allow for non-steady state conditions such
as calm winds and strong temperature inversions. Therefore, CALPUFF was
used to estimate air dispersion and deposition under these conditions. However,
CALPUFF gave similar peak, 24 hour, and annual average concentrations as ISC-
COMPDEP. Comment on the adequacy of this analysis. Comment on the
conclusions reached. Has this analysis helped to characterize and/or reduce the
uncertainty in the air dispersion modeling associated with non-steady state
meteorological conditions?
4) Atmospheric dispersion modeling was used to estimate air concentrations of
hazardous chemicals for the accident analysis. The SLAB model was used for
vapor releases from spills and the mixing of incompatible wastes. ISC-
COMPDEP was used for releases associated with fires. Comment on the
selection of the models and inputs. Are they appropriate selections?
5) Overall, have adequate sensitivity tests been conducted to demonstrate the
magnitude of variation in concentrations and deposition estimates with model
inputs? Please explain fully.
Human Health Risks
Human Health Risk Assessment includes hazard identification, dose-response evaluation,
exposure assessment, and risk characterization. To develop the risk assessment, potentially
exposed populations have been identified and the magnitude, frequency, and duration of their
exposure quantified. This information was then integrated with the hazard identification and
dose response evaluation for the risk characterization. For this risk assessment, both
carcinogenic and non-carcinogenic health effects have been evaluated. In your review, please
comment on the following issues.
-------
Peer Review Handbook Page C-25
Exposure
1) EPA's Exposure Assessment Guidelines identify certain exposure descriptors that
should be used to characterize exposure estimates. The Guidelines define high
end exposure estimates as those representing individuals above the 90th
percentile on the exposure distribution but not higher than the individual in the
population who has the highest exposure. Bounding exposure estimates are those
that are higher than the exposure incurred by the person in the population with the
highest exposure. Central tendency exposure estimates are defined as the best
representation of the center of the exposure distribution (e.g., arithmetic mean for
normal distributions). Comment on whether or not the WTI exposure assessment
properly characterizes each of the exposure estimates in terms of these
descriptors.
2) The factors that go into estimating a central tendency or high end exposure, once
the population has been defined, include the environmental media concentration,
the intake rate, and the duration and/or frequency of exposure. Comment on
whether or not the WTI exposure assessment does an adequate job of describing
the logical procedure of combining these factors to develop central tendency, high
end, and/or bounding estimates of exposure for each of the exposed
subpopulations.
3) An important factor in an exposure assessment is identifying all of the important
exposure sources. Please comment on the adequacy of the WTI assessment in
identifying the important sources and pathways of exposure.
4) Have the key assumptions for estimation of chemical concentration and for
estimation of exposure been identified? Are the magnitude and direction of effect
correct for the assumptions that have been identified?
5) Supposedly, conservative assumptions have been applied in this assessment to
account for uncertainty. Are the conservative assumptions appropriately factored
into the ultimate characterization of what descriptor best applies to each exposure
estimate? Please comment on whether the uncertainties were confronted in an
adequate manner. If they were not, please state what should be done differently.
Hazard Identification/Dose Response and Risk Characterization
1) To select surrogate compounds for quantitative risk assessment, a two step
process was used in which chemicals were ranked on the basis of emission rate,
toxicity (both cancer and non-cancer), and bioaccumulation potential. Please
comment on this selection process. Are the ranking factors appropriate? Could
-------
Page C-26 Peer Review Handbook
important compounds have been omitted from the analysis based on the ranking
procedure?
2) For the majority of the chemicals of concern, traditional approaches to dose
response evaluation were employed (e.g., use of a slope factor for cancer and use
of a RfD/RfC for non-cancer). However for certain chemicals or groups of
compounds a different methodology was used. Specifically, dioxins, furans,
PAHs, lead, mercury, nickel, chromium, acid gases, and paniculate matter were
given special consideration. Please comment on the methodology used for these
compounds. Was it appropriate? Have the uncertainties associated with the
methodology been adequately characterized? Comment on the assumptions used
due to a lack of chemical specific data.
3) Please comment on the selection of the overall population and the various
subpopulations at risk. Were site specific data, such as the informal home
gardening survey, properly utilized to identify these subpopulations?
4) It is stated in the risk assessment that average risk estimates are based on average
emission rates, average air dispersion/deposition within a subarea, and typical
exposure factors. Further, maximum risks are based on average emission rates,
typical exposure factors, and the maximum air concentration within a subarea.
Please comment on this use of the terms average and maximum risks. Are these
descriptive terms appropriate given the parameters used to derive each? Please
explain fully.
5) Comment on whether or not the non-cancer risks of chemicals of concern have
been adequately addressed by the risk assessment? For example, has an adequate
discussion of endocrine disrupters been provided which either characterizes their
risks or clearly explains why their risks cannot be characterized? Further, have
non-cancer chronic toxicities of dioxins and furans been adequately addressed in
the risk assessment?
6) Please comment on whether or not the uncertainties associated with the additivity
and/or synergy of risks from pollutants emitted together from the WTI facility are
adequately discussed in the risk assessment.
7) Have the key assumptions for estimation of dose and risk been identified? Are
the magnitude and direction of effect correct for the assumptions that have been
identified? Please comment on whether the uncertainties were confronted in an
adequate manner. If they were not, please state what should be done differently.
-------
Peer Review Handbook Page C-27
8) Please comment on the overall adequacy of the risk characterization. Does the
risk characterization include a statement of confidence in the risk assessment
including a discussion of the major uncertainties. Are the hazard identification,
dose-response assessment, and exposure assessment clearly presented? Have
sufficient risk descriptors which include important subgroups been presented and
discussed?
Screening Ecological Risk Assessment
As with the human health risk assessment, the ecological risk assessment pulls together
elements of exposure analysis and dose-response evaluations to develop a risk characterization.
For the Screening level Ecological Risk Assessment (SERA), Ecological Chemicals of Concern
(ECOC) and indicator species have been identified to provide conservative estimates of risk.
Please address the following issues in your review.
1) Are there any components of the SERA which you feel undermine the scientific
validity of the assessment? If so, what are they and can you provide suggestions
to strengthen the identified components?
2) Is the organization of the document clear and does it present the material in a
clear and concise manner consistent with the Framework for Ecological Risk
Assessment (EPA, 1992)? Please explain fully.
3) Uncertainties are discussed in numerous sections of the SERA and compose
Section VIII of the SERA. In each case, do these discussions cover all relevant
and important aspects of the uncertainties which you think should be addressed in
the SERA?
4) In your opinion, what is the weakest and what is the strongest aspect of the
SERA? Can you make any suggestions on how the weakest parts can be
strengthened by the Agency?
5) In Section II, are the stressors, ecological effects, and both the assessment and
measurement endpoints adequately characterized? Are the five emission
scenarios adequate to characterize the exposures for the WTI facility? Are there
other emission scenarios which you think should be included in the SERA?
6) In Section III, is the site characterization adequate to support the SERA? Why or
why not?
7) In Section IV, is the tiered process used to identify the ecological chemicals of
concern (ECOC) from the initial list of potential chemicals considered
-------
Page C-28 Peer Review Handbook
scientifically defensible? Does application of this tiered approach support the
statement made in the SERA "by focusing on the potential risk from the selected
ECOCs, the SERA provides a thorough screening-level evaluation for the WTI
facility?"
8) In Sections V and VI, are the exposure and ecological effects adequately
characterized? Are the most appropriate estimation techniques available used?
Are the assumptions clearly stated? Please explain fully.
9) In Section VIII, are there any major elements missing from the risk
characterization which you think need to be included or which would strengthen
the risk characterization? Does the risk characterization support the summary and
conclusions presented in Section IX?
10) In Section IX, given the assumptions made and the processes used to select and
evaluate chemicals, receptors, and exposure pathways, do you think the SERA
adequately met its objective of not inadvertently underestimating risk?
Accident Analysis
The Accident Analysis for the WTI incinerator involves evaluating the probability of an
emergency incident occurring which results in the release of hazardous waste. The
consequences of this release are also evaluated using exposure and human health effects
information. Unlike the human health risk assessment which has a primary goal of quantifying
risks, the accident analysis typically provides information that can be used to reduce the
likelihood, extent and impact of possible accidents. Please comment on the following issues in
your review of this aspect of the risk assessment.
1) The WTI accident assessment selected five scenarios for quantitative evaluation
that were considered to be of primary concern. The scenarios are an on-site spill,
an on-site fire, an on-site mixing of incompatible waste, an off-site spill, and an
off-site spill and fire. Please comment on the selection of these scenarios. Were
any significant scenarios missed?
2) Specific chemicals were selected to evaluate each scenario. Please comment on
the selections. Would other chemicals have been more appropriate?
3) Chemical specific release rates are calculated for each scenario. Please comment
on the procedures used to estimate the release rates. Was an appropriate approach
used?
-------
Peer Review Handbook Page C-29
4) Atmospheric dispersion modeling was used to estimate air concentrations of
hazardous chemicals. Specifically, the SLAB model was used for vapor releases
from spills and the mixing of incompatible wastes. ISC-COMPDEP was used for
releases associated with fires. Comment on the selection of the models and in-
puts. Are they appropriate selections? Should other models or inputs been used?
5) Please comment on the assessment's conclusions on the severity of consequences
and probability of occurrence. Has the report correctly categorized the severity of
the consequences of the different accident scenarios? Has the assessment
adequately justified the reported probability of occurrence of each of the accident
events?
6) Key assumptions were made in the identification of accident scenarios and the
description of the conservative and typical events. Included were a description of
the magnitude of the effect of the assumptions and direction of the effect. Please
comment on the assumptions. Are they justified? Are the descriptions of the
magnitude and directions of the effects correct? Has the accident assessment
adequately confronted the uncertainties involved in doing this type of analysis? If
not, what else should be done?
7) Comment on the appropriateness of using IDLH values for characterizing the
severity of consequences in the accident analysis. Comment on the
appropriateness of using 10 X LOC for chemicals for which IDLH values have
not been established.
8) In the accident analysis, IDLH (or 10 X LOC) values were used to determine the
downwind distances over which adverse human health effects might occur. To
evaluate the uncertainty introduced by using the IDLH, a sensitivity analysis was
conducted where these distances were recalculated using the LOC (a more
stringent health criteria). Other sources of uncertainty that are identified in the
accident analysis include concentration averaging times, chemical concentrations,
emission rates, and meteorological conditions. For most of these parameters it is
stated that conservative assumptions were used to avoid underestimating risks.
Have the uncertainties inherent in the accident analysis been adequately
characterized? For those parameters where sensitivity analyses were not
conducted, is the conclusion that conservative assumptions have avoided
underestimation valid?
-------
-------
Peer Review Handbook Page D-l
APPENDIX D - GUIDANCE ON REQUESTING A
REVIEW BY THE
US EPA SCIENCE ADVISORY BOARD (SAB)
-------
Page D-2 Peer Review Handbook
Guidance on Requesting a Review
By the US EPA Science Advisory Board (SAB)
Summary
The US EPA Science Advisory Board (SAB or the "Board") annually solicits proposals
for review projects every spring for the following fiscal year. This appendix provides guidance
to Programs and Regions to help them submit requests for SAB reviews. Requests should be
submitted to the SAB in both hard copy and electronic versions usually by mid-June for the
following fiscal year. The requests maybe part of the annual submissions that respond to the
peer review activities of the Agency, or they may be submitted directly to the Board. Although
providing requests at one defined time in the spring helps with SAB planning, we recognize that
proj ects also come to light during other parts of the year. Please contact the SAB staff for details
on making submissions during the remainder of the year (see end of this document for contacts).
Background
A key priority for the Administrator is to base Agency actions on sound scientific data,
analyses, and interpretations. The Administrator issued the Agency's Peer Review Policy to
increase the quality of the technical foundations upon which EPA's regulatory structures are
built. The SAB is a key scientific peer review mechanism available to Programs and Regions in
implementing the Peer Review Policy. However, because the Board has finite resources it
cannot conduct all reviews. This document is designed to help Programs and Regions determine
which projects to submit to the SAB. Note particularly that the SAB focuses on the technical
underpinnings of Agency positions; i.e., risk assessment issues, in contrast to risk management
issues.
The topics that are best suited for the Board's agenda are those that satisfy several of the
following criteria:
1) Integrate science into Agency actions in new ways.
2) Influence long-term technological developments.
3) Impact overall environmental protection.
4) Address novel scientific problems or principles.
5) Address problems that transcend federal-agency or other organizational
boundaries.
6) Strengthen the Agency's basic capabilities.
7) Serve Congressional or other leadership interests.
8) Deal with controversial issues.
In suggesting issues for SAB involvement, Programs and Regions should note the
breadth of SAB activities:
-------
Peer Review Handbook Page D-3
Historically, most of the outputs of the Board are in the form of "full" reports.
They present the findings of peer reviews of Agency document(s) and contain
considerable detail about the findings and recommendations of the Board. They also
address the specific questions posed by the Charge to the Board. "Letter" reports fulfill
the same function as reports, but are simply shorter in length. Due to the need to be more
responsive with advice, the Board has recently begun to produce more short letter reports
than full reports, as they can be produced and finalized in less time.
The SAB has also introduced the "Consultation" as a means of conferring—in
public session—with the Agency on a technical matter before the Agency has begun
substantive work on that issue. The goal is to leaven EPA's thinking on an issue by
brainstorming a variety of approaches to the problem very early in the development
process. There is no attempt or intent to express an SAB consensus or to generate an
SAB report. The Board, via a brief letter simply notifies the Administrator that a
Consultation has taken place.
More recently, the Board introduced a new vehicle for communicating with its
clients — the "Advisory" — which provides, via a formal SAB consensus report, critical
input on technical issues that arise during the Agency's issue development process. The
Advisory generally involves a review of a multi-year Agency project. The intent is to
provide some mid-course assessment to see if the Agency is heading in a scientifically
credible direction. In order to maintain an objective, arms-length relation with the
Agency and its projects, the SAB review of the final product at some point in the future
will include experts who did not participate in producing the Advisory.
The Agenda Setting Process
Each Assistant Administrator and Regional Administrator is normally asked to submit a
list of candidate topics for SAB action/review. A "project sheet" (see attached example) is used
to define each topic that is nominated for SAB review. The project sheet is prepared by the
requesting office and contains the following information:
1) Project title/subject (Descriptive short title of project)
2) Requesting Organization/Office (Primary office requesting review, AA/RA level)
3) Requesting Official (Name and position of senior official requesting review,
usually office or division level - this is the person who may receive a summary
briefing from the Chair following the review)
4) Program Contact (Name/phone number/mail code - this is the principal contact
for SAB Staff to interact with during development of the SAB review)
-------
Page D-4 Peer Review Handbook
5) Background (brief history of the project and why it is important)
6) Tentative Charge (what the SAB is being asked to comment on - usually a set of
questions)
7) Tentative Schedule and Committee (when the review is expected to be conducted,
e.g., Winter 2001; and which SAB committee is appropriate for the review — final
choice as to the review committee is at the discretion of the Board)
8) Budget Estimate (Rough estimates of Agency funding for the subject over the
past 5 years (if applicable) and for the next 2 years (if applicable). The SAB
Executive Committee has asked for this information to help it better appreciate
the level of Agency involvement in and commitment to the issue)
9) Preparer (name, phone, office of preparer of Project Sheet and the date prepared)
The proposed topics will be examined and discussed in a number of forums:
1) The individual SAB Committees - Throughout the late Spring and Summer, the
SAB Committees will be examining options for the following fiscal year,
including all suggestions made by the Agency.
2) The Science Policy Council-Steering Committee (SPC-SO - The SPC-SC usually
meets in early summer to examine the proposals for each fiscal year. The goal is
to provide cross-office critique/integration of the proposals. The SPC-SC will be
used as a forum for continuing discussion throughout the process and throughout
the year as new topics emerge.
3) The Deputy's perspective - In the summer, the Deputy Administrator will review
the requests and provide insights on priorities.
4) The SAB Executive Committee - During its summer meeting, the SAB's EC will
examine the nominated topics, adding its own perspective on an appropriate
agenda, using its selection criteria.
5) The Administrator - In September, the list of proposed topics will be delivered to
the Administrator for information and added insights.
The completed project sheets should be submitted electronically to the SAB Deputy Staff
Director (fowle.jack@epa.gov) and in signed hard copy (mail code 1400A).
-------
Peer Review Handbook Page D-5
For general information, or to contact a specific SAB Staff member, please call the SAB
main phone line: (202) 564-4533. Please ask to be connected to the Staff person handling the
major review area you wish to inquire about (e.g., human health and exposure, drinking
water/water quality, ecology, engineering, modeling, air quality, economics, research, and
radiation).
-------
Page D-6 Peer Review Handbook
US EPA Science Advisory Board
Proposed Project
Project title/subject: Proposed Amendments to the Risk Assessment Guidelines for
Carcinogens
Requesting Organization/Office: Office of Research and Development (ORD)
Requesting Official: Name, Title, Office/Organization
Program Contact: Name, Title, Office/Organization, 202-260-xxxx
Background: EPA's Health Risk Assessment Guidelines provide generic science and science
policy guidance on risk assessment issues for use in all Agency offices. EPA has currently
issued or proposed nine guidelines (or amendments) in this series, all of which have been
submitted to the Science Advisory Board for review.
The current guidelines for carcinogen risk assessment were reviewed by the SAB and issued as
final guidance in 1986. In 1988, the Forum initiated a public process for considering
amendments to these guidelines. A Risk Assessment Technical Panel considered submissions
from the public as well as information developed by experts at two public workshops in revising
these guidelines.
Tentative Charge: Review the amended and expanded guidance, with special emphasis on (a)
weight-of-evidence issues, (b) a new classification system, (c) dose response modeling, and (d)
the use of pharmacokinetic and metabolic data. A more detailed charge will be negotiated with
SAB at a later date.
Tentative Schedule and Committee: Winter, 2001, Environmental Health Committee
Budget: FY 1997 - $xxx and yy FTE
FY 1998 - $xxx and yy FTE
FY 1999 - $xxx and yy FTE
FY 2000 - $xxx and yy FTE
FY 2001 - estimated costs of $xxx and yy FTE each year
Preparer: Name, Title, Office/Organization, 202-260-xxxx
Date: June 1,2000
-------
Peer Review Handbook Page E-l
APPENDIX E - EXAMPLE STATEMENTS OF
WORK FOR CONTRACTS
Users of this Peer Review Handbook need to be aware that the examples
contained in this Appendix are generalized statements of work prepared (and in
some cases modified) to emphasize certain important features (e.g., attention to
conflicts of interest, responsibilities of contractors or contracting officers,
development and use of the charge to peer reviewers). Please be sure that proper
and currently approved contract language is used in any EPA contract document
at the time of award (for example, see EPAAR 1552.212-71, alternate I).
-------
Page E-2 Peer Review Handbook
STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical Review
Contractor for Panel Review of Assistance Agreement or Fellowship Applications
1) Purpose
The purpose of this contract is to purchase peer review services of a contractor with
expertise in Exploratory Research: Environmental Chemistry. The services are for peer
reviewing applications received in response to the Office of Research and Development's
(ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the
evaluation sheets shall be prepared prior to the reviewer's participation in a 3-day panel
discussion to be held in Washington. D.C. on May 5 -7.1997.
2) Statement of Work
ORD's National Center for Environmental Research and Quality Assurance (NCERQA)
is responsible for overseeing the recently expanded research grants and fellowships programs.
Each year NCERQA (alone or in conjunction with other organizations) solicits applications in
each of these programs. The applications to be reviewed under this contract were submitted in
response to the solicitation for the 1997 Science to Achieve Results program. As part of the
selection process, NCERQA must conduct a peer review that is designed to evaluate the
scientific quality of each application; this is accomplished through the ad hoc use of technical
experts.
The peer review services required by this contract necessitate the independent review of a
maximum of 10 applications and the preparation of a typed evaluation summary and an overall
rating for each of these applications. Each evaluation summary shall support and be consistent
with the overall rating that is assigned; it also shall be completed prior to the contractor's
participation in the panel discussions. After the panel discussions for the applications assigned
to the contractor, the contractor shall submit all completed evaluation summaries to the
designated Science Review Administrator (SRA).
The contractor also shall serve as the panel's rapporteur for approximately 6 - 8 of the
applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed
evaluation summary (on-site typing support will be provided by NCERQA) that reflects the
panel's discussion of the respective application as well as the panel's overall rating (the criteria
for the panel's overall rating are the same as those for each peer reviewer's overall rating). As
rapporteur, the contractor shall submit these panel evaluation summaries to the designated SRA
prior to leaving the panel meeting.
-------
Peer Review Handbook Page E-3
Before the contractor shall be allowed to participate in the review process, the contractor
shall have disclosed any actual or potential conflicts of interest and shall have signed and
submitted to NCERQA a Conflict of Interest/Confidentiality Form. The contractor is directed to
assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular
reviewer. Upon receipt of an approved Purchase Order (PO), NCERQA will send the following
items to the contractor:
a) A copy of the Purchase Order or the Purchase Order number
b) The applications assigned to the contractor
c) For grants, a set of abstracts for all the applications being reviewed by the panel
d) For persons reviewing grant applications, a copy of the pertinent section(s) of the
solicitation package to provide background information; for persons reviewing
fellowship applications, information on how to access the solicitation package on
the Internet
e) A sample evaluation form to help the contractor prepare an acceptable evaluation
form for each assigned application
f) A blank evaluation form for each assigned application and the criteria for
completing the form and determining the overall rating
g) A blank and sample invoice as well as instructions for completing and submitting
the invoice to EPA
h) Information on the points of contact for additional information (e. g., NCERQA's
SRA)
i) Logistics information on the location and time of the panel discussions
NCERQA will transmit the above items under a cover letter. In this cover letter,
NCERQA will provide additional details about each item, including (as needed) more specific
instructions for the set of applications assigned to the reviewer.
Each contractor shall be responsible for making his/her own travel reservations for hotel
and transportation.
3) Reviewer Tasks
a) Review the assigned applications using the guidance provided with NCERQA's
evaluation form.
b) Submit completed evaluation forms to the SRA designated in the cover letter
immediately following the panel discussions for the applications assigned to the contractor.
THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY
FOR EACH APPLICATION MUST SUPPORT AND BE CONSISTENT WITH THE
OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONS
-------
Page E-4 Peer Review Handbook
WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE
CONTRACTOR TO REDO THE FORM.
c) For those applications for which the contractor is serving as the panel's
rapporteur, submit a completed panel evaluation summary to the SRA designated in the cover
letter prior to leaving the panel meeting. THE PANEL'S EVALUATION SUMMARY
MUST BE TYPED (ON-SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA)
AND BE CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS
WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE
CONTRACTOR TO REDO THE FORM.
d) Make own airline and hotel accommodations for participation in the panel review
meeting. Round-trip air fare must be a commercial REFUNDABLE ticket.
-------
Peer Review Handbook Page E-5
STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization Tool Report
Work Assignment No.:
Title: Peer Review of Prioritization Tool Report
Work Assignment Manager (WAM):
Name: John Q. Government Employee
Address: Office of Solid Waste
Phone No.: (202) 260-XXXX
Background:
The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the
process of implementing the Waste Minimization National Plan, announced by the Agency on
November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction
over waste management, in keeping with the policy stated in the 1984 amendments to the
Resource, Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act
(PPA). The Plan outlines major goals, objectives, and action items to achieve national
reductions in the generation of hazardous wastes.
One of the objectives of the Plan is to: "develop a framework for setting national
priorities; develop and distribute a flexible screening tool for identifying priorities at individual
facilities; [and] identify constituents of concern." This objective is a key building block in
implementing subsequent objectives of the Plan.
In September 1995, WMB formed the Waste Minimization Prioritization Team, which
includes representatives from EPA regions and states, to implement this objective. The Team
has worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools
that are currently available. The Team plans to summarize this work, along with its
recommendations, in a report (referred to herein as the Prioritization Tool report) that would be
available in draft form in July 1996.
WMB and the Team wish to obtain independent peer review of the Prioritization Tool
report prior to briefing EPA management. The report is being prepared with the support of ICF,
Inc.; therefore, for the peer review to be considered independent, it must be performed by
another contractor.
-------
Page E-6 Peer Review Handbook
Purpose and Scope of Work:
The purpose of this work assignment is to provide support to WMB and the Team in
finalizing the Prioritization Tool report by conducting an independent peer review of the report.
Work Statement:
Task 1 -Management work plan and budget
Within 15 days of CO approval of this work assignment, the contractor shall deliver a
management work plan including aproposed level of effort, schedule, and budget for all tasks.
Task 2 -Provide independent peer review of Prioritization Tool report
The contractor shall provide support to WMB and the Team in preparing the
Prioritization Tool report by performing an independent peer review of the report. The
contractor shall establish a panel of peer reviewers including three senior-level persons who
collectively have extensive expertise in particular areas to be identified by the WAM upon
approval of the work assignment.
Within three weeks of work assignment approval, receipt from the WAM of the
necessary qualifications of peer reviewers (in a TD), and receipt from the WAM of the peer
review "charge" (in a TD), whichever comes latest, the contractor shall identify the three peer
reviewers and prepare a memo that lists the names of the peer reviewers and their affiliations and
includes the peer reviewers' bio's. Within five weeks of WAM approval of the of the peer
reviewers (via a TD) and receipt of the draft Prioritization Tool report from the WAM (via a
TD), whichever comes later, the contractor shall conduct the peer review, assemble the peer
review comments and recommendations in a peer review report organized by charge question,
prepare an introduction to the peer review report with a clear and concise overview of the
comments, and attach to the peer review report any marginal comments the peer reviewers had
on the Prioritization Tool report.
It is not necessary that the peer reviewers jointly reach consensus on their findings and
recommendations, since there maybe limited overlap in the peer reviewers' areas of expertise
and in the charge questions that they focus on. The contractor shall assume, for the purpose of
estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200
pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and
writing comments. EPA plans to provide the report to the contractor in mid-July.
-------
Peer Review Handbook
Page E-7
Deliverables and Schedule:
Task
1
2
3
Deliverable
Work plan and budget
Memo identifying peer reviewers
Peer review report
Schedule
Within 15 days of CO approval of work
assignment
Within 3 weeks of work assignment approval,
receipt of peer reviewer qualifications from
WAM, and receipt of charge from WAM,
whichever comes latest
Within five weeks of WAM approval of peer
reviewers and receipt of draft Prioritization
Tool report from WAM, whichever comes
later
Other Requirements:
CONTRACTOR COMMUNICATIONS
Upon approval of the Work Plan, the contractor shall maintain at least weekly
communications with the Work Assignment Manager regarding the status of work on the Work
Assignment.
CONFLICT OF INTEREST (COD [Revised Section]
The contractor must adhere to the following requirements:
a) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior
to commencement of any work, notify both the CO and PO of any actual or
potential organizational or personal conflicts of interest.
b) Provide a written certification, within 20 days of receipt of a Work Assignment,
QRT, or similar tasking document, that:
1) Either all conflicts of interest have been reported to the CO or that no
conflicts of interest exist. The contractor is directed to assure that none of
the conflicts disclosed are so direct and substantial as to rule out a
particular reviewer.
2) All personnel who perform work under this Work Assignment or relating
to this Work Assignment have been informed of their obligation to report
personal and organizational conflicts of interest to the CO.
-------
Page E-8 Peer Review Handbook
3) The Contractor recognizes its continuing obligation to identity and report
any conflicts of interest arising during performance of this Work
Assignment.
c) If a conflict of interest is identified during performance under this Work
Assignment, the Contractor shall immediately make a full disclosure in writing to
the CO. The disclosure shall include a description of action which the Contractor
has taken or proposes to take, after consultation with the CO, to avoid, mitigate,
or neutralize the conflict of interest.
d) After selecting the peer reviewers but before starting the peer review, submit to
EPA documentation that shows that the contractor has determined if the peer
review candidates:
1) have a conflict of interest or a situation that could create the appearance of
a lack of impartiality in relation to the work product
2) have had or presently have a financial relationship with EPA
and summarizes for EPA its efforts to identify and propose resolution of these
concerns with peer review candidates.
EXPENDITURE OF FUNDS/HOURS
In addition to the requirements of the contract, the contractor shall notify both the Project
Officer and the Work Assignment Manager when 75% of funds or hours for this Work
Assignment have been expended.
INFORMATION COLLECTION
Any other provision of this Work Assignment notwithstanding, the contractor shall not
proceed with any information collection where the same or similar information will be collected
from ten or more public respondents until written approval is received from the Contracting
Officer. This approval will cite an approval number from the Office of Management and Budget
as required by the Paperwork Reduction Act (PRA).
Only Federal agencies and their employees are exempt from the PRA definition of
"public respondent." State agencies and their employees are classified as "public respondents."
Soliciting similar information applies to any collection method, i.e., written, oral,
electronic, etc., and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc.
The PRA applies equally to "willing participants" and participation that is mandated by law.
-------
Peer Review Handbook Page E-9
Any question of applicability of the PRA shall be resolved by submitting a complete
description of the circumstances in a written request to the Contracting Officer. No collection
shall be undertaken until the Contracting Officer provides written notice to the Contractor as to
the applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not
initiate any collection until the requisite approval is received.
The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the
Interagency Reports Management Program as derived from 44 U.S.C. Chapters 29 and 31. All
work performed under this Work Assignment involving federal interagency reporting must be
done in full compliance with these GSA procedures.
CONFIDENTIAL BUSINESS INFORMATION
If this Work Assignment requires use of RCRA Confidential Business Information
(CBI), the contract must specifically authorize the contractor to have access to RCRA CBI and
the contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA
CBI Security Manual and in the contract. The contractor shall identify in the Work Plan budget
all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon
as it is no longer needed under this Work Assignment or before the expiration of the Work
Assignment, whichever occurs first.
PRINTING AND DUPLICATION
The contractor is prohibited from performing any printing under the Government Printing
and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000
impressions of a single-page document or 25,000 impressions of a multiple-page stand-alone
document, is limited to one color (black) copies, and does not exceed the maximum image size
of 10 3/4 by 14 1/4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA
WAM will determine in advance if the work can be performed more cost effectively and under
the job or time constraints at the EPA Print Shop. If the total number of photocopies for this
Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan
the photocopying costs by task and deliverable.
WORK ASSIGNMENT/WORK PLAN BUDGETS
The contractor shall not exceed either the dollar or PL hour budget contained in the
approved Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not
exceed the PL hour budget of the QRT.
-------
Page E-10 Peer Review Handbook
TECHNICAL DIRECTION
The Designated Work Assignment Manager (WAM) on this Work Assignment is
authorized to provide technical direction to the extent allowed under EPAAR (1552.237-71)
(APR 1984) (DEVIATION). Other than the Designated WAM, only the Project Officer and the
Contracting Officer are authorized to provide technical direction.
Technical direction includes:
(1) Direction to the contractor which assists the contractor in accomplishing the
Statement of Work
(2) Comments on and approval/acceptance of reports or other deliverables
Technical direction must be within the contract and the Work Assignment statement of
work. The Project Officer and the WAM do not have the authority to issue technical direction
which (1) institutes additional work outside the scope of either the contract or this Work
Assignment; (2) constitutes a change as defined in the "Changes" clause; (3) causes an increase
or decrease in the estimated cost of the contract or Work Assignment; (4) alters the period of
performance or deliverable due dates; or (5) changes any of the other express terms or conditions
of the contract or Work Assignment.
Technical direction will be issued in writing or confirmed in writing within five (5)
calendar days after verbal issuance. The technical direction memorandum will be provided to
the contractor and copies will be forwarded to the Contracting Officer and the Proj ect Officer. If
the contractor has not received written confirmation within five (5) calendar days of a oral
issuance, the contractor must so notify the Project Officer.
INHERENTLY GOVERNMENTAL FUNCTIONS
The contractor shall not perform any inherently governmental functions (IGF) under this
Work Assignment. If during the course of developing the plan of work, through receipt of
technical direction, or in carrying out the assignment any portion of the effort is considered to
possibly be an inherently governmental function, the contractor must immediately notify the
Project Officer and the Contracting Officer.
OCCUPATIONAL HEALTH AND SAFETY
Facility site visits conducted under a Work Assignment that include on-site inspections
or sampling must be conducted in full compliance with the Department of Labor, Occupational
Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440
(Occupational Health and Safety Manual).
-------
Peer Review Handbook Page E-ll
TRAVEL COSTS
The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal
regulations in incurring allowable travel costs under this Work Assignment, and correspondingly
must at all times seek and obtain Government rates whenever available and observe current
subsistence ceilings.
QUICK RESPONSE TASKS
Each Quick Response Task (QRT) shall be confirmed in writing and approved by the
Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and
the CO within two working days, giving a brief description of the plan of work, including best
estimate of hours (by P-level) and a break-out of costs to accomplish the task.
No task shall exceed a duration of 30 calendar days from start date to completion date.
The level of effort for each task shall be limited to a maximum of 250 labor hours.
Quick Response Task Requests do not change the dollar or professional labor hour
budgets of a Work Assignment.
-------
Page E-12 Peer Review Handbook
STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of Protozoa Method
Development Criteria Document
Period of Performance: Work Plan Approval to August 1, 1997
Work Assignment Manager: Sally Q. Government Employee
Office of Water
U.S. Environmental Protection Agency
LOE: 196 hours
SOW: 2.4
BACKGROUND INFORMATION:
The United States Environmental Protection Agency (EPA), Office of Water is charged with
protecting public health and the environment from adverse exposure to chemicals and microbials
in water media, such as ambient and drinking waters, wastewater/sewage sludge and sediments.
In support of this mission OW's Office of Science and Technology (OST) develops health
standards, health criteria, health advisories, and technical guidance documents for water and
water-related media. Under this work assignment, documents prepared by OST are to undergo
peer review.
Peer review is an important component of the scientific process. It provides a focused, objective
evaluation of a research proposal, publication, risk assessment, health advisory, guidance or
other document submitted for review. The criticism, suggestions and new ideas provided by the
peer reviewers stimulate creative thought, strengthens the reviewed document and confer
credibility on the product. Comprehensive, objective peer reviews leads to good science and
pro duct acceptance within the scientific community.
Under this work assignment, the contractor will receive one document (Protozoa Method
Development Criteria Document) for peer review which is related to human health and
ecological effects.
STATEMENT OF WORK:
Task 1. The contractor shall develop a work plan to address all tasks in this work
assignment. The work plan shall describe the steps that will be taken by the
contractor to provide for peer review, including selection of peer reviewer
candidates with appropriate expertise, determining absence of conflict of interest,
document and reference distribution, establishing schedules, preparing the peer
review report, and submittal of the peer review package. Curriculum vitae for all
persons assigned to complete this work assignment shall be provided. All P
-------
Peer Review Handbook
Page E-13
levels, hours and total costs for each task will be provided and costs greater than
$100.00 shall be itemized in detail.
Task 2. The contractor shall select a group of peer reviewers and determine their
availability for the task and absence of conflict of interest, and establish a
schedule for the peer review. The contractor is directed to assure that none of the
conflicts disclosed are so direct and substantial as to rule out a particular
reviewer. Three peer reviewers shall participate in the review. No single peer
reviewer may charge more than 40 hours to this task. It is fully acceptable for
peer reviewers to commit to less than 40 hours. The peer review will be
conducted for the Protozoa Method Development Criteria Document. Reviewers
selected by and working for the contractor shall be approved by the EPA Project
Officer in writing prior to their beginning work. Minimally, all peer reviewers
shall be accomplished in protozoan methods for sample recovery and analysis
from water. Approval submissions shall include the reviewers' names and
curriculum vitae.
Task 3. The contractor shall arrange for the selected peer reviewers to review the EPA
document. Prepare the charge to the peer reviewers based on technical direction
received from the EPA WAM. Provide the peer reviewers with copies of the
candidate report and all relevant references and instruct the selected peer
reviewers to undertake the review. The WAM will provide the contractor with
the final version of the document to be reviewed.
Task 4. The contractor shall monitor peer reviewers' progress to assure timely
completion. The contractor shall collate peer review comments, and organize the
comments in the peer review "for comments" document. Provide the peer review
document and all materials submitted by the peer reviewers to the EPA WAM.
SCHEDULE AND DELIVERABLES:
Task 1. (Work Plan) 15 days after receipt of work assignment
Task 2. 1 week after work plan approval
Task 3. 1 week after selection of peer reviewers
Task 4. 1 week after receiving comments from the peer reviewers
TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable
to this work assignment must be submitted and approved by the project officer.
-------
-------
Peer Review Handbook Page F-l
APPENDIX F - USEFUL FORMS
Peer Review Conflict of Interest Inquiry
Peer Review Checklist for Determining
Whether a Work Product Needs Peer Review
(Template Provided by Region V)
Peer Review Checklist for Conducting a Peer Review
(Template Provided by Region V)
Volunteer Service Program Participation Agreement
(EPA Form 3100-14)
-------
Page F-2 Peer Review Handbook
SUBJECT: PEER REVIEW CONFLICT OF INTEREST INQUIRY
Dear (Peer Reviewer — insert name):
You have been requested by EPA to serve as a Peer Reviewer for
(name of project) . Your participation in this review will be greatly
appreciated. However, it is possible that your personal affiliations and involvement in particular
activities could pose a conflict of interest or create the appearance that you lack impartiality in
your involvement for this peer review. Although your involvement in these activities is not
necessarily grounds for exclusion from the peer review, you should consult the contact named
below or other appropriate official to discuss these matters. Affiliations or activities that could
potentially lead to conflicts of interest might include:
a) work or arrangements concerning future work in support of industries or other
parties that could potentially be affected by regulatory developments or other
actions based on material presented in the document (or review materials) that
you have been asked to review;
b) your personal benefit (or benefit of your employer, spouse or dependent child)
from the developments or other actions based on the document (or review
materials) you have been asked to review;
c) any previous involvement you have had with the development of the document
(or review materials) you have been asked to review;
d) any financial interest held by you (or your employer, spouse or dependent child)
that could be affected by your participation in this matter; and
e) any financial relationship you have or have had with EPA such as research grants
or cooperative agreements.
Please contact (name and contact info for Agency peer review
official or primary contractor) to discuss any potential conflict of interest issues at your earliest
convenience, but no later than .
[Be sure to date and sign this inquiry]
-------
Peer Review Handbook Page F-3
PEER REVIEW CHECKLIST FOR DETERMINING
WHETHER A WORK PRODUCT NEEDS PEER REVIEW
Instructions: This checklist is based on the Agency's Peer Review Handbook and the October 2000
Region 5 Order "U.S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records
Management, and Work Product Authorization of Scientific and Technical Work Products " which
constitute Region 5 's standard operating procedures for peer review. If you have any questions about
peer review or need clarification when completing this checklist, please refer to the Handbook, available
via the internet at http://www.epa.gov/ord/spc/2peerrev.htm. Figure I on page 2 of the Handbook
includes a useful flow chart and cross references to specific sections of the Handbook that are
applicable to this checklist. You are also encouraged to consult with your Division or Office Peer
Review Coordinator. The Division/Office Peer Review Coordinators will periodically request
information from this checklist in order to update the National Peer Review Database.
1. Title of Work Product:
2. Product Description:,
3. Project Manager:,
Name, Organization and Phone Number
Please circle the
4. Determination if Work Product is Scientific or Technical: appropriate answer
a) Is the work product a scientific, engineering, economic, social yes no
science, or statistical document? (Examples of such documents
include: risk assessments, technical studies and guidance,
analytical methods, scientific database designs, technical models,
technical protocols, statistical surveys/studies, technical
background materials, and research plans and strategies.)
b) Is the work product a scientific or technical document resulting yes no
from a grant, contract or cooperative agreement?
c) Will the work product be used to support a research agenda, yes no
regulatory program, policy position, or other Agency
position or action?
If you answered "no" to all of these questions, your work product is not subject to EPA's
peer review policy for scientific or technical work products and does not need to be placed
on any of the peer review lists. Please proceed to #7 of this checklist. If you answered
"yes" to any of these questions, your work product might need peer review; please
continue on to #5 of this checklist.
-------
Page F-4 Peer Review Handbook
5. Determination if Work Product is a Major Work Product:
Determination of whether a work product is "major" will largely be on a case-by-case
basis. As the continuum of work products covers the range from the obviously major to
those products that clearly don't need peer review, there is no one single, easy yes/no
answer to the test of "major" (see Handbook, Section 2.2.3). There also is no single
definition of "significant." Determination of "major" and "significant" are the
responsibility of the Division or Office Director who is the official Decision Maker.
Please circle the
appropriate answer
a) Does the work product establish a significant precedent, yes no
model, or methodology?
b) Does the work product address significant controversial yes no
issues?
c) Does the work product focus on significant emerging or yes no
"cutting edge" issues?
d) Does the work product have significant cross-Agency or yes no
inter-agency implications?
e) Does the work product involve a significant investment of yes no
agency resources?
f) Does the work product consider an innovative approach or yes no
application for a previously defined problem, process or
methodology?
g) Is the work product required to be peer reviewed by statute yes no
or other legal mandate?
h) Does the work product support a regulatory decision, yes no
policy or guidance of major impact? (Major impact can mean
that it will have applicability to a broad spectrum of regulated
entities and other stakeholders, or that it will have narrower
applicability, but with significant consequences on a smaller
geographic or practical scale.)
i) Is the work product an application of or modification to yes no
an existing, adequately peer reviewed methodology or model
that departs significantly from the situation it was originally
designed to address?
If you answered "yes" to any of these questions, your work product needs peer review
unless special circumstances exist; please continue on to #6. If you answered "no" to all of
these questions, your work product probably does not need peer review. However, peer
review can always be done to improve the quality of the work product. Please proceed to
#7 of this checklist.
-------
Peer Review Handbook Page F-5
6. Determination Whether Circumstances Exist Where a Major Work Product
Would Not Be Peer Reviewed:
Please circle the
appropriate answer
a) Was the work product previously reviewed by recognized yes no
experts or an expert body? (Note: Peer review of an EPA work
product by a recognized refereed journal strengthens the
scientific credibility of the work product but does not eliminate
the need to have the work product itself peer reviewed for issues
and concerns to support an Agency action. See Sections 2.4.4 and
2.4.5 of the Handbook for more details.)
b) Are the scientific or technical methodologies or information yes no
being used commonly accepted in the field of expertise?
c) Has the regulatory activity or action which the work product yes no
supports been terminated or canceled?
d) Is there a statutory or court ordered deadline, or a time yes no
constraint which may limit or preclude peer review of the
work product?
If you answered "yes" to any of these questions, your work product probably does not
require peer review. This decision with the justification needs to be concurred with and
signed off by the Division/Office Director. The decision with the justification must be
retained in the peer review files and noted in Peer Review Work Product List C in the
National Peer Review Database. Continue on to #7. If you answered "no" to all of these
questions, proceed to #8.
7. Next Steps For Work Products That Will Not Be Peer Reviewed:
a) Division/Office Director concurs with the decision that the work
product should not be peer reviewed.
Signature of Division/Office Director and Date Signed
b) A copy of this completed checklist has been given to the Div/Off
Peer Review Coordinator and put in the official peer review files in
the Division/Office.
Signature of Div/Off Peer Review Coordinator and Date Signed
Location of Div/Off Peer Review Files
-------
Page F-6 Peer Review Handbook
c) Work product has been placed on Peer Review Work Product List C
in the National Peer Review Database. (Note: This only applies to those
work products subject to the peer review policy.)
Signature of Div/Off Peer Review Coordinator and Date Signed
If all of the necessary information is complete, you are done. You don't need to proceed
any further with this checklist.
8. Next Steps For Work Products That Will Be Peer Reviewed:
a) Division/Office Director has been consulted and concurs with the decision that
the product should be peer reviewed.
Signature of Division/Office Director and Date Signed
b) A copy of this completed checklist has been given to the Division/Office
Peer Review Coordinator and put in the official peer review files in the
Division/Office.
Signature of Div/Off Peer Review Coordinator and Date Signed
Location of Div/Off Peer Review Files
c) Work product has been placed on Peer Review Work Product List B
in the National Peer Review Database.
Signature of Div/Off Peer Review Coordinator and Date Signed
Because your work product will be peer reviewed, you need to complete a second checklist
entitled "Peer Review Checklist for Conducting a Peer Review"
-------
Peer Review Handbook Page F-7
PEER REVIEW CHECKLIST FOR CONDUCTING A PEER REVIEW
Instructions: This checklist is based on the Agency's Peer Review Handbook and the October 2000
Region 5 Order "U.S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records
Management, and Work Product Authorization of Scientific and Technical Work Products " which
constitute Region 5 's standard operating procedures for peer review. If you have any questions about
peer review or need clarification when completing this checklist, please refer to the Handbook, available
via the internet at http://www.epa.gov/ord/spc/2peerrev.htm. Pages 2-4 of the Handbook contain useful
flowcharts and cross references to specific sections of the Handbook that are applicable to this
checklist. You are also encouraged to consult with your Division or Office Peer Review Coordinator.
The Division/Office Peer Review Coordinators will periodically request information from this checklist
in order to update the National Peer Review Database.
1. Title of Work Product:
2. Product Description:,
3. Project Manager:
Name, Organization and Phone Number
Check the box when
4. Up-front Considerations for Pfenning the Peer Review: item is completed
a) The Div/Office Director has chosen a peer review leader for the project. Q
(Note: The project manager and peer review leader can be the same person.)
Name of Peer Review Leader:
Phone Number:
Organization:
b) The peer review leader has obtained appropriate peer review training
before conducting the peer review.
c) Key questions and issues have been identified to include in the charge
to the peer reviewers.
d) The Div/Office Records Coordinator has been consulted to insure
that all the files, including electronic records, will be created, maintained,
retained, and disposed of appropriately and in accordance with
Div/Office and Agency procedures.
e) A formal peer review record or file has been established, and provisions
have been made to store any electronic records associated with
the work product and peer review.
Location of Record/File:
Provisions for Electronic Records:
-------
Page F-8 Peer Review Handbook
Check the box when item is
completed or circle the
appropriate answer
(NA = not applicable)
f) There is a source of adequate funding to pay for external peer review if
external peer review is necessary and funding is needed. (Note: Contracts Q NA
can be used for peer review services. However, special management controls
are required to ensure proper use of these contracts. See Section 3.6 of the
Handbook for details.)
Source of Funding:
g) Resource limitations may restrict the peer review. (If "yes " was Yes No
selected, a limited peer review might be considered. However, only in
very rare circumstances should resource limitations restrict peer review.
Peer review must be planned for as part of a project's budget.)
h) Amount of time needed for peer review(s) has been allotted given
existing constraints of potential peer reviewers, deadline for the final D
work product, logistics for the peer review, etc.
Length of Time Needed:
5. Develop the Charge to the Peer Reviewers:
a) A clear, focused charge has been formulated that identifies recognized I I
issues, asks specific questions, and invites comments or assistance.
b) The charge has been included in the peer review record. Q
6. Select the Peer Review Mechanism:
a) The work product is novel, complex, controversial, or has great Yes No
cost implications. (If the answer is "yes" to any of the above, serious
thought should be given to conducting an external peer review. If the
answer is "no " to all of the above, internal peer review is probably sufficient.)
b) A determination has been made regarding which components Q
or stages of the work product will be peer reviewed. (Note: Generally,
peer review is recommended for each stage of a product's development.)
Components to be peer reviewed:
c) A peer review mechanism (e.g., internal, external or a combination Q]
of both) has been chosen for the work product or stages of the work
product. Mechanism:
d) The work product either: 1) has been, or is being, generated as part Yes No
of administrative or civil enforcement activities by U.S. EPA, or
2) likely will be used in the future to support administrative or civil
enforcement activities by U.S. EPA. (If the answer is "yes" to either
item above, then the Office of Regional Council (ORC) must be consulted
if the Peer Review Leader believes an external peer review is needed or is
preferable. ORC concurrence should be obtained.)
-------
Peer Review Handbook Page F-9
Check the box when
item is completed,
or circle yes or no
e) The work product is going to be peer reviewed via a refereed, Yes No
scientific journal. (If the answer is "yes, " the work product still
should be considered for peer review because journal peer review
may not cover issues and concerns that the Agency would want peer
reviewed in order to support an Agency action.)
f) Logistics for conducting the peer review (e.g., written comments will Q
be received by mail, or will be collected at a meeting) have been
included in the peer review record.
g) The Div/Off Director has concurred with the recommended method Q
of peer review.
Date of Div/Off Director Concurrence:
h) The concurrence of the Div/Off Director has been included in the Q
peer review record.
7. Determine the Specific Time Line for the Peer Review:
a) A start date for the peer review has been selected. rn
Start Date:
b) The amount of time the peer reviewers will be given to conduct
the peer review has been determined.
Number of Days for Review:
c) A due date for comments from the reviewers has been selected.
Due Date:
d) The amount of time necessary to incorporate comments from the
peer reviewers into the work product has been determined.
Number of Days for Revision:
e) A deadline for final completion of the work product has been
determined.
Due Date:
8. Select the Peer Reviewers:
a) Advice was sought in developing a list of potential peer reviewer
candidates who are independent of the work product and have
appropriate scientific and technical expertise.
b) The expertise required for the peer review has been determined.
c) In reviewing the candidates, a balance and a broad enough
spectrum of expertise were considered.
d) In reviewing the candidates, any potential conflicts of interest
were considered.
-------
Page F-10 Peer Review Handbook
Check the box when
item is completed
e) The peer reviewers have been selected and the process for O
selecting the reviewers, including inquiries and resolution
of potential conflicts of interest, has been documented and included
in the peer review record/file. (Note: Conflict of Interest Inquiry Forms
are available from the Regional and Div/OffPeer Review Coordinators.)
9. Obtain and Transmit Materials for Peer Review:
a) Instructions have been given to the peer reviewers which ask for Q
written comments in a specified format by the specified deadline
that are responsive to the charge.
b) The peer reviewers have been provided with the essential documents, Q
data, and information to conduct their review.
Date Peer Reviewers Given Charge/Materials:
c) The peer reviewers have been instructed not to disclose draft work Q
products to the public.
d) The peer review record/file contains all the materials given to the rn
peer reviewers.
10. Conduct the Peer Review:
a) Written comments have been received from all peer reviewers. rn
Date all comments were received:
b) All clarification or additional information necessary from the peer
reviewers is received.
c) The validity and objectivity of the comments have been evaluated.
d) Appropriate experts/staff/managers have been consulted on the
potential impacts of the comments on the final work product, the
project schedule, and budget.
e) The peer review comments have been included in the peer review
record/file.
11. Consider the Peer Review Comments:
a) Decisions have been made regarding which comments are
accepted and will be incorporated into the final work product, and
which comments will not be incorporated.
b) A memo or other written record has been prepared which responds
to the peer review comments and specifies acceptance or, where
thought appropriate, rebuttal and non-acceptance.
c) The Div/Off Director has concurred with the decisions and written
record on how to incorporate the peer reviewers comments in the
work product and on which comments will not be incorporated.
Date of Div/Off Director concurrence:
-------
Peer Review Handbook Page F-ll
Check the box when
item is completed,
or circle yes or no
d) The concurrence of the Div/Off Director has been included Q
in the peer review record/file.
e) The memo or written record documenting how comments were Q
handled and how the work product was revised has been
included in the peer review record/file.
f) The work product has been revised to incorporate the acceptable rn
comments.
g) The peer review performed during the process of developing the i—i
work product has been summarized and included in the work product.
h) It is necessary to send the revised work product back to the peer Yes No
reviewers. (If the answer is "yes, "proceed to item #11 i. If the
answer is no, proceed to item #12.)
i) Additional comments are received, evaluated, and incorporated ,—,
into the work product, and placed in the peer review record.
12. Consider Other Comments:
a) Prior to finalization, the document needs additional internal Yes No
and/or external programmatic review. (If the answer is "yes," go to
#12b. If the answer is "no," proceed to #13.)
b) Written comments by programmatic reviewers have been received. CH
c) Final decisions have been made regarding which comments are
accepted and will be incorporated into the final work product, and EH
which ones will not be incorporated.
d) A memo or other written record has been prepared which responds
to the programmatic review comments and specifies acceptance or, D
where thought appropriate, rebuttal and non-acceptance.
e) Div/Off Director has concurred with the decisions and written
record on how to incorporate the programmatic comments. Q
Date of Div/Off Director concurrence:
f) The memo or written record has been included in the peer review i—i
record/file.
g) The work product has been revised to incorporate the acceptable D
programmatic comments.
13. Finalize Work Product and Close Out Peer Review:
a) The work product has been completed. EH
b) The Div/Off Director has approved the work product. i—i
Date of Div/Off Director Approval:
c) The Div/Off Director approval has been included in the
peer review record/file.
-------
Page F-12 Peer Review Handbook
Check the box when
item is completed, or
circle yes or no
d) The Div/Off Director has judged the work product to be sufficiently Yes No
controversial, of significant enough interest to outside parties, or of
wide enough distribution, such that it should also be authorized by the
Regional Administrator (RA), or the Deputy RA (DRA). (If the answer
is "yes, "proceed to #13e. If the answer is "no, "proceed to #13f.)
e) The RA or DRA has authorized the work product. i—i
Date of RA or DRA Authorization:
f) The final work product has been included in the peer review ,—,
record/file.
14. Publication and Release of Reports:
a) The Div/Off Director has approved publication or release of the Q
work product.
b) The written approval by the Div/Off Director has been included rn
in the peer review record/file.
c) The Div/Off Director has judged the work product to be sufficiently Yes No
controversial, of significant enough interest to outside parties or of
wide enough distribution, such that its distribution or release
should also be authorized by the RA or DRA. If the answer is
"yes, "proceed to #14d. If the answer is "no, "proceed to #15.
(Note: The Div/Off Director's decision to elevate to the RA or DRA can
be made concurrently with item #13d.)
d) The RA or DRA has authorized distribution or release of the work i—i
product.
Date of RA or DRA Authorization:
15. Retention of Peer Review Files and Records:
a) The Div/Off official procedures for administrative records and the
Agency's record retention schedules have been examined to i—i
determine how long the peer review record/file, including electronic
records, should be retained. (Note: The required time of retention for
final reports and supporting data varies depending upon the nature
of the report, however, final reports which are mission related or have
an EPA number and receive external distribution are generally
permanent federal records.)
b) The Div/Off Records Officer or the Regional Records Officer has been
consulted to help determine how long the peer review record/file, EH
including electronic records, should be retained.
-------
Peer Review Handbook Page F-13
Check the box when
item is completed
c) A location for the completed peer review record/file has been •—•
identified, and provisions have been made to retain electronic
records associated with the work product and peer review.
(Note: This can be the same location and provisions as identified in #4e.)
Location of Record/File:
Provisions for Electronic Records:
d) Someone has been assigned the responsibility for maintaining the
record/file and electronic records, and ensuring that they are either
archived or destroyed appropriately. (Note: This can be the same person
as identified in #4a.)
Contact Name and Phone No:
Organization:
16. Closeout of Checklist:
a) Items #1-15 of checklist have been completed.
Signature of Peer Review Leader and Date Signed
b) A copy of the completed checklist has been given to the Div/Off
Peer Review Coordinator.
Signature of Div/Off Peer Review Coordinator and Date Signed
c) The completed checklist has been included in official peer review
record/file.
d) The work product has been moved from Peer Review Work Product
List B to List A in the National Peer Review Database.
Date Product moved to List A:
-------
Page F-14
Peer Review Handbook
U.S. Environmental Protection Agency
VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT
Name of Program Participant
Citizen of U.S.
D Yes D No
Project Supervisor
Proposed Arrival Date
Sponsored by Educational
Institution
D Yes dl No
Actual Arrival Date
Proposed Termination
Actual Termination
Name and Address of Institution (if "Yes" )
Proposed Organizational Agreement (Include Geographical Location)
Describe Project(s) on Which Participant Will Work, Including Scope and Anti cipated Hour s per Week
Facilities and Equipment to be Made Available by EPA
Degree of Supervision to be Exercised by EPA
Assistance and Degree of Cooperation Required by other Agency Personnel by Participant
Project Supervisor Certifies That Services to be Produced by the Progiam Participant are not Services Provided for Through EPA Agency
Operations
Signature of Supervisor
THE VOLUNTEER AGREES THAT:
a. Any and al claims for compensation from the Government of the United States are waived for any services performed;
b. The Government has anon-exclusive royalty-free license to use or reproduce and patent or copyright material which is
developed as part of and during participation in this program; and
c. He/she will adhere to the administrative instructions and requirements of the agency while on EPA premises.
Participant Signature
PERSONNEL OFFICE
CONCURRENCE
SIGNATURE OF PERSONNEL OFFICER OR DE SIGNEE
DATE
EPA Form 3100-14 (Rev. b 30) Replaces the 7-72 edition and the 12-77 edition which was entitled "Unpaid Work-Study Program"
-------
Peer Review Handbook Page G-l
APPENDIX G - REFERENCES CONCERNING
PEER REVIEW
American Chemical Society and the Conservation Foundation (1985) Issues in Peer Review of
the Scientific Basis for Regulatory Decisions, Washington, DC, November 1985.
Browner, C. (1994) Peer Review Program, Washington, DC, Memorandum issued June 7, 1994.
(NOTE: Attached as Appendix A to this Handbook)
Chubin, D. (1994) Grants peer review in theory and practice, Evaluation Review 18: 20-30.
Chubin, D. and E. Hackett (1990) Peerless Science: Peer Review and Science Policy, Albany,
NY: State University of New York Press.
Jasanoff, S. (1990) The Fifth Branch: Science Advisors as Policymakers, Cambridge, MA:
Harvard University Press.
Kostoff, R. (1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual
Meeting, Baltimore, MD, February 9, 1996.
National Academy of Sciences (1999) Evaluating Federal Research Programs: Research and
the Government Performance and Results Act, Washington, DC: National Academy Press,
January 1999.
National Environmental Policy Institute (1996) Enhancing the Integrity and Transparency of
Science in the Regulatory Process, Washington, DC: National Environmental Policy Institute,
Fall 1996.
National Research Council (1995) Interim Report of the Committee on Research and Peer
Review in EPA, Washington, DC: National Academy Press, March 1995.
National Research Council (2000) Strengthening Science at the U. S. Environmental Protection
Agency: Research Management and Peer Review Practices, Washington, DC: National
Academy Press.
Reilly, W. (1993) Peer-review Policy, Washington, DC, Memorandum issued January 19, 1993.
Rennie, D. (1999) The Development and Rationale of Peer Review, In: Peer Review in Health
Sciences, Fiona Godlee and Tom Jefferson, eds., BMA Books, London.
-------
Page G-2 Peer Review Handbook
Science Advisory Board (1999) An SAB Report: Review of the Peer Review Program of the
Environmental Protection Agency, EPA-SAB-RSAC-00-002, U.S. Environmental Protection
Agency, Science Advisory Board, Washington, DC, November 1999.
Spitzer, H. (1995) Peer Review Practices in the Federal Government, Bethesda, MD:
Environmental Network, report prepared for the American Industrial Health Council, April
26,1995)
USEPA (1992) Safeguarding the Future: Credible Science, Credible Decisions, U.S.
Environmental Protection Agency, Washington, DC, March 1992.
USEPA (1998) Science Policy Council Handbook: Peer Review, EPA 100-B-98-001, U.S.
Environmental Protection Agency, Science Policy Council, Washington, DC, January 1998.
USEPA (1999) EPA 's Selection of Peer Reviewers, EPA OIG Report No. 1999-P-217, U.S.
Environmental Protection Agency, Office of the Inspector General (OIG), Washington, DC,
September 1999.
U.S. General Accounting Office (1994) Peer Review: EPA Needs Implementation Procedures
and Additional Controls, GAO/RCED-94-89, Washington, DC: U.S. Government Printing
Office, February 1994.
U.S. General Accounting Office (1996) Peer Review: EPA's Implementation Remains Uneven,
GAO/RCED-96-236, Washington, DC: U.S. Government Printing Office, September 1996.
U.S. General Accounting Office (1999) Federal Research: Peer Review Practices at Federal
Science Agencies Vary, GAO/RCED-99-99, Washington, DC: U.S. Government Printing
Office, March 1999.
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
-------
Peer Review Handbook
Notes and Comments
------- |