EPA/100/B-06/002
U.S. Environmental Protection Agency


         Peer Review Handbook

                  3rd Edition

      Prepared for the U.S. Environmental Protection Agency
        by Members of the Peer Review Advisory Group,
             for EPA's Science Policy Council
               Science Policy Council
        U.S. Environmental Protection Agency
              Washington, DC 20460

-------
This report is dedicated to the memory of our friend and colleague,
Nash Gerald. Nash was a wonderful colleague who thoughtfully
strived to perfect the use of peer review at EPA.

Nash joined EPA in 1989. In addition to his duties coordinating
science/policy and research for the Office of Air Quality Planning
and Standards, he consistently contributed to considerations of the
Agency's peer review practices. Nash's courteous and
indefatigable approach to resolving details greatly enhanced the
revision of this Handbook. He will be missed.

-------
                               Peer Review Handbook                      Page iii
                                 Disclaimer
This Handbook was developed by EPA to provide guidance to EPA staff and managers who are
planning and conducting peer reviews. This Handbook is intended to improve the internal
management of EPA by providing recommended procedures and approaches for EPA staff and
managers.  This Handbook is a guidance manual and not a rule or regulation. The Handbook
does not replace existing laws or regulation, does not change or substitute for any legal
requirement, and is not legally enforceable. This Handbook does not create or confer legal rights
or impose any legally binding requirements on EPA or any party.  The use of non-mandatory
language such as "may," "can" or "should" in this Handbook does not connote a requirement but
does indicate EPA's strongly preferred approach to ensure the quality of peer reviews conducted
or initiated by EPA. Mention of trade names or commercial products does not constitute
endorsement or recommendation for use.

-------
         Peer Review Handbook                 Page iv
The Peer Review Advisory Group
      Connie Bosnia         ORD
      Jennifer Bo wen        OPEI
      Michele Burgess       OSWER (now at USDA)
      Mary E Clark          OAR
      Kerry Dearfield        ORD (now at USDA)
      Patrick Easter          OSWER
      Daniel Fort            SAB
      Nash Gerald           OAR
      Roland Hemmett       Region 2
      Vincia Holloman       OEI
      Joyce Jatko            ORD (now at Navy)
      Barbara Klieforth       ORD
      Steven  Knott          OPPTS
      Andrew Mclntosh      AO (now at GAO)
      John E  Miller          ORD
      Manisha Patel          OGC
      Mary Reiley           OW
      Nancy Wentworth      OEI

-------
                             Peer Review Handbook                     Page v
                           Table Of Contents


DISCLAIMER	iii

PEER REVIEW ADVISORY GROUP                                              iv

FOREWORD	xiii

LIST OF ACRONYMS	xv

QUICK START GUIDE TO THE PEER REVIEW PROCESS	1
  Figure 1 - Flowchart for Planning a Peer Review	2
  Figure 2 - Flowchart for Conducting a Peer Review	3
  Figure 3 - Flowchart for Completing a Peer Review	4
  Manager's Planning Checklist for Peer Review	5
  Regulatory Action Development Checklist for Workgroups	6

PEER REVIEW GUIDANCE	9

1.  THE NEED FOR PEER REVIEW	11

  1.1    Overview Statement	11

  1.2    Understanding Peer Review	11
    1.2.1    Why Use Peer Review?	11
    1.2.2    What is Peer Involvement?	12
    1.2.3    What is Peer Review?	12
    1.2.4    What is Peer Input?	12
    1.2.5    How is Peer Review Different from Peer Input?	13
    1.2.6    Can Someone Who Provided Peer Input Become an Independent Peer
            Reviewer for the Same Work Product Later in the Process?	13
    1.2.7    Can the Same Peer Reviewer be Used More Than Once if a Product Will Be
            Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used
            Again and Again for Different Products?	13
    1.2.8    How is Peer Review Different from Public Comment?	14
    1.2.9    How is Peer Review Different from Stakeholder Involvement?	14
    1.2.10   What Role Does Peer Review Have in Regulatory Development?	14
    1.2.11   How Does the Tiering Process Affect Peer Review?	15
    1.2.12   Should You Discuss Peer Review in the Analytic Blueprint for Your
            Regulation?	15

-------
                             Peer Review Handbook                      Page vi
  1.2.13    What Role does Peer Review Have in Regulatory Negotiations?	16
  1.2.14    Should You Address Peer Review in the Preamble of a Regulation?	16
  1.2.15    How is Peer Review Documented in the Action Memo for Regulations?	16

1.3.   Peer Review and Information Quality	16

  1.3.1     Overview Statement	16
  1.3.2     How Does Peer Review Ensure the Quality of Information EPA Disseminates? 17
  1.3.3     How Do You Ensure the Quality of Data Supporting EPA's Information
           Products?	17
  1.3.4     Are Work Products Undergoing Peer Review Subject to the Guidelines?	18

1.4    Annual Agency Reporting	19

  1.4.1     What Are EPA's Reporting Practices and What Information Is Publicly
           Available?	19
  1.4.2     What Information Is Provided in the Science Inventory?	19
  1.4.3     How Will the Public Be Allowed to Comment on the Peer Review Plans
           for Influential  Scientific Information and Highly Influential  Scientific
           Assessments?	21
  1.4.4     What Is ORD' s Role in Reviewing the Science Inventory and How Will This
           Review Be Conducted?	21
  1.4.5     How Will EPA Submit an Annual Report to OMB?	21

1.5    The Roles of People and Organizations in Peer Review	22

  1.5.1     Who is Ultimately Accountable for Peer Review?	22
  1.5.2     Who are the Agency Staff involved in Peer Review?	22
  1.5.3     Who are the Decision Makers and What are Their Responsibilities?	22
  1.5.4     Who are the Peer Review Coordinators and What are Their Responsibilities? ... 23
  1.5.5     Who are the Peer Review Leaders and What are Their Responsibilities?	25
  1.5.6     Who are the Peer Reviewers?	26
  1.5.7     What are the Responsibilities of Peer Reviewers?	26
  1.5.8     What is an Independent Peer Reviewer?	26
  1.5.9     When does an Agency Internal Peer Reviewer Qualify as Independent?	27
  1.5.10    What is a Peer Review Panel?	27
  1.5.11    What is a Subj ect Matter Expert?	27
  1.5.12    What is the Role of the Science Policy Council (SPC)?	27
  1.5.13    What is the Role of the Peer Review Advisory Group (PRAG)?	28
  1.5.14    What is the Role of the Office of Research and Development (ORD)?	28

-------
                               Peer Review Handbook                       Page vii
2.  PL ANNING A PEER REVIEW	29

  2.1    Overview Statement	29

  2.2    Determining Which Work Products to Peer Review	29
     2.2.1    What are Scientific and/or Technical Work Products?	29
     2.2.2    What Scientific and/or Technical Work Products Need Peer Review?	30
     2.2.3    How Does One Determine Whether a Scientific and/or Technical Work Product
             is Influential Scientific Information?	30
     2.2.4    How Does One Determine Whether Influential Scientific Information is a
             Highly Influential Scientific Assessment?	32
     2.2.5    What Economic Work Products Need Peer Review?	34
     2.2.6    Should Economic Work Products Prepared in Support of Regulations that are
             Classified as Maj or or Economically Significant be Peer Reviewed?	35
     2.2.7    What Other Economic Work Products Might Benefit from Peer Review?	36
     2.2.8    What Other Social Science Work Products Need Peer Review?	36
     2.2.9    How Should Peer Review be Handled for Products Developed Under an
             Interagency Agreement (IAG)?	37
     2.2.10   Should Products from Contracts, Grants, and Cooperative Agreements
             Receive Peer Review?	37
     2.2.11   How Does Peer Review Apply to Products Generated Through EPA Grants
             or Cooperative Agreements?	38
     2.2.12   Can the Recipient of a Grant or Cooperative Agreement Use Agreement Funds
             to Pay Peer Reviewers of their Work Products?	39
     2.2.13   Do Products Generated Under EPA Grants or Cooperative Agreements Need
             to be Reported in the Science Inventory	40
     2.2.14   Should Site-Specific Decisions be Subject to Peer Review?	40
     2.2.15   Should NEPA Products be Subject to Peer Review?	40
     2.2.16   Should Environmental Regulatory Models be Peer Reviewed?	41
     2.2.17   Should Another Organization's Work Products That Have Been Submitted to
             EPA for Use in Decision Making be Peer Reviewed?	41
     2.2.18   Can Work Products That are Not Determined to be Influential Scientific
             Information or Highly Influential Scientific Assessments Still be
             Peer Reviewed?	42

  2.3    Determining Which Work Products Do Not Receive Peer Review	42
     2.3.1    What Information Is Exempt from the OMB Bulletin Provisions?	42
     2.3.2    Are There Other Circumstances Where Peer Review Is Not Necessary?	43
     2.3.3    For Information That Is Not Exempt, Are There Circumstances Where the
             Peer Review Provisions of the OMB Bulletin Can Be Waived or Deferred?	43
     2.3.4    Does Publication in a  Refereed Scientific Journal Mean That Adequate Peer

-------
                             Peer Review Handbook                      Page viii
           Review Has Been Performed?	43
  2.3.5     Do Voluntary Consensus Standards Undergo Peer Review?	44

2.4    Choosing a Peer Review Mechanism	44
  2.4.1     How Do You Determine the Appropriate Peer Review Mechanism?	44
  2.4.2     What are Some Examples of Internal Peer Review?	46
  2.4.3     What are Some Examples of External Peer Review?	46
  2.4.4     What is the Role of Peer Review by aRefereed Scientific Journal?	47
  2.4.5     Do Agency Work Products Become Candidates for Peer Review when Peer
           Reviewed Journal Articles are Used in Support of that Work Product?	47
  2.4.6     When and How Often Should Peer Review Occur?	48
  2.4.7     What Factors are Considered in Setting the Time Frame for Peer Review?	49
  2.4.8     Which Office/Region or Other Agency is Responsible for Conducting the
           Peer Review?	49

2.5    Creating the Peer Review Record and Peer Review Reports	49
  2.5.1     What is the Peer Review Record?	49
  2.5.2     How Can the Peer Review Record Improve the Peer Review Process?	50
  2.5.3     What Should Be in the Peer Review Record?	50
  2.5.4     How and When Should You Develop and Post Peer Review Reports and
           Agency Responses for Influential Scientific Information and Highly Influential
           Scientific Assessments?	51
  2.5.5     What Should You Do with a Peer Review Record That Pertains to a
           Rulemaking Action?	51
  2.5.6     When Should the Peer Review Record Building Process Begin?	51
  2.5.7     What are the Differences in Record Keeping for a Review by Individuals
           Compared to Review by a Panel?	52
  2.5.8     Where Should the Peer Review Record be Kept and For How Long?	52
  2.5.9     Are Internal Peer Review Comments Included in the Peer Review Record?	52

2.6    Budget Planning	53
  2.6.1     What Budgetary Factors Should You Consider in a Peer Review?	53
  2.6.2     What Input is Needed for the Annual Budget Formulation and Budget
           Execution Process?	53

2.7    Legal Considerations	53
  2.7.1     What Are the Legal Ramifications of the Peer Review Policy?	53
  2.7.2     Is Legal  Advice Needed?	54

2.8    Federal Advisory Committee Act (FACA) Considerations	54
  2.8.1     When Do FACA Requirements Apply to EPA-Run Peer Reviews?	54

-------
                               Peer Review Handbook                      Page ix
    2.8.2    When Are EPA-Run Peer Reviews Not Subject to F AC A?	55
    2.8.3    How Do You Ensure that a Contractor-Run Peer Review Does Not Become
             SubjecttoFACA?	55

3.  CONDUCTING A PEER REVIEW	57

  3.1     Overview Statement	57

  3.2     Charge to the Peer Reviewers	57
    3.2.1    What is a Charge?	57
    3.2.2    What are the Essential Elements of a Charge?	58
    3.2.3    Where Can You Get an Example of a Charge?	58
    3.2.4    Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?	58
    3.2.5    Who Writes the Charge When You Hire a Contractor to Conduct the
             Peer Review?	59
    3.2.6    Is it Okay to  Ask a Contractor to Develop the Charge to the Peer Reviewers? ... 59

  3.3     Time Line	59
    3.3.1    What are the Factors in Scheduling a Peer Review?	59

  3.4     Selection of Peer Reviewers	60
    3.4.1    What are Considerations for Selecting Peer Reviewers?	60
    3.4.2    Where Do You Find Peer Reviewers?	60
    3.4.3    Are External or Internal Peer Reviewers Preferred?	62
    3.4.4    What is Important in the Mix of a Peer Review Panel?	62
    3.4.5    Ethical Standards	63
    3.4.6    What Techniques Help Ensure Disclosure and Appropriate Resolution of
             Conflicts of Interest?	68
    3.4.7    Are There Constraints to Selecting Peer Reviewers?	71
    3.4.8    If State Employees Are Used as Peer Reviewers, Can EPA Pay Them
             for This Service?	72
    3.4.9    Does the Agency Recognize Peer Reviews That External Parties Conduct
             and/or Pay For?	72

  3.5     Materials for Peer Reviewers	73
    3.5.1    What Instructions Do You Give Peer Reviewers?	73
    3.5.2    What Materials Should be Sent to Peer Reviewers?	73
    3.5.3    How Closely can EPA Interact with Peer Reviewers During the Review?	75

  3.6     Peer Review Services	75
    3.6.1    What are Gratuitous Services for Peer Review?	75

-------
                             Peer Review Handbook                    Page x
    3.6.2    Can You Use a Contract to Obtain Peer Review Services?	76
    3.6.3    How Do You Write a Statement of Work for Contracts?	76
    3.6.4    What are Some Management Controls for Contracts?	77
    3.6.5    Can You Identify and/or Select Peer Reviewers When Using a Contract?	80
    3.6.6    How is Travel Handled with Contracts or Purchase Orders?	80
    3.6.7    How is Travel Handled with Special Government Employees?	81

4.  COMPLETING A PEER REVIEW	82

  4.1    Overview	82

  4.2    Final Work Product	82
    4.2.1    How Do You Incorporate Peer Review Comments into the Final
            Work Product?	82
    4.2.2    How Might Peer Review Comments Impact Your Work Product?	83
    4.2.3    What Should the Final Work Product Say About the Peer Review Process?	84
    4.2.4    Can the Identity of Peer Reviewers be Kept Anonymous?	84

  4.3    Completing the Peer Review Record	85
    4.3.1    How Do You Complete the Peer Review Record?	85
    4.3.2    Where Should the Peer Review Records be Kept, and for How Long?	85
    4.3.3    Is Information Regarding a Peer Review Subject to Release under FOIA?	86


APPENDIX A EPA'S PEER REVIEW POLICY  	A-l

APPENDIX B OMB INFORMATION QUALITY BULLETIN FOR PEER REVIEW AND
             PREAMBLE	B-l

APPENDIXC SOUND  SCIENCE AND PEER REVIEW IN RULEMAKING	C-2

  PEER REVIEW (CONDITIONAL TEMPLATE)	C-4
  FRAMEWORK FOR ACTION MEMORANDA	C-5

APPENDIXD EXAMPLES OF PEER REVIEW CHARGES	D-l

  CHARGE EXAMPLE 1 - US EPA  Science Advisory Board (SAB) Consultation on
  Suspended and Bedded Sediments	D-3
  CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children	D-4
  CHARGE EXAMPLE 3 - Economics - Study of Municipal Government Costs and Financial
  Impacts from Environmental Regulations	D-5

-------
                            Peer Review Handbook                     Page xi
  CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National
  Sewage Sludge Survey for Pollutants in Sewage Sludge	D-7
  CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the
  Development of Water Quality Criteria for Suspended and Bedded Sediments (SABS)	D-9
  CHARGE EXAMPLE 6 - Clean Air Scientific Advisory Committee Review of the
  Agency's National Ambient Air Monitoring Strategy	D-12
  CHARGE EXAMPLE 7 - The Multimedia, Multipathway, and Multireceptor
  Risk Assessment (3MRA) Model	D-15

APPENDIX E  GUIDANCE ON REQUESTING A REVIEW BY THE SCIENCE
ADVISORY BOARD (SAB)	E-l

  TABLE 1 - PROCESS FOR SELECTING SCIENCE ADVISORY BOARD (SAB)
  PROJECTS	E-5

APPENDIX F  EXAMPLE STATEMENTS OF WORK FOR CONTRACTS	F-8

  STATEMENT OF WORK - EXAMPLE 1 - Peer Review of Prioritization Tool Report ....F-10
  STATEMENT OF WORK - EXAMPLE 2 External Peer Review of Protozoa Method
  Development Criteria Document	F-17
  STATEMENT OF WORK - EXAMPLE 3 - Technical Support for Peer Review Services.F-19

-------
Peer Review Handbook                    Page xii

-------
                                Peer Review Handbook                      Page xiii
                                   Foreword

Science is the foundation that supports all of our work here at EPA.  Strong, independent science
is of paramount importance to our environmental policies.  The quality of science that underlies
our regulations is vital to the credibility of EPA's decisions and ultimately the Agency's
effectiveness in protecting human health and the environment. One important way to ensure
decisions are based on defensible science is to have an open and transparent peer review process.

Consistent Agency-wide implementation of peer review has been an EPA priority for many
years.  Since issuing the Peer Review Policy in 1993, EPA has taken many steps to support and
strengthen the policy. In January 1998, EPA issued the  1st edition of the Peer Review Handbook
to provide guidance about peer review and peer review processes. The Handbook was updated
in December 2000 to reflect feedback from EPA's science community on improvements to the
usability of the Handbook.  The science community found that the Handbook contributed greatly
to the Agency goal of sound science and substantially improved EPA's peer review process.

In December 2004, the Office of Management and Budget (OMB) issued its "Final Information
Quality Bulletin for Peer Review" (See Appendix B). This OMB Bulletin establishes
government-wide guidance aimed at enhancing the practice of peer review of government
science documents. The Bulletin includes guidance for all Federal agencies on what information
should be subject to peer review, for selecting peer reviewers and for providing opportunities for
public participation. It also defines a peer review planning process to allow the public to
comment on peer review plans.  While incorporating the provisions of the OMB Bulletin, EPA
updated its Peer Review Policy to reiterate the Agency's commitment to peer review.  This
policy encourages and expects peer review of all scientific and technical information that is
intended to inform or support Agency decisions and notes that influential scientific information,
including highly influential scientific assessments, should be peer reviewed in accordance with
this Handbook.

-------
Peer Review Handbook                    Page xiv

-------
                              Peer Review Handbook
                                                         Page xv
                                 Acronyms
AA
AAS
ADP
ANPR
ASTM
BOSC
CBI
CCMP
CENR
CFR
CMM
CO
COI
COR
CREM
CTGs
DEO
DFO
DRA
EA
EC
ECOC
EEAC
EIMS
EIS
ELG
EMAP
EO
EPA
EPAAR
FACA
FAR
FIFRA
FIRMR
FOIA
FR
FTE
GSA
HHS
Assistant Administrator
Advisory and Assistance Services
Action Development Process
Advance Notice of Proposed Rulemaking
American Society of Testing and Materials
Board of Scientific Counselors
Confidential Business Information
Comprehensive Conservation  and Management Plan
Committee on the Environment and Natural Resources
Code of Federal Regulations
Contracts Management Manual
Contract(ing) Officer
Conflict of Interest
Contracting Officer Representative
Council for Regulatory Environmental Modeling
Control Techniques Guidelines
Deputy Ethics Official
Designated Federal Officer
Deputy Regional Administrator
Economic Analysis (previously known as Regulatory Impact Analysis)
Executive Council of the SAB
Ecological Chemicals of Concern
Environmental Economics Advisory Committee
Environmental Information Management System
Environmental Impact Statement
Effluent Limitation Guidelines
Environment Monitoring and Assessment Program
Executive Order
Environmental Protection Agency
EPA Acquisition Regulations
Federal Advisory Committee Act
Federal Acquisition Regulations
Federal Insecticide, Fungicide and Rodenticide Act
Federal Information Resource Management Regulation
Freedom of Information Act
Federal Register
Full Time Equivalent
General  Services Administration
Health and Human Services

-------
                               Peer Review Handbook
                                                          Page xvi
HISA
IDLH
IAG
IGA
IGF
IQGs
IRIS
ISC-COMPDEP
ISI
ISO
LAN
LOG
LOE
NAS
NEPA
NOAA
NODA
NPDES
NFS
NRC
NRMRL
NTTAA
CAM
GAP
OAQPS
OAR
OARM
OCHP
OEI
OGC
OGE
OIA
OIG
OIRA
OMB
ORC
ORD
ORIA
ORMI
OSA
OSP
Highly Influential Scientific Assessment
Immediately Dangerous to Life or Health Concentration
Interagency Agreement
Inherently Governmental Activities
Inherently Governmental Function
Information Quality Guidelines
Integrated Risk Information System
Industrial Source Complex - Complex Terrain Deposition
Influential Scientific Information
International Organization for Standardization
Local Area Network
Level of Concern
Level of Effort
National Academy of Sciences
National Environmental Policy Act
National Oceanic and Atmospheric Administration
Notice of Data Availability
National Pollutant Discharge Elimination System
National Park Service
National Research Council
National Risk Management Research Laboratory
National Technology Transfer and Advancement Act of 1995
Office of Acquisition Management
Office of Atmospheric Programs
Office of Air Quality Planning and Standards
Office of Air and Radiation
Office of Administration and Resources Management
Office of Children's Health Protection
Office of Environmental Information
Office of General Counsel
Office of Government Ethics
Office of International Affairs
Office of the Inspector General
OMB's Office of Information and Regulatory Affairs
White House Office of Management and Budget
Office of Regional Counsel
Office of Research and Development
Office of Radiation and Indoor Air
Office of Regulatory Management and Information
Office of the Science Advisor
Office of Science Policy

-------
                               Peer Review Handbook
                                                          Page xvii
OST
OSW
OTAQ
PAHs
PCB
PCDD
PCDF
PEP
PL
PMCR
PO
PR
PPA
PRA
PRAG
PRPT
Q&A
QRT
RA
RC
RCRA
RFA
RfC
RFC
RfD
RFR
RGB
RIA
RI/FS
ROD
RSAC
SAB
SAME
SAP
SBREFA
SERA
SGE
SI
SOPs
SOW
SPC
Office of Science and Technology
Office of Solid Waste
Office of Transportation and Air Quality
Polynuclear aromatic hydrocarbons
Polychlorinated biphenyl
Polychlorinated dibenzo(p)dioxin
Polychlorinated dibenzofuran
Peconic Estuary Program
Professional Level
Preliminary Model Calibration Report
Purchase Order
Procurement Request
Pollution Prevention Act
Paperwork Reduction Act
Peer Review Advisory Group
Peer Review Product Tracking Database
Question and Answer
Quick Response Task
Regional Administrator
Regional Counsel
Resource Conservation and Recovery Act
Regulatory Flexibility Act
Reference Concentration (Inhalation)
Request for Correction
Reference Dose (Oral)
Request for Reconsideration
Regular Government Employee
Regulatory Impact Analysis (now known as Economic Analysis)
Remedial Investigation/Feasibility Study
Record of Decision
Research Strategies Advisory Committee
Science Advisory Board
Simplified Acquisition Made Easy
Scientific Advisory Panel
Small Business Regulatory Fairness Act
Screening level Ecological Risk Assessment
Special Government Employee
Science Inventory
Standard Operating Procedures
Statement of Work
Science Policy Council

-------
                               Peer Review Handbook                      Page xviii
SPC-SC         Science Policy Council-Steering Committee
SRA            Science Review Administrator
STAR          Science To Achieve Results
TAMS          Toxics Air Monitoring System
TD             Technical Directive
TQM           Total Quality Management
UCL            Upper Confidence Limits
URL            Uniform Resource Locator
US             United States
USC            United States Code
USEPA         United States Environmental Protection Agency
USGS          United States Geological Survey
VSCF          Value of a Statistical Cancer Fatality
VSL            Value of Statistical Life
WAM          Work Assignment Manager
WMB          Waste Minimization Branch
WTI            Waste Technologies Industries

-------
                            Peer Review Handbook                    Page 1
                     Quick Start Guide To
                   The Peer Review Process
This Section provides flowcharts describing the major steps in conducting a peer review and
checklists to help plan a successful peer review. Cross references to the appropriate sections in
the Peer Review Handbook are shown in parentheses and bolded. The flowcharts and checklists
included are:

$           Figure 1 - Flowchart for Planning a Peer Review
$           Figure 2 - Flowchart for Conducting a Peer Review
$           Figure 3 - Flowchart for Completing a Peer Review
$           Manager's Planning Checklist for Peer Review
$           Regulatory Action Development Checklist for Workgroups

-------
                                         Peer Review Handbook
                                        Page 2
                          Figure 1 - Flowchart for Planning a Peer Review
1.   Determine if the work product:
        Is a scientific, engineering, economic,
        social science, or statistical document
        (2.2.1)
        Is influential scientific information (ISI) or
        a highly influential scientific assessment
        (HISA) (2.2.3, 2.2.4, 2.2.5)

2.   Peer Review typically needed for:
    -   ISI, including HISA (2.2.2)
        Other work products unless peer review is
        determined not to be warranted

3.   Peer Review typically not  needed if:
    -   ISI/HISA consists only of science
        previously peer reviewed and the review is
        deemed adequate under the Agency's
        Policy (2.3.2)
        meets criteria for exemption (2.3.1)
        receives waiver (2.3.3)
        otherwise determined not to be warranted

4.   If a work product is subject to peer review:
        Identify criteria/basis  for charge (3.2.1)
    -   Identify key staff (1.5)
        Create a peer review record (2.5)
        Ensure source of funding for the peer
        review (2.6)
        Determine/estimate overall time frame for
        peer review (3.3)
Identify Work Product
and Include in
Science Inventory
i

      Is the Work    \     No
   Product ISI/HISA?
 Would the Work
Product Still Warrant
  Peer Review?
    Is Peer Review    \Yes
   Exempt or Waived?
   Does Work Product
    Consist Only of
   Science Previously
    PeerReviewed?
 Work Product Should be Peer
        Reviewed

-Identify Basis for Charge
-Identify Key Staff
-Create Peer Review Record
-Ensure Availability of Funds
-Detemne Overall Time Frame
                                                         Continue with Conducting a Peer Review

-------
                                       Peer Review Handbook
                                PageS
                       Figure 2 - Flowchart for Conducting a Peer Review
1.   Develop the charge (3.2)
        Determine which key issues to address
        Include in peer review record (2.5.3)

2.   Select a peer review mechanism (2.4):
    -   Internal (2.4.2)
    -   External (2.4.3)
        Mail (e.g., Letter) review (2.4.3)
        Face to face meeting (2.4.3)
        One time or multiple meetings (2.4.6)
        Include logistical information in the peer review
        record (2.5.3)

3.   Determine the specific timeline(3.3):
        When will the review be started
        What are the intermediate check points
        What is the deadline for completion

4.   Select peer reviewers (3.4):
        Determine expertise needed (3.4.1, 3.4.4)
        Determine sources of peer reviewers (3.4.2)
        If ISI/HISA, consider asking public to nominate peer
        reviewers (3.4.2)
        Consider and address the balance of the panel (3.4.4)
        Consider conflicts of interest (3.4.5, 3.4.6)
        Include documentation in peer review record (2.5)

5.   Materials for the peer review (3.5):
    -   Obtain materials from Program for review
        Prepare instructions for peer reviews (3.5.1)
        Forward materials to peer reviewers (3.5.2)
        Include copy of materials in peer review record (2.5)

6.   Conduct the peer review:
        If ISI/HISA, ask reviewers to prepare peer review
        report (2.5.4)
    -   If HIS A, consider seeking public comment on
        work product and allowing the public to present to
        the peer reviewers  (1.4.2, 1.4.3, 1.5.3,2.4.7)
        If HIS A, and if seeking public comment, provide peer
        reviewers with significant public comments
        Include in peer review record (2.5)
    Work Product
(Ready for Peer Review)
  Develop the Charge
 Select the Peer Review
      Mechanism
   Set the Timelines,
  Including Deadlines
 Select Peer Reviewers
 Send Materials to Peer
 Reviewers & Conduct
       Review
 Complete Peer Review
 and Obtain Comments
   of Peer Reviewers
                              Add to Peer Review
                                   Record
     Continue With
   Completing a Peer
       Review
    Note: Some of these steps may occur concurrently.

-------
                                       Peer Review Handbook
                                         Page 4
                       Figure 3 - Flowchart for Completing a Peer Review
1.   Evaluate comments from peer reviewers
    (4.2.1)
        Consider and respond to comments
        Obtain clarification, if needed
        Prepare Agency response (4.3.1, 4.3.1.1)
    -   Include comments in peer review record

2.   Brief your Decision Maker; obtain written
    management approval of response to comments
    (1.5.3,4.2.1)

3.   Comments that are considered, but not used
        Document why not used (4.3.1)
        Include documentation in peer review
        record (2.5.3, 4.3.1)

4.   Comments that are used (2.5.3, 4.2, 4.3)
        Revise the work product by incorporating
        comments
        Send revised work product back to peer
        reviewers, if necessary

5.   Finalize work product (4.3.1)
        Include in peer review record (2.5.3,
        4.3.1)
        Post peer review report and related
        materials (e.g., charge, Agency response)
        on the internet (4.3.1)
        For ISI/HISA that support rulemaking
        - include peer review discussion and
        - certification in preamble to the rule (
        1.2.14,2.5.5,4.2.3)
Receive Comments from Peer
Reviewers
i
r
Brief Decision Maker,
if appropriate
i
r
         YES
                                         NO
Are Peer Reviewers
Comments Used in
  Work Product?
   Document How
   Used and Place
In Peer Review Record

    Incorporate
   Comments Into
   Work Product
                  Document Why Not Used
                  and Place in Peer Review
                        Record
                   Brief Decision Maker on
                   Approach to Handling
                       Comments
                  Finalize the Work Product
                Update the Peer Review Record
                                                                             Done

-------
                                Peer Review Handbook                      Page 5
                 Manager's Planning Checklist for Peer Review

1) Title of Work Product:	
2) What Decision/Rule/Regulation/Action Does this Work Product Support:
3) Designation of Scientific and Technical Work Products
G Is the work product scientific or technical	yes	no?
G Is the work product	influential scientific information (ISI),	highly influential scientific
   assessment (HISA), or	other? (See Section 2.2.3 and 2.2.4 of the Peer Review Handbook
   for an explanation of these terms).

4) Determining What Peer Review is Needed
G If ISI or HISA, peer review is needed.
G If not influential, is peer review still needed?
G What peer review mechanism is needed (internal and/or external)?
G When does the review need to be done?
G How much time will be needed to conduct/complete the review?
G Are there court ordered deadlines or other constraints?
G Has senior management (AA/RA/others) been informed of progress/problems?
G What would constitute success for this review?

5) Determining the Resources for Peer Review
G What is the priority of this project relative to other projects in the same office?
G What resources are needed to conduct the review?
G What are the impacts of the review on personnel?
G Who will lead the peer review?
G Who will conduct the peer review?
G Who will maintain the peer review record?
G Where will the peer review record be kept?
G What mechanism will be used for the peer review?
G Has the charge been developed?
G Has internal and external coordination been initiated/completed?
G Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes
    to the final work product) been made?
G How will results of the review be presented and addressed in the final work product (e.g., in a
    preamble, in an accompanying appendix - as well as changes in the work product itself)?
G Has the work product been entered into the Science Inventory?

6) Comments:	

-------
                                 Peer Review Handbook                      Page 6
           Regulatory Action Development Checklist for Workgroups

This checklist will help workgroups plan for peer review in the larger context of regulatory
development. Each numbered section corresponds to a time period in the regulatory
development process.

1. Peer Review Prior to Proposal

   Tier 1 or Tier 2 Rule1
   G Is the peer review schedule incorporated into the Analytic Blueprint?
   G Does this rule rely upon influential scientific information or a highly influential scientific
   assessment?
   G Will the work product be reviewed using external peer review?

   Tier 3 Rule
   G Is the peer review schedule incorporated in the plans (e.g., the analytic blueprint) for
   producing the action?
   G Does this rule rely upon influential scientific information or a highly influential scientific
   assessment?
   G If an internal mechanism will be used for peer review, is it acceptable according to Section
   2.4 of the Peer Review Handbook?

2. Sending a Proposed Rule Forward for the Administrator's Signature
   G Has peer review been completed?
   G Does the action memo indicate whether the rule relies upon influential scientific
   information or a highly influential scientific assessment?
   G If the proposed rule relies on influential scientific information or a highly influential
   scientific assessment, is there a discussion of the peer review in the preamble of the rule?

3. Before the Proposed Rule Publishes
   G Were the peer review report and any relevant materials included in the docket for this
   rulemaking?

4. Peer Review Prior to Finalization
   G Is a new peer review plan necessary as a result of new regulatory options?
        For further information on tiering and criteria used to determine the appropriate tier for an action, see
http://intranet.epa.gov/adplibrarv/adp/tiering/step2.htm

-------
                                 Peer Review Handbook                      Page 7
5. Sending a Final Rule Forward for the Administrator's Signature
G Has any new peer review been completed?
G Does the action memo indicate whether the rule relies upon influential scientific information or
a highly influential scientific assessment?
G If the final rule relies on influential scientific information or a highly influential scientific
assessment, is there a discussion of the peer review in the preamble of the rule?

6. Before the Final Rule Publishes
G Were the peer review report and any relevant materials included in the docket for this
rulemaking?

-------
Peer Review Handbook                   Page 8

-------
          Peer Review Handbook             Page 9
U.S. Environmental Protection Agency




    PEER REVIEW GUIDANCE

-------
Peer Review Handbook                   Page 10

-------
                                Peer Review Handbook                      Page 11
                      1.  The Need For Peer Review
1.1 Overview Statement

Peer review at the U.S. Environmental Protection Agency (EPA) takes many different forms
depending on the nature of the work product, relevant statutory requirements, and office-specific
policies and practices.  In January 1993, responding to recommendations in the report
Safeguarding the Future: Credible Science, Credible Decisions2, EPA issued an Agency-wide
policy for peer review. In 1994 EPA reaffirmed the central role of peer review and instituted an
Agency-wide implementation program. In 1998, a Peer Review Handbook was created as a
single,  centralized form of implementation guidance for Agency staff and managers.  The Peer
Review Handbook was revised and reissued in December 2000. The newly updated Peer
Review Policy and this 3rd Edition of the Handbook incorporate insights from the use of the 2nd
Edition as well as the provisions of the OMB Final Information Quality Bulletin for Peer Review
that was issued to help agencies enhance their peer review transparency and accountability.
Among the changes introduced by the OMB Bulletin is the use of the terms "influential scientific
information" and "highly influential scientific assessments" (see Sections 2.2.3 and 2.2.4 for
explanations of these terms).
1.2 Understanding Peer Review

       1.2.1  Why Use Peer Review?
Peer review is not free;
however, not doing peer
review can be costly.
Peer review is intended to uncover any technical problems
or unresolved issues in a preliminary (or draft) work product through the use of independent
experts. This information is then used to revise that draft product so that the final work product
will reflect sound technical information and analyses. Peer review is a process for enhancing a
scientific or technical work product so that the decision or position taken by the Agency, based
on that product, has a sound, credible basis.  To be most effective, peer review of a scientific
and/or technical work product should be incorporated into the up-front planning of any action
based on the work product - this includes obtaining the proper resource commitments (people
and money) and establishing realistic schedules.

Peer review of scientific and/or technical work products should not be looked upon as another
"hurdle" in the Agency decision making processes. Although conducting a peer review means
that time and resources have to be included in the  decision making process, the benefits justify
the added cost.  Peer review enhances the credibility and acceptance of the decision based on the
work product.  By ensuring a sound basis for decisions, greater cost savings are realized since
       2EPA/600/9-91/050, March 1992

-------
                                 Peer Review Handbook                      Page 12
decisions will not be challenged as often and extra effort will not be expended to go back and
redo the work product.  So while peer review is not free, the cost of not doing peer review is
usually much more expensive.  Furthermore, not conducting a peer review may potentially place
the Agency in the position of having to decide whether to try to defend a scientifically
questionable position - which can be very costly in terms of both resources, and, more
importantly, credibility.

       1.2.2  What is Peer Involvement?

Peer involvement is the process whereby Agency staff involve subject-matter experts from
outside their program in one or more aspects of the development of work products. Peer
involvement, therefore, constitutes active outreach to and participation by the broad scientific,
engineering, public health, economics and other social science communities beyond the Agency
(external), as well as within the Agency (internal).  Typically, peer involvement takes two
general forms: peer input (ongoing discussions during the development of the work product) and
peer review (an evaluation of a work plan, preliminary draft or the like, or the final objective
expert evaluation of the work product).

       1.2.3  What is Peer Review?

Peer review is a documented critical review of a specific Agency scientific and/or technical work
product. Peer review is conducted by qualified  individuals (or organizations) who are
independent of those who performed the work, and who are collectively equivalent in technical
expertise (i.e., peers) to those who performed the original work.  Peer review is conducted to
ensure that activities are technically supportable, competently performed, properly documented,
and consistent with established quality criteria.  Peer review is an in-depth assessment of the
assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance
criteria, and conclusions pertaining to the specific major scientific  and/or technical work product
and of the documentation that supports them. Peer review may provide an evaluation of a
subject where quantitative methods of analysis or measures of success are unavailable or
undefined such  as research and development. Peer review is usually characterized by a one-time
interaction or a  limited  number of interactions by independent peer reviewers.  Peer review is
encouraged during the early stages of the project or methods  selection, and/or as part of the
culmination of the work product, as appropriate. Regardless  of the timing of peer review, the
goal is ensuring that the final product is technically sound.

1.2.4  What is Peer Input?

Many Agency work products are developed with the input of various scientific and technical
experts inside and outside the Agency. Like the contribution made by peer reviewers, peer input
is valuable and  enhances the scientific or technical basis of the products. Peer input, sometimes
referred to as peer consultation, generally connotes an interaction during the development of an
evolving Agency work  product, providing an open exchange of data, insights, and ideas. Peer

-------
                                Peer Review Handbook                      Page 13
input may be characterized by a continued and iterative interaction with scientific experts during
work product development. A common example of peer input is the input received from
workgroup members during development of a product. Many Agency products are developed
through the efforts of a workgroup, which may include external experts, such as State and Tribal
representatives. These workgroup members have an active, ongoing participation in developing
the work product.  Another example of obtaining peer input is that of an Agency Office sending
a draft work product to a list of stakeholder representatives for general comments (stakeholder
representatives often include experts who could be considered "peers").

       1.2.5  How is Peer Review Different from Peer Input?

The key distinctions between peer input as described above and formal peer review are the
independence of the peer reviewers and their level of involvement. The goal of peer review is to
obtain an independent, third-party review of the product from experts who have not substantially
contributed to its development.  When experts have a material stake in the outcome of the peer
review (such as a regulated party) or have participated substantially in the development of the
product (such as a workgroup member), those experts'  reviews may not qualify as unbiased,
independent peer review and may be better characterized as peer input.
It is clear that peer input provides valuable contributions to the
development of the work product. However, peer input does
not substitute for peer review.  In other words, one cannot
argue that a peer review is not necessary if a work product has
received "enough" peer input.
   Peer Input is not a
substitute for Peer Review
       1.2.6  Can Someone Who Provided Peer Input Become an Independent Peer
             Reviewer for the Same Work Product Later in the Process?

Generally, the answer is no as that expert is no longer independent, but rather a contributor to the
work product. There may be special circumstances where the expertise is so narrow that another
peer reviewer is not available.  The Peer Review Leader (see Section  1.5.5) will normally be
responsible for making this determination and documenting the decision in the peer review
record.

       1.2.7  Can the Same Peer Reviewer be Used More Than Once if a Product Will Be
             Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used
             Again and Again for Different Products?

There is no prohibition on using the same peer reviewer more than once on the same product or
for multiple products of the same office.  However, it is preferable to  use different people each
time to provide a broader perspective.  When using a contractor to provide peer review services,
you should recognize that contractors may have a "pool" of reviewers that they use regularly. If
the same peer reviewers are used repeatedly, they may lose their independence (or the

-------
                                Peer Review Handbook                      Page 14
appearance of independence) from the work product(s). If a peer reviewer is asked to participate
in multiple reviews of the same product it should be noted in the peer review record.

       1.2.8  How is Peer Review Different from Public Comment?

Peer review and public comment are not the same. Public comment solicited from the general
public through the Federal Register or by other means is often required by the Administrative
Procedure Act, other relevant statutes or both. Public comment may also be solicited for policy
purposes.  The Agency takes public comment on some strictly scientific products and almost all
regulatory decisions. Public commenters usually include a broad array of people with an interest
in the technical analysis or the regulatory decision; some are scientific experts (who may provide
some peer input), some are experts in other areas, and some are  interested non-experts.  The
critical distinction is that public comment does not necessarily draw the kind of independent,
expert information and in-depth analyses expected from the peer review process. Public
comment is open to all issues, whereas the peer
review process is limited to consideration of
specified technical issues. While it may be an
important component of EPA's decision making
process, public comment does not substitute for peer
review.
Public comment does not substitute
for peer review.
       1.2.9  How is Peer Review Different from Stakeholder Involvement?

Stakeholder involvement occurs when the Agency works with external interest groups that have
some stake in or concerns about the outcome of the technical work product or regulatory
position. This is an interactive process which usually involves other agencies, industry groups,
regulated-community experts, environmental groups, other interest groups that represent a broad
spectrum of the regulated community, etc., and usually strives for a consensus approach.  The
goal of peer review, on the other hand, is to obtain an
independent, third-party review for ensuring scientific
quality and technical credibility of the work product that
supports a policy or decision.
     Stakeholder involvement is not
     a peer review mechanism.
       1.2.10 What Role Does Peer Review Have in
          Regulatory Development?

Peer review of scientific and/or technical work products that support regulations is an important,
fundamental step in the policy setting and regulatory development processes and affirms the
credibility of the Agency. Because the work products supporting regulations often are subject to
intense scrutiny by the general public and the stakeholders involved, you should plan ahead and
document the peer review process for such work products.

-------
                                 Peer Review Handbook                      Page 15
In general, peer review should be completed prior to issuance of the proposed regulation. In
some cases, work products that support final regulations may need an additional peer review if
those scientific and/or technical work products change significantly after the public comment
period.

A regulation itself is not subject to the Peer Review Policy.  However, if a regulation is
supported by influential scientific information or a highly influential scientific assessment, the
underlying work product should be peer reviewed before EPA issues the proposed regulation.
See Sections 2.2.3 and 2.2.4 for a more complete explanation of what is considered "influential
scientific information" or a "highly influential scientific assessment" and for guidance in
determining whether a work product contains this type of information. Remember, soliciting
comments from stakeholders and the general public after a rule is proposed does not constitute
peer review.

The original data and formal  analytic models used in Economic Analyses (EAs) are subject to
the OMB Bulletin if they constitute influential scientific information or highly influential
scientific assessments.  However, the straightforward application of data, models, and scientific
and economic  analyses used in previously peer-reviewed EAs and economic assessments or
other peer-reviewed products are not subject to additional formal peer review.

       1.2.11  How Does the Rulemaking Tiering Process Affect Peer Review?

Tier 1 and Tier 2 rulemakings are, by definition, important Agency rulemakings3. Therefore, you
should carefully scrutinize work products supporting Tier 1  and Tier 2 rules to determine
whether they should undergo peer review. In almost all cases you should use external peer
review for a work product that is intended to support a Tier  1 or Tier 2 rulemaking. Although
acceptable in certain circumstances for influential scientific information, the use of an internal
peer review mechanism for such work products should be avoided.

Work products supporting Tier 3 rulemakings may also benefit from peer review. For work
products supporting a  Tier 3 rule, internal or external peer review may be appropriate depending
on the nature of the product and other factors. You can find more information on the differences
between internal and external peer review in Section 2.4.

       1.2.12  Should You Discuss Peer Review in the Analytic Blueprint for Your
              Regulation?

Workgroups should specifically address peer review in  each analytic blueprint. For peer review
purposes, development of the analytic blueprint is the process whereby the workgroup identifies
        For further information on tiering and criteria used to determine the appropriate tier for an action, see
http://intranet.epa.gov/adplibrary/adp/tiering/step2.htm

-------
                                 Peer Review Handbook                      Page 16
the supporting scientific and/or technical work products and recommends what kind of peer
review is needed. The analytic blueprint should show the schedule of the peer review in the
context of the schedule for the overall rulemaking.

       1.2.13 What Role does Peer Review Have in Regulatory Negotiations?

Regulatory negotiations are not generally subject to peer review; however, to ensure final
decisions are based on sound and credible science, where possible, work products that support
the negotiation should be subjected to peer review before the negotiation takes place.

       1.2.14 Should You Address Peer  Review in the Preamble of a Regulation?

Generally, yes. For proposed and final regulations that rely on influential scientific information
or a highly influential scientific assessment, you should  discuss the peer review report in the
preamble, as described in the OMB Bulletin. If you are  not the rulewriter, take steps to ensure
the regulatory workgroup is aware of this provision of the OMB Bulletin.  See Appendix C -
Sound Science and Peer Review in Rulemaking, for template information.

       1.2.15 How is Peer Review Documented in the Action Memo for Regulations?

For all rules requiring the Administrator's signature (proposed and final), indicate in the action
memo what kind of peer review took place. See Appendix C for more information.

1.3.   Peer Review and Information Quality

       1.3.1  Overview Statement

EPA is committed to providing quality environmental information to its partners and the public.
This commitment is integral to its mission to protect human health and the environment. One of
its goals is that all parts  of society have access to accurate information sufficient to effectively
participate in managing  human health and environmental risks.  To fulfill this and other
important goals, EPA strives to rely  upon information of appropriate quality for each decision
made.  In early 2002, pursuant to the Information Quality Act4, the OMB developed government-
wide guidance for "ensuring and maximizing" the quality of information Federal agencies
disseminate to the public.5 In late 2002, EPA developed and issued its Information Quality
        The Information Quality Act, also known as the Data Quality Act, was issued by Congress, under Section
515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554;
H.R. 5658)

        Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information
Disseminated by Federal Agencies, OMB, 2002 (67 FR 8452).

-------
                                  Peer Review Handbook                       Page 17
Guidelines6 for ensuring the information that supports EPA's mission is reliable and accurate
(objectivity); is appropriate for the intended use (utility); and, is protected from compromise
(integrity). Peer review, when appropriate, enhances the quality of the information EPA
disseminates.

       1.3.2  How Does Peer Review Ensure the Quality of Information EPA
              Disseminates?

The Agency recognizes peer review as a component of pre-dissemination review that
complements and enhances the "objectivity" and "utility" of EPA's information products.  The
Agency recommends that offices conduct pre-dissemination reviews of information to ensure
that the information is of appropriate quality before it is disseminated to the public. Pre-
dissemination review is especially important for influential scientific information and highly
influential scientific assessments.

       1.3.3  How Do You Ensure the Quality of Data Supporting EPA's Information
              Products?

EPA's Quality System7 provides guidance on systematic planning for work products that involve
the collection of new environmental data or the use of existing environmental data. The
Agency's policies for planning for the appropriate level of quality assurance and peer review
ensure that the information the public receives is reliable and defensible.  Documentation of this
planning occurs at various stages in development of the final product:

    a)  A Quality Assurance Project Plan or similar documentation is developed and approved
       prior to the collection of new data, or use of existing data.  Systematic planning helps
       ensure that the data and information are of the right type, quality and quantity for the
       conclusions reached  in the final work product.

    b)  EPA's Action Development Process (ADP)8 is another mechanism that assists the
       Agency in achieving the  objectivity and transparency of information used in developing
        Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information
Disseminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002),
(http://www.epa.gov/qualitv/informationguidelines')

        EPA's Quality System provides guidance on systematic planning for the management, collection and use
of all environmental data to ensure the data are of the right type, quantity and quality for the intended use. EPA
Quality Manual for Environmental Programs 5360 Al. (May 2000), Section 1.3.1,
http://www.epa.gov/QUALITY/qs-docs/5360.pdf

       o
        EPA's Action Development Process: Guidance for EPA Staff on Developing Quality Actions, EPA June
2004, http://intranet.epa. gov/adplibrary/index.htm

-------
                                 Peer Review Handbook                       Page 18
       regulations, economic and policy decisions.  The stakeholder involvement and
       consultation that are part of the ADP enhance EPA's efforts to disseminate quality
       information that is useful to the public, and are a valuable component of pre-
       dissemination review.

   c)  The guidance entitled General Assessment Factors for Evaluating the Quality of
       Scientific and Technical Information9 is an additional resource for EPA's staff as they
       evaluate the quality and utility of information, regardless of the source. The document
       complements EPA's Information Quality Guidelines by raising the awareness of the
       information-generating public about EPA's ongoing interest in ensuring and enhancing
       the quality of information available for Agency use.

   d)  Peer review may be an important assessment tool. For example, peer review may be used
       by researchers to confirm the robustness of a theory or design before proceeding to the
       next level, such as field sampling or fabrication of new equipment. This is particularly
       helpful in "cutting edge" situations where technical standards or other measures of
       validation may not yet exist.

       1.3.4   Are Work Products Undergoing Peer Review Subject to the Guidelines?

Products that are undergoing peer review are not considered to be disseminated under EPA's
Information Quality Guidelines because they are dynamic documents that are subject to change
and therefore, do not represent EPA's final decision or position. These products  should contain
the following disclaimer:

       This information is distributed solely for the purpose ofpre-dissemination peer
       review  under applicable information quality guidelines. It has not been formally
       disseminated by EPA. It does not represent and should not be construed to
       represent any Agency determination or policy.

In cases where the information is highly relevant to specific policy or regulatory deliberations,
this disclaimer should appear on each page of the work product. Peer review products that are
disseminated are subject to EPA's Information Quality Guidelines.
        General Assessment Factors for Evaluating the Quality of Scientific and Technical Information, EPA
(June 2003), http://www.epa.gov/osa/spc/pdfs/assess2.pdf

-------
                                 Peer Review Handbook                      Page 19
1.4    Annual Agency Reporting

       1.4.1   What Are EPA's Reporting Practices and What Information Is Publicly
              Available?

EPA maintains a database, called the Science Inventory (http://cfpub.epa.gov/si), which is
designed to track and report science activities across the Agency, including their status and peer
review plans (See section 1.4.2).  Offices are expected to keep this information current; agenda
entries for the highly influential scientific assessments and influential scientific information
products should be updated at least every six months.  Real-time updates may occasionally be
necessary, for instance if there is  an imminent change in the timing for the peer review of a high
visibility study or a change in the timing of the public availability of a draft of a highly
influential assessment.

EPA will make its peer review plans for highly influential scientific assessments and influential
scientific information available for public comment through the Science Inventory.  Offices are
to consider any comments that are submitted on the peer review plans for which they are
responsible.  The OMB Bulletin also calls for final peer review reports for the highly influential
scientific assessments and influential scientific information to be developed and made available
to the public.  EPA will make these reports available on the Science Inventory web site, which
will directly link to the peer review agenda entry for that item.  Additional information on the
content of these reports is provided in Section 2.5.

The Deputy Administrator has designated the Office of Research and Development (ORD) to
conduct an annual review of the peer review plans.  As called for in OMB's Peer Review
Bulletin, ORD expects to submit  a report to OMB by December 15 of each year. This report will
include information concerning the peer reviews conducted on the highly influential scientific
assessments and influential scientific information during the previous fiscal year.  ORD will
generate this report from the information in the Science Inventory database.

       1.4.2   What Information Is Provided in the Science Inventory?

The Science Inventory database is designed to track and report peer review and other science
activities across the Agency. The database is the single repository for product-specific peer
review reporting and tracking and uses a common reporting form. Peer review work products
are divided into three categories:  highly influential scientific assessments, influential scientific
information, and other products.

Each AA/RA will designate a Peer Review Coordinator (See section 1.5.4) to ensure that their
science activities and peer review products are fully represented, accurate, and complete.
Specific information is included for each work product, including a summary or abstract of the
product, what the product will be used for, and whether the product will undergo peer review. If
the product will undergo peer review, additional information regarding the peer review plans is

-------
                                 Peer Review Handbook                      Page 20
provided.  If the activity or product is considered to be influential scientific information or a
highly influential scientific assessment, the following information should be provided to address
the provision of the OMB Bulletin that allows the public to view and comment on the Agency's
peer review plans for these activities or products:

   a)  A paragraph including the title, subject and purpose of the activity or product;

   b)  An Agency contact to whom inquiries may be directed to learn the specifics of the peer
       review plan;

   c)  Whether the dissemination is likely to be influential scientific information or a highly
       influential scientific assessment;

   d)  The timing of the review (including deferrals);

   e)  Whether the review will be conducted through a panel or individual letters (or whether an
       alternative procedure will be employed);

   f)  Whether there will be opportunities for the public to comment on the work product to be
       peer reviewed and, if so, how and when these opportunities will be provided;

   g)  Whether EPA will  provide significant and relevant public comments to the peer
       reviewers before they conduct their review;

   h)  The anticipated number of reviewers (3 or fewer; 4-10; or more than 10);

   i)  A succinct description of the primary disciplines or expertise needed in the review;

   j)  Whether reviewers will be selected by EPA or by a designated outside organization;

   k)  Whether the public, including scientific or professional societies, will be asked to
       nominate peer reviewers.

If the product will not undergo peer review, Offices are asked to provide a statement on why peer
review is not necessary (see Section 2.3 for further details).

Each record should be electronically signed by the Peer Review Coordinator.  By signing it, the
Peer Review Coordinator indicates that it is ready for ORD review and, in the case of highly
influential scientific assessments and influential scientific information, is ready to be made
available for public comment.

-------
                                Peer Review Handbook                      Page 21
       1.4.3  How Will the Public Be Allowed to Comment on the Peer Review Plans for
             Influential Scientific Information and Highly Influential Scientific
             Assessments?

The OMB Bulletin directs Federal agencies to make the peer review plans for influential
scientific information and highly influential scientific assessments publicly available for
comment. EPA will provide for this by making electronic files of the peer review plans available
on the Science Inventory.  The public can then comment on these plans and comments will be
directed to the peer review contact identified for the product. Offices are to consider the
comments submitted by the public on the peer review plans for highly influential scientific
assessments and influential scientific information, in accordance with the OMB Bulletin.

       1.4.4  What Is ORD's Role in Reviewing the Science Inventory and How  Will This
             Review Be Conducted?

ORD conducts an annual review of the peer review products in the database. The purpose of this
review is to ensure that the Science Inventory fully represents the Agency's peer review products
and to determine if the peer review decisions are consistent with the guidance presented in this
Handbook. ORD will consult, as necessary, with the appropriate persons in each organization
(see Section 1.5).  ORD will then consolidate the information and findings for the SPC and the
Deputy Administrator. Any conflicts arising from the review will be resolved by the Deputy
Administrator.

       1.4.5  How Will EPA Submit an Annual Report to OMB?

EPA expects to  submit an annual report to OMB by December 15 of each year that summarizes
the peer reviews that were conducted during the previous fiscal year for highly influential
scientific assessments and influential scientific information. Release of any reviewer
information retrieved by a personal identifier will be performed in accordance with the Privacy
Act, 5 USC  552a as amended, and as interpreted in OMB implementing guidance, 40 FR
28,948 (July 9, 1975).  According to the OMB Bulletin, the annual report should include the
following:

a)     The number of peer reviews conducted subj ect to the Bulletin

b)     The number of times alternative procedures were invoked

c)     The number of times waivers or deferrals were invoked (and in the case of deferrals, the
       length of time elapsed between the deferral and the peer review)

d)     Any decision to appoint a peer reviewer pursuant to any exception to the applicable
       independence or conflict of interest standards of the Bulletin

-------
                                Peer Review Handbook                     Page 22
e)     The number of peer review panels that were conducted in public and the number that
       allowed public comment

f)     The number of public comments provided on the peer review plans, and

g)     The number of peer reviewers used who were recommended by professional societies.

1.5    The Roles of People and Organizations in Peer Review

       1.5.1   Who is Ultimately Accountable for Peer Review?

Under the Peer Review Policy, the Administrator has designated the Assistant Administrators
and Regional Administrators (AAs and RAs) to be accountable for implementing the Policy in
their respective organizations. The Deputy Administrator is ultimately responsible for peer
review across the Agency and is the final arbitrator of conflicts and concerns about peer review.

       1.5.2   Who are the Agency Staff involved in Peer Review?

The principal Agency staff involved are Decision Makers (and their line managers), Peer Review
Leaders and Peer Review Coordinators. In addition, there are many other critical staff in each
Office and Region who have responsibility for peer review activities (e.g., Office and Division
Peer Review Coordinators, technical information managers, and,  of course, any Agency staff
who serve as internal peer reviewers).  Finally, ORD has oversight responsibility, as designated
by the Deputy  Administrator, for ensuring the Agency's Peer Review Policy is implemented.

       1.5.3   Who are the Decision Makers and What are Their Responsibilities?

The AA/RA is the ultimate Decision Maker for his/her organization and is accountable for the
decisions regarding the identification of influential scientific information and highly influential
scientific assessments and the mechanism(s) of peer review utilized for each of the products.
The AA/RA may designate Office Directors and/or Division Directors as the front-line Decision
Makers.

Furthermore, the Decision Makers  commit the resources needed to ensure a proper peer review.
Decision Makers are responsible for ensuring that the peer reviews are properly performed and
documented.

In order to ensure greater independence of peer reviews, it is important to strictly separate the
management of work products from the actual peer review of those work products. Therefore,
the Decision Maker and the Peer Review Leader for a work product should not be the same
person.

-------
                                 Peer Review Handbook                      Page 23
The Decision Maker needs appropriate training on how to manage the peer review process. The
Peer Review Coordinator for the Office can advise the Decision Maker on how to obtain the
appropriate training; the training approach varies among the different EPA offices.

Specific responsibilities of the Decision Maker(s) are these:

   a)  Determine which work products in their organization call for peer review and which ones
       are influential scientific information or highly influential scientific assessments;

   b)  Designate (in conjunction with the Project Manager) a Peer Review Leader to organize
       each peer review;

   c)  Provide advice, guidance, and support to the Peer Review Leader in the preparation,
       conduct, and completion of the peer review;

   d)  Ensure that sufficient funds are designated in the office's budget request to conduct the
       peer review; also ensure that adequate resources and/or extramural management support
       are available for the peer review;

   e)  Establish a realistic peer review schedule;

   f)  Designate the stage(s) of product development where peer review is appropriate;

   g)  Consider public comments on the peer review plans and consider and decide if the public
       will be asked to nominate potential peer reviewers for influential scientific information
       and for highly influential scientific assessments;

   h)  For highly influential scientific assessments, decide whether it is feasible and appropriate
       to make the draft scientific assessment available to the public for comment at the same
       time it is  submitted for peer review (or during the peer review process) and whether it is
       feasible and appropriate to sponsor a public meeting where oral presentations on
       scientific issues can be made to the peer reviewers by interested members of the public.
       When public comment on the assessment is sought, provide peer reviewers with
       significant comments raised by the public;

   i)  Ensure all relevant issues and comments raised by the peer reviewer(s) are adequately
       addressed and documented for the record, and where appropriate, incorporated into the
       work product that is used as basis for decision making.

       1.5.4   Who are the Peer Review Coordinators and What are Their Responsibilities?

The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor peer
review activities in his/her respective  Office or Region. This person should be of sufficient

-------
                                 Peer Review Handbook                      Page 24
stature and judgment to have the access to and confidence of all levels of Office or Regional
management. The Peer Review Coordinator is the main contact for his/her organization and can
also direct interested parties to other persons/contacts in the Office on specific work products
(e.g., Peer Review Leader).

Specific responsibilities of the Peer Review Coordinator are these:

   a)   General oversight responsibility for the Office's or Region's peer review process;

   b)   Report peer review activities to the AA/RA;

   c)   Help mediate difficult issues between the organization and others; if an issue cannot be
       resolved, then the Peer Review Coordinator can bring the issue to the attention of the
       appropriate level Decision Makers in each organization for resolution;

   d)   Function as the liaison with ORD and the Science Policy Council (SPC) to:

       1) Represent the  Office/Region before the SPC on peer review issues;

       2) Advise ORD of any changes in the list of work products and peer review mechanisms
          during the annual reporting, and when necessary, at other times;

       3) Participate in Agency peer review training, workshops, etc., as requested and
          disseminate this information to the organization; coordinate and/or present training
          within the organization.

   e)   Submit information on the organization's peer review plans and activities as needed to
       implement the Peer Review Policy (See Section 1.4). On a product specific basis, this
       data submission responsibility may be re-designated within the organization:

       1) Assure that all Science Inventory records  for peer review products are accurate and
          comply with OMB reporting requirements (see Section 1.4.2);

       2) Assure the Science Inventory record addresses all issues and reporting requirements
          for the annual report to OMB and sign the record, making it available to the public
          via the Science Inventory website.

   f)   Establish procedures to ensure that the work product peer review documentation (i.e.,
       peer review record) is filed and maintained in an appropriate manner (see Section 2.5);

   g)   Provide advice, guidance, and support to the various Peer Review Leaders for the
       performance of the peer reviews;

-------
                                Peer Review Handbook                      Page 25
   h)   Distribute Agency-wide peer review guidance and materials to appropriate Office/Region
       personnel, as requested.

       1.5.5  Who are the Peer Review Leaders and What are Their Responsibilities?

The Peer Review Leader is assigned by the Decision Maker to organize, conduct and complete
the peer review for a specific individual work product. The Peer Review Leader will obtain the
assistance and support of the Peer Review Coordinator as well as any others within the Agency
to help perform the peer review. The Peer Review Leader will be chosen on a case by case basis
depending on the work product needing peer review.  The Peer Review Leader cannot be the
Decision Maker.  The Peer Review Leader could be the Project Manager for the work product.

The Peer Review Leader should have appropriate training on how to conduct a peer review
before conducting the peer review.  The Peer Review Coordinator for the Office can advise the
Peer Review Leader on how to obtain the appropriate training; the training approach varies
among the different EPA offices, however, uniform training modules are available.

Specific responsibilities of the Peer  Review Leader are:

   a)  Keep the  Decision Maker informed of the status of a given project; provide the Peer
       Review Coordinator with data for the annual report;

   b)  Organize, conduct, and complete the peer review following Agency procedures:

          1) Establish and maintain the peer review record for the specific individual peer
             review currently being performed (see Section 2.5); this includes providing the
             peer review summary information in the Science Inventory for the Peer Review
             Coordinator to sign when the peer review is completed;

          2) Select the peer reviewers in consultation with others involved with the peer
             review (e.g., Decision Maker) and ensure that conflict of interest issues are
             addressed and documented in the peer review record (see Section  3.4.6);

          3) Advise peer reviewers of their responsibilities, including preparation of a peer
             review report if the product is influential scientific information or a highly
             influential scientific assessment.

   c)  Provide information to the Decision Maker (including all appropriate managers in the
       Peer Review Leader's chain  of command) on the charge, profile of peer reviewers, the
       peer review comments, and a proposal on how to address the comments.  Obtain Decision
       Maker approval on the approach to responding to peer reviewer comments.  Clearly
       identify any peer review comments for the Decision Maker that will not be addressed in

-------
                                 Peer Review Handbook                      Page 26
       the agreed upon approach. (See Section 2.5.3 for specific contents of the peer review
       record):

          1)  Make the peer review report for influential scientific information and highly
              influential scientific assessment publicly available;

          2)  For highly influential scientific assessments, develop the Agency response to the
              peer review and make it publicly available.

   d)  Notify the Peer Review Coordinator that the peer review is completed for the annual
       report;

   e)  Archive the peer review record in a manner consistent with the organization's archiving
       procedures.

When a contractor is used to conduct a peer review, some of the above responsibilities are
assumed by the contractor (see Section 3.6). In addition, when peer reviews are completed
through a Federal Advisory Committee, some of the above responsibilities are assumed by the
Designated Federal Officer (DFO).

       1.5.6   Who are the Peer Reviewers?

Peer reviewers are individuals who have technical expertise in the subject matter of the work
product undergoing peer review. Peer reviewers can come from EPA, another Federal agency,
or from outside of the Federal government.

       1.5.7   What are the Responsibilities of Peer Reviewers?

Peer reviewers need to be willing participants in the peer review process - they should agree to
read all materials, participate fully, act ethically, and protect confidential information that arises.
Peer reviewers should maintain the confidentiality of the product, perform the review in a timely
manner, and be unbiased and objective.  Peer Reviewers should not be given any products or
data that may reveal confidential business information (CBI) unless the peer reviewer is internal
to EPA and has the appropriate clearance (see Section 3.4.7).

       1.5.8   What is an  Independent Peer Reviewer?

An independent peer reviewer is an expert who was not associated with the generation of the
specific work product either directly by  substantial contribution to its development or indirectly
by significant consultation  during the development of
the specific product. The independent peer reviewer,
thus, is expected to be objective (See Sections 1.2.6
and 1.2.7 for further information).
The quality of the peer review is
dependent on the competence and
independence of the reviewers.

-------
                                 Peer Review Handbook                       Page 27
Independence is freedom from institutional or ideological bias regarding the issues under review
and is necessary for objective, fair, and responsible evaluation of the work product. If a selected
reviewer has a particular scientific or technical perspective, it may be desirable to balance the
review with peer reviewers with other perspectives.  In addition to being independent, ideally,
peer reviewers should be free of real or perceived conflicts-of-interest.  Conflict of interest is
different from independence and is discussed in Section 3.4.5 and 3.4.6. If there are potential
conflicts of interest (real or perceived), they should be fully understood and addressed by the
peer review leader (or other appropriate official) to ensure a credible peer review.

        1.5.9    When does an Agency Internal Peer Reviewer Qualify as Independent?

An Agency independent internal peer reviewer is one who comes from a different organizational
unit than the one where the review question or document originates.  A different organizational
unit usually denotes, at minimum, a different Office (i.e.,  above Division level in programs;
above Branch level in Regions) within the organization.  In particular, a reviewer should not
come from within the chain of command, either upward or downward.  Agency staff may not
serve as peer  reviewers for the final peer review of a highly influential scientific assessment
unless a specific exception is granted under Section III.3.C of the OMB Bulletin.
         1.5.10   What is a Peer Review Panel?

A peer review panel can range from a few individuals to ten or more, depending on the issue
being investigated, the time available and any limitations on resources. Individuals who serve as
peer reviewers should have appropriate scientific and technical expertise such that the review
panel covers the broad spectrum of expertise necessary to address the issues/questions presented
in the charge.

       1.5.11    What is a Subject Matter Expert?

A subject matter expert is one who has specific scientific and technical expertise in the matter
under review. The importance of scientific and technical expertise in the subject matter is
obvious, however, knowledge or just "knowing" about the subject area is not equivalent to
expertise in the  subject matter.  For Agency decisions, a multi-disciplinary group of experts
corresponding to the disciplines that contribute to complex Agency decisions is often necessary
for a full and complete peer review. For example, a risk assessment that relies on both animal
and human data usually would call for experts in both areas for a complete review.  For
economic analyses, experts from the economic disciplines are necessary.

         1.5.12  What is the Role of the Science Policy Council (SPC)?

According to the Peer Review Policy statement: "The Science Policy Council is responsible for
overseeing Agency-wide implementation of this policy, including: promoting consistent

-------
                                Peer Review Handbook                     Page 28
interpretation; assessing Agency-wide progress; developing recommendations for revisions of
the policy as necessary; and issuing the Peer Review Handbook., which provides additional
information and procedures on implementing this policy." The SPC meets its responsibilities
through coordination with the Peer Review Coordinators, the Peer Review Advisory Group
(PRAG) and the Office of Research and Development (ORD).

The SPC, PRAG and ORD are not responsible for identifying specific products for peer review
or determining the level of review or mechanism for that review; those functions are the
responsibility of management within each Office or Region.

        1.5.13   What is the Role of the Peer Review Advisory Group (PRAG)?

The Science Policy Council has created the PRAG to assist in the implementation of the
Agency's Peer Review Policy.  The primary role of the PRAG is to provide interpretation of the
policy and to assist the SPC and Agency Offices and Regions in preparing updates to the Peer
Review Handbook.

        1.5.14  What is the Role of the Office of Research and Development (ORD)?

The Deputy Administrator has designated the  Office of Research and Development (ORD) to
provide oversight to the Program Offices and Regions in the collection and review of
information that is contained in the  Science Inventory (for detailed information, see Section 1.4).

-------
                                 Peer Review Handbook                       Page 29
                          2.  Planning A Peer Review
2.1    Overview Statement

Planning a peer review is a critical first step to ensuring a successful peer review of a work
product.  The initial step is to determine whether your work product should be peer reviewed.
Once you have determined that a peer review will be conducted, the Decision Maker and Peer
Review Leader need to plan an appropriate review.  This includes the determination of resources
(budget and personnel), schedule for the completion of the peer review, mechanism for peer
review, development of the peer review record, and, finally, the selection of peer reviewers.

2.2    Determining Which Work Products to Peer Review

       2.2.1   What are Scientific and/or Technical Work Products?

The first  step in determining which work products should be peer reviewed is to identify
products  that are scientific and/or technical in nature.  The term scientific and/or technical work
products  is generally consistent with the term "scientific information" in the OMB Bulletin10.
Scientific and/or technical work products are used to support a research agenda, regulatory
program, policy position or other Agency position or action. Scientific and/or technical work
products  include economic and social science work products.  Categories of work products
include, for example: risk assessments,  technical studies and guidance, analytical methods,
scientific database designs, technical models, technical protocols, statistical survey/studies,
technical background materials, technical guidance  (except for guidance providing  policy
decisions), research plans, and research strategies (re journal articles, see section 2.3.4).

Products  that would not be considered scientific and/or technical work products can include
those: that address procedural matters (e.g., planning, reporting, coordination, notification); that
are primarily policy statements (e.g., relocation policy); that are conference proceedings (unless
the proceedings are used as the scientific basis for an Agency action or decision); or that are
decision  documents, e.g., Environmental Impact Statement (EIS), Record of Decision (ROD),  or
Economic Analysis (EA).  For any of these, the decision document itself is not subject to the
Peer Review Policy, but its underlying scientific and/or technical models, data, and/or work
products  upon which these documents are based are candidates for peer review. In  addition, the
following Agency documents are not considered scientific and/or technical work products under
the Peer Review Policy: strategic plans, analytic blueprints, and goals documents.
        OMB defines "scientific information" as "factual inputs, data, models, analyses, technical information, or
scientific assessments based on the behavioral and social sciences, public health and medical sciences, life and earth
sciences, engineering, or physical sciences.. ." (OMB Bulletin, Section 1.5.)

-------
                                 Peer Review Handbook                       Page 30
Making final determinations concerning which work products are scientific and/or technical is a
responsibility of the Decision Maker (See Section 1.5.3). All scientific and/or technical work
products should be entered into the peer review component of the Science Inventory.

       2.2.2   What Scientific and/or Technical Work Products Need Peer Review?

The principle underlying the Peer Review Policy is
that all influential scientific and technical work
products used in decision making will be peer
reviewed. The process for identifying which
products are "influential" and then determining the
mechanism of review will take into account various
criteria and the circumstances surrounding the use of
the work product. To maintain flexibility,  the Decision Maker(s) for peer review should consider
the full field of possible work products to identify those additional products that might still
warrant peer  review as well as the full spectrum of peer review mechanisms for each product.
When in doubt about whether a
work product merits peer review,
decide to peer review it.
OMB defines influential to mean information that an agency reasonably can determine will have
or does have a clear and substantial impact on important public policies or private sector
decisions. EPA has provided further guidance on making determinations with respect to whether
a document is "influential" in its Information Quality Guidelines.11  Once a product has been
identified as being influential scientific information, the Decision Maker should determine
whether it meets OMB's definition of a "highly influential scientific assessment" (see Section
2.2.4). The OMB Bulletin calls for additional peer review procedures for highly influential
scientific assessments.

       2.2.3   How Does  One Determine Whether a Scientific and/or Technical Work
              Product is  Influential Scientific Information?

Generally, determinations  whether a scientific and/or technical work product is "influential" will
occur on a case-by-case basis. The continuum of work products covers the range from the
obviously influential, which clearly need peer review, to those products which clearly are not
influential and don't need  peer review.  The majority of EPA's work products fall in-between
those two ends  of the continuum and should be closely evaluated and assessed with respect to
certain factors (see below). The Decision Maker should make the judgment as to whether a work
product is influential scientific information. There is no easy,  single "yes/no" test that applies to
the whole continuum of work products for determining whether a work product is influential
  Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information
Disseminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002),
(http://www.epa.gov/qualitv/informationguidelines')

-------
                                 Peer Review Handbook                      Page 31
scientific information. A useful rule of thumb is that if there is any doubt about whether a work
product needs peer review, go ahead and subject it to peer review.

Generally, scientific and/or technical work products that are used to support a regulatory
program or policy position and that meet one or more of the following factors would be
considered to be influential scientific information:

   a)  Establishes a significant precedent, model, or methodology;

   b)  Likely to have an annual effect on the economy of $100 million or more, or adversely
      affect in a material way the economy, a sector of the economy, productivity, competition,
      jobs, the environment, public health or safety, or State, Tribal, or Local governments or
      communities;

   c)  Addresses significant controversial issues;

   d)  Focuses on significant emerging issues;

   e)  Has significant cross-Agency/interagency implications;

   f)  Involves a significant investment of Agency resources;

   g)  Considers an innovative approach for a previously defined problem/process/methodology;

   h)  Satisfies a statutory or other legal mandate for peer review.

Historically, EPA has defined products subject to peer review in terms of having "major impact."
"Major impact" means that the work product may have applicability to a broad spectrum of
regulated entities and other stakeholders, or that it will have narrower applicability, but with
significant consequences on a smaller geographic or practical scale. The Agency developed the
previously listed factors to consider when determining if a product had a "major impact." The
Agency has also linked its use of the term "influential" to the term "major" in its Information
Quality Guidelines. The OMB Peer Review Bulletin provides similar factors to use in
determining if a scientific assessment is highly influential, thereby creating some overlap
between the two definitions.  Scientific assessments that meet the criteria of both influential and
highly influential should be considered as highly influential.

Typically, an influential scientific and/or technical work product that has a reasonable likelihood
of supporting a regulatory decision  or policy/guidance of major impact may also have a clear and
substantial impact on private sector decisions. The scientific and/or technical work products that
underlie many of the Agency's major rulemakings, policy and guidance documents of general
applicability would be designated "influential" under this construct because of their far-reaching
impacts.

-------
                                  Peer Review Handbook
                                    Page 32
The novelty or controversy associated with the work product may determine whether it is
influential scientific information.  Influential scientific information may be novel or innovative,
precedential, controversial, or emerging ("cutting edge"). An application of an existing,
adequately peer-reviewed methodology or model to a situation that departs significantly from the
situation it was originally designed to address may make peer review appropriate.  Similarly, a
modification of an existing, adequately peer-reviewed methodology or model that departs
significantly from its original approach may also make peer review appropriate. Determining
what constitutes a "significant departure" as used in this Section is the responsibility of the
Decision Maker.

In summary, an influential scientific or technical work product has a major impact, involves
precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory
obligation to conduct a peer review.

       2.2.4  How Does One Determine Whether Influential Scientific Information is a
              Highly Influential Scientific Assessment?

Once a scientific and/or technical work product has been categorized as "influential scientific
information," the Decision Maker should determine whether the product meets OMB's definition
of a highly influential scientific assessment (see also Figure 4). As with the categorization of a
work product as influential  scientific information, the decision whether or not to elevate a work
product to the highly influential category occurs on a case-by-case basis and is largely  based on
evaluation of the same factors as identified above.
Note that "highly influential scientific
assessments" differ from "influential
scientific information" both in terms of
their degree of influence and in
substance.  OMB defines highly
influential scientific assessments as
influential scientific information that
"the agency or the Administrator
determines to be a scientific assessment
that:

   a)   Could have a potential impact of
       more than $500 million in any
       year, or

   b)   Is novel, controversial, or
       precedent-setting or has
       significant interagency interest."
             Influential Scientific Information
               Highly Influential Scientific
             Figure 4-EPA Work Products
This diagram depicts the relation between all EPA work
  products and those considered Influential Scientific
Information or Highly Influential Scientific Assessments

-------
                                  Peer Review Handbook                      Page 33
OMB defines a scientific assessment as "an evaluation of a body of scientific or technical
knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions,
and/or applies best professional judgment to bridge uncertainties in the available information"12.

The OMB Bulletin includes the following examples of assessments that could be considered
highly influential scientific assessments:

a)     State-of-science reports;

b)     Technology assessments;

c)     Weight-of-evidence analyses;

d)     Meta-analyses;

e)     Health, safety, or ecological risk assessments;

f)     Toxicological characterizations of substances;

g)     Integrated assessment models;

h)     Hazard determinations;

i)     Exposure assessments.

OMB's definition  of a highly influential scientific assessment overlaps significantly with the
factors you should consider when assessing whether a work product is influential scientific
information. The decision to categorize a product as a highly influential scientific assessment
first depends on determining whether the influential scientific information is a "scientific
assessment," and then on assessing the likely degree of the product's impact.

The more far-reaching or significant the impacts of a scientific assessment are, the more
appropriate it is to categorize the product as a highly influential scientific assessment. The
determination  as to whether the degree of novelty, precedent, and controversy associated with a
particular assessment warrants designation as a highly influential scientific assessment should be
made by the Decision Maker. If the influential scientific assessment involves significant issues
that are truly "cutting edge," it may be appropriate to designate it as a highly influential scientific
assessment. The greater the "significant departure," as discussed above, the more appropriate it
is to include the product as a highly influential scientific assessment.  These are subjective
determinations that should be made by the Decision Maker.
       12
         OMB Bulletin, Section 1.7.

-------
                                 Peer Review Handbook                       Page 34
OMB also directs agencies to consider the degree of potential economic impact of a scientific
assessment in deciding whether it should be categorized as a highly influential scientific
assessment. As explained above, OMB  defines highly influential scientific assessments as those
scientific assessments that may have an  impact of more than $500 million in any year.

       2.2.5   What Economic Work Products Need Peer Review?

Economic analyses (EA) and reports are considered scientific and/or technical work products,
and as such it may be appropriate to peer review them.  If those work products are ISIs, then they
should be peer reviewed if they have not already been subject to adequate peer review (See
section 2.2.3). The following economic work products, should generally be peer reviewed.

   a)  Internal Agency guidance for conducting economic and financial analysis;

   b)  Economic and financial methodologies that will serve as a principal method or protocol
       used to conduct economic analyses within a program;

   c)  Unique or novel applications of existing economic and financial methodologies,
       particularly those that are recognized to be outside of mainstream economic practices;

   d)  Broad-scale economic assessments of regulatory programs, such as those required by
       Congressional mandates (e.g., the Clean Air Act reports to Congress on benefits and
       costs);

   e)  Stated preference (e.g.,  contingent valuation) and revealed preference surveys (e.g.,
       recreational travel cost surveys)  developed to assist in the economic analysis of a
       regulation or program;

   f)  National surveys of costs and expenditures for environmental protection (e.g., financial
       needs surveys, pollution abatement expenditures surveys);

   g)  Economic research plans developed to assess and advance the state-of-science in
       economic theory, methodologies or modeling (in particular, the technical feasibility of
       the plan's components);

   h)  Meta-analyses (i.e., re-analyses of existing published literature and supporting data on the
       measurement of economic benefits, costs and impacts);

   i)  Data and analytical models underlying economic analyses, particularly those supporting
       economically significant rules, if the models and corresponding use of the data have not
       been previously subject to adequate peer review.

-------
                                Peer Review Handbook                      Page 35
If these types of economic work products support a highly influential scientific assessment, you
should conduct an external peer review.  External peer reviews can be provided by the Science
Advisory Board's Environmental Economics Advisory Committee, other appropriate outside
organizations, or individuals that have expertise in the technical economic issues raised in the
economic work product.

       2.2.6   Should Economic Work Products Prepared in Support of Regulations that
              are Classified as "Major" or "Economically Significant" be Peer Reviewed?

Generally, if an EA applies accepted, previously peer-reviewed methods in a straightforward
manner, it would not undergo an additional peer review. EAs prepared to support "major" or
"economically significant" regulations (these terms are defined at the end of this Section)
typically do not utilize innovative or untried economic methods.  It is unnecessary to conduct
peer reviews of straightforward applications or transfers of accepted, previously peer-reviewed
economic methods or analyses. The procedures used to transfer or adapt an economic work
product are generally established by separate economic guidance documents which have been
peer reviewed. Therefore, EAs that are developed using these procedures do not normally
undergo an additional peer review,  even those prepared in support of "major" and "economically
significant" rules. To the extent that the underlying data and models are considered "influential
scientific information," they are subject to peer review, including questions regarding the
applicability of the specific data set or model for the use.

Even when peer review is not needed, additional peer input can be beneficial in the development
of economic work products for "major" and "economically significant"  rules. At present, some
peer input of these analyses is already likely to be included as part of the regulatory development
process, including input received from other EPA offices represented on the workgroup for the
rule, from the Agency's Regulatory Steering Committee, and from the public as part of the
public comment process for the rule. But there may be added benefit to employing additional
peer input procedures, such as actively soliciting input from economists in other Federal
Agencies on the quality and completeness of the economic analysis.

Under Section 3(f)(l) of Executive Order 12866, "economically significant" rules are those that
may have an annual  effect on the economy of $100 million or more or adversely affect in a
material way the economy, a sector of the economy, productivity, competition, jobs, the
environment, public health or safety, or State, Local, or Tribal governments or communities.
The term "major," as defined in the Congressional Review Act, means a rule that has resulted in
or is likely to result in: an annual effect on the economy of $100 million or more; a major
increase in costs or prices for consumers,  individual industries, Federal, State, or Local
government agencies, or geographic regions; or significant adverse effects on competition,
employment, investment,  productivity, innovation, or on the ability of United States-based
enterprises to compete with foreign-based enterprises in domestic and export markets.

-------
                                Peer Review Handbook                      Page 36
       2.2.7  What Other Economic Work Products Might Benefit from Peer Review?

There may be other economic work products not covered in the preceding sections for which
peer review might be useful.  Examples of such work products are presented below; however, by
listing these examples, there is no intent to establish a presumption of peer review for these work
products.

    a)  Analyses measuring the economic impacts and effectiveness of adopting market-based or
       economic incentives as regulatory management instruments;

    b)  The technical analyses supporting economic policies established under other government
       organizations (e.g., economic models used to study transportation, economic
       development, and international trade policies).

Most of these types of economic work products do not exhibit the degree of complexity, or
establish an innovative or untried approach, that would warrant a peer review.  However, other
factors, such as the potential significance of the analysis for cross-Agency or interagency
practices, or the significance of the issues addressed, may make peer review desirable.

       2.2.8  What Other Social Science Work Products Need Peer Review?

Typically, a social science work product is one that includes empirical, logic-based approaches
to answer technical questions about human motivation, human behavior, social interactions, and
social processes, which are relevant to the environmental issues being addressed.  The term
"behavior" includes  overt actions; underlying psychological processes such as cognition,
emotion, temperament and motivation; and bio-behavioral interactions.  The term "social"
includes sociocultural, socioeconomic,  and sociodemographic status; biosocial interactions; and
the  various levels of social context from small groups to complex cultural systems. Examples of
social science work products include analyses and/or evaluations related to such topics as
pollution prevention, risk communication,  environmental information, environmental justice,
quality of life, decision-making, and public participation.

The following social science work products should normally undergo peer review:

    a)  Internal Agency guidance for conducting social impact assessments and other
       community cultural assessments related to different environmental protection approaches
       such as community-based watershed protection (heretofore referred to as social
       assessments);

    b)  New social science methodologies that will serve as a principal method or protocol used
       to conduct social assessments;

-------
                                 Peer Review Handbook                      Page 37
   c)  Unique or novel applications of existing social science methods such as surveys, focus
       groups, interviews, network analyses, comparative analyses, and content analyses;

   d)  New national surveys of values, perceptions and preferences related to environmental
       protection;

   e)  Innovative research or analyses that address the human dimensions of environmental
       protection or environmental change in terms of social trends, future predictions and/or
       behavioral generalizations;

   f)  Social science research plans developed to assess and advance the state-of-science in
       social science theory, methodologies or modeling (in particular, the technical feasibility
       of the plan's components).

       2.2.9  How Should Peer Review be Handled for Products Developed Under an
             Interagency Agreement (IAG)?

Under an interagency agreement (IAG), EPA provides funds to another agency for that agency to
use for a specific purpose.  The receiving agency's guidance for peer review may likely be
different from EPA Peer Review Policy, although the OMB Bulletin establishes some minimal
common guidance for the Federal government. Regardless, if EPA plans to use any work
products from that agreement, EPA should decide whether those documents need review under
EPA Peer Review Policy.

       2.2.10 Should Products from Contracts, Grants, and Cooperative Agreements
             Receive Peer Review?

If there is a scientific and/or technical work product resulting from a grant, contract, or
cooperative agreement and it is considered influential scientific information or a highly
influential scientific assessment and it will likely  be used in Agency decision-making, the work
product needs peer review. Since it would probably result in a perceived, if not real, conflict of
interest, the group that is generating the work product usually cannot conduct or perform the peer
review of its own work product. Exceptions may be made in certain instances for organizations
that have adequate and well established recognized procedures for peer review, such as the
National Academy of Sciences  (NAS). In practice,  the Agency may need to peer review the
product on its own, or arrange with an independent third group (e.g., via another extramural
vehicle) to conduct the peer review. The Agency should not use scientific and/or technical work
products from contracts, grants, or cooperative agreements to support decision making unless the
work products are peer reviewed for both scientific and technical rigor and applicability to the
specific use to be made of the product.

Be aware that contracts are very different from grants and cooperative agreements. Please  note
that there are important legal restrictions on the direct use of work products developed under

-------
                                 Peer Review Handbook                      Page 38
grants and cooperative agreements in the Agency's decision-making process (see
http://intranet.epa.gov/ogd/) for additional information).

       2.2.11  How Does Peer Review Apply to Products Generated Through EPA Grants
              or Cooperative Agreements?

It may be desirable to peer review scientific and/or technical work products that are generated
through EPA grants or cooperative agreements, but special considerations apply.

EPA may legally provide financial assistance for research that is intended to stimulate or support
development of scientific knowledge that is not primarily for EPA's direct use or benefit.  The
resulting work products might be widely disseminated either through publication in scientific
journals or through other means, as opposed to a report tailored to EPA's specific needs and
requirements.  EPA can consider these work products just as it can review other published
scientific works when formulating its programs and policies.  Further, EPA retains a royalty free,
nonexclusive and irrevocable right to use the work products for Federal purposes, even if the
recipient has copyrighted the material (see 40 CFR 30.36(a)).

EPA may determine that the recipient's work product is influential scientific information
because: 1) it will be used to support an EPA program or policy position, and 2) it meets one or
more of the seven factors outlined in Section 2.2.3. In this situation, the work product should be
peer reviewed.

The following are options for peer reviewing the product:

   a)   EPA can have the product peer reviewed with the participation of the assistance
       agreement recipient/author. In this case, EPA could arrange for an independent peer
       review of the product itself, or may contract with a third party to conduct the review.
       EPA would also enter into a contract with the author  (formerly the recipient), which
       would task the author to prepare a response to the peer reviewers' comments, and to
       revise or prepare an addendum to the product in response to peer reviewer comments as
       determined  appropriate by EPA.

       A caveat to this approach is that it may  be difficult to get the recipient/author to agree to
       allow EPA to determine how to revise the product  in  response to the peer review
       comments.

   b)   EPA can have the product peer reviewed without the  participation of the recipient/author.
       EPA could arrange for the peer review itself,  or could contract with a third party to
       conduct the review.  In this case, however, the work product would not be revised to
       incorporate the peer review comments.  Instead, EPA would receive the comments and
       prepare a statement that documents EPA's own response to the comments.  The EPA
       Decision Maker who is using the work product to support an EPA program or policy

-------
                                Peer Review Handbook                      Page 39
       decision should be provided information on both the conclusions of the recipient's work
       product and EPA's own conclusions from the peer review.

   c)   Recipients can get their products peer reviewed on their own.  Recipients may determine
       on their own that peer review would benefit the credibility of their product. Provided
       EPA agrees that a peer review would further the public purpose of the assistance
       agreement, EPA may include funds for the peer review in the agreement. (See 2.2.10 for
       additional information.) Alternatively, the recipient may make arrangements for, and
       fund, an independent peer review of their product. In either case, EPA would need to
       evaluate whether the peer review process undertaken by the recipient was acceptable for
       the purposes for which EPA was planning to use the work product.  EPA may accept the
       peer review if it determines that it is of appropriate quality and that EPA could defend the
       peer review as if it were conducted by  EPA itself.

Under options a) or b), issues may arise over obtaining access for peer reviewers to the raw data
used by the recipient to generate the work product. Under 40 CFR 30.36(c)(2), EPA has a right
to obtain raw data produced by a non-profit organization or university under an assistance
agreement, even where the agreement does not specifically provide for this access. Nevertheless,
it may be prudent to include a specific term in the assistance agreement clarifying this  point to
avoid misunderstandings.  EPA may have to pay for obtaining access to the data if its transmittal
imposes additional costs on the recipient. Assuming our use of the data is incidental to the
principal purpose of the agreement, we also have a specific right to  authorize peer reviewers to
use the data for Federal purposes under 40 CFR 30.36(c)(2).

EPA cannot obtain access to the raw data if EPA specifically bargains away this right in the
assistance agreement.

Again, consult the Office of General Counsel (OGC) for help in drafting appropriate language
for your assistance agreement.

       2.2.12  Can the Recipient of a Grant or Cooperative Agreement Use Agreement
              Funds to Pay Peer Reviewers of their Work Products?

As noted in Section 2.2.11, provided EPA agrees that a peer review would further the public
purpose of the assistance agreement, EPA may include funds for the peer review in the
agreement. A payment to peer reviewers in exchange for their review of a scientific and/or
technical work product is allowable as a fee for professional services under assistance
agreements. (To accurately  characterize this cost, however, it is important that the payment be
referred to as a fee, rather than an honorarium.)  See OMB Circular A-21, Section J, item 32,
Professional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, Item
39, Professional Services Costs (Non-Profit Organizations), and OMB Circular A-87,
Attachment B, Item 33, Professional Services  Costs (State, Local and Indian Tribal
Governments).  See http://www.whitehouse.gov/OMB/circulars/ for further details.

-------
                                Peer Review Handbook                     Page 40
       2.2.13 Do Products Generated Under EPA Grants or Cooperative Agreements Need
             to be Reported in the Science Inventory

As a matter of practice, EPA organizations are encouraged to include scientific work being done
under grants and cooperative agreements in the Science Inventory so that Agency staff are aware
of the ongoing work. If a grant or cooperative agreement product is likely to be used in Agency
decision making (assuming this use is incidental to the principal purpose of the agreement), it
should generally be considered a candidate for peer review and noted as such in the Science
Inventory.

       2.2.14 Should Site-Specific Decisions be Subject to Peer Review?

A site-specific decision itself is not subject to peer review and does not need peer review as
described in the Peer Review Policy. However, if a site-specific decision is supported by
influential scientific information  or a highly influential scientific assessment, that work product
should be peer reviewed. Generally speaking, you should closely examine how the underlying
scientific and/or technical work product is adapted to the site-specific circumstances.

       2.2.15 Should NEPA Products be Subject to Peer Review?

Although an Environmental Impact Statement (EIS) prepared under the requirements of the
National Environmental Policy Act (NEPA) receives extensive review through the "scoping" and
interagency review processes that are part of NEPA, this is not usually considered peer review.
If the underlying scientific and/or technical data, models, analyses, or work products are
influential scientific information  or a highly influential scientific assessment, then the relevant
portions should  be peer reviewed.

In general, the Agency's role in preparing the NEPA  document may suggest the extent of review
the document should get (See sections 2.2.3 and 2.2.4). If EPA is developing the document as
part of an EPA action/decision (EPA is  the Lead Agency under NEPA), and the document is
influential scientific information  or a highly influential scientific assessment, then it should
receive independent peer review. If not, (the document is not influential,  will have little impact,
is non-controversial, etc.), then peer input might be appropriate.

On the other hand, if EPA is reviewing  an EIS from another agency (EPA is not the Lead
Agency under NEPA), it is likely that we are reviewing the EIS for conflicts with EPA policy
and general environmental concerns. In such a case, EPA should ask if the underlying scientific
and/or technical work product that supports the EIS has been peer reviewed to avoid concerns
about the full credibility and soundness of the EIS based on the science and technical support.
EPA should work with the other organization/agency to ensure that scientific and/or technical
work products receive peer review adequate for EPA purposes.

-------
                                Peer Review Handbook                     Page 41
       2.2.16 Should Environmental Regulatory Models be Peer Reviewed?

Generally, yes.  Specific guidelines for the peer review of environmental regulatory models have
been published by the Agency.  These can be found on the EPA web site under the Science
Policy Council home page (http://www.epa.gov/osa/spc/). In 2000, the Science Policy Council
established the Agency's Council for Regulatory Environmental Modeling (CREM) which
promotes consistency and consensus among environmental modelers and users.  The CREM is a
good resource for you to contact for specific questions regarding peer review of environmental
regulatory models (http://cfpub.epa.gov/crem/).

       2.2.17 Should Another Organization's Work Products That Have Been Submitted
             to EPA for Use in Decision Making be Peer Reviewed?

Any scientific and/or technical work product that is used in Agency decision making and is
considered influential scientific information or a highly influential scientific assessment becomes
a candidate for peer review regardless of whether the work product is produced by the Agency or
another organization. Therefore, all work products important to EPA environmental decision
making that are independently generated by other organizations (e.g., other Federal agencies,
interagency groups, State and Tribal bodies, environmental groups, industry, educational
institutions, international bodies) need to be considered as candidates for peer review.

If possible, when EPA knows that a work product is being generated by  another organization and
may be of interest to EPA for future use, the appropriate EPA office(s) should work with that
organization, and others, as appropriate (e.g., the states), to promote the use of peer review. For
example, the Office of International Affairs (OIA), as well as the impacted Program or Regional
Office(s), should be included when international products are being considered for EPA use.

It is hoped that if the other organization has the work product independently peer reviewed, the
peer review will meet the intent of the Agency's Peer Review Policy and EPA's proposed use of
the product (i.e., the peer review is basically equivalent to what EPA would do).  Agency staff
from the appropriate office(s) should examine closely the particulars of the peer review to ensure
independence and a conscious effort to incorporate the peer reviewers' comments into the final
work product.  If there are perceived, or real, conflicts of interest, this may preclude the use of
that peer review and, in those instances, another peer review would be needed. See Section 3.4.9
for considerations of when an outside party conducts and/or funds peer review of their own work
product and submits it to the Agency.

If the outside organization does not have the work  product peer reviewed and EPA decides it
should be peer reviewed, the appropriate EPA office(s) should ensure peer review of that work
product occurs prior to the Agency's use of the work product in decision making.  Peer review
can be accomplished by asking the outside organization to do so, or, if it declines, EPA may
conduct or arrange for the peer review. If EPA is conducting or arranging the peer review, the
product should be entered in the Science Inventory.

-------
                                 Peer Review Handbook                      Page 42
       2.2.18  Can Work Products That are Not Determined to be Influential Scientific
              Information or Highly Influential Scientific Assessments Still be Peer
              Reviewed?

Yes. Scientific and/or technical work products that are not categorized as influential scientific
information or a highly influential scientific assessment may nonetheless be candidates for peer
review. For example, a project manager may decide to use peer review because of particular
program needs and goals.  Peer review may also be warranted because it adds substantial value
to the work product.

2.3    Determining Which Work Products Do Not Receive Peer Review

       2.3.1   What Information Is Exempt from the OMB Bulletin Provisions?

The following information does not need to be peer reviewed, even if it might be considered a
highly  influential scientific assessment or influential scientific information:

    a)  Information related to national security, foreign affairs, or negotiations involving
       international trade  or treaties where peer review would interfere with the need for secrecy
       or promptness;

    b)  Information disseminated in the course of an individual adjudication or permit
       proceeding (including a registration, approval, licensing, site-specific determination),
       unless peer review is practical and appropriate and the information is scientifically or
       technically novel or likely to have precedent-setting influence on future adjudications
       and/or permit proceeding;

    c)  Information involving a health or safety dissemination that is time-sensitive;

    d)  A regulatory impact analysis or regulatory flexibility analysis subject to interagency
       review under Executive Order 12866, except for underlying data and analytical models
       used;

    e)  Routine statistical information (e.g., periodic demographic and economic statistics) and
       analyses of these data to compute standard indicators and trends;

    f)  Accounting, budget, actuarial, and financial information, and;

    g)  Information disseminated in connection with routine rules that materially alter
       entitlements, grants, user fees, or loan programs, or the rights and obligations of
       recipients thereof.

-------
                                Peer Review Handbook                      Page 43
       2.3.2  Are There Other Circumstances Where Peer Review Is Not Necessary?

There may be circumstances in which it is not necessary for a work product to undergo peer
review. For example, peer review is not generally conducted:

   a)  For work that has been previously reviewed in a manner consistent with OMB's Bulletin
       and this Handbook. For example, a cancer risk assessment methodology or an exposure
       modeling technique that was the subject of earlier peer review of appropriate technical
       merit would not generally undergo additional peer review, even if the product supported a
       significant Agency  decision;

   b)  If an application of an adequately peer-reviewed work product does not depart
       significantly from its scientific or technical approach;

   c)  When the scientific and/or technical methodologies or information being used are
       commonly accepted in the field of expertise, e.g., many products supporting Control
       Techniques Guidelines (CTGs) and Effluent Limitation Guidelines (ELGs). Such use
       should provide the appropriate documentation to support the commonly held view.

       2.3.3  For Information That Is Not Exempt, Are There Circumstances Where the
             Peer Review Provisions of the OMB Bulletin Can Be Waived or Deferred?

The Administrator may waive or defer the peer review provisions of the OMB Bulletin for
influential scientific information or highly influential scientific assessments if there is a
compelling rationale for the waiver or deferral. The use of waivers is expected to be limited to
unusual and compelling situations not otherwise covered by the exemptions, such as situations
where unavoidable legal deadlines prevent full implementation of the Bulletin's peer review
provisions. According to the Bulletin, deadlines found in consent decrees will not ordinarily
warrant waiver of the provisions because those deadlines should be negotiated to permit time for
conducting a peer review. Deferral of some or all of the peer review provisions may be an
appropriate way to accommodate immovable deadlines. If any of the OMB Bulletin provisions
are deferred, peer review should be conducted as soon as practicable thereafter. Deferrals of
peer review of influential scientific information and highly influential scientific assessments
should be approved by the Administrator.

If peer review of a work product is not planned, an explanation should be included in the
database entry for that work product in the Science Inventory.

       2.3.4  Does Publication in  a Refereed Scientific Journal Mean That Adequate Peer
             Review Has Been Performed?

Scientific papers (articles) that are peer reviewed by a credible refereed scientific journal may
not generally need to undergo further peer review. However, you should expect that there may

-------
                                Peer Review Handbook
                    Page 44
be cases where EPA determines that a more rigorous or transparent review process is necessary,
(e.g., if a particular journal review process did not address questions that EPA determines should
be addressed before using or disseminating the information, such as the extent of uncertainty
inherent in the findings, the sensitivity to model choice, or the applicability of the data presented
for use in the Agency's models). Also see Section 2.4.4 for additional information.

       2.3.5  Do Voluntary Consensus Standards Undergo Peer Review?

Generally, no.  The National Technology Transfer and Advancement Act of 1995 (NTTAA)
directs EPA to use available voluntary consensus standards in its regulatory activities unless to
do so would be inconsistent with applicable law or otherwise impractical.  For purposes of the
NTTAA, voluntary consensus standards are defined as technical standards (e.g., materials
specifications, test methods, sampling procedures, and business practices) that are developed or
adopted by voluntary consensus bodies (such as ISO or ASTM).  The general purpose of the
NTTAA is to reduce private and governmental costs by avoiding having the government
"reinvent the wheel" in the development of technical standards. Voluntary consensus standards
would normally not undergo peer review because the underlying process used by issuing
organizations to develop and approve these standards is generally considered adequate for
purposes of the Agency's peer review policy.  EPA reserves the right to conduct a peer review if
it determines that the particular standard it wants to use is not a voluntary consensus standard for
purposes of the NTTAA.
2.4    Choosing a Peer Review Mechanism

       2.4.1  How Do You Determine the Appropriate
             Peer Review Mechanism?
The mechanism of the peer
review should match the
importance and complexity of
the work product.
During the planning of a peer review, the Decision Maker
and the Peer Review Leader may consider several mechanisms for the peer review of scientific
and/or technical work products. These options range from consultations with EPA colleagues
not involved in developing the product to a large and formal panel of outside subject matter
experts. The peer review effort might be a focused one-time evaluation, or could encompass
several examinations over the course of a project.  In principle, peer review provides the greatest
credibility for influential scientific information and highly influential scientific assessments
when it involves well-qualified external reviewers, is intensive in its examination, and operates
through a more or less formal and transparent process.  As a practical matter, however, time and
resource considerations in many cases impose limitations on what can be reasonably achieved.
Arranging for the most appropriate and feasible peer review will involve good judgment
regarding the extent to which the peer review will improve the credibility of the product as well
as consideration of substance, time, and resources. Developing a peer review plan that provides
for appropriate depth, timing, and content is an important matter for early consideration by the
Decision Maker and Peer Review Leader. Note that use of peer input or public or stakeholder
involvement does not constitute peer review.

-------
                                 Peer Review Handbook                      Page 45
While the following information is described in more detail elsewhere, these are important
considerations in determing a peer review mechanism.  The approach best suited to a specific
work product will depend on the nature of the topic and the intended final product. Generally,
the more novel or complex the science or technology, the greater the cost implications of the
impending decision, and the more controversial the issue, then the stronger the indication is for a
more extensive and involved peer review and for external peer review in particular.  Certain
work products may clearly lend themselves to extensive external peer review; generally these
will be products with large impacts (e.g., those that support Tier 1  and Tier 2 rulemakings).
Other work products may not need a large scale external peer review and may utilize a less
involved, less resource intensive review.  The peer review of some products may be better served
with some form of internal peer review or a combination of internal and external peer review.

It is important to make the choice of peer review mechanism at the time that the work is planned
(for products supporting rule makings, at the analytic blueprint stage) so that peer  review costs
and time can be budgeted into the work plan. Essentially, the level of peer review should match
the impact and complexity of the work product. For example, a rule under development carries
considerable weight and deserves careful handling and attention; therefore, the supporting work
product deserves similar care and attention for its peer review.  Factors that need to be
considered include: use of internal vs. external peer reviewers; individual letter reviews vs. panel
reviews; the number of reviewers; the timing of the peer review. No single peer review
mechanism is likely to work best in all situations.  Some useful guidance includes:

   a)   Influential scientific information intended to support the most important decisions, or
       work products that have special importance in their own right, external peer review is the
       approach of choice.  Generally, the more complex, novel and/or controversial the
       product, or the higher impact it has, the more the Decision Maker should consider
       implementing a peer review involving external experts;

   b)   Highly influential scientific assessments are expected to undergo external peer review.
       When time and resources allow, panels are preferable;

   c)   Work products that are less complex, novel, or controversial, or have a lower impact
       might be subject to a less  extensive, less resource-intensive review processes;

   d)   Group  discussion with peer reviewers can be very helpful at some point in the peer
       review process as it allows interaction among peer reviewers with different perspectives
       and expertise.  Public panels  are more transparent than closed discussions.  On the other
       hand, simply soliciting individual comments is easier, faster, and less expensive.
       Individual review is probably more appropriate for peer review at the early stages of a
       product's development or for products with less impact and complexity;

   e) More reviewers are necessary for more complex projects (more disciplines)  and for more
       controversial topics (differences in scientific perspective within a discipline).

-------
                              Peer Review Handbook                      Page 46
f)   Strict time constraints, such as a court-ordered deadline, can make a less involved or
    formal peer review mechanism imperative. Decision Makers and Peer Review Leaders
    should make maximum efforts to ensure that such a process is systematic and objective.

    2.4.2  What are Some Examples of Internal Peer Review Mechanism?

 a)  Individual independent experts from within the Agency (e.g., ORD experts on non-cancer
    effects of lead (Pb) review a draft article on benchmark dose completed by a Program
    Office);

 b)  An ad hoc panel of independent experts from within the Agency (e.g., an independent
    internal workgroup convened to examine the case for the classification of a chemical as a
    carcinogen);

 c)  Technical merit review by scientists in an Agency laboratory (e.g., an initial review of the
    risk assessment for a regional incinerator by Agency scientists).

    2.4.3  What are Some Examples of External Peer Review Mechanisms?

 a)  Independent experts from outside the Agency (e.g., a letter review by outside scientists);

 b)  An ad hoc panel of independent experts outside the Agency (e.g., a group is convened to
    develop a consensus on the carcinogenicity of a particular industrial chemical);

 c)  Agency-sponsored  peer review workshops (e.g., a review of potential indicators of
    ecosystem damage);

 d)  Review by an established Federal Advisory Committee Act mechanism such as the
    Science Advisory Board (SAB), FIFRA Scientific Advisory Panel (SAP), ORD's Board
    of Scientific Counselors (BOSC), or the Clean Air Scientific Advisory Committee
    (CASAC) - e.g., a review of a criteria document for a particular chemical risk;

 e)  Agency-appointed special board or commission (e.g., a review of the risk assessment
    methodology prepared by the Clean Air Act Commission on Risk Assessment). Note:
    The Office of General Counsel should be consulted regarding EPA's  authority to
    establish and finance the activities of a commission or board.

 f)  Review by the NAS (e.g., a review of the state of current knowledge  about children's
    health risks from pesticide exposures).

 There are other bodies that may provide external commentary on Agency work products but
 are not considered peer review mechanisms, such as interagency committees (e.g.,  a review

-------
                                 Peer Review Handbook                      Page 47
    of prospective research plans by the Committee on the Environment and Natural Resources
    coordinated by the White House), committees convened by another Federal agency or
    government organization (e.g., a review of the Dioxin Reassessment by the Health and
    Human Services Committee to Coordinate Environmentally Related Programs), and/or
    reviews initiated by non-governmental groups (e.g., a Society of Risk Analysis review of
    cancer guidelines).

       2.4.4   What is the Role of Peer Review by a Refcreed Scientific Journal?

Peer review of journal articles (written by EPA or non-EPA authors) performed by a credible,
refereed scientific journal contributes to the scientific and technical credibility of the reviewed
product.  Generally, EPA considers peer review by such journals as adequate for reviewing the
scientific credibility and validity of the findings (or data) in that article, and therefore, a
satisfactory form of peer review13.  However, in some cases, peer review of an Agency work
product that uses these articles may be conducted (see Section 2.4.5).

EPA-authored journal articles, whether used in an Agency work product or not, are included in
the Science Inventory to highlight the extensive work EPA produces in the scientific literature.

Prior to submitting an article to a journal for peer review, EPA employees are encouraged to
have the article internally peer reviewed (see Section 2.4.2); such internal peer review is already
common practice in certain parts of EPA.  Articles may also need examination in accordance
with any organizational clearance procedures, especially when the author is presenting him or
herself as an EPA employee. For EPA employees, conflict of interest law and policy will also
apply.

       2.4.5   Do Agency Work Products Become Candidates for Peer Review when Peer-
              Reviewed Journal Articles are Used in Support of that Work Product?

In most instances, Agency work products are candidates for peer review even when supported by
peer-reviewed journal article(s). Although the use of articles that have been peer reviewed by a
credible journal strengthens the scientific and technical credibility of any work product in which
the article(s) appears or is referenced, it does not automatically eliminate the need to consider
whether the work product itself should be peer reviewed.  In most cases, journal peer review may
not cover issues and concerns that the Agency may want peer reviewed to support an Agency
action. Under these circumstances, the scientific and/or technical work product in which the
article(s) appears or is referenced becomes a candidate for peer review. A journal article
authored by EPA employees should be used in the same manner as an article published by
anyone else in a credible, well-recognized journal.
13 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information
Disseminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002),
(http://www.epa.gov/qualitv/informationguidelines')

-------
                                Peer Review Handbook                      Page 48
If an Agency work product is based solely on a single article that has received peer review by a
credible journal (e.g., where a model is suggested for a singular use that fits a specific Agency
need), peer review of the Agency work product may or may not be necessary depending on how
closely you apply the findings from the article.  If an Agency work product is based on two or
more articles that have received peer review by a credible journal(s), the Agency work product
generally should be considered for peer review.  Decisions to make (or not make) a work product
a candidate for peer review should be documented in the peer review record.

One important factor to remember with regard to the use of articles that have received journal
peer review concerns the availability of documentation from that peer review. Ideally, EPA
should maintain a clear, easily accessible record of the peer review to ensure the credibility and
validity of the peer review (see Section 2.5 for details  on the peer review record). However, the
documentation from a journal peer review may not normally be available to the Agency, so such
documentation is not expected in the peer review record.

       2.4.6  When and How Often Should Peer Review Occur?

The Decision Maker and Peer Review Leader have significant discretion in deciding on the
timing and the frequency of peer review.  Options abound, each with merits depending on the
context and specified peer review objectives. In many situations, a single  peer review event,
beginning when the final draft work product becomes  available, is the approach taken. However,
it is increasingly apparent that peer review performed  earlier in the work product development
stages provides a superior approach for some work products. There may be substantial
incremental benefit to conducting more than one peer  review during the whole process of work
product development, particularly where it involves complex tasks, has decision branching
points, or could be expected to produce controversial findings.  In addition, early review could be
beneficial at the stage of research design or data collection planning where the product involves
extensive primary data collection.  The Decision Maker and Peer Review Leader should
determine when the peer review(s) should occur, considering the type of work product under
development and at what point in its development process a peer review would be most
beneficial.

Other types of work products that could benefit from early, up-front peer review in their
development are scientific and technical planning products. Examples of such products are
research proposals, plans, and strategies.  Also, while  not products per se,  ongoing research
programs can be peer reviewed.

Remember though, that while more than one peer review can be beneficial, the distinction
between peer input and peer review should be kept in  mind.  Experts providing input during the
development or planning stages of the work product generally do not become peer reviewers of
that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction).

-------
                                Peer Review Handbook                      Page 49
       2.4.7  What Factors are Considered in Setting the Time Frame for Peer Review?

Several factors impact how quickly a peer review may be desired. These include deadlines for
completion of a project, research program, or rulemaking, funding availability, availability of
quality peer reviewers, the possible need to seek public comment on the peer review product, and
statutory and/or court-ordered deadlines.

Peer review sometimes leads to new information and analyses. Reviewers may make
recommendations for new research that would alter the work product and thus modify the
scientific/technical basis for the action or rule it supports. For this reason, a completed peer
review is desirable before issuing any proposal for public comment.  If that is not logistically
possible because of court or statutory deadlines, or other appropriate reasons, the Decision
Maker should make  every effort to complete the peer review before the close of the comment
period. Because peer review comments on such work products could be of sufficient magnitude
to warrant a revision to the proposed action or rule, Decision Makers should exercise diligence in
completing the peer  review prior to the proposal stage whenever possible.

Whenever feasible and appropriate, Offices should make a draft scientific assessment that is
highly influential available to the public for comment at the same time it is submitted for peer
review (or during the peer review process) and sponsor a public meeting where oral presentations
on scientific issues can be made to the peer reviewers by interested members of the public.
When employing a public comment process as part of the peer review, Offices should, whenever
practical, provide peer reviewers with access to public comments that address  significant
scientific or technical issues. To ensure that public participation does not unduly delay activities,
Offices should specify time limits for public participation throughout the peer review process.

       2.4.8  Which Office/Region or Other Agency is Responsible for Conducting the
             Peer  Review?

The organization of  the Decision Maker is normally responsible for conducting the peer review.
Responsibility for conducting a peer review can be negotiable when more than one Agency
Office or Region or  other agencies are involved. Often, the degree of involvement by any of the
organizations and agencies and their ability to fund peer review will determine who has the lead
for the peer review.

2.5    Creating the Peer Review Record and Peer Review Reports

       2.5.1  What is the Peer Review Record?

The peer review record is the formal record (file) of decision on the conduct of the peer review,
including the  type of peer review performed and an explanation of how the peer review
comments were addressed. It includes sufficient documentation for an uninvolved person to
understand what actually happened and why.  The Peer Review Leader (with the program

-------
                                Peer Review Handbook                      Page 50
manager if there is one) creates a separate, clearly marked peer review file within the overall file
for development of the work. Once the peer review is completed, it is the responsibility of the
Peer Review Leader to ensure that the peer review record is filed and maintained in accordance
with the organization's document retention procedures.

The Peer Review Record is separate from the entry in the Science Inventory Database.  While
some information from the peer review record appears in the database, the paper peer review
record is the official record of the peer review.

       2.5.2  How Can the Peer Review Record Improve the Peer Review Process?

A good peer review record allows future reference to what happened during the peer review, and
helps Decision Makers make appropriate use of peer reviewer input. In addition, a good record
helps ensure that EPA's Peer Review Policy is implemented.  The Peer Review Leader is
responsible for ensuring that the peer review record for individual work products is collected and
maintained until completion of the peer review effort.

       2.5.3  What Should Be in the Peer Review Record?

The peer review record should include all materials considered by the individual peer reviewers,
the peer review report, and other input. Such materials include, at a minimum (see also Section
4.3.1):

   a)  The draft work product submitted for peer review;

   b)  Materials and information (including the charge) given to the peer reviewers;
   c)  The peer review report, which summarizes the peer review findings and contains
       information about the peer reviewers (such as reviewers' names, affiliations, and a
       statement concerning potential conflicts and their resolution, if applicable);

   d)  Logistical information about conduct of the peer review (such as times and locations of
       meetings);

   e)  A memorandum, or other written record, approved by the Decision Maker, responding to
       the peer review comments specifying acceptance or, where thought appropriate, rebuttal
       and non-acceptance.  The Office should prepare a written response to the peer review
       report addressing each comment.

   f)   The final work product.

When deciding  if particular materials should be included in the record, the Peer Review Leader
should consider whether the materials would help reconstruct the peer review process and results
at a later time. If the materials may be helpful,  they should be part of the peer review record.

-------
                                 Peer Review Handbook                      Page 51
       2.5.4   How and When Should You Develop and Post Peer Review Reports and
              Agency Responses for Influential Scientific Information and Highly
              Influential Scientific Assessments?

Offices are expected to make peer review reports of the influential scientific information and
highly influential scientific assessments publicly available to implement the provisions of the
OMB Bulletin. Offices should instruct peer reviewers to prepare a report that describes the
nature of their review and the nature of their findings and conclusions.  The peer review report
should either (a) include a verbatim copy of each reviewer's comments (either with or without
specific attributions) or (b) represent the views of the group as a whole, including any disparate
and dissenting views, although attribution of comments to names is not necessary. The names of
the reviewers and their organizational affiliations should be included in the report. For highly
influential scientific assessments, the report should also include the charge to the reviewers and a
short paragraph on both the credentials and relevant experiences of each peer reviewer.

EPA will post or provide a link to the peer review reports on the Science Inventory website
(http://cfpub.epa.gov/si/) along with all materials related to the peer review (charge statement
and Agency response).  The credibility of the final work product is likely to be enhanced if the
public understands how the Agency addressed the specific concerns raised by the peer reviewers.
Offices should consider preparing a written response for inclusion in the peer review report. For
highly influential scientific assessments, the OMB Bulletin explicitly calls for Offices to prepare
a written response to the peer review report explaining (a) the agency's agreement or
disagreement with the views expressed in the report, (b) the actions that have or will be
undertaken to respond to the report, and (c) the reasons the Office believes those actions satisfy
any key concerns or recommendations in the report. These responses will also be posted on the
Science Inventory website.

       2.5.5   What Should You Do with a Peer Review Record That Pertains to a
              Rulemaking Action?

The Peer Review Leader should coordinate with his/her program's docket office to see that
proper docketing procedures are followed for a peer review of a work product supporting a rule.
If EPA relies on influential scientific information or a highly influential  scientific assessment to
support a regulatory action, the preamble should include a discussion of how EPA implemented
the provisions of the OMB Bulletin. See Appendix C for a template  to use for this purpose.

       2.5.6   When Should the Peer Review Record Building Process Begin?

An early start at developing and maintaining a peer review record will help ensure the record is
complete and helpful.  Ideally, the record begins when the decision to peer review a work
product is made. The Peer Review Leader should construct the peer  review record from this
point on - doing so will avoid potentially time-consuming reconstruction at a later point. Note
that the peer review record is not complete until it contains a copy of the final work product

-------
                                Peer Review Handbook                      Page 52
which addresses the peer review comments. If a product is not peer reviewed, a record should
also be created which explains why the product was not peer reviewed.

       2.5.7  What are the Differences in Record Keeping for a Review by Individuals
             Compared to Review by a Panel?

Generally, the content of the peer review record would be the similar. In the case of a review by
individuals, the peer review record would typically contain each individual's comments, while
for a panel review, the record typically contains a summary or other synthesis documenting the
panel's deliberations.

       2.5.8  Where Should the Peer Review Record be Kept and For How Long?

During the active conduct of the peer review, the Peer Review Leaders maintain the peer review
record themselves until the peer review is complete.  Minimally, the file should be maintained
until one year after the completed peer review is reported in the next annual reporting cycle.
After that, the peer review record should be maintained for a reasonable period of time, as
delineated by the organization's document retention policy. Establishment and maintenance of
the archive where the peer review records ultimately reside are an organization's responsibility
(i.e., not that of an individual program manager or Peer Review Leader).  Generally, to allow
flexibility, individual  Offices and Regions should decide the appropriate level of organizational
responsibility and how to ensure the record is routinely available.  The peer review record may
be kept with other records relating to the overall project, as long as it is easily and separately
identifiable.
There are also specific procedures regarding the use of dockets for record-keeping; however,
these are not covered in this Peer Review Handbook. The  documents contained in the peer
review record should be maintained in accordance with the Agency's record-keeping retention
schedule for such records. For details, see EPA's National Records Management Program at
http://www.epa.gov/records.  One long-term archiving  mechanism may be the formal archiving
at the Federal Records Center.

       2.5.9  Are Internal Peer Review Comments Included in the Peer Review Record?

To be considered a legitimate peer review, internal EPA peer reviews should be formally
conducted and documented. Such a process would be consistent with the guidance found in this
Handbook for planning, conducting, and completing a peer review. When you follow this formal
process to obtain peer review from EPA peers (see Section 1.5.9), then the whole record of that
internal peer review should be included in the peer review  record. This includes all the materials
detailed in Section 2.5.3 (also see Section 4.3.1).  Conducting a formal internal peer review is not
the same thing as informal input from your EPA colleagues (i.e., "colleagues down the hall"),
nor peer input from Agency personnel helping to develop the work product, nor organizational
review and clearance processes.  Such inputs from these informal processes should not be placed

-------
                                Peer Review Handbook                      Page 53
in the peer review record. The peer review record should contain only the information obtained
when you conduct a formal internal peer review.

In some cases, an internal EPA peer review may be followed by a separate external peer review.
In such a case, however, the external peer review will stand as the peer review of record since it
is viewed as more independent in nature, has broader fields of available expertise which can be
brought to bear on the issues and often includes greater depth for specific disciplines.
2.6    Budget Planning
                                                       normal cost of doing business.
       2.6.1  What Budgetary Factors Should You                  J     s
             Consider in a Peer Review?
                                                       pegf
Resources necessary for the implementation of the Peer Review Policy should be requested
through the usual Agency budgetary processes. The budget formulation process within the
Executive Branch is followed, after appropriation bills are passed by Congress, by budget
execution. These two processes provide opportunities to secure resources for activities carried
out by Headquarters and Regional offices, including peer review. The work products which will
undergo peer review should have adequate funding requests for the coming fiscal year.
Similarly, adequate funding should appear in the actual approved operating plan to ensure the
peer review  can be conducted. For purposes of budget planning, the costs of peer review would
include the FTE cost of staff, the contract or other costs associated with the use of outside peer
reviewers, and the administrative costs of conducting  a review (copying, travel expenses, etc).

       2.6.2   What Input is Needed for the Annual Budget Formulation and Budget
              Execution Process?

Senior Management in Program Office and Regions (including Decision Makers and budget
officers) should ensure that budget requests include anticipated resources for peer review. Peer
review should be considered as a normal part of doing business. Peer review resource
considerations should also be addressed in the analytic blueprint for Agency rule-making actions.

2.7    Legal Considerations

       2.7.1   What Are the Legal Ramifications of the Peer Review Policy?

The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. Except where
provided otherwise by law, peer review is not a formal part of or substitute for notice and
comment rulemaking or adjudicative procedures.  EPA's decision to conduct peer review in any
particular case is wholly within the Agency's discretion.  Similarly, nothing in the Policy creates
a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA

-------
                                Peer Review Handbook                      Page 54
decisions rely on scientific and/or technical work products that have been subjected to peer
review, the remarks of peer reviewers should be included in the record for that decision.

      2.7.2  Is Legal Advice Needed?

AA/RA staff and management should work regularly and closely with individual OGC/Regional
Counsel (RC) staff assigned to Agency activities.  Peer Review Leaders should initially consult
with their usual OGC/RC advisors for legal advice or referral.  Headquarters attorneys have
specialties in specific areas and can be consulted as needed (e.g., FACA considerations,
contractual responsibilities; ethics and potential conflicts of interest).

2.8   Federal Advisory Committee Act (FACA) Considerations

The Federal Advisory Committee Act, 5 USC App. 2, imposes certain open meeting (public
announcement in the Federal Register), balanced membership, and chartering requirements (with
the approval of the General Services Administration (GSA)) before the Agency establishes,
controls or manages an "advisory committee" for advice or recommendations. Peer review
carried out by formal and established (chartered) Federal advisory committees, such as the
Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel (SAP), is always subject
to FACA requirements.  However, FACA considerations do not apply to every EPA and
contractor-run peer review.

In the next Section you will find information on the applicability of FACA to EPA- or
contractor-run peer reviews.
      2.8.1  When Do FACA  Requirements Apply to EPA-Run Peer Reviews?

In most cases, Federal Advisory Committee Act (FACA) requirements apply to EPA-run peer
reviews that are conducted by formal and established (chartered) Federal advisory committees,
such as the Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel (SAP).
These groups  have the following characteristics:

      a)     The group is established, controlled, or managed by EPA;

      b)     The group has a fixed membership, established purpose, and a set agenda; and

      c)     The group strives to  produce collegial, rather than individual, advice to EPA.

EPA-run peer reviews that are  not originally intended to be subject to FACA, but which exhibit
the above characteristics, may unintentionally become subject to FACA.  Questions concerning
the applicability of FACA to peer review meetings should be addressed to the FACA experts in
the Cross-Cutting Issues Law Office of OGC, or the appropriate Office of Regional Counsel.

-------
                                Peer Review Handbook                     Page 55
       2.8.2  When Are EPA-Run Peer Reviews Not Subject to FACA?

If EPA conducts a peer review with the purpose of obtaining advice from the individual peer
reviewers and not for the purpose of obtaining a peer review product from the group (as a
collective or consensus body), the peer review, in most cases, would not be subject to FACA.
Peer review participants provide only their own views or recommendations and do not vote nor
do they provide collective or consensus recommendations to EPA.  When referring to the
recommendations of the individual reviewers, EPA should not characterize these
recommendations using terms such as "collective" or "consensus." As a general matter,
workshops and "letter reviews" that seek individual views or comments are usually not subject to
the requirements of FACA.

In addition to ensuring that peer reviewers only provide comments as individuals,  EPA officials
may wish to lessen the potential for a challenge under FACA by seeking balanced participation
at peer review meetings, and allowing interested members of the public to attend, and ensuring
that they have access to appropriate materials.

Non-FACA meetings may be announced in the Federal Register (providing that it is clear in the
notice that such meetings are not subject to FACA) as it provides the public with useful
information and a point of contact concerning the peer review. In addition, non-F AC A (as well
as FACA) meetings should also be advertised via other avenues (e.g., the Web, local
newspapers, and mailing lists).
       2.8.3  How Do You Ensure that a Contractor-Run Peer Review Does Not Become
             Subject to FACA?

Committees (or other peer review groups) established, controlled or managed by an outside
organization (such as by an EPA contractor) to provide that outside organization with advice and
recommendations (that will be submitted eventually to EPA as a contractor report) are not
subject to FACA. Although FACA should not apply to contractor-run peer reviews, there are
things that you (i.e., EPA) can inadvertently do that may invoke FACA.

The following are considerations that you should be aware of when a contractor runs a peer
review (e.g., letter review, panel, workshop, etc.) for EPA:

   a)  The outside party's peer review may be subject to FACA if EPA establishes, manages or
       controls the peer review group (e.g., EPA selects panel members, runs the meeting, etc.);

   b)  EPA should not provide contractors with a draft agenda or suggested format for meetings.
       EPA contractors should manage and control the process, including running any meetings;

-------
                             Peer Review Handbook                      Page 56
c) At the request of the EPA contractor, EPA can provide a briefing to the peer reviewers
   (e.g., in a conference call with the contractor on the line) on the history or background of
   the development of the document.  EPA should only provide technical or background
   information and not use the call to take over the contractor's peer review group. Not only
   should the contractor be  on the line, but it should be very clear to all participants that the
   contractor is in charge of the call. The contractor, not EPA, should invite people to
   participate, make all administrative arrangements, conduct the meeting, and control the
   agenda;

d) EPA employees may attend the peer review panel meetings or workshops. However,
   they may not take over the control of the meeting. The contractor may call on them to
   speak when appropriate,  but EPA staff should limit their participation to providing
   technical and/or background information, and not attempt to, or appear to, take over the
   contractor's meeting;

e) Since FACA does not apply when a contractor establishes, controls, or manages a peer
   review, the contractor need not avoid terms such as "collective" or "consensus" when
   reporting agreement among its peer reviewers (subcontractors);

f)  EPA may provide comments to the contractor on the contractor's peer review only to the
     extent that the Agency is verifying that the contractor has satisfactorily completed the
   report in accordance with the work assignment. EPA should not attempt to make changes
     in the contractor's conclusions; this would compromise the independence of the peer
                            review conducted by the contractor.

-------
                                Peer Review Handbook                     Page 57
                         3.  Conducting A Peer Review
3.1    Overview Statement

The success and usefulness of any peer review depends on the quality of the draft work product
submitted for peer review, the care given to the statement of the issues or "charge," the match
between the peer review draft product and the form of peer review, the match between the peer
review draft product and the scientific/technical expertise of the reviewers, and Agency use of
peer review comments in the final product. It is not simply enough to conduct a peer review;
each of the  foregoing elements deserves serious attention.

Figure 2 shows the order of activities for conducting a peer review.  The charge should be
completed before selection of the peer reviewers to ensure that they have the appropriate
expertise to address the questions raised.

3.2    Charge to the Peer Reviewers

       3.2.1   What is a Charge?

 As part of each peer review, the Peer Review Leader formulates a clear, focused charge that
identifies the technical and scientific issues on which the Agency would like feedback and
invites suggestions for improving the document as a whole. This request signals the Agency's
receptivity to expert recommendations.  The charge to peer reviewers usually makes two general
requests. First, it focuses the review by presenting
specific questions and concerns surrounding such
issues as the comprehensiveness of the literature
reviewed, the soundness of the method proposed, the
scientific support for the assumptions employed, and
the sensitivity of the results to alternative assumptions.
Secondly, it invites general comments on the entire
work product.  The reviewers should be asked to ensure
that the potential  implications of the uncertainties for the technical conclusions drawn are clearly
and transparently presented.  The  specific and general comments should focus on the scientific
and technical merits of the work product and, where germane, whether the scientific/technical
studies have been applied in a sound manner. Remember, the peer review is not for the decision
or action itself, but for the underlying scientific and/or technical work product; reviewers should
not be asked to provide advice on policy. Additionally, Focused questions greatly simplify the
task of collating,  analyzing and synthesizing peer review comments on a topical basis.  The
questions should  be specific enough to encourage helpful comments, including constructive
alternatives, but not so specific that they preclude creative responses (e.g., simply asking whether
or not the reviewer agrees with the model choice or conclusion).  Understanding the reviewer's
Time is well-spent preparing a good
charge, as the charge is crucial for
an effective peer review.

-------
                                Peer Review Handbook                      Page 58
reasoning is pivotal to the Agency's ability to address the reviewer's concern as well as crafting
specific improvements to the work product. The peer reviewers' written responses to these
questions help the Agency create a peer review record. As a general rule, time is well-spent
preparing a good charge, as the charge is crucial for an effective peer review.

       3.2.2  What are the Essential Elements of a Charge?

   a)  A brief overview or introduction (describe what the work product is, how it was
       developed, how it will be used)

   b)  As needed, a brief description or listing of any background materials provided to the peer
       reviewers

   c)  The specific issues or questions to be addressed by the peer reviewer(s), including
       logistical details such as:

          1) The due date of reviewers' comments;

          2) The format of reviewer responses (e.g., MS Word, Adobe Acrobat, e-mails, or
             text files);

          3) The point of contact, in case peer reviewers have questions.

       3.2.3  Where Can You Get an Example of a Charge?

Appendix D - Examples of Peer Review Charges, contains examples of successful charges that
cover a variety of issues. Appendix E - Guidance on Requesting a Review by the Science
Advisory Board, provides guidance for obtaining Science Advisory Board (SAB) services.

       3.2.4  Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?

Yes. EPA may decide to obtain stakeholder input on the charge to the peer reviewers, but EPA
makes the final  determination on what elements to include in the charge to ensure that it meets
EPA's needs for the peer review.

   a)  If you obtain stakeholder input,  include interested parties to the extent feasible based
       upon statutory, regulatory, budgetary and/or time constraints.  Do not limit input to one
       stakeholder or one side of a controversial issue (e.g.,  a responsible party or environmental
       group).

   b)  If EPA has hired  a contractor to perform the peer review, it should still be EPA personnel
       who obtain stakeholder input and provide the list of charge questions to the contractor.

-------
                                 Peer Review Handbook                      Page 59
   c)  If you form a committee of stakeholders to help develop the charge, be aware that your
       committee may become subject to the requirements of FACA (see Section 2.8).

       3.2.5  Who Writes the Charge When You Hire a Contractor to Conduct the Peer
             Review?

In general, if EPA hires a contractor to perform the peer review, EPA should allow the contractor
independence in conducting it.  However, with regard to the charge, EPA should provide the
contractor with  a substantive list of questions that EPA wants included in the charge letter to
ensure that the peer review meets EPA's needs. If the EPA project manager has them ready, the
list of charge questions can be incorporated into the Statement of Work.  Based on this list, the
contractor can then prepare and submit the actual charge letter to the peer reviewers.  Prior to
submitting the charge letter to the reviewers, the contractor should be directed to give EPA an
opportunity to review the charge letter to ensure that it meets EPA's needs. EPA cannot submit
the charge directly to the peer reviewers when the review is being conducted by a contractor.

       3.2.6  Is it Okay to Ask a Contractor to Develop the Charge to the Peer Reviewers?

No. EPA should provide the charge questions to the contractor as discussed above.  The
contractor, though, may provide assistance and advice in the development of the charge.

3.3    Time Line

       3.3.1  What are the Factors in Scheduling a Peer Review?

The peer review schedule is a critical feature of the process. The schedule should take into
account the availability of a quality draft work  product, availability of appropriate experts, time
available for peer review comments, deadlines  for the final work product, and logistical aspects
of the peer review (e.g., contracting procedures).

The schedule for peer review should take into account the overall rulemaking (or other decision
making) schedule.  For rules, in particular those in Tier 1 and Tier 2, the scheduling of the peer
review should be included in the development of the analytic blueprint. Peer review sometimes
leads to new information and analyses, or recommendations for new research that would alter the
work product and thus modify the scientific/technical basis for the action. For this reason, it is
usually advisable to complete the peer review before taking public comment on the rule, or at
least before the  close of the public comment period.  Note, however, that whenever feasible and
appropriate, offices should make a draft highly influential scientific assessment available to the
public for comment during the peer review process, and, if feasible and appropriate, sponsor a
public meeting where oral presentations on  scientific issues can be made to the peer reviewers by
interested members of the public. When employing a public comment process as part of the peer
review, Offices  should provide the reviewers access to the public's  comments that address
scientific or technical issues.

-------
                                 Peer Review Handbook                      Page 60
3.4    Selection of Peer Reviewers

       3.4.1   What are Considerations for Selecting Peer Reviewers?

The choice of peer reviewers should be based primarily upon on the reviewers' expertise,
knowledge, skills, and experience and should include specialists from multiple disciplines, as
appropriate. The group of reviewers should be sufficiently broad and diverse to fairly represent
the relevant scientific and technical perspectives and fields of knowledge; they should represent
a balanced range of technically legitimate points of view. Each peer reviewer should have
recognized expertise  that bears on the subject matter under discussion. It may sometimes be
appropriate, as in the case of F AC A committees, to consider "affiliation" (e.g., industrial,
academic, or environmental community).

The selection of independent peer reviewers is also critical to an effective peer review. EPA
should always make every effort to use peer reviewers who do not have any conflict of interest
or an appearance of a lack of impartiality, and who are completely independent (see Section
3.4.5). However, the very need to have experienced individuals on a peer review, along with the
desire to have appropriate technical balance and representation, means that the field of potential
peer reviewers may include those that are less than independent, have a  conflict of interest, or
might at least appear to lack impartiality.  Sometimes selecting individuals  who have served a
variety of organizations rather than a single one for an extended period may provide expertise
with diverse perspective.  The emphasis on independence and expertise  applies equally to
government experts and experts from the larger scientific community

To ensure that the selected reviewers have the appropriate expertise, knowledge, skills, and
experience, the charge to the reviewers should be determined in advance of the selection of the
reviewers.  Selection of peer reviewers should be made by identifying reviewers with the
appropriate expertise and then narrowing the field of potential peer reviewers to those
individuals that are independent, do not have a conflict of interest and do not appear to lack
impartiality.

You should also consider requesting that the public, including scientific and professional
societies, nominate peer reviewers.

       3.4.2   Where Do You Find Peer Reviewers?

Recommendations for potential peer reviewers can be identified by a number of organizations.
These include external groups such as the affected party(ies), special interest groups, public
interest groups, environmental groups, professional societies, trade or business associations,
State organizations or agencies, Native American Tribes, colleges  and universities, the National
Research Council, and other Federal agencies with an involvement in or familiarity with the
issue. Agency associated groups include the staff of the SAB or the SAP, and relevant scientific
and technical  experts from Program or Regional Offices.

-------
                                Peer Review Handbook                      Page 61
In certain circumstances, existing organizations such as the SAB, SAP or BOSC may be used to
conduct a peer review. These groups establish their own criteria for accepting work. If you use
such an organization, you should coordinate directly with the organization (e.g., see Appendix E
- Guidance on Requesting a Review by the Science Advisory Board, for its procedures). SAB,
SAP and BOSC conduct formal, public, external peer reviews.

Another method which might be used to find peer reviewers is public solicitation. The Science
Inventory can assist the public solicitation process by announcing the opportunity for public
nominations and by providing background on the review topic the public may find useful for
identifying potential reviewers they may wish to nominate. EPA's Science Advisory Board uses
the Internet to solicit names for both ad hoc and standing advisory committees. These names,
along with  short biographical sketches, are also posted so that the public may not only nominate,
but also comment on potential advisory committee members. See "Advisory Committee
Meetings and Report Development: Process for Public Involvement" available from SAB's
website at http://www.epa.gov/sab/pdf/sabso_04_001.pdf

Occasionally, a member of the scientific community will offer his/her services for peer review
during an ongoing peer review. These offers may be at no cost or based on an expectation that
reimbursement will be made. Disposition of these unsolicited offers should be handled on a case
by case basis by the Peer Review Leader, and as necessary, in consultation with the Peer Review
Coordinator, the Office of General Counsel (OGC), and appropriate Decision Makers.

If you use a contract mechanism to conduct a peer review, the contractor may have its own pool
of scientific and technical experts for peer review. EPA may provide contractors with
information on potential peer reviewers for conducting a peer review, if such a listing is prepared
in alphabetical order. EPA should not require that the contractor select from a prepared list, nor
require that the contractor receive EPA approval before selecting any given peer reviewer
(sometimes known as a "subcontractor"). However, EPA should review the list of peer
reviewers for conformance to work assignment specifications and adherence to conflict of
interest and appearance of a lack of impartiality concerns.  Furthermore, when utilizing a
contract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor
to a specific subcontractor (or peer reviewer), nor is EPA permitted to direct the peer reviewer
(subcontractor). All interactions with the peer reviewers are to be coordinated through the prime
contractor.  See Section 3.6 for further information.

Keep in mind that for contracting purposes, contractors are required to obtain Contracting
Officer (CO) approval of subcontractors and the CO generally seeks the input of the Contracting
Officer Representative (the COR may be the Peer Review Leader in many cases) before
approving the use of subcontractors.  In this case, as noted above, the Peer Review Leader should
review the list of potential reviewers for conformance to the work assignment specifications and
adherence to conflict of interest and appearance of a lack of impartiality concerns.

-------
                                 Peer Review Handbook                      Page 62
       3.4.3   Are External or Internal Peer Reviewers Preferred?

External peer reviewers are generally preferred for influential scientific information, and are
expected for highly influential scientific assessments.  For some work products, like those
reviewed at interim steps, either external or internal peer review may be appropriate.  Selection
of internal peer reviewers should be based upon technical expertise, available time and
affiliation.  That is, they should not come from the same Office or group producing the product
or have any other connection with the product or document being peer reviewed.  The choice of
reviewers should be analyzed on a case-by-case basis. Reviewers employed by other Federal
and State agencies may possess unique or indispensable expertise. Don't forget that for highly
influential scientific assessments, the use of internal peer reviewers is constrained by the OMB
Bulletin. External peer reviewers should be selected based upon technical expertise as well,
however, you should not use individuals who have been involved in the development of the work
product. (See Section 1.2.6; see also Sections 1.2.7 to 1.2.9).

       3.4.4   What is Important in the Mix of a Peer Review Panel?

A peer review panel or group can number from just a few individuals to ten or more, depending
on the issue, the time and resources available, and the broad spectrum of expertise necessary to
treat the range of issues/questions in the charge.  Objective technical expertise and no conflict of
interest or appearance of a lack of impartiality are critical characteristics for selecting peer
reviewers.  Naturally, experts whose understanding of the specific technical area(s) being
evaluated are necessary; nevertheless, it is also important to include a broad enough spectrum of
other related experts to completely evaluate the relevant impacts on other less obvious concerns
(i.e., to comment not only if the job is being done right, but also whether the right job is being
done). For example, for health related peer reviews, experts  in such fields as ecology and
economics  may provide very useful insights.  Although persons who are familiar with and have a
substantial  reputation in the field are often called upon repeatedly to be reviewers, it is important
to keep  a balance with new people who bring fresh perspectives to the  review of a work product.
The idea here is to avoid the repeated use of the same reviewer on multiple assessments unless
his/her participation is essential and cannot be obtained elsewhere.

There is usually a continuum of views on any issue. To the extent practicable, selected experts
should have technically legitimate points of view that fall along the continuum. A review panel
should include experts that are considered "mainstream" (nearer the center of the continuum) as
well as those further along the continuum while generally avoiding the extremes.  This will  help
maintain a balanced review panel, while allowing a broad range of views to be expressed and
discussed.  A balanced panel will allow consensus building if consensus is the object of a
particular peer review; if not, it provides a spectrum of views for the Agency to evaluate.  As
such, the group of reviewers should be sufficiently broad and diverse to fairly represent the
relevant scientific and technical perspectives and fields of knowledge.

-------
                                Peer Review Handbook                     Page 63
As a general rule, experts who have made public pronouncements or have had a predominant
influence on the position for a given organization on an issue, those who have clearly "taken
sides," may have an appearance of a lack of impartiality (see "Ethical Standards" below) and
should be avoided.

       3.4.5  Ethical Standards

As discussed previously (see Section 2.4), there are various mechanisms through which peer
reviews may be accomplished. Internal peer reviews can be conducted by independent experts
from within EPA, either individually or as ad hoc peer review panels. External peer reviews can
be conducted by individual experts (e.g., through letter reviews) who are either Regular
Government Employees (RGEs) of another agency, experts hired as EPA Special Government
Employees (SGEs), or experts whose services are procured through a contract mechanism.  In
addition, external peer review panels can be convened through a contract mechanism or under
the auspices of a Federal Advisory Committee. Finally, peer reviews may be conducted by
outside organizations such as the National Academy of Sciences (NAS).

To ensure a credible peer review process, high ethical standards are applied to each of these
mechanisms. These standards are embodied in the various laws, regulations, and other
requirements that apply to peer reviewers who are either RGEs, SGEs, contractors, or who are
employed by outside organizations (e.g., NAS).

There are specific regulations that describe ethics considerations for contractor personnel.  The
Federal Acquisition Regulations (FAR), the Environmental Protection Agency Acquisition
Regulations (EPAAR), and other internal Agency documents define ethical standards, such as
Conflict of Interest (COI), and describe the Agency's policies and procedures.  This Peer Review
Handbook provides some guidance regarding the application of this information to peer review
(see Section 3.6 and Appendix F - Example Statements of Work for Contracts); however,
specific questions should be addressed to the Office of Acquisition Management.

For peer reviews conducted by outside organizations, the peer review leader or other appropriate
official should be thoroughly familiar with the ethics policies and requirements of the
organization conducting the review. For example, the NAS National Research Council  has
published their "Policy on Committee Composition and Balance and Conflicts of Interest."

The remainder of this section is focused on the laws and regulations governing the standards of
ethical conduct for  RGEs and SGEs. The standards for government employees are very
thorough and, as  such, may help the peer review leader or other appropriate official to identify
issues that may be more generically of concern with respect to the selection of peer reviewers.

-------
                                 Peer Review Handbook                      Page 64
       Conflicts of Interest:

Public service is a public trust. As such, it is vitally important that government employees
maintain the highest ethical standards. While there are many aspects of ethical behavior, for the
purposes of peer review we are mostly concerned with compliance with applicable Federal ethics
statutes and regulations. The basic regulations are issued by the Office of Government Ethics
(OGE) and are occasionally supplemented with additional regulations by EPA.

Compliance with Federal ethics regulations can be a confusing and complex process.  However,
the process is usually concerned with determining whether a RGE or a potential SGE has an
ethics issue with respect to a "conflict of interest" or "an appearance of a lack of impartiality."
Each of these two concepts will be explained in more  detail below.  However, sometimes the two
concepts are confused.  A "conflict of interest" is concerned with matters of financial interest.
There is either a conflict of interest or there is not. An "appearance  of a lack of impartiality"
(sometimes mischaracterized as a "potential conflict of interest"), can be financial or not
financial in nature.  The determination of an "appearance issue" is not quite as clear-cut as a
"conflict of interest" and calls for more of a judgment on the part of the determining official as to
whether one exists.

Let's look at each of these two concepts in detail.

Generally, a conflict of interest arises when the person's financial interests are affected by
his/her participation in a particular matter, when he/she, his/her associates or other individuals
whose interests are imputed (as provided under Federal law) would  derive benefit from
incorporation of their point of view in an Agency product, and/or when their professional
standing and status or the significance of their principal area of work might be affected by the
outcome of the peer review.  Whenever there are questions about conflicts of interest, you should
contact the appropriate Deputy Ethics Official, Ethics Assistant or the Designated Agency Ethics
Official for clarification and assistance. For a list of EPA's Deputy Ethics Officials, see
http://intranet.epa.gov/ogc/ethics/deos.htm.

For Financial Conflict of Interest (COI) issues, the basic 18 USC  208 provision states that:

   An employee is prohibited from participating personally and substantially in an official
   capacity in any particular matter in which he, to his knowledge, or any person whose interests
   are imputed to him under this statue has a financial interest,  if the particular matter will have
   a direct and predictable effect on that interest.

For a conflict of interest to be present, all elements in  the above provision must be present. If an
element is missing, it is unlikely that there is a conflict of interest issue. However, the
regulations concerning an appearance of a lack of impartiality may still apply and need to be
considered.

-------
                                 Peer Review Handbook                       Page 65
One very important aspect to consider is whether a charge to an advisory committee or other peer
review panel constitutes a "particular matter." A "particular matter" refers to matters that will
involve deliberation, decision, or action that is focused upon the interests of specific people, or a
discrete and identifiable class of people.  It does not refer to consideration or adoption of broad
policy options directed to the interests of a large and diverse group of people. A particular
matter of general applicability means a particular matter that is focused on the interests of a
discrete and identifiable class of persons, but does not involve specific parties. A particular
matter involving specific parties means any judicial or other proceeding, application, request for
a ruling or other determination,  contract, claim, controversy, investigation, charge,  accusation,
arrest or other particular matter  involving a specific party or parties in which the United  States is
a party or has a direct and substantial interest. Participating personally means direct participation
in the matter to be reviewed. Participating substantially refers to involvement that is of
significance to the matter under consideration. For more information regarding the  definition of a
particular matter, see the Office of Government Ethics document "Standards of Ethical Conduct
for Employees of the Executive Branch" available at http://www.usoge.gov.

Another consideration is whether there will be a direct and predictable effect on the financial
interests of the reviewers.  A direct effect on a participant's financial interest exists  if a close
causal link exists between any decision or action to be taken in the matter and any expected
effect of the matter on the financial interest. A particular matter does not have a direct effect if
the chain of causation is attenuated or is  contingent upon the occurrence of events that are
speculative or that are independent of, and unrelated to, the matter.  A particular matter that has
an effect on a financial interest only as a consequence of its effects on the general economy is not
considered to have a direct effect.  A predictable effect exists if there is an actual, as opposed to a
speculative, possibility that the matter will affect the financial interest.

The question of whether a charge is a particular matter involving a specific party or parties or
whether it is a particular matter  of general applicability has important ramifications for reviewers
who are Special Government Employees (SGEs) as defined under 18 USC  202(a). SGEs
serving on FACA (Federal Advisory Committee Act) committees are covered by certain
exemptions from the conflict of interest statutes.  The most significant of these is 5  CFR
2640.203(g), which pertains to certain financial interests arising from an SGEs's outside
employment.  Specifically, this  exemption permits SGEs serving on FACA committees to
participate in particular matters  of general applicability (e.g., development of general regulations,
policies or standards) where the disqualifying interest arises from the SGEs non-Federal
employment or prospective employment (but NOT due to financial  holdings or consultancies).

Furthermore, the question of whether a charge is a particular matter involving a specific  party or
parties,  or whether it is a particular matter of general applicability, has other ramifications for
SGEs (and other Federal employees for that matter).  For example, if a charge is a particular
matter involving a specific party or parties, there are certain restrictions regarding obtaining U.S.
Government contracts/grants as well as future representation back to the Federal government.
For more information, see 18 USC  207 or contact your Deputy Ethics Official/Ethics Assistant.

-------
                                 Peer Review Handbook                      Page 66
Even if an SGE is determined to have a conflict of interest, that does not mean that it is
impossible for her or him to participate in an advisory committee or other peer review.  For
example, in some cases it is possible to obtain a waiver for her or his participation. It is also
possible to have a particular SGE be recused from certain portions of a review that would
otherwise create a conflict of interest.  However, you should always weigh the advantages and
disadvantages of this approach very carefully. In addition, public disclosure of these conflicts in
a manner compliant with the Privacy Act and other applicable regulations and practice, will add
to the integrity of the advisory committee and peer review process.  Additional information for
SGEs may be found in "Conflict of Interest and the Special Government Employee: A Summary
of Ethical Requirements Applicable to SGEs" available from the U.S. Office of Government
Ethics (U.S. OGE) at http://www.usoge.gov.

       Appearance of a Lack of Impartiality:

The Code of Federal Regulations at 5 CFR  2635.502(a) states that:

   Where an employee knows that a particular matter involving specific parties is likely to have
  a direct and predictable effect on the financial interest of a member of his household, or
  knows that a person with whom he has a covered relationship is or represents a party to such
  matter, and where the person determines that the circumstances would cause a reasonable
  person with knowledge of the relevant facts to question his impartiality in the matter, the
  employee should not participate in the matter unless he has informed the agency designee of
  the appearance problem and received authorization from the agency designee. "

Further,  2635.502(a)(2) states that,

  An employee who is concerned that circumstances other than those specifically described in
  this section would raise a question regarding his impartiality should use the process
  described in this section to determine whether he should or should not participate in a
  particular matter.

Even though circumstances for specific candidates may not raise an issue of a conflict of interest,
each advisory committee member or peer reviewer should be evaluated to ensure that an
appearance of a lack of impartiality does not preclude their participation.

How do you evaluate candidates for appearance issues?  In addition to collecting confidential
financial disclosure forms (described below) and public comment (where appropriate), it may be
necessary on occasion to seek additional information from the potential advisory committee
member. For example, you might want to consider asking additional questions which will help
you to determine whether there is an appearance issue. Some suggested questions would be:

-------
                                 Peer Review Handbook                      Page 67
   a)  Do you know of any reason that you might be unable to provide impartial advice on the
       matter to come before the Panel or any reason that your impartiality in the matter might
       be questioned?

   b)  Have you had any previous involvement with the review document(s) under
       consideration? If so,  please identify and describe that involvement.

   c)  Have you served on previous advisory panels, committees or subcommittees that have
       addressed the topic under consideration?  If so please identify those activities.

   d)  Have you made any public statements (written or oral) on the issue?  If so, please identify
       those statements.

   e)  Have you made any public statements that would indicate to an observer that you have
       taken a position on the issue under consideration? If so, please identify those statements.

If an SGE has an appearance of a lack of impartiality, it may still be possible to make a written
determination that she or he may serve on an advisory committee or to ensure that a particular
advisory committee member or peer reviewer is recused from certain areas of a review. See your
Deputy Ethics Official/Ethics Assistant for more information.

       A Few More Ethical Considerations:

In addition to the above considerations, you should check for a few more items to ensure
compliance with Federal ethics laws:

       Restrictions on Representation:

There are two statutes (18 USC  203 and 18 USC  205) that impose related restrictions on the
outside activities of RGEs and SGEs, particularly involving activities involving the
representation of others before the Federal government.  Section 203 prohibits an employee from
receiving, agreeing to receive, or soliciting compensation for representation services before any
court or Federal agency/entity in connection with any particular matter in which the United
States is a party or has a direct and substantial interest.  Section 205 prohibits an employee from
personally representing anyone before  any court  or Federal agency or other specified Federal
entity,  in connection with any particular matter in which the United States is a party or has a
direct and substantial interest.

Fortunately, these restrictions are limited in their application to SGEs.  One of the most
significant limitations is that SGEs are restricted  only in connection with particular matters
involving specific parties.  Furthermore, the restrictions on SGEs are narrowly drawn to focus
only on those matters in which the SGE actually  participated for the Government, as well  as, in
some cases, those matters actually pending in the SGE's own agency (in our case, EPA).

-------
                                 Peer Review Handbook                      Page 68
       Emoluments Clause of the U.S. Constitution:

The Department of Justice (DOJ) originally held that SGEs may be subject to the "Emoluments
Clause" of the United States Constitution (Article I, Section 9, clause 8) which prohibits persons
who "hold office of profit or trust" in the Federal government from having any position or
receiving any payment from a foreign government.  However,  an opinion from DOJ/Office of
Legal Counsel (March 9, 2005) states that SGEs who do not have access to classified
information and who serve in purely advisory positions  do not hold an "office of profit or trust"
in the Federal government and are not subject to the Emoluments Clause. Therefore, it is
extremely unlikely that any SGE performing a peer review for EPA would be subject to the
Emoluments Clause. As always, check with your Deputy Ethics Official/Ethics  Assistant if you
need more information.

       Putting  It All Together:

The fine art of determining whether an advisory committee member or other peer reviewer has a
conflict of interest or an appearance of a lack of impartiality can be difficult, confusing and
complex.  Needless to add, it is also important that any decision that is made concerning advisory
committee members or peer reviewers be appropriately  documented and all applicable Federal
record regulations and Privacy Act regulations be strictly adhered to.

The matter of obtaining a fair and credible peer review,  as well as maintaining the credibility of
the Agency and the Agency's scientific products, is of paramount importance. Peer review
leaders are strongly encouraged to obtain peer reviewers who do not have a conflict of interest or
an appearance of a lack of impartiality. Finding a totally independent peer reviewer is a difficult
and often daunting task.  See Section 3.4.6 for additional steps to take to ensure a credible  peer
review.

For additional information on putting together an advisory panel (e.g., for highly influential
scientific assessments), please refer to the SAB brochure "Overview of the Panel Formation
Process at the Environmental Protection Agency Science Advisory Board, September 2002" at
http://www.epa.gov/sab/pdf/ec02010.pdf.

       3.4.6  What Techniques Help Ensure Disclosure and Appropriate Resolution of
             Conflicts of Interest?

Before finalizing the selection of reviewers, the Peer Review Leader should ascertain whether
each potential peer reviewer's involvement in certain activities could pose a conflict of interest
(COI) or create  an appearance of a lack of impartiality.  One way of identifying conflicts is to
ask potential reviewers about work and clients (both current and prior) that might create conflicts
or the appearance of a lack of impartiality in carrying out peer review activities.  When the peer
review process is being conducted by a contractor, the requirement for addressing peer
reviewers' possible conflicts of interest should be highlighted in the Statement of Work or the

-------
                                Peer Review Handbook                      Page 69
work ordering instrument (i.e., Work Assignment, Delivery Order, Task Order, etc.) and is a
matter that is bound by contractual clauses with the Contracting Officer as the final Decision
Maker in contracting matters.

You should take care to avoid ethical issues with respect to advisory committee members or
other peer reviewers and the work product under review. Remember, each situation is unique
and should be treated on a case-by-case basis. The following are some considerations that
should be addressed in evaluating ethical issues:

   a) Paying attention to the employment, financial, and professional affiliations of the
      participants;

   b) Exploring the issue directly with each participant before the review process takes place;

   c) Disclosing publicly at the beginning of meetings any previous involvement with the
      issue;

   d) Obtaining confidential financial disclosure forms.  In the cases of regular government
      employees (RGBs), use either the OGE Form 450 or the SF-278 Form as appropriate. In
      the case of Special Government Employees (SGEs), use EPA Form 3110-48
      (Confidential Financial Disclosure Form for Special Government Employees Serving on
      Federal Advisory Committees at the U.S. Environmental Protection Agency). All these
      must be  filed annually (with some exceptions). Remember, you can always discuss any
      conflicts of interest or appearances of a lack of impartiality with your Deputy Ethics
      Official, Ethics Assistant or Deputy Agency Ethics Official as appropriate, but it is
      important to remember that there is no "attorney-client privilege" for these discussions;

   e) Collecting additional information through public comment, additional ethics questions
      posed to potential advisory panel members and other appropriate means;

   f) In the case of non-Federal peer reviewers not on a FACA advisory panel (e.g., SAB, SAP
      or BOSC), providing them a copy of the peer review COI inquiry. This form is sent to
      each prospective peer reviewer by the Peer Review Leader (or contractor, in the case of
      contractor-run reviews) to advise them of the need to address COI issues prior to the
      actual review taking place. A follow-up contact with the Peer Review Leader (or
      contractor, in the case of contractor-run reviews) is then made to discuss any relevant
      issues.  The Peer Review Leader then documents this effort in the peer review record; this
      includes a summary provided by the contractor documenting their inquiries and efforts.

The Peer Review Leader should  ensure that the peer review COI inquiry took place and that it
appears in the peer review record.

-------
                                 Peer Review Handbook                      Page 70
However, it should be strictly understood that Federal records maintained on individuals (defined
as any item, collection, or grouping of information about an individual that is maintained by an
agency, including, but not limited to, her or his education, financial transactions, medical history,
and criminal or employment history and that contains his name, or the identifying number,
symbol, or other identifying particular assigned to the individual, such as a finger or voice print
or a photograph) are protected by the Privacy Act of 1974 (5 USC  552a - Records maintained
on individuals).  A copy can be found at http://www.usdoj.gov/04foia/privstat.htm.  This Act
states that no agency shall disclose any record which is contained in a system of records by any
means of communication to any person, or to another agency, except pursuant to a written
request by,  or with the prior written consent of, the individual to whom the record pertains.
There are certain exceptions (e.g., pursuant to certain civil or criminal law enforcement activity).

Thus, the peer review leader should ensure that any documentation in the peer review record
does not contain information subject to the Privacy Act. Furthermore, Peer Review Leaders
should ensure that any applicable EPA record-keeping procedures are followed (for details, see
EPA's National Records Management Program, http://www.epa.gov/records/).  For information
on Freedom of Information Act (FOIA) requests, please see http://www.epa.gov/foia.  If you
have any questions, be sure to ask your appropriate Office of General Counsel and/or Office
contract official(s).

Established peer review groups such as the Science Advisory Board (SAB), the Scientific
Advisory Panel (SAP) and the Board of Scientific Counselors (BOSC) provide useful models for
addressing balance and ethics issues.  Assistance in identifying conflicts of interest/appearance
of a lack of impartiality and in providing an appropriate response can be obtained from the
Deputy Agency Ethics Official (located in the Office of the General Counsel) as well as any of
the Agency's deputy ethics officials (DEOs) or Ethics Assistants. For assistance in evaluating
conflicts of interest or appearance issues in a contractual situation, obtain the involvement of the
Contracting Officer and the resources available within the Office of Acquisition Management.

Of course, ethical issues do not necessarily arise merely because a peer reviewer knows
something about the subject matter. In fact, experts with a stake in the outcome - and therefore  a
conflict or an appearance issue - may be some of the most knowledgeable and up-to-date experts
because they have concrete reasons to maintain their expertise. Such experts could be used
provided conflicts of interest are properly disclosed and appropriate waivers from the Office of
Government Ethics are obtained or, in the case of an appearance of a lack of impartiality, proper
disclosure is made. In some cases, however, the conflict or appearance issue may be so direct
and substantial as to rule out a particular expert. For instance, a potential peer reviewer may
have a client or employer with a direct financial stake in the particular specific party matter
under review, such as a Federal grant  or contract to the potential peer reviewer or his/her
employer that relates to the matter under review or the potential peer reviewer's or their
company's work on a specific chemical under review. However, review of a general
methodology that applies to numerous chemicals would not necessarily raise such a concern.

-------
                                Peer Review Handbook                     Page 71
(Note: COI language should be made part of contracts/statements of work (SOW) or purchase
orders (PO) - see Section 3.6.4.)

A Peer Review Leader may also want to adopt measures that will prevent creation of conflicts as
the peer review is underway.  Any measures contemplated that involve a contractual action must
be coordinated with the cognizant Contracting Officer. Some measures might include clauses in
a contract or purchase order that require reviewers to receive advance approval on future work,
or place limits on such work, while they are performing the current peer review. Note that at
some level these types of measures will discourage experts from serving as peer reviewers. (See
Section 3.6.4 for further information dealing with contracts and suggestions for appropriate
management controls.)

       3.4.7  Are There Constraints to Selecting Peer Reviewers?

Sometimes the schedule for a peer review is accelerated due to a court-ordered deadline or other
time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain external peer
reviewers in time to conduct a full external peer review.  It may even be impossible to conduct a
small scale internal peer review using just a few individuals.  Mechanisms for identifying and
using a small number of peer reviewers should be developed so that quick, effective peer review
can be included for even the most rapidly moving products.

Another possible constraint involves confidential business information (CBI).  There are
different definitions and types of CBI, depending on the statute that governs your action (e.g.,
Toxic Substances Control Act). To evaluate certain Agency-generated studies properly, some
peer reviewers may need access to CBI. However, unless the reviewers are Federal employees
with CBI clearance, the Agency does not have the independent authority to disclose CBI to them.
Therefore, whenever contemplating the use of outside peer reviewers, Agency  staff should
determine whether the reviewers will need access to CBI. If they do not have CBI clearance, the
Office of the General Counsel should be consulted on whether it is practical to obtain the consent
of CBI submitters  to disclose the information to peer reviewers.

Offices need to be aware of the requirements of the Federal Advisory  Committee Act (FACA)
when establishing peer review mechanisms (see Section 2.8).  Federal advisory committees that
are subject to chartering by the General Services Administration must hold meetings that are
open to the public, and have balanced membership requirements. The Office of the General
Counsel (OGC) should be consulted regarding the applicability of FACA to peer review panels.

Occasionally, you may want to appoint a non-U.S. citizen to an advisory committee. Citizens of
other countries may be appointed as representative members of an advisory committee.
However, whether a non-U.S.  citizen may be appointed as a special government employee (SGE)
is a complicated question involving the Appropriations Act.

-------
                                Peer Review Handbook                      Page 72
Section 605 of the General Provisions of the Treasury and General Government Appropriations
Act provides, with some exceptions, that no appropriations can be used "to pay the compensation
of any employee of the US Government" unless the person is a U.S. citizen or has filed a
declaration that they intend to become a U.S. citizen.  There  are a few exceptions to this
restriction.  One exception is for "nationals of those countries allied with the United States in a
current defense effort." The appropriations restriction also provides that citizens of specific
countries, refugees from certain countries, and students from China also are not subject to the
restriction.  If you want to hire a foreign national as an SGE, first contact OGC so that they can
determine whether any exception applies to the person you are considering. Generally, to
determine whether a country is allied with the U.S. in a current defense effort, OGC will consult
with the Department of State.

If the foreign nation in question does not have a qualifying defense relationship with the U.S. and
no other exceptions apply to the individual, we may still hire the person as an SGE, but they
would have to agree in writing to forgo any compensation. For an SGE serving without
compensation, we may reimburse the individual for his/her expenses (travel and per diem), but
we may not make payments for "home work time" or time spent at advisory committee meetings.
The main point is to contact OGC if a foreign national is being considered for a FACA so that
OGC can determine whether the person may be compensated.

  3.4.8   If State Employees Are Used as Peer Reviewers, Can EPA Pay Them for This
          Service?

In some cases, this may be possible. First, the Peer Review candidate should determine if the
State agency has a policy on whether its employees can perform this type of work, and whether
the State policy will allow compensation. In most instances, EPA can also pay travel expenses
(consult with your administrative staff for details). If the State person is not being paid for their
peer review services, they must sign an agreement stating that they do not expect payment (see
Section 3.6.1  - Gratuitous Services).

       3.4.9  Does the Agency Recognize Peer Reviews That External Parties Conduct
             and/or Pay For?

There may be instances where parties external to EPA will want to conduct and/or pay for a peer
review on a particular work product.  This may look benign at first blush, but is a very complex
and sensitive  situation that can raise significant concerns for conflicts of interest or an
appearance of a lack of impartiality for interested parties "paying" for a peer review of their own
work product. While the Agency cannot prevent external parties from conducting and paying for
a peer review, it is desirable that any such peer review is consistent with the intent of the
Agency's Peer Review Policy and implements the principles and guidance in this Handbook.  If
the external party submits their work product and accompanying peer review, the materials
should be treated by the Agency as anything else submitted for the Agency's evaluation (i.e.,

-------
                                 Peer Review Handbook                      Page 73
evaluation for scientific credibility and validity, as well as consistency with the provisions of the
Peer Review Handbook).

3.5    Materials for Peer Reviewers

       3.5.1  What Instructions Do You Give Peer Reviewers?

The Peer Review Leader is responsible for ensuring that peer reviewers understand their
responsibilities (see Sections 3.2 and 3.6 if a contract is involved):

   a)  Advise the Agency of actual or potential organizational or personal conflicts of interest or
       other matters that would create the appearance of a lack of impartiality (see Section
       3.4.6);

   b)  Provide written comments in specified format by the specified deadline that are
       responsive to the charge;

   c)  Comply with any requests for not disclosing draft work products to the public;

   d)  Inform on the planned extent of disclosure of names and attribution of comments.

       3.5.2  What Materials Should be Sent to Peer Reviewers?

For a peer review to be successful, peer reviewers should receive several documents at the
beginning of the  process.  Typically, the most important among these documents are the charge
letter and the  current work product. The charge letter describes what the peer reviewers are
being asked to do, and should serve to focus and structure the review.  The work product is, of
course, the material being subject to peer review.

Remember, documents should not be provided directly to a potential peer reviewer if that
reviewer is going to be working under a contract or purchase order.  In the case of a contract, the
Agency provides the work product with associated background material to be peer reviewed to
the prime contractor who in turn distributes these documents to the peer reviewers. In the case of
a purchase  order, the "charge or statement of work" must be part of the PO (purchase order) and
the provision  of any documents should be coordinated with the purchasing agent handling the
order.

Essential documentation for each peer reviewer includes:

    1)  A current copy of the work product to be peer reviewed with associated background
       material.  The work product should be of the best possible scientific/technical quality to
       ensure an adequate and  useful peer review;

-------
                                 Peer Review Handbook                      Page 74
   2)  A clear charge or statement of work, seeking informed comment on identified specific
       issues to properly focus the efforts of the peer reviewers and ensure that their individual
       efforts can be compared or contrasted;

   3)  Information concerning the process that you use for the peer review, including the due
       date of reviewer comments, the format of those responses, and a point of contact in case
       the peer reviewer has questions.  Responses should be written and submitted to the Peer
       Review Leader by an agreed upon deadline.  In certain rare cases,  oral commentary may
       be sufficient; however, in such cases, a follow-up written response for the record should
       be made;

   4)  For highly influential scientific assessments,  copies of significant public comments raised
       on scientific issues;

   5)  In some cases, Agency materials being peer reviewed will be available to the public, even
       if they are marked as drafts.  For example, all materials reviewed by the SAB and the
       FIFRA SAP are available. Agency managers may also decide that a broad accessibility
       has benefits for the Agency. Offices should include a disclaimer on  draft work products
       that are submitted for peer review (See Section 1.3.4). In other cases, confidentiality
       needs to be maintained. In these cases, each  peer reviewer should be informed of the
       need for confidentiality with regard to the release of Agency products that are stamped as
       "DRAFT" or "DRAFT - Do Not Cite, Quote, or Release."  Premature release of draft
       Agency products, views, or positions may be inappropriate and can be damaging to the
       credibility of the Agency or the peer reviewer.  Although it may not have legal effect,
       such language should be included in the charge to the peer reviewers when necessary.
       Other mechanisms to use in discouraging premature release include a disclaimer that
       appears in a separate section at the front of the document and creating the document with
       watermarks clearly delineating DRAFT status (or a header or footer that states DRAFT
       status) on every page.  In addition, in any solicitation for peer reviewers, the necessity for
       confidentiality and the non-release of materials should be emphasized.

Useful, but not critical materials that may be sent to peer reviewers include:

   1)  The name, address, and phone and fax numbers, and/or e-mail address of all peer
       reviewers working on the specific review;

   2)  A bibliography and/or any particularly relevant scientific articles from the literature;

   3)  A work product that  has line numbering added in the margin for ease in providing and
       referencing comments.

Peer Reviewers should be given what is necessary to complete their task - they should not be
overburdened with excess material.

-------
                                Peer Review Handbook                      Page 75
       3.5.3  How Closely can EPA Interact with Peer Reviewers During the Review?

   a)  When EPA Conducts the Peer Review - The Peer Review Leader normally has
       administrative contacts with the reviewers during the development and conduct of the
       peer review. In some cases (e.g., SAB peer review), peer reviewers may also receive a
       briefing on the product to be peer reviewed. Otherwise, the Peer Review Leader and
       other EPA staff should not contact the reviewers during the course of the review. Such
       contact can lead to perceived conflicts or inappropriate direction that could compromise
       the independence of the review.

   b)  When the Contractor Conducts the Peer Review - If peer review is conducted under a
       contract mechanism, EPA should limit direct contact to the prime contractor's designated
       representative and should not have general  contact and direction to the contractor's staff
       or peer reviewers (sub-contractors).  Note, when a peer review is conducted under a
       contract, there are constraints where EPA staff are prohibited from contacting peer
       reviewers to avoid personal services arrangements. Personal services contracts exist
       when the nature of the relationship between the contractor and EPA can be characterized
       as an employer-employee relationship. Any communications with peer reviewers should
       be coordinated through the prime contractor.

3.6    Peer Review Services

A range of peer review services are available to the Agency including internal, external
(gratuitous  services, contracts, purchase order), and Special Government Employee (SGE)
mechanisms.  The mechanism  selected is generally based on the nature of the scientific or
technical work product.

Peer review services are "advisory and assistance services", as defined in Federal Acquisition
Regulation  (FAR) 37.201.  These types of services require special approvals and management
oversight. See Figure  5, Item B Management Approvals, in Chapter 7 of the Contracts
Management Manual (CMM) for current approval  levels.  The CMM is available on the intranet
at http://intranet.epa.gov/oamintra/policy/cmm.pdf.

       3.6.1  What are Gratuitous Services for Peer Review?

The provision of peer review products or services to EPA without compensation is provided as
so-called "gratuitous"  services. If a person wishes to perform peer review services for EPA
without compensation, EPA must ask them to sign an agreement whereby the person agrees to
provide the prescribed peer review services  as gratuitous services, with no expectation of
receiving compensation for these services from EPA. An agreement, such as EPA Form 3100-14
or a gratuitous services contract, must be executed  because the Antideficiency Act (31 USC 
1342) prohibits the Agency's acceptance of "voluntary" services.  For a copy of EPA Form
3100.14, see http://intranet.epa.gov/oas/fmsd/forms/f2.htm. "Voluntary services" are services

-------
                                 Peer Review Handbook                      Page 76
provided to EPA without an agreement in advance that such services are provided at no cost to
EPA. Note that persons cannot waive compensation (i.e., agree to provide gratuitous services)
for which there is a statutory right to payment, unless a law permits the waiver.  For situations
concerning State employees, see Section 3.4.9.  Examples of laws that permit services without
compensation are 5 USC  3109 for experts or consultants and 5 USC  3111 for student
volunteers. If you are dealing with  such gratuitous services, contact the appropriate OGC
attorney for advice.

       3.6.2   Can You Use a Contract to Obtain Peer Review Services?

The Agency may obtain peer review services through a contract or purchase order. Typically,
peer review services would be available under a "mission contract," i.e., a contract with a broad
scope covering a variety of services. It is also possible to have a contract or purchase order
solely for peer reviews.

A contract is awarded if the cost is over $100,000. If the cost is under $100,000, then a purchase
order is issued. For assistance in preparing the necessary pre-award contract documents, consult
Chapter 7 of the CMM.

For assistance in preparing simplified acquisition packages for purchase orders, the Office of
Acquisition Management (OAM) has a guide called SAME: Simplified Acquisition Made Easy,
which is available on the intranet at http://intranet.epa.gov/oamintra/policy/index.htm.

       3.6.3   How Do You Write a Statement of Work for Contracts?

For general assistance in preparing the Statement of Work  (SOW), consult Chapter 11 of the
CMM. The SOW should clearly specify that the contractor is responsible for preparing peer
review evaluations and set forth guidelines  for the peer review of scientific or technical
documents. The contractor may perform the peer review with in-house staff, subcontractors or
consultants. Any guidelines for performing peer reviews to ensure soundness and defensibility
should be developed by the Program Office and made part of the contract. The  contractor would
then ensure that the peer reviews adhere to  the guidelines.

The SOW should include the list of questions that EPA wants the contractor to include in the
charge to the peer reviewers. While the contractor will be the one that will prepare and  send the
formal charge to the peer reviewers, EPA should provide the list of questions to the contractor
(see Section 3.2 for general discussion on charge to peer reviewers).

The SOW cannot simply define the role of the prime contractor as arranging for the services of
others to perform peer reviews and logistics for meetings.  Unless the prime contractor is clearly
tasked with responsibility for performing peer reviews, individual peer reviewers' fees and
associated travel expenses are not payable under the contract.

-------
                                 Peer Review Handbook                      Page 77
EPA may pay for the reviewer's comments or evaluation, and also for attendance at a meeting
with the Agency and other reviewers to discuss the results of the peer review.  If the SOW calls
for the preparation of comments or an evaluation, and specifies a meeting with the Agency and
other peer reviewers to discuss the results of the peer review, payment is appropriate.  The peer
reviewer's attendance at the meeting would then be part of contract performance.

Example statements of work are found in Appendix F.

       3.6.4  What are Some Management Controls for Contracts?

Management controls ensure several things:

   1)   the contractor does not perform inherently governmental activities (IGA);

   2)   the contractor's work is free from conflicts of interest;

   3)   if provided to the contractor, confidential business information or sensitive information is
       appropriately safeguarded, and;

   4)   improper relationships with  contractor employees and subcontractors are avoided.

Agency and Federal acquisition regulations prohibit  contractors from performing IGA. The
Office of Management and Budget Circular A-76,  revised May 29, 2003, defines "inherently
governmental activities" as activities that are so intimately related to the public interest as to
mandate performance by government personnel. These activities require the exercise  of
substantial official discretion in the  application of government authority and/or in making
decisions for the government.

The Agency cannot award contracts for IGA; however, certain functions, such as those involved
in peer review, may  cross over into  IGA if not properly managed.  With peer review services, the
Agency is seeking only a contractor's recommendations, advice or analysis of a document, not a
determination of what the policy should be. Determining Agency policy is an IGA. Agency
officials make the official Agency decision regarding acceptability and/or quality of the
document.  To ensure that Agency officials are not improperly influenced by recommendations
in the peer review, the contract should include management controls.  One possible control
would be to direct the peer reviewers to submit with  their evaluations or comments a description
of the procedures used to arrive at their recommendations, a summary of their findings, a list of
sources relied upon, and make clear and  substantiate the methods and considerations upon which
their recommendations are based. To the extent possible, the contract should set forth any
guidelines or criteria for performance of the peer review. Agency officials should document
their evaluations of the quality and validity of the peer review. You should ensure that there is a
clear record of their review of the contractor's work and that Agency personnel made the final

-------
                                Peer Review Handbook                      Page 78
decisions. Such records of review could include notes from reviews of draft and final documents
by Agency personnel, minutes from progress meetings with contractors, and so forth.

Conflict of Interest (COI) - Another important factor is that the objectivity of the peer review
should not be improperly influenced or undermined by the contractor performing the review. To
identify and avoid or mitigate actual or potential COI, the contract should include controls.
Inclusion of Agency-developed individual and organizational COI clauses in the contract or
purchase order is critical for peer review services.  Usually, the EPA contracting officer (CO)
will include COI solicitation provisions and contract clauses as a matter of course without
involvement by the EPA Project Officer.  As a safeguard, the Project Officer should:

   a)  Highlight the conflict of interest requirements in the Statement of Work (SOW) for the
       procurement of the peer review services;

   b)  Review the solicitation/contract to make sure that the appropriate conflict of interest
       clauses have been included, particularly EPAAR clause 1552.209-73, Notification of
       Conflicts of Interest Regarding Personnel;

   c)  Work with the CO to develop contract-specific language regarding the  peer review to
       assist the contractor with identifying actual or potential conflicts of interest that might
       impair the objectivity of peer reviewers.  For example, a contractor may be advised to
       consider the following questions and issues when determining if a proposed peer
       reviewer may have an actual or potential conflict of interest:

         1)    The sources and nature (obtained from a brief description of the work) of any
              compensated and non-compensated employment of the panel member and their
              spouse, including any government service, for the preceding two years.

         2)    The sources of research support and project funding, including from any
              government source, for which the panel member served as the Principal
              Investigator, Significant Collaborator, Project Manager or Director during the
              preceding two years.  For the panel member's spouse,  a general description of
              research and project activities in the preceding two years.

         3)    The compensated consulting activities of the panel member during the preceding
              two years, including the names of clients if compensation provided 15% or more
              of annual compensation. For the panel member's spouse, a general description of
              consulting activities for the preceding two years.

         4)    The sources of compensated expert witness activities of the panel member and a
              brief description of the issue and testimony during the preceding two years. For
              the panel member's spouse, a general description of expert testimony provided in
              the preceding two years.

-------
                                 Peer Review Handbook                      Page 79
         5)    The assets, including stocks, bonds, real estate, business, patents, trademarks, and
              royalties, for the panel member, their spouse and dependent children.
              Specifically, the financial holdings that collectively had a fair market value
              greater than $15,000 at any time during the preceding two year period (excluding,
              for example, well-diversified mutual funds, money market funds, treasury bonds
              and personal residences).

         6)    The liabilities over $10,000 owed by the panel member, their spouse, and
              dependent children at any time in the preceding one year (excluding, for example,
              a mortgage on a personal residence, home equity loans, automobile and consumer
              loans).

         7)    A brief description of any public statements and/or positions of the panel member
              on, or closely related to, the matter under review.

         8)    A brief description of any previous involvement of the panel member with the
              development of the document (or review materials) the individual has been asked
              to review (including previous peer reviews).

         9)    A brief description of any other information that might reasonably raise a question
              about an actual or potential personal conflict of interest or bias including any
              financial benefit that might be gained by the panel member (or anyone whose
              interests are imputed to the panel member) as a result of the outcome of the
              review.

Confidential Business Information (CBI)/Privacy Act Protected Information and Other
Sensitive Information - When peer reviewers are not employees of the United States
Government, it is unlikely that EPA will have authority to give reviewers access to confidential
business information in the absence of consent for such disclosure by the CBI  submitter.
Therefore, all documents provided to non-Federal reviewers must be screened for information
claimed as CBI.  Even where business information has not been explicitly claimed as CBI, if it is
of a kind where the submitter might be expected to object to its release, prior to release the
submitter must be asked whether it wishes to assert a claim, unless the submitter has previously
been informed that failure to assert a CBI claim may result in disclosure without notice.  If the
contractor should have access to CBI for your peer review, notify the CO so that the appropriate
clauses can be included in the contract or purchase order. These clauses  will clearly identify any
required procedures or processes prior to release of any protected information, including any
requirements for confidentiality agreements, as well as limits on use and disclosure of the data by
contractor personnel.

Personal services - Unless your organization has statutory authority to engage in personal
service contracts, avoid treating contractor employees as Agency employees. For example,
deciding who to hire, supervising, and assigning tasks to contractor employees are personal

-------
                                 Peer Review Handbook                      Page 80
services. For additional information, program officials should consult EPA Order 1901.1 A, Use
of Contractor Services to Avoid Improper Contracting Relationships.  It is very important that all
Agency employees follow this order when interacting with contractors. The Order is available
on the intranet at http://epawww.epa.gov/oamintra/policy/eo 19011 a.pdf.

To avoid these improper relationships, program officials should write well-defined SOWs.  The
SOWs should set forth in detail a description of the work to be performed independently,
including the manner in which it will be evaluated. The SOW should set forth what work is to be
performed not how the work is to be performed.  Technical direction may be used to clarify
ambiguous provisions to ensure efficient and effective contractor performance, and is not
considered supervision  or assignment of tasks.

       3.6.5   Can You Identify and/or Select Peer Reviewers When Using a Contract?

The prime contractor is responsible for selecting who will perform the peer review (EPA's  doing
so may invoke FACA; see Section 2.8), whether these individuals work directly  for the
contractor or are consultants or subcontractors to the prime contractor.  Interfering in this process
is a violation of Federal and Agency acquisition regulations.  In accordance with FAR clause
52.244-2 Subcontracts,  contractors may be required to obtain Contracting Officer (CO) consent
or approval of subcontractors and the CO generally seeks the input of the work assignment
manager (WAM - the WAM may be the Peer Review Leader in many cases) before  approving
the use of subcontractors.

EPA can establish criteria for the sort of individuals that might participate on a peer review
panel.  However, the Agency should not be involved in the selection of individual peer
reviewers, and should avoid commenting on the contractor's selection of peer reviewers other
than to determine whether the panel, once selected, meets the criteria established.  EPA may
identify a pool of qualified subcontractors and consultants to the prime contractor (listed in
alphabetical order).  If a list is provided, note on the list that this is a suggested list and other
qualified candidates may exist who are not on the list. This is to prevent the impression that the
prime contractor can only choose someone on our list. The prime contractor is required to
include a conflicts of interest clause substantially similar to the conflicts of interest clause in the
primary contract in its subcontract to the peer reviewer.

       3.6.6   How is Travel Handled with Contracts or Purchase  Orders?

Funds obligated on a contract or purchase order are available to pay for the costs of producing
the peer review including the travel costs and fee of the  peer reviewer.

EPA acquires peer reviews through simplified acquisitions issued directly to peer reviewers or
through contracts with companies, which provide the peer review services. By issuing a
purchase order or awarding a contract for peer review services, EPA may pay not only for the
peer review services/comments, but also for participation in a meeting with the Agency and other

-------
                                Peer Review Handbook                      Page 81
reviewers to discuss comments. The scope of work of the contract must require the contractor or
individual peer reviewer, as appropriate, to provide peer review services and indicate whether the
contractor or peer reviewer will be required to discuss a specific peer review work product with
the Agency and/or with other peer reviewers. Participation in a meeting to discuss a peer review
work product would then be part of the contract's performance.  Thus, the contract may serve as
the mechanism to pay for peer review services and associated travel expenses to provide
comments to EPA.

       3.6.7  How is Travel Handled with Special Government Employees?

The term Special Government Employee (SGE) is defined in 18 USC  202(a) as an officer or
employee of an agency who performs temporary duties, with or without compensation, for not
more than 130 days in a period of 365 days, either on a full-time or intermittent basis.

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through
the issuance of invitational travel orders (5 USC  5703).  These invitational travel and per diem
expenses should be charged to an appropriate EPA travel account. The Federal Travel
Regulations govern the invited travelers reimbursement.

Members of the SAB, SAP, and other FACA advisory committees are  often classified  as SGEs.
It is not appropriate to reimburse travel or per diem expenses of advisory committee members
(SGEs) through a contract.

-------
                                Peer Review Handbook                     Page 82
                      4.  Completing A Peer Review
4.1    Overview
Performance of the formal peer review of the work product is not the final stage in the product's
development. Rather, it is an important stage in developing the work product, with the final
version (with all comments addressed) representing the true end of the peer review. As a result,
the peer review process closes with four major activities:

      a)     Evaluating peer review comments and recommendations

      b)  Utilizing peer review comments for completing the final document,

      c)  Organizing and maintaining a record of the peer review, and

      d)  Completing the peer review record in the Science Inventory.

Careful attention to all of these elements, singly and together, assures a credible and transparent
peer review process.  Conversely, inattention can nullify the peer review effort.  A well-planned
peer review applied to a reasonable quality starting work product, followed by a responsible
utilization of peer review suggestions in the final product assures a credible and defensible
product for use in Agency decision-making.

The peer review of a work product is not complete until the peer review comments are
incorporated into the final version, or reasons are stated why such comments are not to be
incorporated. However, for the purposes of documentation to the Agency's Science Inventory
only, the work product can be designated in the Science Inventory as having completed peer
review once the peer review report is available on-line.

The peer review record is complete only when  it contains a copy of the final work product (when
there is one) that addresses the peer review comments and a copy of the documentation outlining
why comments were not incorporated, if any were not incorporated . (See Sec. 2.5.3 for a list of
the minimum documentation for the peer review record.)

4.2   Final Work Product

      4.2.1  How Do You Incorporate Peer Review Comments into the Final Work
             Product?

The Peer Review Leader should carefully evaluate and analyze all peer review comments and
recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers

-------
                                Peer Review Handbook                     Page 83
simplifies organizing and analyzing comments. Also, any other issues that are raised should be
identified and evaluated.

The validity and objectivity of the comments should be evaluated. Analyses may include
consultation with other experts and staff within the Office and Agency. Adequate documentation
is needed to show that comments are accepted or rejected. The documentation can be brief, but
should address the legitimate, valid comments, whether accepted for incorporation in the final
work product or not. The peer review record should contain a document describing the Agency's
response to the peer review comments.  The Agency's response to the peer review report for
highly influential scientific assessments should be posted on the Science Inventory.

The Peer Review Leader should brief the Decision Maker (including all appropriate managers in
the Peer Review Leader's chain of command) on the  charge, profile of peer reviewers, the peer
review comments, and provide a proposal on how to  address the peer review comments.  It is the
responsibility of the Peer Review Leader to obtain Decision Maker approval of the approach to
addressing the peer review  comments. The Peer Review Leader should clearly identify for the
Decision Maker any major  peer review comments that will not be accepted and why, as well as
any controversial comments that should be resolved.

Comments that have significant impact on time, budgetary, or resource needs are particularly
important and should be evaluated in consultation with
management. These comments may lead to allocation of
additional resources and a revised schedule for the
     ,  ..    f.,      ,     ,   , -f         .  f   ,,         peer review comments are incorporated
completion or the work product it appropriate tor the        .     ,   _             ,       ^
r.   ,    ,     ,  ,    ,   ,  ,                             into the final work product.
tmal work products intended use.                                J         ^
                                                       The peer review is not complete until the
       4.2.2  How Might Peer Review Comments
             Impact Your Work Product?

Peer review, when appropriate, enhances the quality of the information EPA disseminates by
ensuring the information that governs EPA's mission is reliable and accurate (objective) and is
appropriate for the intended use (utility). A variety of changes to a work product may result
from the comments provided during peer review:

   a)  Peer review comments and recommendations may entail significant impacts on the
       planned project schedule, budget, or other resource needs.  Management decisions to
       adjust one or more of these areas may be appropriate.
   b)  The substantive issues or concerns expressed by peer reviewers may suggest that wider
       scientific and technical consultation is needed to ensure the adequacy of the work product
       relative to its intended use.  If the Agency agrees with the reviewers, additional resources
       and an extended delivery schedule may be necessary.

-------
                                 Peer Review Handbook                      Page 84
   c)  The peer review comments and recommendations on a final product may provide a basis
       for bringing the associated project to closure. At times the Agency and its partners and
       stakeholders may not be able to come to consensus about a scientific or technical aspect
       of a work product. The scientific or technical issue in question can be noted in the charge
       to the reviewers as an area for which the Agency is specifically seeking peer review.

       4.2.3  What Should the Final Work Product Say About the Peer Review Process?

If the product has been peer reviewed, you should describe the peer review in the document.
Frequently, this will be part of a description of the process of developing the product along with
descriptions of other public participation processes. It can be brief and does not need to describe
the process or discuss the peer-review comments  in great detail.  The description can be included
in an introduction, preamble, or appendix.  For influential scientific information and highly
influential scientific assessments that support rulemaking, the peer review should be discussed in
the preamble.

When there are  significant peer-review comments, and particularly if they are not being
accepted, the document should generally discuss the issue and describe the reasons for the
Agency's choices in the appropriate sections of the document. The level of detail that is
provided is a matter of judgment and should reflect the importance and degree of controversy of
the issue.

If a scientific or technical work product has not been peer-reviewed, this fact should be noted in
the document, perhaps in an introduction or description of its scope. This section should briefly
indicate the reasons that peer review was not conducted.

Derivative products  of scientific or technical products (such as fact sheets, press releases, and
brochures) do not need to discuss whether the underlying products were peer reviewed.

       4.2.4  Can the Identity of Peer Reviewers be Kept Anonymous?

Generally, no.  At a minimum, the names and affiliations of peer reviewers should be listed in
the peer review report  for influential scientific information and highly influential scientific
assessments. Depending on the peer review process used, the reviewers names and affiliations
may  be made available to the public prior to the review commencing. Release of any reviewer
information retrieved by a personal identifier must be performed in accordance with the Privacy
Act,  5 USC  552a as amended,  and as interpreted in OMB implementing guidance, 40 FR
28,948 (July 9,  1975).

Though the identity of reviewers becomes known at some point in the peer review process, the
attribution of specific comments to any given reviewer is kept confidential to the extent possible.
In the ordinary course  of events, you can often discuss  comments received without attributing the
comments to a specific reviewer. However, if a matter goes to litigation, the litigating parties

-------
                                 Peer Review Handbook                      Page 85
can discover the names of anyone who contributed to a Federal product, including peer
reviewers. Also, in most cases, though specific comments are not attributed to specific peer
reviewers, the names and  affiliations of the peer reviewers are included in the Peer Review
Report, which is included in the Science Inventory. Therefore, it is not possible to totally shield
peer reviewers.  As noted  previously, peer reviewers should be aware of EPA plans for releasing
their names, credentials, and the extent of attribution of comments.

If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review Leader
should inform the peer reviewer of the above possible eventualities.  The Agency will in the
ordinary course of events  attempt to maintain the confidentiality of the peer reviewers and their
attributable comments from public disclosure, but it is recognized in many instances, for
example open public meetings, litigation, and Agency responses to Freedom  of Information Act
(FOIA) requests, this can't be assured. Remember, the Agency is committed to working with the
fullest possible transparency to the public (except where statutorily constrained, such as with
confidential business information). It is recognized that this may be a deterrent to possible peer
reviewers, but this is a reality that has to be understood.

4.3   Completing the Peer Review Record

      4.3.1  How Do You Complete the Peer Review Record?

Once the Peer Review Leader has completed the peer review  and the final work product (where
one is prepared), the peer  review record is brought up to date  and then archived according to that
organization's procedures (see Section 4.3.2). The peer review record should be indexed and
maintained in an organization's archive (repository).  The location of the peer review record
should be readily identifiable so interested parties can locate and obtain materials easily and
quickly. The peer review record should be placed in any associated  established public docket, if
so required, in addition to the organizational archive.  As a courtesy, a copy of the revised work
product may be sent to the peer reviewers for information.

The Peer Review Leader should collect the materials identified in Section 2.5.3 and submit them
for archiving. Finally, the Agency Science Inventory entry for the project should be completed
and updated, including attachment of electronic copies of the  documents.
       4.3.2  Where Should the Peer Review Records be Kept, and for How Long?

Though the Science Inventory will become the primary public active record and archive for the
peer review record, the Peer Review Leader is responsible for maintaining the on-site record.
During the active conduct of the peer review, the Peer Review Leaders maintain the peer review
record themselves until the peer review is totally completed.  Minimally, the file should be
maintained until one year after the completed peer review is reported in the next annual
reporting. After that, the peer review record should be maintained for a "reasonable period of

-------
                                 Peer Review Handbook                      Page 86
time."  Establishment and maintenance of the archive where the peer review records ultimately
reside are an organization's responsibility (i.e., not that of an individual program manager or
Peer Review Leader).

Generally, to allow flexibility, individual Offices and Regions should decide the appropriate
level of organizational responsibility and how to ensure the record will meet "routinely
available."  The peer review record may be kept with other records relating to the overall project,
as long as it is easily and separately identifiable.  The peer review record should be maintained in
accordance with the Agency's record-keeping schedule for such records. One long term
archiving mechanism may be the formal archiving at the Federal Records Center; the Science
Inventory will also maintain an electronic copy of the record.

       4.3.3  Is Information Regarding a Peer Review Subject to Release under FOIA?

Yes, it is subject to release if EPA receives a Freedom of Information Act (FOIA) request, unless
the peer review information meets the criteria for an exemption under the FOIA.  (See
http ://www. epic. org/open_gov/foia/us_foia_act. html).

-------
        Peer Review Handbook          Page A-l
       Appendix A




EPA's Peer Review Policy
       January 31, 2006

-------
Peer Review Handbook                   Page A-2

-------
                                  Peer Review Handbook                      Page A-3
          ^        UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                                        3 1 20Df
MEMORANDUM

SUBJECT: Peer Review Program

TO:    Assistant Administrators
              General Counsel
              Inspector General
              Associate Administrators
              Regional Administrators
              Staff Office Directors

       We have made tremendous strides in improving our peer review program at EPA
since the Agency's Peer Review Policy was reaffirmed in 1994. Today I am updating the
Peer Review Policy to emphasize the critical role that peer review plays in our efforts to
ensure that EPA's decisions rest on sound, credible science and data (see attached policy
statement).

       Peer review at EPA takes several different forms, ranging from informal
consultations with Agency colleagues who were not involved in developing the product
to the formal, public processes of the Science Advisory Board (SAB) and the FIFRA
Scientific Advisory Panel (SAP). In any form, peer review assists EPA's work by
bringing independent expert experience and judgment to bear on issues before the
Agency to the benefit of the final product.

       In  1994 the Science Policy Council (SPC) and its Steering Committee were asked
to undertake an initiative to ensure that EPA has a comprehensive Agency-wide program
for implementing its Peer Review Policy.  I commend the SPC for its diligence and
success in  meeting this objective. The SPC has made substantial improvements in the
Peer Review Handbook, sponsored training of Agency managers and staff in peer review
procedures, identified scientific and technical work products that merit peer review, and
developed a publicly available data base of the peer review activities across the Agency.
EPA has a strong and well-recognized peer review program as  a direct result of these
efforts.
                              VS.-;.,-' Jl,,| 'I-- -,(""!.) -rlUKA'.'. ..

-------
                                   Peer Review Handbook                      Page A-4
       In 2004 the Office of Management and Budget (OMB) issued a "Final
Information Quality Bulletin for Peer Review" that contains provisions for peer review at
all federal agencies. The OMB Bulletin applies to influential scientific information and
highly influential scientific assessments. The SPC has updated the Agency's Peer
Review Handbook, in part  to incorporate the provisions of the OMB Bulletin, and to
reflect the experience gained from implementing the program over the last decade.

       I ask that you continue to implement fully the provisions of our Peer Review
Policy, and I expect the Science Policy Council to continue its role in overseeing and
strengthening EPA's peer review program. We must ensure that our decisions are based
on the highest quality, peer-reviewed scientific and technical information.
Attachment

cc:     Science Policy Council
       Science Policy Council Steering Committee

-------
                                 Peer Review Handbook                     Page A-5
                     PEER REVIEW AND PEER INVOLVEMENT
              AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY

       This document establishes the policy of the United States Environmental Protection
Agency (EPA) for peer review of scientifically and technically based work products, including
economic and social science products, that are intended to inform Agency decisions. Peer
review, a form of peer involvement, is one process through which EPA staff augment their
capabilities by inviting independent subject-matter experts to provide objective evaluation of the
work product.

PEER REVIEW

       EPA strives to ensure that the scientific and technical bases of its decisions meet two
important criteria: (1) they are based upon the best current knowledge from science, engineering,
and other domains of technical expertise; and (2) they are credible. Peer review, a process based
on the principles of obtaining the best technical and scientific expertise with appropriate
independence, is central to sound science and helps the Agency meet these important criteria.
Peer review occurs when scientifically and technically based work products are evaluated by
relevant experts who were not involved in creating the product. Properly applied, peer review
not only enriches the quality of work products but also adds a degree of credibility that cannot be
achieved in any other way. Furthermore, peer review early in the development of work products
in some cases may conserve future resources by steering the development along the most
efficacious course.

       Peer review generally takes one of two approaches:

         Internal, in which the reviewers are independent experts from inside EPA.

         External, in which the reviewers are independent experts from outside EPA.

POLICY STATEMENT

       Peer review of all scientific and technical information that is intended to inform or
support Agency decisions is encouraged and expected. Influential scientific information,
including highly influential scientific assessments, should be peer reviewed in accordance with
the Agency's Peer Review Handbook. All Agency managers are accountable for ensuring that
Agency policy and guidance are appropriately applied in determining if their work products are
influential or highly influential, and for deciding the  nature, scope, and timing of their peer
review. For highly influential scientific assessments, external peer review is the expected
procedure.  For influential scientific information intended to support important decisions, or for
work products that have special importance in their own right,  external peer review is the
approach of choice.  Peer review is not restricted to the nearly final version of work products; in
fact, peer review at the planning stage can often be extremely beneficial.

-------
                                 Peer Review Handbook                     Page A-6
LEGAL EFFECT

       This policy statement does not establish or affect legal rights or obligations. Rather, it
confirms the importance of peer review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the policy. On a continuing
basis, Agency management is expected to evaluate the policy as well as the results of its
application throughout the Agency and undertake revisions as necessary. Therefore, the policy
does not stand alone; nor does it establish a binding norm that is finally  determinative of the
issues addressed.

IMPLEMENTATION

       The Science Policy Council is responsible for overseeing Agency-wide implementation
of this policy, including: promoting consistent interpretation; assessing Agency-wide progress;
developing recommendations for revisions of the policy as necessary; and issuing the Peer
Review Handbook, which provides additional information and procedures on implementing this
policy.  Assistant Administrators, Regional Administrators, and other senior managers remain
ultimately responsible for ensuring the appropriate application of Agency policy and guidance in
identifying work products subject to peer review, determining the type and timing of such
review,  documenting the process and  outcome of each peer review, ensuring that the Science
Inventory is kept current, and otherwise implementing the policy within their organizational
units.

       The policy is effective immediately.
          APPROVED:L^      '}     L	V^PfiATF     *W  M  2006
          STEPHEN L JOHN SON ADMI!

-------
             Peer Review Handbook          Page B-l
             Appendix B
OMB Information Quality Bulletin for
     Peer Review and Preamble

-------
                                 Peer Review Handbook                      Page B-2
       OFFICE OF MANAGEMENT AND BUDGET
Final Information Quality Bulletin for Peer Review

INTRODUCTION

This Bulletin establishes that important scientific information shall be peer reviewed by qualified
specialists before it is disseminated by the federal government. We published a proposed
Bulletin on September 15, 2003. Based on public comments, we published a revised proposal
for additional comment on April 28, 2004.  We are now finalizing the April version, with minor
revisions responsive to the public's comments.

The purpose of the Bulletin is to enhance the quality and credibility of the government's
scientific information. We recognize that different types of peer review are appropriate for
different types of information.  Under this Bulletin, agencies are granted broad discretion to
weigh the benefits and costs of using a particular peer review mechanism for a specific
information product.  The selection of an appropriate peer review mechanism for scientific
information is left to the agency's discretion. Various types of information are exempted from
the requirements of this Bulletin, including time-sensitive  health and safety determinations, in
order to ensure  that peer review does not unduly delay the release of urgent findings.

This Bulletin also applies stricter minimum requirements for the peer review of highly influential
scientific assessments, which are a subset of influential scientific information.  A scientific
assessment is an evaluation of a body of scientific or technical knowledge that typically
synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional
judgment to bridge uncertainties in the available information. To ensure that the Bulletin is not
too costly or rigid, these requirements for more intensive peer review apply only to the more
important scientific assessments disseminated by the federal government.

Even for these highly influential scientific assessments, the Bulletin leaves significant discretion
to the agency formulating the peer review plan. In general, an agency conducting a peer review
of a highly influential scientific assessment must  ensure that the peer review process is
transparent by making available to the public the  written charge to the peer reviewers, the peer
reviewers' names, the peer reviewers' report(s), and the agency's response to the peer reviewers'
report(s). The agency selecting peer reviewers must ensure that the reviewers possess the
necessary expertise. In addition, the agency must address  reviewers' potential conflicts of
interest (including those stemming from ties to regulated businesses and other stakeholders) and
independence from the agency. This Bulletin requires agencies to adopt or adapt the committee
selection policies employed by the National Academy of Sciences (NAS)14 when selecting peer
reviewers who are not government employees.  Those that are government employees are subject
to federal ethics requirements.   The use of a transparent process, coupled with the selection of
14 National Academy of Sciences, "Policy and Procedures on Committee Composition and Balance and Conflicts of
Interest for Committees Used in the Development of Reports," May 2003:  Available at:
http://www.nationalacademies.org/coi/index.html.

-------
                                 Peer Review Handbook                      Page B-3
qualified and independent peer reviewers, should improve the quality of government science
while promoting public confidence in the integrity of the government's scientific products.

PEER REVIEW

Peer review is one of the important procedures used to ensure that the quality of published
information meets the standards of the scientific and technical community. It is a form of
deliberation involving an exchange of judgments about the appropriateness of methods and the
strength of the author's inferences.15 Peer review involves the review of a draft product for
quality by specialists in the field who were not involved in producing the draft.

The peer reviewer's report is an evaluation or critique that is used by the authors of the draft to
improve the product. Peer review typically evaluates the clarity of hypotheses, the validity of the
research design, the quality of data collection procedures, the robustness of the methods
employed, the appropriateness of the methods for the hypotheses being tested, the extent to
which the conclusions follow from the analysis,  and the strengths and limitations of the overall
product.

Peer review has diverse purposes.  Editors of scientific journals use reviewer comments to help
determine whether a draft scientific article is of sufficient quality, importance, and interest to a
field of study to justify publication.  Research funding organizations often use peer review to
evaluate research proposals.  In addition, some federal agencies make use of peer review to
obtain evaluations of draft information that contains important scientific determinations.

Peer review should not be confused with public comment and other stakeholder processes. The
selection of participants in a peer review is based on expertise, with due consideration of
independence and conflict of interest. Furthermore, notice-and-comment procedures for agency
rulemaking do not provide an adequate substitute for peer  review, as some experts - especially
those most knowledgeable in a field  may not file public  comments with federal agencies.

The critique provided by a peer review often suggests ways to clarify assumptions, findings, and
conclusions.  For instance, peer reviews can filter out biases and identify oversights, omissions,
and inconsistencies.16 Peer review also may  encourage authors to more fully acknowledge
limitations  and uncertainties.  In some cases, reviewers might recommend major changes to the
draft, such  as refinement of hypotheses, reconsideration of research design, modifications of data
collection or analysis methods, or  alternative conclusions.  However, peer review does not
always lead to specific modifications in the draft product.  In some cases, a draft is in excellent
shape prior to being submitted for review.  In others, the authors do not concur with changes
suggested by  one or more reviewers.
15 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving
Regulatory Decision Making. Carnegie Commission, New York, 1993: 75.
16
  William W. Lowrance, Modern Science and Human Values. Oxford University Press, New York, NY 1985: 85.

-------
                                 Peer Review Handbook                      Page B-4
Peer review may take a variety of forms, depending upon the nature and importance of the
product. For example, the reviewers may represent one scientific discipline or a variety of
disciplines; the number of reviewers may range from a few to more than a dozen; the names of
each reviewer may be disclosed publicly or may remain anonymous (e.g., to encourage candor);
the reviewers may be blinded to the authors of the report or the names of the authors may be
disclosed to the reviewers; the reviewers may prepare individual reports or a panel of reviewers
may be constituted to produce a collaborative report; panels may do their work electronically or
they may meet  together in person to discuss and prepare their evaluations; and reviewers may be
compensated for their work or they may donate their time as a contribution to science or public
service.

For large, complex reports, different reviewers may be assigned to different chapters or topics.
Such reports may be reviewed in stages, sometimes with confidential reviews that precede a
public process of panel review.  As part of government-sponsored peer review, there may be
opportunity for written and/or oral public comments on the draft product.

The results of peer review are often only one of the criteria used to make decisions about journal
publication, grant funding, and information dissemination.  For instance, the editors of scientific
journals (rather than the peer reviewers) make final decisions about a manuscript's
appropriateness for publication based on a variety of considerations.  In research-funding
decisions, the reports of peer reviewers often play an important role, but the final decisions about
funding are often made by accountable officials based on a variety of considerations.  Similarly,
when a government agency sponsors peer review of its own draft documents, the peer review
reports are an important factor in information dissemination decisions but rarely are the sole
consideration.  Agencies are not expected to cede their discretion with regard to dissemination or
use of information to peer reviewers; accountable agency officials must make the final decisions.

THE NEED FOR  STRONGER PEER REVIEW POLICIES

There are a multiplicity of science advisory procedures used at federal agencies and across the
wide variety of scientific products prepared by agencies.17  In response to congressional inquiry,
the U.S. General Accounting Office (now the Government Accountability Office) documented
the variability in both the definition and implementation of peer review across agencies.18 The
Carnegie Commission on Science, Technology and Government19 has highlighted the
importance of "internal" scientific advice (within the agency) and "external" advice (through
scientific advisory boards and other mechanisms).
17 Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers. Harvard University Press, Boston, 1990.
18 U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal Agencies Vary.
GAO/RCED-99-99, Washington, D.C., 1999.
19 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving
Regulatory Decision Making. Carnegie Commission, New York, 1993: 90.

-------
                                  Peer Review Handbook                        Page B-5
A wide variety of authorities have argued that peer review practices at federal agencies need to
                20
be strengthened.    Some arguments focus on specific types of scientific products (e.g.,
assessments of health, safety and environmental hazards).21 The Congressional/Presidential
Commission on Risk Assessment and Risk Management suggests that "peer review of economic
and social science information should have as high a priority as peer review of health, ecological,
and engineering information."22

Some agencies have formal peer review policies, while others do not. Even agencies that have
such policies do not always follow them prior to the release of important scientific products.

Prior to the development of this Bulletin, there were no government-wide standards concerning
when peer review is required and, if required, what type of peer review processes are
appropriate.  No formal interagency mechanism existed to foster cross-agency sharing of
experiences with peer review practices  and policies.  Despite the importance of peer review for
the credibility of agency scientific products, the public lacked a consistent way to determine
when an important scientific information product is being developed by an agency, the type of
peer review planned for that product, or whether there would be an opportunity to provide
comments and data to the reviewers.

This Bulletin establishes minimum standards  for when peer review is required for scientific
information and the types of peer review that should be considered by agencies in different
circumstances. It also establishes a transparent process for public disclosure of peer review
planning, including a web-accessible description of the peer review plan that the agency has
developed for each of its forthcoming influential scientific disseminations.

LEGAL AUTHORITY FOR THE BULLETIN

This Bulletin is issued under the Information Quality Act and OMB's general authorities to
oversee the quality of agency information, analyses, and regulatory actions. In the Information
Quality Act,  Congress directed OMB to issue guidelines to "provide policy and procedural
20 National Academy of Sciences, Peer Review in the Department of Energy - Office of Science and Technology.
Interim Report, National Academy  Press, Washington, D.C., 1997; National Academy of Sciences, Peer Review in
Environmental Technology Development: The Department of Energy - Office of Science and Technology. National
Academy Press, Washington, D.C., 1998; National Academy of Sciences, Strengthening Science at the U.S.
Environmental Protection Agency: Research-Management and Peer-Review Practices. National Academy Press,
Washington, D.C. 2000; U.S. General Accounting Office, EPA's Science Advisory Board Panels: Improved
Policies and Procedures Needed to Ensure Independence and Balance. GAO-01-536, Washington, D.C., 2001; U. S.
Environmental Protection Agency, Office of Inspector General, Pilot Study: Science in Support of Rulemaking
2003-P-00003. Washington, D.C., 2002; Carnegie Commission on Science, Technology, and Government, In the
National Interest: The Federal Government in the Reform of K-12 Math and Science Education. Carnegie
Commission, New York, 1991; U.S. General Accounting Office. Endangered Species Program:  Information on
How Funds Are Allocated and What Activities are Emphasized. GAO-02-581, Washington, D.C. 2002.
21 National Research Council, Science and Judgment in Risk Assessment National Academy Press, Washington,
D.C., 1994.
22 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report,
Volume 2, Risk Assessment and Risk Management in Regulatory Decision-Making. 1997:103.

-------
                                 Peer Review Handbook                      Page B-6
guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility and
integrity of information" disseminated by Federal agencies.  Pub. L. No. 106-554,  515(a).  The
Information Quality Act was developed as a supplement to the Paperwork Reduction Act, 44
U.S.C.  3501 et seq., which requires OMB, among other things, to "develop and oversee the
implementation of policies, principles, standards, and guidelines to ... apply to Federal agency
dissemination of public information."  In addition, Executive Order 12866, 58 Fed. Reg. 51,735
(Oct. 4, 1993), establishes that OIRA is "the repository of expertise concerning regulatory
issues," and it directs OMB to provide guidance to the agencies on regulatory planning. E.O.
12866,  2(b). The Order also requires that "[e]ach agency shall base its decisions on the best
reasonably obtainable scientific, technical, economic, or other information."  E.O. 12866, 
l(b)(7). Finally, OMB has authority in certain circumstances to manage the agencies under the
purview of the President's Constitutional authority to supervise the unitary Executive Branch.
All of these authorities support this Bulletin.

THE REQUIREMENTS OF THIS BULLETIN

This Bulletin addresses peer review of scientific information disseminations that contain findings
or conclusions that represent the official position of one or more agencies of the federal
government.

Section I:  Definitions

Section I provides definitions that are central to this Bulletin. Several terms are identical to or
based on those used in OMB's government-wide information quality guidelines, 67 Fed. Reg.
8452 (Feb. 22, 2002), and the Paperwork Reduction Act, 44 U.S.C.  3501 etseq.

The term "Administrator" means the Administrator of the Office of Information and Regulatory
Affairs in the Office of Management and Budget (OIRA).

The term "agency" has the same meaning as in the Paperwork Reduction Act, 44  U.S.C.  
3502(1).

The term "Information Quality Act" means Section 515 of Public Law 106-554 (Pub. L. No.
106-554,  515, 114 Stat. 2763, 2763A-153-154 (2000)).

The term "dissemination" means agency initiated or sponsored distribution of information to the
public.  Dissemination does not include distribution limited to government employees or agency
contractors or grantees; intra- or inter-agency use or sharing of government information; or
responses to requests for agency records under the Freedom of Information Act, the Privacy  Act,
the Federal Advisory Committee Act, the Government Performance and Results Act, or similar
laws.  This definition also excludes distribution limited to correspondence with individuals or
persons, press releases, archival records, public filings, subpoenas and adjudicative processes.  In
the context of this Bulletin, the definition of "dissemination" modifies the definition in OMB's
government-wide information quality guidelines to address the need for peer review prior to

-------
                                 Peer Review Handbook                      Page B-7
official dissemination of the information product.  Accordingly, under this Bulletin,
"dissemination" also excludes information distributed for peer review in compliance with this
Bulletin or shared confidentially with scientific colleagues, provided that the distributing agency
includes an appropriate and clear disclaimer on the information, as explained more fully below.
Finally, the Bulletin does not directly cover information supplied to the government by third
parties (e.g., studies by private consultants, companies and private, non-profit organizations, or
research institutions such as universities).  However, if an agency plans to disseminate
information supplied by a third party (e.g., using this information  as the basis for an agency's
factual determination that a particular behavior causes a disease), the requirements of the
Bulletin apply, if the dissemination is "influential".

In cases where a draft report or other information is released by an agency  solely for purposes of
peer review, a question may arise as to whether the draft report constitutes an official
"dissemination" under information-quality guidelines.  Section I instructs agencies to make this
clear by presenting the following disclaimer in the report:

       "THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE-
       DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY
       GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY]. IT
       DOES NOT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY
       AGENCY DETERMINATION OR POLICY."

In cases where the information is highly relevant to specific policy or regulatory
deliberations, this disclaimer shall  appear on each page of a draft report. Agencies also
shall discourage state, local, international and private organizations from using
information in draft reports that are undergoing peer review. Draft influential scientific
information presented at scientific meetings or shared confidentially with colleagues for
scientific input prior to peer review shall include the disclaimer:  "THE FINDINGS AND
CONCLUSIONS IN THIS REPORT (PRESENTATION) HAVE NOT BEEN FORMALLY
DISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY
AGENCY DETERMINATION OR POLICY."

An information product is not covered by the Bulletin unless it represents an official view of one
or more departments or agencies of the federal government. Accordingly, for the purposes  of
this Bulletin, "dissemination"  excludes research produced by government-funded scientists (e.g.,
those supported extramurally or intramurally by federal agencies or those working in state or
local governments with federal support) if that information is not represented as the views of a
department or agency (i.e., they are not official government disseminations).  For influential
scientific information that does not have the imprimatur of the federal government, scientists
employed by the federal government are required to include in their information product a clear
disclaimer that "the findings and conclusions in this report are those of the author(s)  and do not
necessarily represent the views of the funding agency." A similar disclaimer is advised for non-
government employees who publish government-funded research.

-------
                                 Peer Review Handbook                       Page B-8
For the purposes of the peer review Bulletin, the term "scientific information" means factual
inputs, data, models, analyses, technical information, or scientific assessments related to such
disciplines as the behavioral and social sciences, public health and medical sciences, life and
earth sciences, engineering, or physical sciences. This includes any communication or
representation of knowledge such as facts or data, in any medium or form, including textual,
numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes
information that an agency disseminates from a web page, but does not include the provision of
hyperlinks on a web page to information that others disseminate.  This definition excludes
opinions, where the agency's  presentation makes clear that an individual's opinion, rather than a
statement of fact or of the agency's findings and conclusions, is being offered.

The term "influential scientific information" means scientific information the agency reasonably
can determine will have or does have a clear and substantial impact on important public policies
or private sector decisions. In the term "influential scientific information," the term "influential"
should be interpreted consistently with OMB's government-wide information quality guidelines
and the information quality guidelines of the agency.  Information dissemination can have a
significant economic impact even if it is not part of a rulemaking. For instance, the economic
viability of a technology can be influenced by the government's characterization of its attributes.
Alternatively, the federal government's assessment of risk can directly or indirectly influence the
response actions of state and local agencies or international bodies.

One type of scientific information is a scientific assessment. For the purposes of this Bulletin,
the term "scientific assessment" means an evaluation of a body of scientific or technical
knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions,
and/or applies best professional judgment to bridge uncertainties in the available information.
These assessments include, but are not limited to, state-of-science reports; technology
assessments; weight-of-evidence analyses; meta-analyses; health, safety, or ecological risk
assessments; toxicological characterizations of substances; integrated  assessment models; hazard
determinations; or exposure assessments. Such assessments often draw upon knowledge from
multiple disciplines.  Typically, the data and models used in scientific assessments have already
been subject to some form of peer review (e.g., refereed journal peer review or peer review under
Section II of this Bulletin).

Section II: Peer Review of Influential Scientific Information

Section II requires each agency to subject "influential" scientific information to  peer review prior
to dissemination.  For dissemination of influential scientific information, Section  II provides
agencies broad discretion in determining what type of peer review is appropriate and what
procedures should be employed to select appropriate reviewers.  Agencies are directed to chose a
peer review mechanism that is adequate, giving due consideration to the novelty and complexity
of the science to be reviewed, the relevance of the information to decision making, the extent of
prior peer reviews, and the expected benefits and costs of additional review.

-------
                                  Peer Review Handbook                       Page B-9
The National Academy of Public Administration suggests that the intensity of peer review
should be commensurate with the significance of the information being disseminated and the
likely implications for policy decisions.23  Furthermore, agencies need to consider tradeoffs
between depth of peer review and timeliness.24 More rigorous peer review is necessary for
information that is based on novel methods or presents complex challenges for interpretation.
Furthermore, the need for rigorous peer review is greater when the information contains
precedent-setting methods or models, presents conclusions that are likely to change prevailing
practices, or is likely to affect policy decisions that have a significant impact.

This tradeoff can be considered in a benefit-cost framework.  The costs of peer review include
both the direct costs of the peer review activity and those stemming from potential delay in
government and private actions that can result from peer review. The benefits of peer review  are
equally clear: the insights offered by peer reviewers may  lead to policy with more benefits and/or
fewer costs. In addition to contributing to strong science, peer review, if performed fairly and
rigorously, can build consensus among stakeholders and reduce the temptation for courts and
legislators to second-guess or overturn agency actions.25  While it will not always be easy for
agencies to quantify the benefits and costs of peer review, agencies are encouraged to approach
peer review from a benefit-cost perspective.

Regardless of the peer review mechanism chosen, agencies should  strive to ensure that their peer
review practices are characterized by both scientific integrity and process integrity. "Scientific
integrity," in the context of peer review, refers to such issues as "expertise and balance of the
panel members; the identification of the scientific issues and clarity of the charge to the panel;
the quality, focus and depth  of the discussion of the issues by the panel;  the rationale and
supportability of the panel's findings; and the accuracy and clarity  of the panel report." "Process
integrity" includes such issues as "transparency and openness, avoidance of real or perceived
conflicts of interest, a workable process for public comment and involvement," and adherence to
defined procedures.26
When deciding what type of peer review mechanism is appropriate for a specific information
product, agencies will need to consider at least the following issues: individual versus panel
review; timing; scope of the review;  selection of reviewers; disclosure and attribution; public
participation; disposition of reviewer comments; and adequacy of prior peer review.
23 National Academy of Public Administration, Setting Priorities, Getting Results: A New Direction for EPA,
National Academy Press, Washington, D.C., 1995:23.
24 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report,
1997.
25 Mark R. Powell, Science at EPA: Information in the Regulatory Process. Resources for the Future, Washington,
D.C., 1999: 148, 176; Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers. Harvard University
Press, Boston, 1990: 242.
26ILSI Risk Sciences Institute, "Policies and Procedures: Model Peer Review Center of
Excellence," 2002:  4. Available at http://rsi.ilsi.org/file/Policies&Procedures.pdf.

-------
                                 Peer Review Handbook                      Page B-10
Individual versus Panel Review

Letter reviews by several experts generally will be more expeditious than convening a panel of
experts. Individual letter reviews are more appropriate when a draft document covers only one
discipline or when premature disclosure of a sensitive report to a public panel could cause harm
to government or private interests.  When time and resources warrant, panels are preferable, as
they tend to be more deliberative than individual letter reviews and the reviewers can learn from
each other.  There are also multi-stage processes in which confidential letter reviews are
conducted prior to release of a draft document for public notice and comment, followed by a
formal panel review.  These more rigorous and expensive processes are particularly valuable for
highly complex, multidisciplinary, and more important documents, especially those that are
novel or precedent-setting.

Timing of Peer Review

As a general rule, it is most useful to consult with peers early in the process of producing
information. For example, in the context of risk assessments, it is valuable to have the choice of
input data and the specification of the model reviewed by peers before the agency invests time
and resources in implementing the model and interpreting the results.  "Early" peer review occurs
in time to "focus attention on data inadequacies in time for corrections.

When an information product is a critical component of rule-making, it is important to obtain
peer review before the agency announces its  regulatory options so that any technical corrections
can be made before the agency becomes invested in  a specific approach or the positions of
interest groups have hardened. If review occurs too  late, it is unlikely to contribute to the course
of a rulemaking. Furthermore, investing in a more rigorous  peer review early in the process
"may provide net benefit by reducing the prospect of challenges to a regulation that later may
trigger time consuming and resource-draining litigation."27

Scope of the Review

The "charge" contains the instructions to the peer reviewers regarding the objective of the peer
review and the specific advice sought.  The importance of the information, which shapes the goal
of the peer review, influences the charge. For instance, the goal of the review might be to
determine the utility of a body of literature for drawing certain conclusions about the feasibility
of a technology or the safety of a product. In this context, an agency might ask reviewers to
determine the relevance of conclusions drawn in one context for other contexts (e.g., different
exposure conditions or patient populations).
27 Fred Anderson, Mary Ann Chirba Martin, E Donald Elliott, Cynthia Farina, Ernest Gellhorn, John D. Graham, C.
Boyden Gray, Jeffrey Holmstead, Ronald M. Levin, Lars Noah, Katherine Rhyne, Jonathan Baert Wiener,
"Regulatory Improvement Legislation: Risk Assessment, Cost-Benefit Analysis, and Judicial Review," Duke
Environmental Law and Policy Forum. Fall 2000, vol. XI (1): 132.

-------
                                 Peer Review Handbook                      PageB-11
The charge to the reviewers should be determined in advance of the selection of the reviewers.
In drafting the charge, it is important to remember the strengths and limitations of peer review.
Peer review is most powerful when the charge is specific and steers the reviewers to specific
technical questions while also directing reviewers to offer a broad evaluation of the overall
product.

Uncertainty is inherent in science, and in many cases individual studies do not produce
conclusive evidence. Thus, when an agency generates a scientific assessment, it is presenting its
scientific judgment about the accumulated evidence rather than scientific fact.28 Specialists
attempt to reach a consensus by weighing the accumulated evidence.  Peer reviewers can make
an important contribution by distinguishing scientific facts from professional judgments.
Furthermore, where appropriate, reviewers should be asked to provide advice on the
reasonableness of judgments made from the scientific evidence. However, the charge should
make clear that the reviewers are not to provide advice on the policy (e.g., the amount of
uncertainty that is acceptable or the amount of precaution that should be embedded in an
analysis).  Such considerations are the purview of the government.29

The charge should ask that peer reviewers ensure that scientific uncertainties are clearly
identified and characterized. Since not all uncertainties have an equal effect on the conclusions
drawn, reviewers should be asked to ensure that the potential implications of the uncertainties for
the technical conclusions drawn are clear. In addition, peer reviewers might be asked to consider
value-of-information analyses that identify whether more research is likely to decrease key
uncertainties.30 Value-of-information analysis was suggested for this purpose in the report of the
Presidential/Congressional Commission on Risk Assessment and Risk Management.31  A
description of additional research that would appreciably influence the conclusions of the
assessment can help an agency assess and target subsequent efforts.

Selection of Reviewers

Expertise.  The most important factor in selecting reviewers is expertise:  ensuring that the
selected reviewer has the knowledge, experience, and skills necessary to perform the review.
Agencies shall ensure that, in cases where the document being  reviewed spans a variety of
scientific disciplines or areas of technical expertise, reviewers who represent the necessary
spectrum of knowledge are chosen.  For instance, expertise in applied mathematics and statistics
is essential in the review of models, thereby allowing an audit of calculations and claims of
28 Mark R. Powell, Science at EPA: Information in the Regulatory Process. Resources for the Future, Washington,
D.C., 1999: 139.
29 Ibid.
30 Granger Morgan and Max Henri on, "The Value of Knowing How Little You Know,"
Uncertainty:  A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis,
Cambridge University Press, 1990: 307.
31 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report,
1997, Volume 1: 39, Volume 2: 91.

-------
                                  Peer Review Handbook                       Page B-12
significance and robustness based on the numeric data.32 For some reviews, evaluation of
biological plausibility is as important as statistical modeling.  Agencies shall consider requesting
that the public, including scientific and professional societies, nominate potential reviewers.

Balance. While expertise is the primary consideration, reviewers should also be selected to
represent a diversity of scientific perspectives relevant to the subject.  On most controversial
issues, there exists a range of respected scientific viewpoints regarding interpretation of the
available literature. Inviting reviewers with competing views on the science may lead to  a
sharper, more focused peer review.  Indeed, as a final layer of review, some organizations (e.g.,
the National Academy of Sciences) specifically recruit reviewers with strong opinions to test the
scientific strength and balance of their reports. The NAS policy on committee composition and
balance33 highlights important considerations associated with perspective, bias, and objectivity.

Independence. In its narrowest sense, independence in a reviewer means that the reviewer was
not involved in producing the draft document to  be reviewed. However, for peer review of some
documents, a broader view of independence is necessary to ensure credibility of the process.
Reviewers are generally not employed by the agency or office producing the document.  As the
National Academy of Sciences has stated, "external experts often can be more open, frank,  and
challenging to the status quo than internal reviewers, who may feel constrained by  organizational
concerns."34  The Carnegie Commission on Science, Technology, and Government notes that
"external science advisory boards serve a critically important function in providing regulatory
agencies with expert advice on a range of issues."35 However, the choice of reviewers requires a
case-by-case analysis.  Reviewers employed by other federal and state agencies may possess
unique or indispensable expertise.
A related issue is whether government-funded scientists in universities and consulting firms have
sufficient independence from the federal  agencies that support their work to be appropriate  peer
reviewers for those agencies.36 This concern can be mitigated in situations where the scientist
initiates the hypothesis to be tested or the method to be developed, which effectively creates a
buffer between the scientist and the agency. When an agency awards grants through a
competitive process that includes peer review, the agency's potential to influence the scientist's
research is limited. As such, when a scientist is awarded a government research grant through an
investigator-initiated, peer-reviewed competition, there generally should be no question as to that
scientist's ability to offer independent scientific advice to the agency on other projects. This
contrasts, for example, to a situation in which a scientist has a consulting or contractual
arrangement with the agency or office sponsoring a peer review.  Likewise, when  the agency
32 William W. Lowrance, Modern Science and Human Values. Oxford University Press, New York, NY 1985: 86.
33 National Academy of Sciences, "Policy and Procedures on Committee Composition and Balance and Conflicts of
Interest for Committees Used in the Development of Reports," May 2003:  Available at:
http://www.nationalacademies.org/coi/index.html.
34 National Research Council, Peer Review in Environmental Technology Development Programs: The Department
of Energy's Office of Science and Technology. National Academy Press, Washington, D.C., 1998: 3.
35 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving
Regulatory Decision Making. Carnegie Commission, New York, 1993: 90.
36 Lars Noah, "Scientific 'Republicanism': Expert Peer Review and the Quest for Regulatory Deliberation. Emory
Law Journal. Atlanta, Fall 2000:1066.

-------
                                 Peer Review Handbook                      Page B-13
and a researcher work together (e.g., through a cooperative agreement) to design or implement a
study, there is less independence from the agency. Furthermore, if a scientist has repeatedly
served as a reviewer for the same agency, some may question whether that scientist is
sufficiently independent from the agency to be employed as a peer reviewer on agency-
sponsored projects.

As the foregoing suggests, independence poses a complex set of questions that must be
considered by agencies when peer reviewers are selected.  In general, agencies shall make an
effort to rotate peer review responsibilities across the available pool of qualified reviewers,
recognizing that in some cases repeated service by the same reviewer is needed because of
essential expertise.

Some agencies have built entire organizations to provide independent scientific advice while
other agencies tend to employ ad hoc scientific panels on specific  issues.  Respect for the
independence of reviewers may be enhanced if an agency collects names of potential reviewers
(based on considerations of expertise and reputation for objectivity) from the public, including
scientific or professional societies.  The Department of Energy's use of the American  Society of
Mechanical Engineers to identify potential peer reviewers from a variety of different scientific
societies provides an example of how professional societies can assist in the development of an
independent peer review panel.37

Conflict of Interest.  The National Academy of Sciences defines "conflict of interest"  as any
financial or other interest that conflicts with the service of an individual on the review panel
because it could impair the individual's objectivity or could create an unfair competitive
advantage for a person or organization.38 This standard provides a useful benchmark for
agencies to consider in selecting peer reviewers. Agencies shall make a special effort to examine
prospective reviewers' potential financial conflicts, including significant investments, consulting
arrangements, employer affiliations and grants/contracts. Financial ties of potential reviewers to
regulated entities (e.g., businesses), other stakeholders, and regulatory agencies shall be
scrutinized when the information being reviewed is likely to be relevant to regulatory  policy.
The inquiry into potential conflicts goes beyond financial investments and business relationships
and includes work as an expert witness, consulting arrangements,  honoraria and sources of grants
and contracts. To evaluate any real or perceived conflicts  of interest with potential reviewers and
questions regarding the independence of reviewers, agencies are referred to federal ethics
requirements, applicable standards issued by the Office of Government Ethics, and the prevailing
practices of the National Academy of Sciences.  Specifically, peer reviewers who are federal
employees (including special government employees) are subject to federal requirements
governing conflicts of interest.  See, e.g., 18 U.S.C.  208; 5 C.F.R. Part 2635 (2004). With
37 American Society for Mechanical Engineers, Assessment of Technologies Supported by the Office of Science and
Technology. Department of Energy: Results of the Peer Review for Fiscal Year 2002. ASME Technical Publishing,
Danvers, MA, 2003.
38 National Academy of Sciences, "Policy and Procedures on Committee Composition and
Balance and Conflicts of Interest for Committees Used in the Development of Reports," May
2003:  Available at: http://www.nationalacademies.org/coi/index.html.

-------
                                 Peer Review Handbook                      Page B-14
respect to reviewers who are not federal employees, agencies shall adopt or adapt the NAS
policy for committee selection with respect to evaluating conflicts of interest.39  Both the NAS
and the federal government recognize that under certain circumstances some conflict may be
unavoidable in order to obtain the necessary expertise. See, e.g., 18 U.S.C.   208(b)(3); 5
U.S.C. App.  15 (governing NAS committees). To improve the transparency of the process,
when an agency determines that it is necessary to use a reviewer with a real or perceived conflict
of interest, the agency should consider publicly disclosing those conflicts.  In such situations, the
agency shall inform potential reviewers of such disclosure at the time they are recruited.

Disclosure and Attribution: Anonymous versus Identified

Peer reviewers must have a clear understanding of how their comments will be conveyed to the
authors of the document and to the public. When peer review of government reports is
considered, the case for transparency is stronger, particularly when the report addresses an issue
with significant ramifications for the public and private sectors.  The public may not have
confidence in the peer review process when the names and affiliations of the peer reviewers are
unknown. Without access to the comments of reviewers, the public is incapable of determining
whether the government has seriously considered the comments of reviewers and made
appropriate revisions.  Disclosure of the slate of reviewers and the substance of their comments
can strengthen public confidence in the peer review process.  It is common at many journals and
research funding agencies to disclose annually the slate of reviewers. Moreover, the National
Academy of Sciences now discloses the names of its peer reviewers, without disclosing the
substance of their comments. The science advisory committees to regulatory agencies typically
disclose at least a summary of the comments of reviewers as well as their names and affiliations.

For agency-sponsored peer review conducted under Sections II and III, this Bulletin strikes a
compromise by requiring disclosure of the identity  of the reviewers, but not public attribution of
specific comments to specific reviewers.  The agency has considerable discretion in the
implementation of this compromise (e.g., summarizing the views of reviewers as a group or
disclosing individual reviewer comments without attribution). Whatever approach is employed,
the agency must inform reviewers in advance of how it intends to address this issue. Information
about a reviewer retrieved  from a record filed by the reviewer's name or other identifier may be
disclosed only as permitted by the conditions of disclosure enumerated in the Privacy  Act, 5
U.S.C.   552a as amended, and as interpreted in OMB implementing guidance, 40 Fed. Reg.
28,948 (July 9, 1975).

Public Participation

Public comments can be important in shaping expert deliberations. Agencies may decide that
peer review should precede an opportunity for  public comment to ensure that the public  receives
the most scientifically strong product (rather than one that may change substantially as a result of
peer reviewer suggestions). However, there are situations in which public participation in peer
39
  Ibid.

-------
                                 Peer Review Handbook                      Page B-15
review is an important aspect of obtaining a high-quality product through a credible process.
Agencies, however, should avoid open-ended comment periods, which may delay completion of
peer reviews and complicate the completion of the final work product.

Public participation can take a variety of forms, including opportunities to provide oral
comments before a peer review panel or requests to provide written comments to the peer
reviewers.  Another option is for agencies to publish a "request for comment" or other notice in
which they solicit public comment before a panel of peer reviewers performs its work.

Disposition of Reviewer Comments

A peer review is considered completed once the agency considers and addresses the reviewers'
comments. All reviewer comments should be given consideration and be incorporated where
relevant and valid.  For instance, in the context of risk assessments, the National Academy of
Sciences recommends that peer review include a written evaluation made available for public
inspection.40 In cases where there is a public panel, the agency should plan publication of the
peer review report(s) and the agency's response to peer reviewer comments.

In addition, the credibility  of the final scientific report is likely to be enhanced if the public
understands how the agency  addressed the specific concerns raised by the peer reviewers.
Accordingly, agencies should consider preparing a written response to the peer review report
explaining: the  agency's agreement or disagreement, the actions the agency has undertaken or
will undertake in response to the report, and (if applicable) the reasons the agency believes those
actions satisfy any key  concerns or recommendations in the report.

Adequacy of Prior Peer Review

In light of the broad range  of information covered by Section II, agencies are directed to choose  a
peer review mechanism that  is adequate, giving due consideration to the novelty and complexity
of the science to be reviewed, the relevance of the information to decision making, the extent of
55prior peer reviews, and the expected benefits and costs of additional review.

Publication in a refereed scientific journal may mean that adequate peer review has been
performed.  However, the intensity of peer review is highly variable across journals. There will
be cases in which an agency  determines that a more rigorous or transparent review process is
necessary.  For instance, an agency may determine a particular journal review process did not
address questions (e.g., the extent of uncertainty inherent in a finding) that the agency  determines
should be addressed before disseminating that information.  As such, prior peer review and
publication is not by itself sufficient grounds for determining that no further review is  necessary.
40 National Research Council, Risk Assessment in the Federal Government:  Managing the Process. National
Academy Press, Washington, D.C., 1983.

-------
                                  Peer Review Handbook                       Page B-16
Section III:  Peer Review of Highly Influential Scientific Assessments

Whereas Section II leaves most of the considerations regarding the form of the peer review to the
agency's discretion, Section III requires a more rigorous form of peer review for highly
influential scientific assessments. The requirements of Section II of this Bulletin apply to Section
III, but Section III has some additional requirements, which are discussed below.  In planning a
peer review under Section III, agencies typically will have to devote greater resources and
attention to the issues discussed in Section II, i.e., individual versus panel review; timing; scope
of the review; selection of reviewers; disclosure and attribution; public participation; and
disposition of reviewer comments.

A scientific assessment is considered "highly influential" if the agency or the OIRA
Administrator determines that the dissemination could have a potential impact of more than $500
million in any one year on either the public or private sector or that the dissemination is novel,
controversial, or precedent-setting, or has significant interagency interest.  One of the ways
information can exert economic impact is through the costs or benefits of a regulation based on
the disseminated information. The qualitative aspect of this definition may be most useful in
cases where it is difficult for an agency to predict the potential economic effect of dissemination.
In the context of this Bulletin, it may be either the approach used in the assessment or the
interpretation of the information itself that is novel or precedent-setting. Peer review can be
valuable in establishing the bounds of the scientific debate when methods or interpretations are a
source of controversy among interested parties. If information is covered by Section III, an
agency is required to adhere to the peer review procedures specified in Section III.

Section III (2) clarifies that the principal findings, conclusions and recommendations in official
reports of the National Academy of Sciences that fall under this Section are generally presumed
not to require additional peer review.  All other highly influential scientific assessments require a
review that meets the requirements of Section III of this Bulletin.

With regard to the selection of reviewers, Section III(3)(a) emphasizes consideration of expertise
and balance. As discussed in Section II, expertise refers to the required knowledge, experience
and skills required to perform the review whereas balance refers to the need for diversity in
scientific perspective and disciplines.  We emphasize that the term "balance" here refers not to
balancing of stakeholder or political interests but rather to a broad and diverse representation of
respected perspectives and intellectual traditions within the scientific community, as discussed in
the NAS policy on committee composition and balance.41

Section III(3)(b) instructs agencies to consider barring participation by scientists with a conflict
of interest. The conflict of interest standards for Sections II and III of the Bulletin are identical.
As discussed under Section II, those peer reviewers who are federal employees, including
Special  Government Employees, are subject to applicable statutory and regulatory standards for
41 National Academy of Sciences, "Policy and Procedures on Committee Composition and Balance and Conflicts of
Interest for Committees Used in the Development of Reports," May 2003:  Available at:
http://www.nationalacademies.org/coi/index.html.

-------
                                 Peer Review Handbook                      Page B-17
federal employees. For non-government employees, agencies shall adopt or adapt the NAS
policy for committee member selection with respect to evaluating conflicts of interest.

Section III(3)(c) instructs agencies to ensure that reviewers are independent of the agency
sponsoring the review.   Scientists employed by the sponsoring agency are not permitted to
serve as reviewers for highly influential scientific assessments.  This does not preclude Special
Government Employees, such as academics appointed to advisory committees,  from serving as
peer reviewers. The only exception to this ban would be the rare situation in which a scientist
from a different agency of a Cabinet-level department than the agency that is disseminating the
scientific assessment has expertise, experience and skills that are essential but cannot be obtained
elsewhere. In evaluating the need for this exception, agencies shall use the NAS criteria for
assessing the appropriateness of using employees of sponsors (e.g., the government
scientist must not have  had any part in the development  or prior review of the scientific
information and must not hold a position of managerial or policy responsibility).

We also considered whether a reviewer can be independent of the agency if that reviewer
receives a substantial amount of research funding from the agency sponsoring the review.
Research grants that were awarded to the scientist based on investigator-initiated, competitive,
peer-reviewed proposals do not generally raise issues of independence. However, significant
consulting and contractual relationships with the agency may raise issues of independence or
conflict, depending upon the situation.

Section III(3)(d) addresses concerns regarding repeated  use of the same reviewer in multiple
assessments.  Such repeated use should be avoided unless a particular reviewer's expertise is
essential.  Agencies should rotate membership across the available pool of qualified reviewers.
Similarly, when using standing panels of scientific advisors, it is suggested that the agency rotate
membership among qualified scientists in order to obtain fresh perspectives and reinforce the
reality and perception of independence from the agency.

Section 111(4) requires agencies to provide reviewers with sufficient background information,
including access to key studies, data and models, to perform their role as peer reviewers.  In this
respect, the peer review envisioned in Section III is more rigorous than some forms of journal
peer review, where the  reviewer is often not provided access to underlying data or models.
Reviewers shall be informed of applicable access, objectivity, reproducibility and other quality
standards under federal information quality laws.

Section 111(5) addresses opportunity  for public participation in peer review, and provides that the
agency shall, wherever possible, provide for public participation. In some cases, an assessment
may be so sensitive that it is critical that the agency's assessment achieve a high level of quality
before it is publicized.  In those situations, a rigorous yet confidential peer review process may
be appropriate,  prior to public release of the assessment. If an agency decides to make a draft
assessment publicly available at the onset of a peer review process, the agency  shall, whenever
possible, provide a vehicle for the public to provide written comments, make an oral presentation
before the peer reviewers, or both. When written public comments are received, the agency shall

-------
                                 Peer Review Handbook                      Page B-18
ensure that peer reviewers receive copies of comments that address significant scientific issues
with ample time to consider them in their review. To avoid undue delay of agency activities, the
agency shall specify time limits for public participation throughout the peer review process.

Section 111(6) requires that agencies instruct reviewers to prepare a peer review report that
describes the nature and scope of their review and their findings and conclusions. The report
shall disclose the name of each peer reviewer and a brief description of his or her organizational
affiliation, credentials and relevant experiences.  The peer review report should either summarize
the views of the group as a whole (including any dissenting views) or include a verbatim copy of
the comments of the individual reviewers (with or without attribution of specific views to
specific names).  The agency shall also prepare a written response to the peer review report,
indicating whether the agency agrees with the reviewers and what actions the agency has taken
or plans to take to address the points made by reviewers.  The agency is required to disseminate
the peer review report and the agency's response to the report on the agency's website, including
all the materials related to the peer review such as the charge statement, peer review report, and
agency response to the review.  If the scientific information is used to support a final rule then,
where practicable, the peer review report shall be made available to the public with enough time
for the public to consider the implications of the peer review report for the rule being considered.

Section 111(7) authorizes but does not require an agency to commission an entity  independent of
the agency to select peer reviewers and/or manage the peer review process in accordance with
this Bulletin. The entity may be a scientific or professional society, a firm specializing in peer
review, or a non-profit organization with experience in peer review.

Section IV:  Alternative Procedures

Peer review as described in this Bulletin is only one of many procedures that agencies can
employ to ensure an appropriate degree of pre-dissemination quality of influential scientific
information. For example, Congress has assigned the NAS a special role in advising the federal
government on scientific and technical issues. The procedures of the NAS are generally quite
rigorous, and thus agencies should presume that major findings, conclusions, and
recommendations of NAS reports meet the performance standards of this Bulletin.

 As an alternative to complying with Sections II and III of this Bulletin, an agency may instead
 (1) rely on scientific information produced by the National Academy of Sciences, (2)
 commission the National Academy of Sciences to peer review an agency draft scientific
 information product, or (3) employ an alternative procedure or set of procedures, specifically
 approved by the OIRA Administrator in consultation with the Office of Science and Technology
 Policy (OSTP), that ensures that the scientific information product meets applicable
 information-quality standards.

 An example of an alternative procedure is to commission a respected third party other than the
 NAS (e.g., the Health Effects Institute or the National Commission on Radiation Protection and
 Measurement) to conduct an assessment or series of related assessments.  Another example of

-------
                                 Peer Review Handbook                      Page B-19
 an alternative set of procedures is the three-part process used by the National Institutes of Health
 (NIH) to generate scientific guidance.  Under that process, a scientific proposal or white paper is
 generated by a working group composed of external, independent scientific experts; that paper is
 then forwarded to a separate external scientific council, which then makes recommendations to
 the agency. The agency, in turn, decides whether to adopt and/or modify the proposal.  For
 large science agencies that have diverse research portfolios and do not have significant
 regulatory responsibilities, such as NIH, an acceptable alternative would be
 to allow scientists from one part of the agency (for example, an NIH institute) to participate in
 the review of documents prepared by another part of the agency, as long as the head of the
 agency confirms in writing that each of the reviewers  meets the NAS criteria relating to
 the appropriateness of using employees of sponsors (e.g., the government scientist must not
 have had any part in the development or prior review  of the scientific information and must not
 hold a position of managerial or policy responsibility). The purpose of Section IV is to
 encourage these types of innovation in the methods used to ensure pre-dissemination quality
 control of influential scientific information.

The mere existence of a public comment process (e.g., notice-and-comment procedures under the
Administrative Procedure Act) does not constitute adequate peer review or an "alternative
process," because it does not assure that qualified, impartial specialists in  relevant fields have
performed a critical evaluation of the agency's draft product.42

Section V:  Peer Review Planning

 Section V requires agencies to begin a systematic process of peer review  planning for influential
 scientific information  (including highly influential scientific assessments) that the agency plans
 to disseminate in the foreseeable future. A key feature of this planning process is a web-
 accessible listing of forthcoming influential scientific  disseminations (i.e., an agenda) that is
 regularly updated by the agency. By making these plans publicly available, agencies will be
 able to gauge the extent of public interest  in the peer review process for influential scientific
 information, including highly influential scientific assessments. These web-accessible agendas
 can also be used by the public to monitor agency compliance with this Bulletin.

 Each entry on the agenda shall include a preliminary title of the planned report, a short
paragraph describing the subject and purpose of the planned report, and an agency contact
person.  The agency shall provide its prediction regarding whether the dissemination will be
"influential scientific information" or a "highly influential scientific  assessment," as the
designation can influence the type of peer review to be undertaken.  The agency shall discuss the
timing of the peer review, as well as the use of any deferrals. Agencies shall include entries in
the agenda for influential scientific information, including highly influential scientific
assessments, for which the Bulletin's requirements have been deferred or waived. If the agency,
in consultation with the OIRA Administrator, has determined that it is appropriate to use a
42 William W. Lowrance, Modern Science and Human Values. Oxford University Press, New York, NY 1985: 86.

-------
                                 Peer Review Handbook                      Page B-20
Section IV "alternative procedure" for a specific dissemination, a description of that alternative
procedure shall be included in the agenda.

Furthermore, for each entry on the agenda, the agency shall describe the peer review plan.  Each
peer review plan shall include: (i) a paragraph including the title, subject and purpose of the
planned report, as well as an agency contact to whom inquiries may be directed to learn the
specifics of the plan; (ii) whether the dissemination is likely to be influential scientific
information or a highly influential scientific assessment; (iii) the timing of the review (including
deferrals); (iv) whether the review will be conducted through a panel or individual letters (or
whether an alternative procedure will  be exercised); (v) whether there will be opportunities for
the public to comment on the work product to be peer reviewed, and if so, how and when these
opportunities will be provided; (vi) whether the agency will provide significant and relevant
public comments to the peer reviewers before they conduct their review; (vii) the anticipated
number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct description of the
primary disciplines or expertise needed in the review; (ix) whether reviewers will be selected by
the agency or by a designated outside organization; and (x) whether the public, including
scientific or professional societies, will be asked to nominate potential peer reviewers.  The
agency shall provide a link from the agenda to each document made public pursuant to this
Bulletin.  Agencies shall link their peer review agendas to the U.S. Government's official web
portal :firstgov at http://www.FirstGov.gov

Agencies should update their peer review agendas at least every six months. However, in some
cases  particularly for highly influential scientific assessments and other particularly  important
information   more frequent updates  of existing entries on the agenda, or the addition of new
entries to the agenda, may be warranted.  When new entries are added to the agenda of
forthcoming reports and other information, the public should be provided with sufficient time to
comment on the agency's peer review plan for that report or product.  Agencies shall consider
public comments on the peer review plan. Agencies are encouraged to offer a listserve or similar
mechanism for members of the public who would like to be notified by email each time an
agency's peer review agenda has been updated.

The peer review planning requirements of this Bulletin are designed to be  implemented in
phases.  Specifically, the planning requirements of the Bulletin will go into effect for documents
subject to Section III of the  Bulletin (highly influential scientific assessments) six months after
publication.  However, the planning requirements for documents subject to Section II of the
Bulletin do not go into effect until one year after publication.  It is expected that agency
experience with the planning requirements of the Bulletin for the smaller scope of documents
encompassed in Section III  will be used to inform implementation of these planning
requirements for the larger scope of documents covered under Section II.

Section VI:  Annual Report

Each agency shall prepare an annual report that summarizes key decisions made pursuant to this
Bulletin.  In  particular, each agency should provide to OIRA the following: 1) the number of

-------
                                 Peer Review Handbook                      Page B-21
peer reviews conducted subject to the Bulletin (i.e., for influential scientific information and
highly influential scientific assessments); 2) the number of times alternative procedures were
invoked; 3) the number of times waivers or deferrals were invoked (and in the case of deferrals,
the length of time elapsed between the deferral and the peer review); 4) any decision to appoint a
reviewer pursuant to any exception to the applicable independence or conflict of interest
standards of the Bulletin, including determinations by the Secretary or Deputy Secretary
pursuant to Section III (3) (c); 5) the number of peer review panels that were conducted in public
and the number that allowed public comment;  6) the number of public comments provided on the
agency's peer review plans; and 7) the number of peer reviewers that the agency used that were
recommended by professional societies.

Section VII: Certification in the Administrative Record

If an agency relies on influential scientific information or a highly influential  scientific
assessment subject to the requirements of this Bulletin in support of a regulatory action, the
agency shall include in the administrative record for that action a certification that explains how
the agency has complied with the requirements of this Bulletin and the Information Quality Act.
Relevant materials are to be placed in the administrative record.

Section VIII:  Safeguards, Deferrals, and Waivers

Section VIII recognizes that individuals serving as peer reviewers have a privacy interest in
information about themselves that the government maintains and retrieves by name or identifier
from a system of records. To the extent information about a reviewer (name,  credential,
affiliation) will be disclosed along with his/her comments or analysis, the agency must comply
with the requirements of the Privacy Act, 5 U.S.C. 552a, as amended, and OMB Circular A-130,
Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a
published System of Records Notice. Furthermore, the peer review must be conducted in a
manner that respects confidential business information as well as intellectual property.

Section VIII also allows for a deferral or waiver of the requirements of the Bulletin where
necessary.  Specifically, the agency head may waive or defer some or all of the peer review
requirements of Sections II or III of this Bulletin if there is a compelling rationale for waiver or
deferral.  Waivers will seldom be warranted under this provision because the Bulletin already
provides significant safety valves, such as: the exemptions provided in Section IX, including the
exemption for time-sensitive health and safety information; the authorization  for alternative
procedures in Section IV; and the overall flexibility provided for peer reviews of influential
scientific information under Section II. Nonetheless, we have included this waiver and deferral
provision to ensure needed flexibility in unusual and compelling situations not otherwise covered
by the exemptions to the Bulletin, such as situations where unavoidable legal  deadlines prevent
full compliance with the Bulletin before information is disseminated. Deadlines found in
consent decrees agreed to by agencies after the Bulletin is issued will not ordinarily warrant
waiver of the Bulletin's requirements because those deadlines should be negotiated to permit
time for all required procedures, including peer review. In addition, when an agency is

-------
                                 Peer Review Handbook                      Page B-22
unavoidably up against a deadline, deferral of some or all requirements of the Bulletin (as
opposed to outright waiver of all of them) is the most appropriate accommodation between the
need to satisfy immovable deadlines and the need to undertake proper peer review.  If the agency
head defers any of the peer review requirements prior to dissemination, peer review should be
conducted as soon as practicable thereafter.

Section IX:  Exemptions

There are a variety of situations where agencies need not conduct peer review under this
Bulletin. These include, for example, disseminations of sensitive information related to certain
national security, foreign affairs, or negotiations involving international treaties and trade where
compliance with this Bulletin would interfere with the need for secrecy or promptness.

This Bulletin does not cover official disseminations that arise in adjudications and permit
proceedings, unless the agency determines that peer review is practical and appropriate and that
the influential dissemination is scientifically or technically novel (i.e., a major change in
accepted practice) or likely to have precedent-setting influence on future adjudications or permit
proceedings. This exclusion is intended to cover, among other things, licensing, approval and
registration processes for specific product development activities as well as site-specific
activities. The determination as to whether peer review is practical and appropriate is left to the
discretion of the agency. While this Bulletin is not broadly applicable to adjudications, agencies
are encouraged to hold peer reviews of scientific assessments supporting adjudications to the
same technical standards as peer reviews covered by the Bulletin, including transparency and
disclosure of the data and models underlying the assessments.  Protections apply to confidential
business information.

The Bulletin does not cover time-sensitive health and safety disseminations, for example, a
dissemination based primarily on data from a recent clinical trial that was adequately peer
reviewed before the trial began. For this purpose, "health" includes public health, or plant or
animal infectious diseases.

This Bulletin covers original data and formal analytic models used by agencies in Regulatory
Impact Analyses (RIAs). However,  the RIA documents themselves are already reviewed
through an interagency review process under E.O.  12866 that involves application of the
principles and methods defined in OMB Circular A-4. In that respect, RIAs are excluded from
coverage by this Bulletin, although agencies are encouraged to have RIAs reviewed by peers
within the government for adequacy and completeness.

The Bulletin does not cover accounting, budget, actuarial, and financial information including
that which is generated or used by agencies that focus on interest rates, banking, currency,
securities, commodities, futures, or taxes.

-------
                                 Peer Review Handbook                      Page B-23
Routine statistical information released by federal statistical agencies (e.g., periodic demographic
and economic statistics) and analyses of these data to compute standard indicators and trends
(e.g., unemployment and poverty rates) is excluded from this Bulletin.

The Bulletin does not cover information disseminated in connection with routine rules that
materially alter entitlements, grants, user fees, or loan programs, or the rights and obligations of
recipients thereof.

If information is disseminated pursuant to an exemption to this Bulletin, subsequent
disseminations are not automatically exempted. For example, if influential scientific information
is first disseminated in the course of an exempt agency adjudication, but is later disseminated in
the context of a non-exempt rulemaking,  the subsequent dissemination will be subject to the
requirements of this Bulletin even though the first dissemination was not.
Section X:  OIRA and OSTP Responsibilities
OIRA, in consultation with  OSTP, is responsible for overseeing agency implementation of this
Bulletin. In order to foster learning about peer review practices across agencies, OIRA and
OSTP shall form an interagency workgroup on peer review that meets regularly, discusses
progress and challenges, and recommends improvements to peer review practices.

Section XI: Effective Date  and Existing Law

The requirements of this Bulletin, with the exception of Section V,  apply to information
disseminated on or after six months after publication of this Bulletin. However, the Bulletin
does not apply to information that is already being addressed by an agency-initiated peer review
process (e.g., a draft is already being reviewed by a formal scientific advisory committee
established by the agency).  An existing peer review mechanism mandated by law should be
implemented by the agency in a manner as consistent as possible with the practices and
procedures outlined in this Bulletin.  The requirements of Section V apply to  "highly influential
scientific assessments," as designated in Section III of the Bulletin, within six months of
publication of the final Bulletin. The requirements in Section V apply to documents subject to
Section II of the Bulletin one year after publication of the final Bulletin.

 Section XII:  Judicial Review

This Bulletin is intended to  improve the internal management of the Executive Branch and is not
 intended to, and does not, create any  right or benefit, substantive or procedural, enforceable at
 law or in equity, against the United States,  its agencies or other entities, its officers or
 employees, or any other person.

-------
Peer Review Handbook                   Page B-24

-------
                                Peer Review Handbook                     Page B-25
Bulletin for Peer Review

I.      Definitions.

For purposes of this Bulletin 
  1. the term "Administrator" means the Administrator of the Office of Information and
Regulatory Affairs in the Office of Management and Budget (OIRA);
  2. the term "agency" has the same meaning as in the Paperwork Reduction Act, 44 U.S.C.  
3502(1);
  3. the term "dissemination" means agency initiated or sponsored distribution of information to
the public (see 5 C.F.R. 1320.3(d) (definition of "Conduct or Sponsor")). Dissemination does
not include distribution limited to government employees or agency contractors or grantees;
intra- or inter-agency use or sharing of government information; or responses to requests for
agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory
Committee Act, the Government Performance and Results Act or similar law. This definition
also excludes distribution limited to correspondence with individuals or persons, press releases,
archival records, public filings, subpoenas and adjudicative  processes. The term "dissemination"
also excludes information distributed for peer review in compliance with this Bulletin, provided
that the distributing agency includes a clear disclaimer on the information as follows: "THIS
INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE-
DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY
GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY].
IT DOES NOT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY
AGENCY DETERMINATION OR POLICY." For the purposes of this Bulletin,
"dissemination" excludes research produced by government-funded scientists (e.g., those
supported extramurally or intramurally by federal agencies or those working in state or local
governments with federal support) if that information does not represent the views of an agency.
To qualify for this exemption, the information should display a clear disclaimer that "the
findings and conclusions in this report are those of the author(s) and do not necessarily represent
the views of the funding agency";
  4. the term "Information Quality Act"  means Section 515  of Public Law  106-554 (Pub. L. No.
106-554,  515, 114 Stat. 2763, 2763A-153-154 (2000));
  5. the term "scientific information" means factual inputs, data, models, analyses, technical
information, or scientific assessments based on the behavioral and social sciences, public health
and medical sciences, life and earth sciences, engineering, or physical sciences. This includes
any communication or representation of knowledge such as  facts or data, in any medium or form,
including textual, numerical, graphic, cartographic, narrative, or audiovisual forms.  This
definition includes information that an agency disseminates  from a web page, but does not
include the provision of hyperlinks to information that others disseminate.  This definition does
not include opinions, where the agency's presentation makes clear that what is being offered is
someone's opinion rather than fact or the agency's views;
  6. the term "influential scientific information" means scientific information the agency
reasonably can determine will have or does have a clear  and substantial impact on important
public policies or private sector decisions; and

-------
                                 Peer Review Handbook                      Page B-26
  7. the term "scientific assessment" means an evaluation of a body of scientific or technical
knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions,
and/or applies best professional judgment to bridge uncertainties in the available information.
These assessments include, but are not limited to, state-of-science reports; technology
assessments; weight-of-evidence analyses; meta-analyses; health, safely, or ecological risk
assessments; toxicological characterizations of substances; integrated assessment models; hazard
determinations; or exposure assessments.

II.     Peer Review of Influential Scientific Information.

  1. In General:  To the extent permitted by law, each agency shall conduct a peer review on all
influential scientific information that the agency intends to disseminate.  Peer reviewers shall be
charged with reviewing scientific and technical matters, leaving policy determinations for the
agency. Reviewers shall be informed of applicable access, objectivity, reproducibility and other
quality standards under the federal laws governing information access and quality.
  2. Adequacy of Prior Peer Review: For information subject to this section of the Bulletin,
agencies need not have further peer review conducted on information that has already been
subjected to adequate peer review. In determining whether prior peer review is adequate,
agencies shall give due consideration to the novelty and complexity of the science to be
reviewed, the importance of the information to decision making, the extent of prior peer reviews,
and the expected benefits and costs of additional review. Principal findings, conclusions and
recommendations in official reports of the National Academy of Sciences are generally
presumed to have been adequately peer reviewed.
  3. Selection of Reviewers:
       a. Expertise and Balance: Peer reviewers shall be selected based on expertise, experience
and skills, including specialists from multiple disciplines, as necessary. The group of reviewers
shall be sufficiently broad and diverse to fairly represent the relevant scientific and technical
perspectives and fields of knowledge.  Agencies shall consider requesting that the public,
including scientific and professional societies, nominate potential reviewers.
       b. Conflicts: The agency - or the entity selecting the peer reviewers - shall (i)  ensure that
those reviewers serving as federal employees (including special government employees) comply
with applicable federal ethics requirements; (ii) in selecting peer reviewers who are not
government employees, adopt or adapt the National Academy of Sciences policy for committee
selection with respect to evaluating the potential for conflicts (e.g., those arising from
investments; agency, employer, and business affiliations; grants, contracts and  consulting
income). For scientific information relevant to specific regulations, the agency shall examine a
reviewer's financial ties to regulated entities (e.g., businesses), other  stakeholders, and the
agency.
       c. Independence: Peer reviewers shall not have participated in development of the work
product. Agencies are encouraged to rotate membership on standing panels across the pool of
qualified reviewers. Research grants that were awarded to scientists based on investigator-
initiated, competitive, peer-reviewed proposals generally do not raise issues as  to independence
or conflicts.

-------
                                 Peer Review Handbook                      Page B-27
  4. Choice of Peer Review Mechanism: The choice of a peer review mechanism (for example,
letter reviews or ad hoc panels) for influential scientific information shall be based on the
novelty and complexity of the information to be reviewed, the importance of the information to
decision making, the extent of prior peer review, and the expected benefits and costs of review,
as well as the factors regarding transparency described in 11(5).
  5. Transparency: The agency  or entity managing the peer review  shall instruct peer
reviewers to prepare a report that describes the nature of their review and their findings and
conclusions. The peer review report shall either (a) include a verbatim copy of each reviewer's
comments (either with or without specific attributions) or (b) represent the views of the group as
a whole, including any disparate and dissenting views. The agency shall disclose the names of
the reviewers and their organizational affiliations in the report. Reviewers shall be notified in
advance regarding the extent of disclosure and attribution planned by the agency. The agency
shall disseminate the final peer review report on the agency's website along with all materials
related to the peer review (any charge statement, the peer review report, and any agency
response). The peer review report shall be discussed in the preamble to any related rulemaking
and included in the administrative record for any related agency action.
  6. Management of Peer Review Process and Reviewer Selection: The agency may commission
independent entities to manage the peer review process, including the selection of peer
reviewers, in accordance with this Bulletin.

III.    Additional Peer Review Requirements for Highly  Influential Scientific Assessments.
  1. Applicability:  This section applies to influential scientific information that the agency or
the Administrator determines to be a scientific assessment that:
       (i) could have a potential impact of more than $500 million in any year, or
       (ii) is novel,  controversial, or precedent-setting or has significant interagency interest.
  2. In General: To the extent permitted by law, each agency shall conduct peer reviews on all
information subject to this Section. The peer reviews shall satisfy the requirements of Section  II
of this Bulletin, as well as the additional requirements found in this Section. Principal findings,
conclusions and recommendations in official reports of the National Academy of Sciences that
fall under this Section are generally presumed not to require  additional peer review.
  3. Selection of Reviewers:
       a. Expertise and Balance: Peer reviewers shall be selected based on expertise, experience
and skills, including specialists from multiple disciplines, as  necessary. The group of reviewers
shall be sufficiently  broad and diverse to fairly represent the  relevant scientific and technical
perspectives and fields of knowledge. Agencies shall consider requesting that the public,
including scientific and professional societies, nominate potential reviewers.
       b. Conflicts:  The agency - or the entity selecting the peer reviewers - shall (i) ensure that
those reviewers serving as federal employees (including special government employees) comply
with applicable federal ethics requirements; (ii) in selecting peer reviewers who are not
government employees, adopt or adapt the National Academy of Sciences' policy for committee
selection with respect to evaluating the potential for conflicts (e.g., those arising from
investments; agency, employer, and business affiliations; grants, contracts and consulting
income).  For scientific assessments relevant to specific regulations, a reviewer's financial ties to
regulated entities (e.g., businesses), other stakeholders, and the agency shall be examined.

-------
                                 Peer Review Handbook                       Page B-28
       c. Independence: In addition to the requirements of Section II (3)(c), which shall apply to
all reviews conducted under Section III, the agency  or entity selecting the reviewers 
shall bar participation of scientists employed by the sponsoring agency unless the reviewer is
employed only for the purpose of conducting the peer review (i.e., special government
employees). The only exception to this bar would be the rare case where the agency determines,
using the criteria developed by NAS for evaluating use of "employees of sponsors," that a
premier government scientist is (a) not in a position of management or policy responsibility and
(b) possesses essential expertise that cannot be obtained elsewhere. Furthermore, to be eligible
for this exception, the scientist must be employed by a different agency of the Cabinet-level
department than the agency that is disseminating the scientific information. The agency's
determination shall be documented in writing and approved, on a non-delegable basis, by the
Secretary or Deputy Secretary of the department prior to the scientist's appointment.
       d. Rotation: Agencies shall avoid repeated use of the same reviewer on multiple
assessments unless his or her participation is essential and cannot be obtained elsewhere.
  4. Information Access:  The agency  or entity managing the peer review  shall provide the
reviewers with sufficient information   including background information about key studies or
models  to enable them to understand the data, analytic procedures, and assumptions used to
support the key findings or conclusions of the draft assessment.
  5. Opportunity for Public Participation:  Whenever feasible and appropriate, the agency shall
make the draft scientific assessment available to the public for comment at the same time it is
submitted for peer review (or during the peer review process) and sponsor a public meeting
where oral presentations on scientific  issues can be made to the peer reviewers by interested
members of the public.  When employing a public comment process as part of the peer review,
the agency shall, whenever practical, provide peer reviewers with access to public comments that
address significant scientific or technical issues. To ensure that public participation does not
unduly delay agency activities, the agency shall clearly specify time limits for public
participation throughout the peer review process.
  6. Transparency: In addition to the requirements specified in 11(5), which shall apply to all
reviews conducted under Section III, the peer review report shall include the charge to the
reviewers and a short paragraph on both the credentials and relevant experiences of each peer
reviewer.  The agency shall prepare a written response to the peer review report explaining (a)
the agency's agreement or disagreement with the views expressed in the report, (b) the actions
the agency has undertaken or will undertake in response to the report,  and (c) the reasons the
agency believes those actions satisfy the key concerns  stated in the report (if applicable). The
agency shall disseminate its response  to the peer review report on the agency's website with the
related material specified in Section 11(5).
  7. Management of Peer Review Process and Reviewer Selection: The agency  may commission
independent entities to manage the peer review process, including the  selection of peer
reviewers, in accordance with this  Bulletin.

-------
                                 Peer Review Handbook                      Page B-29
IV.    Alternative Procedures.

As an alternative to complying with Sections II and III of this Bulletin, an agency may instead:
(i) rely on the principal findings, conclusions and recommendations of a report produced by the
National Academy of Sciences; (ii) commission the National Academy of Sciences to peer
review an agency's draft scientific information; or (iii) employ an alternative scientific procedure
or process, specifically approved by the Administrator in consultation with the Office of Science
and Technology Policy (OSTP), that ensures the agency's scientific information satisfies
applicable information quality standards. The  alternative procedure(s) may be applied to a
designated report or group of reports.

V.     Peer Review Planning.

  1. Peer Review Agenda: Each agency shall  post on its website, and update at least every six
 months, an agenda of peer review plans. The  agenda shall describe all planned and ongoing
 influential scientific information subject to this Bulletin. The agency shall provide a link from
 the agenda to each document that has been made public pursuant to this Bulletin. Agencies are
 encouraged to offer a listserve or similar mechanism to alert interested members of the public
 when entries are added or updated.
  2. Peer Review Plans: For each entry on the agenda the agency shall describe the peer review
plan. Each peer review plan shall include: (i) a paragraph including the title, subject and purpose
of the planned report, as well as an agency contact to whom inquiries may be directed to learn
the specifics of the plan; (ii) whether the dissemination is likely to be influential scientific
information or a highly influential  scientific assessment; (iii) the timing of the review (including
deferrals); (iv) whether the review will be conducted through a panel or individual letters (or
whether an alternative procedure will be employed); (v) whether there will be opportunities for
the public to comment on the work product to be peer reviewed, and if so, how and when these
opportunities will be provided; (vi) whether the agency will provide significant and relevant
public comments to the peer reviewers before they  conduct their review; (vii) the anticipated
number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct description of the
primary disciplines or expertise needed in the review; (ix) whether reviewers will be selected by
the agency or by a  designated outside organization; and (x) whether the public, including
scientific or professional societies, will be asked to nominate potential peer reviewers.
  3. Public Comment: Agencies shall establish a mechanism for allowing the public to comment
on the adequacy of the peer review plans. Agencies shall consider public comments on peer
review plans.

VI.    Annual Reports.

Each agency shall provide to OIRA, by December 15 of each year, a summary of the peer
reviews conducted by the agency during the fiscal year.  The report should include the following:
1) the number of peer reviews conducted subject to the Bulletin (i.e., for influential scientific
information and highly influential scientific assessments); 2) the number of times alternative
procedures were invoked; 3) the number of times waivers or deferrals were invoked (and in the

-------
                                 Peer Review Handbook                      Page B-30
case of deferrals, the length of time elapsed between the deferral and the peer review); 4) any
decision to appoint a reviewer pursuant to any exception to the applicable independence or
conflict of interest standards of the Bulletin, including determinations by the Secretary pursuant
to Section III(3)(c);  5) the number of peer review panels that were conducted in public and the
number that allowed public comment; 6) the number of public comments provided on the
agency's peer review plans; and 7) the number of peer reviewers that the agency used that were
recommended by professional societies.

VII.   Certification in the Administrative Record.

If an agency relies on influential scientific information or a highly influential scientific
assessment subject to this Bulletin to support a regulatory action, it shall include in the
administrative record for that action a certification explaining how the agency has complied with
the requirements of this Bulletin and the applicable information quality guidelines.  Relevant
materials shall be placed in the administrative record.

VIII.  Safeguards, Deferrals, and Waivers.

  1. Privacy: To the extent information about a reviewer (name, credentials, affiliation) will be
disclosed along with his/her comments or analysis, the agency shall comply with the
requirements of the Privacy Act, 5 U.S.C.  522a as amended, and OMB Circular A-130,
Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a
published System of Records Notice.
  2. Confidentiality: Peer review shall be conducted in a manner that respects (i) confidential
business information and (ii) intellectual property.
  3. Deferral and Waiver: The agency head may waive or defer some or all of the peer review
requirements of Sections II and III of this Bulletin where warranted by a compelling rationale. If
the agency head defers the peer review requirements prior to dissemination, peer review shall be
conducted as soon as practicable.

IX.    Exemptions.

Agencies need not have peer review conducted on information that is:
  1. related to certain national security, foreign affairs,  or negotiations involving international
trade or treaties where compliance with this Bulletin would interfere with the need for secrecy or
promptness;
  2. disseminated in the course of an individual agency adjudication or permit proceeding
(including a registration, approval, licensing, site-specific determination), unless the agency
determines that peer review is practical and appropriate and that the influential dissemination is
scientifically or technically novel or likely to have precedent-setting influence on future
adjudications and/or permit proceedings;
  3. a health or safety dissemination where the agency determines that the dissemination is time-
sensitive (e.g., findings based primarily on data from a recent clinical trial that was adequately
peer reviewed before the trial began);

-------
                                 Peer Review Handbook                      Page B-31
  4. an agency regulatory impact analysis or regulatory flexibility analysis subject to interagency
review under Executive Order 12866, except for underlying data and analytical models used;
  5. routine statistical information released by federal statistical agencies (e.g., periodic
demographic and economic statistics) and analyses of these data to compute standard indicators
and trends (e.g., unemployment and poverty rates);
  6. accounting, budget, actuarial, and financial information, including that which is generated
or used by agencies that focus on interest rates, banking, currency, securities, commodities,
futures, or taxes; or
  7. information disseminated in connection with routine rules that materially alter entitlements,
grants, user fees, or loan programs, or the rights and obligations of recipients thereof.

X.     Responsibilities of OIRA and OSTP.

 OIRA, in consultation with OSTP, shall be responsible for overseeing implementation of this
 Bulletin.  An interagency group, chaired by OSTP and OIRA, shall meet periodically to foster
 better understanding about peer review practices and to assess progress in implementing this
 Bulletin.

XL    Effective Date and Existing Law.

The requirements of this Bulletin, with the exception of those in Section V (Peer Review
Planning), apply to information disseminated on or after six months following publication of this
Bulletin, except that they do not apply to information for which an agency has already provided a
draft report and an associated charge to peer reviewers. Any existing peer review mechanisms
mandated by law shall be employed in a manner as consistent as possible with the practices and
procedures laid out herein. The requirements in Section V apply to "highly influential scientific
assessments," as designated in Section III of this Bulletin, within six months of publication of
this Bulletin. The requirements in Section V apply to documents subject to Section II of this
Bulletin one year after publication of this Bulletin.

XII.   Judicial Review

This Bulletin is intended to improve the internal management of the executive branch, and is not
intended to,  and does not, create any right or benefit, substantive or procedural, enforceable at
law or in equity, against the United States,  its agencies or other entities, its officers or employees,
or any other

-------
Peer Review Handbook                   Page C-l

-------
           Peer Review Handbook          Page C-2
          Appendix C

Sound Science and Peer Review in
       Rulemaking Policy

-------
                                Peer Review Handbook                      Page C-3
         SOUND SCIENCE AND PEER REVIEW IN RULEMAKING

Several provisions of the December 2004, OMB Bulletin for peer review made it necessary for
the Office of Policy, Economics, and Innovation (OPEI) to create conditional template language
(Attachment A).  This language should be used by rulewriters in the preamble of regulations that
rely on influential scientific information or a highly influential scientific assessment, which are
two categories of information defined in Section 2.2 of this handbook.

For proposed and final regulations that rely on influential scientific information or a highly
influential scientific assessment, rulewriters should use the template as a model to discuss peer
review in the preamble where appropriate.  In addition, peer review leaders should communicate
with rulewriters and workgroup chairs to ensure that all appropriate peer review material is
included in the docket, and that template language is included in the preamble.

OPEI also revised the Action Memorandum Framework to include a discussion of peer review
for influential scientific information or a highly influential scientific assessment (Attachment B).

-------
                               Peer Review Handbook                     Page C-4
                                                                     ATTACHMENT A
A
               Peer Review (Conditional Template)
Read this first (but DO NOT insert it in your preamble):

The OMB Final Information Quality Bulletin for Peer Review directs EPA to include a
discussion of the peer review report and how the Agency complied with the provisions of the
Bulletin in the preamble of rulemakings that are supported by influential scientific information or
highly influential scientific assessments. Peer review reports should either (a) include a verbatim
copy of each reviewer's comments (either with or without specific attributions) or (b) represent
the views of the group as a whole, including any disparate and dissenting views. The Agency
should disclose the names of the reviewers and their organizational affiliations in the report and
should notify the reviewers in advance regarding the extent of the disclosure and attribution
planned by the Agency.  You should ensure that the peer review report is placed in the docket to
comply with the OMB Bulletin.

Use this template if your proposed or final rule is based on a work product containing influential
scientific information or a highly influential  scientific assessment. This language should appear
in the Supplementary Information section of regulatory preambles under General
Information.  You may want to include the language under the heading:

             Did EPA conduct a peer review before issuing this notice?
II   PROPOSED & FINAL ACTIONS: If you used a highly influential scientific
assessment of influential scientific information to support this rulemaking, insert this into
the preamble of your proposed or final rule, advanced notice of proposed rulemaking, or
other substantive action:
   This regulatory action was supported by [influential scientific information or a highly
   influential scientific assessment]. Therefore, EPA conducted a peer review in accordance
   with OMB's Final Information Quality Bulletin for Peer Review.  [Insert a brief
   description of the peer review process along with any other relevant information.] The
   peer review report is located in the docket for today's action. According to the report, [insert
   a brief discussion of the peer review report. For more information about the peer
   review report, see the Peer Review Handbook.]

-------
                                Peer Review Handbook                     Page C-5
                                                                      ATTACHMENT B

                    FRAMEWORK FOR ACTION MEMORANDA

PURPOSE OF THIS FRAMEWORK

This Framework provides guidance on the format and contents of Action Memoranda that
accompany packages presented for signature.  A draft of the memo also accompanies the Final
Agency Review (FAR) package.  The Framework was developed to promote a consistent
approach to these memoranda and to preserve the flexibility necessary to accommodate the
unique needs of each program while ensuring that critical information is provided to Agency
managers in a clear and concise manner.  Generally, the Action Memoranda should not exceed 5
pages, and follows the format outlined in the Agency's Correspondence Manual
(http://epawww.epa.gov/rmpolicy/ads/manuals/1320.pdf).

As indicated by the use of non-mandatory language such as "should," "recommend," and "may,"
this Framework provides recommendations and does not impose any legally binding
requirements.  Recommendations for each section of Action Memoranda are discussed below.
Programs may include information on additional topics if they are relevant to a given action, e.g.,
information quality issues, etc.

Action Memoranda are considered internal, deliberative documents.  At the bottom or top of
each page insert this reminder: "Internal Document Only; Do Not Cite, Quote or Release."

A description  of the items to be included in an Action Memorandum follows.

DEADLINE

Indicate if there is a deadline of any sort and what that deadline is. It may be court imposed,
statutory, or any one of a number of other deadlines, e.g., upcoming national meeting,
Administrator's priority, etc.

OVERVIEW

The overview section of an Action Memorandum provides background information which
briefly describes the rule, characterizes the environmental issue(s) or public health problem(s)
being addressed, and summarizes the history of the action. It should explain why EPA is taking
this action, and where appropriate, cover the following points:

$    Define any court or statutory deadline;
$    Identify whether the action amends the Code of Federal Regulations, and if so, explain
     what kind of amendment (procedural, etc.);
$    Identify  other actions underway that will affect this particular program or sector;

-------
                                Peer Review Handbook                     Page C-6
$    Describe the specific environmental issue(s), public health problem(s) and/or statutory
     requirements being addressed, and the goal intended by taking this action;
$    Describe what the action does, and specifically, how the regulated community is affected
     (performance standards, specific requirements); and
$    Describe how flexible the implementation of the action will be for States and regulated
     entities.

IMPACTS

Summarize the costs and benefits, including a discussion of any non-monetized benefits, or
contingent valuation and/or non-quantified benefits, of the action and the results of any economic
analyses. Explain the roles of cost-benefit and cost-effectiveness analyses in shaping the
approach chosen.

Discuss the economic impacts on all affected entities including, where appropriate, possible
impacts on other Federal agencies, State, or Local governments, and Tribes.

Summarize the results of the regulatory flexibility analysis and impacts on small entities (small
businesses,  small communities and small not-for-profit entities), including what was done to
minimize those impacts. If applicable, summarize the SBREFA Panel recommendations, and
how they were addressed in the rulemaking.

Describe the reporting and record-keeping burden and what the Agency has done to reduce it.

STAKEHOLDER INVOLVEMENT

Briefly discuss the role of both government entities and private sector stakeholders in the
development of the action. Summarize the concerns they have raised and what the Agency has
done to address them, or explain why we can't.

INTERNAL REVIEW

Identify whether the action was developed under Tier 1, 2, or 3.  If the action was Tier 1 or 2,
attach the summary memo from the Final Agency Review. Describe any outstanding issues from
Final Agency Review or other internal review. Identify Program Offices or Regions that
participated in the development of the action, along with any outstanding issues and why they
cannot be resolved or accommodated.  Also, provide the basis for any decision made to not
address an identified cross-media impact.

-------
                                 Peer Review Handbook                      Page C-7
PEER REVIEW

If you did not use influential scientific information or a highly influential scientific assessment as
defined by the Peer Review Handbook to support the action, include the following statement in
the Action Memorandum:

     There were no influential or highly influential products supporting this action as defined by
     the Agency's Peer Review Handbook.

If you did use influential scientific information or a highly influential scientific assessment to
support the action, include the following statement:

     [Insert Name ofAAship] has followed the Agency's Peer Review Policy with respect to the
     underlying [influential scientific information or highly influential scientific assessment]
     supporting this action.

You may add any details you think are important, but you generally should not modify this
compliance statement. If you used influential scientific information or a highly influential
scientific assessment, but were not fully able to meet the Peer Review Policy, explain why.

PLAIN LANGUAGE

Briefly describe which elements of plain language you used in the action, e.g., question and
answer format, addressing the reader as "you," using short sentences, etc.

OMB TRANSACTION

Identify the OMB determination and whether the action went to OMB for review under
Executive Order 12866.  Highlight significant issues resulting  from OMB's review under
Executive Order 12866, including any significant issues raised by other agencies participating in
the interagency review under Executive  Order 12866.  Explain any substantive changes made to
the action as a result of recommendations from OMB or the other agencies. If OMB agreed that
your action was not  significant and, therefore, not  subject to EO 12866  review, or if OMB
waived review of your rule, please so indicate.  If OMB was otherwise involved, e.g., was
briefed or received a courtesy copy for informal review, describe the results of this interaction.

ANTICIPATED EXTERNAL REACTION

Describe the type of response anticipated from the various audiences interested or impacted by
the action. Identify both the involved stakeholders and the nature of their expected response.
Characterize the likely reaction to the action by all interested parties including industry;
environmental groups; Congress; State, Local, and Tribal governments; and OMB. Explain what
the Agency has  done to mitigate anticipated adverse  reactions.

-------
                                Peer Review Handbook                      Page C-8
STAFF CONTACT

Give the name and phone number of the person to contact if there are questions about the
package.

RECOMMENDATION

Identify the action the Administrator is expected to take (i.e., sign the rule or other document).

NOTE: Where an Action Memo accompanies another document (i.e., a rule or FR document) to
be signed by the Administrator (or other authorized official), you should not include a
concurrence line at the bottom of the Action Memo or anything else that might cause the Action
Memo to be misinterpreted to be a Decision Memo (which it is not).  The FR notice (or other
document such as an order), after being signed, will contain the Agency's decision or action (if
any), not the Action Memo, which is only a pre-decisional briefing document.

-------
Peer Review Handbook                   Page C-9

-------
           Peer Review Handbook          Page D-l
           Appendix D




Examples Of Peer Review Charges

-------
                              Peer Review Handbook                    Page D-2
 Appendix D - Examples Of Peer Review Charges
     Please Note - certain questions that are posed in charges can be responded to with a yes or
no answer. Clearly, this is not the type of response the Agency generally wants, therefore, it is
important to phrase charge questions carefully to ensure that you receive a fully satisfactory and
thoughtful response.  Where a yes or no answer might be expected, be sure to ask for a full
explanation supporting the yes or no answer.

     Charges can run the gamut from rather simplistic to highly complex.  The examples shown
here cover a variety of types. Examples  1 through 4 have less complex questions and are looking
for the overall quality of the efforts.  Examples 5 through 7 have numerous technical questions
that need to be addressed and are therefore more complex in their nature.

Other charges that have been used can be found on the SAB website at http://www.epa.gov/sab
and the SAP website at http://www.epa.gov/scipoly/sap/index.htm

-------
                                 Peer Review Handbook                      Page D-3
CHARGE EXAMPLE 1 - US EPA Science Advisory Board (SAB) Consultation on
Suspended and Bedded Sediments

Background:

The Ecological Processes and Effects Committee (EPEC) of the EPA Science Advisory Board
was asked by the Health and Ecological Criteria Division of the Office of Science and
Technology, Office of Water, to provide a consultation on potential approaches for a strategy for
developing water-quality criteria for Suspended and Bedded Sediments (SABS). A request for
nominations for consultant panel members to provide additional expertise to EPEC appeared in
the Federal Register on July 30, 2003 (68 FR 44758-44760).

In 1976, EPA issued a water quality criteria recommendation under the Clean Water Act for
solids and turbidity. For a variety of reasons, the States seldom, if ever, use this criterion.  It is
questionable whether this criterion would achieve intended protection for all different designated
uses for water bodies. SABS occurs naturally in streams in a wide range of concentrations -
levels that might be perfectly normal in one water body would be indicative of impairment in
another.

Although most States currently have water quality criteria that can be applied to manage SABS,
these are typically based on turbidity, suspended solids or settle-able solids, and their
effectiveness for dealing with all water quality impairments caused by SABS, especially as
benchmarks for aquatic life protection based on natural levels, is questionable. In recent
consultation with State representatives, the need for new water quality criteria for SABS or
methodologies for deriving them on a site-specific basis was identified as one of the highest
priorities for the water quality criteria program. As  a result, the EPA Office of Water has
concluded that to better manage SABS  in all types of water bodies and for all designated uses,
State and Tribal water quality managers need new and updated water quality criteria and
information for SABS.

Charge to the panel:

While many questions and much research remain, EPA seeks the opportunity for a consultation
with the Science Advisory Board to gain advice and recommendations on the best potential
approaches to developing water quality criteria for suspended and bedded sediments as will be
described in a draft Strategy for Developing Water  Quality Criteria for Suspended and Bedded
Sediments (SABS) to be prepared by the Office of Water. The Office of Water is also seeking
recommendations on additional criteria development and approaches for uses of water bodies
other than aquatic life, and it is also seeking advice on any potential criteria derivation
methodology not included in the Strategy.

-------
                                 Peer Review Handbook                       Page D-4
CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children

Background

EPA established the Office of Children's Health Protection (OCHP) in 1997 to support its efforts
to increase the protection of children's health throughout its programs. One of the many difficult
issues the Office is addressing is the appropriate treatment of children's health effects in the
economic analyses performed by the Agency. Policy analysis efforts at the Agency often rely on
the benefits transfer technique, and very few of the Agency's benefit transfers have explicitly
addressed children's health issues.  In addition, no accepted systematic process for conducting
benefits transfer currently exists. To assist the Agency in its efforts, this paper discusses the
benefits transfer technique as it applies to estimating values for children's health.  The first
section provides some general background on the technique, and its application to estimate
health-related values.  The second section raises important general issues to consider when
conducting a benefits transfer for children's health values. The last section discusses the
implications of using the benefits transfer method to estimate values for children's health. The
scarcity and state of existing child-oriented health valuation literature  suggests that it may be
necessary to transfer adult-oriented values to estimate child-related values (Neumann and
Greenwood 1999). However, the results of this paper suggest that transfer of these value
estimates to children at best provides estimates for a scoping analysis.  In cases where these
scoping exercises indicate that children's health values may be a crucial component in the policy
analysis, primary research should be undertaken to estimate child-related values.

a)       Assess the appropriateness of transferring health benefit values estimated for adult
     populations to children. Describe the specific issues that arise in these transfers. Which
     variables or situations improve or decrease the appropriateness of transferring benefit
     values from adults to children?

b)       Does the analysis support the proposition that the value of children's health effects
     should be estimated differently than adult health effects? Why or why not?

c)       Identify issues for further research that would improve our ability to estimate values for
     children's health effects.

-------
                                Peer Review Handbook                      Page D-5
CHARGE EXAMPLE 3 - Economics - Study of Municipal Government Costs and
Financial Impacts from Environmental Regulations

Background

Municipalities play a major role in supplying environmental services. Local governments have
taken responsibility for providing drinking water, sewage treatment, and waste disposal in a
majority of communities.  Over the past fifteen to twenty years, most of the mandates found in
the Federal environmental legislation enacted in the early 1970s have been met. The increase in
the number of people served and improvements in the quality of local environmental services
have been considerable, as has the investment in public infrastructure to meet these laws.

Recent revisions to the environmental legislation have established a broader and more stringent
set of standards to be met by suppliers of environmental services.  As a  result, many Local
governments are now faced with having to maintain all or some part of their public services at a
higher level  of performance.  To meet these new standards will require additional investments in
capital, and increases in rates charged to customers for environmental services.

Improvements in environmental services are but one of several demands being made of local
public infrastructure. Studies prepared on public infrastructure needs and the availability of
funds to meet these needs indicate that there will be an excess demand for money to rebuild and
improve upon the existing stock of public infrastructure. Therefore, it is important to recognize
that additional environmental requirements will have to compete with other infrastructure needs
(e.g., highways, bridges), as well as other public services (e.g., police, education, health and
welfare programs) provided at the local level.

Given the increasing demand for public services, this study examines what additional
investments  the new environmental legislation will require Local governments to undertake, and
the likelihood that they will face difficulties raising the necessary funds through capital markets
and revenues from customers. The economic impacts of individual EPA actions are considered
during the regulatory process in those situations permitted by environmental statutes. The
unique feature of this study is its attempt to estimate the cumulative costs and impacts of meeting
a combined set of EPA requirements, and to determine whether they will place a significant
burden on the fiscal conditions of Local governments, and require them to significantly increase
existing charges for improved environmental services.

Please find attached a copy of the draft study and appendices for your review.  This version of
the report reflects Agency comments received on an earlier draft.  We expect that this version  of
the report, with some additional minor modifications, will be the final version. Your comments
will be useful in preparing the final version of the report and discussing the findings of the
reports with  the public.

To assist in your review of the report, we ask that you pay particular attention to the following
questions:

-------
                            Peer Review Handbook                      Page D-6
a) Do State and Local governments and financial markets consider household costs
   (measured as a percent of household income) and selected municipal financial
   information (debt service to general revenues or taxable property values) when
   evaluating the ability of enterprise systems and municipalities to issue bonds or obtain
   loans?

b) Having selected a series of financial indicators, are the criteria used in the  analysis
   acceptable?

c) Do the results support our conclusions? What additional conclusions can be reached
   from the analysis?

d) What modifications would you suggest be made to the recommendation section?

-------
                                Peer Review Handbook                      Page D-7
CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National
Sewage Sludge Survey for Pollutants in Sewage Sludge

Background Information

The United States Environmental Protection Agency (EPA), Office of Water is charged with
protecting public health and the environment from adverse exposure to chemicals and microbial
agents in water media, such as ambient and drinking waters, wastewater/sewage sludge, and
sediments. In support of this mission the Office of Water/Office of Science and Technology
(OST) develops human health and environmental standards, health criteria, health advisories, and
technical guidance documents for water and water-related media.

Peer review is an important component of the scientific process.  It provides a focused, objective
evaluation of a research proposal, publication, risk assessment, health advisory, guidance or
other document submitted for review.  The criticisms, suggestions and new ideas  provided by
the peer reviewers ensure objectivity, stimulate creative thought, strengthen the reviewed
document, and confer scientific credibility  on the product. Comprehensive, objective peer
review leads to good science  and product acceptance within the scientific community and is
required for the technical document that support Agency regulations and guidelines.

Targeted National Survey of Pollutants in Sewage Sludge

As EPA described in the December 2003 Federal Register notice (68 FR 75531), EPA
plans to conduct a targeted survey of pollutants (select pollutants) in sewage sludge.  EPA has
designed a targeted survey of selected chemical pollutants for which adequate data are available,
or will be available, for conducting an exposure and hazard assessment concerning their presence
in treated sewage  sludge.  Microbial pollutants (pathogens and indicator organisms) in sewage
sludge are also included in the survey design.

A survey will provide feedback for updating the science and technology of sewage sludge
applied to land, disposed of in a surface disposal unit, or incinerated.  The new concentration
data would be used to assess human and ecological risks of identified, unregulated pollutants
found in  sewage sludge and identify pollutants for potential regulation.  In addition, the data on
microbial constituents in sewage sludge will help in determining the effectiveness of pathogen
reduction and elimination processes that are specified in the 40 CFR Part 503 Standards.

EPA is committed in FY 2005 to starting a limited analytical survey of pollutants found in
sewage sludge.  EPA expects this survey to address the pollutants identified by the exposure and
hazard screening assessment as presenting  a potential hazard, as identified in the current section
405(d)(2)(C) biennial review (68 FR 75531).  The Agency has evaluated the extent to which
sampling and analytical methodologies will allow expansion of the survey scope within available
resources to include  additional pollutants.

Questions for  Peer Review Charge

-------
                                 Peer Review Handbook                      Page D-8
a)            The sample design proposes excluding facilities with certain treatment types (i.e.,
       less than secondary treatment, partial treatment, and ponds), remotely located facilities
       (i.e., outside the contiguous states), and non-publicly owned facilities (i.e., Tribal and
       private). Do these exclusions appear to be appropriate?  If you disagree with EPA's
       proposed exclusions, your response should explain which groups should be included and
       why (e.g., what information will be lost if they are excluded).

b)            The sample design proposes including facilities that the Clean Watershed Needs
       Survey (CWNS) identifies as biosolids handling facilities, because preliminary review
       indicates that they have the same operations and characteristics as other POTWs for
       purposes of biosolids sampling. Do you agree with the inclusion of the biosolids
       handling facilities? If you disagree, your response should explain how their
       characteristics (e.g., type of treatment) would differ from other POTWs.

c)            The sample design proposes including facilities with no discharge, because they
       also may produce biosolids as a result of treatment. Do you agree with the inclusion of
       no discharge facilities?  If you disagree, your response should explain how their
       characteristics would differ from other POTWs.

d)            The sample design assumes that flow and mass are highly correlated.  Do you
       agree that this is  a reasonable assumption? If you disagree, your response should identify
       possible data sources for mass data, or describe alternative approaches to estimating mass
       values based upon available data.

e)            The sample design proposes facility selection based upon the binomial
       distribution. Do you agree that this is the  appropriate approach?  If you disagree, your
       response should describe your objections to the rationale for selecting the binomial
       distribution. Your response also should recommend another approach, describe this
       approach (unless it was included as an alternative in the  sample design), and provide
       rationale for its selection.

-------
                                Peer Review Handbook                     Page D-9
CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the
Development of Water Quality Criteria for Suspended and Bedded Sediments (SABS)

BACKGROUND INFORMATION:

EPA's Office of Science and Technology (OST) has water quality criteria programs for aquatic
life, human health, and for nutrients (total N, P, DO, chlorophyll, and clarity). EPA is initiating a
National Strategy for the Development of Water Quality Criteria for Suspended and Bedded
Sediments (SABS).  SABS occur naturally in all types of waterbodies.  In appropriate amounts,
sediment is essential to aquatic ecosystems. However, imbalanced SABS conditions have
repeatedly ranked high as a major cause of waterbody impairment. An imbalance of sediments
resulting from human activities can impact ecological integrity at several scales and trophic
levels.

EPA's Strategy for Water Quality Standards and Criteria, dated August 2003, is a document
which sets priorities to strengthen the foundation for protecting and restoring the Nation's
waters.  Imbalance in loading of SABS to the aquatic systems is now considered one of the
greatest causes of water quality impairment in the Nation.  Turbidity, suspended solids,
sediment, and siltation have been consistently listed as dominant polluting factors in 305(b)
Water Quality Reports in rivers and streams, lakes and reservoirs, ponds, wetlands and ocean
shoreline waters. In 1998, for example, approximately 40% of assessed river miles in the U.S.
were not meeting their water quality goals due to excessive sediments.  One of the top ten
priorities listed in the document is producing and implementing a strategy for the development of
suspended and bedded sediment criteria. These priorities were developed based on extensive
listening sessions and discussions with States and other partners.

To assess the effects of this growing environmental stressor, a number of States have identified
the specific measurements and levels at which sediment impairment occurs in their waters, but
many States have not adopted such SABS criteria into their State water quality standards.
Information from an EPA survey conducted in 2001 indicates that numeric sediment criteria of
some type were identified in 32 of 53 States, Territories and Washington, DC. But many States
have not adopted such sediment criteria into their water quality standards. As a result, SABS
problems are identified but perhaps underestimated, and few controls (water quality standards,
permit limits, TMDLs) are in place to reduce the overall impact of SABS.

Also, as part of the current effort to develop national  SABS criteria, the EPA Science Advisory
Board met on October 2, 2003 to discuss the strategy, including various approaches for
establishing criteria. The  Science Advisory Board recommended that several approaches to
SABS criteria development should be synthesized in  an overall approach that can identify
impaired SABS conditions as compared to  appropriate reference conditions.  These
recommendations are presented in the draft strategy document.

The methodologies presented in this draft strategy will be used by the States, Tribes, and
Territories to better manage SABS in water bodies across the country. This strategy presents

-------
                                 Peer Review Handbook                      Page D-10
EPA's blueprint for achieving the goals of developing sediment criteria methodologies and the
implementation of these criteria into water quality standards.

National Strategy for the Development of SABS Criteria

The Strategy for the development of SABS criteria describes the approach that EPA is planning
to take for developing scientific information relating to SABS and working with States to
encourage adoption of SABS criteria into State water quality standards pursuant to Section
303(c) of the Clean Water Act. The draft strategy for developing water quality criteria for SABS
is modeled on the approach used by EPA to develop water quality criteria for nutrients, where
appropriate.  However, the strategy does not establish dates by which States must develop water
quality standards for SABS.

States, under pressure to develop and issue total maximum daily loads (TMDLs) for SABS
impaired water bodies, are moving forward on their own to develop new and improved SABS
criteria.  The draft strategy highlights some of the State programs and looks for approaches and
methods that may be useful, either directly, or with adaptation, to the entire nation. Because
current States' efforts may serve as examples  for development of numeric SABS criteria, a
summary of States' programs are provided. In order to describe a possible SABS criteria
development process, an Oregon case study was prepared for a hypothetical aquatic system,
following a watercourse from the watershed to the receiving water (Section III.F of the draft
strategy).  The case study illustrates sediment criteria development through theoretical and actual
examples.

EPA does not anticipate developing ecoregional SABS  criteria for all waterbody types and all
geographic regions. The responsibility to develop and adopt SABS criteria, based on the
information  and methodologies presented in the draft strategy, will rest with States. EPA's draft
national strategy includes the following major elements:

a)           Outline of actions to be taken by EPA, States, Territories, and Tribes to develop
      criteria,  adopt criteria into water quality standards, and manage for SABS.

b)           Identification of appropriate indicators of water resource impairment due to
             imbalances of SABS.

c)           A process to stratify waterbodies by waterbody type, region, and designated uses
             for the development of SABS criteria.

d)           A series of approaches and technical tools for deriving SABS criteria specific to a
             region, waterbody-type,  and designated use.

e)           A list of resources from  which States  and Tribes can find assistance for deriving
             SABS criteria.

-------
                                 Peer Review Handbook                      Page D-11
f)             A case study may be developed to illustrate SABS criteria development,
              addressing both the theoretical basis and an example using a synthesis of
              approaches.

Following peer review and incorporation of peer review comments, the U.S. Environmental
Protection Agency (EPA) will publish a draft National Strategy for the Development of Water
Quality Criteria for Suspended and Bedded Sediment (SABS) and will ask the scientific
community and the public for its views on the scientific direction of its SABS strategy. Peer
review is an important component of the scientific process. It provides a focused, objective
evaluation of a research proposal, publication, risk assessment, health advisory, guidance or
other document submitted for review. The criticisms, suggestions and new ideas provided by
the peer reviewers ensure objectivity, stimulate creative thought, strengthen the reviewed
document, and confer scientific credibility on the product.  Comprehensive, objective peer
review leads to good  science and product acceptance within the scientific community and is
required for the technical document that support Agency regulations and guidelines.

The Peer Reviewers charge is as follows:

a)             Does EPA's draft national strategy adequately describe methods for assessing the
       expected  SABS condition of a waterbody, for developing SABS criteria and for assessing
       attainment of criteria by States.

b)             Is a regional and watershed approach, as described in the strategy, likely to
       accomplish the desired goal of addressing the sediment imbalance problems in the
       nation's surface waters? If not, what alternatives do you suggest?

c)             Does the draft strategy present a rational, practical and scientifically responsible
       approach that can be implemented by States and Tribes, given their resources?

d)             Is the state-of-the-science sufficient to write technical methods manuals that will
       allow States, Tribes and territories to determine the best method(s) for developing SABS
       criteria into water quality standards pursuant to Section 303(c) and 304(a) of the CWA?

e)             Are the assessment endpoints and indicators described in the strategy reflective of
       the current or best available science?

f)             Are the six major elements of the draft strategy adequate to help States, Tribes,
       and territories develop and adopt SABS criteria?

g)             Does the case study illustrate sediment criteria development through the
       theoretical and actual examples presented?

-------
                                Peer Review Handbook                      Page D-12
CHARGE EXAMPLE 6 - CASAC Review of the Agency's National Ambient Air
Monitoring Strategy

Background.

The draft National Ambient Air Monitoring Strategy (NAAMS or Strategy) was revised after the
Agency's prior consultation with the former CASAC National Ambient Air Monitoring Strategy
(NAAMS) Subcommittee in July 2003. This revision incorporates EPA's responses to that
Subcommittee's recommendations. The primary recommendations from the NAAMS
Subcommittee included a request for an implementation plan, and added emphasis on rural- and
ecosystem-oriented monitoring, support for the National Core Monitoring Network
(NCore) Level 1 program, and training and quality assurance to enhance data consistency across
the Nation.

The Strategy foresees moving resources from programs of decreasing value to those of a higher
value which respects the partnership across EPA, State, Local, and Tribal (SLT) agencies
retaining stability for the monitoring programs and allowing SLT flexibility. The transition to the
NCore network creates a need for training that addresses new methods, information transfer
technologies, and an effective quality assurance program. There are programmatic and technical
areas where  some type of training or a transfer of information is required. These training needs
will be offered by various mechanisms, e.g., satellite broadcasts and videos; hands-on sessions;
guidance documents; vendor training of instrumentation; web-based training; and workshops.
Broadening the Agency's outreach to the health effects, atmospheric scientists and ecosystem
assessment communities are included.

The implementation plan incorporates action-oriented components of the Strategy, e.g.,
regulation revisions, training, funding, and outreach approaches to facilitate the implementation
of the NCore program. The revised monitoring regulations will provide  a legal basis for moving
forward and will also alleviate some of the unnecessary burdens faced by monitoring agencies
and enhance the ability to introduce new technologies into our networks. Additionally, the
regulations introduce the NCore system of: multiple-tiered monitoring stations;  adjustment of
minimum requirements for specific pollutant measurements; new methods performance
specifications; periodic network assessments; and new quality assurance procedures. Written
Meeting Materials. OAQPS has posted written review and background materials for this
Subcommittee meeting on EPA's Ambient Monitoring Technology Information Center
(AMTIC) Web site.  The Final Draft NAAMS document, which was updated following the July
2003 meeting of the former CASAC NAAMS Subcommittee, is at
http://www.epa.gov/ttn/amtic/files/ambient/monitorstrat/allstrat.pdf Additional background
materials for this meeting are found on the "CASAC File Area" page  of the AMTIC Web site at
http://www.epa.gov/ttn/amtic/casacinf html. Furthermore, it is our understanding that the SAB
Staff Office will post a copy of the final agenda and charge to the  Subcommittee for this
advisory meeting on the SAB Web site at http://www.epa.gov/sab (under "Meeting Agendas")
and the Subcommittee's page at http://www.epa.gov/sab/panels/casac aamm subcom.html,
respectively, in advance of the Subcommittee's meeting.

-------
                                Peer Review Handbook                      Page D-13
Charge to the Panel:

For this advisory meeting, OAQPS is requesting that the CASAC AAMM Subcommittee provide
expert advice and recommendations on the following charge questions, which focus on key
implementation issues:

       a)     The CASAC has expressed its support for the Agency's proposal to redesign the
             routine PM monitoring network to support PM precursor gas measurements (CO,
             SO2, NO/NOy, NH3, HNO3) at NCore Level II multiple-pollutant sites, and for
             air quality management decisions and to obtain relevant exposure data for
             research programs.

             Questions: Given limited budgetary resources, does this represent both an
             appropriate and adequate balance, as reflected by the relative resource allocations
             provided in Section 11, "Draft Implementation Plan," of the Final Draft NAAMS
             Document? In addition, are the relative adjustments in the training and guidance
             approaches proposed in the draft implementation plan consistent with the overall
             objectives of the Strategy?

       b)     The implementation plan proposes a series of communication actions to advance
             the NCore Level 2 network, in order to more directly support long-term health
             effects research and provide better support to ecosystem assessments through an
             increased level of coordination.

             Questions: Does the Subcommittee have additional suggestions for addressing
             this need for integration and communication to the broader community of "users,"
             including scientific researchers (i.e.,  human health, atmospheric, ecological) and
             State, Local and Tribal (SLT) Agency representatives? More specifically, what is
             the most effective manner for EPA both to reach-out to this broad user
             community and, where appropriate, to incorporate their feedback and design input
             on such issues as monitoring site locations and parameters?

       c)     One of the remaining technical issues relates to harmonizing rural- and urban-
             based 2.5 chemical speciation networks such that both categories of networks
             utilize consistent sampling and analysis protocols. For example, EPA is
             considering converting all of the Speciation Trends Network (STN) speciation
             sites to Interagency Monitoring of Protected Visual Environments (IMPROVE)
             samplers and IMPROVE laboratory and sample handling protocols.

             Question: What are strengths and weaknesses of this approach?
       d)     As EPA implements the National Ambient Air Monitoring Strategy to address
             multiple monitoring objectives, it will be looking to spatially optimize the
             ambient monitoring networks. This may mean that some redundant monitors in

-------
                    Peer Review Handbook                      Page D-14
adjacent, but separate, geopolitical areas (e.g., neighboring counties) are
"divested" from a given network. Although technically sound, these divestments
could result in data gaps which might, in turn, adversely impact regulatory
decision-making. The Agency is willing to adopt alternative approaches for
assessing regulatory issues such as non-attainment designations, so long as such
approaches are scientifically justifiable; hence, the rationale for initiating
discussion of these issues with the CASAC.

Question: Is it scientifically acceptable to generate air quality surfaces through
modeled observations and/or integrated predictive/observational fields that would
be of appropriate uncertainty for use in the regulatory decision-making process?

-------
                                Peer Review Handbook                      Page D-15
CHARGE EXAMPLE 7 - The Multimedia, Multipathway, and Multireceptor Risk
Assessment (3MRA) Model

Background:

The Office of Solid Waste (OSW) asked the Science Advisory Board to review the Multimedia,
Multipathway, and Multireceptor Risk Assessment (3MRA) Model and this request was accepted
by the Board, becoming SAB Project #03-13. A "Widecast" Federal Register notice requesting
nominations for the Panel was published on April 11, 2003 (68 FR 17797).

In brief, the development of the 3MRA methodology has a history in the Hazardous Waste
Identification Rule. EPA plans to use the modeling system to help inform managers on a variety
of decisions in the waste program, such  as setting concentration-based exit criteria for wastes in
the hazardous waste management regulations, or deciding whether technology-based  standards
are protective of human health and the environment.

In December 1995, EPA's Office of Solid Waste proposed to amend existing regulations for
disposal of listed hazardous wastes under the Resource Conservation and Recovery Act(RCRA).
This Hazardous Waste Identification Rule (HWIR) was designed to establish constituent-specific
exit levels for low risk solid wastes that are currently captured in the RCRA subtitle C hazardous
waste system. The EPA's Science Advisory Board (SAB) reviewed the proposed HWIR
methodology for calculating exit concentrations, concluding that the methodology "lacks the
scientific defensibility for its intended regulatory use," the SAB also made recommendations
that, when implemented, should provide an adequate scientific basis for establishing a risk-based
methodology applicable at the national level for the waste program.

In response, the Office of Solid Waste (OSW) collaborated with the Office of Research and
Development (ORD) to develop and document a sound science foundation, supporting data for
an assessment, and related software technology for an integrated, multimedia modeling system
(entitled 3MRA) following the recommendations of the SAB and other reviewers. The
Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) modeling system
represents a collection of science-based  models and databases that have been integrated into a
software infrastructure that is based on the FRAMES (Framework for Risk Analysis in
Multimedia Environmental Systems) concept, which provides a computer-based  environment for
linking environmental models and databases and managing the large amounts of information
within the system, including the visualization of outputs. This integrated multimedia modeling
system provides national-level estimates of human and ecological risks resulting from long-
term(chronic) chemical release from land-based waste management units. Over 45 experts
participated in the peer review process of the underlying science within the 3MRA modeling
system.

Charge to the Panel:

-------
                                Peer Review Handbook                      Page D-16
The SAB Staff Office Director and OSW staff concurred that the panel should focus upon the
following four areas: assessment methodology, 3MRA modeling system, modeling system
evaluation, and modeling system documentation. The panel will respond to the following charge
questions:

a)     Charge Question 1: While EPA had the assessment methodology peer reviewed prior to
             the development of the 3MRA modeling system, does the SAB have any
             additional comments about the methodology as implemented?

b)     Charge Question 2a: Does the 3MRA modeling system provide a tool for performing
             national risk assessments that facilitates consistent use of the science and provides
             a mechanism for reproducing results?

c)     Charge Question 2b: Does the 3MRA modeling system provide Decision Makers
             sufficient flexibility for understanding the impacts on potential chemical
             exemption levels by allowing varying measures of protection based on the
             number of receptors and/or number of sites protected, types of human and
             ecological receptors, and distance?

d)     Charge Question 2c: Does the 3MRA modeling system provide appropriate information
             for setting national risk-based regulations for the waste program?

e)     Charge Question 3a: Is the software development and verification testing approach
             implemented for the 3MRA modeling system sufficient to ensure confidence that
             the modeling results reflect the modeling system design?

f)     Charge Question 3b: Given the thorough evaluations that EPA has implemented using the
             available data resources and technologies, while also recognizing the real world
             limitations that apply to validating the 3MRA modeling system, have we
             reasonably demonstrated through methodology design, peer review, quality
             control, sensitivity analyses, and model comparison, that the 3MRA modeling
             system will produce scientifically sound results of high utility and acceptance
             with respect to multimedia regulatory applications?

g)     Charge Question 4: Has EPA made substantive progress, relative to 1995, in designing
             and preparing documentation for the  3MRA modeling system? Does the SAB
             have additional suggestions for improving the presentation of the comprehensive
             set of materials related to this modeling system?

-------
              Peer Review Handbook          Page E-l
             Appendix E

Guidance on Requesting a Review by the
     Science Advisory Board (SAB)

-------
Peer Review Handbook                  Page E-2

-------
                               Peer Review Handbook                     Page E-3
       APPENDIX E - GUIDANCE ON REQUESTING A

REVIEW BY THE SCIENCE ADVISORY BOARD (SAB)

Introduction

The Science Advisory Board Staff Office (SAB SO) solicits nominations for project proposals to
be reviewed by three of the Agency's independent scientific advisory committees the Science
Advisory Board (SAB); the Clean Air Scientific Advisory Committee (CASAC); and the
Advisory Council on Clean Air Compliance Analysis (Council).  This Appendix provides
guidance to EPA Program Offices and Regions regarding the project proposal nomination and
selection process. More detailed information about the functions and advisory process for the
SAB can be found in the Implementation Plan for the New Structural Organization of the EPA
Science Advisory Board (EPA-SAB-04-002) available on the SAB website at
http://www.epa.gov/sciencel/pdf/sab04002.pdf

Background

A key priority for the Administrator is to base Agency actions on sound scientific data, analyses,
and interpretations, as well as independent peer review. The Administrator issued the Agency's
Peer Review Policy to increase the quality of the technical foundations upon which EPA's
regulatory structures are built.  The SAB, CASAC, and the Council provide mechanisms for
EPA to receive independent scientific peer review and advice regarding the development and use
of science at EPA. The following information is intended to help programs and Regions
determine which scientific advisory projects to submit to the SAB, CASAC and Council.

The SAB, CASAC, and Council provide several kinds of advisory functions including the
following kinds of activities:

       Consultation - Provides non-consensus, oral advice on a technical issue before EPA
      begins substantive work on that issue.

       Advisory - Provides written advice on EPA's technical works-in-progress.

       Peer Review - Conducts a review of EPA's final draft technical reports (e.g.,
      guidelines, assessments, research strategies) or work products (e.g., analytical methods,
      models, databases).

       Commentary -  Provides forward-looking advice on an important technical or
      emerging issue in the form of a short communication (SAB function).

       Original Study - Conducts original work on an emerging or overarching topic of
      importance to EPA (SAB function).

-------
                                 Peer Review Handbook                      Page E-4
         Other Activities - Receives information briefings from EPA and conducts scientific
       workshops on specific technical subject matters.

Because resources are always limiting, the SAB Staff Office uses several criteria for selecting
project proposals proposed by the Agency.  Advisory project proposals best suited for
consideration by the SAB, CAS AC, and Council are those that meet several of the following
criteria:
         General Criterion
              S Provides an opportunity to make a difference in the science that supports the
              Agency's mission

         Client-Related  Criteria
              S Supports major regulatory or risk management initiatives
              S Serves leadership interests (e.g. the Administrator or Congress)
              S Supports EPA strategic priorities

         Science-Driven Criteria
              S Involves scientific approaches that are new to the Agency
              S Addresses areas of substantial uncertainties

         Problem-Driven Criteria
              S Involves major environmental risks
              S Relates to emerging environmental issues
              S Exhibits a long-term outlook

         Organizational Criteria
              S Serves as a model for future Agency methods
              S Requires the commitment of substantial resources to scientific or technological
              development
              S Transcends organizational boundaries, within or outside EPA (includes
              international boundaries)
              S Strengthens the Agency's basic capabilities.

In addition, the SAB Staff Office considers the overall mix of the nominated project proposals
for a specific fiscal year, as well as the time and available resources needed to take on the
projects.  Table 1 depicts the processes for identifying, nominating, and selecting Agency
nominations, emerging issues for strategic advice, and de novo studies. The SAB, CASAC and
Council employ similar processes, but projects for the latter two advisory committees are linked
to specific statutory mandates.

Process for Submitting Nominations
       Any Office desiring to take a product, activity, or issue to the SAB, for a peer review,
advisory, or consultation,  is requested to complete the two-step process, described below.

-------
                  Peer Review Handbook
Page E-5
TABLE 1 - PROCESS FOR SELECTING SCIENCE ADVISORY BOARD (SAB) PROJECTS
~~~~^___^ STEPS TO SAB
TYPE ~~---^^^ PROJECT
OF ^~"~~~~~~^ELECTION
PROJECT "~---~^
AGENCY REQUESTS


IDENTIFYING EMERGING
ISSUES FOR SAB STRATEGIC
ADVICE AND DENOVO
STUDIES




IDENTIFICATION
Achieved through
discussions within EPA
Offices to determine
projects which would
benefit from external
review
Achieved through
Interactions with:
EPA Program/Regional
Offices
Scientific Community
Public
Science Advisory Board
SAB Staff Office


FORA FOR DISCUSSION
AND NOMINATION
Staff Office receives
nominations from EPA
Assistant
Administrators (AAs)
or Regional
Administrators (RAs)
- Annually
- As needed basis
SAB Annual Meeting
SAB Committee
Meetings
Board Meetings
Meetings with Public
Science Policy Council
EPA Program/Regional
Offices
Scientific Workshops

PRIORITIZATION AND SELECTION
Achieved through joint discussions
with:

- Science Policy Council
- EPA Program/Regional offices
- SAB
Selection based on:
- Statutory Requirement
- Agency needs
Achieved through joint discussions
with:
- Science Policy Council
- EPA Program/Regional offices
- SAB
To identify projects for further
consideration.
- Strategic Advice by SAB
- De novo SAB project
Selection based on:
- Impact on Agency ' s mission
- Innovative nature of project
- Agency needs

DEFINING THE
SCOPE AND CHARGE
Achieved through joint
discussions with:

-SAB Staff Office
- the requesting EPA
Program/Regional office
Achieved through joint
discussions with:
- Designated SAB
Committee
- EPA Program/
Regional Office



APPROVAL
SAB Staff Office considers:
- Availability of resources

SAB Staff Office considers:
- Availability of resources





-------
Step 1 - Project Identification and Nomination. Each year, the Assistant Administrators and
Regional Administrators are asked to send the SAB Staff Office Director a memorandum that
lists all advisory project nominations, with the highest priority nominations for the next fiscal
year identified.  SAB Project Sheets (see Item 2 below) for nominated projects should be
attached to the memorandum. The SAB Staff Office and the Science Policy Council also request
this memorandum to identify four to six other high-priority science activities that the Assistant
Administrator or Regional Administrator has decided not to nominate for SAB peer review. The
latter information is requested, so that the SAB may better understand nominated projects within
the context of high-priority science activities at EPA.  The SAB Staff Office requests this
information to assist the Science Policy Council in its discussions and decisions about the most
appropriate set of projects for the Agency to bring to the Board, CASAC , or the Council.

Step 2 - Electronic Project Sheet. Nominators are asked to submit an electronic project sheet
for each individual project to be considered for SAB,  CASAC, or Council attention.  The project
sheets are created after establishing or updating the related Peer Review Project or Science
Activity in the Agency's Science Inventory.  The Science Inventory entries must be approved by
their Peer Review Coordinator or Science Activity Coordinator for the specific Program Office
or Region. Project sheets should be filled out for all desired projects, including previously
submitted projects for which no project planning meeting has occurred between the Program
Office or Region and the SAB Staff Office Director or the Associate Director for Science.  The
electronic Project sheet may be accessed and completed through EPA's Science Inventory
database.  The information fields required for the Electronic data sheet are  provided in Table 2.

Process for Keeping Informed about the Decisions Made

       After receiving project nominations, the SAB  Staff Office will discuss project priorities
with each EPA Program Offices and Region, and discuss overall Agency priorities with the EPA
Science Policy Council.  The Staff Office will also discuss project priorities with the SAB, which
includes the CASAC and Council chairs.  The SAB Staff Office will then develop a
recommended list of advisory projects based on EPA  and SAB rankings taking into  account
available resources. The SAB Staff Office will consult with the EPA Administrator and Deputy
Administrator to develop an annual operating plan that includes the highest priority projects.
Additionally,  projects may be added or deleted at any time during the fiscal year, as requested by
EPA senior executive management, Congress, and the SAB.

       EPA Staff may refer to the  SAB Product Database in an ongoing way to identify all  the
projects nominated for the Board and the status of decisions made on them. The database will
also provide ongoing information about the status of SAB reports and meetings related to
projects accepted for Board advice in Fiscal Year 2006.

-------
       EPA Staff with questions about the SAB Product Database or the process for submitting
nominations to the EPA SAB may contact Angela Nugent (nugent.angela@epa.gov, 202-343-
9981) or Ms. Patricia Thomas (thomas.patricial.epa.gov, 202-343-9974).
              Table 2:  Information Fields for SAB Project Sheet

 1.    Project Title
 2.    Project Short Title
 3.    Fiscal Year SAB Activity Desired to Begin
 4.    Quarter SAB Activity Desired to Begin
 5.    Requesting Organization
 6.    Requesting Office
 7.    Requesting Official
 8.    Requesting Official's Title
 9.    Program Contact
 10.  Program Contact's Phone
 11.  Program Contact's Mail Code
 12.  Background  for this Project:
 13.  Tentative Charge:
 14.  Applicable GPRA Goal and Objective
 15.  Description of and citation for any legal obligation/directive for SAB Review:
 16.  Principal interested and affected parties
 17.  Type of SAB advice requested
 18.  Why Should  the SAB Advise on this Project?
 19.  Disciplinary  Expertise
 20.  Budget:
         FY
         Extramural Budget
         FTE
 21.  Past Peer Reviews:
 22.  Quality Management/Quality Assurance:

-------
                         Appendix F

       Example  Statements  Of Work for
                           Contracts
Users of this Peer Review Handbook need to be aware that the examples contained in this
Appendix are generalized statements of work prepared (and in some cases modified) to
emphasize certain important features (e.g., attention to conflicts of interest, responsibilities of
contractors or contracting officers, development and use of the charge to peer reviewers).  Please
be sure that proper and currently approved contract language is used in any EPA contract
document at the time of award (for example, see EPAAR 1552.212-71, alternate I).

-------
Peer Review Handbook                  Page F-9

-------
                                Peer Review Handbook                      Page F-10
STATEMENT OF WORK - EXAMPLE 1 - Peer Review of Prioritization Tool Report

Work Assignment No.:

Title:  Peer Review of Prioritization Tool Report

Work Assignment Manager (WAM):

       Name:        John Q. Government Employee
       Address:     Office of Solid Waste
       Phone No.:    (202) 564-XXXX

Background:

The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the process of
implementing the Waste Minimization National Plan, announced by the Agency on November
18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste
management, in keeping with the policy stated in the 1984 amendments to the Resource,
Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act (PPA).  The
Plan outlines major goals, objectives, and action items to achieve national reductions in the
generation of hazardous wastes.

One  of the objectives of the Plan is to: "develop a framework for setting national priorities;
develop and distribute a flexible screening tool for identifying priorities at individual facilities;
[and] identify constituents of concern." This objective is a key building block in implementing
subsequent objectives of the Plan.

In September 1995, WMB formed the Waste Minimization Prioritization Team, which includes
representatives from EPA Regions  and states, to implement this objective. The Team has
worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools that
are currently available.  The Team plans to summarize this work, along with its
recommendations, in  a report (referred to herein as the Prioritization Tool report) that would be
available in draft form in July 1996.

WMB and the Team wish to obtain independent peer review of the Prioritization Tool report
prior to briefing EPA management.  The report is being prepared with the support of ICF, Inc.;
therefore, for the peer review to be  considered independent, it must be performed by another
contractor.

Purpose and Scope of Work:

The purpose of this work assignment is to provide support to WMB and the Team in finalizing
the Prioritization Tool report by conducting an independent peer review of the report.

-------
                                Peer Review Handbook                     PageF-11
Work Statement:

Task 1 -Management work plan and budget

Within 15 days of CO approval of this work assignment, the contractor shall deliver a
management work plan including a proposed level of effort, schedule, and budget for all tasks.

Task 2 -Provide independent peer review of Prioritization Tool report

The contractor shall provide support to WMB and the Team in preparing the Prioritization Tool
report by performing  an independent peer review of the report. The contractor shall establish a
panel of peer reviewers including three senior-level persons who collectively have extensive
expertise in particular areas to be identified by the WAM upon approval of the work assignment.

Within three weeks of work assignment approval, receipt from the WAM of the necessary
qualifications of peer reviewers (in a TD), and receipt from the WAM of the peer review
"charge" (in a TD), whichever comes latest, the contractor shall identify the three peer reviewers
and prepare a memo that lists the names of the peer reviewers and their affiliations and includes
the peer reviewers' bio's. Within five weeks of WAM approval of the of the peer reviewers (via
a TD) and receipt of the draft Prioritization Tool report from the WAM (via  a TD), whichever
comes later, the contractor shall conduct the peer review, assemble the peer review comments
and recommendations in a peer review report organized by charge question, prepare an
introduction to the peer review report with a clear and concise overview of the comments, and
attach to the peer review report any marginal comments the peer reviewers had on the
Prioritization Tool report.

It is not necessary that the peer reviewers jointly reach consensus on their findings and
recommendations, since there may be limited overlap in the peer reviewers' areas of expertise
and in the charge questions that they focus on. The contractor shall assume,  for the purpose of
estimating costs, that  the draft Prioritization Tool report is roughly 100 pages in length with 200
pages of appendices,  and that each peer reviewer will spend 40 hours in reviewing the report and
writing comments. EPA plans to provide the report to the contractor in mid-July.
Deliverables and Schedule:

-------
                                 Peer Review Handbook
Page F-12
Task
1
2
3
Deliverable
Work plan and budget
Memo identifying peer reviewers
Peer review report
Schedule
Within 15 days of CO approval of work assignment
Within 3 weeks of work assignment approval, receipt of peer reviewer
qualifications from WAM, and receipt of charge from WAM,
whichever comes latest
Within five weeks of WAM approval of peer reviewers and receipt of
draft Prioritization Tool report from WAM, whichever comes later
Other Requirements:

CONTRACTOR COMMUNICATIONS

Upon approval of the Work Plan, the contractor shall maintain at least weekly communications
with the Work Assignment Manager regarding the status of work on the Work Assignment.

CONFLICT OF INTEREST (COD  [Revised Section]

The contractor must adhere to the following requirements:

       a)     Upon receipt of a Work Assignment,  QRT, or similar tasking document, and prior
             to commencement of any work, notify both the CO and Project Officer of any
             actual or potential organizational or personal conflicts of interest.

       b)     Provide a written certification, within 20 days of receipt of a Work Assignment,
             QRT, or similar tasking document, that:

             1)     Either all conflicts of interest have been reported to the CO or that no
                    conflicts of interest exist. The contractor is directed to assure that none of
                    the conflicts disclosed are so direct and substantial as to rule out a
                    particular reviewer.

             2)     All personnel who perform work under this Work Assignment or relating
                    to this Work Assignment have been informed of their obligation to report
                    personal and organizational conflicts of interest to the CO.

             3)     The Contractor recognizes its  continuing obligation to identify and report
                    any conflicts of interest arising during performance of this Work
                    Assignment.

       c)     If a conflict of interest  is identified during performance under this Work
             Assignment, the Contractor shall immediately make a full disclosure in writing to
             the CO.  The disclosure shall include  a description of action which the Contractor

-------
                                 Peer Review Handbook                      Page F-13
              has taken or proposes to take, after consultation with the CO, to avoid, mitigate,
              or neutralize the conflict of interest.

       d)     After selecting the peer reviewers but before starting the peer review, submit to
              EPA documentation that shows that the contractor has determined if the peer
              review candidates:

              1)     have a conflict of interest or a situation that could create the appearance of
                    a lack of impartiality in relation to the work product, and

              2)     have had or presently have a financial relationship with EPA.

                    This documentation summarizes for EPA the Contractor's efforts to
              identify and propose resolution of these concerns with peer review candidates.

EXPENDITURE OF FUNDS/HOURS

In addition to the requirements of the contract, the contractor shall notify both the Project Officer
and the Work Assignment Manager when 75% of funds or hours for this Work Assignment have
been expended.

INFORMATION  COLLECTION

Any other provision of this Work Assignment notwithstanding, the contractor shall not proceed
with any information collection where the same or similar information will be collected from ten
or more public respondents until written approval is received from the Contracting Officer.  This
approval will cite an approval number from the Office of Management and Budget as required
by the Paperwork Reduction Act (PRA).

Only Federal agencies and their employees  are exempt from the PRA definition  of "public
respondent."  State agencies and their employees are classified as "public respondents."

Soliciting similar information applies to any collection method, i.e., written, oral, electronic,  etc.,
and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc. The PRA applies
equally to "willing  participants" and participation that is mandated by law.

Any question of applicability of the PRA shall be resolved by submitting a complete description
of the circumstances in a written request to the Contracting Officer.  No collection shall be
undertaken until the Contracting Officer provides written notice to the Contractor as to the
applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not
initiate any collection until the requisite approval is received.

The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the
Interagency Reports Management Program  as derived from 44 USC Chapters 29 and 31.  All

-------
                                Peer Review Handbook                     Page F-14
work performed under this Work Assignment involving Federal interagency reporting must be
done in full compliance with these GSA procedures.

CONFIDENTIAL BUSINESS INFORMATION

If this Work Assignment requires use of RCRA Confidential Business Information (CBI), the
contract must specifically authorize the contractor to have access to RCRA CBI and the
contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA CBI
Security Manual and in the contract. The contractor shall identify in the Work Plan budget all
estimated costs for dealing with CBI requirements.  All CBI must be returned to EPA as soon as
it is no longer needed under this Work Assignment or before the expiration of the Work
Assignment, whichever occurs first.

PRINTING AND DUPLICATION

The contractor is prohibited from performing any printing under the Government Printing and
Binding Regulations.  Duplication is allowed to the extent it does not exceed 5,000 impressions
of a single-page document or 25,000 impressions of a multiple-page stand-alone document, is
limited to one color (black) copies, and does not exceed the maximum image size of 10 3/4 by 14
1/4 inches.  For all duplication jobs in excess of 5,000 impressions,  the EPA WAM will
determine in advance if the work can be performed more cost effectively and under the job or
time constraints at the EPA Print Shop.  If the total number of photocopies for this Work
Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the
photocopying costs by task and deliverable.

WORK ASSIGNMENT/WORK PLAN BUDGETS

The contractor shall not exceed either the dollar or PL hour budget contained in the approved
Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not exceed the PL
hour budget of the QRT.

TECHNICAL DIRECTION

The Designated Work Assignment Manager (WAM) on this Work Assignment is authorized to
provide technical direction to the extent allowed under EPAAR (1552.237-71) (APR 1984)
(DEVIATION). Other than the Designated WAM, only the Project Officer and the Contracting
Officer are authorized to provide technical direction.

Technical direction includes:

      (1)    Direction to the contractor which assists the contractor in accomplishing the
      Statement of Work;

      (2)    Comments on and approval/acceptance of reports or  other deliverables.

-------
                                Peer Review Handbook                      Page F-15
Technical direction must be within the contract and the Work Assignment statement of work.
The Project Officer and the WAM do not have the authority to issue technical direction which

       (1)    institutes additional work outside the scope of either the contract or this Work
             Assignment;

       (2)    constitutes a change as defined in the "Changes" clause;

       (3)    causes an increase or decrease in the estimated cost of the contract or Work
             Assignment;

       (4)    alters the period of performance or deliverable due dates; or

       (5)    changes any of the other express terms or conditions of the contract or Work
             Assignment.

Technical direction will be issued in writing or confirmed in writing within five (5) calendar
days after verbal issuance. The technical direction memorandum will be provided to the
contractor and copies will be forwarded to the Contracting Officer and the Project Officer. If the
contractor has not received written confirmation within five (5) calendar days of an oral
issuance, the contractor must notify the Project Officer.

INHERENTLY GOVERNMENTAL FUNCTIONS

The contractor shall not perform any inherently governmental functions (IGF) under this Work
Assignment.  If during the course of developing the plan of work, through receipt of technical
direction, or in carrying out the assignment any portion of the effort is considered to possibly be
an inherently  governmental function, the contractor must immediately notify the Project Officer
and the Contracting Officer.

OCCUPATIONAL HEALTH AND SAFETY

Facility site visits  conducted under a Work Assignment that include on-site inspections or
sampling must be  conducted in full compliance with the Department of Labor, Occupational
Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440
(Occupational Health and Safety Manual).

-------
                                Peer Review Handbook                     Page F-16
TRAVEL COSTS

The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal
regulations in incurring allowable travel costs under this Work Assignment, and correspondingly
must at all times seek and obtain Government rates whenever available and observe current
subsistence ceilings.

QUICK RESPONSE TASKS

Each Quick Response Task (QRT) shall be confirmed in writing and approved by the Project
Officer. The contractor shall respond by letter to the Project Officer with copies to the WAM
and the CO within two working days, giving a brief description of the plan of work, including
best estimate of hours (by P-level) and a break-out of costs to accomplish the task.

No task shall exceed a duration of 30 calendar days from start date to completion date. The level
of effort for each task shall be limited to a maximum of 250 labor hours.

Quick Response Task Requests do not change the dollar or professional labor hour budgets of a
Work Assignment.

-------
                                Peer Review Handbook                      Page F-17
STATEMENT OF WORK - EXAMPLE 2 - External Peer Review of Protozoa Method
Development Criteria Document

Period of Performance:    Work Plan Approval to August 1, 1997

Work Assignment Manager:      Sally Q. Government Employee
                                                            Office of Water
                                                            U.S. Environmental
                                 Protection Agency

LOE:                      196 hours
SOW:                    2.4

BACKGROUND INFORMATION:

The United States Environmental Protection Agency (EPA), Office of Water is charged with
protecting public health and the environment from adverse exposure to chemicals and microbials
in water media, such as ambient and drinking waters, wastewater/sewage sludge and sediments.
In support of this mission OW's Office of Science and Technology (OST) develops  health
standards, health criteria, health advisories, and technical guidance documents for water and
water-related media. Under this work assignment, documents prepared by OST are  to undergo
peer review.

Peer review is an important component of the scientific process.  It provides a focused, objective
evaluation of a research proposal, publication,  risk assessment, health advisory, guidance or
other document submitted for review.  The criticism, suggestions and new ideas provided by the
peer reviewers stimulate creative thought, strengthens the reviewed document and confer
credibility on the product. Comprehensive, objective peer reviews leads to good science and
product acceptance within the scientific community.

Under this work assignment, the contractor will receive one document (Protozoa Method
Development Criteria Document) for peer review which is related to human health and
ecological effects.

STATEMENT OF WORK:

Task  1.     The contractor shall develop a work plan to address all tasks in this work
            assignment. The work plan shall  describe the steps that will be taken by the
            contractor to provide for peer review, including selection of peer reviewer
            candidates with appropriate expertise, determining absence of conflict of interest,
            document and reference distribution, establishing schedules, preparing  the peer
            review report, and submittal of the peer review package.  Curriculum vitae for all
            persons assigned to complete this work assignment shall be provided. All P levels,

-------
                                Peer Review Handbook                      Page F-18
            hours and total costs for each task will be provided and costs greater than $100.00
            shall be itemized in detail.

Task 2.     The contractor shall select a group of peer reviewers and determine their
            availability for the task and absence of conflict of interest, and establish a schedule
            for the peer review. The contractor is directed to ensure that none of the conflicts
            disclosed is so direct and substantial as to rule out a particular reviewer. Three peer
            reviewers shall participate in the review. No single peer reviewer may charge more
            than 40 hours to this task.  It is fully acceptable for peer reviewers to commit to less
            than 40 hours. The peer review will be conducted for the Protozoa Method
            Development Criteria Document.  Reviewers selected by and working for the
            contractor  shall be approved by the EPA Project Officer in writing prior to their
            beginning work. Minimally, all peer reviewers shall be accomplished in protozoan
            methods for sample recovery and analysis from water. Approval submissions shall
            include the reviewers' names and curriculum vitae.

Task 3.     The contractor shall arrange for the selected peer reviewers to review the EPA
            document.  Prepare the charge to the peer reviewers based on technical direction
            received from the EPA WAM. Provide the peer reviewers with copies of the
            candidate report and all relevant references and instruct the selected peer reviewers
            to undertake the review. The WAM will provide the contractor with the final
            version of the document to be reviewed.

Task 4.     The contractor shall monitor peer reviewers' progress to ensure timely completion.
            The contractor shall collate peer review comments, and organize the comments in
            the peer review "for comments" document.  The contractor shall provide the peer
            review document and all materials submitted by the peer reviewers to  the EPA
            WAM.

SCHEDULE AND DELIVERABLES:

Task 1. (Work Plan)  15 days after receipt of work assignment

Task 2.      1 week after work plan approval

Task 3.      1 week after selection of peer reviewers

Task 4.      1 week after receiving comments from the peer reviewers

TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable
to this work assignment must be submitted and approved by the Project Officer.

-------
                                Peer Review Handbook                     Page F-19
STATEMENT OF WORK - EXAMPLE 3 - Technical Support for Peer Review Services

INTRODUCTION

Background

The mission of the United States Environmental Protection Agency's Office of Water (OW) is to
protect public health and the environment from adverse effects of pollutants (e.g., chemicals and
microorganisms) in media such as ambient water, drinking water, wastewater, sewage sludge and
sediments.  In fulfillment of this mission, OW's Office of Science and Technology (OST)
develops effluent guidelines, human health and ecological criteria, health advisories, directs the
national program for adoption of Water Quality Standards, develops prevention and remediation
strategies to reduce the risk to human health and aquatic life resulting from contaminated fish
and sediment, provides technical support to the Total Maximum Daily Load program, analytical
methods for detection and quantification of pollutants in the environment and assesses,
remediates or otherwise manages and communicates the risks associated with exposure to
contaminants and microbial  pathogens.

In January 1993, responding to recommendations in the report "Safeguarding the Future:
Credible Science, Credible Decisions," Administrator William Reilly issued an Agency-wide
policy for peer review. Administrator Carol Browner confirmed and reissued the policy on June
7, 1994 and instituted an Agency-wide implementation program. The Agency  policy guides the
involvement of peers in the Agency's development of products. This involvement may be
through peer consultation or peer review of products.  The goal of the Peer Review Policy is to
implement a comprehensive Agency-wide program to ensure that major scientific and technical
work products receive critical scrutiny from scientific and technical experts as part of the overall
decision making process.

Purpose

The purpose of this contract is to provide peer consultation and peer review support services to
OST consistent with the Agency Peer Review Policy.  (Other offices within the Office of Water
can utilize this contract for efforts within this Statement of Work, if the OST Project Officer
determines capacity is sufficient, based upon OST projections and requirements). The purpose of
peer review is to uncover any technical problems or unresolved issues for use in revising a
preliminary product so that the final work product will reflect sound technical information and
analyses. These processes enhance the quality of and provide for objective review and
evaluation of scientific or technical work by knowledgeable experts who have no vested interest
in related policy or implementation issues.

Scientific and technical work products identified for peer review may address one or more of the
following:

       o    Support maj or regulatory decisions or policy/guidance of maj or impact

-------
                                 Peer Review Handbook                      Page F-20
       o    Establish a significant precedent, model or methodology
       o    Address a controversial issue
       o    Focus on a significant emerging issue
       o    Have significant cross-Agency/interagency implications
       o    Involve significant investment of Agency resources
       o    Consider an innovative approach for a previously defined
            problem/process/methodology
       o    Satisfy a  statutory or other legal mandate for peer review.
       o    Other products deemed appropriate by OST

The contractor is responsible for preparing reports on peer review evaluations and shall provide
support for all stages or portions of the peer review process. The term peer review shall mean
both peer consultation and peer review of products. Peer review will include initial research
protocol or work plan, preliminary review of an interim draft, review of a draft final work
product and final evaluation of the work product. All peer review procedures, products, and
responsibilities shall be in accordance with the latest revision of the "U.S. Environmental
Protection Agency, Peer Review Handbook''

Statement of Work

1.0    Administration and Management

1.1    Candidates for peer review include scientific, engineering and economic documents,
methodologies, strategies or positions that are used to support a research agenda, regulatory
program, policy position or other OST decision. OST has several general categories of scientific
and technical work products which are subject to peer review: Criteria Methodologies/Criteria
and Advisories; Engineering and Technical  Studies; Reports to Congress; Technical Guidance
and Methodologies; Economic Evaluations; Statistical Analyses; Environmental Models;
Technical Publications, Products, and Approaches; Analytical Methods; and  Policies and
Strategies.  The contractor shall provide the necessary professional and  technical personnel for
review of the following types of products:

       o    Toxicological Assessments
       o    Exposure Assessments
       o    Analytic Methods
       o    Statistical Analyses
       o    Dose-Response Assessments
       o    Microbial Risk Assessments
       o    Environmental Sampling Methods
       o    Monte Carlo Analyses
       o    Risk Assessment Guidelines
       o    Engineering and Technology Analyses
       o    Economic Analyses
       o    Technology Analyses

-------
                                 Peer Review Handbook                      Page F-21
1.2    The contractor shall appoint a person to serve as the Program Manager for this contract.
The Program Manager shall be the main point of contact for the contract; shall be responsible for
the performance of work under this contract, and shall be designated a key person in accordance
with the Key Personnel Clause in EPAAR 1552.237-72(APR 1984). The contractor shall
manage all aspects of this contract and shall be solely responsible for the direction and selection
of all contractor, subcontractor and consultant personnel who may be involved in the
performance of the contract.

1.3    The contractor shall furnish the facilities, materials, equipment and necessary
professional, technical and support personnel for performance of work required under this
contract. The contractor shall develop and institute internal management controls that will
ensure that, in the performance of this contract, the contractor will not provide personal services
nor perform inherently governmental functions. For these purposes, personal services are
defined in EPA Order 1900.1 dated October 31, 1985, and inherently governmental functions are
defined in EPA Order 1900.2 dated October 22, 1990, and OFPP Policy Letter 92-1, dated
December 1, 1992.

2.0 Peer Review Support

2.1 Selection of Peer Reviewers

The contractor shall provide Peer Review Services. Peer reviewers  will be selected for
independence and scientific/technical expertise. The scientific/technical peer reviewers shall be
recognized nationally or internationally in their field and have a general knowledge of
environmental science issues as well as the specific knowledge, expertise or experience as
required by the Work Assignment. An independent peer reviewer is an expert who was not
associated with the generation of the specific work product either directly by substantial
contribution to its development or indirectly by consultation during  the development of the
specific product (see Note below). Knowledge about the subject area is not equivalent to
expertise in the subject matter. For Agency decision, a multi-disciplinary group of experts
corresponding to the disciplines that contribute to complex Agency  decisions is often necessary
for a full and complete peer review.  For example, a risk assessment that relies on both animal
and human data often requires experts in both areas for a complete review. The Project Officer
(PO)AVork Assignment Manager (WAM) will provide the contractor with the document to be
reviewed.  The PO/WAM will, when available, provide the contractor with names of known
experts (listed in alphabetical order) in the required disciplines for the contractor to make final
selection.

With submission of the work plan or within the timeframe identified in the Work Assignment,
the contractor shall identify a group of candidate peer reviewers, either from the pool provided
by EPA, from contractor in-house staff experts, or from a list independently identified by the
contractor (or a combination) and determine their availability for the task.  The number of peer
reviewers required for each review will be identified in the work assignment.  The contractor
shall submit its selection of peer reviewers, along with their curriculum vitae to the EPA

-------
                                 Peer Review Handbook                       Page F-22
PO/WAM. The Agency will review the proposed candidates for consistency with the
requirements of the work assignment, based either on qualifications, conflicts of interest, or past
direct involvement with the work under review, however, EPA shall refrain from suggesting
particular individuals to replace such candidates. Once the reviewers are confirmed, the
PO/WAM will provide the contractor with the final version of the charge and copies of relevant
reference materials.

The contractor shall not be involved in the review of documents prepared by the contractor, or its
subcontractor or consultants. It is the responsibility of the contractor to ensure that all peer
reviews are conducted in a manner to avoid all actual, potential or apparent conflicts of interest.
The contractor shall obtain and provide written certification from all peer reviewers prior to the
commencement of work that no personal or organizational conflicts of interest exist.

It is the Agency's policy to have panels of experts that represent diverse constituencies, e.g.
industry, academia, public interest groups, Federal (non-EPA) Agencies, State and Local
governments, so long as the disciplinary expertise requirements of the panel are also met. The
contractor shall attempt in all cases to fulfill this criteria. Cultural diversity is also to be sought,
however, disciplinary expertise is the number one criterion for selection.

Note:  Peer reviewers of a product shall be independent experts which have not been
substantially involved in the development of the product.  In special cases where the number of
experts in a disciplinary  area is very small or not available at the specifically designated time; the
contractor may  select a peer reviewer who has been a peer consultant or prior reviewer of the
product.  If this selection is necessary, the contractor shall submit documentation of the prior
involvement in  the development process of the product by the potential reviewer, along with a
control plan to avoid bias or lack of impartiality, for EPA CO (Contracting Officer) and Project
Officer review and consent. If special circumstances exist where the expertise is so narrow that
another peer reviewer isn't available, the EPA CO/PO, along with the Peer Review Leader, will
determine the appropriate involvement and controls for the potential reviewer, so that no
perception of bias or conflict exists.

2.2    Information and Charge to Peer Reviewers

EPA will provide to the contractor written technical direction identifying the charge to be given
to the peer reviewers.  The peer reviewers shall  consider EPA-supplied and EPA-specified
documents along with EPA-provided instructions on the exact nature of the assignment (as
outlined in the EPA Work Assignment).  The charge to peer reviewers will usually make two
general requests.  First, it will focus the review by presenting specific questions and concerns
that the Agency expects the reviewers to address.  Secondly, it invites general comments on the
entire work product. The contractor shall provide the charge to the peer reviewers along with the
product to be peer reviewed and all  reference materials, and instruct the peer reviewers to
undertake the review. Where appropriate, raw data and/or data in summary form will be provided
and reviewers will be expected to rely on their assessment of such data in performing their
review. Each peer reviewer must be informed of the need for confidentiality with regard to the

-------
                                Peer Review Handbook                      Page F-23
release of Agency products that are stamped as "DRAFT" or "DRAFT - Do Not Cite, Quote or
Release."  The peer reviewer and the contractor will agree that the draft shall not be given to
others who are not under contractor direction without the written approval of the PO/WAM.
Any requests for such drafts should be forwarded to the EPA Contracting Officer for action.

2.3    Peer Review Report

The contractor shall monitor peer reviewers' progress to ensure timely completion.  All peer
review comments shall be submitted to the EPA PO/WAM for the particular work assignment.
Peer reviewers will be required to submit with their evaluations or comments a description of the
procedures used to arrive at their recommendations; a summary of their findings; a list of sources
relied upon;  and make clear and substantiate the methods and considerations upon which their
recommendations are based. When a document assigned under a task order requires more than
one peer reviewer, the contractor shall summarize all review comments and present this
summary in  an integrated manner that facilitates an overall evaluation (e.g., an annotated copy) ,
and shall collate them into a peer review report. The contractor shall be tasked with the
compilation, categorization, and summarization of the comments. The peer review report shall
be submitted to EPA in a specified format by a specified deadline. The format and deadline will
be provided  in the work assignment issued for the work. The contractor shall submit to the EPA
PO/WAM the peer review report and all materials submitted by the peer reviewers.  EPA shall
evaluate and analyze all peer review comments and recommendations to ensure technical
soundness and adherence to Agency policy.  Agency officials should document their evaluations
of the quality and validity of the peer review.

2.4    Peer Review Meetings

The contractor shall discuss peer review comments with the EPA PO, WAM,  and/or other peer
reviewers in telephone conferences, peer review workshops, and/or other  meetings.  Telephone
conferences, workshops and meetings shall be conducted in accordance with work assignment
specifications, which may require taping of and transcription of any proceedings. The scientific
expertise required shall include the broad array of the scientific disciplines relevant to the
specific document under development or review. In instances when a document or issue paper is
reviewed, the contractor may be required to meet and document widely varying topics which will
be defined specifically by the work assignment. The contractor shall organize and provide
support in arranging and conducting peer review meetings. The contractor shall obtain all post-
meeting comments, collect and compile all comments and suggested document revision, and
return all required information/documentation to the PO/WAM.  Occasionally, searches of
recent literature and hard copy or microfiche retrieval of germane papers may be required.

-------
                                 Peer Review Handbook                      Page F-24
2.5  Peer Review Meeting Support

Peer Review meeting logistical support includes bringing together appropriate experts to address
various issues and concerns raised by EPA for selected products. The contractor may be
required to (if specified in the work assignment):

    1)    prepare and distribute presentation materials, including pre-conference information ,
         agenda, pre-registration forms, fact sheets, overheads, slide presentations, etc.;
    2)    arrange for meeting space when government space is not available, reserve meeting
         rooms, copying and audiovisual equipment, furniture/room setups, microphones, and
         related services to accommodate meeting needs;
    3)    provide for, prepare, and convey all materials such as name badges, table tents, visual
         aids, displays, etc., needed for pre-conference and onsite activities;
    4)    inspect the conference site in accordance with expected facilities and services;
    5)    handle registration, distribution of meeting materials, and attendance list;
    6)    provide on-site recorder and/or facilitator;
    7)    arrange reporting , transcription and/or note taking services, typing, reproduction, and
         related support services as required by meeting needs, including rapid photocopying for
         important materials to be copied at meetings;
    8)    operate audiovisual equipment, microphones, other presentation related support
         equipment, and/or assist participants in operating such equipment;
    9)    handle close-down of the site, reviewing all bills for accuracy, and completing
         payments for equipment rental, etc.
    10)  A draft and final report may be prepared, in accordance with the requirements as
         specified in the work assignment, and in accordance with the specifications of
         Statement  of Work Parts 2.3 and 2.4 herein.
    11)  If required, distribute draft proceeding summaries to the participants for comment and
         make final revisions to the document.

The length of the meeting, expected attendance,  seating arrangements and audiovisual equipment
needs will be specifically identified in the work assignment issued by the CO.

The contractor shall arrange travel only in accordance with the authority and limitations of the
Section H Clause  entitled "Approval of Contractor Travel", i.e., use of contract funds to
reimburse travel is strictly limited to logistical support for peer reviewers who directly  contribute
to the requirements  of specific contract performance related to the statement of work of the
contract which supports the mission of OST. In  providing logistical support within the scope,
the contractor may claim transportation, lodging and subsistence for the personnel of the
contractor (including subcontractors and consultants with a contractual agreement with the
prime), who, by reason of their role within the scope of the Work Assignment are required to
attend the meeting.  The contractor may be expected to reserve a block of lodging rooms for all
reviewers and participants in a meeting. However, those participants who are Federal employees
must then make their own room reservations, travel arrangements, and travel funding
arrangements.

-------