UnifedStates ; i,::,;J*
Environmental Protectio'n
Agency  | :• ; -    i ]
iAdmirif'trator
-1(11:02) jl'.t'-^;
Guidance for OdrtaTOtWHa
Peer Review/ of
Regulatory IV!c5|lel$ g; : ;

               Agency Task Fc^rce on !
               Environmental Regulatory
               Modeling

-------

-------
               UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                          WASHINGTON, D.C. 20460
                            JUL  I 5 1994
                                                         OFFICE OF
                                                       THE ADMINISTRATOR
MEMORANDUM
SUBJECT:   Establishment of Agency-wide Guidance for Conducting
           External Peer Review of Environmental Regulatory
           Models

TO:        Assistant Administrators
           General  Counsel
           Inspector General
           Associate Administrators
           Regional Administrators
           Deputy Assistant Administrators
           Deputy Regional Administrators

     As Chairperson of the Agency's Science Policy Council,  I am
pleased to issue the attached "Guidance for Conducting External
Peer Review of  Environmental Regulatory Models" for Agency-wide
use.  The  Guidance complements the Agency policy for peer review
of scientifically  and technically based work products.

     The Guidance  should be considered and referenced as
appropriate in  the current development of standard operating
procedures for  peer review when those  procedures involve model
development and/or application.

     Agency managers who rely on modeling results may use the
Guidance to help determine which products need peer review.
Portions of this document may also prove helpful to those outside
EPA who rely on environmental modeling.

     The Agency Task Force on Environmental Regulatory Modeling
(ATFERM) developed the Guidance as part of its charge to address
the need for improvement in the development and use of modeling
for environmental  decision making.   The Science Advisory Board
(SAB) originally expressed that need in its resolution on
modeling  (SAB-EEC-89-012).

     Comments and  questions regarding  this Guidance may be
directed to its principal authors:  Michael Firestone (OPPTS),
Alan Cimorelli  (Region 3),  and John Yearsley (Region 10).
                          Robert M.  Sussma.n
                          Deputy Administrator
                                                             Printed on Recycled Pa/x

-------
Attachment

cc:  Science Policy Council
     Science Policy Council Steering Committee
     Agency Task Force/Environmental Regulatory Modeling
     Council of Science Advisors
     Science Advisory Board Executive Committee

-------
NOTE
     The following Guidance has been updated since April 24, 1994
when it was transmitted to the Deputy Administrator, to reflect
the latest  (June 7, 1994) statement regarding EPA's Peer Review
Policy.

-------

-------
     AGENCY  GUIDANCE  FOR



   CONDUCTING EXTERNAL  PEER
   REVIEW  OF  ENVIRONMENTAL
      REGULATORY  MODELING
Developed by:



   Agency Task Force on Environmental



   Regulatory Modeling (ATFERM)

-------

-------
Acknowledgements

      This guidance document was developed by the Peer Review Subcommittee of the
Agency Task Force on Environmental Regulatory Modeling (ATFERM). The principal
authors included:

Michael P. Firestone, Science Advisor, Office of Prevention, Pesticides and Toxic
Substances

Alan Cimorelli, Lead Meteorologist, Air and Radiation Division, Region III

John Yearsley, Environmental Scientist, Environmental Services Division, Region X

      We would like to acknowledge the contributions of other members of the Peer
Review Subcommittee including: Robert Ambrose, Environmental Research Laboratory
at Athens, Office of Research and Development; Russell Kinerson, Office of Science
and Technology, Office of Water;  Joseph Tikvart, Office of Air Quality Planning and
Standards, Office of Air and  Radiation; and Lawrence Zaragoza, Hazardous Site
Evaluation Division, Office of Solid Waste and Emergency  Response.

      The authors are especially grateful for the support provided by ATFERM's
Executive Secretary Mary Lou Melley, Office of Solid Waste and Emergency Response,
and ATFERM's co-chairs,  Larry  Reed, Office of Solid Waste and Emergency Response,
and Stephen Cordle, Office of Research and Development.

      We would like to express our appreciation to William Raub, Science Advisor to
the Administrator, for his sound guidance regarding peer review and his assistance in
refining the final document.  We also thank Donald Barnes, Science Advisory Board
Director, and Lynn Goldman, Assistant Administrator for Prevention, Pesticides and Toxic
Substances, for their guidance with respect to peer review  policy and practices at the
Environmental Protection Agency.

       Finally, the authors acknowledge the contributions provided by the external peer
review comments developed by the Environmental Engineering Committee of the
Science Advisory Board, including Richard Con way, Chair, James Mercer, Modeling Peer
Review Subcommittee Chair, and Kachig Kooyoomjian, EPA's Executive Secretary, and
the internal peer review support provided by the Agency's  Council of Science Advisors.

-------

-------
AGENCY  GUIDANCE  FOR  CONDUCTING  EXTERNAL  PEER REVIEW OF
ENVIRONMENTAL REGULATORY  MODELING

I.      INTRODUCTION

       According to EPA's Peer Review Policy Statement dated June 7, 1994, (see
attachment):

       Major scientifically and technically based work products related to Agency
       decisions normally should be peer-reviewed. Agency managers within
       Headquarters, Regions,  laboratories and field components determine and are
       accountable for the decision whether to employ peer review in particular
       instances and, if so, its character, scope, and timing.  These decisions are made in
       conformance with program goals and priorities, resource constraints, and statutory
       or court-ordered deadlines. For those work products that are intended to support
       the most important decisions or that have special importance in their own right,
       external  peer review is the procedure of choice. Peer review is not restricted to
       the penultimate version of work  products;  in fact,  peer review at the planning
       stage can often be extremely beneficial.

By contrast, the policy specifically excludes "non-major or non-technical matters that
Agency managers consider as they make decisions."

       Clearly, environmental models (i.e., fate  and transport,  estimation of contaminant
concentrations in soil, groundwater, surface water and ambient air, exposure assessment)
that may form part of the scientific basis for regulatory decision making  at EPA are
subject to  the peer review policy. However, it can not be more strongly stressed that
peer review should only be considered  for judging the scientific credibility of the model
including applicability, uncertainty, and utility (including the potential for mis-use) of
results, and not for directly advising the Agency on specific  regulatory decisions
stemming  in part from consideration of  the model output.

       The purpose of this guidance is to provide a resource for those program managers
responsible for implementing the peer review process.  More specifically, this guidance
is provided as an aid in evaluating the need and, where appropriate, conducting external
peer review related to the development and/or application of environmental regulatory
modeling.  This specific guidance for modeling has been prepared to complement
general peer review guidance currently  being developed by the Agency's Council of
Science Advisors (Council). Thus, Section II (framework for peer review) and Section IV
(peer review mechanisms and general criteria) reflect the latest Council guidance, and
will be revised in the future as Agency-wide guidance on  peer review evolves.

       The relationship of external peer review to the process  of model  development and
application, including consideration of peer review at various  stages in the process is
described  in Section III of this guidance.

-------
      Section V concerning "Documentation of the Peer Review Process" has been
included in response to comments from the Agency's Science Advisory Board who cited
the need for more detailed guidance on the mechanics of the review process.

      The specific elements of what could be covered in an external peer review of
model development and application are presented in Section VI of this guidance. These
elements are not meant to be prescriptive or limit the nature of peer review, but rather
are intended as an aid to improve the thoroughness and consistency of peer review.

      To reiterate one of the major recommendations in the March 1992 report
commissioned by former Administrator William K. Reilly re: "Safeguarding the Future:
Credible Science, Credible Decisions (The Report of the Expert Panel on the Role of
Science at EPA)":

      "Quality assurance and peer review should be applied to the planning and results
      of all scientific and technical efforts to obtain data used for guidance and
      decisions at EPA, including such efforts in the program and regional offices.  Such
      a requirement  is essential if EPA is to be perceived as a credible, unbiased source
      of environmental and health information, both in the United States and
      throughout the world."

      In conclusion,  this document is intended to provide guidance to program
managers in their efforts to consider, and apply where appropriate, external peer review
to environmental regulatory modeling.  The guidance contained in this document must
not be construed as rigid requirements.1


II.    FRAMEWORK FOR PEER REVIEW OF ENVIRONMENTAL REGULATORY
MODELING

      Peer review can be an  important tool in assisting the Agency to document the
quality  and  credibility of the science upon which its regulatory and policy decisions are
made.  Modeling to provide the scientific support for environmental regulatory decision
making at EPA can be thought to conceptually involve three stages  including: (A) model
development;
(B) model application;  and (C) consideration of modeling results in decision making.
 The guidance set out in this document is not final Agency action. It is not intended, nor can it be relied upon,
 to create any rights enforceable by any party hi litigation with the United States. EPA officials may decide to
 follow the guidance provided herein, or act at variance with the guidance, based on an analysis of specific
 circumstances.  The Agency also reserves the right to change this guidance at any time without public notice.

-------
       External peer review, as discussed in more detail below, is generally relevant to
the first stage of model development and may be relevant in appropriate cases to the
second stage of model application.  Although external peer review is not directly
germane to the regulatory or policy decision itself, it is important at this third stage to
bring forward information regarding prior peer review comments  and the Agency's
response related to model development and/or application.  This  information may aid the
decision maker in interpreting and weighing the  utility of modeling results along with all
the other considerations (e.g., field data;  risk, cost, and benefits information;
requirements to use best available technology; environmental justice issues; etc.) in
reaching a regulatory decision.

       Broader guidance regarding the applicability of peer  review, as discussed in the
Introduction, is currently being developed by the Council of Science Advisors.  As this
broader guidance develops and evolves in the future, the
framework for external peer review of environmental regulatory modeling, as discussed
below, will be revised.

Ai     Model Development

       Models are developed for a variety of reasons, including:

       1.     use as research tools to explore new scientific issues;

       2.     simplification and/or refinement of  existing model paradigms or software;

       3.,    use as screening tools;  and

      4.     to estimate compliance with regulatory requirements (e.g., National
             Ambient Air Quality Standards).

      This guidance document, by its terms, does not directly address models
developed for reasons other than to support regulator/ decision making (e.g., research
tools). Models developed expressly for and used exclusively within a research program
should be subject to essentially the same review  process as  other research results (e.g.,
informal  critique by scientific colleagues, formal appraisal by senior scientists and
managers, publication in refereed journal, etc.).   However,  if the purpose of a research
model is expanded in the future to develop scientific information for Agency decision
making, then the program manager should consider arranging for reassessment of the
model in accordance with the guidance presented below.

       From a general scientific perspective, a well-conceived model is one that provides
an acceptable mathematical approximation of a physical, chemical, biological, social or
economic system.  Model development frequently touches on the interface involving the

-------
state-of-the-art in several areas including:  (1) the technical and scientific understanding
of processes and mechanisms;  (2) applicable solution techniques (e.g., analytical,
numerical);  (3) computer science and technology; and (4) the Agency's need for
scientific decision-support tools.  The goal of model development is to provide methods
which can be applied to improve Agency analysis and decision making. As such, the
development of environmental models cannot be performed in a scientific vacuum.
There must be a coherent class of applications in mind against which the appropriateness
of the science can be judged.

      At the model development stage, a key step is to define and compare to existing
models, the set of conditions under which the use of a model is scientifically defensible -
this is known as the "application niche."  Peer review of model development would be
expected to include evaluation of the application niche, along with consideration of
uncertainty and other areas of model performance. Approaching peer review from this
perspective should help the decision maker understand the limitations of the scientific
basis of the  model and confidence in its results.  It is only with this  firm knowledge that
the Agency can develop sound regulatory and policy decisions.

      External peer review of major technical issues related to  environmental regulatory
modeling is generally a lengthy process.  This is precisely why external peer review
should be identified as an important and  integral aspect of an action plan for model
development. At first glance external  peer review might be expected to slow down the
process.  However, initiating peer review at early stages may, in fact, save time by
redirecting misguided initiatives, identifying alternative approaches,  or providing  strong
technical support for a potentially controversial position.

B..    Model Application

      The middle stage,  model application, begins with examining the stated application
niche and its applicability to current needs and/or exploring whether a model can be
tailored to fit a new niche.  For existing models, especially models developed outside of
EPA, peer review may be appropriate  to the extent that either:  1) new information
becomes available which calls into question the appropriateness of the previously
defined application niche;  or 2) a model might be considered for application outside the
niche for which it was originally developed.

       Peer review of a model's applicability should, where possible, be planned well in
advance of any decision making involving use of the model's results.  In this way, the
formation of "sound science" is distinct from the regulatory decision in which it is
considered. The results of such a peer review can aid in the ultimate judgment by a
decision maker to consider whether or how to use a particular  model's result in the
regulatory process.

-------
       Normally, the first application of a model should undergo peer review.  For
subsequent applications, a program manager should consider the scientific/technical
complexity and/or novelty of the particular circumstances as compared to prior
applications (also see Section IV-B).  Peer review of all similar applications should be
avoided because this would likely waste precious time and monetary resources while
failing to provide the decision maker with any new relevant scientific information upon
which to base a regulatory or policy selection.  Nevertheless, a program manager may
consider conducting peer review of applications upon which costly decisions are based
or applications which are likely to end up in litigation.

       As an alternative to peer review of model application, the Office of Air Quality
Planning and Standards has developed a formal review and acceptance  procedure
through rulemaking to evaluate the utility of alternative models for a particular
application.

C     Environmental Regulatory Decision Making

       The final stage involves  consideration of modeling results in the decision making
process. This  stage may include consideration of:  (1) selecting among available models
and assumptions/standard defaults;  and/or (2) whether/how to consider modeling results
in the regulatory process.  Information previously derived from peer review of scientific
issues may provide key information for the decision maker in understanding the
uncertainties and  utility (and potential for mis-use) of modeling results.  When possible,
public discussion  of the scientific and technical underpinnings associated with the earlier
stages of model development and/or application in advance of the final  stage of
regulatory decision making is expected to reduce  criticism regarding the Agency's use of
"sound science."

       Environmental regulatory and policy decisions also involve other scientific and
non-scientific factors, and are by law, required to  be made by the responsible Agency
decision makers.  Thus, such decisions are not appropriate subjects for scientific peer
review. Rather, the process of public comment is frequently employed  by the Agency at
this stage.
III.   RELATIONSHIP OF EXTERNAL PEER REVIEW TO THE PROCESS OF
      ENVIRONMENTAL REGULATORY MODEL DEVELOPMENT AND APPLICATION

      The Agency's peer review policy notes that "properly applied, peer review not
only enriches the quality of work products but also atdds a degree of credibility that
cannot be achieved in other way. Further, peer review early in the development of work
products in some cases may conserve future resources by steering development along the
most efficacious course." Since this subject guidance focuses on the role of external peer

-------
review, its relationship to other levels of peer review can best be understood by
considering where external peer review may fit into the total process of environmental
regulatory modeling.  The following paradigm is meant to illustrate the general process:
Step 0:
Step Ob:
(optional)
Step 1:
Step 1b:
(optional)
Step 2:
Step 3:
Step 4:
The program manager within the originating office (AA-ship or
Region) identifies elements of the regulatory process which would
benefit from the use of environmental models.  A review/solicitation
of currently available models and related research should be
conducted.  If it is concluded that the development of a new model
is necessary, a research/development work plan would be prepared.

The program manager may consider internal and/or
external peer review of the research/development concepts to
determine whether they are of sufficient merit and whether the
model is likely to achieve the stated purpose.

The originating office develops a new or revised model or evaluates
the possible novel application of model developed for a different
purpose.

The program manager may consider external peer
review of the technical or theoretical basis prior to final
development, revision or application at this stage.  For model
development, this review should evaluate the stated  application
niche.

Initial Agency-wide (internal) peer review/consultation of model
development and/or proposed application may be undertaken by the
originating office.

Model design, default parameters, etc. and/or intended application
are revised (if necessary) based on consideration of internal peer
review comments.

External peer review is considered by the originating office.

Model design, default parameters, etc. and/or intended application
are revised (if necessary) based on consideration of external peer
review comments.

Final  Agency-wide evaluation/consultation may be implemented by
the originating office.  This step should consist of consideration of
external peer review comments and documentation of the Agency's
response to scientific/technical issues.

-------
(Note:  Steps 2 and 4 are relevant where there is either an internal Agency standing or
ad hoc peer review committee or process).

      A program manager may decide that peer review (step 2 -internal, and step 3 -
external) should take place at more than one time during the processes of model
development and model application (i.e., optional steps Ob and/or 1b). This decision
would depend on the nature and complexity of scientific issues that are presented.

      Additionally, a program manager may also consider the utility of employing the
broader concept of peer involvement - that is, to augment staff efforts by soliciting
subject-matter experts from outside the program.

      Based in part on the results of a scientific peer review of model development and
application, as well as other non-technical issues, the program manager would be
responsible for advising the decision maker as to the consistent, equitable, and
appropriate use of the model and its output for environmental regulatory purposes.
IV.   MECHANISMS AND GENERAL CRITERIA FOR CONDUCTING EXTERNAL PEER
      REVIEW

      The mechanisms and general criteria for conducting external peer review
presented below are based upon generic peer review guidance currently being
developed by the Council of Science Advisors. As this broader guidance develops and
evolves in the future, this Section will be revised.

/V    Peer Review Mechanisms

      Mechanisms for accomplishing external peer review include, but are not limited
to, the following:

      i.     Using an ad hoc technical panel of at least three scientists;

      ii.     Using an established external peer review mechanism such as the Science
             Advisory Board or Scientific Advisory Panel;  or

      iii.    Holding a technical workshop.

      Qualifications for peer reviewers will likely van/ for model development versus
model application.  For the former, the emphasis may be toward modelers, while for the
latter, the emphasis may be toward scientists with technical expertise in other disciplines
(e.g., statistics, field monitoring, etc.).

-------
                                        8
       New models, or significant modifications and/or new applications of established
models, should be considered for publication in refereed journals.  However, this step
should be supplemental to and not a substitute for the peer review mechanisms
presented above.

E5.     General Criteria

       General criteria to be considered for determining when and by what mechanism
to initiate an external  peer review of the development and/or application of
environmental regulatory models include:
      i.      Use of model results as a basis for major regulatory or policy/guidance
             decision making;

      ii.     Significant investment of Agency resources;

      iii.    Inter-Agency or cross-Agency implications/applicability;

      iv.    Treatment of a new scientific issue; or

      v.     Novelty and/or complexity of the science.
V.
DOCUMENTATION OF THE PEER REVIEW PROCESS
       It is important to prepare documentation for each peer review conducted and
make this available to the decision maker along with the modeling results. The record
should include the following information:

       A.     Identity of the peer reviewers, their relevant expertise, and their
             institutional affiliation;

       B.     Questions/issues posed to the peer reviewers;

       C.     Reviewer comments, either as a collection of individual statements or as a
             consensus statement; and

       D.     Agency response to  peer review comments including rationale.

-------
VI.   SPECIFIC ELEMENTS OF EXTERNAL PEER REVIEW FOR ENVIRONMENTAL
      REGULATORY MODELING

      This Section of the guidance addresses the specific elements that should be
considered to provide the Agency with consistent scientific/technical external peer
review of environmental regulatory model development and/or application.

      Such elements could include, but are not limited to:

A..    Model Purpose/Objectives

      The first step in evaluating a model used for environmental regulatory purposes is
to clearly understand the broad context in which a model  is intended to be used.

      i.      What is the regulatory context in which the model will be used and what
             broad scientific questions is the model  intended to answer (e.g., evaluating
             the range of human respiratory exposure resulting from air toxics release
             from a power plant)?

      ii.     What is the model's application niche (i.e., the particular physical,
             chemical, and/or biological system and set of defining conditions for which
             the model is expected to be scientifically defensible)?

      iii.    What are the model's strengths and weaknesses?  How well does the
             model fit its intended application niche compared to existing models and
             available data sets?  It is important that the Agency avoid duplicating prior
             efforts.

B..    Major  Defining and Limiting Considerations

      With a clear understanding of the broad purpose and objectives of a model in a
regulatory context, the scientific context needs to be addressed.

      i.      Which processes are characterized by the model (e.g., transport, diffusion,
             chemical reactions, removal mechanisms,  etc.)?

      ii.     What are the important temporal and spatial scales?  Is the grid resolution
             appropriate for the problem?

      iii.    What is the level of aggregation?

-------
                                        10
C    Theoretical Basis for the Model

      Once the regulatory and scientific contexts of a model have been defined, the
basis for problem solving must be formulated.

      i.     What algorithms are used within the model and how were they derived?
             What is the mechanistic basis?

      ii.     What is the method of solution (numerical, analytic)?

      iii.    What formulations are used for those processes which are parameterized?

      iv.    How does the basis for problem solving compare to existing models?
             What is the scientific rationale?

      v.     What are the shortcomings of the modeling approach (e.g.,  missing or over-
             simplification of key processes, restrictive dimensionality, etc.)?
D.
Parameter Estimation
      Parameter estimation may be based on case specific data or in their absence, on
default values.  In the latter case, it is important to understand how parameter defaults
were established.

      i.      What methods were used for parameter estimation?

      ii.     What data were available for parameter estimation?

      iii.    What methods were used to estimate parameters for which there were no
             data?

      iv.    What is the reliability of parameter estimates?

      v.     What are the boundary conditions and are they appropriate?
E..     Data Quality/Quantity

       All models require the input of various types and amount of data.  Models may
also rely on experimental data to help shape their computational algorithms.  In large
measure, the utility of a model for regulatory purposes depends on the quality, quantity,
and spatial and temporal adequacy of data used in its design and in support of its
application.

-------
F.
                                         11

       Questions related to model design include:

       i.     What data were utilized in the design of the model?

       ii.     How can the adequacy of the data be defined in terms of quality, quantity,
             and spatial and temporal applicability taking into account the regulatory
             objectives of the model?

       Questions related to model application include:

       i.     What kinds of data are required to apply the model?

       ii.     To what extent are these data available and what are the key data gaps?

       iii.    Have data quality objectives been defined?  If so, are they scientifically
             defensible?

       iv.    Is the quantity of data sufficient to address the likely variability?  What
             statistical analyses were performed and are they appropriate?

       v.     To what extent are the data suitable with regard to estimating spatial and
             temporal effects?
      VI.
       Do additional data need to be collected and for what purpose?

Key Assumptions
      The applicability of a model depends on the adequacy of its basic underlying
assumptions.

      i.      What are the key assumptions?

      ii.     What is the basis for each key assumption and what is the range of
             possible alternatives?
G.
iii.    How sensitive is the model toward modifying key assumptions?

Modet Performance Measures
      The most basic test of a model's adequacy is to understand how well its results
compare with real world measurements.

      i.     What criteria have been used to assess model performance?

-------
                                        12

      ii.     Did the data bases used in the performance evaluation provide an
             adequate test of the model in terms of applicability to the modeling niche?

      iii.    How accurate can the model be expected to perform? Does the model
             exhibit any overall bias throughout the range of its predictions?  Bias is an
             important test of the model's formulation since intrinsic system uncertainty
             is not present.

      iv.    How well does the model address, distinguish, and report variability and
             uncertainty in its output?  Which parameters and key assumptions are  most
             significant in  determining the model's variability and uncertainty?

      v.     How does the model perform relative to other models in this application
             niche?

JhL    Model Documentation and Users Guide

      The utility of model for regulatory purposes depends  on the availability of a clear
documentation report and a comprehensive users guide. Do these cover:

      i      Model applicability and limitations?

      ii.     Data input?

             Interpretation of  results? and
in.
       IV.
      Documentation of the model code and other key aspects such as
      verification testing?
JL      Retrospective

       A retrospective analysis of the "big picture" may sometimes reveal insights that an
analysis of individual components of a model may miss.

       i.     Does the model satisfy its intended scientific and regulatory objectives?
       11.
       in.
       Is there any available scientific evidence to suggest changes to either the
       model design and/or key parameters and assumptions prior to its use for
       regulatory purposes?

       How robust (i.e., not overly sensitive toward small changes in modifying
       key assumptions or input data) are the model predictions?

-------
                                        13
       iv.    How well does the model output quantify the overall uncertainty resulting
             from limitations/simplifications in its design; use of standard assumptions;
             availability of supporting data;  etc.?

       v.     What key research is necessary to refine or improve the model and/or the
             data bases upon which it relies?
      The elements provided above are not meant to be prescriptive or limit the nature
of external peer review. Rather the purpose for their inclusion is to provide modeling-
related guidance for peer review protocols which will be consistent with more general
guidance being developed by the Council of Science Advisors.
Attachment 1:
"Peer Review and Peer Involvement at the U.S.
Environmental Protection Agency" dated June 7, 1994

-------

-------
                 PEER REVIEW AND PEER INVOLVEMENT
          AT  THE U.  S.  ENVIRONMENTAL PROTECTION AGENCY
     This document  establishes the  policy of the  United States
Environmental  Protection   Agency   (EPA)  for   peer  review  of
scientifically  and  technically  based  work products  that  are
intended to support Agency decisions.  Peer review is presented in
the context of the broader concept, peer involvement.
BACKGROUND

     The  report  "Safeguarding  the  Future:  Credible  Science,
Credible Decisions"1 focused on the state of science at EPA.  The
panel of experts who prepared the report  emphasized the importance
of peer review, especially external peer review, and the need for
broader and more systematic use of it at EPA to evaluate scientific
and  technical  work  products.    Their  specific  recommendation
regarding peer review reads as follows:

     "Quality assurance and peer review  should be applied to
     the planning and results of all scientific and technical
     efforts to obtain data used for guidance and decisions at
     EPA, including such efforts in the  program and regional
     offices.  Such a requirement is essential if EPA is to be
     perceived as a credible, unbiased source of environmental
     and health  information,  both in  the  United States and
     throughout the world."

In  response  to  this  recommendation,   then-Administrator  Reilly
directed staff to  develop  an EPA-wide policy statement,  which he
issued in January,  1993.  The paragraphs below preserve the core of
that earlier statement while updating it  to specify the role of the
Science  Policy  Council  in guiding  further implementation  of the
policy.    Effective  use  of  peer  review  is  indispensable  for
fulfilling  the  EPA mission  and therefore  deserves high-priority
attention from program managers and scientists within all pertinent
Headquarters  and Regional Offices.
       EPA/600/9-91/050, March 1992.

-------
PEER INVOLVEMENT AND PEER REVIEW

     EPA  strives to  ensure  that the  scientific  and technical
underpinnings of its decisions  meet  two important criteria: they
should  be based upon  the best  current knowledge  from science,
engineering, and other domains of technical  expertise; and they
should be judged credible by those who deal with the Agency.  EPA
staff therefore frequently rely upon peer involvement  — that is,
they augment their capabilities by inviting relevant subject-matter
experts from outside the program to become involved  in  one or more
aspects  of the  development  of  the work products  that  support
policies and actions.

     One  particularly  important type of  peer involvement occurs
when scientifically and technically based  work products undergo
peer review —   that  is,  when  they  are  evaluated  by relevant
experts  from outside  the  program who  are  peers of  the program
staff,   consultants, and/or  contractor  personnel who prepared the
product.   Properly applied,  peer review not  only  enriches the
quality of work products but also adds a degree of credibility that
cannot be achieved in any other way.  Further,  peer review early in
the development of work products in some cases may conserve future
resources by steering  the  development  along the most  efficacious
course.

     Peer review generally takes one of two forms.  The  review team
may consist primarily  of relevant experts from within  EPA, albeit
individuals who have no other involvement  with respect  to the work
product that is to be evaluated  (internal  peer review) .   Or the
review  team may  consist primarily  of independent  experts from
outside EPA  (external  peer review).
POLICY STATEMENT

     Major  scientifically  and technically  based  work products
related  to Agency decisions  normally should  be peer-reviewed.
Agency managers within Headquarters,  Regions,  laboratories,  and
field components  determine and are  accountable for the decision
whether to employ  peer review  in particu ar instances and, if so,
its  character,  scope, and  timing.   These decisions  are made in
conformance   with   program   goals   and  priorities,   resource
constraints, and statutory  or  court-ordered deadlines.  For those
work products  that  are  intended to support the  most important
decisions  or that have  special  importance  in their  own right,
external peer  review is  the procedure of choice.   Peer review is
not  restricted to the penultimate  version of  work products; in
fact, peer review at the  planning  stage can  often be extremely
beneficial.

-------
SCOPE

     Agency managers routinely make regulatory and other decisions
that  necessarily  involve  many  different considerations.   This
policy applies to major work products that are primarily scientific
and technical in nature and may contribute to the basis for policy
or regulatory decisions.  By contrast, this  policy does not apply
to nonmajor or nontechnical matters that Agency  managers consider
as they make decisions.  Similarly, this policy  does not apply to
these ultimate decisions.

     This policy  applies where appropriate,  as determined by the
National and Regional Program Managers, to major scientifically and
technically based work  products initiated subsequent to the date of
issuance.  Peer review should be employed  to  the  extent reasonable
to relevant  work products that  currently  are under development.
This policy does not apply to the bases for past  decisions, unless
and  until  the  relevant  scientific  and  technical   issues  are
considered anew in the Agency's decision-making  processes.

     Except where  it  is required by  law,  formal peer review (as
distinguished from the  Agency's normal internal review procedures)
should be conducted in a manner that will  not cause EPA to miss or
need extension of  a statutory or court-ordered deadline.   Agency
managers still may undertake peer review if it  can  be conducted
concurrently with necessary rulemaking steps.

LEGAL EFFECT

     This policy statement  does not establish  or affect  legal
rights or obligations.   Rather, it confirms the importance of peer
review  where  appropriate,  outlines  relevant  principles,   and
identifies factors Agency staff should consider in implementing the
policy.  On a continuing basis,  Agency management is  expected to
evaluate the  policy as  well  as  the  results of its  application
throughout  the  Agency  and   undertake  revisions  as  necessary.
Therefore,  the policy does not stand alone; nor does it establish
a  binding  norm  that  is  finally  determinative of  the  issues
addressed.   Minor variations in its application  from one instance
to another  are  appropriate  and  expected;  they  thus  are not  a
legitimate basis for delaying or complicating action on otherwise
satisfactory scientific, technical, and regulatory products.

     Except where provided otherwise by law,  peer review is not a
formal part of or substitute for notice and comment rulemaking or
adjudicative procedures.  EPA's  decision  whether to  conduct  peer
review  in  any  particular  case  is  wholly  within  the  Agency's
discretion.    Similarly,  nothing  in this  policy creates a  legal
requirement that EPA respond to  peer  reviewers.   However,  to  the
extent that  EPA  decisions  rely on scientific  and technical  work
products that have been subjected  to  peer  review,  the remarks  of
peer reviewers should be included in the record for that decision.

-------
IMPLEMENTATION
     The  Science Policy  Council  is  responsible  for overseeing
Agency-wide implementation. Its responsibilities include promoting
consistent  interpretation, assessing  Agency-wide  progress,  and
develocina  recommendations   for  revisions  of  the  policy  as
developing
necessary
     The Science  Policy Council will  oversee  a peer-review work
group,  which  will   include  representatives  from program  units
throughout EPA to effect a consistent, workable implementation of
the  policy.    The work group  will  assist  the programs  in  (1)
formulating  and,  as   necessary,   revising  standard  operating
procedures (SOPs)  for peer review consistent with  this policy; (2)
identifying work products that  are subject to review; and (3)  for
each major work product, selecting an appropriate  level and timing
of peer review.

     In assisting the  programs,  the  work  group will  take into
account statutory and court deadlines, resource implications,  and
availability of disinterested peer reviewers. The group will work
closely with Headquarters offices and the Regional Offices toward
ensuring effective,  efficient uses of peer  review  in supporting
their mission  objectives.   However,  the Assistant Administrators
and  Regional  Administrators  remain  ultimately  responsible  for
developing SOPs, identifying work products subject to peer review,
determining  the type and timing of  such  review,  documenting  the
process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.

     Because  peer review  can  be  time-consuming and  expensive,
Agency managers  within Headquarters,  Regions,  laboratories,  and
field components are  expected  to plan carefully with respect to its
use   —   taking   account   of   program   priorities,   resource
considerations, and  any other relevant constraints as well as  the
policy goal of achieving high-quality, credible underpinnings  for
decisions.  External peer reviewers should be chosen carefully to
ensure an independent and objective evaluation.  The affiliations
of peer reviewers should be identified on the public record, so as
to avoid undercutting the credibility of the  peer-review process by
conflicts of interest.

     This policy  is  effective immediately.   The peer-review work
group  mentioned  above  will  identify the   focal point  to whom
comments and questions should be addressed and,  from time to time,
will provide further information about implementation activities.
     APPROVED:
                                                  DATE
               CAROL M. BROWNER, ADMINISTRATOR

-------

-------

-------