UnifedStates ; i,::,;J*
Environmental Protectio'n
Agency | :• ; - i ]
iAdmirif'trator
-1(11:02) jl'.t'-^;
Guidance for OdrtaTOtWHa
Peer Review/ of
Regulatory IV!c5|lel$ g; : ;
Agency Task Fc^rce on !
Environmental Regulatory
Modeling
-------
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
JUL I 5 1994
OFFICE OF
THE ADMINISTRATOR
MEMORANDUM
SUBJECT: Establishment of Agency-wide Guidance for Conducting
External Peer Review of Environmental Regulatory
Models
TO: Assistant Administrators
General Counsel
Inspector General
Associate Administrators
Regional Administrators
Deputy Assistant Administrators
Deputy Regional Administrators
As Chairperson of the Agency's Science Policy Council, I am
pleased to issue the attached "Guidance for Conducting External
Peer Review of Environmental Regulatory Models" for Agency-wide
use. The Guidance complements the Agency policy for peer review
of scientifically and technically based work products.
The Guidance should be considered and referenced as
appropriate in the current development of standard operating
procedures for peer review when those procedures involve model
development and/or application.
Agency managers who rely on modeling results may use the
Guidance to help determine which products need peer review.
Portions of this document may also prove helpful to those outside
EPA who rely on environmental modeling.
The Agency Task Force on Environmental Regulatory Modeling
(ATFERM) developed the Guidance as part of its charge to address
the need for improvement in the development and use of modeling
for environmental decision making. The Science Advisory Board
(SAB) originally expressed that need in its resolution on
modeling (SAB-EEC-89-012).
Comments and questions regarding this Guidance may be
directed to its principal authors: Michael Firestone (OPPTS),
Alan Cimorelli (Region 3), and John Yearsley (Region 10).
Robert M. Sussma.n
Deputy Administrator
Printed on Recycled Pa/x
-------
Attachment
cc: Science Policy Council
Science Policy Council Steering Committee
Agency Task Force/Environmental Regulatory Modeling
Council of Science Advisors
Science Advisory Board Executive Committee
-------
NOTE
The following Guidance has been updated since April 24, 1994
when it was transmitted to the Deputy Administrator, to reflect
the latest (June 7, 1994) statement regarding EPA's Peer Review
Policy.
-------
-------
AGENCY GUIDANCE FOR
CONDUCTING EXTERNAL PEER
REVIEW OF ENVIRONMENTAL
REGULATORY MODELING
Developed by:
Agency Task Force on Environmental
Regulatory Modeling (ATFERM)
-------
-------
Acknowledgements
This guidance document was developed by the Peer Review Subcommittee of the
Agency Task Force on Environmental Regulatory Modeling (ATFERM). The principal
authors included:
Michael P. Firestone, Science Advisor, Office of Prevention, Pesticides and Toxic
Substances
Alan Cimorelli, Lead Meteorologist, Air and Radiation Division, Region III
John Yearsley, Environmental Scientist, Environmental Services Division, Region X
We would like to acknowledge the contributions of other members of the Peer
Review Subcommittee including: Robert Ambrose, Environmental Research Laboratory
at Athens, Office of Research and Development; Russell Kinerson, Office of Science
and Technology, Office of Water; Joseph Tikvart, Office of Air Quality Planning and
Standards, Office of Air and Radiation; and Lawrence Zaragoza, Hazardous Site
Evaluation Division, Office of Solid Waste and Emergency Response.
The authors are especially grateful for the support provided by ATFERM's
Executive Secretary Mary Lou Melley, Office of Solid Waste and Emergency Response,
and ATFERM's co-chairs, Larry Reed, Office of Solid Waste and Emergency Response,
and Stephen Cordle, Office of Research and Development.
We would like to express our appreciation to William Raub, Science Advisor to
the Administrator, for his sound guidance regarding peer review and his assistance in
refining the final document. We also thank Donald Barnes, Science Advisory Board
Director, and Lynn Goldman, Assistant Administrator for Prevention, Pesticides and Toxic
Substances, for their guidance with respect to peer review policy and practices at the
Environmental Protection Agency.
Finally, the authors acknowledge the contributions provided by the external peer
review comments developed by the Environmental Engineering Committee of the
Science Advisory Board, including Richard Con way, Chair, James Mercer, Modeling Peer
Review Subcommittee Chair, and Kachig Kooyoomjian, EPA's Executive Secretary, and
the internal peer review support provided by the Agency's Council of Science Advisors.
-------
-------
AGENCY GUIDANCE FOR CONDUCTING EXTERNAL PEER REVIEW OF
ENVIRONMENTAL REGULATORY MODELING
I. INTRODUCTION
According to EPA's Peer Review Policy Statement dated June 7, 1994, (see
attachment):
Major scientifically and technically based work products related to Agency
decisions normally should be peer-reviewed. Agency managers within
Headquarters, Regions, laboratories and field components determine and are
accountable for the decision whether to employ peer review in particular
instances and, if so, its character, scope, and timing. These decisions are made in
conformance with program goals and priorities, resource constraints, and statutory
or court-ordered deadlines. For those work products that are intended to support
the most important decisions or that have special importance in their own right,
external peer review is the procedure of choice. Peer review is not restricted to
the penultimate version of work products; in fact, peer review at the planning
stage can often be extremely beneficial.
By contrast, the policy specifically excludes "non-major or non-technical matters that
Agency managers consider as they make decisions."
Clearly, environmental models (i.e., fate and transport, estimation of contaminant
concentrations in soil, groundwater, surface water and ambient air, exposure assessment)
that may form part of the scientific basis for regulatory decision making at EPA are
subject to the peer review policy. However, it can not be more strongly stressed that
peer review should only be considered for judging the scientific credibility of the model
including applicability, uncertainty, and utility (including the potential for mis-use) of
results, and not for directly advising the Agency on specific regulatory decisions
stemming in part from consideration of the model output.
The purpose of this guidance is to provide a resource for those program managers
responsible for implementing the peer review process. More specifically, this guidance
is provided as an aid in evaluating the need and, where appropriate, conducting external
peer review related to the development and/or application of environmental regulatory
modeling. This specific guidance for modeling has been prepared to complement
general peer review guidance currently being developed by the Agency's Council of
Science Advisors (Council). Thus, Section II (framework for peer review) and Section IV
(peer review mechanisms and general criteria) reflect the latest Council guidance, and
will be revised in the future as Agency-wide guidance on peer review evolves.
The relationship of external peer review to the process of model development and
application, including consideration of peer review at various stages in the process is
described in Section III of this guidance.
-------
Section V concerning "Documentation of the Peer Review Process" has been
included in response to comments from the Agency's Science Advisory Board who cited
the need for more detailed guidance on the mechanics of the review process.
The specific elements of what could be covered in an external peer review of
model development and application are presented in Section VI of this guidance. These
elements are not meant to be prescriptive or limit the nature of peer review, but rather
are intended as an aid to improve the thoroughness and consistency of peer review.
To reiterate one of the major recommendations in the March 1992 report
commissioned by former Administrator William K. Reilly re: "Safeguarding the Future:
Credible Science, Credible Decisions (The Report of the Expert Panel on the Role of
Science at EPA)":
"Quality assurance and peer review should be applied to the planning and results
of all scientific and technical efforts to obtain data used for guidance and
decisions at EPA, including such efforts in the program and regional offices. Such
a requirement is essential if EPA is to be perceived as a credible, unbiased source
of environmental and health information, both in the United States and
throughout the world."
In conclusion, this document is intended to provide guidance to program
managers in their efforts to consider, and apply where appropriate, external peer review
to environmental regulatory modeling. The guidance contained in this document must
not be construed as rigid requirements.1
II. FRAMEWORK FOR PEER REVIEW OF ENVIRONMENTAL REGULATORY
MODELING
Peer review can be an important tool in assisting the Agency to document the
quality and credibility of the science upon which its regulatory and policy decisions are
made. Modeling to provide the scientific support for environmental regulatory decision
making at EPA can be thought to conceptually involve three stages including: (A) model
development;
(B) model application; and (C) consideration of modeling results in decision making.
The guidance set out in this document is not final Agency action. It is not intended, nor can it be relied upon,
to create any rights enforceable by any party hi litigation with the United States. EPA officials may decide to
follow the guidance provided herein, or act at variance with the guidance, based on an analysis of specific
circumstances. The Agency also reserves the right to change this guidance at any time without public notice.
-------
External peer review, as discussed in more detail below, is generally relevant to
the first stage of model development and may be relevant in appropriate cases to the
second stage of model application. Although external peer review is not directly
germane to the regulatory or policy decision itself, it is important at this third stage to
bring forward information regarding prior peer review comments and the Agency's
response related to model development and/or application. This information may aid the
decision maker in interpreting and weighing the utility of modeling results along with all
the other considerations (e.g., field data; risk, cost, and benefits information;
requirements to use best available technology; environmental justice issues; etc.) in
reaching a regulatory decision.
Broader guidance regarding the applicability of peer review, as discussed in the
Introduction, is currently being developed by the Council of Science Advisors. As this
broader guidance develops and evolves in the future, the
framework for external peer review of environmental regulatory modeling, as discussed
below, will be revised.
Ai Model Development
Models are developed for a variety of reasons, including:
1. use as research tools to explore new scientific issues;
2. simplification and/or refinement of existing model paradigms or software;
3., use as screening tools; and
4. to estimate compliance with regulatory requirements (e.g., National
Ambient Air Quality Standards).
This guidance document, by its terms, does not directly address models
developed for reasons other than to support regulator/ decision making (e.g., research
tools). Models developed expressly for and used exclusively within a research program
should be subject to essentially the same review process as other research results (e.g.,
informal critique by scientific colleagues, formal appraisal by senior scientists and
managers, publication in refereed journal, etc.). However, if the purpose of a research
model is expanded in the future to develop scientific information for Agency decision
making, then the program manager should consider arranging for reassessment of the
model in accordance with the guidance presented below.
From a general scientific perspective, a well-conceived model is one that provides
an acceptable mathematical approximation of a physical, chemical, biological, social or
economic system. Model development frequently touches on the interface involving the
-------
state-of-the-art in several areas including: (1) the technical and scientific understanding
of processes and mechanisms; (2) applicable solution techniques (e.g., analytical,
numerical); (3) computer science and technology; and (4) the Agency's need for
scientific decision-support tools. The goal of model development is to provide methods
which can be applied to improve Agency analysis and decision making. As such, the
development of environmental models cannot be performed in a scientific vacuum.
There must be a coherent class of applications in mind against which the appropriateness
of the science can be judged.
At the model development stage, a key step is to define and compare to existing
models, the set of conditions under which the use of a model is scientifically defensible -
this is known as the "application niche." Peer review of model development would be
expected to include evaluation of the application niche, along with consideration of
uncertainty and other areas of model performance. Approaching peer review from this
perspective should help the decision maker understand the limitations of the scientific
basis of the model and confidence in its results. It is only with this firm knowledge that
the Agency can develop sound regulatory and policy decisions.
External peer review of major technical issues related to environmental regulatory
modeling is generally a lengthy process. This is precisely why external peer review
should be identified as an important and integral aspect of an action plan for model
development. At first glance external peer review might be expected to slow down the
process. However, initiating peer review at early stages may, in fact, save time by
redirecting misguided initiatives, identifying alternative approaches, or providing strong
technical support for a potentially controversial position.
B.. Model Application
The middle stage, model application, begins with examining the stated application
niche and its applicability to current needs and/or exploring whether a model can be
tailored to fit a new niche. For existing models, especially models developed outside of
EPA, peer review may be appropriate to the extent that either: 1) new information
becomes available which calls into question the appropriateness of the previously
defined application niche; or 2) a model might be considered for application outside the
niche for which it was originally developed.
Peer review of a model's applicability should, where possible, be planned well in
advance of any decision making involving use of the model's results. In this way, the
formation of "sound science" is distinct from the regulatory decision in which it is
considered. The results of such a peer review can aid in the ultimate judgment by a
decision maker to consider whether or how to use a particular model's result in the
regulatory process.
-------
Normally, the first application of a model should undergo peer review. For
subsequent applications, a program manager should consider the scientific/technical
complexity and/or novelty of the particular circumstances as compared to prior
applications (also see Section IV-B). Peer review of all similar applications should be
avoided because this would likely waste precious time and monetary resources while
failing to provide the decision maker with any new relevant scientific information upon
which to base a regulatory or policy selection. Nevertheless, a program manager may
consider conducting peer review of applications upon which costly decisions are based
or applications which are likely to end up in litigation.
As an alternative to peer review of model application, the Office of Air Quality
Planning and Standards has developed a formal review and acceptance procedure
through rulemaking to evaluate the utility of alternative models for a particular
application.
C Environmental Regulatory Decision Making
The final stage involves consideration of modeling results in the decision making
process. This stage may include consideration of: (1) selecting among available models
and assumptions/standard defaults; and/or (2) whether/how to consider modeling results
in the regulatory process. Information previously derived from peer review of scientific
issues may provide key information for the decision maker in understanding the
uncertainties and utility (and potential for mis-use) of modeling results. When possible,
public discussion of the scientific and technical underpinnings associated with the earlier
stages of model development and/or application in advance of the final stage of
regulatory decision making is expected to reduce criticism regarding the Agency's use of
"sound science."
Environmental regulatory and policy decisions also involve other scientific and
non-scientific factors, and are by law, required to be made by the responsible Agency
decision makers. Thus, such decisions are not appropriate subjects for scientific peer
review. Rather, the process of public comment is frequently employed by the Agency at
this stage.
III. RELATIONSHIP OF EXTERNAL PEER REVIEW TO THE PROCESS OF
ENVIRONMENTAL REGULATORY MODEL DEVELOPMENT AND APPLICATION
The Agency's peer review policy notes that "properly applied, peer review not
only enriches the quality of work products but also atdds a degree of credibility that
cannot be achieved in other way. Further, peer review early in the development of work
products in some cases may conserve future resources by steering development along the
most efficacious course." Since this subject guidance focuses on the role of external peer
-------
review, its relationship to other levels of peer review can best be understood by
considering where external peer review may fit into the total process of environmental
regulatory modeling. The following paradigm is meant to illustrate the general process:
Step 0:
Step Ob:
(optional)
Step 1:
Step 1b:
(optional)
Step 2:
Step 3:
Step 4:
The program manager within the originating office (AA-ship or
Region) identifies elements of the regulatory process which would
benefit from the use of environmental models. A review/solicitation
of currently available models and related research should be
conducted. If it is concluded that the development of a new model
is necessary, a research/development work plan would be prepared.
The program manager may consider internal and/or
external peer review of the research/development concepts to
determine whether they are of sufficient merit and whether the
model is likely to achieve the stated purpose.
The originating office develops a new or revised model or evaluates
the possible novel application of model developed for a different
purpose.
The program manager may consider external peer
review of the technical or theoretical basis prior to final
development, revision or application at this stage. For model
development, this review should evaluate the stated application
niche.
Initial Agency-wide (internal) peer review/consultation of model
development and/or proposed application may be undertaken by the
originating office.
Model design, default parameters, etc. and/or intended application
are revised (if necessary) based on consideration of internal peer
review comments.
External peer review is considered by the originating office.
Model design, default parameters, etc. and/or intended application
are revised (if necessary) based on consideration of external peer
review comments.
Final Agency-wide evaluation/consultation may be implemented by
the originating office. This step should consist of consideration of
external peer review comments and documentation of the Agency's
response to scientific/technical issues.
-------
(Note: Steps 2 and 4 are relevant where there is either an internal Agency standing or
ad hoc peer review committee or process).
A program manager may decide that peer review (step 2 -internal, and step 3 -
external) should take place at more than one time during the processes of model
development and model application (i.e., optional steps Ob and/or 1b). This decision
would depend on the nature and complexity of scientific issues that are presented.
Additionally, a program manager may also consider the utility of employing the
broader concept of peer involvement - that is, to augment staff efforts by soliciting
subject-matter experts from outside the program.
Based in part on the results of a scientific peer review of model development and
application, as well as other non-technical issues, the program manager would be
responsible for advising the decision maker as to the consistent, equitable, and
appropriate use of the model and its output for environmental regulatory purposes.
IV. MECHANISMS AND GENERAL CRITERIA FOR CONDUCTING EXTERNAL PEER
REVIEW
The mechanisms and general criteria for conducting external peer review
presented below are based upon generic peer review guidance currently being
developed by the Council of Science Advisors. As this broader guidance develops and
evolves in the future, this Section will be revised.
/V Peer Review Mechanisms
Mechanisms for accomplishing external peer review include, but are not limited
to, the following:
i. Using an ad hoc technical panel of at least three scientists;
ii. Using an established external peer review mechanism such as the Science
Advisory Board or Scientific Advisory Panel; or
iii. Holding a technical workshop.
Qualifications for peer reviewers will likely van/ for model development versus
model application. For the former, the emphasis may be toward modelers, while for the
latter, the emphasis may be toward scientists with technical expertise in other disciplines
(e.g., statistics, field monitoring, etc.).
-------
8
New models, or significant modifications and/or new applications of established
models, should be considered for publication in refereed journals. However, this step
should be supplemental to and not a substitute for the peer review mechanisms
presented above.
E5. General Criteria
General criteria to be considered for determining when and by what mechanism
to initiate an external peer review of the development and/or application of
environmental regulatory models include:
i. Use of model results as a basis for major regulatory or policy/guidance
decision making;
ii. Significant investment of Agency resources;
iii. Inter-Agency or cross-Agency implications/applicability;
iv. Treatment of a new scientific issue; or
v. Novelty and/or complexity of the science.
V.
DOCUMENTATION OF THE PEER REVIEW PROCESS
It is important to prepare documentation for each peer review conducted and
make this available to the decision maker along with the modeling results. The record
should include the following information:
A. Identity of the peer reviewers, their relevant expertise, and their
institutional affiliation;
B. Questions/issues posed to the peer reviewers;
C. Reviewer comments, either as a collection of individual statements or as a
consensus statement; and
D. Agency response to peer review comments including rationale.
-------
VI. SPECIFIC ELEMENTS OF EXTERNAL PEER REVIEW FOR ENVIRONMENTAL
REGULATORY MODELING
This Section of the guidance addresses the specific elements that should be
considered to provide the Agency with consistent scientific/technical external peer
review of environmental regulatory model development and/or application.
Such elements could include, but are not limited to:
A.. Model Purpose/Objectives
The first step in evaluating a model used for environmental regulatory purposes is
to clearly understand the broad context in which a model is intended to be used.
i. What is the regulatory context in which the model will be used and what
broad scientific questions is the model intended to answer (e.g., evaluating
the range of human respiratory exposure resulting from air toxics release
from a power plant)?
ii. What is the model's application niche (i.e., the particular physical,
chemical, and/or biological system and set of defining conditions for which
the model is expected to be scientifically defensible)?
iii. What are the model's strengths and weaknesses? How well does the
model fit its intended application niche compared to existing models and
available data sets? It is important that the Agency avoid duplicating prior
efforts.
B.. Major Defining and Limiting Considerations
With a clear understanding of the broad purpose and objectives of a model in a
regulatory context, the scientific context needs to be addressed.
i. Which processes are characterized by the model (e.g., transport, diffusion,
chemical reactions, removal mechanisms, etc.)?
ii. What are the important temporal and spatial scales? Is the grid resolution
appropriate for the problem?
iii. What is the level of aggregation?
-------
10
C Theoretical Basis for the Model
Once the regulatory and scientific contexts of a model have been defined, the
basis for problem solving must be formulated.
i. What algorithms are used within the model and how were they derived?
What is the mechanistic basis?
ii. What is the method of solution (numerical, analytic)?
iii. What formulations are used for those processes which are parameterized?
iv. How does the basis for problem solving compare to existing models?
What is the scientific rationale?
v. What are the shortcomings of the modeling approach (e.g., missing or over-
simplification of key processes, restrictive dimensionality, etc.)?
D.
Parameter Estimation
Parameter estimation may be based on case specific data or in their absence, on
default values. In the latter case, it is important to understand how parameter defaults
were established.
i. What methods were used for parameter estimation?
ii. What data were available for parameter estimation?
iii. What methods were used to estimate parameters for which there were no
data?
iv. What is the reliability of parameter estimates?
v. What are the boundary conditions and are they appropriate?
E.. Data Quality/Quantity
All models require the input of various types and amount of data. Models may
also rely on experimental data to help shape their computational algorithms. In large
measure, the utility of a model for regulatory purposes depends on the quality, quantity,
and spatial and temporal adequacy of data used in its design and in support of its
application.
-------
F.
11
Questions related to model design include:
i. What data were utilized in the design of the model?
ii. How can the adequacy of the data be defined in terms of quality, quantity,
and spatial and temporal applicability taking into account the regulatory
objectives of the model?
Questions related to model application include:
i. What kinds of data are required to apply the model?
ii. To what extent are these data available and what are the key data gaps?
iii. Have data quality objectives been defined? If so, are they scientifically
defensible?
iv. Is the quantity of data sufficient to address the likely variability? What
statistical analyses were performed and are they appropriate?
v. To what extent are the data suitable with regard to estimating spatial and
temporal effects?
VI.
Do additional data need to be collected and for what purpose?
Key Assumptions
The applicability of a model depends on the adequacy of its basic underlying
assumptions.
i. What are the key assumptions?
ii. What is the basis for each key assumption and what is the range of
possible alternatives?
G.
iii. How sensitive is the model toward modifying key assumptions?
Modet Performance Measures
The most basic test of a model's adequacy is to understand how well its results
compare with real world measurements.
i. What criteria have been used to assess model performance?
-------
12
ii. Did the data bases used in the performance evaluation provide an
adequate test of the model in terms of applicability to the modeling niche?
iii. How accurate can the model be expected to perform? Does the model
exhibit any overall bias throughout the range of its predictions? Bias is an
important test of the model's formulation since intrinsic system uncertainty
is not present.
iv. How well does the model address, distinguish, and report variability and
uncertainty in its output? Which parameters and key assumptions are most
significant in determining the model's variability and uncertainty?
v. How does the model perform relative to other models in this application
niche?
JhL Model Documentation and Users Guide
The utility of model for regulatory purposes depends on the availability of a clear
documentation report and a comprehensive users guide. Do these cover:
i Model applicability and limitations?
ii. Data input?
Interpretation of results? and
in.
IV.
Documentation of the model code and other key aspects such as
verification testing?
JL Retrospective
A retrospective analysis of the "big picture" may sometimes reveal insights that an
analysis of individual components of a model may miss.
i. Does the model satisfy its intended scientific and regulatory objectives?
11.
in.
Is there any available scientific evidence to suggest changes to either the
model design and/or key parameters and assumptions prior to its use for
regulatory purposes?
How robust (i.e., not overly sensitive toward small changes in modifying
key assumptions or input data) are the model predictions?
-------
13
iv. How well does the model output quantify the overall uncertainty resulting
from limitations/simplifications in its design; use of standard assumptions;
availability of supporting data; etc.?
v. What key research is necessary to refine or improve the model and/or the
data bases upon which it relies?
The elements provided above are not meant to be prescriptive or limit the nature
of external peer review. Rather the purpose for their inclusion is to provide modeling-
related guidance for peer review protocols which will be consistent with more general
guidance being developed by the Council of Science Advisors.
Attachment 1:
"Peer Review and Peer Involvement at the U.S.
Environmental Protection Agency" dated June 7, 1994
-------
-------
PEER REVIEW AND PEER INVOLVEMENT
AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY
This document establishes the policy of the United States
Environmental Protection Agency (EPA) for peer review of
scientifically and technically based work products that are
intended to support Agency decisions. Peer review is presented in
the context of the broader concept, peer involvement.
BACKGROUND
The report "Safeguarding the Future: Credible Science,
Credible Decisions"1 focused on the state of science at EPA. The
panel of experts who prepared the report emphasized the importance
of peer review, especially external peer review, and the need for
broader and more systematic use of it at EPA to evaluate scientific
and technical work products. Their specific recommendation
regarding peer review reads as follows:
"Quality assurance and peer review should be applied to
the planning and results of all scientific and technical
efforts to obtain data used for guidance and decisions at
EPA, including such efforts in the program and regional
offices. Such a requirement is essential if EPA is to be
perceived as a credible, unbiased source of environmental
and health information, both in the United States and
throughout the world."
In response to this recommendation, then-Administrator Reilly
directed staff to develop an EPA-wide policy statement, which he
issued in January, 1993. The paragraphs below preserve the core of
that earlier statement while updating it to specify the role of the
Science Policy Council in guiding further implementation of the
policy. Effective use of peer review is indispensable for
fulfilling the EPA mission and therefore deserves high-priority
attention from program managers and scientists within all pertinent
Headquarters and Regional Offices.
EPA/600/9-91/050, March 1992.
-------
PEER INVOLVEMENT AND PEER REVIEW
EPA strives to ensure that the scientific and technical
underpinnings of its decisions meet two important criteria: they
should be based upon the best current knowledge from science,
engineering, and other domains of technical expertise; and they
should be judged credible by those who deal with the Agency. EPA
staff therefore frequently rely upon peer involvement — that is,
they augment their capabilities by inviting relevant subject-matter
experts from outside the program to become involved in one or more
aspects of the development of the work products that support
policies and actions.
One particularly important type of peer involvement occurs
when scientifically and technically based work products undergo
peer review — that is, when they are evaluated by relevant
experts from outside the program who are peers of the program
staff, consultants, and/or contractor personnel who prepared the
product. Properly applied, peer review not only enriches the
quality of work products but also adds a degree of credibility that
cannot be achieved in any other way. Further, peer review early in
the development of work products in some cases may conserve future
resources by steering the development along the most efficacious
course.
Peer review generally takes one of two forms. The review team
may consist primarily of relevant experts from within EPA, albeit
individuals who have no other involvement with respect to the work
product that is to be evaluated (internal peer review) . Or the
review team may consist primarily of independent experts from
outside EPA (external peer review).
POLICY STATEMENT
Major scientifically and technically based work products
related to Agency decisions normally should be peer-reviewed.
Agency managers within Headquarters, Regions, laboratories, and
field components determine and are accountable for the decision
whether to employ peer review in particu ar instances and, if so,
its character, scope, and timing. These decisions are made in
conformance with program goals and priorities, resource
constraints, and statutory or court-ordered deadlines. For those
work products that are intended to support the most important
decisions or that have special importance in their own right,
external peer review is the procedure of choice. Peer review is
not restricted to the penultimate version of work products; in
fact, peer review at the planning stage can often be extremely
beneficial.
-------
SCOPE
Agency managers routinely make regulatory and other decisions
that necessarily involve many different considerations. This
policy applies to major work products that are primarily scientific
and technical in nature and may contribute to the basis for policy
or regulatory decisions. By contrast, this policy does not apply
to nonmajor or nontechnical matters that Agency managers consider
as they make decisions. Similarly, this policy does not apply to
these ultimate decisions.
This policy applies where appropriate, as determined by the
National and Regional Program Managers, to major scientifically and
technically based work products initiated subsequent to the date of
issuance. Peer review should be employed to the extent reasonable
to relevant work products that currently are under development.
This policy does not apply to the bases for past decisions, unless
and until the relevant scientific and technical issues are
considered anew in the Agency's decision-making processes.
Except where it is required by law, formal peer review (as
distinguished from the Agency's normal internal review procedures)
should be conducted in a manner that will not cause EPA to miss or
need extension of a statutory or court-ordered deadline. Agency
managers still may undertake peer review if it can be conducted
concurrently with necessary rulemaking steps.
LEGAL EFFECT
This policy statement does not establish or affect legal
rights or obligations. Rather, it confirms the importance of peer
review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the
policy. On a continuing basis, Agency management is expected to
evaluate the policy as well as the results of its application
throughout the Agency and undertake revisions as necessary.
Therefore, the policy does not stand alone; nor does it establish
a binding norm that is finally determinative of the issues
addressed. Minor variations in its application from one instance
to another are appropriate and expected; they thus are not a
legitimate basis for delaying or complicating action on otherwise
satisfactory scientific, technical, and regulatory products.
Except where provided otherwise by law, peer review is not a
formal part of or substitute for notice and comment rulemaking or
adjudicative procedures. EPA's decision whether to conduct peer
review in any particular case is wholly within the Agency's
discretion. Similarly, nothing in this policy creates a legal
requirement that EPA respond to peer reviewers. However, to the
extent that EPA decisions rely on scientific and technical work
products that have been subjected to peer review, the remarks of
peer reviewers should be included in the record for that decision.
-------
IMPLEMENTATION
The Science Policy Council is responsible for overseeing
Agency-wide implementation. Its responsibilities include promoting
consistent interpretation, assessing Agency-wide progress, and
develocina recommendations for revisions of the policy as
developing
necessary
The Science Policy Council will oversee a peer-review work
group, which will include representatives from program units
throughout EPA to effect a consistent, workable implementation of
the policy. The work group will assist the programs in (1)
formulating and, as necessary, revising standard operating
procedures (SOPs) for peer review consistent with this policy; (2)
identifying work products that are subject to review; and (3) for
each major work product, selecting an appropriate level and timing
of peer review.
In assisting the programs, the work group will take into
account statutory and court deadlines, resource implications, and
availability of disinterested peer reviewers. The group will work
closely with Headquarters offices and the Regional Offices toward
ensuring effective, efficient uses of peer review in supporting
their mission objectives. However, the Assistant Administrators
and Regional Administrators remain ultimately responsible for
developing SOPs, identifying work products subject to peer review,
determining the type and timing of such review, documenting the
process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.
Because peer review can be time-consuming and expensive,
Agency managers within Headquarters, Regions, laboratories, and
field components are expected to plan carefully with respect to its
use — taking account of program priorities, resource
considerations, and any other relevant constraints as well as the
policy goal of achieving high-quality, credible underpinnings for
decisions. External peer reviewers should be chosen carefully to
ensure an independent and objective evaluation. The affiliations
of peer reviewers should be identified on the public record, so as
to avoid undercutting the credibility of the peer-review process by
conflicts of interest.
This policy is effective immediately. The peer-review work
group mentioned above will identify the focal point to whom
comments and questions should be addressed and, from time to time,
will provide further information about implementation activities.
APPROVED:
DATE
CAROL M. BROWNER, ADMINISTRATOR
-------
-------
------- |