UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON. D.C. 20460
EPA-SAB-EEC-LTR-93-008
OFFICE OF THE ADMINISTRATOR
July 8, 1993 SCIENCE ADVISORY BOARD
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: Review of Draft Agency Guidance for Conducting External Peer
Review of Environmental Regulatory Modeling
Dear Ms. Browner:
In response to the January 1989 Report of the Science Advisory Board
(SAB) entitled "Resolution on the Use of Mathematical Models by EPA for
Regulatory Assessment and Decision-Making," (EPA-SAB-EEC-89-012), an ad hoc
Agency Task Force on Environmental Regulatory Modeling (ATFERM) was created
under the Risk Assessment Council. ATFERM encompasses all offices, regions,
and laboratories within EPA, and represents the first time modeling activities have
been coordinated at this scale within the Agency. The SAB was asked to review
the draft document entitled "Agency Guidance for Conducting External Peer
Review of Environmental Regulatory Modeling," (January 26, 1993 draft), one of
the first products of ATFERM which support better management of model
development and application.
Concurrently, and in response to the March 1992 Report on the Expert
Panel on the Role of Science at EPA, "Safeguarding the Future: Credible Science,
Credible Decisions," (EPA/600/9-91/050), a Council of Science Advisors (CSA) was
established within the Agency. The first major task of the CSA was to develop an
Agency Policy on Peer Review. This policy eventually will consist of a general
peer review guidance. The modeling guidance has been developed ahead of the
general peer review guidance; thus, it refers to a document on general peer review
that the SAB cannot evaluate. For this reason, the SAB went beyond the charge
of reviewing the guidance on peer reviewing models, and has made comments on
the general peer review process.
This review by the Modeling Peer Review Subcommittee (MPRS) took place
on March 3-4 at the Environmental Engineering Committee (EEC) meeting in
Washington, DC. During discussions on March 3, 1993, it was pointed out that
peer review of models is just one initiative in the general area of model use within
-------
the Agency. This review has focused on a portion of this overall program,
specifically peer review. Thus, some of the more general comments on models,
included here for completeness, may be more appropriate for other components of
the modeling initiative.
1. Charge
The charge to the SAB's EEC/MPRS was stated as two questions and a list
of issues:
a) How well does the guidance address its goals of being a resource for
Agency managers implementing external peer review of environmental
regulatory modeling?
b) Does the guidance provide the proper balance between being too
restrictive versus not providing enough detail?
The specific issues addressed included the following:
a) Key elements of a modeling peer review;
b) Modeling framework for addressing peer review;
c) Relationship of peer review to the over all process of model
development and application; and
d) Model-specific peer review mechanisms and criteria.
2. Model Peer Review Process
In general, the subject manual represents a good balance between over
prescriptive and under prescriptive guidance for external peer review of modeling.
The authors correctly recognize that guidance often becomes de facto regulations,
and appropriately stress that this guidance should not be construed as rigid. The
authors are encouraged not to yield to the temptation to specify more prescriptive
guidance that would lose the flexibility necessary to allow peer review of the range
of regulatory models used by the Agency.
The authors also correctly identify many of the concerns associated with
implementing such a comprehensive external peer review process. In particular,
the document points out the concerns associated with slowing the regulatory
process and the need to plan the external peer review process early in the activity.
It is possible for the process to be abused to stifle activity within the Agency. The
authors may want to consider techniques designed to avoid this potential problem.
The Agency needs to establish some reasonable time frames to complete reviews,
as well as checks and balances within the review process.
-------
3. Application of Review Process
The guidance specifies that external peer review is generally relevant to
model development and may be used on a case-by-case basis to applications of
models. The application of complicated environmental models is subject to errors,
even if the development of the model has been thoroughly peer reviewed. These
errors may be more severe than those encountered in the development of models.
Some common examples of application problems include improper input data,
improper boundary condition specification, poor documentation of inputs and
assumptions, the applicability and appropriateness of using default values,
documentation and justification of adjusting model inputs to improve model
performance, and exercising the model outside the range of its validity.
While there are applications that should be peer-reviewed, the guidance
should document reasonable rationales where peer review is unnecessary. These
rationales could include common and routine calculations. Guidance on
applications that should require external peer review might include applications on
which costly decisions are based or applications that may end in litigation. This
would include non-routine model applications which have significant impacts that
are run by states or their contractors under the direction of EPA. An example of
this would be the application of complex photochemical grid models used in State
Implementation Plan (SIP) development. Guidance also might be offered on how
stake-holders who are not satisfied with potential model applications can have
input into whether or not the external peer review process occurs. Guidance
should also be given as to what constitutes adequate model performance.
As emphasized by the SAB's EEC/MPRS, it is essential that applications
that will have a significant impact also be subjected to peer review. An example
pointed out by the SAB's Executive Committee included an entire category of
applications that will have hundreds of millions of dollars of impacts that will not
be covered by the peer review process as it is presently envisioned. These are the
applications to develop the SIPs for nonattainment areas for the criteria air
pollutants. For most of the criteria pollutants, this exercise consists of applying a
routine dispersion model, but for ozone, it involves the application of a complex
photochemical grid model that is essentially a research project. The model runs
are not made by the EPA; they are made by the state agencies or their
contractors. However, the EPA dictates which model to use, and EPA (OAQPS
and the Regional Offices) does not actually "turn the crank," they control the
process. These applications should be included in the peer review process, because
they involve the use of a highly sophisticated, complex model which lacks adequate
performance evaluations.
It is further noted by the SAB's Executive Committee, with regard to the
SIPs, particularly with regard to ozone, that decisions are being made in the
absence of sufficient input data. In these cases, EPA allows the use of "default"
values in lieu of actual input data, and that the evaluation of this practice needs
to be included in the peer review process.
-------
The SAB's EEC/MPRS believes that when models are used in the regulatory
decision-making process, there is a role for peer review to insure that model
results are not misused, and that the decision is consistent with the modeling
effort. Although the use of models has been divided into three stages in the
guidance (development, application, and use in decision making), in actuality, the
use of models is often a continuum across these stages, which are not distinct.
The guidance attempts to have the scientific basis of potential future regulatory
decisions for which a model may be used reviewed in earlier stages (e.g., model
development). This is a risky approach, as it may be impossible to predict how
the model will ultimately be used in the decision making process. It is more
prudent for the Agency to have a review on the use of models at this last stage
(regulatory decisions), as well as earlier stages, to insure that the use of models is
appropriately incorporated into the decision-making.
4. General Peer Review Process Closure
There is clear omission in the guidance on what should happen after the
external peer review process is completed. While this subject may be generally
applicable to Agency-wide peer review processes, it also is necessary to include it
in the model peer review guidance. It should address how the issues raised in the
peer review process will be resolved. An impartial third party is necessary to
review the proposed resolution. This impartial third party could be senior
management, CSA, or a continuing form of ATFERM, so long as they are expert
in the issues raised by the review. To expedite the development process, the third
party should be within EPA.
5. Documentation
Documentation of the entire peer review process is extremely important.
Without documentation of the review(s) and the response(s) to the reviewer
comments, there is neither rigor in the process nor evidence that the review
actually occurred or was taken into account. There is a need in the guidance
document to specify a standardized approach to documenting* the review process,
especially the resolution of reviewer comments.
6. Adequacy of Review Criteria
It is not clear that the elements of the peer, review process are suitable for
all of the peer review mechanisms listed in the guidance document. Specifically,
one external peer review process involves the use of peer-reviewed scientific
journals that should not be constrained to the elements posed in the guidance
document. Also, the journal review process may require much more time
(sometimes one to two years) than the Agency schedule can allow. Clearly, this
mechanism cannot give the complete answers to the questions posed in the
elements of the peer review process, even though it is the most accepted form of
external peer review. The Agency should try to reconcile the differences and
describe how this mechanism should be used to achieve the desired end-product.
-------
Specific guidance may be required on what constitutes a peer-reviewed
journal or what constitutes a qualified audience in a technical workshop.
Guidance could also be provided on what constitutes a "qualified" peer reviewer
and what mix or diversity of scientific backgrounds is required for an adequate
review. It should also be stressed that qualified peer reviewers will be different
for development and application reviews. For the former, modelers may only be
required, but for the latter, scientists and engineers familiar with the specific
domain, source emissions and data analysis experts may also be required. Finally,
guidance should be provided on the number of reviewers (it probably should be a
minimum of three) that are required to give a balanced review.
7. Adequacy of Review Elements
It is not clear that the extensive peer review elements that are defined in
the guidance are equally important or that the list is complete. Models are, at
best, mathematical simplifications or estimations of the current state of
understanding associated with the processes and effects of interest. The most
important element to the review process is the verification of the model against
available data in the range of conditions of interest. It is not clear that other
elements are important if the model does not accurately estimate the processes
and effects. The ability of the model to estimate the results over a wide range of
conditions (so that the agreement between predicted and observed values is
unlikely to be due to compensating errors) is the key measure of success of
modeling without compensating errors (the issue of compensating errors is a
serious problem with some air quality models) and should be an important part of
the peer review process.
Specific items that should be added to the review elements follow.
a) "Model purpose/objectives": is the model needed, i.e., has the model
selection process overlooked an existing code that can perform the
same function (with or without modification)? It is important that
the Agency not duplicate efforts.
b) "Major defining and limiting considerations": (1) what are the
important parameters and their quantity (e.g., temporal and spatial
scales), and the effects of other parameters on these (e.g., temperature
effects) and (2) is the conceptual model correct?
c) "Theoretical basis for model": (1) evaluation of scientific foundation
of modeling approach and equations, (2) identification of shortcomings
of modeling approach (e.g., missing processes, restrictive
dimensionality, over-simplification of processes), (3) is there a better
(or simpler) modeling approach which could satisfy objectives and
have less limitations, (4) is the level of scientific understanding
consistent with the modeling objectives, (5) what equations are solved
-------
and do they capture the natural system, and (6) stability and
application of the solution?
d) "Parameter estimation": (1) what is the sensitivity of the results to
the estimation or variation in parameters and (2) what are the
boundary conditions and (3) are they appropriate?
e) "Model performance measures": (1) has the model conserved mass
(model should have a mass balance check), (2) has model uncertainty
been adequately addressed, (3) what criteria will be used to determine
adequate model performance, and (4) if model performance is not
adequate, what criteria will be used to encourage modelers to
document and justify adjustments, rather than arbitrarily "adjust"
inputs to improve performance?
f) "Model documentation and users guide": (1) is the model
documentation and level of complexity consistent with the educational
background of proposed users and (2) was the model tested by
someone other than the model developer. Verification tests should be
included as part of the documentation?
8. Agency Task Force on Environmental Regulatory Modeling (ATFERM)
The ad hoc Agency Task Force on Environmental Regulatory Modeling
(ATFERM), created under the Risk Assessment Council, responds to the January
1989 Report of the Science Advisory Board (SAB) entitled "Resolution on the Use
of Mathematical Models by EPA for Regulatory Assessment and Decision-Making,"
(EPA-SAB-EEC-89-012). Since ATFERM represents the first time modeling
activities have been coordinated at an Agency scale, the SAB supports the concept
of ATFERM. Unfortunately, ATFERM is to complete its mission within 12
months. A model-coordinating activity within the Agency is needed on a
continuing basis.
9. Recommendations
a) Some form of ATFERM should continue to exist after the 12-month
initial time period is complete. A model-coordinating activity within
the Agency is needed on a continuing basis. This recommendation
was initially made in the SAB's modeling resolution (EPA-SAB-EEC-
89-012) , and was repeated in a more recent review on usage of
computer models in the hazardous waste and Superfund programs
(EPA-SAB-EEC-91-016). One function of ATFERM would be to
monitor the effectiveness of the model peer-review process.
b) The steps of the general peer review process, including (1) a
documented response to the review comments, (2) an independent
evaluation of the review comments, and (3) the documented response
-------
to insure that the review comments were satisfactorily considered,
need to be adequately addressed. The final step should be an internal
(within EPA) function to expedite the review process in order not to
slow model development. This closure step could be performed by
senior management, CSA, or a continuing form of ATFERM, so long
as they are expert in the issues raised by the review.
c) The introduction of the guidance on modeling peer review needs to
include a discussion on model types and codes used within the
Agency. Specifically, the discussion should address the variety of
model types used within the Agency, including those for air, surface
water, groundwater, multimedia, and the codes for associated
activities such as risk assessment and Monte Carlo simulation. This
discussion should also include a description of the types of
applications for which these models can be used. There needs to be
an indication as to how the peer review process accommodates this
diversity of models.
d) The focus on the guidance for modeling peer review is on model
development, but equal emphasis also needs to be placed on review of
model applications. Such emphasis should not imply that all model
applications need external peer review, but for those applications that
do require external peer review, the current guidance is inadequate.
Routine calculations or model applications that follow standard
operating procedures would not normally require external review.
Applications where external review would be considered include:
situations where multimillion dollar decisions are based on model
results (e.g., hazardous waste site remediation and SIP development
using complex photochemical grid models) or model results where
future litigation is anticipated, and these should include those
applications run by states and their contractors under the direction of
EPA.
e) The SAB's EEC/MPRS recommends that when models are used in the
regulatory decision-making process, there be a role for peer review to
insure that model inputs (and default values) are valid and justified,
adjustments to inputs to improve performance are documented and
justified, that model results are not misused, and that the decision be
consistent with the modeling effort.
f) The peer review process early in the modeling activity should include
assessment of the need for a new model, where model development is
anticipated. An important role of ATFERM is to reduce the amount
of duplication that exists within the Agency, especially in the area of
model development.
-------
g) In the paradigm on external peer review, there needs to be more
emphasis on documentation of the whole process.
h) Some guidance needs to be provided as to what constitutes adequate
model performance.
i) ATFERM has made a good attempt not to be overly prescriptive;.
Specific information that might be added include: (1) profiles for
typical reviewers, (2) required number of reviewers (minimum of
three), and (3) when Beta testing is necessary.
j) EPA should consider making it a requirement to peer-review all
models.
We are pleased to have had the opportunity to be of service to the Agency.
We trust that these comments will help in your guidance of this important
program, and look forward to your response.
/^»w
Dr. Raymond C. Loehr, Chair
Executive Committee
Science Advisory Board
Sincerely,
Mr. Richard A. Conway, Chair
Environmental Engineering Committee
Science Advisory Board
1*0.
Dr. James W. Mercer, Chair
Modeling Peer Review Subcommittee
Environmental 'Engineering Committee
Science Advisory Board
8
-------
NOTICE
This report has been written as a part of the activities of the Science
Advisory Board, a public advisory group providing extramural scientific
information and advice to the Administrator and other officials of the
Environmental Protection Agency. The Board is structured to provide a balanced,
expert assessment of scientific matters related to problems facing the Agency.
This report has not been reviewed for approval by the Agency; hence, the
comments of this report do not necessarily represent the views and policies of the
Environmental Protection Agency or of other federal agencies. Any mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.
-------
Science Advisory Board
Modeling Peer Review Subcommittee
Environmental Engineering Committee
Chair:
Dr. James W. Mercer, President, GeoTrans, Inc., Sterling, VA
Members and Consultants:
Dr. Linda M. Abriola, Associate Professor, University of Michigan, Ann Arbor, MI
Dr. Calvin C. Chien, Principal Consultant, E.I. duPont de Nemours & Company,
Wilmington, DE
Dr. James H. Johnson, Chairman, Department of Civil Engineering, Howard
University, Washington, DC
Dr. Jo Ann Lighty, Assistant Professor, University of Utah, Salt Lake City, UT
Dr. June Fabryka-Martin, Los Alamos National Laboratory, Los Alamos, NM
Dr. Wm Randall Seeker, Senior Vice President, Energy & Environmental Research
Corp., Irvine, CA
Dr. Mitchell J. Small, Professor, Depts. of Civil Engineering and Engineering &
Public Policy, Carnegie-Mellon University, Pittsburgh, PA
Science Advisory Staff:
Dr. K. Jack Kooyoomjian, Designated Federal Official, U.S. EPA, Science Advisory
Board (A101F), 401 M Street, SW, Washington, DC 20460
Mrs. Diana L. Pozun, Staff Secretary, Environmental Engineering Committee
Dr. Donald G. Barnes, Staff Director, Science Advisory Board
n
-------
ABSTRACT.
The Modeling Peer Review Subcommittee (MPRS), along with its parent
Environmental Engineering Committee (EEC) of the Science Advisory Board (SAB)
has prepared a letter report on its March 3 and 4, 1993 review of the draft,
entitled "Agency Guidance for Conducting External Peer Review of Environmental
Regulatory Modeling." This draft guidance was prepared by an ad hoc Agency
Task Force on Environmental Regulatory Modeling (ATFERM), which was created
under the Agency's Risk Assessment Council.
The MPRS found that, in general the guidance provides an appropriate
level of detail in the guidelines for specific elements to be addressed by the
reviewer, but that more detailed guidance is needed on the mechanics of the
review process. The MPRS cited an omission as to what should happen after the
external peer review process is completed, in order to address issues raised in the
peer review. However, the Agency was urged not to become overly prescriptive, as
to the specific details of the model peer review where the guidance could become
de facto regulation.
The MPRS stressed the importance of documenting the entire peer review
process. Among the recommendations made, the MPRS also stressed that specific
guidance was needed on what constitutes a "qualified" peer reviewer, that the
model should be verified against available data over the range of conditions of
interest, that there is a role for peer review to insure that model results are not
misused and that the decision is consistent with the modeling effort. The MPRS
also recommended that some form of the ATFERM should continue to exist, that a
•model-coordinating activity within the Agency is needed on a continuing basis, and
that the Agency should consider making the peer-review guidance into policy.
Key Words: Environmental Regulatory Modeling, External Peer Review, Models,
Peer Review,
-------
DISTRIBUTION. LIST
Deputy Administrator
Director, Office of Cooperative Environmental Management
Assist Administrators
EPA Regional Administrators
EPA Laboratory Directors
Deputy Assistant Administrator for Air and Radiation
Director, Office of Air Quality Planning and Standards (OAQPS)
Director, Office of Atmospheric Programs
Director, Center for Environmental Research Information (CERI)
Director, Office of Radiation and Indoor Air (ORIA)
Deputy Assistant Administrator for Policy, Planning and Evaluation (OPPE)
Director, Office of Policy Analysis (OPA)
Director, Office of Regulatory Management and Evaluation (ORME)
Director, Office of Strategic Planning and Environmental Data (OSPED)
Deputy Assistant Administrator for Office of Prevention, Pesticides and Toxic
Substances
Director, Office of Pollution Prevention and Toxics (OPPT)
Deputy Assistant Administrator for Office of Research and Development (ORD)
Director, Office of Environmental Processes and Effects Research (OEPER)
Director, Office of Environmental Engineering and Technology
Demonstration (OEETD)
Deputy Director, OEETD
Director, Office of Monitoring, Modeling and Quality Assurance (OMMSQA)
Director, Office of Technology Transfer and Regulatory Support (OTTRS)
Deputy Assist Administrator for Office of Solid Waste and Emergency Response
(OSWER)
Director, Office of Emergency and Remedial Response (OERR)
Deputy Director, OERR
Director, Office of Solid Waste (OSW)
Deputy Director, OSW
Deputy Assistant Administrator for Water
Director, Office of Science and Technology (OST
Director, Office of Wetlands, Oceans and Watersheds (OWOW)
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
------- |