UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
EPA-SAB-EEC-89-012
OFFICE OF
January 13, 1989 THE ADMINISTRATOR
Honorable Lee M. Thomas
Administrator
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, D.C. 20460
Dear Mr. Thomas:
The Environmental Engineering Committee of the Science
Advisory Board has prepared the attached resolution for your
consideration on the use of mathematical modeling for regulatory
assessment and decision-making. This is the second time the
Science Advisory Board has acted on the issue of modeling; a 1984
letter called the Agency's attention to this important concern.
Over the last few years the Environmental Engineering
Committee has reviewed a number of EPA environmental modeling
studies. In doing so, the Committee has noted a number of
problems in the development and implementation of models within
the Agency that were common to modeling efforts sponsored by a
variety of offices. The Committee believed that these common
problems would be best called to the Agency's attention through a
more general resolution on modeling.
Drafts of the resolution were presented and widely discussed
at a series of Committee and Executive Committee meetings during
1988. For instance, an earlier draft of the resolution was
quoted at length in the Radiation Advisory Committee's recent
report on the sources and transport of radionuclides. While
encouraging the overall approach of quantitative risk assessment
and modeling for environmental decision-making, this Committee
noted a number of common problems in the use of models by the
Agency. The following items summarize the main points that are
addressed in the attached resolution:
1. There should be a better balance between field and
laboratory data collection efforts and modeling analysis
for effective environmental assessment;
2. Models for regulatory assessment and decision-making
which incorporate state-of-the-art scientific under-
standing of the environmental processes involved should
be developed and used;
3. There should be better confirmation of models with
laboratory and field data;
-------
4 . Sensitivity and uncertainty analysis of environmental
models and their predictions should be conducted to
understand level of confidence in model predictions, as
well as to identify key areas of future study;
5 . An Agency-wide task-group to assess and guide model use
by EPA should be formed;
6. There should be an increased effort to hire and support
engineers and scientists with modeling development and
application skills;
7 . There is a need for systematic management of model use
within EPA and a careful review of emerging technologies
such as personal computer-based models and expert
systems; and
8 . Peer review at various levels should be coordinated to
ensure proper development and application of models.
The resolution identifies a number of ways in which the
use of models by the EPA can be improved. The Committee
believes that successful implementation of these recommendations
will require the establishment of a formal institutional
mechanism with responsibility for review, oversight and,
coordination of model use in EPA.
We are pleased to have had the opportunity to be of service
to the Agency, and look forward to your response on this issue.
Sincerely,
c,
Raymond C. Loehr, Chairman
Executive Committee
Science Advisory Board
Richard A. Conway, Chairman
Environmental Engineering Committee
Science Advisory Board
Mitchell J. Small, ^Chairman
Modeling Resolution Subcommittee
Science Advisory Board
Attachment: Modeling Resolution
cc: John A. Moore
Donald G. Barnes
-------
EPA-SAB-EEC-89-012
RESOLUTION ON THE USE
OF MATHEMATICAL MODELS BY EPA
FOR REGULATORY ASSESSMENT AND DECISION-MAKING
by the
Environmental Engineering Committee
Science Advisory Board
U.S. Environmental Protection Agency
January 1989
-------
1
INTRODUCTION
The use of mathematical models for environmental decision making has increased significantly in
recent years. The reasons for this are many, including scientific advances in the understanding of certain
environmental processes, the wide availability of computational resources, the increased number of
scientists and engineers trained in mathematical formulation and solution techniques, and a general
recognition of the power and potential benefits of quantitative assessment methods.
Within the US Environmental Protection Agency (EPA) environmental models which integrate
release, transport, fate, ecological effects and human exposure are being used for rule making decisions
and regulatory impact assessments. This report is directed to the development and validation of such
models, an issue which was first addressed in December 1984 by Norton Nelson, Chairman of the
Executive Committee of the SAB. In a letter to the EPA Administrator, William Ruckelshaus, it was
recommended that a systematic effort of model validation be initiated, including an identification of the
appropriate balance between monitoring and modeling. It was further recommended that the relative
utility of exposure modeling approaches be evaluated in the form of case studies in various media
including model validation and uncertainty analysis.
The Environmental Engineering Committee reaffirms and amplifies these recommendations, based
on review of a number of integrated environmental modeling studies during the past few years. Examples
include a review of the report, "Comparison of Risks and Costs of Hazardous Waste Alternatives:
Methods Development and Pilot Studies" (SAB-EEC Report, July 1985); a review of the Code for
Transport in the Unsaturated Zone (FECTUZ) and its potential use for determining whether a waste is
hazardous for listing decisions (SAB-EEC-88-030); a review of risk-based regulations for alternative
disposal and reuse options for sewage sludge (SAB-EEC-87-013, SAB-EEC-87-015); a review of the
Underground Storage Tank (UST) Release Simulation Model (SAB-EEC-88-029); and a review of the
draft risk screening analysis for mining wastes (SAB-EEC-88-028). While encouraging the overall
approach of modeling for environmental decision-making by EPA and acknowledging the progress made
by various offices within the Agency, the committee noted a number of problems in the development and
application of models, including: the increased reliance on models rather than background data collection
and analysis, an inadequate level of laboratory and field validation for models employed, a lack of studies
quantifying the uncertainties associated with model predictions, and concurrently, the potential misuse of
particular uncertainty analysis techniques. The following resolutions address these issues, and identify
the need for an institutional mechanism within EPA to ensure their implementation.
-------
RESOLUTIONS
1. A balanced program of field and laboratory data collection and modeling analysis is
required for effective environmental assessment.
The realistic characterization of an environmental problem requires the collection of laboratory and
field data - the more complex the problem, the more extensive and in-depth are the required studies. In
some cases involving more complex issues, future projections of environmental effects, larger
geophysical regimes, inter-media transfers, or subtle ecological effects, mathematical models of the
phenomena provide an essential element of the analysis and understanding. However, the models
cannot stand alone; adequate data are required. Indeed, a major function of mathematical models is as a
tool to design field studies, interpret the data and generalize the results.
A number of recent studies of integrated exposure and risk reviewed by the committee have
exhibited an over-reliance on models at the expense of the acquisition of needed data. This trend should
be reversed.
2. Mathematical models for regulatory assessment and decision-making should
Incorporate, to the extent possible, the state-of-the-art scientific understanding of the
environmental problem.
Mathematical models should ideally be based on a fundamental representation of the physical,
chemical and biological processes affecting environmental systems. In the regulatory domain, there may
be a need to sacrifice model complexity and rigor because of inadequate process insight, the need for
computational efficiency, or because of a lack in available supporting data. There should not, however,
be too ready a willingness to abandon fundamental, scientific approaches simply because the required
research and data are too difficult to obtain in a short time-span. If this were done, two undesirable
results would likely occur. First, an improperly formulated model can lead to serious misjudgements
concerning environmental impacts and the effectiveness of proposed regulations. In this regard, a bad
model can be worse than no model at all. Second, by accepting an improperly formulated model, the use
of a weak scientific approach can become institutionalized within the Agency, and the opportunity to
motivate the needed research and data collection can be lost. Rather, shortcomings in process
understanding and available data should serve as an incentive for research and data acquisition to
improve the foundations for models.
-------
It must be recognized that research and data acquisition to support state-of-the-art model
development and validation is a long-term, iterative process involving many scientific and engineering
disciplines. A commensurate, long-term commitment to support this effort is required from the Agency.
3. There is a need for models used in regulatory applications to be confirmed with
laboratory and field data.
There are a number of steps needed to confirm the accuracy and utility of an environmental model.
As a preliminary step, the elements of the basic equations and the computational procedures employed to
solve them should be tested to ensure that the model generates results consistent with its underlying
theory. The confirmed model should then be calibrated with field data and subsequently validated with
additional data collected under varying environmental conditions. After the particular regulatory program
has been implemented, field surveys and long-term monitoring should be conducted for comparison with
model projections. The stepwise procedure of checking the numerical consistency of a model, followed
by field calibration, validation and a posteriori evaluation should be an established protocol for
environmental quality models in all media, recognizing that the particular implementation of this may
differ, for surface water, air and ground water quality models. It is also recognized that the degree and
extent to which the process of validation is conducted for a model depends on the significance of the
environmental issue and the consequence of an erroneous decision concerning the problem.
It is recommended that EPA establish a general model validation protocol and provide sufficient
resources to test and confirm models with appropriate field and laboratory data.
4. Sensitivity and uncertainty analysis of environmental models and their predictions
should be performed to provide decision-makers with an understanding of the level of confidence
In model results, and to Identify key areas for future study.
A number of methods have been developed in recent years for quantifying and interpreting the
sensitivity and uncertainty of models. These methods require careful application, as experience with
uncertainty analysis techniques is somewhat limited, and there is a significant potential for misuse of the
procedures and misinterpretation of the results. Potential problems include the tendency to confuse
model uncertainty with temporal or spatial variation in environmental systems, the tendency to rely on
model uncertainty analysis as a low-cost substitute for actual scientific research, and the tendency to
ignore important uncertainties in model structure when evaluating uncertainties in model parameters. To
address the latter issue, sensitivity analysis of a broader nature is required, considering the impact of
-------
alternative model assumptions and omitted processes. As is the case for model validation, the extent to
which sensitivity and uncertainty analysis should be performed depends upon the importance of the
environmental issue and the relative role of the model in determining the regulatory decision.
Consideration of model sensitivity and uncertainty should be included in all modeling studies. The
implications of errors in model structure, as well as errors in model parameters, should be evaluated to
determine possible effects on the ultimate regulatory decision.
5. There Is a need for a central coordinating group within the EPA to assess the status of
environmental models currently used or proposed for use In regulatory assessment, and to
provide guidance In model selection and use by others in the Agency.
In the selection of mathematical models for regulatory applications, a thorough understanding of
the capabilities, limitations and degree of validation of available models is required. There have been
instances where a model developed for a particular purpose was used in a new application without the
appropriate steps taken to properly adapt and validate the model in the new problem setting. Conversely,
there are cases where available computational programs for models have been ignored and new, but
similar, procedures developed at unnecessary effort and expense. Recognizing the need for improved
model selection and use, the Sources, Fate and Transport Subcommittee of the SAB Research
Strategies Committee (SAB-EC-88-040, SAB-EC-88-040A) recommended that EPA formalize
mechanisms for review and acceptance of environmental models for all media. Methods such as those
used by the EPA Office of Air Quality Planning and Standards (EPA-450/2-78-027R) were recommended.
This would involve identifying tested or recommended models for particular media or environmental
settings, establishing procedures for demonstrating the acceptability of alternative models, and instituting
a Model Clearinghouse to compile and test models, conduct periodic workshops to ensure consistency in
modeling guidance, and promote the use of the most appropriate models and data bases. The
Subcommittee also indicated the need to identify currently applied models where improved validation is
needed, and to develop a priority list for these validation efforts.
To address the issues of model validation and model selection and use within the EPA, we
suggest the establishment of a task-group on mathematical models for environmental quality assessment.
Such a group would evaluate the state-of-the-art of models in each of the media, as well as emerging
multi-media models, evaluate environmental models used by other government agencies, and provide
oversight for model development, validation and application within the EPA. The group would also rank
current models as to their relative importance and need for further validation studies. This Agency-wide
task-group should be established as soon as possible.
-------
6. EPA must him and support engineers and scientists with appropriate model
development and application skills.
This issue is closely linked to the recommendation of the SAB Research Strategies Committee that
EPA increase the numbers and sharpen the skills of the scientists and engineers who conduct
environmental research (SAB-EC-88-080). Modeling is not a separate discipline, rather it is a particular
skill that is part of the overall environmental science and engineering approach to problem-solving. There
has been a tendency to allocate the development of models to the computer specialist, who frequently
lacks the understanding of the basic equations and their significance to the environmental problem.
Similarly, there has been a tendency to presume that the users of models need not understand the basis
for the models. This is incorrect. The proper development and application of models requires engineers
and scientists trained in the fundamental principles of the environmental transport problem and
computational methods, so that they can develop and work with the model in an informed manner, not
just as a black box which is manipulated to obtain numerical output. Note that often the most critical and
effective application of models is made by users not involved in the development of the model, as they
are more likely to question and challenge the implicit perspectives and assumptions of the model
approach. As such, skilled model developers and model users are both required for effective problem-
solving.
The Agency should increase its efforts to hire and retain engineers and scientists who are qualified
in the area of model development and model use, having both broad and problem-specific skills. The
EPA should support their efforts through the program of the Agency task-force on modeling discussed in
the previous resolution.
7. The need for a systematic management of model use within the EPA is heightened by the
introduction of new computer systems and modeling technologies.
The wide availability of personal computers has brought increasing numbers of models to an
increasing number of potential users. As a result, the problem of ensuring code validity and proper model
use is that much more difficult. Special challenges are also raised by the growing technology of
knowledge-based expert systems. Expert systems allow the automation of a wide range of scientific
analysis and inference, and are currently being developed for a variety of environmental engineering
-------
problems. EPA should require strict review and critique of expert systems, recognizing that they can, in
many ways, be treated like other environmental models. They are tools to aid the decision-maker, they
must be rigorously confirmed with field data prior to regulatory application, they require a careful
consideration of model sensitivity and uncertainty, and they require trained users familiar with both the
fundamental physical principles of the environmental system being considered and the way in which the
expert system uses this information to arrive at its recommendation for a design or regulatory decision.
The recommended EPA task group on modeling should pay particular attention to emerging
technologies, such as personal computer-based models and expert systems. Careful review, oversight
and validation are needed for these beneficial, but relatively untested approaches to environmental
modeling.
8. Peer review at various levels is required to ensure proper model development and
application.
Peer review is an essential element of all scientific studies, including modeling applications. Peer
review is appropriate in varying degrees and forms at different stages of the model development and
application process. The basic scientific representation incorporated in the model should be based on
formulations which have been presented in the peer reviewed scientific literature. Ideally, the model itself
and initial test applications should also be presented in peer-reviewed papers. However, this is not
always possible given the pace of scientific development and regulatory need. Peer review panels are
thus often required to review the scientific capabilities of proposed models and their intended
applications. These expert panels should include some combination of internal Agency staff and outside
experts. Innovative approaches to model review should be considered, such as the use of "round robin'
reviews in which the same modeling task is addressed by a number of independent groups, or the use of
benchmark data-sets for testing model accuracy.
The recommended EPA task group on modeling should identify the needs for peer review of
models and establish procedures for coordinating the necessary peer review panels.
SUMMARY
The resolutions presented in this report address critical issues that must be confronted to improve
the use of models by the EPA. These issues include the need for a better balance between data
collection and modeling, the use of state-of-the-art models, the need for model confirmation and
-------
sensitivity and uncertainty analysis, the need for a central coordinating group to provide oversight and
guidance on model use within the Agency, the need for more scientists and engineers with modeling
skills, the need for review of new modeling technologies, and the need for peer review of model
development and application. Many of the recommendations in this report can be implemented by
individuals and individual offices within the EPA. This will undoubtedly lead to a more effective use of
models by the Agency. The Committee believes, however, that a full and successful response to these
resolutions will require the establishment of a formal, institutional mechanism which can promote better
review and coordination of model use throughout the EPA. The actual structure of this group and its
relationship to previous or ongoing initiatives is an issue that requires further consideration by the
Agency. It is hoped that these resolutions will provide further motivation and direction for this effort.
------- |