United States
Environmental Protection
Agency
Robert S. Kerr Environmental
Research Laboratory
Ada OK 74820
Research and Development
EPA/600/SR-93/011 May 1993
Project Summary
Quality Assurance and Quality
Control in the Development and
Application of Ground-water
Models
Paul K.M. van der Heijde and Osman A. Elnawawy
This report describes quality assur-
ance and code testing in ground-water
modeling. The quality assurance pro-
cedures presented cover both develop-
ment and application of ground-water
modeling codes. An important part of
quality assurance is code testing and
performance evaluation. The section on
code testing and performance evalua-
tion discusses past efforts to test
ground-water simulation codes and
document their performance and pre-
sents the three-level testing procedure
developed by the International Ground
Water Modeling Center and the Center's
approach to developing benchmarks for
the first two test levels.
This Project Summary was devel-
oped by EPA's Robert S. Kerr Environ-
mental Research Laboratory, Ada, OK,
to announce key findings of the research
project that is fully documented in a
separate report of the same title (see
Project Report ordering information at
the back).
Introduction
Ground-water modeling has become an
important methodology in support of the
planning and decision-making processes
involved in ground-water management.
The effective application of computer simu-
lation codes in modeling field problems is
a qualitative procedure, a combination of
science and art. A successful model ap-
plication requires a combination of knowl-
edge of scientific principles, mathematical
methods, and site characterization paired
with expert insight in the modeling pro-
cess, often to be provided within the frame-
work of a multi-disciplinary team effort. As
participants at the workshop on "Modeling
for Water Management" organized by the
European Institute for Water (Como, Italy,
May 21-22,1987) formulated: "Modeling
imposes discipline by forcing all concerned
to be explicit on goals, criteria, constraints,
relevant processes, and parameter val-
ues."
Ground-water models provide an ana-
lytical framework for obtaining an under-
standing of the mechanisms and controls
of ground-water systems and the pro-
cesses that influence their quality, espe-
cially those caused by human intervention
in such systems. For managers of water
resources, models may provide essential
support for planning and screening of al-
ternative policies, regulations, and engi-
neering designs affecting groundwater.
This is particularly evident with respect to
ground-water resources development,
ground-water protection, and aquifer res-
toration.
In discussing ground-water modeling,
distinction should be made between model
development and model application. Model
development consists of three compo-
nents: (I) research aimed at obtaining a
quantitative understanding of the studied
ground-water system; (2) software devel-
opment; and (3) model testing and evalu-
ation. Often, model development, and
particularly code development, is driven
by immediate and long-term needs of
ground-water resources management.
Model application is part of a larger set of
activities aimed at solving site- or prob-
lem-specific issues and includes such ac-
tivities as data collection, interpretation and
Printed on Recycled Paper
-------
storage, system conceptualization and
model design, formulation of alternative
problem solving scenarios and engineer-
ing designs, and post-simulation analysis.
Although a consensus may exist as to
what ground-water modeling entails, the
definition of a "model" per se is somewhat
nebulous. In hydrogeology, the term
"ground-water model" has become syn-
onymous with conceptual ground-water
models, mathematical ground-water mod-
els (including analytical and numerical
models), computer models, and simula-
tion models. Furthermore, the term
"ground-water model" may apply either to
a computer code without site-specific data
or to the representation of a site-specific
system using such a generic code, to-
gether with pertinent data.
In the full report a ground-water model
is defined as a non-unique, simplified,
mathematical description of the subsur-
face component of a local or regional hy-
drologic system, coded in a computer
programming language, together with a
quantification of the simulated system in
the form of boundary conditions, system
and process parameters, and system
stresses. The generalized computer code
usable for different site- or problem-spe-
cific simulations is referred to as a (com-
puter) simulation code or a generic
simulation model. A ground-water model-
ing study is defined as the development
and use of a ground-water model (i.e.,
code and data) to solve specific ground-
water management problems. Sometimes,
such a ground-water model is the result
of the application of one or more simula-
tion codes to a generalized ground-wa-
ter management problem; e.g., in support
of promulgating government-mandated
regulations. Generalizing such a man-
agement problem may be based on the
use of concepts and data describing an
"average" or "hypothetical" site represent-
ing targeted sites.
Sometimes a model is described in
terms of the mathematical solution tech-
nique employed. Most commonly used
terms are "analytical model," "semi-ana-
lytical model," and "numerical model." An
analytical model is a model in which the
solution of the mathematical problem (gov-
erning equation and boundary conditions)
results in a closed-form or analytical ex-
pression for the state variable, continuous
in the space and time domains. In a nu-
merical model a solution for the math-
ematical problem is found, discrete in both
the space and time domains, by using
numerical approximations of the govern-
ing partial differential equation(s). In a
semianalytical model complex analytical
solutions are approximated by numerical
techniques, resulting in a discrete solution
in either the space or time domain.
Developing efficient and reliable soft-
ware and applying such tools in ground-
water management requires a number of
steps, each of which should be taken con-
scientiously and reviewed carefully. Tak-
ing a systematic, well defined and
controlled approach to all steps of the
model development and application pro-
cess is essential for its successful utiliza-
tion in management. Quality Assurance
(QA) provides the mechanisms and frame-
work to ensure that decisions are based
on the best available data and (modeling-
based) analyses.
Sections in the full report provide back-
ground information on quality assurance
and define the role of QA in ground-water
modeling. They present a functional and
practical quality-assurance methodology,
written from the perspective of the model
user and the decision-maker in need of
technical information on which to base
decisions. An important part of quality as-
surance is code testing and performance
evaluation. The section on code testing
and performance evaluation presents the
three-level testing procedure developed by
the International Ground Water Modeling
Center, the development of test problems
and related benchmarks for the first two
test levels, and a discussion of the imple-
mentation of the testing procedure.
Quality Assurance in Ground-
water Modeling
Quality assurance in ground-water mod-
eling is the procedural and operational
framework put in place by the organiza-
tion managing the modeling study, to as-
sure technically and scientifically adequate
execution of all project tasks included in
the study, and to assure that all modeling-
based analysis is verifiable and defen-
sible. QA in ground-water modeling is
crucial to both model development and
model use and should be an integral part
of project planning and be applied to all
phases of the modeling process.
The two major elements of quality as-
surance are quality control (QC) and qual-
ity assessment. Quality control refers to
the procedures that ensure the quality of
the final product. These procedures in-
clude the use of appropriate methodology
in developing and applying computer simu-
lation codes, adequate verification and
validation procedures, and proper usage
of the selected methods and codes. To
monitor the quality control procedures and
to evaluate the quality of the studies, qual-
ity assessment is applied. Each project
should have a quality assurance plan (QA
plan), listing the measures planned to
achieve the project's quality objectives.
"Quality assurance" is a term used in
many different disciplines and environ-
ments. Its meaning and implementation
differs from field to field. For example,
there is a significant difference between
QA in software engineering, software qual-
ity assurance (SQA), and QA in industrial
production. Also, there are significant dif-
ferences between data QA and software
QA procedures.
Literally, quality assurance assures the
quality of the product (code, model) or
activity of concern (modeling). A more
workable description is that QA (in model-
ing) guarantees that the quality of the
model-based analysis and advice (to de-
cision-makers) satisfies quantitative qual-
ity criteria or measures. As the principal
idea behind QA is accountability, and the
main mechanism is maintaining records
(hard copy and electronic files, reports) of
all activities and results, a more proper
term might be quality documentation.
Taken in a broad sense, QA provides a
methodological and administrative frame-
work to do the best we can within the
limitations of our current understanding of
nature and available technology.
That QA always assures acceptable
quality of a code development project or a
modeling study is an idle hope. However,
adequate QA can provide safeguards
against faulty codes or improper model-
ing. Regulators and decision-makers
should understand that there is no way to
guarantee that modeling-based advice is
entirely correct, nor that the simulation
code used (or any scientific model or
theory, for that matter) can ever be proven,
verified or validated in the strictest sense
of these terms. Rather, a model can only
be invalidated by disagreement of its pre-
dictions with independently derived obser-
vations regarding real systems.
It should be noted that a major role of
QA/QC is to provide communication be-
tween the modeler and his/her peers, and
between modeler and decision-maker, giv-
ing the latter a sense of the accuracy,
uncertainty, and reliability of the modeler's
advice. Therefore, QA should not apply to
the work of junior modelers only, but
should also be adhered to by expert mod-
elers.
There are various cautions to be made.
QA should never become so stifling that
experienced modelers are discouraged to
take new avenues not previously explored,
or that an inappropriately large part of the
budget of a project is consumed by re-
sponding to bureaucratic requirements.
-------
When QA regulations become bureaucratic
red tape, the time and cost of QA may
take away precious resources from the
data collection and problem analysis ac-
tivities. Furthermore, the risk is present
that QA deteriorates and becomes only a
checklist installing false confidence in mod-
eling results.
Code Testing and Evaluation
Procedures
The usefulness of predictive simulations
based on ground-water models is often
limited by our inability to indicate and quan-
tify the reliability of such model results.
Researchers have developed various tech-
niques to assess confidence levels for
model predictions, so that water resources
managers can account for uncertainties in
the decision-making process. For example,
several investigators present a methodol-
ogy based on the application of decision
analysis to engineering design in a
hydrogeological environment. The meth-
odology involves the coupling of a deci-
sion model based on a risk-cost-benefit
objective function, a simulation model for
ground-water flow and contaminant trans-
port, and an uncertainty model that en-
compasses both geological uncertainty and
parameter uncertainty.
One area of concern is the credibility of
the simulation codes used and the ge-
neric models they represent. As discussed
in the full report, an important aspect of
the credibility of a simulation code is its
reliability. The reliability of codes is estab-
lished by applying a comprehensive, sys-
tematic review and testing procedure. The
quality assurance aspects of such a pro-
cedure have been discussed in Section
2.2 of the full report. Another section pre-
sents a systematic code verification and
performance testing protocol, based on
the use of analytical solutions and syn-
thetic data sets as benchmarks. Although
the full report provides some example test
problems, it does not contain actual bench-
marks. A comprehensive set of bench-
marks for two- and three-dimensional
ground-water flow and transport models
will be presented in a follow-up report.
Conclusions
There is an urgent need for compre-
hensive, systematic testing of all types of
ground-water models and for the estab-
lishment of a verification and validation
protocol. Ground-water management de-
cisions should be based on the use of
technically and scientifically sound meth-
ods of data collection, information pro-
cessing, and interpretation. Because few
experimental investigations have tested
multidimensional theories, conceptualiza-
tion, and associated computer codes, it is
extremely important to conduct further re-
search aimed at developing and execut-
ing verification and validation studies for
prominent ground-water models. It may
be argued that from a ground-water man-
agement point of view further efforts should
be directed towards model testing studies
rather than toward the development of
more complex models.
In recent years, the International Ground
Water Modeling Center has developed a
testing procedure and methodology for
model evaluation as part of its efforts to
implement a comprehensive quality as-
surance program. The current project at-
tempts to systematically analyze the
scientific considerations and collect the
technical elements for implementation of
such a methodology. The next step is the
application of this comprehensive meth-
odology to actual computer codes.
'U.S. Government Printing Office: 1993 — 750-071/60244
-------
Paul K.M. van der Heijde is with the Colorado School of Mines, Golden, CO 80401;
OsmanA. Elnawawyis with Indiana University/Purdue University at Indianapolis,
Indianapolis, IN 46204.
Joseph R. Williams is the EPA Project Officer (see below).
The complete report, entitled "Quality Assurance and Quality Control in the
Development and Application of Ground-water Models," (Order No. PB93-
178226; Cost: $19.50; subject to change) will be available only from
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at
Robert S. Kerr Environmental Research Laboratory
U.S. Environmental Protection Agency
Ada, OK 74820
United States
Environmental Protection Agency
Center for Environmental Research Information
Cincinnati, OH 45268
Official Business
Penalty for Private Use
$300
BULK RATE
POSTAGE & FEES PAID
EPA
PERMIT No. G-35
EPA/600/SR-93/011
------- |