United States Science Advisory EPA-SAB-EE&3WJ13
Environmental Board (A-1Q1) J« 1»3
&EPA AN SAB REPORT:
REVIEW OF THE
ASSESSMENT
FRAMEWORK FOR
GROUND-WATER MODEL
APPLICATIONS
REVIEW OF THE OFFICE OF
SOLID WASTE AND EMERGENCY
RESPONSE ASSESSMENT
FRAMEWORK FOR GROUND-
WATER MODEL APPLICATIONS
-------
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C, £0460
OFFICE OF THE ADMINISTRATOR
SCIENCE ADVISORY BOARD
EPA-SAB-EEC-93-013
June 21, 1993
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, D.C, 20460
Subject: Science Advisory Board Review of the OSWER Assessment
Framework for Ground-Water Model Applications
Dear Ms. Browner:
The Science Advisory Board (SAB) has completed its review of the Office of
Solid Waste and Emergency Response (OSWER) draft guidance, "Assessment
Framework for Ground-Water Model Applications," dated October 1992 (hereafter
cited as the Framework). The document was addressed at a meeting of the
Modeling Project Subcommittee (MPS) of the Environmental Engineering
Committee (EEC) in Arlington, VA on January 14-15, 1993. Technical
presentations were given by OSWER's Information Management Staff (IMS), their
contractors and consultants, and EPA Region III and V technical staff,
Subcommittee member and consultant review comments were verbally conveyed to
these participants at the close of the meeting.
In accordance with the "charge to the Subcommittee," the MPS review
focused on the scientific underpmninp and completeness of the substance of the
Framework; whether the Framework provides help from the project management
perspective for managing model applications; whether the use of the Framework
will serve to aid the OSWER staff in improving the management of its modeling
activities; and whether there is additional information or direction which should be
added to improve the Framework, especially with regard to project management
requirements for the modeling team, and to the modeling code and public domain
issues,
The OSWER IMS are to be commended for their thoughtful approach to the
development of the Framework, The MPS strongly supports this effort and
encourages the extension of this framework, or the development of other
frameworks, for additional types of model applications. The Framework represents
-------
a significant advance in QSWEE's approach to the management and use of
mathematical models in Superfund remediation planning and hazardous waste
programs of the Office of Solid Waste (OSW).
Overall, the Framework is based on sound science and its developers have
appreciated the need for a comprehensive and flexible document. The modeling
philosophy represented in the Framework is reasonable and the language and level
of detail are generally suitable for the intended users. In some of the sections,
however, the Framework contains too much detail which may tend to reduce the
general applicability of the guidance and potentially foster its misuse as a
document to supplant experienced professional judgement.
The MPS believes that this Framework will be a useful tool for OSWEE in
the management and assessment of modeling applications. It will, in fact, assist in
providing a basis for common agreement among diverse users as to which
components are likely to constitute a good modeling study, and it could help to
improve the level of awareness and quality of modeling applications. In this
effort, OSWBE has responded positively to past guidance from the SAB/EEC and
is to be commended for its early involvement of Agency Regional Office personnel
in the development of the Framework.
The MPS is concerned, however, that the Framework could be
inappropriately formalized as a required checklist to be used as the basis for
accepting or evaluating the quality or appropriateness of a particular modeling
study. We stress that modeling is not a linear process; that it is an iterative,
evolutionary approach to the refinement of our understanding of a physical
system, and a tool for the application of this understanding to the prediction of
system response. Clearly, no guidance document is a substitute for modeling
education and experience, The Framework should not be used to promote
modeling by inexperienced people, nor be relied upon to supplant experienced
professional judgement or measurement. The MPS has made a number of
recommendations in the report which follows to reduce the likelihood that the
Framework will be misinterpreted or misused, and to enhance its value to
managers as a guidance document.
While many useful EPA publications related to ground-water are contained
in the Framework, it appears that a number of related documents were overlooked
in its development. A number of specific documents were cited by the MPS to be
consulted in the revision of the Framework and incorporated in its reference list.
The MPS was not asked to review the bound document entitled "Ground-
Water Modeling Compendium," and was only able to inspect it in a cursory
manner, as this document was not distributed prior to tlie meeting of January 14,
1993. Because the Framework represents a rather broad overview of modeling
practice, it seems likely that the user-community receiving the Compendium will
interpret the Framework in terms of the example model application descriptions
and assessments embodied in the Compendium. It is observed by the MPS that
-------
the modeling studies presented In the Compendium predate the Framework and
may not be illustrative of Its reasoned application. Therefore, to avoid any
potential interpretational pitfalls, the MPS recommends the publication of a
revised Framework as a separate document, and OSWER staff should be
encouraged to seek out good modeling application case histories to illustrate
applicable points covered in the Framework.
To avoid Framework user misconceptions about the predictive accuracy of
models, the MPS recommends that the document language be refined to clarify the
precise meaning of phrases, such as "required level of confidence," "performance
target," "quality assurance," and "model verification."
A number of specific concerns and suggestions, along with suggested
language revisions are offered by the MPS, With appropriate revisions as
suggested, the MPS believes that the Framework will become a useful document
for management of ground-water modeling applications within the Agency.
We appreciate the opportunity to conduct this review and look forward to
your response to the scientific advice contained herein, and in particular to the
items stressed in this forwarding letter to you.
Sincerely,
Dr. Kaymond C, Loehr, Chair
Executive Committee
Science Advisory Board
Mr. Richard A, Conway, Chair
Environmental Engineering Committee
Science Advisory Board
Dr. Linda M. Abriola, Chair
Modeling Project Subcommittee
Environmental Engineering Committee
Science Advisory Board
-------
-------
NOTICE
This report has been written as a part of the activities of the Science
Advisory Board, a public advisory group providing extramural scientific
information and advice to the Administrator and other officials of the
Environmental Protection Agency. The Board is structured to provide a balanced,
expert assessment of scientific matters related to problems facing the Agency.
This report has not been reviewed for approval by the Agency, hence, the
comments of this report do not necessarily represent the views and policies of the
Environmental Protection Agency or of other federal agencies. Any mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.
-------
-------
ABSTRACT
The Modeling Project Subcommittee (MPS) of the Environmental
Engineering Committee (EEC) of the Environmental Protection Agency's (EPA)
Science Advisory Board (SAB) reviewed the Agency's Office of Solid Waste and
Emergency Response (OSWER) draft guidance entitled "Assessment Framework for
Ground-Water Model Applications," dated October, 1992. (hereafter cited as the
Framework). The MPS met on January 14 and 15, 1993 and reviewed the
Framework,
In accordance with its charge, the MPS review focused on the scientific
correctness and completeness of the substance of the Framework; whether the
Framework provides guidance to OSWER for managing model applications; and
whether there is additional information or direction which should be added to
improve the Framework.
The MPS strongly supports this effort and encourages the extension of this
Famework, or the development of other frameworks, for additional types of model
applications. The Framework represents a significant advance in OSWER's
approach to the management and use of mathematical models in Superfund
remediation planning.
In order to improve the usefulness and flexibility of the Framework and to
reduce the likelihood that the Framework be used inappropriately, the MPS
suggests the following: that the introduction be expanded to clarify intended users
and uses; that graphic illustrations be added to emphasize feedback involved in the
modeling process; that the Framework be distributed as a separate document; and
that additional references be consulted in its revision. Particular language
revisions are also recommended to improve clarity and comprehensiveness,
With revision, the MPS believes that the Framework will be a useful
guidance document for OSWER management of ground-water model applications.
Key Words: Assessment Framework, , Assessment Framework for Ground-Water
Model Application, Ground-Water Models, Hazardous Waste, Superfund
-------
-------
MODELING PROJECT SUBCOMMITTEE
ENVIRONMENTAL ENGINEERING COMMITTEE
of the
SCIENCE ADVISORY BOARD
CHAIR
Dr. Linda M. Abriola, Associate Professor, Department of Civil and Environmental
Engineering, University of Michigan, Ann Arbor, Michigan
MEMBERS AND CONSULTANTS
Dr. Calvin C. Chien, Principal Consultant, duPont Corporate Remediation Group,
E.L duPont de Nemours & Company, Wilmington, Delaware
Dr. Leonard F. Konikow, (Federal Liaison), U.S. Geological Survey, Water
Resources Division, Reston, VA
Dr. James W. Mercer, President, GeoTrans, Inc., Sterling, VA
Dr. Paul V. Roberts, Professor of Environmental Engineering, Department of Civil
Engineering, Stanford University, Stanford, CA
Dr. Mitchell J. Small, Professor, Departments of Civil Engineering and
Engineering and Public Policy, Carnegie-Mellon University, Pittsburgh, PA
Science Advisory Board Staff
Dr. K Jack Kooyoorajian, Designated Federal Official, U.S. EPA, Science Advisory
Board (A101-F), 401 M Street, S.W, Washington, D.C, 20460
Mis. Diana L. Pozun, Staff Secretary
Dr. Donald G. Barnes, Staff Director
in
-------
-------
TABLE OF CONTENTS
1. EXECUTIVE SUMMARY .......... . ---- ...... ...... ......... 1
2, INTRODUCTION ........... ............. ....... ...... ... - 3
2.1 General Assessment ..... ........... ..... ....... ...... «
3. BECOMMENDATIQNS ....... . .......... ......... .......
3.1 Expansion Of The Framework Introduction ........ ......... a
3,2 Graphic Representation Of Concepts ...... ..... . ...... > > «
3,3 Balance Of Treatment ..... ..... . . .......... .......... 7
3.4 Reference To Previous Work ....... »,.....,.... ........ - °
3.5 Interrelationship Of The Framework and Its Compendium ....... »
3.6 Clarification Of Terminology , ......... ..... ............. 9
3.7 Sections 14 and 39 ............ .................... iy
4. CONCLUSIONS ---- . ......... . . ....... ....... ---- ll
APPENDIX A - DETAILED TECHNICAL COMMENTS
,,,,.,,*,,..-.«**--- *\.*JL
APPENDIX B - THE CHARGE ...................... ...... . , - - B-l
APPENDIX C - REFERENCES CITED ... ..... . ..... ............ c~l
APPENDDI D - GLOSSARY OF TERMS AND ACRONYMS .... ........ D-l
IV
-------
-------
1. EXECUTIVE SUMMARY
The Science Advisory Board (SAB) has completed its review of the Office of
Solid Waste and Emergency Response (QSWER) draft guidance, entitled
"Assessment Framework for Ground-Water Model Applications," dated October
1992 (hereafter cited as the Framework) (See Appendix C; Reference 6). The
document was reviewed and discussed at a meeting of the Modeling Project
Subcommittee (MPS) of the Environmental Engineering Committee (EEC) in
Arlington, VA on January 14-15, 1993. At this meeting, technical presentations on
the development and application of the Framework were given by OSWER's
Information Management Staff (IMS), their contractors and consultants, and EPA
Region III and V technical staff. Subcommittee member and consultant review
comments were verbally conveyed to these participants at the close of the meeting
in a debriefing session.
In accordance with the "charge to the Subcommittee," (See Appendix B for
the complete charge) the MPS review focused on:
a) the scientific underpinnings and completeness of the substance of the
Framework,
b) whether the Framework provides help from the project management
perspective for managing model applications,
c) whether the use of the Framework will serve to aid the OSWER staff
in improving the management of its modeling activities, and
d) whether there is additional information or direction which should be
added to improve the Framework, especially with regard to project
management requirements for the modeling team, and to the
modeling code and public domain issues,
The OSWER Information Management Staff are to be commended for their
thoughtful approach to the development of the Framework. The MPS strongly
supports this effort and encourages the extension of this framework, or the
development of other frameworks, for additional types of model applications.
The Framework represents a significant advance in OSWER's approach to the
management and use of mathematical models in Superfund remediation planning.
Overall, the Framework is based on sound science and its developers have
appreciated the need for a comprehensive and flexible document. The modeling
philosophy represented in the Framework is reasonable and for the most part, the
language and level of detail are suitable for the intended users. In some of the
sections, however, the Framework contains a level of detail which the MPS
believes to be inappropriate. Here, the presence of too much detail may tend to
-------
reduce the general applicability of the guidance and could potentially foster its
misuse as a document to supplant experienced professional judgement.
The MPS believes that this Framework will be a useful tool for OSWER
management and its consultants in the management and assessment of modeling
applications. It will, in fact, provide a basis for common agreement among diverse
users as to which components constitute a good modeling study, and it could help
to improve the level of awareness and quality of modeling applications. In this
effort, OSWEH has responded positively to past guidance from the SAB/EEC and
is to be commended for its early involvement of Agency Regional Office personnel
in the development of the Framework.
The MPS is concerned, however, that the Framework could be
inappropriately formalized as a required checklist to be used as the basis for
accepting or evaluating the quality or appropriateness of a particular modeling
study. We would like to stress that modeling is not a linear process; that it is an
iterative, evolutionary approach to the refinement of our understanding of a
natural system, and a tool for the application of this understanding to the
prediction of system response. Clearly, no guidance document is a substitute for
modeling education and experience. The Framework should not be used to
promote modeling by inexperienced people, nor be relied upon to supplant
experienced professional judgement or measurement, The MPS has made a
number of recommendations in the report which follows to reduce the likelihood
that the Framework will be misinterpreted or misused, and to enhance its value to
managers as a guidance document.
While many useful EPA publications related to ground-water are referenced
in the Framework, it appears that a few related documents were overlooked in its
development. It is important that use be made of previously published materials
which are relevant to the subject. Therefore, a number of specific documents were
cited by the MPS to be incorporated in its reference list.
The MPS was not asked to review the bound document entitled "Ground-
Water Modeling Compendium," and was only able to inspect it in a cursory
manner, as this document was not distributed prior to the meeting of January 14,
1993. Because the Framework represents a rather broad overview of modeling
practice, it seems likely that the user-community receiving the compendium will
interpret the Framework in terms of the example model application descriptions
and assessments embodied in the Compendium. It is observed by the MPS that
the modeling studies presented in the Compendium predate the Framework and
may not be illustrative of its reasoned application. Therefore, to avoid any
potential interpretational pitfalls, the MPS recommends the publication of a
revised Framework as a separate document, OSWER staff should be encouraged
to seek out good modeling application case histories to illustrate applicable points
covered in the Framework,
-------
To avoid Framework user misconceptions about tlie predictive accuracy of
models, the MPS recommends that the document language be refined to clarify the
precise meaning of phrases, such as "required level of confidence," "performance
target," "quality assurance," and "model verification."
A number of specific concerns and suggestions, along with suggested
language revisions are offered by the MPS in Appendix A. With appropriate
revisions as suggested, the MPS believes that the Framework will become a useful
document for management of ground-water modeling applications within the
Agency,
2. MTEQDUCTION
The Science Advisory Board (SAB) has completed its review of the Office of
Solid Waste and Emergency Response (OSWEE) draft guidance entitled
"Assessment Framework for Ground-Water Model Applications" dated October,
1992 (hereafter cited as the Framework) (See Appendix C; Reference 6), The
document was reviewed and discussed at a meeting of the Modeling Project
Subcommittee (MPS) of the Environmental Engineering Committee in Arlington,
VA, on January 14-15, 1993. At this meeting, technical presentations on the
development and application of the Framework were given by OSWEK's
Information Management Staff, their contractors and consultants, and EPA Begion
III and V technical staff. Subcommittee member review comments were
summarized for these participants in a public session at the close of the meeting.
The Framework is a product of QSWER'a Pilot Study on Model
Management. This Pilot Study was designed to provide "useful information on
existing modeling practices and models to EPA staff, contractors, and the
regulated community," (EPA-500-B-92-006) (See Appendix C; Eeference 7). The
Pilot Study is part of an ongoing OSWER initiative on modeling which was
motivated by the SAB's January 1989 Report, "Resolution on Use of Mathematical
Models by EPA for Regulatory Assessment and Decision-Making," (See Appendix
C; Reference 9). Earlier products of this OSWER initiative have been reviewed by
the SAB/EEC (See, for example, EPA-SAB-EEC-91-016, see Appendix C; Reference
10).
The stated objective of the Framework is "to support the use of models as
tools for aiding decision-making". The Framework is being disseminated as an
integral part of a document entitled "Ground-Water Modeling Compendium: Model
Fact Sheets, Descriptions, Applications, and Assessment Framework (EPA-500-B-
92-006, October 1992) (See Appendix C; Reference 7) (hereafter cited as the
Compendium). It is intended that use of the Compendium will "help promote the
appropriate use of models and therefore sound and defensible modeling within the
hazardous waste/Superfund programs."
-------
The MPS was not asked to review the Compendium. Thus, the SAB/MPS
review focused solely on the Framework, In accordance with the "charge to the
Subcommittee," the MPS was asked to assess the scientific correctness and
completeness of the substance of the Framework; whether the Framework provides
help from the project management perspective for managing model applications;
whether the use of the Framework will serve to aid the OSWEE staff in
improving the management of its modeling activities; and whether there is
additional information or direction which should be added to improve the
Framework, especially with regard to project requirements for the modeling team
and to the modeling code and public domain issues, (See Appendix B for the
charge to the SAB), The report which follows herein is the consensus response of
the MPS to this charge.
2,1 General Assessment
The Framework represents a significant advance in OSWER's approach to
the management and use of mathematical models in the waste management
programs, including Superfund remediation, Overall, OSWER has done an
excellent job of formulating criteria for modeling application assessment. In this
effort, OSWER has responded positively to past guidance from the SAB/EEC (See
Appendix A; References 8, 9, & 10) and is to be commended for its early
involvement of Regional Office Personnel in the development of the Framework.
The Framework is based on sound science and its developers have
appreciated the need for a comprehensive and flexible document. The MPS
believes that this Framework will be a useful tool for OSWER management and
consultants in the management and assessment of modeling applications. It will
provide a basis for common agreement among diverse users as to which
components constitute a good modeling study and it could help to improve the
level of awareness and quality of models applications. This perspective on the
intended use of the Framework is also shared by OSWER/IM staff.
The MPS is concerned, however, that the Framework could be
inappropriately formalized as a required checklist to be used as the basis for
accepting or evaluating the quality or appropriateness of a particular modeling
application. Clearly, no guidance document is a substitute for modeling education
and experience. The Framework should not be viewed as an endorsement for the
application of models by inexperienced people nor should it be relied upon to
supplant experienced professional judgment.
The following recommendations are made to reduce the likelihood that the
Framework will be misinterpreted or misused and to enhance its value to
managers as a guidance document. These recommendations involve six general
issues:
a) Expansion of the Framework introduction,
-------
b) Graphic representation of concepts,
c) Balance of treatment,
d) Reference to previous work,
e) Interrelationship of the Framework and its Compendium, and
0 Clarification of terminology.
Some substantive comments on particular assessment criteria are also
included, in response to the charge to the Subcommittee.
In the interest of providing a concise, readable report, detailed technical
comments which illustrate and elaborate upon the general recommendations are
included in Appendix A. These technical comments are referenced where
appropriate in the body of the report.
3. RECOMMENDATIONS
3.1 Expansion Of The Framework Introduction
Although it is recognized that a concise guidance document is desirable, the
MPS strongly recommends that the introduction to the Framework be
substantially expanded to clarify the objectives and intended applications of the
guidance. In this revised introduction, the following points should be addressed:
a) The focus of the current Framework should be explicitly identified as
pertaining to the application of groundwater flow and advective
transport models,
b) The objectives of the Framework should be clearly described,
c) The intended (primary and secondary) users of the Framework should
be identified,
d) The potential way in which the Framework could be employed by
each of these users and the appropriate stages (timing) for its
application in the modeling process should be described,
e) The intended flexibility of the guidance and its status as a "living
document," subject to expansion and revision, should be emphasized,
and
-------
f) A brief discussion of the value of modeling would also be appropriate
in the introduction. The Framework, in its present form, focuses
almost exclusively on the use of models for prediction and evaluation
of alternative remedial schemes. While prediction is often the
endpoint of the modeling process, the value of modeling is not limited
to this goal The modeling process can enhance one's understanding
of the natural system, help in the refinement of a conceptual model,
facilitate hypothesis testing, help check consistency of data sets, help
identify critical (controlling) processes, and aid in the planning of site
characterization (data collection).
Points (a) through (e) above were addressed in the technical briefings
presented to the MPS by OSWER personnel and consultants on January 14, 1993.
OSWER is encouraged to use the viewgraph materials from these briefings to help
in the expansion of the Framework introduction, Particular attention should be
directed toward the viewgraph materials which consider the limitations of the
document.
Particularly useful insights regarding the need for and application of the
Framework were provided in a presentation by Dr. Luanne Vanderpool, EPA
Region V Geologist. Dr. Vanderpool's presentation included a discussion of how
the Framework is likely to be used in practice and by whom. She indicated that
she is acquainted with a number of instances where ground-water models had been
used to evaluate sites in a manner that was either inappropriate or difficult for
EPA management to assess. The principal purpose of the Framework in this
application is to provide guidance to EPA management in the review of model
applications. Her presentation also included a good discussion of the stages in the
modeling process at which the review criteria could be applied. These stages
included the initial proposal for model use; the early application of the model; at
completion of the model application; and in a post-review, after decisions based
upon the model application were implemented. Dr, Vanderpool emphasized that
applications of the Framework at early stages of the modeling process could be
most beneficial. Her presentation also addressed how the Framework might be
applied differently, depending upon the group which performs the model study
(i.e., the EPA, an EPA contractor, or a consultant for the Potentially Responsible
Party (PRP) and the individual(s) who performs the review). Effort should be
made to incorporate the above insights into the Framework introduction.
3,2 Graphic Representation Of Concepts
Although the Framework is designed as a comprehensive and flexible
document, its format creates the unfortunate impression that the intended use of
this document is as a once-through checklist. This impression is conveyed
specifically by the outline organization, the lack of cross-referencing to indicate
important points of feedback, and the identification of individual items by boxes
(D). Modeling, however, is not a linear process. It is an iterative, evolutionary
-------
approach to the refinement of our understanding of a natural system and a tool
for the application of this understanding to the prediction of system response.
Typically, model type and complexity will evolve during the course of a
project as more is learned about a system. In order to emphasize this iterative
nature of modeling, the MPS recommends that figures (logic flow diagrams) be
added to the Framework text to represent the important stages, the
interrelationships, and feedback loops in the modeling process. This use of figures
would help clarify the meaning of many statements in the guidance and would
help reduce the tendency to use the framework in a linear, once-through
(sequential) fashion. These figures should also incorporate the ways in which
modeling can be integrated into data collection and decMon-making, thereby
providing manager-users a broad conceptual view of the potential role of models in
site assessment and in remediation planning. The boxes identifying individual
sections of the Framework should be deleted.
3.3 Balance Of Treatment
Overall, the modeling philosophy represented in the Framework is
reasonable and the language and level of detail are suitable to the intended users.
The Framework developers have recognized the importance of maintaining
comprehensiveness and flexibility in the document. It is necessary that the
Framework provide sufficient detail to clarify the important concepts for all-users,
while, at the same time, avoiding over-prescription of model components or
structure, thereby retaining a balanced, flexible, concise guidance document. Such
a balance is often difficult to achieve.
In some of its sections, the Framework contains a level of detail which the
MPS believes to be inappropriate to the above objectives. Here, the presence of
too much detail will tend to reduce the general applicability of the guidance and
potentially foster its misuse as a document to supplant experienced professional
judgment. Two important sections which the MPS finds overly detailed are
highlighted below, and other more minor suggestions are included, by section
number, in Appendix A.
Section 24, under "Conceptual Model Development," itemizes required field
data. In this list, there is a marked lack of balance between the collection of flow
and transport-related information, with most of the emphasis placed on flow data.
Clearly, no data list can be exhaustive, nor are all listed data equally
important or relevant to a specific system or model. Thus, to increase the
comprehensive and flexible use of this guidance, the MPS recommends deletion of
the third and fourth order subheadinp in this section. Second order subheadings
should then be reorganized and reworded to reflect more balance between flow and
transport data. A suggested specific format is given in Appendix A.
-------
There is a serious omission in Section 24 which also needs to be rectified.
Here the potential role which heterogeneity plays in conceptual model development
is not considered. The importance of quantitative and qualitative information on
the variability of properties must be acknowledged. The suggested wording in
Appendix A is intended to address this issue,
The section titled, "Model Set-up and Input Estimation" contains too much
detail pertaining to model structure. Here the focus is on the application of finite
difference and finite element models, to the exclusion of other numerical or
analytical approaches. The Framework, however, is not intended as a finite
difference or finite element modeling primer. It is recommended that this entire
section be rewritten to emphasize documentation of modeling decisions, rather
than specific rules for model grid development and temporal discretization.
3.4 Reference To Previous Work
Although the Framework contains an extensive reference list, it appears
that a number of related reports were overlooked. It is important that use be
made of the major publications relevant to the subject. The reports listed below
which discuss many issues pertinent to the Framework, should be consulted in the
revisions of this document:
Keely, J.F, (1987). (See Appendix C; Reference 1) presents numerous useful
schematics that illustrate modeling concepts and their interrelationships
U.S. Office of Technology Assessment, (1982). (See Appendix C; Reference 12)
reviews the use of all models in water resources, including ground-water models.
Van der Hepe, Paul, K.M. and Richard A, Park, (1986). (See Appendix C;
Reference 13).
NRG, 1990 (See Appendix C; Reference 5) is an authorative review by the
NRC/NAS, sponsored in part by EPA, which summarizes the salient aspects of
mathematical modeling applications for remedial assessments, This document
discusses many of the same issues of importance in the Framework.
The Framework contains a useful list of EPA Publications related to
ground-water modeling. In addition, footnotes direct the user to other references
for more information. Some of the footnoted references, such as draft documents,
however, may not be easily accessible. The MPS urges OSWER to make
provisions for user access to this information part of their overall management
plan. The reference list should be viewed, like the Framework, as a dynamic
entity to be updated and refined with time and experience.
8
-------
3,5 Interrelationship Of The Framework and Its Compendium
The Assessment Framework is being disseminated as an integral part
(Section 2.0) of a bound document titled, "Ground-Water Modeling Compendium:
Model Fact Sheets, Descriptions, Applications, and Assessment Framework"
(EPA-500-B-92-006), October, 1992 (See Appendix C; Reference 7).
The MPS was not asked to review the Compendium and indeed, was able to
inspect it only in a cursory manner, as this document was not distributed prior to
the meeting on January 14, 1993. Because the Framework represents a rather
broad overview of modeling practice, it seems likely that the user community
receiving the Compendium will interpret the Framework in terms of the example
model application descriptions and assessments embodied in the Compendium,
The MPS is concerned that the Framework has been distributed widely in
this form prior to SAB review. Modeling studies presented in the Compendium
predate the Framework and may not be illustrative of its reasoned application. To
avoid the interpretational pitfalls described above, the MPS recommends the
publication of a revised Framework as a separate document,
OSWER should be encouraged, however, to seek out good modeling
application case histories to illustrate points covered in the Framework, In fact,
many case histories are a part of the literature or the public record. The MPS
believes that a critical analysis of selected case histories in the context of the
Framework could be of substantial value to the intended users,
3.6 Clarification Of Terminology
A number of phrases used in the Framework could easily be misinterpreted
by a user unfamiliar with models and their limitations. These phrases include
"required level of confidence," "performance target," "quality assurance," and "model
verification." Indeed, within the peer-reviewed modeling literature itself, there is
substantial disagreement as to the precise meaning of these terms (See for
example, Appendix C; Reference 3),
To avoid Framework user misconceptions about the predictive accuracy of
models, the MPS recommends that document language be refined to clarify the
meaning of such phrases. Specific concerns and some suggested language revisions
are listed by section number in Appendix A.
There is a great deal of emphasis in the Framework on "model verification"
and "quality assurance," While the listed modeling activities associated with these
goals are entirely appropriate and consistent with good modeling practice, they
cannot assure that a model reliably represents a given physical system. Certain
activities are necessary to build confidence in a model application but they are
insufficient to guarantee accurate predictions.
9
-------
When a physical system is subject to new stresses (as during the application
of a remedial strategy), errors in the conceptual model which had little impact
during the calibration phase may become dominant sources of error for the
prediction phase. Similarly, while use of qualified personnel, careful
documentation of the modeling process, and peer review are undoubtedly
important aspects of sound modeling management, they cannot assure a quality
(i.e. accurate) model prediction. Because, then, a specific model of a physical
system can never be completely "verified," it becomes important to identify
uncertainties in model input parameters and conceptual assumptions and to
explore the implications of these uncertainties on model predictions. These issues
should be stressed in the Framework guidance.
For a more complete discussion, see: Konikow, Leonard F. and John D,
Bredehoeft, 1992. (Appendix C; Reference 3).
3,7 Sections 14 and 39
As part of its charge, the MPS was particularly requested to provide
recommendations pertaining to Sections 14 and 39 of the Framework, Section 14,
listed under "Project Management," focuses on the desirable qualifications of the
modeling team (See Appendix C; Reference 6).
The MPS believes that this section would benefit from OSWER's review of
the previously referenced USEPA Ground-Water Modeling Policy Study Group
report (van der Heyde and Park, 1986, See Appendix C; Reference 13), This
document discusses the types of staff required to perform groundwater modeling in
a section titled, "Technology Transfer and Training to Sustain and Improve
Expertise of Agency Personnel" (pp, 52-58).
Another publication (OTA, 1982. See Appendix C; Reference 12) considers
specific education and training requirements for model developers, users, and
managers involved in water resources modeling applications. These two reports
cited above also address issues relating to the hiring and retention of qualified
personnel,
The MPS recommends that the above references be used to refine the
statement in Section 14 concerning required modeling team experience.
Section 39, under "Model (Code) Selection," relates to the accessibility of
models and suggests that a selected model should be in the "public domain." The
use of proprietary vs. public domain codes is discussed in van der Heijde and Park
(1986) (See Appendix C; Reference 13). In this report, it is recommended (p. 42)
that a general framework of nondiscriminatory criteria should be established by
the Agency to apply to all codes. The MPS agrees with this 1986 Study Group's
recommendation and suggests that a set of criteria be developed to replace the
existing language in Section 39 and to eliminate the need for making a distinction
between public domain and proprietary models. These criteria should include:
10
-------
a) Publication and peer review of the model conceptual and
mathematical framework,
b) Full model documentation and visibility of underlying model
assumptions,
c) Incorporation of a mass balance,
d) Testing of the code, and
e) Model availability for independent (third party) evaluation.
In the formulation of model selection criteria, it should be noted that, if
ground-water models are to be relied upon to form expert opinions in litigation,
the other side will be permitted access to the foundation of those opinions,
including the source code. This was the ruling of the Special Master in the case
of United States of America gt a] vs. Hooker Chemicals & Plastics Corporation et
al (Love Canal) on November 30, 1989 (See Appendix C; Reference 11), At the
same time, the Special Master granted a Protective Order so that the code could
not be used by the opposing side for any purpose other than the trial. Thus,
when selecting a ground-water model for litigation purposes, access to the source
code is an important consideration.
For further information on the use of ground-water models in litigation, see;
Kezsbom, A. and A.V. Goldman, 1991. (See Appendix C; Reference 2).
4. CONCLUSIONS
With appropriate revisions, as outlined above, the MPS believes that the
Framework will become a useful document for management of ground-water
modeling applications within the Agency, QSWER/IM staff are to be commended
for their thoughtful approach to the development of this guidance. The use of
experts and the early involvement of regional staff have helped to produce a
comprehensive and flexible document.
The MPS strongly supports this effort and encourages the extension of this
Framework (or the development of other frameworks) for additional types of
model applications. While the Framework will undoubtedly help improve existing
modeling management practice, it must be cautioned that this guidance cannot
supplant the pressing need for the hiring and retention of qualified modeling
personnel in the Agency, This point was addressed in an earlier Modeling
Eesolution report of the SAB/EEC (EPA-SAB-EEC-89-012) (See Appendix C -
Reference i).
To aid OSWER in its overall management of ground-water models, it is
further recommended by the MPS that a repository be established to electronically
11
-------
archive site-specific model application, data files, A discussion of this concept is
contained in National Research Council, 1992. (See Appendix C; Reference 4).
The SAB/EEC/MPS has appreciated the opportunity to review the draft
Framework document and would be pleased to review future products of the
OSWEE Pilot Study on Model Management.
12
-------
APPENDIX A - DETAILED TECHNICAL COMMENTS
(By Section Number)
This represents a compilation of detailed technical comments which
illustrate and elaborate upon the general recommendations contained in the body
of this report. They are itemized by the corresponding section number in the
Framework document (See Appendix C; Reference 6),
Modeling Application Objectives:
It is recommended that a section be added prior to number 1, relating to
the establishment of a need for the modeling study.
2. The management decision objectives and the role and need for a modeling
study in the pursuit of these objectives should first be established, considering
applicable regulatory and policy issues.
3. It is difficult to distinguish here between model functions and modeling
objectives. Perhaps function should be defined in glossary.
5. This section should be rephrased to reflect the fact that the "required" level
of analysis (level of model complexity) may be revised as a better understanding of
the site/problem/data is developed.
In this section, some discussion of the factors which influence model
complexity might also be appropriate. These factors include:
a) The importance of the decisions which will be influenced by the
model results,
b) The sensitivity of these decisions to the range of possible or likely
outcomes of the modeling, and
c) The availability of time and resources for the modeling application.
7. This section is ambiguous. It is not clear how the required level of
confidence can be specified in advance in a meaningful way nor how to determine
whether the desired level of confidence is achieved. It is suggested that this item
be deleted.
8, More clarification is needed on the meaning of the term, "performance
target." If, as stated in the glossary, this means model accuracy, then this second
phrase should be defined. The Framework should address the possibility that the
performance target may not be obtainable.
A-l
-------
12, It is recommended that this section be combined with section 5.
13. It is recommended that this section be combined with section 3.
Project M&iiflggmept:
Here one important aspect of project management is not considered, namely
financial budgeting and cost control. Related areas of project management for
computer modeling would include management and procurement of adequate
modern computer equipment and facilities.
17, The meaning of the word, "independent," must be clarified. Does
independent mean that people who provide peer review cannot work for the same
contractor or agency or that they cannot have worked on the same project?
Clarification must be given for the meaning of Quality Assurance (QA) in the
present context. Components of QA discussed here include (a) staffing with
qualified people; (b) peer review; and (e) appropriate documentation. Are other
components envisioned? Caution should be exercised against the establishment of
a rigid QA bureaucracy of personnel and procedures which could adversely impact
project efficiency and progress,
19. This item seems rather obvious and inconsistent with the tone of the rest of
the Framework, and it is suggested that it be deleted,
20, Here the need to provide enough information for reproducibility should be
emphasized,
21. Parts of this section are very detailed. In this detail, there is a lack of
balance between flow and transport. It is suggested that the subheadings
"Background Chemical Quality* and "Chemical Parameters" be added under
Conceptual Model and Ground-Water Model Construction, respectively, to improve
the balance.
The subheading "Sources and Sinks" should be rephrased as "Water Sources
and Sinks." The documentation should include a section highlighting assumptions,
their relation to reality, and their potential impact on the solution.
The Public Domain vs. Proprietary Model subheading should be revised for
consistency with Section 39, as discussed in the main body of this report.
Conceptual Model Development:
24, Here too much detail is included. It is recommended that third and fourth
order subheadings be deleted and that the second order subheadings be reworded
to be more inclusive of and flexible to varying model application scenarios.
Acknowledgment must also be made of the potential importance of heterogeneity.
A suggested revised subsection is; "Quantification and Qualification of Field Data."
A-2
-------
The conceptual model should be based upon a quantification of field data as
well as other qualitative data that includes information on the nature and
variability of:
a) Aquifer system (Distribution and configuration of aquifer and
confining formations),
b) Hydrologic boundaries,
c) Hydraulic and chemical properties of formations,
d) Fluid potential and properties,
e) Contaminant sources and properties, and
f) Fluid sources and sinks.
34. It is recommended that this section be deleted. The important aspects of it
are already contained within Sections 23 and 24. Further, the wording here would
seem to imply an unwarranted rigidity in the conceptual model, that parameters
should not be modified during or after calibration.
Model (Code) Selection;
35, Recommended rewording: "The selected model (code) should be described
with regard to its flow and contaminant transport and transformation processes".
36. This section should be clarified to indicate that it is the reliability of the
generic model (code) that is being assessed here. It may be appropriate to
reference the International Ground Water Modeling Center (IGWMC) Data Base
here as a footnote.
Model Setup and Input Estimation:
46/47, Here the level of detail should be reduced, as discussed in the main body of
the report.
58. The meaning of water budget must be clarified here. Is this the water
budget of the model or the real system?
61. Although it may be desirable, it is almost never feasible in real field
applications to obtain an independent set of field observations.
Overall Effectiveness:
This section should include a statement describing another measure of
effectiveness, that the modeling results are consistent with most available data,
A-3
-------
Glossary:
Calibration:
Change wording to indicate that the range of error does not have to be
pre-established.
Tranaformaiaon:
Add reference to biological transformations.
Uncertainty
Replace existing entry with the following definition; process to identify
uncertainties m model input parameters and conceptual assumptions, and the
implications of these on the uncertainty in model predictions, including potential
impacts on the decisions which will be made based on these predictions,
Verification:
See discussion in main body of this report.
References:
Two references cited in the Framework are incomplete and consequently
difficult for the user to find: Bauer et al, 1984, Middleton and Hiller, 1990.
A-4
-------
25
's Charge
It the substance of the Framework scientifically correct?
Does the Framework address ground-water model
application activities with as much completeness as is
acceptable and necessary?
J Does the Framework provide help from the project
management perspective for managing model
applications?
o
Will the use of the Framework serve to aid OSWER in
improving the management of its modeling activities?
Is there additional information or direction which
should be added to #14 (Project Management -
requirements for modeling team) and #39 (model code
in the public domain)?
Technical Pr*»»nt«tfon and IHMUMlon of th* Franwworfc J»mi«r» 14,
-------
APPENDIX C - REFERENCES CITED
1) Keely, J.F. 1987, The Use of Models in Managing Ground-Water Protection
Programs, EPA/600/8-87/003, ORD, Ada, OK. 72p.
2) Kezsbom, A. and A.V, Goldman, 1991. "The boundaries of groundwater
modeling under the law: Standards for excluding speculative expert
testimony," Tort & Insurance Law Journal. Vol. XXVII, No. 1, pp. 109-126.
3) Konikow, Leonard F. and John D. Bredehoeft,"Ground-Water Models Cannot
be Validated," Advances in Water Resources 15, 75-83, 1992,
4) National Research Council, A review of ground water modeling needs for
the U.S. Army, Washington, D.C., 1992.
5) National Research Council, Ground Water Models; Scientific and Regulatory
Applications, National Academy Press, Washington, DC, 1990, 303 pp.
6) U.S. EPA, "Draft Assessment Framework for Ground Water Model
Applications," Office of Emergency and Remedial Response, October 1992
(NOTE; This Draft Assessment Framework was sent by Richard J.
Guimond, Assistant Surgeon General, USPHS and Deputy Assistant
Administrator of the Office of Solid Waste and Emergency Response on
October 5, 1992 to Office Directors and Regional Waste Management
Division Directors, along with a proposed Policy Directive on the Use of
Computerized Models m the Hazardous Waste/Superfund Programs. On
October 26, 1992 Richard J. Guimond sent a formal request to Dr. Donald
G. Barnes, Director of the SAB for review of the Draft Guidance
"Assessment Framework for Ground Water Model Applications" in Support
of an OSWER Policy on the Use of Models).
7) U.S. EPA, "Ground-Water Modeling Compendium: Model Fact Sheets,
Descriptions, Applications, and Assessment Framework" (EPA-500-B-92-006),
October, 1992.
8) U.S. EPA/SAB Memo from Conway, Small and Kooyoomjian of the SAB to
Mr. Asa R. Frost, Jr., Director of Information Management, U.S. EPA,
Office of Solid Waste and Emergency Response, Washington D.C, Pertaining
to the SAB's Consultation on Review of Selection Criteria for Participation
of EPA Staff on the Proposed Agency Task Force on Modeling, April 29,
1991
9) U.S. EPA/SAB Report of the Environmental Engineering Committee,
entitled, "Resolution on Use of Mathematical Models by EPA for Regulatory
Assessment and Decision-Making," (EPA-SAB-EEC-89-012), January, 13,
1989
C-l
-------
10) U.S. EPA/SAB Report of the Environmental Engineering Committee,
entitled, "Usage of Computer Models ill the Hazardous Waste and Superfund
Programs," (EPA-SAB-EEC-91-016), September 6, 1991
11) United States of America et al vs. Hooker Chemicals & Plastics Corporation
etal (Love Canal) on November 30, 1989.
12) U.S. Office of Technology Assessment, "Use of Models for Water Resources
of the United States," U.S. Government Printing Office, Washington, D. C.,
1982
13) Van der Hepe, Paul, K.M. and Richard A. Park, 1986. U.S. EPA Ground
Water Modeling Policy Study Group, International Ground Water Modeling
Center, Holcomb Research Institute, Butler University, Indianapolis, IN
C-2
-------
APPENDIX D - GLOSSARY OF TEEMS AND ACRONYMS
EEC
EPA
IGWMG
IM
IMS
MPS
NAS
NEC
osw
OSWER
OTA
PRP
QA
SAB
U.S.
USGS
Environmental Engineering Committee (SAB/EEC, also
referred to as "The Committee")
U.S. Environmental Protection Agency (U.S. EPA, or "The Agency")
International Ground Water Modeling Center
Information Management (U.S. EPA/OSWER)
Information Management Staff (U.S. EPA/OSWER)
Modeling Project Subcommittee (U.S. EPA/SAB/EEC)
National Academy of Sciences
National Research Council
Office of Solid Waste (U.S. EPA)
Office of Solid Waste and Emergency Response (U.S. EPA)
U.S. Congressional Office of Technology Assessment
Potentially Responsible Party
Quality Assurance
Science Advisory Board (U.S. EPA)
United States
United States Geological Survey
D-l
-------
DISTRIBUTION LIST
Deputy Administrator
Assist Administrators
EPA Regional Administrators
EPA Laboratory Directors
Deputy Assist Administrator for Office of Solid Waste and Emergency Response
(OSWER)
Director, Office of Solid Waste (OSW)
Deputy Director, OSW
Director, Office of Emergency and Remedial response (OERR)
Deputy Director, OERR
Deputy Assistant Administrator for Office of Prevention, Pesticides and Toxic
Substances
Deputy Assistant Administrator for Office of Research and Development (ORD)
Director, Office of Environmental Engineering and Technology
Demonstration (OEETD)
Deputy Director, OEETD
Director, Office of Monitoring, Modeling and Quality Assurance (OMMSQA)
Director, Center for Environmental Research Information (CERI)
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
-------
------- |