United States ....'-.
Environmental Protection
Agency j
So|id Waste'AntL ,,:<
Emorgency Response
Report of the AgencyTam
on Environmental Regulatory
Modeling
Guidance, Support Needs, Draft Criteria
and Charter
Agency Task Force on
Environmental Regulatory
Modeling
-------
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON. D.C. 20460
SPR 25
MEMORANDUM
SUBJECT: Transmittal of the Report of the Agency Task
Force on Environmental Regulatory Modeling
FROM:
Stev^coroieVCo-Chair
Offiqe of .-Research and Development (8401)
.aW
La-fry j3-r"""Reed, Co-Chair
'"-Office'-.Q-f' Solid Waste and Emergency Response (5204G)
THROUGH:
TO:
Gary J. Foley ,>_..
Acting Assistant Administrator^
Office of Research and Development
Elliott P. Laws
Assistant Administrator
Office of Solid Waste and Emergency Response
Robert M. Sussman
Deputy Administrator
This memorandum transmits the final report (attached) of the
Agency Task Force on Environmental Regulatory Modeling (ATFERM)
for your consideration and approval. The report is also being
transmitted to the Science Policy Council's Steering Committee so
that its recommendations may also be addressed by that Council.
The report includes conclusions and recommendations for
improvement in the development and .use of modeling as well as a
guidance document which will help ensure the scientific integrity
of the use of models. In addition, the report recommends and
ATFERM requests your approval for the establishment of a
permanent Committee on Regulatory Environmental Modeling (CREM)
which will liaison with the Science Policy Council and will
require new resources.
-------
Background.
In 1992, the Assistant Administrators of the Offices of
Solid Waste and Emergency Response and of Research and
Development, recognizing the importance of the use of modeling in
environmental decision-making, requested the formation of an
Agency-wide task force to address modeling issues and needs. The
Science Advisory Board had previously issued a resolution on
modeling which described the need for improvement in the
development and use of modeling. In response to the Assistant
Administrators' request, EPA's Deputy Administrator established
an ad hoc Agency Task Force on Environmental Regulatory Modeling
(ATFERM).
Since that time, the Task Force members, representing most
of the Agency's programs and regions, have developed a number of
products, issue papers, conclusions and recommendations. These
products are based on the Task Force's original charge to address
"acceptability criteria for model use,..Agency requirements for
peer review,..expansion of training and technical support" and
the "advisability of establishing a permanent Agency-wide Expert
Panel on Environmental Modeling".
Contents
Section I of the report describes the status and need for
training and technical support for modeling. Section II contains
a draft set of "acceptability criteria" for models.
Section III of the report contains the final version of the
"Guidance for Conducting External Peer Review of Environmental
Regulatory Modeling". The Guidance provides a resource for EPA
managers who utilize modeling and who are responsible for
implementing peer review. The Guidance has been reviewed by the
Science Advisory Board, the Council of Science Advisors, and, at
your request, the Assistant and Regional Administrators. You
have been asked, in a separate memorandum, to transmit the final
version of the Guidance Agency-wide.
Section IV, described below, contains a proposal and draft
charter for a permanent group on environmental modeling.
Recommendations for a Permanent Group
The Task Force requests that you consider and approve its
recommendation to establish a permanent group, or Committee on
Regulatory Environmental Modeling (CREM), as described in Section
IV of the report. The Committee's purpose is to provide the
Agency with consistent yet flexible modeling guidance and tools
to support environmental decision making.
-------
The Task Force recommends that the chair of the permanent
Committee be a liaison to the Steering Committee of the Science
Policy Council. As described in Section IV of the report,.the
Task Force also recommends that the executive secretariat and
staff for the permanent Committee be placed in the Office of
Research and Development and that an initial sum of $300,000 be
available through a pro-rated percentage of the Agency's
Headquarter's programs and its Regions. The proposed operating
plan for the first two years is also found in Section IV,
Appendix II.
Along with approving the establishment of the Committee, we
are asking you to appoint its chair, in your role as Chair of the
Science Policy Council.
Please respond with your approval and/or questions regarding
the proposal. If you need additional information or-desire a
briefing, please notify either of us. . - '
Attachment
cc: Agency Task Force on Environmental Regulatory Modeling
Science Advisory Board Executive Committee
Assistant Administrators
Regional Administrators
Science Policy Council Steering Committee
-------
-------
FINAL REPORT
From The
AGENCY TASK FORCE ON
ENVIRONMENTAL REGULATORY MODELING
To The
DEPUTY ADMINISTRATOR
March 1994
-------
-------
TABLE OF CONTENTS
Page
INTRODUCTION AND SUMMARY 1
Purpose ,. '.-...- 1
Background 1
Activities . 2
Summary of Conclusions from the Report 2
Summary of Products and Recommendations 5
SECTION I - TRAINING AND TECHNICAL SUPPORT NEEDS 1-1
Introduction 1-1
Training 1-1
Direct Support Services 1-3
Information Exchange 1-5
Resource Needs 1-8
SECTION II - MODEL USE ACCEPTABILITY CRITERIA II-1
Introduction II-l
Definitions II-2
Approaches Considered II-2
Acceptability Criteria II-5
Conclusions And Recommendations II-8
SECTION III - AGENCY GUIDANCE FOR CONDUCTING EXTERNAL PEER
REVIEW OF ENVIRONMENTAL REGULATORY MODELING . . . III-l
Introduction III-l
Framework For Peer Review of Environmental Regulatory Modeling III-2
Relationship Of External Peer Review To The Process of Environmental
Regulatory Model Development And Application III-5
Mechanisms and General Criteria for Conducting External Peer Review III-7
Documentation Of The Peer Review Process III-8
Specific Elements of External Peer Review For Environmental Regulatory
Modeling III-8
Agency Policy on Peer Review 111-13
SECTION IV - PROPOSED CHARTER FOR A PERMANENT COMMITTEE
ON REGULATORY ENVIRONMENTAL MODELING IV-1
Rationale For A Permanent Group IV-1
Draft Proposed Charter IV-4
Appendix I: Options And Issues IV-9
Appendix II: Resources IV-11
REFERENCE - SUPPORTING DOCUMENTS R-l
"Resolution on Use of Mathematical Models by EPA for Regulatory
Assessment and Decision Making" (SAB-EEC-89-012). 1989 R-l
"Task Force on Environmental Regulatory Modeling." Memorandum
from Henry Habicht, Deputy Administrator and Chairperson,
Risk Assessment Council. March 2, 1992 R-10
ATFERM Final Report Page i
-------
-------
INTRODUCTION AND SUMMARY
Purpose
This report from the Agency Task Force on Environmental Regulatory Modeling (ATFERM)
presents the Task Force's conclusions and recommendations on how environmental models
and modeling activity can be improved to produce greater contributions to U.S. EPA's
regulatory mission. In addition, the report contains three products which will help ensure the
scientific integrity of the use of models.
Background
Although environmental simulation models have been used to support decision-making in
EPA for many years, their use has accelerated in concert with the ready availability of
personal computers and the demand of decision-makers for quantitative answers. At the
same time, the number of trained modelers has not kept pace. Consequently, the Agency
and others have become increasingly concerned about consistency and quality in model
development, selection, and application. For instance, the Science Advisory Board expressed
its concerns in "Resolution on Use of Mathematical Models by EPA for Regulatory
Assessment and Decision Making" in 1989 (SAB-EEC-89-012).
Since model-supported decision-making can have enormous environmental and economic
ramifications, EPA took action. The Assistant Administrator for Research and Development
and the Assistant Administrator for Solid Waste and Emergency Response jointly requested
the Deputy Administrator, as the Chair of the former Risk Assessment Council, to establish a
task force to examine the issues.
In March 1992, the Deputy Administrator approved the creation of an ad hoc "Task Force
on Environmental Regulatory Modeling," charging it to "...complete within 12 months a
recommendation to the Agency on specific actions that should be taken to satisfy the needs
for improvement in the way that models are developed and used in policy and regulatory
assessment and decision-making." In the same memorandum, the Deputy Administrator
asked the Task Force to "...report on the advisability of establishing a permanent Agency-
wide Expert Panel on Environmental Modeling." In addition, the following are the areas
were to be addressed:
Acceptability criteria for model use, generally and in particular circumstances
Formal technical and policy guidance on model development
Agency requirements for peer review and for documentation of models prior to
use
, Expansion of training and technical support activities for EPA personnel who
oversee model applications
ATFERM Final Report Page 1
-------
Activities
With a 12-month timetable and the constraint that all Task Force members were volunteering
their time while carrying on all of their normal duties, the Deputy Administrator's charge
was broken down into four objectives: (1) draft a set of "acceptability criteria" for models;
(2) assess the need for technical support for model users; (3) draft guidance on external peer
review of modeling; and (4) decide whether a permanent, agency-wide panel on modeling
was needed.
Task Force members divided up into subgroups to address each of the four objectives.
Monthly teleconference meetings were held to track progress, discuss key issues, and
exchange comments on each of the products being developed. By June of 1993 draft
documents presenting the Task Force's conclusion on the first four objectives were prepared
and circulated for comment. From July 1993 through October 1993 each of the four
documents was refined. Table 1 lists the Task Force membership and identifies which
members contributed to each product. The full text of each report or product is included in
this report.
Summary of Conclusions from the Report
Major Conclusions
The four sections of this report contain many conclusions. Below is a summary of the major
conclusions. In addition, each section may include recommendations based on those
conclusions.
Training and Technical Support Needs
There is a need to support those models developed and used to further specific program
objectives. Such support could come from Agency technical experts or by training that
makes use of the latest technologies. (Page 1-4)
Personnel responsible for model use or interpretation need to be properly trained in the
exercise of that responsibility. (Page 1-2)
More technical support needs to be provided to model users in general, .so that these
decision-support tools can be fully exploited. (Page 1-8)
Technical support can be provided in the form of training, direct help to users, and
information transfer. (Page 1-1)
Short-term technical support needs could be met by panels of experts (or forums) until more
formal support programs are needed and instituted. (Page 1-4)
ATFERM Final Report Page 2
-------
Table 1: ATFERM Members, 1992-1993
(Alphabetically, by last name)
Co-Chairs:
Members:
Steve Cordle; Co-Chair; ORD
Larry Reed; Co-Chair; OSWER
Bob Ambrose; ORD/ERL-1,3
Tim Barry; OPPE
Barbara Brown; Region I
David Burden; ORD/RSKERL-1
Dorothy Canter; OSWER
Alan Cimorelli; Region III-3
Mimi Dannel; Region VI-1*
Michael Firestone; OPPTS-3*
Rick Johnson; OARM
Russell Kinerson; OW-2,3
Will LaVeille; ORD
Felix Locicero; Region II
Mary Lou Melley; OSWER-**
Don Miller; Region VII
Linda Ross, Region II-
Zubair Saleem; OSWER
Mark Tedesco; Region II
Joe Tikvart; OAR-1,2,3
Wayne Valentine; Region II-
Luanne Vanderpool; Region V-2*,l
Thomas Wadell; Region I
Richard Walentowicz; ORD-2
Joe Williams; ORD/RSKERL-"
John Yearsley; Region X-3
Larry Zaragoza; OSWER-3
Key:
1 = Training and Technical Support
2 = Model Use Acceptability Criteria
3 = Peer Review
1* = Training and Technical Support Leader
2* = Model Use Acceptability Criteria Leader
3* = Peer Review Leader
** = Task Force Coordinator
= Designated Representative
ATFERM Final Report
Page3
-------
Model Use Acceptability Criteria
Model code acceptability should be judged on the basis of appropriateness, accessibility,
reliability, and usability. (Page II-5)
There is a need for a "Model Information System" which lists models that meet the
acceptability criteria. (Page n-7)
Once a collection of acceptable models is assembled, there is a need for a process to
periodically assess the models being used to support rule-making decisions and regulatory
impact assessments. (Page II-8)
Agency Guidance For Conducting External Peer Review Of Environmental
Regulatory Modeling
Peer review is an important tool in EPA's campaign to document the quality and credibility
of the science upon which its regulatory and policy decisions are based. (Page III-2)
Not all managers who must consider the utility of peer reviews are aware of their
importance. Many of those who are aware do not have a clear description of the procedures
by which peer review can be carried out successfully. In particular, guidance on external
peer review is needed now. (Page III-l)
There is a need to begin external peer review as early in the model development phase as
possible, in order to maximize its value. (Page III-3)
External peer review of a model's applicability needs to be conducted well in advance of any
decision-making that depends upon the model's results. (Page III-4)
Information gathered from the peer review of scientific issues is critical in understanding the
uncertainties and usefulness of a model in regulatory decision-making. Therefore, such
information needs to be available to the decision-maker before decisions based on the model
are made. (Page HI-4)
Proposed Charter For A Committee On Regulatory Environmental Modeling
There is an increase in 'the use of computerized environmental models to support EPA policy
and regulatory decision-making. (Page IV-1)
Improper development, application, and use of models will undercut EPA's sound science
objective and abuse the potential of models as decision-making tools. (Page IV-1)
There is a need for a centralized focus such as an Agency council on regulatory
environmental modeling to promote the goal of providing EPA's senior policy-makers with
ATFERM Final Report Page 4
-------
a set of well-developed, well-documented, and well-understood modeling tools to support
environmental decision-making. (Page IV-2)
Summary of Products and Recommendations
Sections 1 through 4 of this report present the Task Force's findings and recommendations
on each of the four objectives discussed above, in "Activities". Each presentation is divided
into four parts background, problem statement, recommendations, and full text. Here is a
very brief summary of each product.
"Training and Technical Support Needs" asserts that a coherent, multi-faceted program of
technical support should be developed throughout the Agency. Program offices, regions, and
laboratories all have responsibilities in this area.
"Model Use Acceptability Criteria" establishes four criteria for determining when an
environmental regulatory model may be acceptable for Agency use.
"Agency Guidance For Conducting External Peer Review Of Environmental Regulatory
Modeling" describes when external peer review of modeling may be appropriate. It sketches
the general process of model development and application, showing the points at which
external peer review can be applied. It also lists specific elements of an external peer review
process that can be adapted to each particular review exercise.
"Proposed Charter For A Committee on Regulatory Environmental Modeling" describes and
recommends that a body be created by the Deputy Administrator and/or the new Science
Policy Council (SPC) that would be charged with carrying on the work begun by the Task
Force. Several options and recommendations for placing the Committee within EPA's
organizational structure are offered, together with an examination of its staffing needs and
initial budget.
ATFERM Final Report Page 5
-------
-------
SECTION I - TRAINING AND TECHNICAL SUPPORT NEEDS
Introduction
The field of environmental modeling is extremely dynamic. New models are constantly
being developed and existing models updated. In addition, many program areas are
evolving, resulting in changes in the type and number of models utilized by program staff.
These factors, along with many others, contribute to the increasing need for technical
support.
The purpose of this report is to identify critical areas of technical support needs and to
suggest ways of meeting these needs.
This report assumes that an Agency goal is to equip personnel involved in conducting or
evaluating environmental modeling with the tools required to thoroughly understand and
execute these models. In order to provide adequate technical oversight, proficiency in
modeling and interpreting model output is needed.
Technical support generally falls into three categories: training, direct support services, and
information transfer. In the sections to follow, each of these areas will be described along
with suggestions for providing the necessary services. Attention must be given to
determining what mix of training and technical support is appropriate for each office.
Information is also presented regarding the specialized resource needs of environmental
modeling.
Training
Myriad environmental models are currently used for decision making in various Agency
programs. In recent years, the use of personal computer software has made complex models
more accessible and more efficient for the user. It can be argued, however, that the ready
availability of software in the absence of proper training can be counter-productive. While
this view may seem elitist at first, the analogous situation of making a GC/MS (gas
chromatograph/mass spectrometer) available to an untrained chemist seems absurd. Because
environmental models are available to nonspecialists, the educational and training needs
required to responsibly operate the software and interpret the results mus.t be considered.
Basic Educational Requirements
Generally, a quantitative, interdisciplinary background is most helpful as a starting point in
environmental modeling. Specific areas to focus on include mathematics, physics, chemistry,
soil science, fluid mechanics, geology, and hydrogeology. Education in these environmental
sciences and engineering is an important step to becoming an effective model user.
However, it is difficult to specify an exact program since educational programs evolve over
time.
ATFERM Final Report Page I-1
-------
Recommendation: The Agency should ensure that personnel responsible for model use or
interpretation have an adequate background education. Quantitative, interdisciplinary
academic training is most helpful as a starting point in environmental modeling.
Professional Short Courses
Short courses can effectively introduce the new user to model theory, methods of application,
and realistic case studies. Since they usually last from two to five days, these courses
provide time-efficient, intense, focused contact between users and model experts. Short
courses are most effective if the user has the appropriate background education requirements;
they cannot substitute for basic education, in a discipline.
Historically, short courses have been offered directly by EPA research and program offices,
by universities, and by consulting firms. Unfortunately, the cost of university and consulting
firm courses is often prohibitive for State and Federal employees. Typical registration costs
are $600 - $1,200 for a 3-4 day course.
Recommendation: A model that is intended for wide use by States or the regulated
community should be supported by EPA-sponsored professional short courses. In situations
where development of an EPA-sponsored course is not feasible, adequate training and travel
funds should be allocated for EPA personnel to attend university or private sector courses.
Self-study Programs
A self-study program may be the most flexible alternative because it can be pursued at the
user's convenience. The effectiveness of such a program depends on the student, time
commitments, the available model documentation and bibliography for the underlying theory,
as well as other informational materials.
Self study programs can be offered in a variety of formats. For example, videotaped lectures
have been utilized by some programs. However, it is sometimes difficult to maintain the
necessary level of concentration throughout the lecture due to the nature of the material. As
a result, while the cost is fairly low and distribution is simple, a video course may not
provide a highly effective learning environment. A more interactive system, such as
computer-based tutorials, may prove to be a more viable self-study format, even though the
initial development cost could be significantly higher than a video presentation.
Recommendation: Each program area should evaluate the degree of model use and determine
the most effective methods of providing self-study courses, such as video taped lectures or
interactive, computer-based tutorials.
Satellite Transmission of Courses
Training courses transmitted via satellite are currently offered through the Office of Air
Quality Planning and Standards (OAQPS). During calendar year 1993, approximately 500
ATFERM Final Report Page 1-2
-------
hours of programming have been broadcast. It is expected that 100 Regional and State
facilities will eventually be capable of receiving the courses following an initial investment of
roughly $9,000 for the downlinking equipment.
Transmission costs are currently $2,500 per day for use of the uplinking facilities and
approximately $500 per hour for satellite time. However, the system is converting to the use
of compressed digital signals, and transmission costs are projected to drop to $100 to $125
per hour. Given the large audience which can be reached, the use of satellite courses is
extremely cost effective. In addition to cost considerations, satellite viewing of training
courses may be more conducive to learning than some self study methods since they offer the
opportunity for interaction with other students as well as the trainer.
Recommendation: The Agency should continue to support and investigate the expanded use
of satellite training courses.
Direct Support Services
Often, the Regions and States need individualized ad hoc assistance for particular model
applications. Direct support may include activities such as software consultation, providing
advice on data needs, or assisting with model output interpretation. This type of assistance is
essential for the day-to-day operation of programs which utilize environmental modeling.
Integrated Model Evaluation System (IMES) and Exposure Models Library on CD-
ROM
ORD's Integrated Model Evaluation System (IMES) is available to users as an aid in
selection of appropriate fate and dispersion models. This is a PC-based system which helps
select suitable models in the various media (air, surface water, ground water, non-point
source, and multi-media) for a user's particular scenario or site. The system assists the user
in selection by matching model capabilities to site characteristics and information needs. It
also contains an extensive database on model applications, validation, and features.
The Exposure Models Library on CD-ROM is an extension of the IMES. The disc (with a
storage capacity of close to 600 MB) allows for an efficient, cost effective, and
environmentally sound distribution mechanism of the: various models (and their
documentation and user manuals) used in EPA programs. It contains many of the models
described in the IMES and includes source code as well as sample input and output files.
Recommendation: The Agency should continue the support of IMES and the Exposure
Models Library as both a tool in model selection and a vehicle for training on initial
modeling concepts.
ATFERM Final Report Page 1-3
-------
ORD Technical Support
Historically, model distribution and much technical support has been provided by ORD field
offices. For example, the Corvallis, Oregon, field office of ERL-Newport supports EPA
marine plume models and answers general questions on mixing zone theory. The Center for
Exposure Assessment Modeling in Athens, Georgia, offers support for a variety of widely
used exposure assessment models. The Center for Subsurface Modeling Support (CSMoS) at
the R. S. Kerr Environmental Research Laboratory (RSKERL) in Ada, Oklahoma provides
ground water and vadose zone modeling software and technical support to public agencies
and private companies throughout the nation.
Models developed by these and other ORD offices are tools to assist in developing sound
public policy and support programs' missions. If the models are misapplied or their results
are misinterpreted, unsound program and policy decisions may result.
In the past, ORD technical support and model distribution were often provided by both
Agency and contractor staff. Recent concerns have been raised about the use of contractors
to provide EPA advice in response to user calls. As a result, in some laboratories, on-site
contractor support is no longer used and some technical support activity has been
discontinued.
Recommendation: Program offices that develop models should also provide for their proper
use.
EPA offices (Office of Research and Development, Office of Administration and Resources
Management, Office of Acquisition Management, and Office of Inspector General) should
develop and agree upon appropriate contract guidelines and contract vehicles for providing
model user technical support services.
Technical Assistance Panels
Technical assistance panels (TAP) are composed of Agency and contract personnel with
specialized areas of expertise. The Surface Water Assessment Technical Team or SWAT
Team is one such TAP. The SWAT Team, funded by the Total Maximum Daily Load
(TMDL) program, provides short-term expertise to EPA Regions, States and local
governments that are developing TMDLs. The Regional TMDL Coordinator communicates
all requests for assistance to the Headquarters contact. Then, the Headquarters contact
determines which SWAT Team member would be most suitable for the particular application.
The estimated annual budget for the SWAT Team is $100,000.
The user community originally suggested this concept. So far, customers have been very
pleased with the response time and the results.
Recommendation'. Other program areas should be encouraged to explore the benefits of
establishing panels of experts that could provide short-term technical support.
ATFERM Final Report Page 1-4
-------
Technical Forums/Centers
The Superfund and RCRA Technical Support Project (TSP) uses two Regional Forums,
seven Technical Support Centers and an Environmental Response Team to provide direct
scientific and engineering technical assistance to regional Superfund and RGRA staff.
TheTSP is funded and managed by OSWER's Technology Innovation Office (TIO).
TSPs provide a wide range of support: site visits, the development or review of sampling
plans, the review of contractor work plans and reports, data interpretation, development or
evaluation of remedial alternatives, and the development or review of model applications.
The Engineering Forum and the Ground Water Forum are composed of regional technical
personnel. While neither is focused specifically on modeling, ground water modeling is one
of the concerns of the Ground Water Forum. The two Forums serve as communication
networks for technical information transfer between the Regions and the Centers. They work
to improve technical consistency among the regional Superfund and RCRA programs and are
a source of technical information for the regions, holding semi-annual meetings and monthly
conference calls.
The Technical Support Centers provide technical assistance in response to regional requests.
Questions or other brief requests are phoned directly to the appropriate Center by EPA staff
project managers. More substantive requests (while usually initiated through a phone
conversation) require a written technical assistance request. Since the project began in 1987,
the TSP had responded to over 1,400 requests for technical assistance at Superfund sites.
Recommendations: Program areas other than Superfund and RCRA should explore the
benefits of establishing a similar technical transfer/technical support organization.
To facilitate the formation of additional forums, the Agency should identify broad categories
of environmental models that support local, State and Federal environmental regulations.
Such categories might include ground water, surface water, air, and multimedia models. For
those categories where needs for technical assistance: and for exchange of information are
identified, Regional technical forums should be encouraged.
Information Exchange
In the dynamic field of environmental modeling, new models continue to be developed while
existing models are updated. In addition, many program areas are expanding, resulting in
changes in the type and number of models utilized by program staff. As a result, there is an
increasing need for information exchange. The following tools are among those which may
be used to facilitate this exchange.
ATFERM Final Report Page 1-5
-------
Technical Guidance Manuals
Technical guidance manuals for each program area are essential for providing information
about models. Distribution of guidance manuals is a common method of providing
information to the States and the general public. The primary drawback is that the available
manuals are often out of date.
Recommendation: Technical guidance manuals are essential tools for technology transfer.
The resources necessary for updating these manuals should be provided. These manuals
should be incorporated with the IMES and Exposure Models Library CD-ROM.
Bulletin Board Systems
Bulletin Board Systems can be a cost effective means of providing a variety of services. For
example, a number of models can be downloaded from a bulletin board system in a matter of
minutes, while obtaining these models by mail can take several weeks. Bulletin boards are
also often used for notifying the modeling community of training opportunities and the
availability of recent publications. Users can access bulletin boards directly via modem or
indirectly via local area networks (LANs) that connect to them.
Recommendation: Use of interactive Internet nodes, as well as electronic bulletin board
systems should be encouraged as a means for distributing information such as model
availability and training opportunities, thus providing access to the public as well as Federal
and State personnel.
Technical Library
Professional publications and journals also play an important role in information transfer.
Given the high cost of dues, many government employees, both Federal and State, do not
maintain memberships in professional societies or personally subscribe to trade publications.
However, it is imperative that Agency personnel remain abreast of developments in their
fields.
Recommendation: Since an adequate university library is not always available or convenient,
the Agency should compile a list of professional journals and periodicals which should be
available at every EPA library.
Clearinghouses
The Office of Air Quality Planning and Standards Model Clearinghouse is the focal point for
reviewing specific proposed actions which involve interpretation of guidance on the use of air
quality models or deviations from that guidance. The Clearinghouse functions within the
organizational structure of EPA and communicates with the Regional Offices. Any
ATFERM Final Report Page 1-6
-------
coordination with State and Local agencies or individual sources on Clearinghouse activities
is the responsibility of the Regional Offices. This is handled in three ways:
(1) The Clearinghouse, at the request of a Regional Office, reviews the proposed
use of a "non-guideline" model for technical soundness and national
consistency.
(2) The Clearinghouse screens State implementation plans for adherence to
modeling policy, identifies and recommends resolutions.
(3) The Clearinghouse communicates significant decisions involving the
interpretation of modeling guidance to regulatory model users through an
annual "Clearinghouse report" that identifies significant decisions and the
circumstances involved. This report serves to improve consistency in future
decisions and is a source of technical information for the Regional Offices.
Recommendation: Clearinghouses similar to the Office of Air Quality Planning and
Standards Model Clearinghouse should be encouraged. This clearinghouse serves an
important function by providing guidance that merges the technical and policy aspects of
model applications.
The ORD Integrated Model Evaluation System (IMPS) is another example for a centralized
model information system applicable to all media models. It contains information on model
validation studies/status, uncertainty analyses, an extensive bibliographic database, and details
of model applications.
Recommendation: The IMES should be further enhainced to incorporate the acceptability
criteria (discussed elsewhere in this report), and could be considered (to models) as
analogous to the way IRIS (Integrated Risk Information System) is to chemical compounds.
The Agency should adopt guidelines for preparing reports on model applications and for
formulating policies requiring such reports to be prepared and made available through a
clearinghouse or technical library.
As stated under "Technical Forums," the Agency should identify broad categories of
environmental models that provide support for local, State, and Federal environmental
regulations. An inventory of applications should be developed which defines and implements
the paradigms appropriate to each of the categories. Distribution of this inventory of
applications could be accomplished via a clearinghouse and/or through EPA libraries.
Joint Federal Interagency Model Clearinghouse
Numerous Federal agencies (e.g., USGS, USAGE, USDA, etc.) support the development,
distribution, and use of environmental models because they are important scientific tools
which enhance the understanding of environmental problems. For instance, the U.S. Army
ATFERM Final Report . Page 1-7
-------
Corps of Engineers identified mathematical models of ground water flow and contaminant
transport as essential to designing remedial actions for contaminated sites. The U.S. Army
Corps of Engineers prepared a report which characterizes the technical capabilities in
modeling. It identifies three stages of producing a site-specific model of the processes that
control ground water flow and contaminant transport. In addition, the National Research
Council recently recommended developing a research program focused on the application of
ground water modeling in support of remediation efforts by the Corps.
Recommendations: Since many Federal agencies utilize environmental modeling it is
recommended that the concept of a Joint Federal Interagency Model Clearinghouse be
explored. This clearinghouse would be responsible for model distribution and would direct
the user to the appropriate Agency and office for technical support.
Resource Needs
Adequate resources, including technical personnel and computer hardware and software, are
required to support applications of a reasonable set of environmental regulatory models
within the Agency. At present, the availability of technical personnel and computer
hardware and software varies from Region to Region and program to program. Therefore, a
review of the existing resources throughout the Agency is necessary. In addition, the
resource needs of the model user community should be identified and options for meeting
those needs should be developed.
Minimum requirements for computer hardware and software should be developed and a
suitable contract vehicle made available for procuring those products. The requirements and
equipment should be routinely evaluated to ensure that the equipment reflects the current
technology.
Recommendation: The Agency should conduct a requirements survey and identify options for
filling the needs of model users; such needs include technical personnel and computer
hardware and software.
Immediate needs for technical support may be addressed by intra-Agency teaming as has
been, the case in the implementation of the SWAT team for TMDL development.
ATFERM Final Report Page 1-8
-------
SECTION H - MODEL USE ACCEPTABILITY CRITERIA
Introduction
The use of mathematical models for environmental decision making has increased
significantly in recent years. Modeling has become an important methodology in support of
the planning and decision-making processes involved in environmental management. Within
the U.S. Environmental Protection Agency (EPA) environmental models are being used to
support rule-making decisions and regulatory impact assessments.
One of the questions managers continue to ask when models are utilized is "Is this an EPA-
accepted model?" This question is also asked by model reviewers. The response that a
model code has previously been used does not fully answer the question. Model codes are
accepted or rejected on a case-by-case basis. Except for the Office of Air Quality Planning
and Standards (OAQPS), there really is no formal mechanism to evaluate model acceptability
nor is there a list of EPA-accepted models. Without the establishment of a set of model code
selection criteria and a process to evaluate model codes, users and decision-makers will
continue with re-evaluations and uncertainty about the acceptability of models being applied
and the results therefrom. These constant examinations are redundant and introduce
inconsistencies.
A comprehensive set of criteria for model selection could reduce inconsistency in model
selection. A process to identify acceptable model codes categorized by application niche (and
potentially to identify those that are preferable and those that are redundant) will ease the
burden on the Regions and States applying the models to their programs. The Science
Advisory Board (SAB) has also voiced similar concerns about the Agency's application and
use of models.
In 1988, the SAB (SAB-EC-88-040, SAB-EC-88-040A) recommended that EPA formalize
mechanisms for the review and acceptance of environmental models for all media. SAB's
"Resolution on Use of Mathematical Models by EPA for Regulatory Assessment and
Decision-Making" (SAB-EEC-89-012) reiterated this recommendation and made as one of its
main points: "There is a need for a central coordinating group within the EPA to assess the,
status of environmental models currently used or proposed for use in regulatory assessment
and to provide guidance in model selection and use by others in the Agency".
Clearly, development of a mechanism to determine acceptability of environmental models is
only one aspect of improving the Agency's use of environmental models for regulation..
Establishment of acceptability criteria, establishment of model documentation requirements,
and identification of application niches are other steps that need to be taken. The issue of the
appropriate application of a model, that is, the selection of models that satisfy the particular
scenario and the match between model capabilities, site characteristics and information needs,
is a related but separate issue that must be determined on a case-by-case basis. Model code
may be deemed acceptable genetically, yet its application in a particular situation can be
ATFERM Final Report Page II-l
-------
unacceptable. For example, a model code's acceptability is determined within the context of
a particular application niche (or niches); outside of that niche the model may not be
acceptable. Even within an appropriate application niche, a large discrepancy between the
model's data requirements and the available site data would suggest that the model may not
be the best choice for the specific situation. Finally, selection of an acceptable model for a
particular situation does not prevent the misuse of the model during its application.
Definitions
The definition of "model" is somewhat ambiguous. The term is used variously as a synonym
for conceptual model, mathematical model, computer model and simulation models. The
term "model" may apply to either a computer code without site-specific data, or to the
representation of a specific system using such a generic code, together with pertinent data.
For this report, a model is defined as a non-unique, simplified, mathematical description of
the physical system, coded in computer programming language together with a quantification
of the simulated system (in the form of boundary conditions, system and process parameters,
and system stresses). The generalized computer code (software) usable for different site- or
problem-specific simulations is referred to as a model code or generic model. A Conceptual
Model is an interpretation or working description of the characteristics and dynamics of a
physical system. The Application Niche of a model code is the particular prototype physical
system or set of defining conditions for which the model code is scientifically defensible.
The objective of code Verification is to check the correctness and accuracy of the
computation algorithms used to solve the governing equations, and to assure that the
computer code is fully operational. Verification of a model code is separate from
verification of a model application. Model application verification involves testing the
accuracy and predictive capabilities of the calibrated model on a data set independent of the
data set used for calibration.
The objective of Validation of a model is to determine how well the mathematical
representation of the physical processes of the model code describes the actual system
behavior. Given the uncertainties of conceptualization and parameter estimation inherent in
environmental modeling, models are never literally validated, instead they are invalidated.
Calibration is the process of adjusting model parameters within physically defensible ranges
until the resulting predictions give a best possible good fit to observed data.
Approaches Considered
Several programs of the Agency including the Office of Air Quality Planning and Standards
(OAQPS) and the Office of Solid Waste and Environmental Response (OSWER) have
considered the issue of model acceptability criteria and approaches to the question of model
selection. From these initiatives, much can be learned.
ATFERM Final Report Page II-2
-------
Air Quality Models
OAQPS has developed air quality models suitable for regulatory application, QAQPS, in
response to the requirements of the Clean Air Act and in response to comments to include
other models in its Guideline on Air Quality Models, developed some criteria (published in
March 1980 Federal Register notice) which additional models had to satisfy before they
would be considered by the Agency. This is a mechanism by which non-EPA models can be
considered for inclusion in the Guidelines. Solicitation of new, improved models is
continuous. Under this program, models submitted to the Agency for consideration and
evaluation must meet the following criteria.
1. The model must be computerized and functioning in a common FORTRAN language
suitable for use on a variety of computer systems,
2. The model must be documented in a user's guide that outlines the appropriate theory
upon which it is based, identifies the mathematics of the model, data requirements and
the program operating characteristics at a level of detail comparable to that available
for currently recommended models.
3. The model must be accompanied by a complete test data set including input
parameters and output results. The test data must be included in the user's guide as
well as provided in computer-readable form,
4. The model must be useful to typical users. Such users should be able to operate the
computer program(s) from available documentation,
5. The model documentation must include a comparison with data or with other well
established analytical techniques.
6. The developer must be willing to make the model available to users at reasonable cost
or make it available for public access through the National Technical Information
Service; the model can not be proprietary.
Submittal of models for consideration also requires specific information on any related
documentation concerning past applications and performance of the model. Models are
organized into several categories and subjected to intensive evaluations by category or
application niche. The evaluation process includes statistical measures of model performance
in comparison with measured air quality data and, where possible, peer review. When a
single model is found to perform better than others in a given category, it is recommended in
the Guidelines on Air Quality Models for application in that category as a preferred model,
If no model is found to clearly perform better through the evaluation exercise, then the
preferred model is selected on the basis of other factors such as past use, public familiarity,
cost or resource requirements, and availability. The models not specifically recommended
for use in a particular category are summarized in a separate index of the Guidelines: these
ATFERM Final Report Page 11-3
-------
models must be subjected to a case-specific evaluation before they can be used for regulatory
applications. Other models may be used but require justification on a case-by-case basis
using consistent pre-established model evaluation procedures.
It must be emphasized that these criteria represent the minimum requirements for Agency
consideration. Models meeting these criteria then undergo evaluation by the Agency's
experts and a judgement on their acceptability is made.
OSWER Assessment Framework
During the past few years, The Office of Solid Waste and Emergency Response (OSWER)
has pursued activities related to improving the management of modeling in the Superfund and
RCRA programs. Products of this effort have included the "Ground-Water Modeling
Compendium", the "Assessment Framework for Ground-Water Model Applications", and a
survey of model use in the waste management programs. The Assessment Framework
addresses the use and review of ground-water model applications. It contains a series of
assessment criteria focus on the activities and thought processes that should be a part of a
model application. Within the Assessment Framework are factors related to model code
selection. Relevant criteria include:
The reliability of the model code should be assessed including a review of:
Peer review of the model's theory (or the publication of the theory in a peer-
reviewed journal)
Peer reviews of the model's code
Verification studies (evaluation of the model results against laboratory test,
analytical solutions, or other well accepted models)
Relevant field tests (application and evaluation of the model to site-specific
conditions for which extensive data sets are available)
The model's (code) acceptability in the user community as evidenced by the
quantity and type of use.
The usability of the model code should be assessed including the availability of:
The model binary code
The model source code
Pre- and post- processors
Existing data resources
Standardized data formats
Complete user instruction manual
Sample problems
ATFERM Final Report Page E-4
-------
Necessary hardware
Transportability across platforms
User support
Key assumptions
Acceptability Criteria
Before an environmental model code is used as a planning or decision-making tool, its
credentials should be established through a systematic and detailed examination of the generic
model relative to a comprehensive set of selection criteria. Model selection is more than just
choosing the "best" model of a physical system. The selection process must also take into
account such real-world constraints as available resources for the project. Acceptability must
be determined within the context of the appropriate application niche and modeling objectives
(e.g. screening study or detailed investigation).
A selected model must first accurately represent the significant features of the physical
system being considered. Passing this test, the next aspect of model selection is the cost and
effort required for implementation, which must be balanced against the potential benefits to
be achieved by the model. There is obvious tension between an optimal technical
representation of the physical system and the cost of implementation, as increasing model
resolution (fineness of temporal and spatial scale of model prediction) and model accuracy
usually involve increased expense and effort. A model selection decision of acceptability
must reflect both the characteristics of the site and pollutant and also reflect available
resources and acceptable levels of uncertainty.
Proposed criteria for models code acceptability are presented below. These criteria are
generic; it is anticipated that the criteria will be refined for specific media (air, surface
water, ground water). While many of the elements in the set of criteria are objective, other
elements are more subjective. Appropriate performance measures need to be established for
the subjective criteria. The criteria fall into four broad categories: Appropriateness (is the
model code appropriate for the physical system being described?), Accessibility (what is the
availability of the code?), Reliability (credibility, verification, validation, and performance
evaluations), and Usability (relating to the functional and operational attributes of the code).
A draft report prepared for ORD (Current and Suggested Practices in the Validation of
Exposure Assessment Models. Office of Health and Environmental Assessment, September
30, 1987) considered ways to evaluate and rank reliability of models in more detail than is
appropriate here.
1. Appropriateness
a. What is the model's application niche? Why was this model code developed?
What programmatic questions are the model code attempting to answer?
ATFERM Final Report Page II-5
-------
b. What are the strengths, weaknesses and applicability of this model relative to
its application niche? How does this model perform relative to other models
in this application niche?
2. Accessibility
a. Is the model code (source code and executable, compiled version) available
free or at a nominal cost? If model results will be used to form expert
opinions, all parties should have access to the source code of the model.
3. Reliability
a. Has the theoretical basis for the model code been peer reviewed?
b. Have the algorithms and methods of solution of the model code been peer
reviewed? Have the program structure and program logic of the model code
been peer reviewed?
c. Has the model code undergone verification testing (evaluation of model results
against laboratory tests, analytical solutions, synthetic test data sets, or other
well accepted models)?
d. Has the model code been applied and tested against relevant field data?
e. What is the model code's acceptability in the user community? Does there
exist a number of successful applications of this model code? Have successful
peer-reviewed applications of this code been published?
f. How well does the model code perform in terms of accuracy, bias and
precision?
g. How well does the model code perform in terms of convergence, stability of
solution algorithms, and the absence of terminal failures?
4. Usability
a. Is the model code well structured and internally documented? Where possible,
self-explanatory parameter, variable, subroutine, and function names should be
used.
b. Is full model documentation available? Good documentation should include:
i. Model description (including intended application niche(s), theoretical
framework, underlying assumptions, and limiting conditions)
ATFERM Final Report Page II-6
-------
ii. Model input data definitions, descriptions, and format requirements
iii. Type and format of output data provided
iv. Description of routines, source listing
v. Complete execution instructions, error message explanation, trouble-
shooting guide and description of hardware requirements
vi. Sample model runs (input data, sets and output results)
vii. Code verification and validation information.
c. Is user support available for this model code (including user training and
continuing support and development of the model code)?
d. What kinds of data are required for running the model? To what extent are
these data available? How critical to the model is the quality of the data?
e. Are pre- and post-processors available for this model code?
g. How tedious are the model setup and input preparation processes?
h. What computational resources are required to run the model code? How
computationally efficient is the model code (i.e. What is the achieved
numerical accuracy versus memory requirements and code execution time)?
Information about the various models' codes their application niche(s) their capabilities and
data requirements, and how they satisfy the acceptability criteria could be compiled and
incorporated into a "Model Information System." This system would be an updated source
of model information accessible to EPA staff, Stale staff, and contractors (possibly via an
electronic bulletin board). Information on new models and on revisions of existing models
should be added to the information system as such information becomes available.
Potentially there would be a separate database for each media (air, surface water,
groundwater) with cross references to a database for multimedia models.
The "Model Information System" should build on already existing databases and resources
such as the databases developed by the EPA-sponsored International Ground Water Modeling
Center, EPA's Integrated Modeling Evaluation System (IMES), EPA's Center for Exposure
Assessment Models (CEAM), and EPA's Center for Subsurface Modeling Support
(CSMOS). Such a Model Information System should be an cross-program effort; its overall
management could be the responsibility of a permanent Agency wide Environmental Modeling
Committee.
ATFERM Final Report . Page II-7
-------
Conclusions and Recommendations
Environmental models are increasingly being used to support regulatory decisions and impact
assessments. There is a continuing need to improve the approach to modeling for
environmental decision-making by the EPA and to reduce the potential for environmental
model misuse. The issue of model acceptability for regulatory use is one of the more critical
aspects of understanding and considering model use in the Agency.
There is a strong need for the Agency to provide guidance to users and decision-makers for
selecting environmental models to support planning and decision-making efforts of the
Agency. The set of criteria for model acceptance proposed in this report is the foundation
for such guidance. Creation of a "Model Information System" that includes information on
models (air, ground water, and surface water) and how the models satisfy the points of the
acceptability criteria is a necessary adjunct to establishing the criteria. A set of criteria
without any readily accessible source of model information would be of little value. The
Agency should institute a group (permanent Agencywide Environmental Modeling
Committee) to establish model acceptability criteria (including performance measures) and a
"Model Information System".
There is also a need for the Agency to systematically assess environmental models being used
to support rule-making decisions and regulatory impact assessments. While establishing a
model code evaluation process is relatively simple conceptually, it is logistically complex and
staff-intensive. Establishment and implementation of such a process would require a level of
commitment considerably beyond that required to establish a "Model Information System".
Careful consideration of the appropriateness and resource requirements of any evaluation
process is needed. Following establishment of acceptability criteria and the Model
Information System (MIS), the permanent Agencywide Environmental Modeling Committee
should consider the issue of implementing a model code evaluation process and upgrading the
MIS.
ATFERM Final Report Page II-8
-------
NOTE
The following Guidance has been updated since April 24, 1994
when it was transmitted to the Deputy Administrator, to reflect
the latest (June 7, 1994} statement regarding EPA's Peer Review
Policy.
-------
-------
SECTION III - AGENCY GUIDANCE FOR CONDUCTING EXTERNAL PEER
REVIEW OF ENVIRONMENTAL REGULATORY MODELING
I. INTRODUCTION
According to EPA's Peer Review Policy Statement dated June 7, 1994, (see attachment):
Major scientifically and technically based work products related to Agency decisions
normally should be peer-reviewed. Agency managers within Headquarters, Regions,
laboratories and field components determine and are accountable for the decision
whether to employ peer review in particular instances and, if so, its character, scope,
and tuning. These decisions are made in conformance with program goals and
priorities, resource constraints, and statutory or court-ordered deadlines. For those
work products that are intended to support the most important decisions or that have
special importance in their own right, external peer review is the procedure of choice.
Peer review is not restricted to the penultimate version of work products; in fact,
peer review at the planning stage can often be extremely beneficial.
By contrast, the policy specifically excludes "non-major or non-technical matters that Agency
managers consider as they make decisions." ,
Clearly, environmental models (i.e., fate and transport, estimation of contaminant
concentrations in soil, groundwater, surface water and ambient air, exposure assessment) that
may form part of the scientific basis for regulatory decision-making at EPA are subject to the
peer review policy. However, it cannot be more strongly stressed that peer review should
only be considered for judging the scientific credibility of the model including applicability,
uncertainty, and utility (including the potential for mis-use) of results, and not for directly
advising the Agency on specific regulatory decisions stemming in part from consideration of
the model output. *
The purpose of this guidance is to provide a resource for those program managers
responsible for implementing the peer review process. More specifically, this guidance is
provided as an aid in evaluating the need and, where appropriate, conducting external peer
review related to the development and/or application of environmental regulatory modeling.
This specific guidance for modeling has been prepared to complement general peer review
guidance currently being developed by the Agency's Council of Science Advisors (Council).
Thus, Section II (framework for peer review) and Section IV (peer review mechanisms and
general criteria) reflect the latest Council guidance, and will be revised in the future as
Agency-wide guidance on peer review evolves.
The relationship of external peer review to the process of model development and
application, including consideration of peer review at various stages in the process js
described in Section III of this guidance.
ATFERM Final Report ' Page III-l
-------
Section V concerning "Documentation of the Peer Review Process" has been included in
response to comments from the Agency's Science Advisory Board who cited the need for
more detailed guidance on the mechanics of the review process.
The specific elements of what could be covered hi an external peer review of model
development and application are presented hi Section VI of this guidance. These elements
are not meant to be prescriptive or limit the nature of peer review, but rather are intended as
an aid to improve the thoroughness and consistency of peer review.
To reiterate one of the major recommendations in the March 1992 report commissioned by
former Administrator William K. Reilly re: "Safeguarding the Future: Credible Science,
Credible Decisions (The Report of the Expert Panel on the Role of Science at EPA)":
"Quality assurance and peer review should be applied to the planning and results of
all scientific and technical efforts to obtain data used for guidance and decisions at
EPA, including such efforts in the program and regional offices. Such a requirement
is essential if EPA is to be perceived as a credible, unbiased source of environmental
and health information, both hi the United States and throughout the world."
In conclusion, this document is intended to provide guidance to program managers in their
efforts to consider, and apply where appropriate, external peer review to environmental
regulatory modeling. The guidance contained hi this document must not be construed as
rigid requirements.1
H. FRAMEWORK FOR PEER REVIEW OF ENVIRONMENTAL REGULATORY
MODELING
Peer review can be an important tool hi assisting the Agency to document the quality and
credibility of the science upon which its regulatory and policy decisions are made. Modeling
to provide the scientific support for environmental regulatory decision-making at EPA can be
thought to conceptually involve three stages including: (A) model development;
(B) model application; and (C) consideration of modeling results in decision-making.
External peer review, as discussed hi more detail below, is generally relevant to the first
stage of model development and may be relevant hi appropriate cases to the second stage of
model application. Although external peer review is not directly germane to the regulatory
or policy decision itself, it is important at this third stage to bring forward information
regarding prior peer review comments and the Agency's response related to model
development and/or application. This information may aid the decision maker in interpreting
The guidance set out in this document is not final Agency action. It is not intended, nor can it be relied upon, to
create any rights enforceable by any party in litigation with the United States. EPA officials may decide to follow
the guidance provided herein, or act at variance with the guidance, based on an analysis of specific circumstances.
The Agency also reserves the right to change this guidance at any time without public notice.
ATFERM Final Report Page III-2
-------
and weighing the utility of modeling results along with all the other considerations (e.g.,
field data; risk, cost, and benefits information; requirements to use best available
technology; environmental justice issues; etc.) in reaching a regulatory decision.
Broader guidance regarding the applicability of peer review, as discussed in the Introduction,
is currently being developed by the Council of Science Advisors. As this broader guidance
develops and evolves hi the future, the framework for external peer review of environmental
regulatory modeling, as discussed below, will be revised.
A. Model Development
Models are developed for a variety of reasons, including:
1. use as research tools to explore new scientific issues;
2. simplification and/or refinement of existing model paradigms or software;
3. use as screening tools; and
4. to estimate compliance with regulatory requirements (e.g., National Ambient
Air Quality Standards).
This guidance document, by its terms, does not directly address models developed for
reasons other than to support regulatory decision-making (e.g., research tools). Models
developed expressly for and used exclusively within a research program should be subject to
essentially the same review process as other research results (e.g., informal critique by
scientific colleagues, formal appraisal by senior scientists and managers, publication in
refereed journal, etc.). However, if the purpose of a research model is expanded hi the
future to develop scientific information for Agency decision-making, then the program
manager should consider arranging for reassessment of the model hi accordance with the
guidance presented below.
From a general scientific perspective, a well-conceived model is one that provides an
acceptable mathematical approximation of a physical, chemical, biological, social or
economic system. Model development frequently touches on the interface involving the
state-of-the-art in several areas including: (1) the technical and scientific understanding of
processes and mechanisms; (2) applicable solution techniques (e.g., analytical, numerical);
(3) computer science and technology; and (4) the Agency's need for scientific decision-
support tools. The goal of model development is to provide methods which can be applied to
improve Agency analysis and decision-making. As such, the development of environmental
models cannot be performed in a scientific vacuum. There must be a coherent class of
applications hi mind against which the appropriateness of the science can be judged.
At the model development stage, a key step is to define and compare to existing models, the
set of conditions under which the use of a model is scientifically defensible - this is known as
ATFERM Final Report Page III-3
-------
the "application niche." Peer review of model development would be expected to include
evaluation of the application niche, along with consideration of uncertainty and other areas of
model performance. Approaching peer review from this perspective should help the decision
maker understand the limitations of the scientific basis of the model and confidence in its
results. It is only with this firm knowledge that the Agency can develop sound regulatory
and policy decisions.
External peer review of major technical issues related to environmental regulatory modeling
is generally a lengthy process. This is precisely why external peer review should be
identified as an important and integral aspect of an action plan for model development. At
first glance external peer review might be expected to slow down the process. However,
initiating peer review at early stages may, hi fact, save time by redirecting misguided
initiatives, identifying alternative approaches, or providing strong technical support for a
potentially controversial position.
15.. Model Application
The middle stage, model application, begins with examining the stated application niche and
its applicability to current needs and/or exploring whether a model can be tailored to fit a
new niche. For existing models, especially models developed outside of EPA, peer review
may be appropriate to the extent that either: 1) new information becomes available which
calls into question the appropriateness of the previously defined application niche; or 2) a
model might be considered for application outside the niche for which it was originally
developed.
Peer review of a model's applicability should, where possible, be planned well in advance of
any decision-making involving use of the model's results. In this way, the formation of
"sound science" is distinct from the regulatory decision in which it is considered. The
results of such a peer review can aid hi the ultimate judgment by a decision maker to
consider whether or how to use a particular model's result in the regulatory process.
Normally, the first application of a model should undergo peer review. For subsequent
applications, a program manager should consider the scientific/ technical complexity and/or
novelty of the particular circumstances as compared to prior applications (also see Section
IV-B). Peer review of all similar applications should be avoided because this would likely
waste precious tune and monetary resources while failing to provide the decision maker with
any new relevant scientific information upon which to base a regulatory or policy selection.
Nevertheless, a program manager may consider conducting peer review of applications upon
which costly decisions are based or applications which are likely to end up in litigation.
As an alternative to peer review of model application, the Office of Air Quality Planning and
Standards has developed a formal review and acceptance procedure through rulemaking to
evaluate the utility of alternative models for a particular application.
£L Environmental Regulatory Decision-Making
ATFERM Final Report Page III-4
-------
The final stage involves consideration of modeling results in the decision-making process.
This stage may include consideration of: (1) selecting among available models and
assumptions/standard defaults; and/or (2) whether/how to consider modeling results in the
regulatory process. Information previously derived from peer review of scientific issues may
provide key information for the decision maker hi understanding the uncertainties and utility
(and potential for mis-use) of modeling results. When possible, public discussion of the
scientific and technical underpinnings associated with the earlier stages of model development
and/or application hi advance of the final stage of regulatory decision-making is expected to
reduce criticism regarding the Agency's use of "sound science."
Environmental regulatory and policy decisions also involve other scientific and non-scientific
factors, and are by law, required to be made by the responsible Agency decision makers.
Thus, such decisions are not appropriate subjects for scientific peer review. Rather, the
process of public comment is frequently employed by the Agency at this stage.
III. RELATIONSHIP OF EXTERNAL PEER REVIEW TO THE PROCESS OF
ENVIRONMENTAL REGULATORY MODEL DEVELOPMENT AND
APPLICATION
The Agency's peer review policy notes that "properly applied, peer review not only enriches
the quality of work products but also adds a degree of credibility that cannot be achieved in
other way. Further, peer review early in the development of work products hi some cases
may conserve future resources by steering development along the most efficacious course."
Since this subject guidance focuses on the role of external peer review, its relationship to
other levels of peer review can best be understood, by considering where external peer review
may fit into the total process of environmental regulatory modeling. The folio whig paradigm
is meant to illustrate the general process:
Step 0: The program manager within the originating office (AA-ship or Region)
identifies elements of the regulatory process which would benefit from
the use of environmental models. A review/solicitation of currently
available models and related research should be conducted. If it is
concluded that the development of a new model is necessary, a
research/development work plan would be prepared.
Step Ob: The program manager may consider internal and/or external peer
(optional) review of the research/development concepts to determine whether they
are of sufficient merit and whether the model is likely to achieve the
stated purpose.
Step 1: The originating office develops a new or revised model or evaluates the
possible novel application of model developed for a different purpose.
Step Ib: The program manager may consider external peer review of the
ATFERM Final Report Page III-5
-------
(optional) technical or theoretical basis prior to final development, revision or
application at this stage. For model development, this review should
evaluate the stated application niche.
Step 2: Initial Agency-wide (internal) peer review/ consultation of model
development and/or proposed application may be undertaken by the
originating office.
Model design, default parameters, etc. and/or intended application are
revised (if necessary) based on consideration of internal peer review
comments.
Step 3: External peer review is considered by the originating office.
Model design, default parameters, etc. and/or intended application are
revised (if necessary) based on consideration of external peer review
comments.
Step 4: Final Agency-wide evaluation/consultation may be implemented by the
originating office. This step should consist of consideration of external
peer review comments and documentation of the Agency's response to
scientific/technical issues.
(Note: Steps 2 and 4 are relevant where there is either an internal Agency standing or ad
hoc peer review committee or process).
A program manager may decide that peer review (step 2 -internal, and step 3 - external)
should take place at more than one time during the processes of model development and
model application (i.e., optional steps Ob and/or Ib). This decision would depend on the
nature and complexity of scientific issues that are presented.
Additionally, a program manager may also consider the utility of employing the broader
concept of peer involvement that is, to augment staff efforts by soliciting subject-matter
experts from outside the program.
Based in part on the results of a scientific peer review of model development and application,
as well as other non-technical issues, the program manager would be responsible for advising
the decision maker as to the consistent, equitable, and appropriate use of the model and its
output for environmental regulatory purposes.
IV. MECHANISMS AND GENERAL CRITERIA FOR CONDUCTING EXTERNAL
PEER REVIEW
The mechanisms and general criteria for conducting external peer review presented below are
based upon generic peer review guidance currently being developed by the Council of
ATFERM Final Report Page III-6
-------
Science Advisors. As this broader guidance develops and evolves in the future, this Section
will be revised.
A.. Peer Review Mechanisms
Mechanisms for accomplishing external peer review include, but are not limited to, the
following:
i. Using an ad hoc technical panel of at least three scientists;
ii. Using an established external peer review mechanism such as the Science
Advisory Board or Scientific Advisory Panel; or
iii. Holding a technical workshop.
Qualifications for peer reviewers will likely vary for model development versus model
application. For the former, the emphasis may be: toward modelers, while for the latter, the
emphasis may be toward scientists with technical expertise in other disciplines (e.g.,
statistics, field monitoring, etc.).
New models, or significant modifications and/or new applications of established models,
should be considered for publication in refereed journals. However, this step should be
supplemental to and not a substitute for the peer review mechanisms presented above.
B.. General Criteria
General criteria to be considered for determining when and by what mechanism to initiate an
external peer review of the development and/or application of environmental regulatory
models include:
i. Use of model results as a basis for major regulatory or policy/guidance
decision-making;
ii. Significant investment of Agency resources;
iii. Inter-Agency or cross-Agency implications/applicability;
ATFERM Final Report Page III-7
-------
iv. Treatment of a new scientific issue; or ,
v. Novelty and/or complexity of the science.
V. DOCUMENTATION OF THE PEER REVIEW PROCESS
It is important to prepare documentation for each peer review conducted and make this
available to the decision maker along with the modeling results. The record should include
the following information:
A. Identity of the peer reviewers, their relevant expertise, and their institutional
affiliation;
B. Questions/issues posed to the peer reviewers;
C. Reviewer comments, either as a collection of individual statements or as a
consensus statement; and
D. Agency response to peer review comments including rationale.
VI. SPECIFIC ELEMENTS OF EXTERNAL PEER REVIEW FOR
ENVIRONMENTAL REGULATORY MODELING
This Section of the guidance addresses the specific elements that should be considered to
provide the Agency with consistent scientific/technical external peer review of environmental
regulatory model development and/or application.
Such elements could include, but are not limited to:
A., Model Purpose/Objectives
The first step hi evaluating a model used for environmental regulatory purposes is to clearly
understand the broad context in which a model is intended to be used.
i. What is the regulatory context in which the model will be used and what broad
scientific questions is the model intended to answer (e.g., evaluating the range
of human respiratory exposure resulting from air toxics release from a power
plant)?
ii. What is the model's application niche (i.e., the particular physical, chemical,
and/or biological system and set of defining conditions for which the model is
expected to be scientifically defensible)?
ATFERM Final Report Page III-8
-------
iii. What are the model's strengths and weaknesses? How well does the model fit
its intended application niche compared to existing models and available data
sets? It is important that the Agency avoid duplicating prior efforts.
B.. Major Defining and Limiting Considerations
With a clear understanding of the broad purpose and objectives of a model in.a regulatory
context, the scientific context needs to be addressed.
i. Which processes are characterized by the model (e.g., transport, diffusion,
chemical reactions, removal mechanisms, etc.)?
ii. What are the important temporal and spatial scales? Is the grid resolution
appropriate for the problem?
iii. What is the level of aggregation?
C.. Theoretical Basis for the Model
Once the regulatory and scientific contexts of a model have been defined, the basis for
problem solving must be formulated.
i. What algorithms are used within the model and how were they derived? What
is the mechanistic basis? .
ii. What is the method of solution (numerical, analytic)?
iii. What formulations are used for those processes which are parameterized?
iv. How does the basis for problem solving compare to existing models? What is
the scientific rationale?
v. What are the shortcomings of the modeling approach (e.g., missing or over-
simplification of key processes, restrictive dimensionality, etc.)?
D. Parameter Estimation
Parameter estimation may be based on case specific data or in their absence, on default
values. In the latter case, it is important to understand how parameter defaults were
established.
i. What methods were used for parameter estimation?
ii. What data were available for parameter estimation?
ATFERM Final Report Page III-9
-------
iii. What methods were used to estimate parameters for which there were no data?
iv. What is the reliability of parameter estimates?
v. What are the boundary conditions and are they appropriate?
R Data Quality/Quantity
All models require the input of various types and amount of data. Models may also rely on
experimental data to help shape their computational algorithms. In large measure, the utility
of a model for regulatory purposes depends on the quality, quantity, and spatial and temporal
adequacy of data used hi its design and hi support of its application.
Questions related to model design include:
i. What data were utilized hi the design of the model?
ii. How can the adequacy of the data be defined hi terms of quality, quantity, and
spatial and temporal applicability taking into account the regulatory objectives
of the model?
Questions related to model application include:
i. What kinds of data are required to apply the model?
ii. To what extent are these data available and what are the key data gaps?
iii. Have data quality objectives been defined? If so, are they scientifically
defensible?
iv. Is the quantity of data sufficient to address the likely variability? What
statistical analyses were performed and are they appropriate?
v. To what extent are the data suitable with regard to estimating spatial and
temporal effects?
vi. Do additional data need to be collected and for what purpose?
ATFERM Final Report Page IIMO
-------
R Key Assumptions
The applicability of a model depends on the adequacy of its basic underlying assumptions.
i. What are the key assumptions?
ii. What is the basis for each key assumption and what is the range of possible
alternatives?
iii. How sensitive is the model toward modifying key assumptions?
G. Model Performance Measures
The most basic test of a model's adequacy is to understand how well its results compare with
real world measurements.
i. What criteria have been used to assess model performance?
ii. Did the data bases used hi the performance evaluation provide an adequate test
of the model in terms of applicability to the modeling niche?
iii. How accurate can the model be expected to perform? Does the model exhibit
any overall bias throughout the range of its predictions? Bias is an important
test of the model's formulation since intrinsic system uncertainty is not
present. .
iv. How well does the model address, distinguish, and report variability and
uncertainty in its output? Which parameters and key assumptions are most
significant in determining the model's variability and uncertainty?
v. How does the model perform relative to other models in this application niche?
H. Model Documentation and Users Guide
The utility of model for regulatory purposes depends on the availability of a clear
documentation report and a comprehensive users guide. Do these cover:
i Model applicability and limitations?
ii. Data input?
iii. Interpretation of results? and
iv. Documentation of the model code and! other key aspects such as verification
testing?
ATFERM Final Report , PageIII-11
-------
L Retrospective
A retrospective analysis of the- "big picture" may sometimes reveal insights that an analysis
of individual components of a model may miss.
i. Does the model satisfy its intended scientific and regulatory objectives?
ii. Is there any available scientific evidence to suggest changes to either the model
design and/or key parameters and assumptions prior to its use for regulatory
purposes?
iii. How robust (i.e., not overly sensitive toward small changes in modifying key
assumptions or input data) are the model predictions?
iv. How well does the model output quantify the overall uncertainty resulting from
limitations/simplifications hi its design; use of standard assumptions;
availability of supporting data; etc.?
v. What key research is necessary to refine or improve the model and/or the data
bases upon which it relies?
The elements provided above are not meant to be prescriptive or limit the nature of external
peer review. Rather the purpose for their inclusion is to provide modeling-related guidance
for peer review protocols which will be consistent with more general guidance being
developed by the Council of Science Advisors.
Attachment 1: "Peer Review and Peer Involvement at the U.S.
Environmental Protection Agency" dated June 7, 1994
ATFERM Final Report Page 111-12
-------
PEER REVIEW AND PEER INVOLVEMENT
AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY
This document establishes the policy of the United States
Environmental Protection Agency (EPA) for, peer review of
scientifically and technically based work products that are
intended to support Agency decisions. Peer review is presented in
the context of the broader concept, peer involvement.
BACKGROUND
The report "Safeguarding the Future: Credible Science,
Credible Decisions"1 focused on the state of science at EPA. The
panel of experts who prepared the report emphasized the importance
of peer review, especially external peer review, and the need for
broader and more systematic use of it at EPA to evaluate scientific
and technical work products. Their specific recommendation
regarding peer review reads as follows:
"Quality assurance and peer review should be applied to
the planning and results of all scientific and technical
efforts to obtain data used for guidance and decisions at
EPA, including such efforts in the program and regional
offices. Such a requirement is essential if EPA is to be
perceived as a credible, unbiased source of environmental
and health information, both in the United States and
throughout the world." ,
In response to this recommendation, then-Administrator Reilly
directed staff to develop an EPA-wide policy statement, which he
issued in January, 1993. The paragraphs below preserve the core of
that earlier statement while updating: it to specify the role of the
Science Policy Council in guiding further implementation of the
policy. Effective use of peer review is indispensable for
fulfilling the EPA mission and therefore deserves high-priority
attention from program managers and scientists within all pertinent
Headquarters and Regional Offices.
EPA/600/9-91/050, March 1992.
-------
PEER INVOLVEMENT AND PEER REVIEW
EPA strives to ensure that the scientific and technical
underpinnings of its decisions meet two important criteria: they
should be based upon the best current knowledge from science,
engineering, and other domains of technical expertise; and they
should be judged credible by those who deal with the Agency. EPA
staff therefore frequently rely upon peer involvement that is,
they augment their capabilities by inviting relevant subject-matter
experts from outside the program to become involved in one or more
aspects of the development of the work products that support
policies and actions.
One particularly important type of peer involvement occurs
when scientifically and technically based work products undergo
peer review that is, when they are evaluated by relevant
experts from outside the program who are peers of the program
staff, consultants, and/or contractor personnel who prepared the
product. Properly applied, peer review not only enriches the
quality of work products but also adds a degree of credibility that
cannot be achieved in any other way. Further, peer review early in
the development of work products in some cases may conserve future
resources by steering the development along the most efficacious
course.
Peer review generally takes one of two forms. The review team
may consist primarily of relevant experts from within EPA, albeit
individuals who have no other involvement with respect to the work
product that is to be evaluated (internal peer review) . Or the
review team may consist primarily of independent experts from
outside EPA (external peer review).
POLICY STATEMENT
Major scientifically and technically based work products
related to Agency decisions normally should be peer-reviewed.
Agency managers within Headquarters, Regions, laboratories, and
field components determine and are accountable for the decision
whether to employ peer review in particu'ar instances and, if so,
its character, scope, and timing. These decisions are made in
conformance with program goals and priorities, resource
constraints, and statutory or court-ordered deadlines. For those
work products that are intended to support the most important
decisions or that have special importance in their own right,
external peer review is the procedure of choice. Peer review is
not restricted to the penultimate version of work products; in
fact, peer review at the planning stage can often be extremely
beneficial.
-------
SCOPE
Agency managers routinely make regulatory and other decisions
that necessarily involve many different considerations. This
policy applies to major work products that are primarily scientific
and technical in nature and may contribute to the basis for policy
or regulatory decisions. By contrast, this policy does not apply
to nonmajor or nontechnical matters that Agency managers consider
as they make decisions. Similarly, this policy does not apply to
these ultimate decisions.
This policy applies where appropriate, as determined by the
National and Regional Program Managers, to major scientifically and
technically based work products initiated subsequent to the date of
issuance. Peer review should be employed to the extent reasonable
to relevant work products that currently are under development.
This policy does not apply to the bases for past decisions, unless
and until the relevant scientific and technical issues are
considered anew in the Agency's decision-making processes.
Except where it is required by law, formal peer review (as
distinguished from the Agency's normal internal review procedures)
should be conducted in a manner that will not cause EPA to miss or
need extension of a statutory or court-ordered deadline. Agency
managers still may undertake peer review if it can be conducted
concurrently with necessary rulemaking steps.
LEGAL EFFECT
This policy statement does not establish or affect legal
rights or obligations. Rather, it confirms the importance of peer
review where appropriate, outlines relevant principles, and
identifies factors Agency staff should consider in implementing the
policy. On a continuing basis, Agency management is expected to
evaluate the policy as well as the results of its application
throughout the Agency and undertake revisions as necessary.
Therefore, the policy does not stand alone; nor does it establish
a binding norm that is finally determinative of the issues
addressed. Minor variations in its application from one instance
to another are appropriate and expected; they thus are not a
legitimate basis for delaying or complicating action on otherwise
satisfactory scientific, technical, and regulatory products.
Except where provided otherwise by law, peer review is not a
formal part of or substitute for notice and comment rulemaking or
adjudicative procedures. EPA's decission whether to conduct peer-
review in any particular case is wholly within the Agency's
discretion. Similarly, nothing in this policy creates a legal
requirement that EPA respond to peer reviewers. However, to the
extent that EPA decisions rely on scientific and technical work
products that have been subjected to peer review, the remarks of
peer reviewers should be included in the record for that decision.
-------
IMPLEMENTATION
The Science Policy Council is responsible for overseeing
Agency-wide implementation. Its responsibilities include promoting
consistent interpretation, assessing Agency-wide progress, and
developing recommendations for revisions of the policy as
necessary.
The Science Policy Council will oversee a peer-review work
group, which will include representatives from program units
throughout EPA to effect a consistent, workable implementation of
the policy. The work group will assist the programs in (1)
formulating and, as necessary, revising standard operating
procedures (SOPs) for peer review consistent with this policy; (2)
identifying work products that are subject to review; and (3) for
each major work product, selecting an appropriate level and timing
of peer review.
In assisting the programs, the work group will take into
account statutory and court deadlines, resource implications, and
availability of disinterested peer reviewers. The group will work
closely with Headquarters offices and the Regional Offices toward
ensuring effective, efficient uses of peer review in supporting
their mission objectives. However, the Assistant Administrators
and Regional Administrators remain ultimately responsible for
developing SOPs, identifying work products subject to peer review,
determining the type and timing of such review, documenting the
process and outcome of each peer review, and otherwise implementing
the policy within their organizational units.
Because peer review can be time-consuming and expensive,
Agency managers within Headquarters, Regions, laboratories, and
field components are expected to plan carefully with respect to its
use taking account of program priorities, resource
considerations, and any other relevant constraints as well as the
policy goal of achieving high-quality, credible underpinnings for
decisions. External peer reviewers should be chosen carefully to
ensure an independent and objective evaluation. The affiliations
of peer reviewers should be identified on the public record, so as
to avoid undercutting the credibility of the peer-review process by
conflicts of interest.
This policy is effective immediately. The peer-review work
group mentioned above will identify the focal point to whom
comments and questions should be addressed and, from time to time,
will provide further information about implementation activities.
APPROVED: >~t -t^L^U^t^^^- DATE: JMM 7
CAROL M. BROWNER, ADMINISTRATOR
-------
SECTION IV - PROPOSED CHARTER FOR A PERMANENT COMMITTEE ON
REGULATORY ENVIRONMENTAL MODELING
Rationale For A Permanent Group
Problem Statement
EPA decision-making is supported by a growing number of computerized environmental
models. Use of such models has been prompted by the increasing complexity of the
environmental programs EPA manages; by the fact that task-specific data almost invariably
are incomplete and sometimes unobtainable within the constraints of technology, time, and
other resources; and by the sophistication of the regulated community. Use of such models
has also been fueled in part by wider access among Agency personnel to desktop computers
capable of running such programs.
While the value of computerized environmental models to EPA's policy and regulatory
decision-making is indisputable, the widespread and informal employment of highly
sophisticated, often narrowly focused models can lead to misuse and misinterpretation. Such
failings undermine EPA's commitment to sound science. This must be avoided.
To that end the following issues require attention and action at the highest levels within each
Office:
The Agency needs a formal internal process by which to judge the quality and
acceptability of environmental models;
EPA needs a recognized body Agency-wide to perform or confirm such
judgments;
The Agency needs a standard methodology for model development;
The evolution and life-cycles of models utilized by EPA should be defined and
managed consistently; and
The level of experience and the type of training needed to properly use and
interpret a model should be known and documented.
A permanent Committee on Regulatory Environmental Modeling is proposed to guide and
facilitate these and related efforts.
Charter Draft 1 Page IV-1
-------
Ad Hoc Task Force
EPA management recognizes the importance of these models and their appropriate
applications. The Science Advisory Board issued its modeling resolution in 1989. In
response, the Assistant Administrators of the Offices of Solid Waste and Emergency
Response and of Research and Development requested formation of a task force on
environmental modeling. In March 1992, EPA's Deputy Administrator established an ad
hoc Task Force on Environmental Regulatory Modeling.
The Task Force was charged "to complete within twelve months a recommendation to the
Agency on specific actions it should take to satisfy the needs for improvement in the way
that models are developed and used in policy and regulatory assessment and decision-
making."
During the timeframe, the Task Force developed products on:
Agency Guidance for Conducting External Peer Review of Environmental
Regulatory Modeling
Acceptability Criteria for Model Use
Technical Support Needs
Permanent Group for Environmental Modeling
The Deputy Administrator also requested the Task Force to "report on the advisability of
establishing a permanent Agency-wide Expert Panel on Environmental Modeling." In
response to this request, the Task Force has developed the following proposal of establishing
a permanent group as well as language for a draft charter.
The Task Force recommends establishing a permanent group to focus on environmental
modeling. To address the unmet needs of the Agency, the permanent group should have the
following attributes:
Authority/ Speaks for Agency on inter-Agency modeling activities
Autonomy Coordinates with all offices and regions
Coordinates with other Agency-wide science groups, (e.g., Science
Advisory Board, Risk Assessment Forum, Council of Science Advisors,
etc.)
Develops policy options for consideration by the new Science Policy
council (SPC)
Visibility Reports to the SPC through its Steering Committee
Resources Has committed and available resources to support activities
Charter Draft 1 Page IV-2
-------
In developing the draft charter, the Task Force reviewed lists of elements found in existing
charters. The elements included in the draft charter for the permanent group on
environmental modeling are based on a review of the:
Risk Assessment Forum Charter
Environmental Monitoring Management Council Charter
Council of Science Advisors Interim Charter
Charter of the former Risk Assessment Council
The Risk Assessment Forum Charter was the primary model for drafting the environmental
modeling charter.
Charter Draft 1 Page IV-3
-------
-------
Draft Proposed Charter
U.S. Environmental Protection Agency
Committee on Regulatory Environmental Modeling
Charter
Purpose
The Committee on Regulatory Environmental Modeling (CREM) is established to promote
consensus on mathematical modeling issues and to ensure that this consensus is incorporated
into appropriate modeling guidance.1 The purpose of the Committee is to provide the
Agency with consistent yet flexible modeling tools to support environmental decision making.
To fulfill this purpose, the Committee assembles modeling experts from throughout the
Agency to study and report on these issues from a cross media perspective.
Scope of Activity
CREM Activities may include:
providing expertise to the EPA in developing Agency-wide positions on
modeling issues;
providing a central point of focus to address cross media issues that are
beyond the purview of any single program office; and
promoting consistency in model development and application in order to
eliminate duplication, confusion, and uncertainty, thus empowering personnel
to develop sound scientific input for consideration in environmentally-related
decision-making.
As used in this Charter, the terms "modeling" and "model(s)" are limited to the regulatory environmental context.
Other modeling activities such as those associated with economics and demography are outside the scope of CREM.
Charter Draft 1 Page IV-4
-------
Responsibilities
Responsibilities of the Committee on Regulatory Environmental Modeling are defined to
accommodate the dynamic nature of model development, technology, and advances in
environmental management. Responsibilities include:
Agency Positions:
Cross Media:
Consistency:
Agency Expertise:
Empowering EPA
Personnel
developing and articulating the Agency response to significant
modeling issues
facilitating resolution of modeling issues
identifying Agency modeling priorities and long term
strategies for the Agency
developing modeling guidance for adoption by the Agency
responding to and addressing concerns regarding the
determination of levels of confidence in a wide range of
models in different media
facilitating modification and application of models to address
health and ecological risks that are not chemical specific
participating in Agency review and peer review of modeling
efforts
promoting scientific integrity of the modeling process across
the Agency
promoting consistency in model guidance, testing, and
evaluation
promoting continuity in the technical support and distribution
of models
promoting the establishment of protocols for post audits of
applications
responding to requests from the Deputy Administrator,
Assistant and Regional Administrators, and the SPC
promoting continuity in the availability of modeling
information
identifying issues appropriate for the Committee on
Regulatory Environmental Modeling
encouraging the development of easier access to and use of
models as well as training and technical support
promoting more common interfaces for setting up model runs
promoting a repository for model application and use within
the Agency
Charter Draft 1
Page IV-5
-------
Membership
The overall composition reflects a balance of scientific disciplines and Agency experience.
Selection of members is based on experience and expertise in environmental modeling and
underlying disciplines, such as groundwater, drinking water, hazardous waste engineering,
exposure assessment, surface water, air dispersion, information systems, and computer
software. The Committee on Regulatory Environmental Modeling has representatives from
each of the major programs and regional offices.
Criteria
Members should satisfy one or more of the following criteria:
1. The member should have an advanced degree(s) in engineering, science or
mathematics, and/or an established record of peer-reviewed publication on
model development or application, and/or substantial experience in utilization
of models,
2. The member should have at least three years of actual (i.e., "hands-on")
experience with mathematical models. The experience should involve
activities that include developing, reviewing, or using models or model
applications.
3. The member should have at least three years' experience in working with
environmental issues.
4. The member should have knowledge of the organizational structure of EPA, its
science policies, and Regional operations.
Membership Term
Members must be appointed by an Assistant Administrator or a Regional Administrator who
commits to support participation by the member. The membership term is two years.
Members may serve multiple terms without limit.
Organization
Reporting/Authority
The Committee on Regulatory Environmental Modeling is established by the EPA Deputy
Administrator in his role as Chair of the Science Policy Council (SPC). The Committee
reports through the SPC's Steering Committee. Amendments to the charter must be approved
by the SPC.
Charter Draft 1 Page IV-6
-------
The Committee reports are referred to the SPC for consideration of policy and procedural
issues. Committee products or recommendations become Agency policy upon acceptance and
concurrence by the Deputy Administrator as Chair of the SPC.
Chair
The Chair of the Committee is appointed by the Deputy Administrator in his role as Chair of
the SPC and serves as liaison to the SPC's Steering Committee.
Executive Secretariat
The Executive Secretariat for the Committee works under the direction of the Chair and
manages and directs staff members to assure that all logistical and operational needs are met.
Meetings
The Committee on Regulatory Environmental Modeling meetings are held on a regular basis,
at least quarterly. Meeting summary reports are reported from each meeting to the SPC. An
agenda is distributed by the Committee staff at least one week before each meeting, along
with any papers to be reviewed at that meeting.
Subcommittees or work groups may be formed by the Committee to accomplish short term
tasks or address ongoing issues. The Chairpersons of such subcommittees, working with the
Committee staff, are responsible for processing their reports and for timely presentation to
the full Committee on Regulatory Environmental Modeling.
Procedures for the Committee on Regulatory Environmental Modeling
Decisions
Decisions on issues are the result of consensus from members of the Committee and its
subcommittees and work groups. Decisions of the Committee are submitted to the SPC
through it Steering Committee (hereafter abbreviated as SPC/SC) for concurrence.
Selecting Committee Issues
New issues are proposed at any time by any member and may also originate from the
SPC/SC.
The Committee is responsible for developing a yearly operating plan that specifies tasks,
activities, and products and is consistent with the Agency Strategic Plan.
Each year, the Committee presents the operating plan and budget to the SPC for review and
approval.
Charter Draft 1 Page IV-7
-------
Products
Processing Committee Products
The Chairperson of each subcommittee or work group is responsible for developing and
adhering to an approved work plan. The subcommittee or work group coordinates with the
Committee staff throughout development of the project and briefs the Committee at regular
intervals.
Reviewing Products
Products, their peer review comments, and their responses are submitted to the SPC/SC for
evaluation of programmatic and regional impacts.
Distributing Products
Completed Committee products are transmitted to the SPC/SC and disseminated by it
throughout the Agency.
In general, draft products are available to non-participants only with the approval of the
Committee on Regulatory Environmental Modeling.
Public Participation
Committee meetings are not required to be open to the public, but the Committee may
sponsor public workshops, as appropriate.
Charter Draft 1 Page IV-8
-------
APPENDIX I
Options And Conclusions
In response to the Deputy Administrator's request to "report on the advisability of
establishing a permanent Agency-wide Expert Panel on Environmental Modeling," the Task
Force developed the following options to analyze the advantages and disadvantages of
establishing a permanent group.
Bases on this analysis, the Task Force concluded and recommends that a permanent
group be established.
Options for Task Force Continuation
Option
Advantages
Disadvantages
No Permanent Group
No apparent resource
commitment
No organizational changes
Continued inconsistency and
duplication of current efforts
No EPA body to develop
standards and guidance for the
Agency
No consistency in model
development
No cross media perspective
Ad Hoc Group
Formal EPA body to focus on
modeling issues
Convene when needed
Identify objectives and tasks to
accomplish within a specific
timeframes
Timeframe limits issues that can
be addressed
No ongoing Agency group to
provide expertise or focus
Unable to establish and move
toward long range goals
Permanent Group
Formal EPA body to focus on
modeling issues
Ongoing Agency support and
expertise
Continuity in Agency guidance
and standards
Focus on cross media issues
Promote quality science
Requires resource commitment
Charter Draft 1
Page IV-9
-------
Options for Placement of Committee
After concluding that a permanent group is the best alternative for addressing modeling needs
within EPA, the Task Force examined two options for placement of the group within the
Agency's structure: (1) a technical panel under the Risk Assessment Forum ; and (2) a
Committee on Regulatory Environmental Modeling reporting directly to the Science Policy
Council.
The Task Force concluded that an SPC-related Committee is the better option based on
the following advantages: (1) the focus is not limited to risk assessment issues; (2) SPC is
able to provide an Agency-level policy review of modeling issues; and (3) SPC provides a
cross-media basis for coordination of intra-Agenc.y and inter-Agency modeling activities.
Options for Placement of Executive Secretariat
The Task Force examined the following options for the placement of the Executive
Secretariat and support staff: (1) AA-ship of the Chair; (2) ORD AA's office; and (3)
rotation among the AA's Offices.
The Task Force prefers option (2), the ORD AA's office, since a similar science group,
the Risk Assessment Forum, is placed there.
Resources
The Task Force recommends providing the Coimmittee on Regulatory Environmental
Modeling with two full time equivalent (FTE) positions and $300,000 for its first two
years.
The rationale, a proposed estimated budget, and a descriptions of the projects are discussed
below in Appendix II.
Source of Resources
The Task Force examined a number of alternatives for the source of funding for the
Committee on Regulatory Environmental Modeling, including (1) the Administrator's budget;
(2) pro-rated percentage of resources from all AAs and RAs; and (3) the AA-ship of the
Chair.
The Task force prefers option (2), the pro-rated percentage of resources from all AAs
and RAs, since it reflects the widespread use of models across the Agency.
Charter Draft i Page IV-10
-------
APPENDIX n
Resources
A resource commitment is necessary to the successful establishment of any Agency-level
modeling group. Resources will support the permanent assignment of an executive director
and staff and may also provide for contractor assistance.
The Task Force examined the resources of the Risk Assessment Forum which, nine years
ago, initially required $140,000 and 1.5 full time employee (FTE) position. Today its
current resources are one million dollars and nine FTE.
The Task Force recommends providing the Committee on Regulatory Environmental
Modeling with two full time equivalent (FTE) positions and $300,000 for its first two years.
The proposed estimated budget is as follows:
Fiscal Years 1994 through 1996
1. PC&B for two full time equivalent (FTE) positions with additional funds for travel and
other administrative costs
The Committee on Regulatory Environmental Modeling, with its workgroups and/or
panels will address the following initiatives:
Refine the model acceptability criteria, peer review it, secure approval, and
distribute as Agency guidance.
Examine the Air Program's model management program and identify any
aspects which could be instituted Agency-wide.
Distribute the External Peer Review Guidance prepared by ATFERM. Work
to promote widespread awareness and acceptance of the guidance.
Prepare strategic and tactical plans for future Council activities, including an
analysis of the resources necessary to achieve them.
2. Funds for contracts and/or Inter-Agency Agreements to support:
An inter-program workshop to present the Task Force's findings and to
continue the dialogue about model use within the Agency. The workshop
would also provide program offices with the opportunity to present information
on their use of modeling and modeling issues.
$40.000
Charter Draft 1 Page IV-11
-------
Initiation of the development of an Agency inventory of environmental
regulatory models which relate to EPA's programs, meet the draft criteria
standards now set, and have identified technical support, documentation, and
training. Include in this inventory the Air Program's approved models.
$95.000
Establishment of a communications and awareness program regarding the
modeling products available.
$35.000
Survey of current modeling technical support and training programs. Identify
alternatives for improving such support, particularly in light of new
technologies. Care must be taken to ensure the proper mix of technical
support and training for each program office, since the most effective
combination may vary between offices. Development of a long-term strategy
and a short-term action plan for an Agency initiative on technical support for
modeling. Implementation of some short-term activities where feasible.
$130.000
Charter Draft 1 Page IV-12
-------
-------
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON. 0 C 20460
MAR 2832
MEMORANDUM
SUBJECT: Task Force on Environmental Regulatory Modeling
,1/1 /A/00 ,A^
FROM: F. Henry Habicht Jl^^C-^^
Chairperson, RisloAssessmeni: Council
Deputy Administrator
TO: Assistant Administrators
Regional Administrators
The Environmental Protection Agency uses numerous computer
models to support decision-making. These models originate from a
number of sources and have a vide variety of users. In 1989, the
Science Advisory Board (SAB) issued a resolution on modeling,
citing the need for improvement in th« way that decision-making
needs are factored into the model development process and in the
way that models are applied in actual decisions. The Offices of
Solid Waste and Emergency Response and of Research and
Development have examined these issues and concur with the SAB.
To respond to these needs, the Risk Assessment Council is
creating an ad hoc Agency Task Force on Environmental Regulatory
Modeling. The purpose of the Task Force will be to complete
within 12 months a recommendation to the Agency on specific
actions it should take to satisfy the needs for improvements in
the way that models are developed and used in policy and
regulatory assessment and decision-making.
In its deliberations, the Task Force should explicitly
consider and develop where appropriate the following:
o acceptability criteria for model use
generally and in particular circumstances
o formal technical and policy guidance on model
development
o Agency requirements for peer review and for
documentation of models prior to use
o expansion of training and technical support
activities for EPA personnel who oversee
model applications
-------
- 2 -
I encourage the ad hoc Task Force to examine and evaluate
any other activities that seem appropriate. In particular, the
Task Force should report on the advisability of establishing a
permanent Agency-wide Expert Panel on Environmental Modeling.
Such a panel could assume the responsibility for updating the
criteria, guidelines, and requirements, and for overseeing the
adequacy of training and technical support activities.
Please designate a representative to serve on the ad hoc
Agency Task Force. Ensure that the representative you designate
for the Task Force has the education and experience described on
the attached list of criteria. The Task Force will convene
within six weeks and is expected to complete its assignment
within twelve months.
This is a high priority initiative, affecting our ability to
make correct and defensible regulatory decisions. Kelp to ensure
its success by selecting an appropriate representative for the ad
hoc Task Force, and by supporting the results of its efforts
including consideration of a permanent Expert Panel.
Please provide the name of your designated representative to
Asa R. Frost, Jr., Director of OSWER Information Management (OS-
110, FTS 260-6760) within three weeks of the date of this
memorandum.
Att»chm.nt
VTS
-------
ATTACHMENT
CRITERIA FOR MEMBERSHIP
IN THE RISK ASSESSEMENT COUNCIL AD HOC TASK FORCE
ON ENVIRONMENTAL REGULATORY MODELING
.1. The member is to be appointed by an Assistant
Administrator or a Regional Administrator.
2. The member should have an advanced degree(s) in
engineering/ science or mathematics, and/or an
established record of peer-reviewed publications on
model development or application, and/or substantial
experience in utilization of models.
3. The member should have at least three years of" actual
(i.e., "hands-on") experience with mathematical models.
The experience should involve activities which include
developing, reviewing, or using models or model
applications.
4. The member should have at least three years' experience
in working with environmental issues.
5. The member should have knowledge of the organizational
structure of EPA, its science policies and Regional
operations.
. The member should be clear on the intended level of
detail of the study and be prepared (i.e., have the
ability and the time) to support the technical effort,
as well as the willingness to travel occasionally.
Specifically, the member should be prepared and make
available at least a time commitment of 5% of work time
during the period of the study,
-------
-------
United States Of flee of the Administrator EPA-SAB-EEC-89-012
Environmental Protection Science Advisory Board January 1989
Agency Washington, DC 20460
ReP°rtof
the Environmental
Engineering Committee
Resolution on Use of
Mathematical Models
by EPA for Regulatory
Assessment and
Decision-Making
-------
-------
January 13, 1989
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON. D C 20450
EPA-SAB-EEC-89-012
Honorable Lee M. .Thomas OFF.CEOF
, . rHE ADMINISTRATOR
Administrator
U.S. Environmental Protection Agency
401 M Street, S.W.
Washington, D. C. 20460
Dear Mr. Thomas:
The Environmental Engineering Committee of the Science
Advisory Board has prepared the attached resolution for your
consideration on the use of mathematical modeling for regula-
tory assessment and decision-making. This is the second time the
Science Advisory Board has acted on the issue of modeling; a 1984
letter called the Agency's attention to this important concern.
Over the last few years the Environmental Engineering
Committee has reviewed a number of EPA environmental modeling
studies. In doing so, the Committee has noted a number of
problems in the development and implementation of models within
the Agency that were common to modeling efforts sponsored by a
variety of offices. The Committee believed that these common
problems would be best called to the Agency's attention through
a more general resolution on modeling.
Drafts of the resolution were presented and widely
discussed at a series of Committee and Executive Committee
meetings during 1988. For instance, an earlier draft of the
resolution was quoted at length in the Radiation Advisory Com-
mittee's recent report on the sources and transport of radionuclides,
while encouraging the overall approach of quantitative risk
assessment and modeling for environmental decision-making, this
Committee noted a number of common problems in the use of models
by the Agency. The following items summarize the main points
that are addressed in the attached resolution:
1. There should be a better balance between field and
laboratory data collection efforts and modeling analysis
for effective environmental assessment;
2. Models for regulatory assessment and decision-making
which incorporate state-of-the-art scientific under-
standing of the environmental processes involved should
be developed and used;
3. There should be better confirmation of models with
laboratory and field data;
-------
4. Sensitivity and uncertainty analysis of environmental
models and their predictions should be conducted to
understand level of confidence in model predictions, as
well as to identify key areas of future study;
5. An Agency-wide task-group to assess and guide model use
by EPA should L><* formed;
6. There should be an increased effort to hire and support
engineers and scientists with modeling development and
application skills;
7. There is a need for systematic management of model use
within EPA and a careful review of emerging technologies
such as personal computer-based models and expert
systems; and
8. Peer review at various levels should be coordinated to
ensure proper development and application of models.
The resolution identifies a number of ways in which the
use of models by the EPA can be improved. The Committee
believes that successful implementation of these recommendations
will require the establishment of a formal institutional
mechanism with responsibility for review, oversicjht and
coordination of model use in EPA.
We are pleased to have had the opportunity to be of service
to the Agency, and look forward to your response on this issue.
Sincerely,
Raymond C. Loehr, Chairman
Executive Committee
Science Advisory Board
Richard A. Conway, Chairman
Environmental Engineering Committee
Science Advisory Board
Attachment: Modeling Resolution
Mitchell J. Smalr, Chairman
Modeling Resolution Subcommittee
Science Advisory Board
cc: John A. Moore
Donald G. Barnes
-------
ABSTRACT
The Environmental Engineering Committee (EEC) of the Science
Advisory (SAB) has prepared a resolution on the use of
mathematical model ing for regulatory assessment and decision-
making. The main points that are addressed in the resolution are
as follows: 1) There should be a better balance between
laboratory and field data collection; 2) Models should be
developed and used which incorporate state-of-the-art scientific
understanding of the processes involved; 3) There should be
better confirmation of models with laboratory and field data; 4)
Sensitivity and uncertainty analysis should be conducted; 5) An
Agency-wide task-group should be formed to assess and guide model
use by EPA; 6) EPA should hire and support engineers and
scientists with modeling skills, 7) Model use needs systematic
management at EPA; and 8) Peer review of models should be
conducted at various levels.
Key Words: models, mathematical models, modeling resolution.
-------
NOTICE
This report has been written as a part of the activities of
the Science Advisory Board, a public advisory group providing
extramural scientific information and advice to the Administrator
and other officials of the Environmental Protection Agency. The
Board is structured to provide a balanced, expert assessment of
scientific matters related to problems facing the Agency. This
report has not been reviewed for approval by the Agency; hence,
the contents of this report do not necessarily represent the
views and policies of the Environmental Protection Agency or of
other Federal agencies. Any mention of trade names or commercial
products do not constitute endorsement or recommendation for use.
-------
Chairperson
ENVIRONMENTAL ENGINEERING COMMITTEE
of the
SCIENCE ADVISORY BOARD
Executive Secretary
* Dr. Raymond C. Loehr
Professor of Civil Engineering
Department
University of Texas
Austin, Texas 78712
Dr. K. Jack Kooyoomjian
Science Advisory Board (A-101F)
Environmental Protection Agency
401 M Street, S.w.
Washington, D.C. 20460
Members
Dr. Joan Berkowitz
President
Risk Science International
1101 - 30th Street, N7W.
Washington, D.C. 20007
Dr. Keros Cartwright
Illinois State Geological Survey
615 Peabody Drive
Champaign, Illinois 61820
** Mr. Richard Conway
Corporate Development Fellow
Research and Development Department
Union Carbide Corporation
P.O. BOX 8361 (770/342)
South Charleston, WV 25303
Dr. Ben B. Ewing
Professor of Environmental Studies
Institute for Environmental
Studies
University of Illinois
408 South Goodwin Avenue
Urbana, Illinois 61801
Dr. William Glaze
Director, School of Public Health
University of California at
Los Angeles
650 Circle Drive South
Los Angeles, California 90024
Dr. Joseph Ling
3 M Company
3 M Community Services
Executive Program
Building 521-11-01
St. Paul, Minnesota 55114
Dr. Donald J. O'Connor
Professor of Environmental
Engineering
Manhattan College
Environmental Engineering
and Science Program
Manhattan College Parkway
Riverdale, New York 10471
Dr. Charles R. O'Melia
Professor of Environmental
Engineering
Department of Geography and
Environmental Engineering
The Johns Hopkins University
Baltimore, Maryland 21218
Dr. Paul V. Roberts
Department of Civil Engineering
Stanford University
Stanford, California 94305-4020
Dr. Thomas T. Shen
New York State Department of
Environmental Conservation
50 Wolf Road, Room 134
Albany, New York 12233
-------
Mr. George P. Green
Manager, Production Services
Public Service Company of Colorado
1800 W. Sheri Lane
Littleton, Colorado 80120
Dr. William Haun
13911 Ridgedale Drive
Suite 343
Minnetonka, Minnesota 55343
Secretary
Mrs. Marie Miller
Science Advisory Board (A101F)
Environmental Protection Agency
401 M Street, S.W.
Washington, D.C. 20460
202/382-2552
Dr. Mitchell J. Small
Assistant Professor
Department of Civil Engineer.
Carnegie-Melon University
Schenley Park
Pittsburgh, Pennsylvania 15213
Dr. Calvin H. Ward
Chairman, Department of
Environmental Science and
Engineering
Rice University
Houston, Texas 77251
* As of November, 1988, Chairman of SAB Executive Committee
** As of November, 1988, Chairman of the SAB Environmental
Engineering Committee
-------
EPA-SAB-EEC-89-012
RESOLUTION ON THE: USE
OF MATHEMATICAL MODELS BY EPA
FOR REGULATORY ASSESSMENT AND DECISION-MAKING
by the
Environmental Engineering (Committee
Science Advisory Board
U.S. Environmental Protection Agency
January 1989
-------
-------
INTRODUCTION
The use of mathematical models for environmental decision making has increased significantly in
/
recent years. The reasons for this are many, including scientific advances in the understanding of certain
environmental processes, the wide availability of computational resources, the increased number of
scientists and engineers trained in mathematical formulation and solution techniques, and a general
recognition of the power and potential benefits of quantitative assessment methods.
Within the US Environmental Protection Agency (EPA) environmental models which integrate release,
transport, fate, ecological effects and human exposure are being used for rule making decisions and
regulatory impact assessments. This report is directed to the development and validation of such models,
an issue which was first addressed in December 1984 by Norton Nelson, Chairman of the Executive
Committee of the SAB. In a letter to the EPA Administrator, William Ruckelshaus, it was recommended
that a systematic effort of model validation be initiated, including an identification of the appropriate
balance between monitoring and modeling. It was further recommended that the relative utility of
exposure modeling approaches be evaluated in the form of case studies in various media including model
validation and uncertainty analysis.
The Environmental Engineering Committee reaffirms and amplifies these recommendations, based on
review of a number of integrated environmental modeling studies during the past few years. Examples
include a review of the report, "Comparison of Risks and Costs of Hazardous Waste Alternatives:
Methods Development and Pilot Studies" (SAB-EEC Report, July 1985); a review of the Code for
Transport in the Unsaturated Zone (FECTUZ) and its potential use for determining whether a waste is
hazardous for listing decisions (SAB-EEC-88-030); a reviow of risk-based regulations for alternative
disposal and reuse options for sewage sludge (SAB-EEC-87-013, SAB-EEC-87-015); a review of the
Underground Storage Tank (UST) Release Simulation Model (SAB-EEC-88-029); and a review of the
draft risk screening analysis for mining wastes (SAB-EEC-88-028). While encouraging the overall
approach of modeling for environmental decision-making by EPA and acknowledging the progress made
by various offices within the Agency, the committee noted a number of problems in the development and
application of models, including: the increased reliance on models rather than background data collection
and analysis, an inadequate level of laboratory and field validation for models employed, a lack of studies
quantifying the uncertainties associated with model predictions, and concurrently, the potential misuse of
particular uncertainty analysis techniques. The following resolutions address these issues, and identify
the need for an institutional mechanism within EPA to ensure their implementation.
-------
RESOLUTIONS
1. A balanced program of field and laboratory data collection and modeling analysis Is
required for effective environmental assessment.
The realistic characterization of an environmental problem requires the collection of laboratory and
field data - the more complex the problem, the more extensive and in-depth are the required studies. In
some cases involving more complex issues, future projections of environmental effects, larger
geophysical regimes, inter-media transfers, or subtle ecological effects, mathematical models of the
phenomena provide an essential element of the analysis and understanding. However, the models
cannot stand alone; adequate data are required. Indeed, a major function of mathematical models is as a
tool to design field studies, interpret the data and generalize the results.
A number of recent studies of integrated exposure and risk reviewed by the committee have exhibited
an over-reliance on models at the expense of the acquisition of needed data. This trend should be
reversed.
2. Mathematical models for regulatory assessment and decision-making should incorporate,
to the extent possible, the state-of-the-art scientific understanding of the environmental problem.
Mathematical models should ideally be based on a fundamental representation of the physical,
chemical and biological processes affecting environmental systems. In the regulatory domain, there may
be a need to sacrifice model complexity and rigor because of inadequate process insight, the need for
computational efficiency, or because of a lack in available supporting data. There should not, however,
be too ready a willingness to abandon fundamental, scientific approaches simply because the required
research and data are too difficult to obtain in a short time-span. If this were done, two undesirable
results would likely occur. First an nproperly formulated model can lead to serious misjudgements
concerning environmental impacts and the effectiveness of proposed regulations. In this regard, a bad
model can be worse than no model at all. Second, by accepting an improperly formulated model, the use
of a weak scientific approach can become institutionalized within the Agency, and the opportunity to
motivate the needed research and data collection can be lost. Rather, shortcomings in process
understanding and available data should serve as an incentive for research and data acquisition to
improve the foundations for models.
-------
It must be recognized that research and data acquisition to support state-of-the-art model
development and validation is a long-term, iterative process involving many scientific and engineering
disciplines. A commensurate, long-term commitment to support this effort is required from the Agency.
3. There Is a need for models used In regulatory applications to be confirmed with laboratory
and field data.
There are a number of steps needed to confirm the accuracy and utility of an environmental model.
As a preliminary step, the elements of the basic equations and the computational procedures employed to
solve them should be tested to ensure that the model generates results consistent with its underlying
theory. The confirmed model should then be calibrated with field data and subsequently validated with
additional data collected under varying environmental conditions. After the particular regulatory program
has been implemented, field surveys and long-term monitoring should be conducted for comparison with
model projections. The stepwise procedure of checking the numerical consistency of a model, followed
by field calibration, validation and a posteriori evaluation should be an established protocol for
environmental quality models in all media, recognizing that the particular implementation of this may differ
for surface water, air and ground water quality models. It is also recognized that the degree and extent to
which the process of validation is conducted for a model depends on the significance of the
environmental issue and the consequence of an erroneous decision concerning the problem.
It is recommended that EPA establish a general model validation protocol and provide sufficient
resources to test and confirm models with appropriate field and laboratory data.
4. Sensitivity and uncertainty analysis of environmental models and their predictions should
be performed to provldo decision-makers with an understanding of the level of confidence in
model results, and to identify key areas for future study.
A number of methods have been developed in recent years for quantifying and interpreting the
sensitivity and uncertainty of models. These methods require careful application, as experience with
uncertainty analysis techniques is somewhat limited, and trwre is a significant potential for misuse of trie
procedures and misinterpretation of the results. Potential problems include the tendency to confuse
model uncertainty with temporal or spatial variation in environmental systems, the tendency to rely on
model uncertainty analysis as a low-cost substitute for actual scientific research, and the tendency to
ignore important uncertainties in model structure when evaluating uncertainties in model parameters. To
address the latter issue, sensitivity analysis of a broader nature is required, considering the impact of
-------
alternative model assumptions and omitted processes. As is the case for model validation, the extent to
which sensitivity and uncertainty analysis should be performed depends upon the importance of the
environmental issue and the relative role of the model in determining the regulatory decision.
Consideration of model sensitivity and uncertainty should be included in all modeling studies. The
implications of errors in model structure, as well as errors in model parameters, should be evaluated to
determine possible effects on the ultimate regulatory decision.
5. There Is a need for a central coordinating group within the EPA to assess the status of
environmental models currently used or proposed for use In regulatory assessment, and to
provide guidance in model selection and use by others In the Agency.
In the selection of mathematical models for regulatory applications, a thorough understanding of the
capabilities, limitations and degree of validation of available models is required. There have been
instances where a model developed for a particular purpose was used in a new application without the
appropriate steps taken to properly adapt and validate the model in the new problem setting. Conversely,
there are cases where available computational programs for models have been ignored and new, but
similar, procedures developed at unnecessary effort and expense. Recognizing the need for improved
model selection and use, the Sources, Fate and Transport Subcommittee of the SAB Research Strategies
Committee (SAB-EC-88-040, SAB-EC-88-040A) recommended that EPA formalize mechanisms for
review and acceptance of environmental models for all media. Methods such as those used by the EPA
Office of Air Quality Planning and Standards (EPA-450/2-78-027R) were recommended. This would
involve identifying tested or recommended models for particular media or environmental settings,
establishing procedures for demonstrating the acceptability of alternative models, and instituting a Model
Clearinghouse to compile and test models, conduct periodic workshops to ensure consistency in
modeling guidance, and promote the use of the most appropriate models and data bases. The
Subcommittee also indicated the need to identify currently applied models where improved validation is
needed, and to develop a priority list for these validation efforts.
To address the issues of model validation and model selection and use within the EPA, we suggest
the establishment of a task-group on mathematical models for environmental quality assessment. Such a
group would evaluate the state-of-the-art of models in each of the media, as well as emerging multi-media
models, evaluate environmental models used by other government agencies, and provide oversight for
model development, validation and application within the EPA. The group would also rank current models
-------
as to their relative importance and.need for further validation studies. This Agency-wide task-group
should be established as soon as possible.
6. EPA must hire and support engineers and scientists with appropriate model development
and application skills.
This issue is closely linked to the recommendation of the SAB Research Strategies Committee that
EPA increase the numbers and sharpen the skills of the scientists and engineers who conduct
environmental research (SAB-EC-88-080). Modeling is not a separate discipline, rather it is a particular
skill that is part of the overall environmental science and engineering approach to problem-solving. There
has been a tendency to allocate the development of models to the computer specialist, who frequently
lacks the .understanding of the basic equations and their significance to the environmental problem.
Similarly, there has been a tendency to presume that the users of models need not understand the basis
for the models. This is incorrect. The proper development and application of models requires engineers
and scientists trained in the fundamental principles of the environmental transport problem and
computational methods, so that they can develop and work with the model in an informed manner, not
just as a black box which is manipulated to obtain numerical output. Note that often the most critical and
effective application of models is made by users not involved in the development of the model, as they
are more likely to question and challenge the implicit perspectives and assumptions of the model
approach. As such, skilled model developers and model users are both required for effective problem-
solving.
The Agency should increase its efforts to hire and retain engineers and scientists who are qualified in
the area of model development and model use, having both broad and problem-specific skills. The EPA
should support their efforts through the program of the Agency task-force on modeling discussed in the
previous resolution.
7. The need for a systematic management of model use wtthln the EPA Is heightened by the
Introduction of new computer systems and modeling technologies.
The wide availability of personal computers has brought increasing numbers of models to an
increasing number of potential users. As a result, the problem of ensuring code validity and proper model
use is that much more difficult. Special challenges are also raised by the growing technology of
knowledge-based expert systems. Expert systems allow the automation of a wide range of scientific
analysis and inference, and are currently being developed for a variety of environmental engineering
-------
problems. EPA should require strict review and critique of expert systems, recognizing that they can, in
many ways, be treated like other environmental models. They are tools to aid the decision-maker, they
must be rigorously confirmed with field data prior to regulatory application, they require a careful
consideration of model sensitivity and uncertainty, and they require trained users familiar with both the
fundamental physical principles of the environmental system being considered and the way in which the
expert system uses this information to arrive at its recommendation for a design or regulatory decision.
The recommended EPA task group on modeling should pay particular attention to emerging
technologies, such as personal computer-based models and expert systems. Careful review, oversight
and validation are needed for these beneficial, but relatively untested approaches to environmental
modeling.
8. Peer review at various levels Is required to ensure proper model development and
application.
Peer review is an essential element of all scientific studies, including modeling applications. Peer
review is appropriate in varying degrees and forms at different stages of the model development and
application process. The basic scientific representation incorporated in the model should be based on
formulations which have been presented in the peer reviewed scientific literature. Ideally, the model itself
and initial test applications should also be presented in peer-reviewed papers. However, this is not
always possible given the pace of scientific development and regulatory need. Peer review panels are
thus often required to review the scientific capabilities of proposed models and their intended applications.
These expert panels should include some combination of internal Agency staff and outside experts.
Innovative approaches to model review should be considered, such as the use of "round robin" reviews in
which the same modeling task is addressed by a number of independent groups, or the use of benchmark
data-sets for testing model accuracy.
The recommended EPA task group on modeling should identify the needs for peer review of models
and establish procedures for coordinating the necessary peer review panels.
SUMMARY
The resolutions presented in this report address critical issues that must be confronted to improve the
use of models by the EPA. These issues include the need for a better balance between data collection
and modeling, the use of state-of-the-art models, the need for model confirmation and sensitivity and
-------
uncertainty analysis, the need for a central coordinating group to provide oversight and guidance on
model use within the Agency, the need for more scientists and engineers with modeling skills, the need
for review of new modeling technologies, and the need for peer review of model development and
application. Many of the recomme'V' Ions in this report can be implemented by individuals and individual
offices within the EPA. This will undoubtedly lead to a more effective use of models by the Agency. The
Committee believes, however, that a full and successful response to these resolutions will require the
establishment of a formal, institutional mechanism which can promote better review and coordination of
model use throughout the EPA. The actual structure of this group and its relationship to previous or
ongoing initiatives is an issue that requires further consideration by the Agency. It is hoped that these
resolutions will provide further motivation and direction for this effort.
-------
------- |