June 1, 1999
EPA-SAB-EC-ADV-99-009
Honorable Carol M. Browner
Administrator
U.S. Environmental Protection Agency
401 M Street, SW
Washington, DC 20460
Subject: Advisory on the Charter for the Council on Regulatory Environmental
Modeling (CREM)
Dear Ms. Browner:
The Environmental Models Subcommittee (EMS), hereinafter referred to as the
"Subcommittee", met February 23 and 24, 1999, to review the draft "Proposed Charter, Council
on Regulatory Environmental Modeling", in a public meeting in Washington, DC. The
Subcommittee concluded that the draft charter provides adequate and appropriate guidance to
help the Agency develop, apply, evaluate and improve, scientifically-based and defensible models
of high quality. The Subcommittee also felt that the proposed CREM charter should help the
Agency communicate its activities to the public, and it made recommendations for improving the
CREM charter.
Further, the Subcommittee supports EPA's proposal to establish the Council on
Regulatory Environmental Modeling (CREM) for the explicit purpose of enhancing the scientific
defensibility of environmental models and also to promote consistency in the development, use
and improvement of models as well as to communicate these activities to the public. The
Subcommittee is unanimous that an entity like the CREM is long overdue and encourages the
Agency to proceed quickly to implementation. Given the past difficulty within EPA of
establishing Agency-wide guidance for model development and use (e.g., the Agency Task Force
on Environmental Regulatory Modeling - ATFERM), the Subcommittee strongly urges EPA
senior management to establish CREM and support its charter.
This recommendation is consistent with past SAB recommendations and continuing
concerns. The SAB has been actively advising the Agency on the use of models in environmental
protection for many years dating back to 1989 (SAB, 1989).
The Board recommended that:
a) EPA establish a general model validation protocol, provide sufficient resources to
test and evaluate models with appropriate field and laboratory data, and
-------
b) EPA form an Agency-wide task-group to assess and guide model use by EPA
We note that the Agency then set up the Agency Task Force on Environmental Regulatory
Modeling (ATFERM) in 1992 and that in the resulting report (EPA, 1994 ) ATFERM concluded
that the Agency had no formal mechanism to evaluate model acceptability and that a
comprehensive set of criteria for model selection was needed to reduce inconsistency in the use of
models by the EPA. If implemented this would also ease the burden on the Regions and States
who apply the models in their programs. The SAB reviewed the ATFERM report and supported
the generic approach to conducting peer review of environmental regulatory modeling, noting that
the "most important element to the review process is the verification of the model against
available data in the range of conditions of interest" with a discussion of compensating errors and
suggesting that guidance be provided concerning adequate model performance (SAB, 1993). The
ATFERM report contained acceptability criteria deemed adequate by the SAB. ATFERM also
prepared "Agency Guidance for Conducting External Peer Review of Environmental Regulatory
Modeling" in 1994 (EPA, 1994) for use across EPA, but the approach was not implemented
within the Agency. The SAB therefore urged the Agency to move forward in consolidating its
gains in modeling (SAB, 1995). In part as a result of this urging, the Agency conducted the
Models 2000 Conference in Athens, GA in December 1997. This conference led to the Models
2000 proposal for the establishment of CREM which the EMS is reviewing in this report. In light
of these events, the Subcommittee commends EPA's proposal for CREM, but it is concerned
about EPA's commitment to the effort and its willingness to follow through and to implement the
CREM.
In this Advisory, the Subcommittee focuses on how the Charter promotes the use of
science with respect to models, and avoids the policy and management aspects of deciding how to
implement a program for evaluating the effectiveness of model use at EPA. However, the
Subcommittee strongly recommends that the CREM be given sufficient authority to do its job, as
well as the appropriate oversight and support from EPA senior management.
Charge 1. Comment on the Adequacy of the CREM Charter to Ensure That Models Are
Useful, Articulate Major Assumptions and Separate Scientific Conclusions from
Policy Judgments
The first charge to the Subcommittee was to "comment on the adequacy of the CREM
Charter for helping the Agency establish a framework to ensure that models developed by the
Agency are useful for their intended purpose, articulate major assumptions and uncertainties, and
separate scientific conclusions from policy judgments".
The CREM charter is largely silent on this issue, especially on the separation of scientific
conclusions from policy judgments. The Subcommittee therefore encourages the Agency to
consider the findings presented in the soon to be published SAB Integrated Risk Project report. It
offers a way to integrate the seemingly disparate factors that must be considered in decision-
making (e.g., science, economics, policy, public values, etc.). The Subcommittee believes that
successful regulatory model development within the Agency requires a clear statement of the
-------
model's designated task. The key for CREM is to collaborate with the Agency Program Offices
and Regions to develop policies and procedures to ensure that the regulatory or other needs that
stimulate model development and use are clearly stated. Model developers should be required to
document the policy and other choices made as they develop EPA models, and likewise model
users should describe the choices they make when they apply the model. Users should also
describe how these choices affect the model output. These same principles also apply to models
developed by others which are to be used by the Agency.
To ensure that regulatory models are useful, CREM should emphasize the role of models
in regulatory problem-solving. Models are tools to help solve problems. An emphasis on
problem-solving will allow EPA to clearly demonstrate how and why they apply models, and to
describe the extent to which EPA models succeed in their assigned tasks. The lessons learned
should be systematically documented for use when models are applied in different situations to
improve existing models, and for future model development.
In considering how CREM might collaborate with the Agency Offices and Regions to help
them implement the policies and procedures CREM develops, the Subcommittee felt that the
CREM might work most effectively if it shares the information it collects, and then encourages
peer influence to effect change. The documentation and promotion of "Exemplary Case Studies"
could be used to highlight the successes of the Agency's activities in developing and applying
models. The intent would be to illuminate general modeling principles applicable to other
modeling activities beyond the specific case studies and purposes for which the models were
originally constructed.
To provide guidance to Agency Program Offices and Regions the Subcommittee felt that
an effective CREM needs to actively monitor how well procedures are followed, and) how well
do they work. CREM also needs to receive feedback and suggestions from Agency Offices and
Regions in order to suggest changes needed to improve modeling procedures to the Science
Policy Council for Agency-wide adoption and use. CREM needs good working relationships with
Agency Offices and the Regions as well as ready access to information concerning the use of
models. Therefore, CREM will need commensurate authority and EPA senior management
support to carry out its chartered responsibilities effectively. A similar collaborative process will
be useful for making changes to the CREM charter as the Agency learns from implementing these
new procedures.
Charge 2. Mechanism to Help Assessors and Managers Ensure That "Unseen" Aspects of
the Assessment Are Brought Forward
In the second charge, the Subcommittee was asked to "comment on the utility of the
Charter for developing a framework under which a mechanism can be developed to help risk
assessors and managers ensure that usually 'unseen' aspects of the assessment are brought
forward to make clear to decision-makers and the public choices made during the modeling and
(the reasons) why they were made".
-------
The Subcommittee interprets this charge to mean that we are being asked to comment on
whether the proposed CREM charter helps promote making transparent the science policy and
other largely non-technical choices made during the development and application of an
environmental regulatory model. The proposed charter does not address this issue, but CREM
could play an important role by providing guidance on how to make clear what are the "unseen"
aspects of a model and ensuring that the case studies recommended earlier address this issue. The
subcommittee recommends that the CREM help the Agency move towards training EPA staff in
the role of models in problem-solving and move away from merely certifying that users have been
trained in the use of various software applications..
There are many ways models may be used to support a decision. They may help scientists
calculate an estimated risk, economists estimate costs and benefits, pollsters analyze the public's
viewpoints on the issue, etc. To the extent that EPA emphasizes certain factors considered during
decision making (e.g., the risk assessment number), and not others (e.g., policy considerations),
Agency assessors, managers, and, equally important, the public are not fully informed about how
specific decisions are reached. The use of models must be viewed in this broader context. To
help promote this recognition, as we have already recommended, CREM could organize its
activities around problem-solving assessment issues, such as: Total Maximum Daily Load
(TMDL) determinations, wasteload allocations, hazardous waste listing/delisting determinations,
site prioritizations, Clean Air Act State Implementation Plan (SIP) approvals, environmental
justice assessments, etc. The Agency's web site might be an appropriate way to do this. The
categories for reporting might then include:
a) Designated task assessment issues and endpoints (which aspects of the problem are
considered, which are not)
b) Problem framing: what are the specific measures used to evaluate these issues and
endpoints
c) State-of-science in understanding the problem and in being able to predict the
impact of a decision or regulation
d) Available data and data needs — data analysis procedures
e) Alternative models available for assessment
f) Basis for choosing a particular model or set of models
These recommendations are similar to those made by SAB's Environmental Engineering
Committee in its Quality Systems report (SAB, 1999).
This would help to put modeling efforts into the broader context of decision support.
Again, as the Subcommittee has already recommended, analysts should be encouraged to be very
explicit about the broad framing of the problem and the roles of science and value judgments in
-------
determining what to include (and model) and what to exclude.
Charge 3. Help Those Outside EPA Understand and Help Evaluate Progress Toward
Achieving Agency Modeling Goals
The third charge to the Subcommittee was to "comment on the utility of the Charter to
help develop goals for the use of models, and a mechanism to help those outside EPA understand
the Agency's models, modeling activities and modeling results and to help them evaluate EPA's
progress toward achieving the Agency's goals".
The Subcommittee recommends that CREM encourage Program Offices and Regions to
collaborate with their constituencies to improve model development and use. The Subcommittee
believes this to be an effective way to help those outside EPA understand and evaluate progress
toward achieving Agency modeling goals. This function should be made explicit in the CREM
charter. An added benefit is that "ownership" will be broadened, and the Agency will be able to
build on the work of others. The section in the charter titled "Empowering the Public" is a good
start towards this goal, and the Subcommittee endorses its inclusion in the proposed charter. In
promoting such outreach CREM should act as a facilitator. It should provide guidance to each
Program Office and Region with regard to their seeking feedback from model users, not only
within the Agency but also from outside the Agency, (including industry, state and local
governments, community groups, academia and environmental organizations). The Agency will,
therefore, need to identify clearly the procedures by which Program Offices and Regions can
effectively gather feedback from the community of model users.
The CREM should likewise act as a facilitator in demonstrating how to share the
information gathered across Agency programs for uses beyond its original intent. As part of this
effort, CREM is strongly encouraged to make the completed Action Team White Papers and
other products public. In fact, careful consideration should be given to persons from outside the
Agency being included as members of these Action Teams. The Subcommittee further
recommends that CREM publish examples of models or model applications that fail evaluation
tests and how EPA learns from these experiences. The payoff for these activities is that the
Agency will be better able to explain and defend its use of environmental regulatory models. EPA
will be able to save resources by leveraging the work of others and enjoy greater benefit from the
money it spends on model development, use and improvement. The practice of sharing
information and cooperation will also help to reduce duplication of modeling effort.
Models are often complex. Therefore, proper training is necessary to ensure effective and
proper use. Because resources for formal training are not always available, the Subcommittee
recommends that tutorials, examples, and case studies be developed for environmental regulatory
models used by EPA. Furthermore, cooperation between EPA, other parts of the public, private
and not-for-profit sectors be encouraged to develop and disseminate training materials. To
-------
summarize, CREM should act as a facilitator in promoting Agency outreach implemented through
the participation of the Program Offices and Regions. However, the CREM itself should not be
charged with performing the function of public outreach.
In addition to the comments in response to the three charge questions, the Subcommittee
recommends adding a sentence on page 2 , under Organization - reporting/Authority, to indicate
who can propose amendments to the CREM charter.
In summary, the Subcommittee strongly urges the Agency to charter and employ CREM
to develop policies and procedures for the development, validation and use of environmental
regulatory models at EPA. The Subcommittee feels that this is necessary and long overdue to
ensure that models used by EPA are of the highest quality and that they are scientifically-based
and defensible. However, the Subcommittee is not convinced that EPA is fully committed and
willing to launch the CREM with the level of senior management support needed for its success.
The Subcommittee believes that a "carrot and stick" approach is the best way for CREM
to accomplish its mission. This can be done by providing incentives and support for those who
provide input and share their modeling efforts through the CREM. In addition, by instituting a
mechanism for full disclosure of modeling activities at the Agency, pressure will be exerted to
improve the quality of these activities. Through a well-designed process of highlighting Agency
modeling efforts in a unique and distinctive manner, CREM can identify where modeling practices
are working well; CREM can also identify gaps and areas that need improvement. To be effective
in this important activity CREM must have input and access to information about model
development and model use in the Programs and Regions. CREM also will need solid backing of
senior EPA management to fulfill its mission. The Subcommittee recommends that CREM be
given this necessary authority and management backing to fulfill its mission.
The Subcommittee commends EPA's proposal for involving the public in this effort. This
process can lead not only to a better understanding of EPA's models, but a better acceptance of
models used in regulatory activities. It also provides a way to tap the work done by others,
thereby leveraging EPA's resources.
The EMS looks forward to the opportunity to continue to work with the Agency as it
increases its efforts to develop, apply and improve high quality, scientifically-based and defensible
regulatory models. We look forward to your response to the advice contained in this Advisory.
Sincerely,
/signed/ /signed/
-------
Dr. Joan Daisey, Chair Dr. Ishwar Murarka, Chair
Science Advisory Board Environmental Models Subcommittee
Science Advisory Board
-------
References Cited
EPA. 1994. Guidance for Conducting External Peer Review of Environmental Regulatory
Models. Agency Task Force on Environmental Regulatory Modeling. EPA 100-B-94-001.
US EPA. Washington, DC. 1994.
EPA. 1994. Report of the Agency Task Force on Environmental Regulatory Modeling.
Guidance, Support needs, Draft Criteria and Charter. EPA 500-R-94-001. Office of Solid
Waste and Emergency Response. US EPA. Washington, DC. March, 1994.
SAB. 1989. Resolution on the Use of Mathematical Models for Regulatory Assessment and
Decision-making. SAB-EEC-89-012. Science Advisory Board. US EPA, Washington,
DC. January, 1989.
SAB. 1993. Review of Draft Agency Guidance for Conducting External Peer review of
Environmental Modeling. SAB-EEC-LTR-93-008. Science Advisory Board. US EPA,
Washington, DC. July, 1993.
SAB. 1995. Commentary on Appropriateness of SAB Reviews of Computer Environmental
Transport and Fate Models Developed for Environmental Decision-Making. SAB-EEC-
COM-95-005. Science Advisory Board. US EPA. Washington, DC. February, 1995.
SAB. 1999. Science Advisory Board Review of the Implementation f the Agency-Wide Quality
System. SAB-EEC-LTR-99-002. Science Advisory Board. US EPA, Washington, DC.
February, 1999.
R- 1
-------
U.S. ENVIRONMENTAL PROTECTION AGENCY SCIENCE ADVISORY BOARD
ENVIRONMENTAL MODELS SUBCOMMITTEE OF THE
EXECUTIVE COMMITTEE
,DE
CHAIR
Dr. Ishwar Murarka, Chief Scientist and President, ISH Inc., Cupertino, CA
MEMBERS
Dr. Steven M. Bartell, Cadmus Group, Inc., Oak Ridge, TN
Dr. Calvin Chien, Senior Environmental Fellow, E.I. DuPont Company, Wilmington,
Dr. Kai-Shen Liu, Epidemiologist, California Department of Health Services, Environmental Health
Laboratory Branch, Berkeley, CA
Dr. Paulette Middleton, Associate Director, Environmental Science and Policy Center, RAND
Corporation, Boulder, CO
CONSULTANTS
Dr. M. Bruce Beck, Professor & Eminent Scholar, Warnell School of Forest Resources, University of
Georgia, Athens Georgia
Dr. Linfield Brown, Professor, Department of Civil and Environmental Engineering, Tufts University,
Medford, MA
Dr. Arthur J. Gold, Professor, Department of Natural Resources Science, University of Rhode Island,
Kingston, RI
Dr. Helen Grogan, Cascade Scientific, Inc., Bend, Oregon
Dr. Wu-Seng Lung, Professor, Department of Civil Engineering, University of Virginia, Charlottesville,
VA
Dr. Jana Milford, Associate Professor, Department of Mechanical Engineering, University of Colorado,
Boulder, CO
Dr. Mitch Small, Department of Civil Engineering & Public Policy, Carnegie Mellon University,
Pittsburgh, PA
Dr. Thomas Theis, Professor & Chair, Department of Civil and Environmental Engineering, Clarkson
University, Potsdam, NY
SCIENCE ADVISORY BOARD STAFF
Dr. John R. Fowle, III, Deputy Staff Director/Designated Federal Officer, Environmental Protection
Agency, Science Advisory Board, Washington, DC
Ms. Karen Martin, Deputy Designated Federal Officer, Environmental Protection Agency, Science
Advisory Board, Washington, DC
Mrs. Dorothy M. Clark, Management Assistant, Environmental Protection Agency, Science Advisory
-------
Board, Washington, DC
-------
NOTICE
This report has been written as a part of the activities of the Science Advisory Board, a
public advisory group providing extramural scientific information and advice to the Administrator
and other officials of the Environmental Protection Agency. The Board is structured to provide
balanced expert assessment of scientific matters related to problems faced by the Agency. This
report has not been reviewed for approval by the Agency; and hence, the contents of this report
do not necessarily represent the views and policies of the Environmental Protection Agency or
other agencies in the Federal government. Mention of trade names or commercial products does
not constitute a recommendation for use.
in
-------
DISTRIBUTION LIST
Administrator
Deputy Administrator
Assistant Administrators
EPA Regional Administrators
EPA Laboratory Directors
EPA Headquarters Library
EPA Regional Libraries
EPA Laboratory Libraries
Library of Congress
National Technical Information Service
Office of Technology Assessment
Congressional Research Service
------- |