United States
                    Environmental Protection
Atmospheric Sciences Research
Research Triangle Park NC 27711
                    Research and Development
EPA/600/S9-86/018 Sept. 1986
&EPA          Project  Summary

                    Workshop  on   Model
                    Evaluation  Protocols:
                    Chairman's  Report
                    W. T. Fennel I
                     This report summarizes the results of
                    a workshop sponsored by the U.S. Envi-
                    ronmental Protection Agency that was
                    held to discuss procedures and proto-
                    cols for evaluating regional-scale acid
                    deposition models. The workshop was
                    the first of three that are planned to
                    assist the U.S. Environmental Protec-
                    tion Agency, the Ontario Ministry of En-
                    vironment, the Atmospheric Environ-
                    ment Service of Canada, and the
                    Electric Power Research Institute in de-
                    signing a model evaluation program.
                    The workshop was asked to consider
                    four major topics:
                      procedures to be used in evaluat-
                       ing the performance of acid deposi-
                       tion models and methodologies for
                       applying these procedures
                      data requirements of these proce-
                       dures and methodologies
                      the probable impact of time and
                       budget constraints on the evalua-
                       tion process
                      possible conflicts between client
                       needs and the probable output of
                       the evaluation program.
                     Each of these topics was considered,
                    and a series of recommendations was
                    made. These recommendations cov-
                    ered the manner in which the evalua-
                    tion should be conducted, the specific
                    tasks that should be undertaken to ef-
                    fect the evaluation, the data require-
                    ments of the recommended evaluation
                    program,  and  the types of questions
                    that can and cannot be answered by
                    such a program.
                     This Project Summary was devel-
                    oped by EPA's Atmospheric Sciences
                    Research Laboratory, Research Triangle
                    Park, NC, to announce key findings of
the research project that is fully docu-
mented in a separate report of the same
title (see Project Report ordering infor-
mation at back).

  The workshop on model evaluation
protocols, sponsored by the U.S. Envi-
ronmental Protection Agency (EPA) and
held in Raleigh, NC, on February 11-13,
1986, was the first of three workshops
planned to assist the EPA, the Ontario
Ministry of Environment (OME), the At-
mospheric Environment Service (AES),
and the Electric Power  Research Insti-
tute (EPRI) in designing a program to
evaluate  the performance of regional-
scale acid deposition models. It was at-
tended by a group of 27 scientists repre-
senting  government  laboratories,
industrial groups,  private contractors,
and universities. An attendance list is
given in Appendix  A of the report. The
workshop participants considered four
major topics.
   Procedures to be used in evaluating
   the performance of acid deposition
   models and methodologies for  ap-
   plying these procedures
   Data requirements of these proce-
   dures and methodologies
   Probable impact of time and bud-
   get constraints on the evaluation
   Possible conflicts  between  client
   needs and the probable output of
   the evaluation  program
  A series of recommendations was
made  regarding the manner in which
the evaluation should be conducted, the
specific tasks that should be undertaken
to effect  the evaluation, the data  re-

quirements of the recommended evalu-
ation program, and the types of ques-
tions that can and cannot be answered
by such a program. These recommen-
dations are presented in the report.
  In  reaching their conclusions, the
workshop participants drew on the re-
sults of two previous activities: the
EPRI-sponsored  Utility Deposition Net-
work (UDN) workshop held in  Novem-
ber 1985 and an  August 1985 workshop
on evaluation of acid deposition models
(sponsored by  EPA,  OME and  AES),
which resulted in  a Concept Plan. The
UDN workshop was convened to review
a technical plan  for a  deposition  moni-
toring network  that would gather the
wet deposition  and aerometric meas-
urements needed for an operational
evaluation of acid deposition  models.
The UDN workshop made specific rec-
ommendations on the type of measure-
ments required  and on the methods to
be used in making them. These recom-
mendations were  incorporated into the
recommendations of evaluation  proto-
cols workshop.
  The Concept Plan, on the other hand,
outlined a series of field studies as well
as a wet and dry deposition monitoring
network that would generate the data
needed for both  operational (how well a
model reproduces actual observations
of deposition and concentration fields
on the time and space scales  needed)
and  diagnostic (how well individual
components of the model simulate ac-
tual  atmospheric processes)  evalua-
tions of the models. This plan, however,
did not contain a set of specific tasks for
its fulfillment, nor  did it indicate the rel-
ative importance of routine monitoring
activities, which generate the  data for
operational  evaluation, compared  to
process studies, which provide the in-
formation needed for diagnostic evalua-
tion. The workshop participants, there-
fore, spent considerable time  in
defining these tasks and in discussing
the general order  of priorities.
  From the clients' point of view, the
primary purpose of regional-scale acid
deposition models is to provide scientif-
ically defensible tools for analyzing the
consequences of  alternative strategies
for controlling emissions of acid deposi-
tion  precursors. Reflecting this need,
the clients posed a set of four key policy
questions that they expect the models
to address. These questions were re-
lated to  (1) deposition loadings,
(2) source attribution, (3) chemical non-
linearity, and (4) detectability.
  During the course of the workshop,
several issues were raised that could be
a source of conflict between the needs
of the clients and what the workshop
participants think is possible in a realis-
tic model evaluation program: conflicts
between model evaluation and model
development, problems in evaluating
complex models, and problems in set-
ting priorities.
  One of the charges given the work-
shop was to set priorities for the model
evaluation process, that is, to determine
which aspects of the models contained
the greatest uncertainty and thus re-
quired the greatest emphasis in an eval-
uation program. Thus, the participants
strongly suggested that sensitivity stud-
ies be conducted as soon as possible in
order to guide  the process of experi-
ment design. Nevertheless, the work-
shop participants recognized that the
clients required preliminary guidance in
this area; consequently, the  tasks de-
scribed in the following sections are or-
dered according to the participants' ini-
tial priority judgments.

General Recommendations

Goals and Management

  The workshop participants agreed
that the original goals of the model eval-
uation program could not be met by a
model  evaluation program subject to
the probable time and budget con-
straints. Thus, these goals were modi-
fied to be more consistent with what
was thought to be achievable. These re-
stated goals are given below:

Deposition Loadings
  Given data  from a  surface-based
monitoring network operating for only a
few years and sampling under a limited
range of chemical conditions, it will not
be possible to determine the accuracy
to which a given model can predict cli-
matologically valid deposition loadings
to a given area. It will, however, be pos-
sible to determine whether such  a
model  can simulate deposition fields
over the period sampled by the network
to an accuracy comparable to the uncer-
tainty to which the monitoring network
can define the actual deposition field. It
should also be possible to quantify ob-
jectively the level of disagreement be-
tween the measurements and alterna-
tive  predictions, assuming  that this
disagreement exceeds the  measure-
ment uncertainty.
Source Attribution
  No method is available for directly
testing or evaluating the  ability of a
model to make this computation. The
models will certainly be able to com-
pute changes  in deposition resulting
from changes  in emissions; and it  is
possible, by numerically tagging the
emissions from a given area, to deter-
mine where the emissions from the
tagged area  may  go.  However, one
must remember that such relationships
apply only to a fixed  distribution of
emissions. Because of the nonlinearity
of the chemistry, changing the emis-
sions at any point  in the modeled do-
main may affect source attributions  at
every other location.

Chemical Nonlinearity
  Short of drastically  changing existing
emission patterns, one cannot truly
evaluate the ability of a model to handle
this issue. The only solution, therefore,
is to perform extensive diagnostic stud-
ies that investigate how accurately
process  modules in the  models repre-
sent the  corresponding natural phe-
nomena. If these modules appear to be
fairly accurate, then there will be confi-
dence in the  ability  of the models to
treat issues such as nonlinearity and
source attribution.

  The ability of a model to make this
sort of computation cannot be directly
evaluated. However, the degree of con-
fidence that one can have in the predic-
tions of the model depends on how well
the various process  modules simulate
the actual atmospheric phenomena.
  In summary, an operational  model
evaluation program  will indicate how
well the  models can simulate current
deposition patterns over a time period
comparable to the length of the surface
monitoring program. A diagnostic eval-
uation program, on the other hand, will
indicate how well the process modules
in a given model simulate the important
physical and chemical processes in at-
mosphere. Diagnostic studies are  es-
sential if we are to have confidence that
the models are capable of simulating
deposition for significantly different
emission  compositions and  distribu-
  The field measurement  program as-
sociated with  the model evaluation
process should last a minimum of two
years, and longer if possible, because a
shorter  measurement program would

not gather a sufficient data set for a use-
ful  evaluation of the models. All data
collected during diagnostic studies
should be released to potential users as
soon as they have been quality audited,
because  experimental data for model
development purposes are critically
lacking. Filling this need was judged
more important than achieving a com-
pletely hands-off diagnostic evaluation
of sequestered data. The first year of
data from the operational  evaluation
program should  be delivered to the
users for model development purposes,
and only the second year of data should
be sequestered for use in  blind evalua-
tion tests.
  The model evaluation program
should be managed by a  highly quali-
fied, disinterested party. The managing
organization should be responsible for
developing the evaluation protocols,
supervising their execution, and report-
ing the results. The National Academy
of Sciences was suggested as the most
acceptable organization for this role.

Recommended Tasks
  The workshop  recommended that
several preliminary tasks be started dur-
ing FY  86 because the  results are
needed for planning the work to  be ac-
complished in FY 87 and  beyond: de-
velop model evaluation methods, de-
velop evaluation protocols, analyze
existing data bases, conduct model sen-
sitivity studies, and conduct emissions

Operational Evaluation
  Given  current budget  restrictions,
highest priority should, be placed  on es-
tablishing and operating  a surface-
based wet deposition and  aerometric
monitoring network, and on performing
other tasks to produce a data base for
operational evaluation of regional-scale
acid deposition  models. The first four
tasks are listed in order of  decreasing
priority. The last  three are necessary
support tasks for any  of the first four.
Some of the tasks  have  a diagnostic
component,  but the workshop partici-
pants felt this component was vital to
interpreting  the operational compari-

Deploy and Operate a Surface-
Based  Monitoring  Network
  Top priority is given  to the task of es-
tablishing and operating  a 30-station
wet deposition and aerometric monitor-
ing network in the northeastern United
States. This network should be main-
tained for a minimum of two years.

Vertical Profiles Over the
Modeling Domain
  In this task, frequent aircraft flights
will be made year-round  over  various
portions of the eastern United States to
obtain information on the vertical distri-
bution, from near the surface to several
thousand feet, of several important spe-

Deploy and Operate Subgrid
Variability Networks
  The purpose of this task is to gather
data on the subgrid variability of precip-
itation  chemistry and ambient concen-
tration of pollutants for interpreting the
results of network  measurements. The
subgrid variability will also form the nu-
cleus for additional diagnostic  studies.
Two subgrid variability networks
should be established: one in Kentucky
or the Ohio River Valley and one on the
U.S./Canada border.  Each network
should consist of an enhanced central
monitoring station surrounded by a
cluster of approximately 100 sequential
precipitation chemistry monitoring sites
and should cover a 200 km2 area. Each
full subgrid network should be operated
for two 2-month intensive periods per
year, and about ten sites should be op-
erated continuously. During the  inten-
sive observation periods, aircraft meas-
urements should be made of the vertical
profiles of the species measured  at the
enhanced station.

Emission Inventories
  Existing emission inventories should
be  updated to correspond to the time
periods being modeled in  the opera-
tional  evaluation studies. In addition,
more extensive  improvements should
be made to the inventories, such as im-
proving VOC inventories, if the FY 86
emission inventory task shows that this
effort is justifiable economically.

Support Tasks
  In addition to these major tasks there
are three additional support tasks:
(1) quality audit  data,  (2) archive data,
and (3) perform evaluations.

Suggested  Protocols
  For the first year of measurements, all
data should  be released as soon as
quality auditing is complete. In the sec-
ond year, the following policies are sug-
  1. Emission inventory updates qual-
 ity audited and released to modelers
  2. Surface monitoring data quality
 audited and sequestered
  3. Subgrid variability data quality au-
 dited and released to modelers
  4. Vertical profile data quality audited
 and sequestered
  5. Comparison of model outputs and
 data overseen by the National Academy
 of Sciences
     surface comparisons made by
 approved objective techniques
     vertical profile comparisons
 made by objective and subjective analy-

 Diagnostic Evaluation
  The primary purpose of the diagnos-
 tic evaluation process is to ensure that
 the  models are  providing an accurate
 simulation of the physical and chemical
 processes that control the transport,
 transformation, and deposition of acidic
 materials. Initially, seven types of diag-
 nostic studies were identified as neces-
 sary for realistically assessing the abil-
 ity  of  acid deposition models to
 simulate wet and dry removal. Two of
 these study classes, subgrid variability
 and vertical profiles, were included in
 the  basic operational evaluation pro-
 gram. The remaining five study classes
 were grouped according to  a prelimi-
 nary assessment of their relative impor-
 tance:  wet deposition modules, dry
 deposition modules (high priority stud-
 ies), atmospheric transport, gas-phase
 chemistry, and treatment of the inflow
 boundary conditions (low priority stud-

 Wet Deposition Module
  These studies  should ensure that the
 physical and chemical processes gov-
"erning  the wet  removal of  acidifying
 gases and aerosols are accurately rep-
 resented in the  models. Intensive pre-
 cipitation scavenging studies should be
 conducted in the two subgrid regions
 during  the intensive observation peri-
 ods. Additional  measurements would
 be  required from radar, cloud-physics-
 equipped aircraft, and enhanced upper-
 air soundings. Data can be analyzed by
 simulating the observed cloud and pre-
 cipitation chemistry fields with diagnos-
 tic  models; comparing observed and
 simulated data will indicate how well
 the model represents the critical proc-
 esses. Given a sufficient number of case
 studies, the parameterization  schemes
 that consistently result in the best repre-

    sentation of the observations should

    Dry Deposition Module
     Model simulations should be com-
    pared with alternative methods for
    measuring or estimating dry deposition
    in the atmosphere, using the dry depo-
    sition core stations as the primary diag-
    nostic reference points. Additional core
    stations should be placed at each of the
    two enhanced subgrid sites, and meas-
    urements for deducing dry deposition
    from air concentration and meteorolog-
    ical measurements should  also be
    made at about four of the subgrid clus-
    ter stations. Using combinations of
    ground-based and aircraft data, esti-
    mates can be made of dry deposition
    fluxes to each of the subgrid areas, and
    these estimates can be compared with
    model computations.

    Atmospheric Transport
     In evaluating transport, two types of
    studies are envisioned: studies of long-
    range horizontal transport  and studies
    of vertical translation  in storms.  The
    most effective tests of a model's ability
    to handle pollutant transport are tracer
    studies. The sampling system must be
    composed of several tracer-sampling
    aircraft  in addition to a ground-based
    sampling network.

    Gas Phase Chemistry
     This diagnostic evaluation, largely in-
    direct, will involve detailed measure-
    ment of hydrocarbon species, reaction
    products, and NOXNOV chemistry at the
       central stations. These measurements
       will be compared with zero-dimensional
       reaction-chemistry simulations, using
       the parameterizations employed by the
       models, as well as more elaborate de-
       scriptions. Comparisons of key ratios of
       reactants and intermediates will be em-
       ployed as the primary tests of reality in
       the submodel  calculations. An alterna-
       tive  experimental approach  is to  per-
       form Lagrangian-rype experiments.
Boundary Conditions

  These studies should determine
whether  model simulations are being
affected by the transport of errors into
the modeling domain through the in-
flow boundaries. A possible method of
determining transport effects would be
to make  a series of research  aircraft
flights along the inflow boundary at var-
ious altitudes in the boundary layer.
          W. T. Pennell is with Battelle Pacific Northwest Laboratories, Richland. WA
          Jack Durham is the EPA Project Officer (see below).
          The complete report, entitled "Workshop on Model Evaluation Protocols:
            Chairman's Report," (Order No. PB 86-217 122/AS; Cost: $9.95, subject to
            change) will be available only from:
                 National Technical Information Service
                 5285 Port Royal Road
                 Springfield, VA 22161
                  Telephone: 703-487-4650
          The EPA Project Officer can be contacted at:
                 Atmospheric Sciences Research Laboratory
                 U.S. Environmental Protection Agency
                 Research Triangle Park, NC27711
United States
Environmental Protection
Center for Environmental Research
Cincinnati OH 45268

Official Business
Penalty for Private Use $300
    .  0000329    PS