United States
                     Environmental Protection
                     Agency
                            X  > .it
Atmospheric Sciences          ^      "_
Research Laboratory           .-     \-
Research Triangle Park NC 27711 '/ f   ^
                     Research and Development
EPA/600/S3-87/008  May 1988
SERA          Project  Summary
                     International  Sulfur  Deposition
                     Model  Evaluation
                    Terry L. Clark, Robin L Dennis, Eva C. Voldner, Marvin P. Olson,
                    Steve K. Seilkop, and Mayer Alvo
                      The International Sulfur Deposition
                    Model Evaluation  (ISDME) project,
                    jointly conducted by the U.S. Environ-
                    mental Protection Agency and Atmos-
                    pheric Environment Service of Environ-
                    ment Canada, assessed the perform-
                    ance of  eleven  linear  chemistry
                    atmospheric models  in  predicting
                    amounts of sulfur wet  deposition.
                    Standardized  model input  data sets -
                    were distributed  to the participating
                    modelers,  who  later submitted sea-
                    sonal and annual 1980 model predic-
                    tions of dry/wet  deposition and air
                    concentrations of sulfur dioxide and
                    sulfate at up to 65 sites across eastern
                    North America.
                      The models were evaluated  in an
                    operational mode using  new, more
                    rigorous approaches, as well as the
                    more conventional distribution statis-
                    tics  recommended  by  the  American
                    Meteorological Society. The  new
                    approaches focused on the ability of the
                    models to  replicate features of  the
                    spatial patterns of sulfur wet deposi-
                    tion, as determined by an interpolation
                    technique  known  as  kriging.  This
                    technique quantified the uncertainties
                    in the observations which  were used
                    in the evaluation process  to identify
                    areas where interpolated predictions
                    were statistically significantly different
                    from the interpolated observations. To
                    supplement the evaluation, predictions
                    of dry deposition amounts and air
                    concentrations of each model were
                    intercompared to identify apparent
                    peculiarities.
                      Finally, a scoring system based on
                    criteria for six  model performance
                    measures was devised to compare
                    seasonal, annual and overall perform-
                    ances of the models. Three clusters of
models,  each with similar overall
scores, were identified.
  This  Project Summary was devel-
oped by EPA's Atmospheric Sciences
Research Laboratory, Research Trian-
gle Park. NC, to announce key findings
of the  research project that is  fully
documented in a separate report of the
same title (see Project Report ordering
information at back).

Introduction
  Computer models have been deve-
loped to estimate  mean air concentra-
tions of sulfur dioxide and sulfate and
sulfur dry and wet deposition amounts
at receptor  points or across grid  cells
approximately 100 km on a side. Many
of these models have been applied to
estimate source-receptor relationships,
particularly for su If ur wet deposition, a nd
total sulfur deposition across geopolitical
entities. In the past, these models could
not be  rigorously evaluated for  their
predictions  of sulfur wet deposition,
because of the paucity of sampling  data.
The increased number of sites in the
North American precipitation chemistry
networks after 1979 affords the oppor-
tunity to more rigorously evaluate these
models  for sulfur wet deposition.
  Recognizing the need to evaluate
existing North American models, the
Office of Research and Development of
the U.S. Environmental Protection
Agency and Atmospheric Environment
Service  of Environment  Canada jointly
conducted the International Sulfur Depo-
sition Model Evaluation (ISDME) Project.
Using  unique methods, this project
evaluated   eleven  linear-chemistry
atmospheric models of sulfur deposition
for each season and the year of 1980.
These  models were either statistical.

-------
Lagrangian or hybrids. The selection of
1980 as the evaluation year was based,
in part, on the availability of the  most
recent  emissions inventory.

Evaluation Approach
  The  approach of the study was  to (i)
compile  and  distribute  standardized
model  input data sets to  the modelers,
(ii) apply the eleven models using these
data sets and (iii) evaluate the models
in a "blind" test  mode,  meaning the
computer code of these models was not
modified specifically to  improve the
results for  1980. The evaluation  data
consisted  of sulfur wet deposition
amounts calculated from  screened pre-
cipitation chemistry data  from the five
rpajor  North American  networks  and
daily air concentrations of sulfur dioxide
and sulfate at four Canadian sites. Dry
deposition data were not available during
this study.
  The ISDME screening and calculation
procedures were very similar to those
recommended by the  Unified Deposition
Data Base Committee established by the
Canadian Research and Monitoring
Coordinating  Committee  and the  U.S.
National Acid Precipitation Assessment
Program. The number of sites with data
passing the ISDME criteria ranged from
32 for  the annual to 46 for the spring
periods.
  Unlike the evaluations of the past, the
ISDME focused on the  ability  of the
models to replicate the spatial patterns
of seasonal,  as well as annual, amounts
of sulfur  wet  deposition within the
uncertainties of the data.  Seasonal and
annual evaluations,  rather than  only
annual evaluations, were conducted to
provide a more stringent test  of the
models. Patterns were  generated  by
interpolating  the site observations and
predictions to half-degree grid cells via
a technique known as  kriging.  This
technique minimizes the  interpolation
errors  and quantifies the uncertainties
arising from both  the interpolation and
the errors of measurement.
  The emphasis on pattern replication is
based  on the fact that  point measure-
ments are not necessarily representative
of the  spatial resolution of the models.
That is, large spatial variability of mete-
orological parameters and air concentra-
tions can cause "local" effects  on the
measurements  at a  site.  As a  result,
these measurements could be represen-
tative of only a point and not the  region.
Therefore,  it is necessary to compare
predictions  and observations on  com-
mensurate  spatial  scales.  Moreover,
pattern  comparisons provide at least a
limited evaluation of the models in data
sparse regions.
  The  second  unique  aspect of  the
evaluation approach was the consider-
ation of the uncertainties of the observed
amounts of sulfur wet deposition. Rather
than comparing  predictions to the best
estimates  of  the  observations, this
evaluation determined whether and by
how much the predictions were outside
the uncertainty limits of the interpolated
observations. Based on these uncertainty
limits, the statistical significance of the
differences was determined. These
uncertainties arise  from  sampling  and
analytical  errors, local or nonregional
effects,  network protocol differences,
and the interpolations of both  the pre-
dictions and observations. The uncer-
tainties  of the annual pattern of sulfur
wet  deposition  ranged  from  approxi-
mately ±50% in the region  of highest
sulfur wet deposition to ±100% across
data sparse  regions of  eastern  North
America.
  The models were  evaluated using six
model  performance measures, which
quantified the differences between the
predicted and observed patterns of sulfur
wet deposition.  The measure receiving
the most weight was the precentage of
the half-degree  cells where the predic-
tions  significantly  differed from  the
measurements. The remaining five
measures were assigned equal weights.
Two of these measures quantified differ-
ences in pattern position, while two
additional  measures quantified  the
differences in  the magnitudes and
locations of the maxima.  The final
measure quantified  differences in  the
seasonal distribution  of the annual
amounts.

Evaluation Results
  The  comparison  of predicted  and
observed patterns of seasonal sulfur wet
deposition revealed that the predicted
patterns tended  to resemble concentric
ellipses and to show less detail than the
observed patterns.  The  predicted sea-
sonal and annual patterns also consist-
ently exhibited maximum  amounts
within the high emissions region of Ohio,
western Pennsylvania and WestVirginia.
This behavior  was  not  evident in  the
patterns of the observations, which
indicated that the location of the max-
imum was  not  anchored, but migrated
from northern West  Virginia to southern
Ontario.
  Depending  on the  model,  distances
separating  the  predicted and observed
locations of the maxima for all seasons
were  less than  170 to 350  km.  Most
models were within the  uncertainty
limits of the magnitude of the seasonal
and annual maxima.
  The  determination  of the  areas of
statistically  significant  differences
between  the  predictions and observa-
tions of sulfur wet deposition revealed
that across 80% of the evaluation region:

• The seasonal predictions of  five of the
  eleven models were within the uncer-
  tainty limits of the observed seasonal
  patterns.

• The seasonal predictions of all but
  three models were within  the same
  uncertainty limits for at least three
  seasons.

• The annual predictions of all but three
  models were within the uncertainty
  limits of the observed annual pattern.

  A  contributing factor to this  high
percentage was the degree  of uncer-
tainty of the  observed pattern of sulfur
wet deposition, which was as high as
+50%  across the area of greatest site |
density. Therefore, the  models in  this
study  were  afforded a rather large
statistical tolerance for differences.
  Regarding the seasonal contributions
to the 1980 amount  of  sulfur  wet
deposition, nearly 40% and 30% of the
annual observed amounts  occurred
during summer (July, August  and Sep-
tember) and spring (April, May and June),
respectively.  Approximately   15%
occurred  during both winter  (January,
February and March)  and  autumn
(October,  November and  December).
Most  of the models are within  10% of
the summer and spring contributions and
within 25% of  the autumn and winter
contributions.  Three models did not
mimic the observed seasonal  variability
as well as the other models.
  Two features of model performance
were  common to the  statistical models
but not to the  deterministic models. First,
the patterns of the sulfur wet deposition,
as determined by the statistical models,
tended to be  oriented more toward the
east-west axis,  as  opposed  to the
southwest-northeast  axis   of  the
observed patterns. On the  other hand,
the orientation of the patterns produced
by the deterministic models tended to be,
closer to that of the observed patterns.

-------
This distinction could be due to the fact
that the statistical  models  base  their
transport simulation on seasonal mean
winds, which in North America tend to
be westerly, rather than on wind mea-
surements  available  twice  a day. The
second  feature common to statistical
models was  the  tendency  to  predict
sulfate air  concentrations higher  than
those of the deterministic models at both
the ISDME  sites and the four Canadian
monitoring sites.
  Finally, the seasonal, annual and
overall performances of the models were
summarized by scoring  and clustering
procedures  based on subjectively deter-
mined criteria values for each of the six
performance  measures.   A  model
received points  if the~values  of the
performance measures did  not  exceed
the criteria  values. Overall performance
scores were based on weighted seasonal
performance  scores. The weights, the
sum of which equalled one, were deter-
mined by the mean relative contributions
of the  observed seasonal amounts of
sulfur wet deposition. Consequently, the
model performance for the summer, the
season of greatest sulfur wet deposition,
received a weight of approximately 0.4.
The distribution of  overall performance
scores  illustrated three distinct clusters
or groups of models. One model having
one of  the highest  overall performance
scores  was deemed to have performed
relatively well,  but used an  empirical
"correction" (or  background)  term
accounting  for 40% to 60%  of the
predicted sulfur wet deposition.

Conclusions
  In conclusion, this study demonstrated
the use of a unique approach to model
evaluation,  which accounts for the
uncertainties of the evaluation data. This
approach is appropriate for future eval-
uations of the recently develped Eulerian
models of wet deposition. By applying
this  unique  approach,  this   study
assessed  the  performance  of  linear-
chemistry atmospheric models of sulfur
deposition to identify the strengths and
weaknesses of each model.
  It is recommended that the reasons
explaining the behavior of these  models
be investigated as a means of identifying
and implementing scientifically defensi-
ble improvements to the models. It is
further recommended that the improved
models be evaluated for subsequent
years when emissions data are available
and  more precipitation chemistry data
exist. Additonal ^1tes would reduce the
interpolation uncertainties  and  thus
provide for a more  rigorous  model
evaluation

-------
    The EPA authors Terry L. Clark and Robin L. Dennis (also the EPA Project
      Officers, see below) are with the Atmospheric Sciences Research Laboratory.
      Research Triangle Park, NC 27711; Eva C. Voldner and Marvin P. Olson are
      with Atmospheric Environment Service, Environment Canada. Downsview,
      Ontario; Steve K. Seilkop is with Analytical Sciences. Inc., Research Triangle
      Park, NC; and Mayer Alvo is with the University of Ottawa, Ottawa. Ontario.
    The complete report, entitled "InternationalSulfur Depositon Model Evaluation,"
      (Order No.  PB 88-190 509/AS;  Cost:  $25.95. subject to change) will be
      available only from:
            National Technical Information Service
            5285 Port Royal Road
            Springfield, VA 22161
            Telephone: 703-487-4650
    The EPA Project Officers can be contacted at:
            Atmospheric Sciences Research Laboratory
            U.S. Environmental Protection Agency
            Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
                                                                                                  U.S.OFFICIAL MAO.
Official Business
Penalty for Private Use $300

EPA/600/S3-87/008
                                                                  ,0lt3 >
           0000529    PS
                                                                                 •6U S GOVERNMENT PRINTING OFFICE. 1988—548-013/1

-------