United States
Environmental Protection
Agency
X > .it
Atmospheric Sciences ^ "_
Research Laboratory .- \-
Research Triangle Park NC 27711 '/ f ^
Research and Development
EPA/600/S3-87/008 May 1988
SERA Project Summary
International Sulfur Deposition
Model Evaluation
Terry L. Clark, Robin L Dennis, Eva C. Voldner, Marvin P. Olson,
Steve K. Seilkop, and Mayer Alvo
The International Sulfur Deposition
Model Evaluation (ISDME) project,
jointly conducted by the U.S. Environ-
mental Protection Agency and Atmos-
pheric Environment Service of Environ-
ment Canada, assessed the perform-
ance of eleven linear chemistry
atmospheric models in predicting
amounts of sulfur wet deposition.
Standardized model input data sets -
were distributed to the participating
modelers, who later submitted sea-
sonal and annual 1980 model predic-
tions of dry/wet deposition and air
concentrations of sulfur dioxide and
sulfate at up to 65 sites across eastern
North America.
The models were evaluated in an
operational mode using new, more
rigorous approaches, as well as the
more conventional distribution statis-
tics recommended by the American
Meteorological Society. The new
approaches focused on the ability of the
models to replicate features of the
spatial patterns of sulfur wet deposi-
tion, as determined by an interpolation
technique known as kriging. This
technique quantified the uncertainties
in the observations which were used
in the evaluation process to identify
areas where interpolated predictions
were statistically significantly different
from the interpolated observations. To
supplement the evaluation, predictions
of dry deposition amounts and air
concentrations of each model were
intercompared to identify apparent
peculiarities.
Finally, a scoring system based on
criteria for six model performance
measures was devised to compare
seasonal, annual and overall perform-
ances of the models. Three clusters of
models, each with similar overall
scores, were identified.
This Project Summary was devel-
oped by EPA's Atmospheric Sciences
Research Laboratory, Research Trian-
gle Park. NC, to announce key findings
of the research project that is fully
documented in a separate report of the
same title (see Project Report ordering
information at back).
Introduction
Computer models have been deve-
loped to estimate mean air concentra-
tions of sulfur dioxide and sulfate and
sulfur dry and wet deposition amounts
at receptor points or across grid cells
approximately 100 km on a side. Many
of these models have been applied to
estimate source-receptor relationships,
particularly for su If ur wet deposition, a nd
total sulfur deposition across geopolitical
entities. In the past, these models could
not be rigorously evaluated for their
predictions of sulfur wet deposition,
because of the paucity of sampling data.
The increased number of sites in the
North American precipitation chemistry
networks after 1979 affords the oppor-
tunity to more rigorously evaluate these
models for sulfur wet deposition.
Recognizing the need to evaluate
existing North American models, the
Office of Research and Development of
the U.S. Environmental Protection
Agency and Atmospheric Environment
Service of Environment Canada jointly
conducted the International Sulfur Depo-
sition Model Evaluation (ISDME) Project.
Using unique methods, this project
evaluated eleven linear-chemistry
atmospheric models of sulfur deposition
for each season and the year of 1980.
These models were either statistical.
-------
Lagrangian or hybrids. The selection of
1980 as the evaluation year was based,
in part, on the availability of the most
recent emissions inventory.
Evaluation Approach
The approach of the study was to (i)
compile and distribute standardized
model input data sets to the modelers,
(ii) apply the eleven models using these
data sets and (iii) evaluate the models
in a "blind" test mode, meaning the
computer code of these models was not
modified specifically to improve the
results for 1980. The evaluation data
consisted of sulfur wet deposition
amounts calculated from screened pre-
cipitation chemistry data from the five
rpajor North American networks and
daily air concentrations of sulfur dioxide
and sulfate at four Canadian sites. Dry
deposition data were not available during
this study.
The ISDME screening and calculation
procedures were very similar to those
recommended by the Unified Deposition
Data Base Committee established by the
Canadian Research and Monitoring
Coordinating Committee and the U.S.
National Acid Precipitation Assessment
Program. The number of sites with data
passing the ISDME criteria ranged from
32 for the annual to 46 for the spring
periods.
Unlike the evaluations of the past, the
ISDME focused on the ability of the
models to replicate the spatial patterns
of seasonal, as well as annual, amounts
of sulfur wet deposition within the
uncertainties of the data. Seasonal and
annual evaluations, rather than only
annual evaluations, were conducted to
provide a more stringent test of the
models. Patterns were generated by
interpolating the site observations and
predictions to half-degree grid cells via
a technique known as kriging. This
technique minimizes the interpolation
errors and quantifies the uncertainties
arising from both the interpolation and
the errors of measurement.
The emphasis on pattern replication is
based on the fact that point measure-
ments are not necessarily representative
of the spatial resolution of the models.
That is, large spatial variability of mete-
orological parameters and air concentra-
tions can cause "local" effects on the
measurements at a site. As a result,
these measurements could be represen-
tative of only a point and not the region.
Therefore, it is necessary to compare
predictions and observations on com-
mensurate spatial scales. Moreover,
pattern comparisons provide at least a
limited evaluation of the models in data
sparse regions.
The second unique aspect of the
evaluation approach was the consider-
ation of the uncertainties of the observed
amounts of sulfur wet deposition. Rather
than comparing predictions to the best
estimates of the observations, this
evaluation determined whether and by
how much the predictions were outside
the uncertainty limits of the interpolated
observations. Based on these uncertainty
limits, the statistical significance of the
differences was determined. These
uncertainties arise from sampling and
analytical errors, local or nonregional
effects, network protocol differences,
and the interpolations of both the pre-
dictions and observations. The uncer-
tainties of the annual pattern of sulfur
wet deposition ranged from approxi-
mately ±50% in the region of highest
sulfur wet deposition to ±100% across
data sparse regions of eastern North
America.
The models were evaluated using six
model performance measures, which
quantified the differences between the
predicted and observed patterns of sulfur
wet deposition. The measure receiving
the most weight was the precentage of
the half-degree cells where the predic-
tions significantly differed from the
measurements. The remaining five
measures were assigned equal weights.
Two of these measures quantified differ-
ences in pattern position, while two
additional measures quantified the
differences in the magnitudes and
locations of the maxima. The final
measure quantified differences in the
seasonal distribution of the annual
amounts.
Evaluation Results
The comparison of predicted and
observed patterns of seasonal sulfur wet
deposition revealed that the predicted
patterns tended to resemble concentric
ellipses and to show less detail than the
observed patterns. The predicted sea-
sonal and annual patterns also consist-
ently exhibited maximum amounts
within the high emissions region of Ohio,
western Pennsylvania and WestVirginia.
This behavior was not evident in the
patterns of the observations, which
indicated that the location of the max-
imum was not anchored, but migrated
from northern West Virginia to southern
Ontario.
Depending on the model, distances
separating the predicted and observed
locations of the maxima for all seasons
were less than 170 to 350 km. Most
models were within the uncertainty
limits of the magnitude of the seasonal
and annual maxima.
The determination of the areas of
statistically significant differences
between the predictions and observa-
tions of sulfur wet deposition revealed
that across 80% of the evaluation region:
• The seasonal predictions of five of the
eleven models were within the uncer-
tainty limits of the observed seasonal
patterns.
• The seasonal predictions of all but
three models were within the same
uncertainty limits for at least three
seasons.
• The annual predictions of all but three
models were within the uncertainty
limits of the observed annual pattern.
A contributing factor to this high
percentage was the degree of uncer-
tainty of the observed pattern of sulfur
wet deposition, which was as high as
+50% across the area of greatest site |
density. Therefore, the models in this
study were afforded a rather large
statistical tolerance for differences.
Regarding the seasonal contributions
to the 1980 amount of sulfur wet
deposition, nearly 40% and 30% of the
annual observed amounts occurred
during summer (July, August and Sep-
tember) and spring (April, May and June),
respectively. Approximately 15%
occurred during both winter (January,
February and March) and autumn
(October, November and December).
Most of the models are within 10% of
the summer and spring contributions and
within 25% of the autumn and winter
contributions. Three models did not
mimic the observed seasonal variability
as well as the other models.
Two features of model performance
were common to the statistical models
but not to the deterministic models. First,
the patterns of the sulfur wet deposition,
as determined by the statistical models,
tended to be oriented more toward the
east-west axis, as opposed to the
southwest-northeast axis of the
observed patterns. On the other hand,
the orientation of the patterns produced
by the deterministic models tended to be,
closer to that of the observed patterns.
-------
This distinction could be due to the fact
that the statistical models base their
transport simulation on seasonal mean
winds, which in North America tend to
be westerly, rather than on wind mea-
surements available twice a day. The
second feature common to statistical
models was the tendency to predict
sulfate air concentrations higher than
those of the deterministic models at both
the ISDME sites and the four Canadian
monitoring sites.
Finally, the seasonal, annual and
overall performances of the models were
summarized by scoring and clustering
procedures based on subjectively deter-
mined criteria values for each of the six
performance measures. A model
received points if the~values of the
performance measures did not exceed
the criteria values. Overall performance
scores were based on weighted seasonal
performance scores. The weights, the
sum of which equalled one, were deter-
mined by the mean relative contributions
of the observed seasonal amounts of
sulfur wet deposition. Consequently, the
model performance for the summer, the
season of greatest sulfur wet deposition,
received a weight of approximately 0.4.
The distribution of overall performance
scores illustrated three distinct clusters
or groups of models. One model having
one of the highest overall performance
scores was deemed to have performed
relatively well, but used an empirical
"correction" (or background) term
accounting for 40% to 60% of the
predicted sulfur wet deposition.
Conclusions
In conclusion, this study demonstrated
the use of a unique approach to model
evaluation, which accounts for the
uncertainties of the evaluation data. This
approach is appropriate for future eval-
uations of the recently develped Eulerian
models of wet deposition. By applying
this unique approach, this study
assessed the performance of linear-
chemistry atmospheric models of sulfur
deposition to identify the strengths and
weaknesses of each model.
It is recommended that the reasons
explaining the behavior of these models
be investigated as a means of identifying
and implementing scientifically defensi-
ble improvements to the models. It is
further recommended that the improved
models be evaluated for subsequent
years when emissions data are available
and more precipitation chemistry data
exist. Additonal ^1tes would reduce the
interpolation uncertainties and thus
provide for a more rigorous model
evaluation
-------
The EPA authors Terry L. Clark and Robin L. Dennis (also the EPA Project
Officers, see below) are with the Atmospheric Sciences Research Laboratory.
Research Triangle Park, NC 27711; Eva C. Voldner and Marvin P. Olson are
with Atmospheric Environment Service, Environment Canada. Downsview,
Ontario; Steve K. Seilkop is with Analytical Sciences. Inc., Research Triangle
Park, NC; and Mayer Alvo is with the University of Ottawa, Ottawa. Ontario.
The complete report, entitled "InternationalSulfur Depositon Model Evaluation,"
(Order No. PB 88-190 509/AS; Cost: $25.95. subject to change) will be
available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
The EPA Project Officers can be contacted at:
Atmospheric Sciences Research Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
U.S.OFFICIAL MAO.
Official Business
Penalty for Private Use $300
EPA/600/S3-87/008
,0lt3 >
0000529 PS
•6U S GOVERNMENT PRINTING OFFICE. 1988—548-013/1
------- |