United States Environmental Protection Agency X > .it Atmospheric Sciences ^ "_ Research Laboratory .- \- Research Triangle Park NC 27711 '/ f ^ Research and Development EPA/600/S3-87/008 May 1988 SERA Project Summary International Sulfur Deposition Model Evaluation Terry L. Clark, Robin L Dennis, Eva C. Voldner, Marvin P. Olson, Steve K. Seilkop, and Mayer Alvo The International Sulfur Deposition Model Evaluation (ISDME) project, jointly conducted by the U.S. Environ- mental Protection Agency and Atmos- pheric Environment Service of Environ- ment Canada, assessed the perform- ance of eleven linear chemistry atmospheric models in predicting amounts of sulfur wet deposition. Standardized model input data sets - were distributed to the participating modelers, who later submitted sea- sonal and annual 1980 model predic- tions of dry/wet deposition and air concentrations of sulfur dioxide and sulfate at up to 65 sites across eastern North America. The models were evaluated in an operational mode using new, more rigorous approaches, as well as the more conventional distribution statis- tics recommended by the American Meteorological Society. The new approaches focused on the ability of the models to replicate features of the spatial patterns of sulfur wet deposi- tion, as determined by an interpolation technique known as kriging. This technique quantified the uncertainties in the observations which were used in the evaluation process to identify areas where interpolated predictions were statistically significantly different from the interpolated observations. To supplement the evaluation, predictions of dry deposition amounts and air concentrations of each model were intercompared to identify apparent peculiarities. Finally, a scoring system based on criteria for six model performance measures was devised to compare seasonal, annual and overall perform- ances of the models. Three clusters of models, each with similar overall scores, were identified. This Project Summary was devel- oped by EPA's Atmospheric Sciences Research Laboratory, Research Trian- gle Park. NC, to announce key findings of the research project that is fully documented in a separate report of the same title (see Project Report ordering information at back). Introduction Computer models have been deve- loped to estimate mean air concentra- tions of sulfur dioxide and sulfate and sulfur dry and wet deposition amounts at receptor points or across grid cells approximately 100 km on a side. Many of these models have been applied to estimate source-receptor relationships, particularly for su If ur wet deposition, a nd total sulfur deposition across geopolitical entities. In the past, these models could not be rigorously evaluated for their predictions of sulfur wet deposition, because of the paucity of sampling data. The increased number of sites in the North American precipitation chemistry networks after 1979 affords the oppor- tunity to more rigorously evaluate these models for sulfur wet deposition. Recognizing the need to evaluate existing North American models, the Office of Research and Development of the U.S. Environmental Protection Agency and Atmospheric Environment Service of Environment Canada jointly conducted the International Sulfur Depo- sition Model Evaluation (ISDME) Project. Using unique methods, this project evaluated eleven linear-chemistry atmospheric models of sulfur deposition for each season and the year of 1980. These models were either statistical. ------- Lagrangian or hybrids. The selection of 1980 as the evaluation year was based, in part, on the availability of the most recent emissions inventory. Evaluation Approach The approach of the study was to (i) compile and distribute standardized model input data sets to the modelers, (ii) apply the eleven models using these data sets and (iii) evaluate the models in a "blind" test mode, meaning the computer code of these models was not modified specifically to improve the results for 1980. The evaluation data consisted of sulfur wet deposition amounts calculated from screened pre- cipitation chemistry data from the five rpajor North American networks and daily air concentrations of sulfur dioxide and sulfate at four Canadian sites. Dry deposition data were not available during this study. The ISDME screening and calculation procedures were very similar to those recommended by the Unified Deposition Data Base Committee established by the Canadian Research and Monitoring Coordinating Committee and the U.S. National Acid Precipitation Assessment Program. The number of sites with data passing the ISDME criteria ranged from 32 for the annual to 46 for the spring periods. Unlike the evaluations of the past, the ISDME focused on the ability of the models to replicate the spatial patterns of seasonal, as well as annual, amounts of sulfur wet deposition within the uncertainties of the data. Seasonal and annual evaluations, rather than only annual evaluations, were conducted to provide a more stringent test of the models. Patterns were generated by interpolating the site observations and predictions to half-degree grid cells via a technique known as kriging. This technique minimizes the interpolation errors and quantifies the uncertainties arising from both the interpolation and the errors of measurement. The emphasis on pattern replication is based on the fact that point measure- ments are not necessarily representative of the spatial resolution of the models. That is, large spatial variability of mete- orological parameters and air concentra- tions can cause "local" effects on the measurements at a site. As a result, these measurements could be represen- tative of only a point and not the region. Therefore, it is necessary to compare predictions and observations on com- mensurate spatial scales. Moreover, pattern comparisons provide at least a limited evaluation of the models in data sparse regions. The second unique aspect of the evaluation approach was the consider- ation of the uncertainties of the observed amounts of sulfur wet deposition. Rather than comparing predictions to the best estimates of the observations, this evaluation determined whether and by how much the predictions were outside the uncertainty limits of the interpolated observations. Based on these uncertainty limits, the statistical significance of the differences was determined. These uncertainties arise from sampling and analytical errors, local or nonregional effects, network protocol differences, and the interpolations of both the pre- dictions and observations. The uncer- tainties of the annual pattern of sulfur wet deposition ranged from approxi- mately ±50% in the region of highest sulfur wet deposition to ±100% across data sparse regions of eastern North America. The models were evaluated using six model performance measures, which quantified the differences between the predicted and observed patterns of sulfur wet deposition. The measure receiving the most weight was the precentage of the half-degree cells where the predic- tions significantly differed from the measurements. The remaining five measures were assigned equal weights. Two of these measures quantified differ- ences in pattern position, while two additional measures quantified the differences in the magnitudes and locations of the maxima. The final measure quantified differences in the seasonal distribution of the annual amounts. Evaluation Results The comparison of predicted and observed patterns of seasonal sulfur wet deposition revealed that the predicted patterns tended to resemble concentric ellipses and to show less detail than the observed patterns. The predicted sea- sonal and annual patterns also consist- ently exhibited maximum amounts within the high emissions region of Ohio, western Pennsylvania and WestVirginia. This behavior was not evident in the patterns of the observations, which indicated that the location of the max- imum was not anchored, but migrated from northern West Virginia to southern Ontario. Depending on the model, distances separating the predicted and observed locations of the maxima for all seasons were less than 170 to 350 km. Most models were within the uncertainty limits of the magnitude of the seasonal and annual maxima. The determination of the areas of statistically significant differences between the predictions and observa- tions of sulfur wet deposition revealed that across 80% of the evaluation region: • The seasonal predictions of five of the eleven models were within the uncer- tainty limits of the observed seasonal patterns. • The seasonal predictions of all but three models were within the same uncertainty limits for at least three seasons. • The annual predictions of all but three models were within the uncertainty limits of the observed annual pattern. A contributing factor to this high percentage was the degree of uncer- tainty of the observed pattern of sulfur wet deposition, which was as high as +50% across the area of greatest site | density. Therefore, the models in this study were afforded a rather large statistical tolerance for differences. Regarding the seasonal contributions to the 1980 amount of sulfur wet deposition, nearly 40% and 30% of the annual observed amounts occurred during summer (July, August and Sep- tember) and spring (April, May and June), respectively. Approximately 15% occurred during both winter (January, February and March) and autumn (October, November and December). Most of the models are within 10% of the summer and spring contributions and within 25% of the autumn and winter contributions. Three models did not mimic the observed seasonal variability as well as the other models. Two features of model performance were common to the statistical models but not to the deterministic models. First, the patterns of the sulfur wet deposition, as determined by the statistical models, tended to be oriented more toward the east-west axis, as opposed to the southwest-northeast axis of the observed patterns. On the other hand, the orientation of the patterns produced by the deterministic models tended to be, closer to that of the observed patterns. ------- This distinction could be due to the fact that the statistical models base their transport simulation on seasonal mean winds, which in North America tend to be westerly, rather than on wind mea- surements available twice a day. The second feature common to statistical models was the tendency to predict sulfate air concentrations higher than those of the deterministic models at both the ISDME sites and the four Canadian monitoring sites. Finally, the seasonal, annual and overall performances of the models were summarized by scoring and clustering procedures based on subjectively deter- mined criteria values for each of the six performance measures. A model received points if the~values of the performance measures did not exceed the criteria values. Overall performance scores were based on weighted seasonal performance scores. The weights, the sum of which equalled one, were deter- mined by the mean relative contributions of the observed seasonal amounts of sulfur wet deposition. Consequently, the model performance for the summer, the season of greatest sulfur wet deposition, received a weight of approximately 0.4. The distribution of overall performance scores illustrated three distinct clusters or groups of models. One model having one of the highest overall performance scores was deemed to have performed relatively well, but used an empirical "correction" (or background) term accounting for 40% to 60% of the predicted sulfur wet deposition. Conclusions In conclusion, this study demonstrated the use of a unique approach to model evaluation, which accounts for the uncertainties of the evaluation data. This approach is appropriate for future eval- uations of the recently develped Eulerian models of wet deposition. By applying this unique approach, this study assessed the performance of linear- chemistry atmospheric models of sulfur deposition to identify the strengths and weaknesses of each model. It is recommended that the reasons explaining the behavior of these models be investigated as a means of identifying and implementing scientifically defensi- ble improvements to the models. It is further recommended that the improved models be evaluated for subsequent years when emissions data are available and more precipitation chemistry data exist. Additonal ^1tes would reduce the interpolation uncertainties and thus provide for a more rigorous model evaluation ------- The EPA authors Terry L. Clark and Robin L. Dennis (also the EPA Project Officers, see below) are with the Atmospheric Sciences Research Laboratory. Research Triangle Park, NC 27711; Eva C. Voldner and Marvin P. Olson are with Atmospheric Environment Service, Environment Canada. Downsview, Ontario; Steve K. Seilkop is with Analytical Sciences. Inc., Research Triangle Park, NC; and Mayer Alvo is with the University of Ottawa, Ottawa. Ontario. The complete report, entitled "InternationalSulfur Depositon Model Evaluation," (Order No. PB 88-190 509/AS; Cost: $25.95. subject to change) will be available only from: National Technical Information Service 5285 Port Royal Road Springfield, VA 22161 Telephone: 703-487-4650 The EPA Project Officers can be contacted at: Atmospheric Sciences Research Laboratory U.S. Environmental Protection Agency Research Triangle Park, NC 27711 United States Environmental Protection Agency Center for Environmental Research Information Cincinnati OH 45268 U.S.OFFICIAL MAO. Official Business Penalty for Private Use $300 EPA/600/S3-87/008 ,0lt3 > 0000529 PS •6U S GOVERNMENT PRINTING OFFICE. 1988—548-013/1 ------- |