EPA-454/R- 96-004
Compilation of Photochemical Models'
Performance Statistics for 11/94 Ozone SIP
Applications
Julv 1996
U.S. Environmental Protection Agency
Office of Air Quality Planning and Standards
Emissions, Monitoring, and Analysis Division
Research Triangle Park, North Carolina 27711
U.S. Environmental Protection Agency
Region 5, Library (f»u2J)
77 West Jackson Boulevard, 12th Ftoor
Chicago, »L 60604-3590
-------
DISCLAIMER
The information of this report has been reviewed in the entirety by the U.S.
Environmental Protection Agency (EPA), and approved for publication as an EPA document.
Mention of trade names, products, or services does not convey, and should not be interpretated
as conveying official EPA approval, endorsement, or recommendation.
f C'P M'"M* j! .r
-------
ACKNOWLEDGMENT
This report has been prepared by Sonoma Technology Incorporated and funded by the U.
S. Environmental Protection Agency under Contract No. 68D30020 with Shao-Hang Chu as the
Work Assignment Manager.
111
-------
TABLE OF CONTENTS
Page
ACKNOWLEDGMENT iii
LIST OF FIGURES vi
LIST OF TABLES vii
1 INTRODUCTION 1-1
2. THE PHOTOCHEMICAL MODEL SIMULATIONS 2-1
3 MODEL PERFORMANCE 3-1
3.1 INDIVIDUAL SIMULATION STATISTICS 3-2
3.2 SIMULATION STATISTICS AVERAGED FOR EACH REGION 3-15
3.3 SIMULATION STATISTICS ON THE HIGHEST OZONE DAY IN
E^CH REGION 3-20
3 4 SIMULATION STATISTICS STRATIFIED BY MODELING
METHODOLOGY AND GEOGRAPHIC SETTING 3-20
3 5 SIMULATION STATISTICS FOR THE UAM-IV. UAM-V. AND SAQM
MODELS 3-31
4 CONCLUSIONS AND RECOMMENDATIONS 4-1
4 1 CONCLUSIONS 4-1
4 2 RECOMMENDATIONS 4-2
5 REFERENCES . 5- J
-------
LIST OF FIGURES
Figure
3-1. The predicted and observed domain-wide maximum ozone concentrations for
SIP applications 3-10
3-2. The number of simulations in each region with performance that met all
three of EPA's ozone performance goals 3-13
3-3 The percentage of simulations in each region with performance that met
all three of EPA's ozone performance goals 3-14
3-4 The average accurac> of the domain-wide peak ozone unpaired in space and
time in each region 3-17
3-5. The average normalized bias in predicted ozone concentrations above 60 ppb
in each region 3-18
3-6 The average normalized gross error in predicted ozone concentrations above
60 ppb in each region 3-19
3-7 The accuracy of the domain-wide peak ozone unpaired in space and time on
the highest ozone day in each region 3-22
3-8 The mean normalized bias in predicted ozone concentrations above 60 ppb on
the highest ozone day in each region 3-23
3-9 The mean normalized gross error in predicted ozone concentrations above
60 ppb on the highest ozone day in each region 3-24
3-10 The predicted and observed domain-wide nv 'mum ozone concentrations
stratified by boundary condition methodolog. 3-28
3-11 The predicted and observed domain-wide maximum ozone concentrations
stratified by wind model type 3-29
3-12 The predicted and observed domain-wide maximum ozone concentrations
stratified by geophysical characteristics 3-30
3-13 The predicted and observed domain-wide maximum ozone concentrations
stratified by the predominance of pollutant transport into the region 3-32
VI
-------
LIST OF TABLES
Table
2-1. Regions included in the study 2-2
3-1. The photochemical models' ozone performance statistics for individual
baseline simulations 3-3
3-2. The extent to which EPA performance goals were achieved in the SIP
applications 3-11
3-3. Average photochemical models' ozone performance statistics by region 3-16
3-4 Photochemical models' ozone performance statistic on high ozone
day b\ region 3-21
3-5 The methodologies used to develop boundary conditions and windfields.
and tne characteristics of the geographic setting 3-25
3-6 Photochemical models' performance statistics for ozone stratified by
boundary condition methodologv windfield model, geographic setiing.
and predominance of pollutant transport into the region 3-27
3-" Comparison of the average model performance statistics for ozone from
the SAQM. UAM-IY. and UAM-V models
Additional statistical parameters for model performance evaluation 4-3
Vll
-------
LIST OF TABLES
Table Page
2-1. Regions included in the study 2-2
3-1. The photochemical models' ozone performance statistics for individual
baseline simulations 3-3
3-2. The extent to which EPA performance goals were achieved in the SEP
applications 3-11
3-3. Average photochemical models' ozone performance statistics by region 3-16
3-4. Photochemical models' ozone performance statistic on high ozone
day b\ region 3-21
3-5. The methodologies used to develop boundary conditions and windfields,
and the characteristics of the geographic setting 3-25
3-6. Photochemical models' performance statistics for ozone stratified by
boundary condition methodology, windfield model, geographic setting,
and predominance of pollutant transport into the region 3-2"
3-7. Comparison of the average model performance statistics for ozone from
the SAQM. UAM-IV. and UAM-V models 3-31
4-1 Additional statistical parameters for model performance evaluation 4-3
Vlll
-------
1. INTRODUCTION
The scientific credibility of photochemical modeling studies depends on the soundness of
the model formulation, the adequacy of the aeometric database, and the accuracy of the model's
predictions for specific applications (Tesche et al., 1990: Roth et al., 1991). The accuracy of
specific applications must be evaluated because the quality and representativeness of input data
can strongly influence the model performance An investigation was carried out to evaluate the
performance of three grid-based photochemical models in recent applications. The photochemical
models were the Urban Airshed Model Version IV (UAM-IV). the Urban Airshed Model vanable
grid version (UAM-V). and the San Joaquin Valley Air Quality1 Model (SAQM). All three models
employed the CB4 chemical mechanism for these applications. The models were applied in 24
ozone nonanainment regions in 1993-1995 to support the development of emissions control
strategies for the 1994 ozone State Implementation Plans (SIPs). The models were used to
simulate the relauonship between nitrogen oxides (NOX) and volatile organic compound (VOC)
emissions, and ambient ozone concentrations under current episodic conditions and for future
emission scenarios The baseline model applications provide an unique opportunity' to examine
photochemical models performance for a large number of regions and episodes. Furthermore, the
SIP simulations were performed using generally consistent methodologies
The U S Enuronmental Protection Agency (EPA) established model performance goals
fo; the SIP applications In order to assess the extent to which the baseline simulations met the
performance goals, the EPA required states to report three statistical measures of the model's
abih'.\ to predict ambient ozone concentrations. These measures are:
1 The normalized accurac> of domain-wide maximum 1 -hr concentration unpaired in space
and time
2 Mear normalized bias of all predicted and observed concentration pairs where the
observed concentration.-, exceed 60 ppb
3 Mean normalized error of all predicted and observed concentration pairs where the
observed concentrations exceed 60 ppb
This report reflects the status of November 1994 SIP applications model performance results. As
the Suites continue their modeling efforts these results are expected to change. This document is
a compilation of the EPA recommended basic model performance statistics for the 24 ozone
nonattamment areas in the November 1994 SIP applications. For a more complete evaluation of
the models' performance both spatial and temporal analyses of the matching between the
predicted and observed concentrations are desirable. Thus, it is important to recognize that the
scope of this evaluation is quite limited The evaluation focuses on three statistical measures of
the models' ahilm to predict ambient ozone concentrations. It does not include an evaluation of
the models' pertormance on other species (such as NO, NO,. NOy, and VOCs) or for other
statistical measures of its performance on ozone, which may be important for establishing the
smtabilm of simulations for use in control strategy development
1-1
-------
2. THE PHOTOCHEMICAL MODEL SIMULATIONS
The statistical performance data for this study were obtained from reports and briefings
submitted to the EPA. Model performance results for the 22 regions listed in Table 2-1 were
included in the database. In some cases, more than one nonattainment area was included in a
single modeling domain and separate statistics were included for each area if they were provided
in the modeling documentation (e.g., in the Houston and Beaumont, Texas areas). In most cases
the reported statistics apply to a single nonattainment area. Data for several other regions were
received but not included as explained below.
The photochemical model simulations were carried out by numerous modeling groups
under the direction of state and local agencies. EPA provided modeling guidelines (EPA. 1991)
to promote consistency between the applications. The guidelines allowed considerable flexibility
so that regions with different types of databases, different geographic settings and meteorological
conditions, and different levels of modeling expertise could complete the applications. As shown
in Table 2-1. most of the simulations were performed with the UAM-IV model (SAI, 1990),
version 6.21. However, simulations for the Lake Michigan Ozone Study (LMOS) region (i.e.. the
four states surrounding Lake Michigan* were performed with the UAM-V model (SAI. 1993.
1994a i Simulations for the greater Atlanta, Georgia region were carried out with the UAM-IV
and UAM-V models Comparable efforts were made in the applications of both models in Atlanta
which ;ujihtates direct comparison of the results. Differences in the predictions from the UAM-
IV and UAM-V models are small for Altanta and the UAM-IV results were used in the summarx
of mode! performance. A comparison of performance statistics for the Atlanta UAM-IV and
UAM-V simulations are included In addition, the SAQM model was used for California's San
Joaqmn Valle> modeling domain The SAQM model is an enhanced version of the Regional Acid
Deposition Model (RADM)
The photochemical simulations were performed for two- to seven-day periods with one or
more days with 1-hr ozone concentrations above 120 ppb. The results for the first day of each
period were excluded from statistical analysis because it takes time for the simulations to become
driven b\ emissions rather than initial concentration estimates. In some large modeling domains
(e.g . San Joaquin Valley), both the first and second days of the simulations were treated as "start-
up1 days. Table 2-1 shows the number of episode days modeled in each region. Only modeled
episode days with observed ozone concentrations above 120 ppb were included in the database.
On average, six episode days were modeled in each region. The number of days modeled varied
significantly between the regions. For example, 17 days were modeled in the New York area and
only one day was modeled in the Sacramento area. The number of episode days modeled was
generally higher in the areas with more severe ozone problems (i.e., in Los Angeles. Houston, and
New York) The total number of episode days included in the database is 131 davs.
2-1
-------
Table 2-1. Regions included in the study.
Reeion
San Diego, CA
Los Anseles. CA
Ventura. CA
San Joaquin Valle\. CA
Sacramento. CA
Phoenix. AZ
Houston. TX
Beaumont. TX
Dallas TX
Baton Rouce. LA
St Louis MO
Lake Michigan
Detroit. MI
.Nashville TN
Louisville. KV
Cincinnati. OH
Atlanta GA
Richmond \ A
Philadelphia. NJ
New York. NY
Baltimore. MD
New Eneland
Model
UAM-IV
UAM-IV
UAM-IV
SAQM
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-I\'
L'AM-I\'
UAM-V
L'AM-IV
UAM-IV
UAM-IV
LAM-TV
LAM-TV
LAM -IV
UAM-IV
UAM-IV
UAM-Pv'
UAM-W
Number of Episode
Days Simulated
2
10
3
9
^
1
2
12
10
9
7
8
8
4
4
->
>
6
4
6
5
17
4
4
Reference
SDAPCD. 1994
SCAQMD, 1994
CARB. 1995a
CARE. 1995c
CARB. 1995b
Briefing
TNRCC. 1994b
TNRCC. 1994b
TNRCC. 1994c
SAI. 1994b
Briefing
LADCO, 1994 &.
Briefinc
Briefinc
Kaminsb. 1995
Bnefine
Bnefine
SAI. 1994c
Bnefine
Georgopoulos. 1995
NYDEC. 1994a.b.c.d
Briefing
Briefing
A.s noted above, the database does not include results for all of the ozone nonattainraent
areas Tne UAM-IV model was applied in El Paso, Texas without emissions for a significant
portion oi the modeling domain (Juarez. Mexico) and the results showed gross underprediciion of
observed ozone concentrations (TNRCC. 1994a). The El Paso simulation results were excluded
because the\ were based on grossly inadequate inputs compared to all other simulations
2-2
-------
considered here Results for the Santa Barbara and Southeast Desert Air Basin of California were
not reported separately from those for Ventura and the South Coast Air Basin of California.
respectively.
The techniques employed to prepare the meteorological and air qualm1 model inputs
varied between the regions. Numerous modelers used prognostic meteorological models, such as
the MM5. CSUMM, and CALRAMS. to develop the hourly 3-dimensional wind fields, while
other modelers chose diagnostic wind models, such as CALMET, U AM-Diagnostic Wind Model.
and the Regional Oxidant Model (ROM) meteorological model Various models were also used
to develop mixing height (RAMMET. MIXEMUP, CALMET, etc.). Likewise, boundary-
concentration inputs were obtained from the EPA- recommended default values, surface
observations, surface and aircraft observations, and regional model estimates (ROM). The
difference in model input preparation procedures were often determined by the extent of the
aerometnc database in the regions and the complexity of the meteorology. Prognostic
meteorological models were used more frequently in areas with coastal meteorology and/or
complex terrain (e.g.. Los Angeles and LMOS) rather than flat areas with continental
meteorolog) To the extem possible, the database was coded with the model input preparatior
procedure codes
The photochemical model performance statistics incorporated into the database were
thus: provided b\ the modeling groups in reports and bnefings to the EPA. STI did not
recompute the statistic trom the model output files and the observed concentration data because
the\ were not made available for use in the stud\. The absence of actual model output and
observation files inhibited qualm assuring the statistical dam and limited the scope of the analysis
to those statistics reported by the modeling groups. While numerous modeling groups reported
more than the required statistics, only the three EPA-required statistics were common to all of the
application reports
2-3
-------
3. MODEL PERFORMANCE
This evaluation of model performance in SIP applications focuses on the models' ability to
predict the domain-wide peak ozone concentration and the concentrations at all locations with
observed ozone concentrations above 60 ppb. The three statistical measures recommended in the
EPA photochemical modeling guidelines (EPA, 1991) are used. The measures are:
1. The normalized accuracy of domain-wide maximum 1-hr concentration unpaired in space
and time (Aj:
100
avmatr-waf peak Oomair.~*idf peak
" aomuir -*idf peak
(l)
Mean normalized bias of all predicted and observed concentration pairs where the
observed concentrations exceed 60 ppb (NBIAS^):
100 A. (Predx, - OBS: ,)
\7?/4 s = V - - - :_
u'.tri A includes all of the predicted and observed concemraii(>
r>ai>> u;i7; observed com fntranor.' abo-.e 60
Mean normalized error of all predicted and observed concentration pairs where the
ob>erved concentrations exceed 60 ppb (NERROR^):
m<,jL, ?nd . - OBS'
\ERROR = - >
A induih'-' all of the predicted and obser\'ed concentratio
\\iil. </ -
-------
The EPA guidelines set statistical performance goals for regulatory' applications of the
UAM. The goal for the peak accuracy unpaired in space and lime is within ±15 percent The goal
for the mean normalized bias is within ±15 percent. The goal for the mean normalized error is
less than 35 percent. These goals are primarily based on historical performance of the model.
rather than analyses of accuracy requirements for intended use of the model (e.g.. analyses that
relate accuracy of baseline simulations to accuracy of the response of the model to emission
changes).
3.1 INDIVIDUAL SIMULATION STATISTICS
The statistical performance data for the 129 individual simulations are shown in Table 3-1
and Figure 3-1 The table indicates the region, the date of the episode day, the model, the
observed domain-wide maximum 1-hr ozone concentration, the predicted domain-wide maximum
1-hr ozone concentration, the accuracy of the domain-wide peak, the mean normalized bias, and
the mean normalized error. The average, standard deviations, minimum, and maximum values are
shown at the bottom of the table. Table 3-2 and Figures 3-2 and 3-3 illustrate the extent to
which the individual simulations met EPA's performance goals.
The results indicate that the models predicted an average domain-wide peak of 172 ±51
ppb \\hen the axerage observed domain-wide peak was 170 ±44 ppb. On average, the accuracy
of the predicted domain-wide peak ozone concentration unpaired in space and time was +2 ±20
percent The near absence of bias on average in the unpaired peak ozone is an excellent result for
air qualm models Even at the extremes, the ozone peak accuracy is only a factor of two in error
The lowest accuracx domain-wide peaks in individual simulations were for October 12. 1991 in
Houston where the ozone was under-predicted by 53 percent (112 ppb predicted versus 240 ppb
observed) and for June 21. 1988 in Nev. York where the ozone was overpredicted by 100 percent
(298 ppb predicted \ersus 149 ppb observed). At least one simulation in every region met the
±15 percent performance goal for the unpaired peak accuracy Overall. 77 percent of the
individual simulation met this performance goal For areas that modeled four or more episode
davs. over 75 percent of the episode days modeled in the following regions met the EPA
performance goal for unpaired peak ozone: Los Angeles. Dallas. St. Louis. Lake Michigan.
Detroit. Atlanta, and Baltimore The statistics and Figure 3-1 show there is considerable scatter
in the unpaired peak predictions. The correlation between the observed and predicted domain-
wide peak is moderate. R: = 0.54. which indicates the models are able to explain 54 percent of the
variance in the peak ozone.
The mean normalized bias for ozone concentration above 60 ppb averaged
-5 ± 16 percent in the SIP applications. The negative bias is consistent with the majority of
previous UAM simulations. The small average bias is comparable to the uncertainty in the ozone
observations (±5 ppb) This level of performance on average is excellent for air quality models.
At the extremes, the mean normalized bias was as low as -45 percent (October 12. 1991 in
Houston) and as high as +55 percent (June 21, 1998 in New York) in individual simulations At
least one simulation in every region met the ±15 percent performance goal for the mean domain-
wide peak ozone was overpredicted by 6.4 percent on average with the ROM-derived boundary
3-2
-------
Table 3-1. The photochemical models' o/one performance statistics for individual baseline simulations.
1'iigc 1 of 7
Region
San Diego, ("A
San Diego, CA
Los Angeles, CAh
Los Angeles, CAh
Los Angeles, CA
Los Angeles, CAh
Los Angeles, CA
Los Angeles, CAh
Los Angeles, CAb
Los Angeles, CAS
Los Angeles, CAh
Los Angeles, CAh
Ventura, CA
Ventura, CA
Ventura, CA
San Joaquin Valley, CA
San Joaquin Valley, CA
Sacramento, CA
Phoenix, AZ
Phoenix, AZ
Houston, TX
Model
HAM IV
HAM IV
HAM IV
UAM IV
UAM IV
UAM IV
UAM [V
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
SAQM
SAQM
UAM-IV
UAM-IV
UAM IV
UAM IV
Hpisode Date
August 2", 1989
September 21, I'>81>
Augusi 27, 1987
August 28, 1987
.fuly 14, 1087
July 15, 1987
June 24, 1987
June 25, 1987
June 6, 1985
June 7, 1985
Septembers, 1987
September 9, 1987
September 17. 1984
September 6, 1984
September 7, 1984
Augusi 5, 1990
August 6, 1990
July 1.3, (990
August 10, 1992
June 14, 1991
August 1, 1990
Observed
Domain wide
Penk
()
220
230
240
.360
360
330
260
140
170
180
150
160
143
157
140
138
Predicted
Domain-wide
Penk
(PI*)
141
135
216
.322
255
288
251
250
392
.382
254
252
143
175
173
139
150
139
134
132
163
Accuracy of
Domain-wide
Peak t inpaireci
(%)
-8.4
-13.5
-10 0
11.0
2.0
31 0
90
4.0
9.0
6.0
-23.0
-3.0
2.1
2.9
-3.9
-7.3
-6.3
-28
-14.6
-5.7
18 1
Mean
Normali/ed
Bias'
(.%)
-12.9
-5.6
-22.0
-8.0
-24.0
-12.0
-25.0
-26.0
7.0
7.0
-27.0
-21.0
-12.0
-12.0
-2.0
1.0
-7.0
-12.0
-5.4
4.4
^to.
-------
Table 3 I. The photochemkal models' o/one performance slalistics for individual baseline simulations.
Page 2 of 7
Region
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Model
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
Fpisode Date
Inly 27, 1 WO
July 28, 1WO
July 2''. |WO
July .10, IWO
July 11, IWO
May 16, |9S8
May 17, 1988
May 18, 1188
October 10, 1991
October II, 1991
October 12, 1991
July 28, 1990
July 30. IWO
July 31, IWO
May 18, l<>88
May 19, 1 988
May 19, 1088
October 10, |9O]
October II, I9<>|
October 12, 199 1
October 1 1, l<><)|
( )bserve
185
179
263
25.3
211
139
153
173
151
111
112
146
174
151
163
139
139
109
127
115
121
Accuracy of
Domain-wide
Peak Unpaired
(%)
2 8
19 3
29.6
32.5
40.7
13.1
-27.1
-17.6
23.8
-30.7
-53.3
13.2
25.2
-33.2
25.4
-22.8
-22.8
-24.3
-11.8
-42.5
II 7
Mean
Normalized
Bias"
(96)
-25.7
-31.3
-14.1
17.7
-11.4
-32.9
-28.6
-36.3
-8.9
-5.4
^t5.0
10.5
23.1
-31.8
-13.9
-27.5
-275
-6.6
-5.7
-42.8
-342
Mean Gross
I'.rror'
(%)
10.1
34.6
32.2
32.6
43.9
36.0
38.0
38.2
21.0
21.7
45.2
19.5
32.5
32.0
16.6
29.9
29.9
17.3
14 1
41 2
14.2
u»
-------
Table 3-1. The photochemical models' o/one performance statistics for individual baseline simulations.
Page 3 of 7
Region
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
St. Louis, MO
St. Louis, MO
St. Louis, MO
St. Louis, MO
St. Louis, MO
Model
UAM-IV
HAM -IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
Hpisode Date
August 1, 1('Q|
August 25, l"88
August 26, 1988
August 27, 1990
August 28, 1990
August 2(>, 1900
August 30, 1900
July 31, 190|
June 18, 1987
August 15, 1989
August 16, 1989
August 21, 1902
July 27, 1989
July 28, 1989
May 24, 1990
May 25, 1990
August 15, 1988
August 16, 1988
August 17, 1988
August 18. 1988
July 7, 1988
( )liser\ IN!
Domain wide
Peak
-------
Table 3-1. The photochemical models' o/one performance statistics for individual baseline simulations
Page 4 of 7
Region
St l^oiiis, MO
St Louis, MO
St Louis, MO
Ijike Michigan
Lake Michigan
Lake Michigan
Ijike Michigan
Lake Michigan
Lake Michigan
Lake Michigan
Lake Michigan
Detroit, MI
Detroit, MI
Detroit, MI
Detroit, MI
Louisville, KY
Ixniisville, KY
l/niisville, KY
Nashville, TN
Nashville, TN
Nashville, TN
Model
UAM-IV
UAM-IV
HAM IV
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
F.pisode Date
July R, I"8R
Inly ", 1988
June 24, 1987
August 26, I°Q|
July 17, 1991
July 18, |99|
July 19, 1991
June 20, 1991
June 21, 1991
June 26, 1991
June 28, 1991
August 2, 1988
August 3, 1988
July 5, 1988
July 7, 1988
luly 20, 1987
June 2.1, 1988
June 27. |990
August 1, 1988
August 3, 1 988
July 9. I <>X8
Observed
Domain- w ide
Peak
'>)
I61
1 26
1 60
I 78
1 30
1 69
1 53
I2I
I 34
I 75
1 33
I44
1 64
1 47
1 68
I5I
1 37
1 48
1 38
1 35
I 38
Predicted
Domain-wide
Peak
(PPh)
1 60
1 40
1 52
1 50
1 42
I6I
I44
1 09
LSI
1 65
1 37
1 42
1 48
1 28
1 64
1 87
1 38
1 60
1 15
132
139
Accuracy of
Domain-wide
Peak I Inpaired
(%)
-1.8
II 1
-5.0
-16.0
2.0
-5.0
-60
-10.0
13.0
-6.0
3.0
-1.4
-9.8
-12.9
-2.4
23.8
0.7
8.1
-16.7
-2.2
07
Mean
Normalized
Bias'
(%)
-7 0
-8.0
7.0
5.0
11.0
4.0
13.0
-7.0
-12.0
3.0
10.0
-12.0
-24.0
-15
-12.0
1.7
-28.1
6.0
-7.2C
-9.4C
-9 ()'
Mean Gross
T.rror'
(%)
20.0
16.0
24.0
14.0
17.0
16.0
18.0
14.0
20.0
14.0
16.0
22
28
20
22
15.9
29.5
16.1
15.1
20.2
20.3
-------
Tahlc 3 1 The photochemical models' o/otie performance statistics (or individual haseline simulations.
Page S of 7
Region
Nashville, TN
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Atlanta, GA
Atlanta, GA
Atlanta, GA
Atlanta, GA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Philadelphia, NJ
Philadelphia, NJ
Philadelphia, NJ
Philadelphia, NJ
Model
HAM IV
HAM IV
HAM IV
UAM IV
UAM-IV
UAM IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
1 piscxlc Date
June 24, l<>88
August 1, 1(>88
August 16, 1988
August 17, 1988
August 18, 1088
August 2, 1988
August 3, 1988
July 31, 1987
August 1, 1987
July 8, 1988
Aug. 10, 1992
July 10, 1988
July 15, 1988
July 6, 1988
July 7, 1988
June 13, 1988
June 14, 1988
July 19, |9Q|
July 20, 1991
July 7, 1988
July 8. 19R8
( rtiservcd
Domnin wide
IVnk
(PI*)
117
155
144
144
159
140
l()9
201
109
186
132
142
141
155
173
144
153
150
ISO
210
210
1'ieduted
Domain-wide
Peak
(PI*)
in
190
158
150
150
180
169
194
191
197
148
127
142
130
146
131
144
160
304
201
274
Accuracy of
Domnin wide
Peak Unpaired
(%)
-17.5
227
9.9
4.0
-5.6
28.3
0.0
-3.4
12.9
6.1
11.9
-10.6
0.7
-16.1
-15.6
-9.0
-5.9
6.7
68.8
-4 3
30 4
Mean
Normalized
Bias*
(%)
-25.2'
-4.K
-3.5
-14.6
-10.0
-3.3
3.5
5.2C
-0.9C
-85C
4.3C
I8.2C
-8.7C
-I5.0r
-15.9C
-12. 3e
-7.4r
24.3
15.3
6.9
20.1
Mean Gross
Hrror'
(%)
26.1
22.5
19.2
21.7
21.6
20.4
19.4
28
13.2
18.4
22.1
20.7
13.3
20.6
18.8
18.2
19.8
35.0
29.6
25.6
30.7
-J
-------
Table 3-1. The photochemical models' o/one performance stalistics for individual baseline simulations.
Page 6 of 7
Region
Philadelphia, NJ
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
Baltimore, MD
Baltimore, MD
Baltimore, MD
Model
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
HAM IV
F'.pisocle Dale
June 15, l(>87
Jul} 10, 1988
July II, 1988
July 17, I00|
July 18, 1901
July 10, 1001
July 20, 1901
July 7, 1988
July 8, 1088
July 9, 1988
June 15, 1987
June 18, 1087
June 19, 1987
June 19, 1988
June 20, 1987
June 20, 1988
June 21, 1988
June 22, 1088
June 20, 1 988
June 30, |088
June 19, |
-------
Table 31. The photochemical models' o/one performance statistics for indn ;88
Inly 8, l('88
Julv II, 1988
12<>.0
129.0
129.0
129.0
( HiM'rved
Douiinn wide
P.-nk
-------
400
Ozone Predictions in SIP Applications
Domain-wide Maximum Concentrations
100
Dashed lines are +/-15 percent
200 300
Observed Concentration (ppb)
400
12/18/95
22 Regions
129 Data Points
Figure 3-1. The predicted and observed domain-wide maximum ozone concentrations for SIP applications.
-------
Table 3-2. The extent to which RPA performance goals were acheived in (he SIP applications.
Page 1 of 2
Region
San Diego
l/».«i Angeles
Ventura
San Joaquin Valley
Sacramento
Phoenix
Houston
Beaumont
Dallas
Baton Rouge
St. Louis
Lake Michigan
Detroit
Ixniisville
Nashville
Cincinnati
Atlanta
Richmond
Philadelphia
Number of
Day-
Modeled
2
10
1
2
1
2
12
10
0
7
8
8
4
3
4
6
4
6
5
Number of Days lor Whu h Performance Cioals
Were At lne\ ed
Peak
Accuracy
Criteria
2
8
?
2
\
2
2
3
8
5
7
7
4
2
2
4
4
4
1
Mean
Hia*
Criteria
T
4
1
-»
1
2
4
4
8
5
7
8
2
2
3
6
4
T
1
Mean
1 iror
Criteria
2
8
1
2
1
2
6
0
9
7
8
8
4
3
4
6
4
6
4
All
Criteria
2
3
3
2
1
2
0
2
7
4
7
7
2
1
2
4
4
3
1
Percent of Modeled Days for Which Perfonnam e
(ioals Were Achieved
Peak
Accuracy
Criteria
100
80
100
100
100
100
17
30
89
71
88
88
100
67
50
67
100
67
60
Mean
Bias
Criteria
100
40
100
100
100
100
33
40
89
71
88
100
50
67
75
100
100
50
20
Mean
Rrror
Criteria
100
80
100
100
100
100
50
90
100
100
100
100
100
100
100
100
100
100
80
All
Criteria
100
30
100
100
100
100
0
20
78
57
88
88
50
33
50
67
100
50
20
-------
Table 3-2. The extent to which FiPA performance goals were acheived in the SIP applications.
Page 2 of 2
Region
New York
Baltimore
New England
Sum
Mean
Minimum
Maximum
Number of
Days
Modeled
17
4
2
Number of Days for Which Pcrfonnance Cioals
Wen- Achieved
Peak
Accuracy
Criteria
9
3
2
87
1
9
Mean
Hias
Criteria
10
4
2
86
1
10
Mean
l-.rror
Criteria
15
4
2
117
1
15
All
Criteria
5
1
2
66
0
7
Percent of Modeled Days for Which Perfonnance
Goals Were Achieved
Peak
Accuracy
Criteria
53
75
100
77.3
16.7
100.0
Mean
Bias
Criteria
59
100
100
75.3
20.0
100.0
Mean
Firror
Criteria
88
100
100
94.9
50.0
100.0
All
Criteria
29
75
100
64.1
0.0
100.0
NO
-------
o"
v/
Ozone Predictions in SIP Applications
Number of Days Meeting EPA Performance Goals
18
16
03
Q 12
.- 10
Q)
JO
1 8
6
4
2
r>
a
IV
1
- -
HfllL
" .
I
Days Meeting Goals
Alt Days Modeled
Region
12/18/95
22 Regions
129 Data Point?
Figure 3-2. The number of simulations in each region with performance that met all three of EPA's ozone performance goals.
-------
Ozone Predictions in SIP Applications
Percent of Days Meeting EPA Performance Goals
100
CD
TJ
O
CO
>»
CO
Q
CD
o
O
w
'Q.
LLJ
CD
O
v_
0>
Q.
Region
12/18/95
22 Regions
129 Data Points
Figure 3-3. The percentage of simulations in each region with performance that met all three of HI'A's ozone performance
goals.
-------
areas that modeled four or more episode days, over 75 percent of the episode days modeled in the
following regions met the EPA performance goal for mean normalized bias: Dallas. SL Louis,
Lake Michigan. Nashville, Cincinnati, Atlanta, and Baltimore.
The mean normalized gross error averaged 24 ± 8 percent in the SIP simulations. This
level of error is somewhat lower than in most historiplUAM applications. In individual
simulations, the gross error was as low as 12 percent (May 24, 1990 in Baton Rouge) and as high
as 56 percent (June 21. 1988 in New York). In most regions, all of the simulations had gross
errors of less than 35 percent for ozone. However, selected simulations for Los Angeles,
Houston. Beaumont. Richmond, and New York had higher gross error than the EPA goal.
Overall, 95 percent of the simulations met the EPA goal for gross error.
Table 3-2 shows the number of simulations that met all three of the EPA's performance
goals. At least one simulation in every region except Houston met all three goals. Overall, 64
percent of the simulations met all three goals. However, several of the areas that modeled a large
number of days had low percentages of simulations achieving all three goals. For example, only 3
of 10 davs in Los .Angeles. 0 of 12 days in Houston, 2 of 10 in Beaumont and 5 of 17 days in
New York met all three goals The simulations for Dallas. Lake Michigan, and St. Louis had the
most days with performance that met all three goals.
3.2 SIMULATION STATISTICS AVERAGED FOR EACH REGION
Table 3-3 and Figures 3-4 through 3-6 showtbe roedel performance statistics averaged
lor each region The) show the domain-wide peak ozdfne was underpredicted by more than 9
percent (1 o > in San Diego. Phoenix, and Beaumont and overpredicted by more than 9 percent in
Louisville. Cincinnati. Philadelphia, and New York on average. The domain-wide peak ozone
was predicted within ±10 percent in the 15 other regions on average. The correlation between the
region a\ erased predicted and observed peaks is high, R: = 0.86. and considerably better than that
for the individual simulations
The grand average mean bias averaged by region is -5.6 percent, which is similar to the
grand average for all individual simulations. In Los Angeles. Houston, Beaumont, and Detroit,
the average predictions have negative biases of 15 percent (1 o) or more. Positive average biases
of 13 percent (+2o) or more are evident in the Philadelphia and New York simulations. The mean
bias in the other 16 regions is between -13 and +3 percent, which is reasonably good.
The grand average mean gross error averaged by region is 22.7 percent which is slightly
better than the average for all individual simulations. The results show that the mean gross error
is generally higher (greater than 25 percent) in the region with high ambient ozone concentrations
(Los Angeles. Houston. Philadelphia. New York, and New England). The simulations for San
Joaquin Valley. Lake Michigan. Sacramento. Dallas. Richmond, and Baltimore have mean gross
error of less than 20 percent on average, which is quite good.
3-15
-------
Table 3-3. Average photochemical models' ozone performance statistics by region
Region
San Diego. CA
Los Angeles. CA
Ventura. CA
San Joaquin Vallev
Sacramento. CA
Phoenix. AZ
Houston. TX
Beaumont. TX
Dallas TX
Bator. Rouee. LA
Si Louis. MO
Lake Michicar,
Detroit. Ml
N^hvilie. TN
Louisville KY
Cincinnati OH
Auir.'o. GA
Ri.-.'.r,, r,J VA
Philadelphia NI
Nev, ^ 01 L M
Baltimore MD
Nev. Eneland
Average
Standard Deviation
Minmiun,
Ma.\imum
Number
of Davs
2
10
3
1
1
1
12
10
Q
7
8
8
4
4
-i
^
6
4
6
<;
17
4
~i
59
3.9
1
17
Observed
Domain-
wide Peak
(ppb)
155
278
163
155
14?
149
179
161
150
137
147
1 50
156
137
145
152
172
151
188
183
162
203
1644
29.7
137
278
Predicted
Domain-
wide Peak
(ppb)
138
286
164
145
139
133
176
138
145
150
143
145
146
125
162
166
183
137
230
200
165
199
1642
367
125
286
Accuracy of
Domain-
wide Peak
Unpaired
(%)
-11
4
0
-7
.3
-10
2
-11
-3
9
->
.3
-7
-9
11
10
7
-4
23
14
3
o
-^
0.3
8.8
-11
23
Mean
Normalized
Bias
(%)
-9
-15
-9
-3
-12
-1
_2'1
-16
-5
.5
-1
3
-16
-13
-7
-5
0
.7
17
13
0
-11
-5.6
8.9
-22
17
Mean
Gross
Error (9P
27
31
-o
16
18
0~>
z*.
35
2~i
19
on
21
16
tL ^
20
21
21
20
19
3d
T7
19
25
01 1
4.9
16
35
3-16
-------
Ll-£
3!
03
-------
Ozone Predictions
Average Bias By Region
All Cases with Observed Ozone Above 60 ppb
OJ
I
t>
oo
15
Ctf
CO
-f-«
CD
O
0>
CL
(15)
Region
^VCj^^X
x*
12/18/95
22 Regions
129 Data Points
Figure 3-5. The average normalized bias in predicted o/,one concentrations above 60 ppb in each legion.
-------
VO
I
o
UJ
-*->
c
o
Q)
Q_
40
35
30
25
20
15
10
5
0
*?
Ozone Predictions
Average Gross Error By Region
All Cases with Observed Ozone Above 60 ppb
Region
12/18/95
22 Regions
129 Data Polnta
Figure 3-6. The average normalised gross error in predicted ozone concentrations above 60 ppb in each region,
-------
normalized bias. Overall. 75 percent of the individual simulations met this performance goal For
3.3 SIMULATION STATISTICS ON THE HIGHEST OZONE DAY IN EACH
REGION
Table 3-4 and Figures 3-7 through 3-9 show the model performance statistics for the
highest ozone day in each region. The results show the domain-wide peak ozone was
underpredicted by 6.6 percent on average on the highest ozone days in each region. On average.
the predicted and observed domain-wide ozone peak concentrations were 183 and 196 ppb on the
highest days in each region. The mean bias and gross error averaged -4.8 and 23.7 percent on the
highest ozone days, respectively. The correlation between the predicted and observed peaks on
the highest ozone days was moderately high, R2 = 0.70. Thus, overall, the performance on the
highest ozone days is not significantly different than the average of all days modeled. The only
difference is the tendency for the models to underpredict the domain-wide peak on the highest
days, which was especial!) evident in New York and Houston.
3.4 SIMULATION STATISTICS STRATIFIED BY MODELING METHODOLOGY
AND GEOGRAPHIC SETTING
Within the modeling community, there is interest in whether the modeling methodologies
and geographic setting significantly influence the model performance. There are many minor
differences in the modeling methodologies in these simulations and in the geographic settings of
the areas Often these differences are difficult to quantify and classih consistently. For this
analysis, we have elected to stratify only on a few variables
The results of these simulations were stratified on (1) the type of wind model, prognostic
or diagnostic, used to de\elop the hourly gridded 3-dimensional windfields and (2) the methods of
determining the boundary conditions, either derived from a regional model (ROM) or from
observations. In addition, the results were stratified on (1) whether the geophysical features \\er;
pnmanh flat land or involved a land-water interface, which generally increases the complexity of
the meteorolog\. and (2) whether regional pollutant transport typicalh has a major or minor efiect
on ozone in the region. Table 3-5 lists the boundary condition methodology, the wind model, the
type of geographic setting, and the predominance of pollutant transport into the region. The data
indicate that prognostic wind models were used more frequently in regions with more complex
meteorology, so these stratification variables are not necessarily independent of one another. It is
also important to recognize that the later classifications, especially the importance of pollutant
transport into the region, are subjective and may not be accurate for all of the episodes in a
region.
As shown in Table 3-6 and Figure 3-10, the stratification on the boundary condition
methodologies indicate a tendency towards underprediction when boundary conditions were
derived from observations rather than from a regional model. The mean bias averaged -9 percent
in the 78 simulations where boundary conditions were derived from observations compared to
+ 1.5 percent mean bias on average in the 51 simulations with ROM-derived boundary conditions.
The gross error was not significantly affected by the boundary condition methodology. The
3-20
-------
Table 3-4. Photochemical models' ozone performance statistic on high ozone day
by region.
Reeion
San Diego. CA
Los- Anceles. CA
Los Aneeles. CA
Ventura. CA
San Joaquin VaJle\
Sacramento. C A
Phoenix. AZ
Houston. TX
Beaumont. TX
D_ .- TX
Bator, Rouee LA
S: Louis. MO
Lake Michiear,
DC-L-;:. Ml
NashuliL T\
N^hMlle. TN
LouisMiie KY
C,n;:r ,-,.-,'.. OH
A^nta. G A
R.j'r.rnond \ A
Philadelphia. M
Philadelphia. NJ
New York. NY
Baltimore. MD
Nev. Encland
Mean (weighted b\ region 1
Standard Deviauon
Minimum
Maximum
O'Served
Domain-wide
Peak (ppK>
156
360
360
180
160
143
157
240
226
:"0
15"
1^3
1 7?
If*
138
138
151
169
201
1 ?*
210
210
244
180
22°
1962
59.8
1380
360 0
Predicted
Domain-wide
Peak tppb)
135
392
382
173
150
139
134
112
151
155
160
160
150
164
139
115
187
169
194
146
201
274
206
169
211
182.7
68.9
1120
392 4
Accuracy of
Domain-wide
Peak
Unpaired (%^
-13
9
6
-4
-6
.3
-15
-53
-33
-9
2
_o
-16
f\
]
-17
24
0
.7
-]h
-4
30
-30
-6
-5
-66
16.5
-53.3
30.4
Mean
Normalized
Bias
(%)
-6
7
7
_*>
-7
-12
.5
-45
-^2
-f-
1
.7
5
-12
-9
_"7
2
4
5
-16
7
20
-3
3
-12
-4.8
12.8
-450
201
Mean Gross
Error
(<£)
31
33
32
21
15
18
24
45
31
^ ^
21
2C>
14
^ *,
20
15
16
19
28
19
26
31
29
15
24
237
7.3
140
452
3-21
-------
Ozone Predictions
Accuracy of Peak Ozone on Highest Ozone Day
Unpaired in Space and Time
K)
"E 30
o>
o
Q.
8 0
i_
O (15)
n
~
(45)
Region
12/18/95
22 Regions
129 Dnta Points
Figure 3-7. The accuracy of the domain-wide peak ozone unpaired in space and time on the highest o'one day in each region.
-------
KJ
Ozone Predictions
Bias on Highest Ozone Day By Region
All Cases with Observed Ozone Above 60 ppb
45 r
Region
12/18/95
22 Regions
129 Data Points
Figure 3-8. The mean normalized bias in predicted ozone concentrations above 60 ppb on the highest ozone day in each
region.
-------
Ozone Predictions
Gross Error on Highest Ozone Day By Region
All Cases with Observed Ozone Above 60 ppb
CO
1
K)
-U
Region
12/18/95
22 Regions
129 Data Points
Figure 3-9. The mean normalized gross error in predicted ozone concentrations above 60 ppb on the highest ozone day in each
region.
-------
Table 3-5. The methodologies used to develop boundary conditions and windfields.
and the characteristics of the geographic setting.
Region
San Diego. CA
Los Angeles. CA
\entura. CA
San Joaquin Yal!e\
Sacramento. CA
Phoenix. A_Z
Ho'jsvr TX
Beaumont. TX
Dallas TX
Ba'.or Rouee- LA
S: L,-u:sMO
L J. K C > t . f"i 1 i! oJ
Deir.r, MI
i Lo'j;s\,:ie K^
N\:sr:Mij. TN
C::,. :.:. ()H
A'..,;i:.. GA
Ri.r.r-. v,.: \ A
Philadeipnia NJ
New ^'ork N^'
Baltimore MD
N'e« EiteJ.irn!
Boundan.1 Condmon
Method
Surface &. Aircraft
Observations
Surface & Aircraft
Observations
Surface Observations
Surface & Aircraft
Observations
Surface & Aircraft
Observations
Surface Observations
Surface Observations
Sun ace Observations
Surface Observations
Surf Ac Aircrar.
Surface Ohse'\auor.j
Surface
-------
C I
conditions and under-predicted by 1 percent with observation-based boundary conditions Furth
stratification of the results based on whether only surface observations or surface and aircraft
observations were used to determine boundary conditions showed only minor differences in the
performance statistics. However, within the group of simulations with boundary conditions
derived from observations, the model performance was slightly better when aircraft and surface
observations were used instead of surface data alone.
Stratification of the model performance on the type of wind model employed showed
larger differences, as shown in Table 3-6 and Figure 3-11. About one third of the simulations
were made using winds derived from prognostic wind models. Simulations made with winds
denved from prognostic models had more negative bias and larger error than those made with
winds denved from diagnostic models. On average, the mean bias was -12 and -0.7 percent in
simulations made with winds denved from prognostic and diagnostic models, respectively. The
mean gross error was 26 and 23 percent in simulations made with winds derived from prognostic
and diagnostic models, respectively The differences in the average domain-wide peak accuracy
were consistent with the other results but smaller (-2 percent with prognostic winds and +4.?
percent with diagnostic winds). It is also worth noting that ozone model performance with the
UAM-DWM wind model was comparable to those for all diagnostic wind models. The under-
and overpredictions of ozone on average may be due to the tendencies for prognostic and
diagnostic models to overestimate and underestimate wind speeds, respectively. The poorer
performance of simulations made with prognostic winds may also reflect the greater complexity of
meteorolog) in the regions where the modelers elected to use this approach. Thus, the results
should be interpreted cautiously because this stratification does not compare wind models in the
same regions and. therefore, other confounding factors, such as the complexity of meteorolog) or
emission inventory problems, may be responsible for the differences.
Stratification of the results on the basis of whether the geophysical characteristics include
onh land or land and water also shows differences, as shown in Table 3-6 and Figure 3-12 The
average gross error in the predictions are 20 and 26 percent on average in regions with onh land
and with land and water, respectively. The other statistics are comparable. Not surprising]),
these results indicate that there is more model error (not bias) in regions with complex
meteorolog).
Lastly, stratification on the extent of pollutant transport into the region indicates there is
negative bias in the regions where regional pollutant transport is a minor factor (see Table 3-6 and
Figure 3-131 On average, the mean bias is -10 percent in these simulations compared to +3
percent in the simulations where regional pollutant transport is typically a more important factor.
The domain-wide peak ozone predictions are more accurate on average in regions where pollutant
transport is a minor factor However, these differences are not large and should be interpreted
cautiously because the stratification is subjective
3-26
-------
Table 3-6. Photochemical models' performance statistics for o/one stratified by boundary condition methodology, windfleld
model, geographic setting, and predominance of pollutant transport into the region.
Stratification Parameter
Boundary conditions
derived from (lie Regional
Oxidanl M(xlel
Boundary conditions
derived from observations
Boundary conditions
derived from surface and
aircraft observations
Boundary conditions
derived from surface
observations
Windfleld derived from
Prognostic Wind Model
Windfield derived from
Diagnostic Wind Models
Windfield derived from
the UAM-DWM
Land-only geographic
setting
Ijind-water gesographic
setting
Significant pollutant
transport into region
Minor pollutant transport
into region
Average
Number of
Tpisode
Days
51
78
25
54
48
81
60
37
92
54
73
129
Mean Domain-wide
Peak Obsi'rv ation
JIT!')
167 1
172 5
200.8
159.2
185.1
161.8
1574
149.9
178.7
1664
173.4
170.5
Menu Domain wide
Peak Prtxlution (ppb)
176 1
I6<>. 5
201 0
154 1
179.5
1 67 6
161.3
150.4
180.8
175.2
169.8
172.1
Mean Arc uracy of
Domain-wide Peak
I Inpaired (%)
6 4
-1 0
-0 3
1 4
-2.0
4.3
2.5
0.5
2.5
64
-1 2
1.9
Mean
Normalized
Bias (%)
1.5
-9.0
-5.9
-10.4
-11.7
-0.7
-3.9
-4.7
^t.8
2.6
-10.2
^.8
Mean (Jross
F-rror (%)
23 7
24.5
23.1
25.0
26.3
22.9
21.9
20.2
25.8
23.3
24 8
24.2
K)
-------
Ozone Predictions
Stratified by Boundary Condition Methodology
400
K)
00
100
Dashed lines are +/- 15 percent
200 300
Observed Concentration (ppb)
Modeled (ROM)
Boundary Conditions
n
Measured and Estimated
Boundary Conditions
400
12/18/95
22 Regions
129 Dnta Points
Figure 3-10. The predicted and observed domain-wide maximum o/.one concentrations stratified by boundary condition
methodology.
-------
Ozone Predictions
Stratified by Wind Modeling Approach
400 r
UJ
t
t-J
vo
QL
100
Dashed lines are +/- 15 percent
200 300
Observed Concentration (ppb)
Prognostic
Wind Model
o
Diagnostic
Wind Model
400
12/18/95
22 Regions
129 Data Points
Figure 3-II. The predicted and observed domain-wide maximum ozone concentrations stratified by wind model type.
-------
Ozone Predictions
Stratified by Geophysical Characteristics
400
U)
I
OJ
o
Land Only
Loss Complex Motoorology
Land and Water
More Complex Meteorology
A
100
Dashed lines are +/- 15 percent
200 300
Observed Concentration (ppb)
400
12/18/95
22 Regions
129 Data Points
Figure 3-12. The predicted and observed domain-wide maximum ozone concentrations stratified by geophysical characteristics.
-------
3.5 SIMULATION STATISTICS FOR THE UAM-IV, UAM-V, AND SAQM
MODELS
As indicated above, most of the simulations were made with the UAM-IV model and there
V.CIN onh one region. Atlanta, where a direct comparison of the UAM-IV and UAM-V models
was made using the same input database (SAI. 1994c). Table 3-7 shows the average ozone
performance statistics for four days of simulations in Atlanta with the two models. The results
show the mean normalized bias and gross error are virtually identical in these simulations (0.0 vs.
-0.6 percent bias and 20.4 vs. 20.9 percent gross error). However, the average accuracy of the
domain-wide peak ozone from UAM-IV and UAM-V models is different, but still reasonably
accurate: +6.9 percent with the UAM-FV model and -3.9 percent with the UAM-V model.
Overall, these results indicate the performance of both models is excellent and comparable in the
Atlanta simulations These results should be interpreted cautiously because the characteristics of
the Atlanta area and ozone episodes probably do not represent a stressful test of the two models
Thai is. technical improvements incorporated into UAM-V may affect the performance mov
significant!) in areas with more complex meteorology or emission patterns, such as Los Angeles
or Lake Michican The cood agreement found in the Atlanta simulations mav not hold for other
TaK; j ?-" Comparison of the average model performance statistics for ozone from the
SA< >M. UAM-IV. and UAM-V models
' :
MtxM'Reeion
I \.\1-I\ in AUanui
I AM-\ in AUanui
S-\(,)V 11; San Jouejuin \ alle>
l'AM-\ us LaU Michiean
l'\M-I\ in 20 Areas
Number
of Dav>.
4
j
2
8
119
Mean Accuracy of
Domain-wide
Peak Unpaired
<<7r!
69
-3 9
-6 8
-? 1
2.4
Mean Normalized
BmsC7r)
0
-06
-30
? 4
-54
Mean Gros^
Error
(<7(\
204
20 9
15 5
16 1
249
Table 3-7 also shows a comparison of the average performance statistics for the SAQM
model in [he San Joaquin Valley simulations and the UAM-IV model in the 20 other regions. The
statistics sho\\ the SAQM simulations have significantly less gross error than the UAM-IV
simulations, but the domain-wide peak ozone is also less accurate than the average from all of the
UAM-IV simulations. The low gross error and near absence of bias in the SAQM simulations is
impress!\e performance and reflects not only the strengths of the SAQM/MM5 models but also
the extensne aerometnc database available for the episodes and the extensive effort to refine
model pcriormance. A process-oriented approach including multi-species comparisons was used
3-31
-------
Ozone Predictions
Stratified by Extent of Pollutant Transport
400
More Pollutant
Transport
Less Pollutant
Transport
A
100
Dashed lines are */- 15 percent
200 300
Observed Concentration (ppb)
400
12/18/95
22 Regions
129 Data Points
Figure 3-13. The predicted and observed domain-wide maximum o/one concentrations stratified by the predominance of
pollutant transport into the region.
-------
over a two-year period to improve model performance for the SAQM simulations and the
performance statistics reflect the large effort.
Average performance statistics for the UAM-V model in the Lake Michigan simulations
are also shown in Table 3-7. The accuracy of the domain-wide peak predictions and the mean
normalized bias are comparable to those for the UAM-IV in the 20 regions; however, the gross
error in the UAM-IV Lake Michigan simulations is significantly lower than the average for the
UAM-IV simulations in other areas (16.1 vs 24.9 percent). Like in the San Joaquin Valley, an
extensive special study aerometric database was available for these episodes and a process-
onented approach including multi-species comparisons was used over a two-year period to
improve model performance for the Lake Michigan simulations. The UAM-V ozone performance
is clearly better than the average UAM-IV performance in other areas on average, and the
difference is probably due to both features of the UAM-V model and the better database and more
extensive model evaluation procedures employed in the Lake Michigan modeling.
3-33
-------
4. CONCLUSIONS AND RECOMMENDATIONS
4.1 CONCLUSIONS
The investigation of photochemical models ozone performance in the SIP applications is
based on three basic performance statistics required by EPA for regulatory applications. The
compilation results for 131 simulation.; in 22 regions showed the average accuracy of the domain-
wide peak ozone unpaired in space and time was +2 ±20 percent. The mean normalized bias in
concentrations above 60 ppb in all simulations was - 5 ±16 percent The mean normalized gross
error in concentrations above 60 ppb in all simulations was 24 ±8 percent.
The EPA estabhshed model performance goals for regulator}' applications of the UAM.
The goals were for the unpaired peak accuracy and mean normalized bias to be within ±15
percent, and for the mean gross error to be less than 35 percent. Almost all (94 percent) of the
simulations met the gross error goal and over half (64 percent) of the simulations met all three
goals. Furthermore, at least one simulation in each region except Houston met all three goals.
Thus, while there were areas with better and poorer model performance, almost all areas had at
leait one episode day with acceptable performance (based on the EPA goals) which could be used
tor control strategy evaluations.
Stratification of the results on selected methodological procedures did not show large
differences Simulations made with boundary conditions derived from observations predicted
lower ozone on average than those with boundary conditions derived from the Regional Oxidant
Model. Simulations made with prognostic wind models had larger error and more negative bias
on average compared to simulations made with windfields derived from diagnostic wind models.
howe\er. the areas where prognostic models were employed are also regions with more complex
meteorolog\ and where larger model error would be expected. The negative bias in ozone with
the prognostic models ma\ be due to their tendency to overestimate wind speeds, but it also could
be due to cither factors.
Stratification of the ozone performance statistics on geographic characteristics also
showed modest differences. For example, there was greater model error on average in regions
with land-water interfaces (26 percent versus 20 percent). This result was expected because
regions with land-water interfaces have more complex meteorology. However, regions where
pollutant transport into the region was expected to be significant had lower model error and less
bias than those without significant pollutant transport. This result may suggest inflow boundary
conditions were adequately estimated in the regions with significant transport in these simulations.
but there could also be other compensating factors which explain the differences. Thus, all of the
stratification results should be interpreted cautiously.
A comparison of results from applications of the UAM-IV and UAM-V models to the
Atlanta regions showed excellent and comparable performance. The average results for the four
simulated days differed only in the accuracy of the domain-wide peak predictions, which were
slightly better with the UAM-V model. Overall, the results were too similar to distinguish
4-1
-------
between the models. The characteristics of the Atlanta region and episodes may not be sufficient
to stressfully test the differences between the UAM-FV and UAM-V models, so the absence of
differences in these test should be interpreted cautiously.
A comparison of the SAQM model performance in the San Joaquin Valley and the UAM-
V model performance in Lake Michigan to the average UAM-TV model performance in 20 other
regions showed the SAQM and UAM-V models had significantly less gross error in predicung
ozone concentrations above 60 ppb. The accuracy of the domain-wide peak concentrations and
the mean normalized bias were similar for the SAQM, UAM-V, and UAM-IV on average. The
better overall performance of the SAQM and UAM-V models in these simulations is probably due
to both features of the models and the better databases and more extensive model evaluation
procedures employed in the modeling in these two areas. Process-oriented approaches including
mulu-species comparisons were used to improve model performance for the simulations and the
performance statistics reflect the more comprehensive efforts.
4.2 RECOMMENDATIONS
The analysis of photochemical grid model performance using three statistics for ozone
provides a good starting point for model evaluation, but it does not represent a comprehensive
approach to model evaluation A more thorough approach would include the following elements
Use of additional model performance statistics, such as those listed in Table 4-1
Evaluation of the model performance for precursor species, such as NO. NO2. NO . CO.
lumped VOC species (OLE. PAR. TOL. XYL. ETH. FORM, and ALD). in addition to
those for ozone
Use of a process-oriented approach that reviews all of the statistical results and graphical
displa\s for baseline runs, alternate baseline runs, and other sensitivity runs, and seeks to
idenuf\ potential problems or inconsistencies, including compensating errors.
In order to carry out more comprehensive analyses, the evaluation team must have access to all of
the model input, model output, and observation files used in the simulations and all of the
modeling documentation, and have appropriate display software to visualize the predictions and
observations
4-:
-------
Table 4-1 Additional statistical parameters for model performance evaluation.
L Ahbreviauon
PAPST
NPAPST
PAPS
NPAPS
PAPT
NPAPT
PA
PNBIAS
PSTRK,*
,,<
PERROK
BUS
ERROR
Parameter
Absolute accuracv of domain-wide maximum 1-hr concentrauons paired in space and ume'
Normalized accuracv of domain-wide maximum 1-hr concentrauons paired in space and ume"
Absolute accuracy of domain-wide maximum 1-hr concentrauons paired in space and unpaired
in t me*
Normalized accuracv of domain-wide maximum 1-hr concentrauons paired in space and
unpaired ume'
Absolute accuracy of domain-wide maximum 1-hr concentrauons paired in ume and unpaired in
space'
Normalized accuracy of domain-wide maximum concentrauons paired in ume and unpaired in
space*
.
Absolute accuracv of domain-wide maximum 1-hr concentrauons unpaired in space and ume
Mean normalized bias of predicted and observed maximum 1-hr concentrauons at all monitoring
stations'
Mcar normalized error of predated and observed maximum 1-hr concentrauons at all
mor.itor.nc stations*
Mean absolute bias o! predicted and observed maximum 1-hr concentrauons at all monitoring
Mean absolute error of predicted and observed maximum 1-hr concentrations at all monitoring
station1.*
Nka:, absolute bus of all predicted and observed concentration pairs where the observed
exceeds a minimum concentration
Mear. absolute error of a!! predicted and obsened concentration pairs where the observed
exceeds a minimum concentration
co p^ cxm^nuaiiorL^ excted the minimum concentraiiorLc
4-3
-------
5. REFERENCES
CARS (1995a) Revisions to the base case and future year Urban Airshed Model simulations
for Ventura County in support of the 1994 State Implementation Plan. Report prepared
by California Air Resources Board, Sacramento. CA.
CARB (1995b) Photochemical modeling of the Greater Sacramento Area in support of the
1994 State Implementation Plan. Report prepared by California Air Resources Board.
Sacramento, CA.
C.ARB (1995c) San Joaquin Valley SIP Modeling: Model Performance. Report prepared by
California Air Resources Board, Sacramento, CA.
EPA (1991) Guidelines for regulatory application of the Urban Airshed Model. Report
prepared by U.S. Environmental Protection Agency, Research Triangle Park. NC,
EPA450/4-91-013.
Georgopoulos P G (1995) Ozone sip modeling technical support documentation summary for
the New Jersey - Philadelphia CMSA area. Report prepared by Ozone Research Center.
Environmental and Occupational Health Sciences Institute, Piscataway, NJ, Technical
ORC-TR9502.
Kammski M.A. (1995) Urban airshed modeling of the middle Tennessee modeling domain:
model performance evaluation. Report prepared by University of Tennessee, Knoxville.
TN
LADCO (1994) Lake Michigan Ozone Study: evaluation of the UASM-V photochemical grid
model in the Lake Michigan region, Version 2.0. Report prepared by Lake Michigan Air
Directors Consortium. Des Plaines. IL.
NYDEC (1994a> New York urban airshed modeling For June 14 to 20. 1987 base case.
New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE3.
NYDEC (1994b) New York urban airshed modeling For July 5 to 11, 1988 base case.
New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE1.
NYDEC (1994c) New York urban airshed modeling For June 18 to 22, 1988 base case.
New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE2.
NYDEC (1994d) New York urban airshed modeling For July 16 to 20, 1991 base case.
New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE4.
Roth P.M.. Reynolds S.. Tesche T.. and Dennis R. (1991) A conceptual framework for
evaluating the performance of grid-based photochemical air quality simulation models.
Report prepared b\ Envair. San Anselmo. CA.
5-1
-------
SAJ (1990) User's guide to the Urban Airshed Model (UAM-IV). Report prepared by
Systems Applications Internationa], San Rafael. CA.
SAJ (1993) User's guide: Lake Michigan Ozone Study photochemical modeling system.
Report prepared by Systems Applications International, San Rafael, CA.
SAJ (1994a) Photochemical modeling of the Lake Michigan region for the 1991 Lake
Michigan Ozone Study (LMOS) using the Nested-grid Urban Airshed Model (UAM-V).
Repon prepared by Systems Applications International, San Rafael, CA.
SAJ (1994b) Application of the Urban Airshed Model to Baton Rouge, Louisiana for three
multi-day ozone episodes. Volume IE: diagnostic/sensitivity analysis and model
performance evaluation. Repon prepared by Systems Applications International. San
Rafael. CA. SYSAPP94/090.
SAJ (1994c) Comparison of the UAM-W and UAM-V photochemical models for three
Atlanta-area ozone episodes. Report prepared by Systems Applications International. San
Rafael. CA.SYSAPP94/106.
SCAQMD (1994) 1994 Air Quality Management Plan, Technical Report V-B: Ozone
Modeling Performance Evaluation. South Coast Air Quality Management District,
Diamond Bar. CA
SDAPCD (1994) Request for ozone reclassification - urban airshed modeling technical
support document. Repon prepared by San Diego Air Pollutant Control District, San
Diego. CA
Tesche T.W.. Lurmann F.W.. Roth P.M.. Georgopoulos P., Seinfeld J.H., and Cass G.
(19901 Improvements of procedures for evaluating photochemical models. Report
prepared for California Air Resources Board. Radian Corp.. Sacramento. CA, ARJ3
Contract No A832-103.
TNRCC (1994a'i El Paso. Texas ozone nonattainment areas base case report performance
e\ aluation Report prepared by Texas Natural Resource Conservation Commission.
Austin. TX.
TNRCC (1994b) Houston/Galveston Beaumont/Port Arthur ozone nonattainment areas base
case repon performance evaluation. Report prepared by Texas Natural Resource
Conservation Commission. Austin, TX.
TNRCC (1994c) Dallas/Ft Worth ozone nonattainment areas base case report performance
evaluation. Report prepared by Texas Natural Resource Conservation Commission,
Austin. TX.
5-2
-------
TECHNICAL REPORT DATA
'Please read Instructions on reverse before corr.ple;ing'
FEPC-FT N:
EPA-454 'R-96-004
3 RECIPIENT'S ACCESSION N;
TITLE ANT. SVETITLE
Compilation of Photochemical Models'
Performance Statistics for 11/94 Ozone SIP
Applications
5 REP-OPT DATE
July 1996
6 PERFORMING ORGANIZATION CODE
S PERFORMING ORGANIZATION REPOFT NC
9 PERFORMING ORGAN!ZATICN NAME ANT ADDRESS
10 PROGRAM ELEMENT NC
Sonoma Technology Inc.
5510 Skylane Boulevard, Suite 101
Santa R c s a, CA 9 5 4~2
11 CONTRACT - GRANT NC
EPA Contract No
SF::;;:FIN- AGEN:- NAME ANT ArrpEr;
U.S. Environmental Protection Agency
Office :f Air Quality Planning and
Standards
Err.iss.ions, Monitoring & Analysis Division
Research Triangle Park, NC 2~'7ll
13 TYPE OF REPOPT ANT PERIOE COVEPET
Final Report
14 SPONSORING AGENT> CODE
EP;
'k Assignment Manaaer: Shao-Hancr Chi
three
devel
The e
s tat i
ccnce
perf o
stati
model
betwe
sica
s ' per
en the
is a compilation of the model performance statistics of
chemical models 'UAK-IY, UA1-1-V, and SAQM) used in the II''94
Implementation Plans (SIP) applications. The models were
24 ozone ncnattainment regions in 1993-1995 to support the
of emissions control strategies for the 1994 ozone SIPs.
icn focuses or. three EPA recommended basic model performance
measures of the models' ability to predict ambient ozone
ens. It does not include an evaluation of the models'
on other species (such as NO, N02, Noy, and VOCs) or other
measures on ozone. For a more complete evaluation of the
formance both spatial and temporal analyses of the matching
predicted and observed concentrations are desirable.
KE'i A"f" ANT POCUMEKT ANALYSIS
b IDENTIFIERS'OPEN ENDED TERMS
c COSATI
Field ^ro
Ozone SIF Applications Modeling
Photochemical Model Performance
CISTFIEVTICN STATEMENT
Release Unlimited
19 SECURITY CLASS IKfport/
Unclassified
2C SECURITY CLASS I Page!
Unclassified
21 NC OF PAGES
156
EPA Fora. 222C-1 (R«v 4-77)
PREVIOUS EDITION IS OBSOLETE
------- |