EPA-454/R- 96-004
  Compilation of Photochemical Models'
Performance Statistics for 11/94 Ozone SIP
                  Applications
                     Julv 1996
         U.S. Environmental Protection Agency
       Office of Air Quality Planning and Standards
       Emissions, Monitoring, and Analysis Division
      Research Triangle Park, North Carolina 27711

           U.S. Environmental Protection Agency
           Region 5, Library (f»u2J)
           77 West Jackson Boulevard, 12th Ftoor
           Chicago, »L 60604-3590

-------
                                   DISCLAIMER
      The information of this report has been reviewed in the entirety by the U.S.
Environmental Protection Agency (EPA), and approved for publication as an EPA document.
Mention of trade names, products, or services does not convey, and should not be interpretated
as conveying official EPA approval, endorsement, or recommendation.
                                   f C'P M'"M*  j! .r

-------
                             ACKNOWLEDGMENT
      This report has been prepared by Sonoma Technology Incorporated and funded by the U.
S. Environmental Protection Agency under Contract No. 68D30020 with Shao-Hang Chu as the
Work Assignment Manager.
                                       111

-------
                        TABLE OF CONTENTS


                                                              Page

ACKNOWLEDGMENT	  iii
LIST OF FIGURES	  vi
LIST OF TABLES  	vii

1  INTRODUCTION	  1-1

2. THE PHOTOCHEMICAL MODEL SIMULATIONS	2-1

3  MODEL PERFORMANCE 	3-1
     3.1   INDIVIDUAL SIMULATION STATISTICS 	3-2
     3.2   SIMULATION STATISTICS AVERAGED FOR EACH REGION	 3-15
     3.3   SIMULATION STATISTICS ON THE HIGHEST OZONE DAY IN
          E^CH REGION  	 3-20
     3 4   SIMULATION STATISTICS STRATIFIED BY MODELING
          METHODOLOGY AND GEOGRAPHIC SETTING	 3-20
     3 5   SIMULATION STATISTICS FOR THE UAM-IV. UAM-V. AND SAQM
          MODELS  	3-31

4  CONCLUSIONS AND RECOMMENDATIONS	4-1
     4 1   CONCLUSIONS   	4-1
     4 2   RECOMMENDATIONS  	4-2

5  REFERENCES            .     	5- J

-------
                                  LIST OF FIGURES
Figure

3-1.    The predicted and observed domain-wide maximum ozone concentrations for
       SIP applications	  3-10

3-2.    The number of simulations in each region with performance that met all
       three of EPA's ozone performance goals  	  3-13

3-3    The percentage of simulations in each region with performance that met
       all three of EPA's ozone performance goals 	3-14

3-4    The average accurac>  of the domain-wide peak ozone unpaired in space and
       time in each region	   3-17

3-5.    The average normalized bias in predicted ozone concentrations above 60 ppb
       in each region	   3-18

3-6    The average normalized gross error in predicted ozone concentrations above
       60 ppb in each region   	   3-19

3-7    The accuracy of the domain-wide peak ozone unpaired in space and time on
       the highest ozone day in each region  	   3-22

3-8    The mean normalized bias in predicted ozone concentrations above 60 ppb on
       the highest ozone day in each region  	   3-23

3-9    The mean normalized gross error in  predicted ozone concentrations above
       60 ppb on the highest ozone day in each region    	3-24

3-10  The predicted and observed domain-wide nv • 'mum ozone concentrations
       stratified by boundary condition methodolog.  	3-28

3-11  The predicted and observed domain-wide maximum ozone concentrations
       stratified by wind model type	3-29

3-12  The predicted and observed domain-wide maximum ozone concentrations
       stratified by geophysical characteristics  	 3-30

 3-13  The predicted and observed domain-wide maximum ozone concentrations
       stratified by  the predominance of pollutant transport into the region   	3-32
                                           VI

-------
                                   LIST OF TABLES


Table

2-1.    Regions included in the study	2-2

3-1.    The photochemical models' ozone performance statistics for individual
       baseline simulations  	3-3

3-2.    The extent to which EPA performance goals were achieved in the SIP
       applications  	3-11

3-3.    Average photochemical models' ozone performance statistics  by region  	 3-16

3-4    Photochemical models' ozone performance statistic on high ozone
       day b\ region     	 3-21

3-5    The methodologies used to develop boundary conditions and windfields.
       and tne characteristics of the geographic setting	3-25

3-6    Photochemical models' performance statistics for ozone stratified by
       boundary condition    methodologv windfield model, geographic setiing.
       and predominance of pollutant transport into the region	3-27

3-"    Comparison of the average model performance statistics for ozone from
       the SAQM. UAM-IY. and UAM-V models  	
       Additional statistical parameters for model performance evaluation	  4-3
                                          Vll

-------
                                  LIST OF TABLES


Table                                                                             Page

2-1.    Regions included in the study	   2-2

3-1.    The photochemical models' ozone performance statistics for individual
       baseline simulations	   3-3

3-2.    The extent to which EPA performance goals were achieved in the SEP
       applications	3-11

3-3.    Average photochemical models' ozone performance statistics by region	3-16

3-4.    Photochemical  models' ozone performance statistic on high ozone
       day b\  region	3-21

3-5.    The methodologies used to develop boundary conditions and windfields,
       and the characteristics of the geographic setting	3-25

3-6.    Photochemical  models' performance statistics for ozone stratified by
       boundary condition  methodology,  windfield model, geographic setting,
       and predominance of pollutant transport into the region  	3-2"

3-7.    Comparison of the average model performance statistics for ozone from
       the SAQM. UAM-IV.  and UAM-V  models	3-31

4-1    Additional statistical  parameters for  model performance evaluation	4-3
                                          Vlll

-------
                                  1. INTRODUCTION
       The scientific credibility of photochemical modeling studies depends on the soundness of
 the model formulation, the adequacy of the aeometric database, and the accuracy of the model's
 predictions for specific applications (Tesche et al., 1990: Roth et al., 1991). The accuracy of
 specific applications must be evaluated because the quality and representativeness  of input data
 can strongly influence the model performance  An investigation was carried out to evaluate the
 performance of three grid-based photochemical models in recent applications.  The photochemical
 models were the Urban Airshed Model Version IV (UAM-IV). the Urban Airshed Model vanable
 grid version (UAM-V). and the San Joaquin Valley Air Quality1 Model (SAQM). All three models
 employed the CB4 chemical mechanism for these applications. The models were applied in 24
 ozone nonanainment regions in 1993-1995 to support the development of emissions control
 strategies for the 1994 ozone State Implementation Plans (SIPs). The models were used to
 simulate  the relauonship  between nitrogen oxides (NOX) and volatile organic compound (VOC)
 emissions, and ambient ozone concentrations under current episodic conditions and for future
 emission scenarios The  baseline model applications provide an unique opportunity' to examine
 photochemical models performance for a large number of regions and episodes. Furthermore,  the
 SIP simulations were performed using  generally consistent methodologies

       The U S Enuronmental Protection Agency (EPA) established  model performance goals
 fo; the SIP applications   In order to assess the extent to which the baseline simulations met the
 performance goals, the EPA required states to report three statistical measures of the model's
 abih'.\ to predict ambient ozone concentrations. These measures are:

   1    The normalized accurac> of domain-wide maximum 1 -hr concentration unpaired in space
       and time

  2    Mear normalized bias of all predicted and observed concentration pairs where the
       observed concentration.-, exceed 60 ppb

  3    Mean normalized error of all predicted and observed concentration pairs where the
       observed concentrations exceed 60 ppb

This report reflects the status of November 1994 SIP applications model performance results. As
 the Suites continue their modeling efforts these results are expected to  change.  This document is
 a compilation of the EPA recommended basic model performance statistics for the 24 ozone
 nonattamment areas in the November 1994 SIP applications. For a more complete evaluation of
the models' performance  both spatial and temporal analyses of the matching between the
predicted and observed concentrations are desirable. Thus, it is important to recognize that the
scope  of this evaluation is quite limited The evaluation focuses on three statistical measures of
the models' ahilm  to predict ambient ozone concentrations.  It does not include an evaluation of
the models' pertormance  on other species (such as NO, NO,. NOy, and VOCs) or for other
statistical measures of its  performance on ozone, which may be important for establishing the
smtabilm  of simulations  for use in control strategy development

                                          1-1

-------
                 2.  THE PHOTOCHEMICAL MODEL SIMULATIONS
       The statistical performance data for this study were obtained from reports and briefings
 submitted to the EPA. Model performance results for the 22 regions listed in Table 2-1 were
 included in the database. In some cases, more than one nonattainment area was included in a
 single modeling domain and separate statistics were included for each area if they were provided
 in the modeling documentation (e.g., in the Houston and Beaumont, Texas areas).  In most cases
 the reported statistics apply to a single nonattainment area. Data for several other regions were
 received but not included as explained below.

       The photochemical model simulations were carried out by numerous modeling groups
 under the direction of state and local agencies. EPA provided modeling guidelines (EPA. 1991)
 to promote consistency between the applications. The guidelines allowed considerable flexibility
 so that regions with different types of databases, different geographic settings and  meteorological
 conditions, and different levels of modeling expertise could complete the applications. As shown
 in Table 2-1. most of the simulations were performed with the UAM-IV model (SAI, 1990),
 version 6.21.  However, simulations for the Lake Michigan Ozone Study (LMOS) region (i.e.. the
 four states surrounding Lake Michigan* were performed with the UAM-V model (SAI.  1993.
 1994a i  Simulations for the greater Atlanta, Georgia region were carried out with the UAM-IV
 and UAM-V models  Comparable efforts were made in the applications of both models in Atlanta
 which ;ujihtates direct comparison of the results. Differences in the predictions from the UAM-
 IV and UAM-V models are small for Altanta and the UAM-IV results were used in the summarx
 of mode! performance. A  comparison of performance statistics for the Atlanta UAM-IV and
 UAM-V simulations are included In addition, the SAQM model was used for California's San
 Joaqmn Valle> modeling domain  The SAQM model is an enhanced version of the Regional Acid
 Deposition Model (RADM)

       The photochemical simulations were performed for two- to seven-day periods with one or
 more days with 1-hr ozone concentrations above 120 ppb. The  results  for the first day of each
 period were excluded from statistical analysis because it takes time for  the simulations to become
 driven b\ emissions rather than initial concentration estimates.  In some large modeling domains
 (e.g . San Joaquin Valley), both the first and second days of the simulations were treated as "start-
 up1 days. Table 2-1 shows the number of episode days modeled in each region.  Only modeled
episode days with observed ozone concentrations above 120 ppb were  included in the database.
On average, six episode days were modeled in each region. The number of days modeled varied
significantly between the regions. For example, 17 days were modeled in the New York area and
only one day was modeled in the Sacramento area. The number of episode days modeled was
generally higher in the areas with more severe ozone problems (i.e., in Los Angeles. Houston, and
New York) The total number of episode days included in the database is 131 davs.
                                          2-1

-------
                         Table 2-1.  Regions included in the study.
Reeion
San Diego, CA
Los Anseles. CA
Ventura. CA
San Joaquin Valle\. CA
Sacramento. CA
Phoenix. AZ
Houston. TX
Beaumont. TX
Dallas TX
Baton Rouce. LA
St Louis MO
Lake Michigan
Detroit. MI
.Nashville TN
Louisville. KV
Cincinnati. OH
Atlanta GA
Richmond \ A
Philadelphia. NJ
New York. NY
Baltimore. MD
New Eneland
Model
UAM-IV
UAM-IV
UAM-IV
SAQM
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-I\'
L'AM-I\'
UAM-V
L'AM-IV
UAM-IV
UAM-IV
LAM-TV
LAM-TV
LAM -IV
UAM-IV
UAM-IV
UAM-Pv'
UAM-W
Number of Episode
Days Simulated
2
10
3
9
^
1
2
12
10
9
7
8
8
4
4
->
•>
6
4
6
5
17
4
4
Reference
SDAPCD. 1994
SCAQMD, 1994
CARB. 1995a
CARE. 1995c
CARB. 1995b
Briefing
TNRCC. 1994b
TNRCC. 1994b
TNRCC. 1994c
SAI. 1994b
Briefing
LADCO, 1994 &.
Briefinc
Briefinc
Kaminsb. 1995
Bnefine
Bnefine
SAI. 1994c
Bnefine
Georgopoulos. 1995
NYDEC. 1994a.b.c.d
Briefing
Briefing
       A.s noted above, the database does not include results for all of the ozone nonattainraent
areas  Tne UAM-IV model was applied in El Paso, Texas without emissions for a significant
portion oi the modeling domain (Juarez. Mexico) and the results showed gross underprediciion of
observed ozone concentrations (TNRCC.  1994a).  The El Paso simulation results were excluded
because the\ were based on grossly inadequate inputs compared  to all other simulations
                                          2-2

-------
 considered here  Results for the Santa Barbara and Southeast Desert Air Basin of California were
 not reported separately from those for Ventura and the South Coast Air Basin of California.
 respectively.

       The techniques employed to prepare the meteorological and air qualm1 model inputs
 varied between the regions.  Numerous modelers used prognostic meteorological models, such as
 the MM5. CSUMM, and CALRAMS. to develop the hourly 3-dimensional wind fields, while
 other modelers chose diagnostic wind models, such as CALMET, U AM-Diagnostic Wind Model.
 and the Regional Oxidant Model (ROM) meteorological model Various models were also used
 to develop mixing height (RAMMET. MIXEMUP, CALMET, etc.). Likewise, boundary-
 concentration inputs were obtained from the EPA-  recommended default  values, surface
 observations, surface and aircraft observations, and regional model estimates (ROM). The
 difference in model input preparation procedures were often determined by the extent of the
 aerometnc database in the regions and the complexity of the meteorology. Prognostic
 meteorological models were used more frequently  in areas with coastal meteorology and/or
 complex terrain (e.g.. Los Angeles and LMOS) rather than flat areas with continental
 meteorolog)  To the extem  possible, the database  was coded with the model input preparatior
 procedure codes

       The photochemical model performance statistics incorporated into the database were
thus:  provided b\  the modeling groups in reports and bnefings to the EPA. STI did not
recompute the statistic trom the model output files and the observed concentration data because
the\ were not made available for use in the stud\.  The absence of actual model output and
observation files inhibited qualm assuring the statistical dam and limited the scope of the analysis
to those statistics reported by the modeling groups. While numerous modeling groups reported
more than  the required statistics, only the three EPA-required statistics were common to all of the
application reports
                                          2-3

-------
                              3.  MODEL PERFORMANCE
       This evaluation of model performance in SIP applications focuses on the models' ability to
 predict the domain-wide peak ozone concentration and the concentrations at all locations with
 observed ozone concentrations above 60 ppb.  The three statistical measures recommended in the
 EPA photochemical modeling guidelines (EPA,  1991) are used. The measures are:

   1.  The normalized accuracy of domain-wide maximum 1-hr concentration unpaired in space
       and time (Aj:
              100
                      avmatr-waf peak      Oomair.~*idf peak
                              " aomuir -*idf peak
(l)
       Mean normalized bias of all predicted and observed concentration pairs where the
       observed concentrations exceed 60 ppb (NBIAS^):
            100 A. (Predx,  -  OBS: ,)
  \7?/4 s  = — V - - - :_
 u'.tri A includes all of the predicted and observed concemraii(>
 r>ai>> u;i7;  observed com fntranor.' abo-.e 60
       Mean normalized error of all predicted and observed concentration pairs where the
       ob>erved concentrations exceed 60 ppb (NERROR^):
              m<,jL,  ?nd .  - OBS'
 \ERROR  = -— >	
       A induih'-' all of the predicted and obser\'ed concentratio
      \\iil.  <•/ -
-------
       The EPA guidelines set statistical performance goals for regulatory' applications of the
UAM. The goal for the peak accuracy unpaired in space and lime is within ±15 percent  The goal
for the mean normalized bias is within ±15 percent.  The goal for the mean normalized error is
less than 35 percent. These goals are primarily based on historical performance of the model.
rather than analyses of accuracy requirements for intended use of the model (e.g.. analyses that
relate accuracy of baseline simulations to accuracy of the response of the model to emission
changes).
3.1    INDIVIDUAL SIMULATION STATISTICS

       The statistical performance data for the 129 individual simulations are shown in Table 3-1
and Figure 3-1  The table indicates the region, the date of the episode day, the model, the
observed domain-wide maximum  1-hr ozone concentration, the predicted domain-wide maximum
1-hr ozone concentration, the accuracy of the domain-wide peak, the mean normalized bias, and
the mean normalized error.  The average, standard deviations, minimum, and maximum values are
shown at the bottom of the table.  Table 3-2 and Figures 3-2 and 3-3 illustrate the extent to
which the individual simulations met EPA's performance goals.

       The results indicate that the models predicted an average domain-wide peak of 172 ±51
ppb \\hen the axerage observed domain-wide peak was 170 ±44 ppb.  On average, the accuracy
of the predicted domain-wide peak ozone concentration unpaired in space and time was +2 ±20
percent  The near absence of bias on average in the unpaired peak ozone is an excellent result for
air qualm models  Even at the extremes, the ozone peak accuracy is only a factor of  two in error
The lowest accuracx domain-wide peaks in individual simulations were for October 12. 1991 in
Houston where  the ozone was under-predicted by 53 percent (112 ppb predicted versus 240 ppb
observed) and for June 21.  1988 in Nev.  York where the ozone was overpredicted by 100 percent
(298 ppb predicted \ersus 149 ppb observed).  At least one simulation in every region met the
±15 percent performance goal for the unpaired peak accuracy   Overall.  77 percent of the
individual simulation met this performance goal  For areas that modeled four or more episode
davs. over 75 percent of the episode days modeled in the following regions met the EPA
performance goal for unpaired peak ozone: Los Angeles. Dallas.  St. Louis. Lake Michigan.
Detroit. Atlanta, and Baltimore The statistics and Figure 3-1 show there is considerable  scatter
in the unpaired  peak predictions. The correlation between the observed  and predicted domain-
wide peak is moderate. R: = 0.54. which indicates the models are  able to explain 54 percent of the
variance in the peak ozone.

       The mean normalized bias for ozone concentration above 60 ppb averaged
-5 ± 16 percent  in the SIP applications. The negative bias is consistent with the majority of
previous UAM  simulations. The small average bias is comparable to the uncertainty in the ozone
observations (±5 ppb) This level of performance on average is excellent for air quality models.
At the extremes, the mean normalized bias was as low as -45 percent (October 12. 1991 in
Houston) and as high as +55 percent (June  21, 1998 in New York) in individual simulations   At
least one simulation in every region met the ±15 percent performance  goal for the mean domain-
wide peak ozone was overpredicted by 6.4 percent on average with the ROM-derived boundary

                                          3-2

-------
                Table 3-1.   The photochemical models' o/one performance statistics for individual baseline simulations.
1'iigc 1 of 7
Region
San Diego, ("A
San Diego, CA
Los Angeles, CAh
Los Angeles, CAh
Los Angeles, CA
Los Angeles, CAh
Los Angeles, CA
Los Angeles, CAh
Los Angeles, CAb
Los Angeles, CAS
Los Angeles, CAh
Los Angeles, CAh
Ventura, CA
Ventura, CA
Ventura, CA
San Joaquin Valley, CA
San Joaquin Valley, CA
Sacramento, CA
Phoenix, AZ
Phoenix, AZ
Houston, TX
Model
HAM IV
HAM IV
HAM IV
UAM IV
UAM IV
UAM IV
UAM [V
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
SAQM
SAQM
UAM-IV
UAM-IV
UAM IV
UAM IV
Hpisode Date
August 2", 1989
September 21, I'>81>
Augusi 27, 1987
August 28, 1987
.fuly 14, 1087
July 15, 1987
June 24, 1987
June 25, 1987
June 6, 1985
June 7, 1985
Septembers, 1987
September 9, 1987
September 17. 1984
September 6, 1984
September 7, 1984
Augusi 5, 1990
August 6, 1990
July 1.3, (990
August 10, 1992
June 14, 1991
August 1, 1990
Observed
Domain wide
Penk
()
220
230
240
.360
360
330
260
140
170
180
150
160
143
157
140
138
Predicted
Domain-wide
Penk
(PI*)
141
135
216
.322
255
288
251
250
392
.382
254
252
143
175
173
139
150
139
134
132
163
Accuracy of
Domain-wide
Peak t inpaireci
(%)
-8.4
-13.5
-10 0
11.0
2.0
31 0
90
4.0
9.0
6.0
-23.0
-3.0
2.1
2.9
-3.9
-7.3
-6.3
-28
-14.6
-5.7
18 1
Mean
Normali/ed
Bias'
(.%)
-12.9
-5.6
-22.0
-8.0
-24.0
-12.0
-25.0
-26.0
7.0
7.0
-27.0
-21.0
-12.0
-12.0
-2.0
1.0
-7.0
-12.0
-5.4
4.4
^to.
-------
                Table 3 I.   The photochemkal models' o/one performance slalistics for individual baseline simulations.
Page 2 of 7
Region
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Houston, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Beaumont, TX
Model
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
Fpisode Date
Inly 27, 1 WO
July 28, 1WO
July 2''. |WO
July .10, IWO
July 11, IWO
May 16, |9S8
May 17, 1988
May 18, 1188
October 10, 1991
October II, 1991
October 12, 1991
July 28, 1990
July 30. IWO
July 31, IWO
May 18, l<>88
May 19, 1 988
May 19, 1088
October 10, |9O]
October II, I9<>|
October 12, 199 1
October 1 1, l<><)|
( )bserve
185
179
263
25.3
211
139
153
173
151
111
112
146
174
151
163
139
139
109
127
115
121
Accuracy of
Domain-wide
Peak Unpaired
(%)
2 8
19 3
29.6
32.5
40.7
13.1
-27.1
-17.6
23.8
-30.7
-53.3
13.2
25.2
-33.2
25.4
-22.8
-22.8
-24.3
-11.8
-42.5
II 7
Mean
Normalized
Bias"
(96)
-25.7
-31.3
-14.1
17.7
-11.4
-32.9
-28.6
-36.3
-8.9
-5.4
^t5.0
10.5
23.1
-31.8
-13.9
-27.5
-275
-6.6
-5.7
-42.8
-342
Mean Gross
I'.rror'
(%)
10.1
34.6
32.2
32.6
43.9
36.0
38.0
38.2
21.0
21.7
45.2
19.5
32.5
32.0
16.6
29.9
29.9
17.3
14 1
41 2
14.2
u»

-------
Table 3-1.   The photochemical models' o/one performance statistics for individual baseline simulations.
Page 3 of 7
Region
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Dallas, TX
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
Baton Rouge, LA
St. Louis, MO
St. Louis, MO
St. Louis, MO
St. Louis, MO
St. Louis, MO
Model
UAM-IV
HAM -IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
Hpisode Date
August 1, 1('Q|
August 25, l"88
August 26, 1988
August 27, 1990
August 28, 1990
August 2(>, 1900
August 30, 1900
July 31, 190|
June 18, 1987
August 15, 1989
August 16, 1989
August 21, 1902
July 27, 1989
July 28, 1989
May 24, 1990
May 25, 1990
August 15, 1988
August 16, 1988
August 17, 1988
August 18. 1988
July 7, 1988
( )liser\ IN!
Domain wide
Peak

-------
Table 3-1.   The photochemical models' o/one performance statistics for individual baseline simulations
Page 4 of 7
Region
St l^oiiis, MO
St Louis, MO
St Louis, MO
Ijike Michigan
Lake Michigan
Lake Michigan
Ijike Michigan
Lake Michigan
Lake Michigan
Lake Michigan
Lake Michigan
Detroit, MI
Detroit, MI
Detroit, MI
Detroit, MI
Louisville, KY
Ixniisville, KY
l/niisville, KY
Nashville, TN
Nashville, TN
Nashville, TN
Model
UAM-IV
UAM-IV
HAM IV
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-V
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
F.pisode Date
July R, I"8R
Inly ", 1988
June 24, 1987
August 26, I°Q|
July 17, 1991
July 18, |99|
July 19, 1991
June 20, 1991
June 21, 1991
June 26, 1991
June 28, 1991
August 2, 1988
August 3, 1988
July 5, 1988
July 7, 1988
luly 20, 1987
June 2.1, 1988
June 27. |990
August 1, 1988
August 3, 1 988
July 9. I <>X8
Observed
Domain- w ide
Peak
'>)
I61
1 26
1 60
I 78
1 30
1 69
1 53
I2I
I 34
I 75
1 33
I44
1 64
1 47
1 68
I5I
1 37
1 48
1 38
1 35
I 38
Predicted
Domain-wide
Peak
(PPh)
1 60
1 40
1 52
1 50
1 42
I6I
I44
1 09
LSI
1 65
1 37
1 42
1 48
1 28
1 64
1 87
1 38
1 60
1 15
132
139
Accuracy of
Domain-wide
Peak I Inpaired
(%)
-1.8
II 1
-5.0
-16.0
2.0
-5.0
-60
-10.0
13.0
-6.0
3.0
-1.4
-9.8
-12.9
-2.4
23.8
0.7
8.1
-16.7
-2.2
07
Mean
Normalized
Bias'
(%)
-7 0
-8.0
7.0
5.0
11.0
4.0
13.0
-7.0
-12.0
3.0
10.0
-12.0
-24.0
-15
-12.0
1.7
-28.1
6.0
-7.2C
-9.4C
-9 ()'
Mean Gross
T.rror'
(%)
20.0
16.0
24.0
14.0
17.0
16.0
18.0
14.0
20.0
14.0
16.0
22
28
20
22
15.9
29.5
16.1
15.1
20.2
20.3

-------
                Tahlc 3  1   The photochemical models' o/otie performance statistics (or individual haseline simulations.
                                                                                                                   Page S of 7
Region
Nashville, TN
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Cincinnati, OH
Atlanta, GA
Atlanta, GA
Atlanta, GA
Atlanta, GA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Richmond, VA
Philadelphia, NJ
Philadelphia, NJ
Philadelphia, NJ
Philadelphia, NJ
Model
HAM IV
HAM IV
HAM IV
UAM IV
UAM-IV
UAM IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
UAM-IV
UAM IV
UAM-IV
1 piscxlc Date
June 24, l<>88
August 1, 1(>88
August 16, 1988
August 17, 1988
August 18, 1088
August 2, 1988
August 3, 1988
July 31, 1987
August 1, 1987
July 8, 1988
Aug. 10, 1992
July 10, 1988
July 15, 1988
July 6, 1988
July 7, 1988
June 13, 1988
June 14, 1988
July 19, |9Q|
July 20, 1991
July 7, 1988
July 8. 19R8
( rtiservcd
Domnin wide
IVnk
(PI*)
117
155
144
144
159
140
l()9
201
109
186
132
142
141
155
173
144
153
150
ISO
210
210
1'ieduted
Domain-wide
Peak
(PI*)
in
190
158
150
150
180
169
194
191
197
148
127
142
130
146
131
144
160
304
201
274
Accuracy of
Domnin wide
Peak Unpaired
(%)
-17.5
227
9.9
4.0
-5.6
28.3
0.0
-3.4
12.9
6.1
11.9
-10.6
0.7
-16.1
-15.6
-9.0
-5.9
6.7
68.8
-4 3
30 4
Mean
Normalized
Bias*
(%)
-25.2'
-4.K
-3.5
-14.6
-10.0
-3.3
3.5
5.2C
-0.9C
-85C
4.3C
I8.2C
-8.7C
-I5.0r
-15.9C
-12. 3e
-7.4r
24.3
15.3
6.9
20.1
Mean Gross
Hrror'
(%)
26.1
22.5
19.2
21.7
21.6
20.4
19.4
28
13.2
18.4
22.1
20.7
13.3
20.6
18.8
18.2
19.8
35.0
29.6
25.6
30.7
-J

-------
                 Table 3-1.   The photochemical models' o/one performance stalistics for individual baseline simulations.
                                                                                                                   Page 6 of 7
Region
Philadelphia, NJ
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
New York, NY
Baltimore, MD
Baltimore, MD
Baltimore, MD
Model
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
UAM-IV
HAM IV
F'.pisocle Dale
June 15, l(>87
Jul} 10, 1988
July II, 1988
July 17, I00|
July 18, 1901
July 10, 1001
July 20, 1901
July 7, 1988
July 8, 1088
July 9, 1988
June 15, 1987
June 18, 1087
June 19, 1987
June 19, 1988
June 20, 1987
June 20, 1988
June 21, 1988
June 22, 1088
June 20, 1 988
June 30, |088
June 19, |
-------
                   Table  31.  The photochemical models'  o/one performance statistics for indn ;88
Inly 8, l('88
Julv II, 1988
12<>.0
129.0
129.0
129.0
( HiM'rved
Douiinn wide
P.-nk

-------
        400
                       Ozone  Predictions in SIP Applications
                      Domain-wide Maximum Concentrations
                   100
Dashed lines are +/-15 percent
     200                   300
Observed Concentration (ppb)
   400

    12/18/95
   22 Regions
129 Data Points
    Figure 3-1. The predicted and observed domain-wide maximum ozone concentrations for SIP applications.

-------
Table 3-2.  The extent to which RPA performance goals were acheived in (he SIP applications.
Page 1 of 2
Region
San Diego
l/».«i Angeles
Ventura
San Joaquin Valley
Sacramento
Phoenix
Houston
Beaumont
Dallas
Baton Rouge
St. Louis
Lake Michigan
Detroit
Ixniisville
Nashville
Cincinnati
Atlanta
Richmond
Philadelphia
Number of
Day-
Modeled
2
10
1
2
1
2
12
10
0
7
8
8
4
3
4
6
4
6
5
Number of Days lor Whu h Performance Cioals
Were At lne\ ed
Peak
Accuracy
Criteria
2
8
?
2
\
2
2
3
8
5
7
7
4
2
2
4
4
4
1
Mean
Hia*
Criteria
T
4
1
-»
1
2
4
4
8
5
7
8
2
2
3
6
4
T
1
Mean
1 iror
Criteria
2
8
1
2
1
2
6
0
9
7
8
8
4
3
4
6
4
6
4
All
Criteria
2
3
3
2
1
2
0
2
7
4
7
7
2
1
2
4
4
3
1
Percent of Modeled Days for Which Perfonnam e
(ioals Were Achieved
Peak
Accuracy
Criteria
100
80
100
100
100
100
17
30
89
71
88
88
100
67
50
67
100
67
60
Mean
Bias
Criteria
100
40
100
100
100
100
33
40
89
71
88
100
50
67
75
100
100
50
20
Mean
Rrror
Criteria
100
80
100
100
100
100
50
90
100
100
100
100
100
100
100
100
100
100
80
All
Criteria
100
30
100
100
100
100
0
20
78
57
88
88
50
33
50
67
100
50
20

-------
                    Table 3-2.  The extent to which FiPA performance goals were acheived in the SIP applications.
Page 2 of 2
Region
New York
Baltimore
New England
Sum
Mean
Minimum
Maximum
Number of
Days
Modeled
17
4
2




Number of Days for Which Pcrfonnance Cioals
Wen- Achieved
Peak
Accuracy
Criteria
9
3
2
87

1
9
Mean
Hias
Criteria
10
4
2
86

1
10
Mean
l-.rror
Criteria
15
4
2
117

1
15
All
Criteria
5
1
2
66

0
7
Percent of Modeled Days for Which Perfonnance
Goals Were Achieved
Peak
Accuracy
Criteria
53
75
100

77.3
16.7
100.0
Mean
Bias
Criteria
59
100
100

75.3
20.0
100.0
Mean
Firror
Criteria
88
100
100

94.9
50.0
100.0
All
Criteria
29
75
100

64.1
0.0
100.0
NO

-------
              o"
             v/
                                  Ozone Predictions in SIP Applications
                            Number of Days Meeting EPA Performance Goals
18
16
03
Q 12
.- 10
Q)
JO
1 8
6
4

2
r>











•a





























IV
1
- -











HfllL
" .



















I
                                                                                          Days Meeting Goals
                                                                                          Alt Days Modeled
                                              Region
    12/18/95
   22 Regions
129 Data Point?
Figure 3-2.   The number of simulations in each region with performance that met all three of EPA's ozone performance goals.

-------
                                    Ozone Predictions in SIP Applications

                             Percent of Days Meeting EPA Performance Goals
          100
      CD
      TJ
      O
      CO
      >»
      CO
      Q

      CD
      •o
      O
      w
      'Q.
      LLJ
      CD
      O
      v_
      0>
      Q.
                                                      Region
     12/18/95

    22 Regions

129 Data Points
Figure 3-3.   The percentage of simulations in each region with performance that met all three of HI'A's ozone performance

            goals.

-------
areas that modeled four or more episode days, over 75 percent of the episode days modeled in the
following regions met the EPA performance goal for mean normalized bias: Dallas. SL Louis,
Lake Michigan. Nashville, Cincinnati, Atlanta, and Baltimore.

       The mean normalized gross error averaged 24 ± 8 percent in the SIP simulations. This
level of error is somewhat lower than in most historiplUAM applications. In individual
simulations, the gross error was as low as 12 percent (May 24, 1990 in Baton Rouge) and as high
as 56 percent (June 21.  1988 in New York). In most regions, all of the simulations had gross
errors of less than 35 percent for ozone.  However, selected simulations for Los Angeles,
Houston. Beaumont. Richmond, and New York had higher gross error than the EPA goal.
Overall, 95 percent of the simulations met the EPA goal for gross error.

       Table 3-2 shows the number of simulations that met all three of the EPA's performance
goals.  At least one simulation in every region except Houston met all three goals.  Overall, 64
percent of the simulations met all three goals. However, several of the areas that  modeled a large
number of days had low percentages of simulations achieving all three goals.  For example, only 3
of 10 davs in Los .Angeles. 0 of 12 days in Houston, 2 of 10 in Beaumont and 5 of 17 days in
New York met all three goals  The simulations for Dallas. Lake Michigan, and St. Louis had the
most days with performance that met all three goals.
3.2    SIMULATION STATISTICS AVERAGED FOR EACH REGION

       Table 3-3 and Figures 3-4 through 3-6 showtbe roedel performance statistics averaged
lor each region  The) show the domain-wide peak ozdfne was underpredicted by more than 9
percent (1 o > in San Diego. Phoenix, and Beaumont and overpredicted by more than 9 percent in
Louisville. Cincinnati. Philadelphia, and New York on average.  The domain-wide peak ozone
was predicted within ±10 percent in the 15 other regions on average.  The correlation between the
region a\ erased predicted and observed peaks is  high, R: = 0.86. and considerably better than that
for the individual simulations

       The grand average mean bias averaged by region is  -5.6 percent, which is similar to the
grand average for all individual simulations. In Los Angeles. Houston, Beaumont, and Detroit,
the average predictions have negative biases of 15 percent (1 o) or more. Positive average biases
of 13  percent (+2o) or more are evident in the Philadelphia and New York simulations. The mean
bias in the other 16 regions  is between -13 and +3 percent, which is reasonably good.

       The grand average mean gross error averaged  by region is 22.7 percent which is slightly
better than the average for all individual simulations.  The results show that the mean gross error
is generally higher (greater  than 25 percent) in the region with high ambient ozone concentrations
(Los Angeles. Houston. Philadelphia. New York, and  New  England).  The simulations for San
Joaquin Valley. Lake Michigan. Sacramento. Dallas. Richmond, and Baltimore have mean gross
error of less than 20 percent on average, which is quite good.
                                         3-15

-------
Table 3-3.  Average photochemical models' ozone performance statistics by region
Region
San Diego. CA
Los Angeles. CA
Ventura. CA
San Joaquin Vallev
Sacramento. CA
Phoenix. AZ
Houston. TX
Beaumont. TX
Dallas TX
Bator. Rouee. LA
Si Louis. MO
Lake Michicar,
Detroit. Ml
N^hvilie. TN
Louisville KY
Cincinnati OH
Auir.'o. GA
Ri.-.'.r,, r,J VA
Philadelphia NI
Nev, ^ 01 L M
Baltimore MD
Nev. Eneland
Average
Standard Deviation
Minmiun,
Ma.\imum
Number
of Davs
2
10
3
1
1
1
12
10
Q
7
8
8
4
4
-i
^
6
4
6
<;
17
4
~i
59
3.9
1
17
Observed
Domain-
wide Peak
(ppb)
155
278
163
155
14?
149
179
161
150
137
147
1 50
156
137
145
152
172
151
188
183
162
203
1644
29.7
137
278
Predicted
Domain-
wide Peak
(ppb)
138
286
164
145
139
133
176
138
145
150
143
145
146
125
162
166
183
137
230
200
165
199
1642
367
125
286
Accuracy of
Domain-
wide Peak
Unpaired
(%)
-11
4
0
-7
.3
-10
2
-11
-3
9
->
.3
-7
-9
11
10
7
-4
23
14
3
o
-^
0.3
8.8
-11
23
Mean
Normalized
Bias
(%)
-9
-15
-9
-3
-12
-1
_2'1
-16
-5
.5
-1
3
-16
-13
-7
-5
0
.7
17
13
0
-11
-5.6
8.9
-22
17
Mean
Gross
Error (9P
27
31
-o
16
18
0~>
z*.
35
2~i
19
on
21
16
tL ^
20
21
21
20
19
3d
T7
19
25
01 1
4.9
16
35
                                    3-16

-------
                                           Ll-£
3!
  03

-------
                                        Ozone Predictions

                                     Average Bias By Region

                            All Cases with Observed Ozone Above 60 ppb
OJ
I
t—>
oo
            15
 Ctf

CO
-f-«

 CD
 O

 0>
CL
           (15)
                                               Region

                                                ^•VCj^^X
                                                                      x*
                                                                             12/18/95
                                                                            22 Regions
                                                                          129 Data Points
       Figure 3-5. The average normalized bias in predicted o/,one concentrations above 60 ppb in each legion.

-------
VO



I—
o
UJ
-*->
c
o
Q)
Q_



40
35
30

25
20
15
10
5
0
                    *?
                                               Ozone Predictions
                                        Average Gross Error By Region
                                 All Cases with Observed Ozone Above 60 ppb

                                                        Region
    12/18/95
   22 Regions
129 Data Polnta
    Figure 3-6.   The average normalised gross error in predicted ozone concentrations above 60 ppb in each region,

-------
normalized bias.  Overall. 75 percent of the individual simulations met this performance goal  For

3.3    SIMULATION STATISTICS ON THE HIGHEST OZONE DAY IN EACH
       REGION

       Table 3-4 and Figures 3-7 through 3-9 show the model performance statistics for the
highest ozone day in each region.  The results show the domain-wide peak ozone was
underpredicted by 6.6 percent on average on the highest ozone days in each region. On average.
the predicted and observed  domain-wide ozone peak concentrations were 183 and 196 ppb on the
highest days in each region. The mean bias and gross error averaged -4.8 and 23.7 percent on the
highest ozone days, respectively.  The correlation between the predicted and observed peaks on
the highest ozone days was moderately high, R2 = 0.70. Thus, overall, the performance on the
highest ozone days is not significantly different than the average of all days modeled. The only
difference is the tendency for the models to underpredict the domain-wide peak on the highest
days, which was especial!)  evident in New York and  Houston.

3.4    SIMULATION STATISTICS STRATIFIED BY MODELING METHODOLOGY
       AND GEOGRAPHIC SETTING

       Within the modeling community, there is interest in whether the modeling methodologies
and geographic setting significantly influence the model performance. There are many minor
differences in the modeling methodologies in these simulations and in the geographic settings of
the areas  Often these differences are difficult to quantify  and classih consistently.  For this
analysis, we have elected to stratify only on a few  variables

       The results of these  simulations were stratified on  (1) the type of wind model, prognostic
or diagnostic, used to de\elop the  hourly gridded 3-dimensional windfields and (2) the methods  of
determining the boundary conditions, either  derived from  a regional model (ROM) or from
observations.  In  addition, the results were stratified on  (1) whether the geophysical features \\er;
pnmanh flat land or involved a land-water interface,  which generally increases the complexity of
the meteorolog\. and (2) whether  regional pollutant transport typicalh has a major or minor efiect
on ozone in the region. Table 3-5 lists the boundary  condition methodology, the wind model, the
type of geographic setting,  and the predominance of pollutant transport into the  region. The data
indicate that prognostic wind models were used more frequently in regions with  more complex
meteorology,  so these stratification variables are not necessarily independent of  one another.  It  is
also important to recognize that the later classifications, especially the importance of pollutant
transport into the region, are subjective and may not be accurate for all of the episodes in a
region.

       As shown in Table  3-6 and Figure 3-10, the stratification on the boundary condition
methodologies indicate a tendency towards underprediction when boundary conditions were
derived from observations rather than from a regional model. The mean bias averaged -9 percent
in the 78 simulations where boundary conditions were derived from observations compared to
+ 1.5 percent mean bias on average in the 51  simulations with ROM-derived boundary conditions.
The gross error was not significantly affected by the boundary condition methodology. The

                                          3-20

-------
Table 3-4.  Photochemical models' ozone performance statistic on high ozone day
          by region.
Reeion
San Diego. CA
Los- Anceles. CA
Los Aneeles. CA
Ventura. CA
San Joaquin VaJle\
Sacramento. C A
Phoenix. AZ
Houston. TX
Beaumont. TX
D_ .- TX
Bator, Rouee LA
S: Louis. MO
Lake Michiear,
DC-L-;:. Ml
NashuliL T\
N^hMlle. TN
LouisMiie KY
C,n;:r ,-,.-,'.. OH
A^nta. G A
R.j'r.rnond \ A
Philadelphia. M
Philadelphia. NJ
New York. NY
Baltimore. MD
Nev. Encland
Mean (weighted b\ region 1
Standard Deviauon
Minimum
Maximum
O'Served
Domain-wide
Peak (ppK>
156
360
360
180
160
143
157
240
226
:"0
15"
1^3
1 7?
If*
138
138
151
169
201
1 ?•*
210
210
244
180
22°
1962
59.8
1380
360 0
Predicted
Domain-wide
Peak tppb)
135
392
382
173
150
139
134
112
151
155
160
160
150
164
139
115
187
169
194
146
201
274
206
169
211
182.7
68.9
1120
392 4
Accuracy of
Domain-wide
Peak
Unpaired (%^
-13
9
6
-4
-6
.3
-15
-53
-33
-9
2
_o
-16
f\
]
-17
24
0
.7
-]h
-4
30
-30
-6
-5
-66
16.5
-53.3
30.4
Mean
Normalized
Bias
(%)
-6
7
7
_*>
-7
-12
.5
-45
-^2
-f-
1
.7
5
-12
-9
_"7
2
4
5
-16
7
20
-3
3
-12
-4.8
12.8
-450
201
Mean Gross
Error
(<£)
31
33
32
21
15
18
24
45
31
^ ^
21
2C>
14
^ *,
20
15
16
19
28
19
26
31
29
15
24
237
7.3
140
452
                                   3-21

-------
                                                Ozone Predictions
                                 Accuracy of Peak Ozone on Highest Ozone Day
                                           Unpaired in Space and Time
K)
          "E  30
          o>
          o
          
          Q.
8    0
i_

O  (15)
                                                          n
                                                                           ~
             (45)
                                                         Region
                                                                                            12/18/95
                                                                                           22 Regions
                                                                                        129 Dnta Points
     Figure 3-7.   The accuracy of the domain-wide peak ozone unpaired in space and time on the highest o'one day in each region.

-------
KJ
                                                Ozone Predictions
                                      Bias on Highest Ozone Day By Region
                                 All Cases with Observed Ozone Above 60 ppb
              45 r	
                                                        Region
    12/18/95
   22 Regions
129 Data Points
    Figure 3-8.   The mean normalized bias in predicted ozone concentrations above 60 ppb on the highest ozone day in each
                region.

-------
                                                Ozone Predictions
                                  Gross Error on Highest Ozone Day By Region
                                  All Cases with Observed Ozone Above 60 ppb
CO
1
K)
-U
                                                         Region
    12/18/95
   22 Regions
129 Data Points
    Figure 3-9.  The mean normalized gross error in predicted ozone concentrations above 60 ppb on the highest ozone day in each
               region.

-------
Table 3-5.  The methodologies used to develop boundary conditions and windfields.
           and the characteristics of the geographic setting.
Region
San Diego. CA
Los Angeles. CA
\entura. CA
San Joaquin Yal!e\
Sacramento. CA
Phoenix. A_Z
Ho'jsvr TX
Beaumont. TX
Dallas TX
Ba'.or Rouee- LA
S: L,-u:sMO
L J. K C > t . f"i 1 i! oJ
Deir.r, MI
i Lo'j;s\,:ie K^
N\:sr:Mij. TN
C::,. :•.:•. ()H
A'..,;i:.. GA
Ri.r.r-. v,.: \ A
Philadeipnia NJ
New ^'ork N^'
Baltimore MD
N'e« EiteJ.irn!
Boundan.1 Condmon
Method
Surface &. Aircraft
Observations
Surface & Aircraft
Observations
Surface Observations
Surface & Aircraft
Observations
Surface & Aircraft
Observations
Surface Observations
Surface Observations
Sun ace Observations
Surface Observations
Surf Ac Aircrar.
Surface Ohse'\auor.j
Surface 
-------
                                            C I
conditions and under-predicted by 1  percent with observation-based boundary conditions  Furth
stratification of the results based on whether only surface observations or surface and aircraft
observations were used to determine boundary conditions showed only minor differences in the
performance statistics.  However, within the group of simulations with boundary conditions
derived from observations, the model performance was slightly better when aircraft and surface
observations were used instead of surface data alone.

       Stratification of the model performance on the type of wind model employed showed
larger differences, as shown in Table 3-6 and Figure 3-11.  About one third of the simulations
were made using winds derived from prognostic wind models. Simulations made with winds
denved from prognostic models had more negative bias and larger error than those made with
winds denved from diagnostic models. On  average, the mean bias was -12 and -0.7 percent in
simulations made with winds denved from prognostic and diagnostic models, respectively. The
mean gross error was 26 and 23 percent in simulations made with winds derived from prognostic
and diagnostic models, respectively  The differences in the average domain-wide peak accuracy
were consistent with the other results but smaller (-2 percent with prognostic winds and +4.?
percent with diagnostic winds). It is also worth noting that ozone model performance with the
UAM-DWM wind model was comparable to those for all diagnostic wind models. The under-
and overpredictions of ozone on average may be due to the tendencies for prognostic and
diagnostic models to overestimate and underestimate wind  speeds, respectively. The poorer
performance of  simulations made with prognostic winds may also reflect the greater complexity of
meteorolog) in  the regions where the modelers elected to use this approach. Thus, the results
should be interpreted cautiously because this stratification does not compare wind models in the
same regions and. therefore, other confounding factors, such as the complexity of meteorolog) or
emission inventory  problems, may be responsible for the differences.

       Stratification of the results on the basis of whether the geophysical characteristics include
onh land or land and water also shows differences, as shown in Table 3-6 and Figure 3-12  The
average gross error in the  predictions are 20 and 26 percent on average in regions with onh land
and with land and water, respectively. The  other statistics are comparable. Not surprising]),
these results indicate that there is more model error (not bias) in regions with complex
meteorolog).

       Lastly, stratification on the extent of pollutant transport into the region indicates there is
negative bias in the regions where regional pollutant transport is a minor factor (see Table 3-6 and
Figure 3-131 On average, the mean bias is -10 percent in these simulations compared to +3
percent in  the simulations where regional pollutant transport is typically  a more important factor.
The domain-wide peak  ozone predictions are more accurate on average in regions where pollutant
transport is a minor factor  However, these differences are not large and should be interpreted
cautiously because the stratification  is subjective
3-26

-------
      Table 3-6.  Photochemical models' performance statistics for o/one stratified by boundary condition methodology, windfleld
                  model, geographic setting, and predominance of pollutant transport into the region.
Stratification Parameter
Boundary conditions
derived from (lie Regional
Oxidanl M(xlel
Boundary conditions
derived from observations
Boundary conditions
derived from surface and
aircraft observations
Boundary conditions
derived from surface
observations
Windfleld derived from
Prognostic Wind Model
Windfield derived from
Diagnostic Wind Models
Windfield derived from
the UAM-DWM
Land-only geographic
setting
Ijind-water gesographic
setting
Significant pollutant
transport into region
Minor pollutant transport
into region
Average
Number of
Tpisode
Days
51
78
25
54
48
81
60
37
92
54
73
129
Mean Domain-wide
Peak Obsi'rv ation
JIT!')
167 1
172 5
200.8
159.2
185.1
161.8
1574
149.9
178.7
1664
173.4
170.5
Menu Domain wide
Peak Prtxlution (ppb)
176 1
I6<>. 5
201 0
154 1
179.5
1 67 6
161.3
150.4
180.8
175.2
169.8
172.1
Mean Arc uracy of
Domain-wide Peak
I Inpaired (%)
6 4
-1 0
-0 3
1 4
-2.0
4.3
2.5
0.5
2.5
64
-1 2
1.9
Mean
Normalized
Bias (%)
1.5
-9.0
-5.9
-10.4
-11.7
-0.7
-3.9
-4.7
^t.8
2.6
-10.2
^.8
Mean (Jross
F-rror (%)
23 7
24.5
23.1
25.0
26.3
22.9
21.9
20.2
25.8
23.3
24 8
24.2
K)

-------
                                                   Ozone Predictions
                                     Stratified by Boundary Condition Methodology
               400
K)
00
                          100
     Dashed lines are +/- 15 percent
       200                   300
Observed Concentration (ppb)
                                                                                                    Modeled (ROM)
                                                                                                  Boundary Conditions
                                                                                                         n
                                                                                                Measured and Estimated
                                                                                                  Boundary Conditions
400
                                                                    12/18/95
                                                                   22 Regions
                                                               129 Dnta Points
     Figure 3-10.  The predicted and observed domain-wide maximum o/.one concentrations stratified by boundary condition
                  methodology.

-------
                                                  Ozone Predictions
                                        Stratified by Wind Modeling Approach
              400 r
UJ
t
t-J
vo
          QL
                           100
     Dashed lines are +/- 15 percent
       200                     300
Observed Concentration (ppb)
                                                                                                        Prognostic
                                                                                                        Wind Model
                                                                                                            o
                                                                                                         Diagnostic
                                                                                                        Wind Model
400

      12/18/95
     22 Regions
 129 Data Points
     Figure 3-II.   The predicted and observed domain-wide maximum ozone concentrations stratified by wind model type.

-------
                                                   Ozone Predictions
                                       Stratified by Geophysical Characteristics
              400
U)
I
OJ
o
                                                                                                    Land Only
                                                                                              Loss Complex Motoorology
                                                                                                        •

                                                                                                  Land and Water
                                                                                              More Complex Meteorology
                                                                                                        A
                          100
     Dashed lines are +/- 15 percent
        200                  300
Observed Concentration (ppb)
400
                                                                     12/18/95
                                                                    22 Regions
                                                                129 Data Points
     Figure 3-12.  The predicted and observed domain-wide maximum ozone concentrations stratified by geophysical characteristics.

-------
 3.5    SIMULATION STATISTICS FOR THE UAM-IV, UAM-V, AND SAQM
       MODELS

       As indicated above, most of the simulations were made with the UAM-IV model and there
 V.CIN onh one region. Atlanta, where a direct comparison of the UAM-IV and UAM-V models
 was made using the same input database (SAI. 1994c).  Table 3-7 shows the average ozone
 performance statistics for four days of simulations in Atlanta with the two models. The results
 show the mean normalized bias and gross error are virtually identical in these simulations (0.0 vs.
 -0.6 percent bias and 20.4 vs. 20.9 percent gross error). However, the average accuracy of the
 domain-wide peak ozone from UAM-IV and UAM-V models is different, but still reasonably
 accurate: +6.9 percent with the UAM-FV model and -3.9 percent with the UAM-V model.
 Overall, these results indicate the performance of both models is excellent and comparable in the
 Atlanta simulations  These results should be interpreted cautiously because the characteristics of
 the Atlanta area and ozone episodes probably do not represent a stressful test of the two models
 Thai is. technical improvements incorporated into UAM-V may affect the performance mov
 significant!) in areas with more complex meteorology or emission patterns, such as Los Angeles
 or Lake Michican  The cood agreement found in the Atlanta simulations mav not hold for other
     TaK; j ?-"  Comparison of the average model performance statistics for ozone from the
               SA< >M. UAM-IV. and UAM-V models
' :



MtxM'Reeion
I \.\1-I\ in AUanui
I AM-\ in AUanui
S-\(,)V 11; San Jouejuin \ alle>
l'AM-\ us LaU Michiean
l'\M-I\ in 20 Areas


Number
of Dav>.
4
j
2
8
119

Mean Accuracy of
Domain-wide
Peak Unpaired
<<7r!
69
-3 9
-6 8
-? 1
2.4



Mean Normalized
BmsC7r)
0
-06
-30
? 4
-54


Mean Gros^
Error
(<7(\
204
20 9
15 5
16 1
249
       Table 3-7 also shows a comparison of the average performance statistics for the SAQM
model in [he San Joaquin Valley simulations and the UAM-IV model in the 20 other regions. The
statistics sho\\ the SAQM simulations have significantly less gross error than the UAM-IV
simulations, but the domain-wide peak ozone is also less accurate than the average from all of the
UAM-IV simulations. The low gross error and near absence of bias in the SAQM simulations is
impress!\e performance and reflects not only the strengths of the SAQM/MM5 models but also
the extensne aerometnc database available for the episodes and the extensive effort to refine
model pcriormance.  A process-oriented approach including multi-species comparisons was used
                                         3-31

-------
                                               Ozone Predictions
                                   Stratified by Extent of Pollutant Transport
          400
                                                                                                     More Pollutant
                                                                                                      Transport
                                                                                                          •

                                                                                                     Less Pollutant
                                                                                                      Transport
                                                                                                          A
                       100
Dashed lines are */- 15 percent
       200                      300
Observed Concentration (ppb)
400
                                                                 12/18/95
                                                                22 Regions
                                                            129 Data Points
Figure 3-13.   The predicted and observed domain-wide maximum o/one concentrations stratified by the predominance of
              pollutant transport into the region.

-------
over a two-year period to improve model performance for the SAQM simulations and the
performance statistics reflect the large effort.

       Average performance statistics for the UAM-V model in the Lake Michigan simulations
are also shown in Table 3-7. The accuracy of the domain-wide peak predictions and the mean
normalized bias are comparable to those for the UAM-IV in the 20 regions; however, the gross
error in the UAM-IV Lake Michigan simulations is significantly lower than the average for the
UAM-IV simulations in other areas (16.1 vs 24.9 percent).  Like in the San Joaquin Valley, an
extensive special  study aerometric database was available for these episodes and a process-
onented approach including multi-species comparisons was used over a two-year period to
improve model performance for the Lake Michigan simulations. The UAM-V ozone performance
is clearly better than the average UAM-IV performance in other areas on average, and the
difference is probably due to both features of the UAM-V model and the better database and more
extensive model evaluation procedures employed in the Lake Michigan modeling.
                                         3-33

-------
                    4.  CONCLUSIONS AND RECOMMENDATIONS
 4.1     CONCLUSIONS

        The investigation of photochemical models ozone performance in the SIP applications is
 based on three basic performance statistics required by EPA for regulatory applications. The
 compilation results for 131 simulation.; in 22 regions showed the average accuracy of the domain-
 wide peak ozone unpaired in space and time was +2 ±20 percent.  The mean normalized bias in
 concentrations above 60 ppb in all simulations was - 5 ±16 percent  The mean normalized gross
 error in concentrations above 60 ppb in all simulations was 24 ±8 percent.

        The EPA estabhshed model performance goals for regulator}' applications of the UAM.
 The goals were for the unpaired peak accuracy and mean normalized bias to be within ±15
 percent, and for the mean gross error to be less than 35 percent.  Almost all (94 percent) of the
 simulations met the gross error goal and over half (64 percent) of the simulations met all three
 goals. Furthermore, at least one simulation in each region except Houston  met all three goals.
 Thus, while there were areas with better and poorer model performance, almost all areas had at
 leait one episode day with acceptable performance (based on the EPA goals) which could be used
 tor control strategy evaluations.

        Stratification of the results on selected methodological procedures did not show  large
 differences  Simulations made with boundary conditions derived from observations predicted
 lower ozone on average than those with boundary conditions derived from  the Regional Oxidant
 Model. Simulations made with prognostic wind models had larger error and more negative bias
 on average compared to simulations made with windfields derived from diagnostic wind models.
 howe\er. the areas where prognostic models were employed are also regions with more complex
 meteorolog\ and where larger model error would be expected. The  negative bias in ozone with
 the prognostic models ma\ be due to their tendency to overestimate  wind speeds, but it also could
 be due to cither factors.

       Stratification of the ozone performance statistics on geographic characteristics also
 showed modest differences. For example, there was greater model error on average in regions
 with land-water interfaces (26 percent versus 20 percent). This result was expected because
 regions with land-water interfaces have more complex meteorology.  However, regions where
 pollutant transport into the region was expected to be significant had lower model error and less
 bias  than those without significant pollutant transport.  This result may suggest inflow boundary
 conditions were adequately estimated in the regions with significant transport in these simulations.
 but there could also be other compensating factors which explain the differences. Thus,  all of the
 stratification results should be interpreted cautiously.

       A comparison of results from applications of the  UAM-IV and UAM-V models to the
 Atlanta regions showed excellent and comparable performance. The average results for the four
simulated days differed  only in the accuracy of the domain-wide peak predictions, which were
slightly better with the UAM-V  model.  Overall, the results were too similar to distinguish

                                          4-1

-------
between the models. The characteristics of the Atlanta region and episodes may not be sufficient
to stressfully test the differences between the UAM-FV and UAM-V models, so the absence of
differences in these test should be interpreted cautiously.

       A comparison of the SAQM model performance in the San Joaquin Valley and the UAM-
V model performance in Lake Michigan to the average UAM-TV model performance in 20 other
regions showed the SAQM and UAM-V models had significantly less gross error in predicung
ozone concentrations above 60 ppb.  The accuracy of the domain-wide peak concentrations and
the mean normalized bias were similar for the SAQM, UAM-V, and UAM-IV on average.  The
better overall performance of the SAQM and UAM-V models in these simulations is probably due
to both features of the models and the better databases and more extensive model evaluation
procedures employed in the modeling in these two areas. Process-oriented approaches including
mulu-species comparisons were used to improve model performance for the simulations and the
performance statistics reflect the more comprehensive efforts.
4.2    RECOMMENDATIONS

       The analysis of photochemical grid model performance using three statistics for ozone
provides a good starting point for model evaluation, but it does not represent a comprehensive
approach to model evaluation A more thorough approach would include the following elements

   •    Use of additional model performance statistics, such as those listed in Table 4-1

   •    Evaluation of the model performance for precursor species, such as NO. NO2. NO . CO.
       lumped VOC species (OLE. PAR. TOL. XYL. ETH. FORM, and ALD). in addition to
       those for ozone

   •    Use of a process-oriented approach that reviews all of the statistical results and graphical
       displa\s for baseline runs, alternate baseline runs, and other sensitivity runs, and seeks to
       idenuf\ potential problems or inconsistencies, including compensating errors.

In order to carry out more comprehensive analyses, the evaluation team must have access to all of
the model input, model output, and observation files used in the simulations and all of the
modeling documentation, and have appropriate display software to visualize the predictions and
observations
                                         4-:

-------
Table 4-1   Additional statistical parameters for model performance evaluation.
L Ahbreviauon
PAPST
NPAPST
PAPS
NPAPS
PAPT
NPAPT

PA
PNBIAS
PSTRK,*
™,,<
PERROK
BUS
ERROR
Parameter
Absolute accuracv of domain-wide maximum 1-hr concentrauons paired in space and ume'
Normalized accuracv of domain-wide maximum 1-hr concentrauons paired in space and ume"
Absolute accuracy of domain-wide maximum 1-hr concentrauons paired in space and unpaired
in t me*
Normalized accuracv of domain-wide maximum 1-hr concentrauons paired in space and
unpaired ume'
Absolute accuracy of domain-wide maximum 1-hr concentrauons paired in ume and unpaired in
space'
Normalized accuracy of domain-wide maximum concentrauons paired in ume and unpaired in
space*
.
Absolute accuracv of domain-wide maximum 1-hr concentrauons unpaired in space and ume
Mean normalized bias of predicted and observed maximum 1-hr concentrauons at all monitoring
stations'
Mcar normalized error of predated and observed maximum 1-hr concentrauons at all
mor.itor.nc stations*
Mean absolute bias o! predicted and observed maximum 1-hr concentrauons at all monitoring
Mean absolute error of predicted and observed maximum 1-hr concentrations at all monitoring
station1.*
Nka:, absolute bus of all predicted and observed concentration pairs where the observed
exceeds a minimum concentration
Mear. absolute error of a!! predicted and obsened concentration pairs where the observed
exceeds a minimum concentration
                co p^ cxm^nuaiiorL^ excted the minimum concentraiiorLc
                                      4-3

-------
                                  5. REFERENCES
 CARS (1995a) Revisions to the base case and future year Urban Airshed Model simulations
       for Ventura County in support of the 1994 State Implementation Plan.  Report prepared
       by California Air Resources Board, Sacramento. CA.

 CARB (1995b) Photochemical modeling of the Greater Sacramento Area in support of the
       1994 State Implementation Plan.  Report prepared by California Air Resources Board.
       Sacramento, CA.

 C.ARB (1995c) San Joaquin Valley SIP Modeling: Model Performance. Report prepared by
       California Air Resources Board, Sacramento, CA.

 EPA (1991) Guidelines for regulatory application of the Urban Airshed Model.  Report
       prepared by U.S. Environmental Protection Agency, Research Triangle Park. NC,
       EPA450/4-91-013.

 Georgopoulos P G (1995) Ozone  sip modeling technical support documentation summary for
       the New Jersey - Philadelphia CMSA area. Report prepared by Ozone Research Center.
       Environmental and Occupational Health Sciences Institute, Piscataway, NJ, Technical
       ORC-TR9502.

 Kammski M.A. (1995) Urban airshed modeling of the middle Tennessee modeling domain:
       model performance evaluation.  Report prepared by University of Tennessee, Knoxville.
       TN

 LADCO (1994) Lake Michigan Ozone Study: evaluation of the UASM-V photochemical grid
       model in  the Lake Michigan region, Version 2.0. Report prepared by Lake Michigan Air
       Directors Consortium. Des Plaines. IL.

 NYDEC (1994a> New York urban airshed modeling For June 14 to 20. 1987 base case.
       New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE3.

 NYDEC (1994b) New York urban airshed modeling For July 5 to 11, 1988 base case.
       New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE1.

 NYDEC (1994c)  New York urban airshed modeling For June 18 to 22, 1988 base case.
       New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE2.

 NYDEC (1994d) New York urban airshed modeling For July 16 to 20, 1991 base case.
       New York State Division of Environmental Conservation, Albany, NY, NYAS-94-SE4.

Roth P.M.. Reynolds S.. Tesche T.. and Dennis R. (1991) A conceptual framework for
      evaluating the performance of grid-based photochemical air quality simulation models.
      Report prepared b\ Envair. San Anselmo. CA.
                                        5-1

-------
SAJ (1990) User's guide to the Urban Airshed Model (UAM-IV). Report prepared by
      Systems Applications Internationa], San Rafael. CA.

SAJ (1993) User's guide: Lake Michigan Ozone Study photochemical modeling system.
      Report prepared by Systems Applications International, San Rafael, CA.

SAJ (1994a) Photochemical modeling of the Lake Michigan region for the 1991 Lake
      Michigan Ozone Study (LMOS) using the Nested-grid Urban Airshed Model (UAM-V).
      Repon prepared by Systems Applications International, San Rafael, CA.

SAJ (1994b) Application of the Urban Airshed Model to Baton Rouge, Louisiana for three
      multi-day ozone episodes.  Volume IE: diagnostic/sensitivity analysis and model
      performance evaluation. Repon prepared by Systems Applications International. San
      Rafael. CA. SYSAPP94/090.

SAJ (1994c) Comparison of the UAM-W and UAM-V photochemical models for three
      Atlanta-area ozone episodes. Report prepared by Systems Applications International. San
      Rafael. CA.SYSAPP94/106.

SCAQMD (1994) 1994 Air Quality Management Plan, Technical Report V-B:  Ozone
      Modeling Performance Evaluation. South Coast Air Quality Management District,
      Diamond Bar. CA

SDAPCD (1994) Request for ozone reclassification - urban airshed modeling technical
      support document. Repon prepared by San Diego Air Pollutant Control District, San
      Diego. CA

Tesche T.W.. Lurmann F.W.. Roth P.M.. Georgopoulos P., Seinfeld J.H., and Cass G.
      (19901 Improvements of procedures for evaluating photochemical models.  Report
      prepared for California Air Resources Board. Radian Corp.. Sacramento. CA, ARJ3
      Contract No A832-103.

TNRCC (1994a'i El  Paso. Texas ozone nonattainment areas base case report performance
      e\ aluation  Report prepared by Texas Natural Resource Conservation Commission.
      Austin. TX.

TNRCC (1994b) Houston/Galveston Beaumont/Port Arthur ozone nonattainment areas base
      case repon performance evaluation. Report prepared by Texas Natural Resource
      Conservation Commission. Austin, TX.

TNRCC (1994c) Dallas/Ft Worth ozone nonattainment areas base case report  performance
      evaluation. Report prepared by Texas Natural Resource Conservation Commission,
      Austin. TX.
                                         5-2

-------
                           TECHNICAL REPORT DATA
           'Please  read Instructions  on  reverse before  corr.ple;ing'
  FEPC-FT N:
  EPA-454 'R-96-004
                                                  3  RECIPIENT'S ACCESSION N;
  TITLE ANT. SVETITLE
  Compilation of Photochemical Models'
  Performance Statistics for  11/94  Ozone SIP
  Applications
                                                  5  REP-OPT DATE
                                                    July  1996
                                                6  PERFORMING ORGANIZATION CODE
                                                  S  PERFORMING ORGANIZATION REPOFT NC
9  PERFORMING ORGAN!ZATICN NAME ANT ADDRESS
                                                  10  PROGRAM ELEMENT NC
Sonoma Technology Inc.
5510 Skylane  Boulevard,  Suite 101
Santa R c s a, CA  9 5 4~2
                                                  11  CONTRACT - GRANT NC
                                                    EPA Contract No
   SF::;;:FIN- AGEN:- NAME ANT ArrpEr;
  U.S. Environmental Protection Agency
  Office :f  Air  Quality Planning  and
    Standards
  Err.iss.ions, Monitoring & Analysis  Division
  Research Triangle  Park,  NC   2~'7ll	
                                                  13  TYPE OF REPOPT ANT PERIOE COVEPET
                                                  Final Report
                                                14 SPONSORING AGENT> CODE
  EP;
       'k Assignment  Manaaer: Shao-Hancr Chi
  three
  devel
  The e
  s tat i
  ccnce
  perf o
  stati
  model
  betwe
     sica
     s '  per
     en the
 is a compilation of the model performance statistics of
chemical models 'UAK-IY, UA1-1-V,  and SAQM) used  in  the II''94
 Implementation Plans  (SIP) applications.  The  models were
24 ozone ncnattainment regions in  1993-1995 to  support the
 of emissions  control strategies for the 1994 ozone  SIPs.
icn focuses  or. three EPA recommended basic model performance
 measures  of  the models' ability to predict ambient  ozone
ens.  It does  not include an evaluation of the  models'
 on other  species (such as NO, N02,  Noy, and VOCs)  or other
 measures  on  ozone.   For a more  complete evaluation  of the
formance both  spatial and temporal analyses of  the matching
 predicted and observed concentrations are desirable.
                             KE'i A"f" ANT POCUMEKT ANALYSIS
                                     b  IDENTIFIERS'OPEN ENDED TERMS
                                                                c COSATI
                                                                Field ^ro
 Ozone SIF Applications Modeling
 Photochemical  Model  Performance
  CISTFIEVTICN STATEMENT
 Release Unlimited
                                     19 SECURITY CLASS IKfport/
                                         Unclassified
                                     2C SECURITY CLASS I Page!
                                         Unclassified
                                                              21  NC OF PAGES
                                                              156
EPA Fora. 222C-1 (R«v  4-77)
                     PREVIOUS EDITION IS OBSOLETE

-------