A Program for the Statistical Evaluation
of Point Source Dispersion Models
Using ASTM D 6589
-------
EPA-454/R-03-006
A Program for the Statistical Evaluation
of Point Source Dispersion Models
Using ASTM D 6589
U.S. ENVIRONMENTAL PROTECTION AGENCY
Office of Air Quality Planning and Standards
Emissions, Monitoring, and Analysis Division
Research Triangle Park, North Carolina 27711
August 2003
EPA - RIP LIBRARY
-------
DISCLAIMER
The information in this document is being reviewed in its entirety by the U.S. Environmental
Protection Agency (EPA), for approval for publication as an EPA document. Mention of trade
names, products, or services does not convey, and should not be interpreted as conveying official
EPA approval, endorsement, or recommendation.
11
-------
PREFACE
The American Society for Testing and Materials (ASTM) has published a Standard
Guide D 6589 for Statistical Evaluation of Atmospheric Dispersion Model Performance. Within
the Annex to this ASTM Guide, a procedure is outlined for assessing the performance of
atmospheric transport and diffusion models to predict the average maximum "centerline"
concentration values of a chemical species that has been released from a point source.
The EPA has developed a FORTRAN program called "Design 2" that implements the
procedure documented within the Annex to ASTM Standard Guide D 6589, for the statistical
evaluation of the performance of transport and diffusion models to simulate the average
"centerline" concentration values.
This report describes the input files and formats of these input files for running Design 2.
The report describes the information provided by the various output files created by Design 2.
The report does not describe the actual statistical procedures of Standard Guide D 6589, as these
are documented in detail within this Guide's Annex.
in
-------
ACKNOWLEDGMENTS
This research described in this report was funded wholly or in part by the United States
Environmental Protection Agency (EPA) through EPA Contract NO. 68-D-98-006, Work
Assignment NO. 4-02 with EC/R Inc., who subcontracted with the Pacific Environmental
Services, Inc, a MACTEC Company (PES). The EPA Work Assignment Officer was John S.
Irwin. The EC/R Project Leader was Steve Fudge. The PES Project Leader was James Paumier.
IV
-------
TABLE OF CONTENTS
Page
Preface iii
Acknowledgments iv
1. INTRODUCTION 1
2. OBSERVED CONCENTRATION DATA 4
3. MODELED CONCENTRATION DATA 8
4. REGIMES 10
5. CONTROL FILE 13
6. RESULTS 17
7. REFERENCES 23
APPENDIX - DIRECTORY STRUCTURE OF DISTRIBUTION CD 24
-------
SECTION 1
INTRODUCTION
Air quality transport and diffusion models have been in use for several decades to
estimate the impacts of pollutants released from point sources. Some of the simpler models are
steady-state, Gaussian plume models that include little or no chemical transformation. During
the model development phase and in subsequent upgrades, model performance is constantly
evaluated.
In the past, the emphasis of the statistical evaluation comparisons has been on the "intended
use." For instance, one of the uses for modeling results is to estimate the highest concentration
values to be expected over a 5-year period, resulting from the operation of a proposed new power
plant (Cox and Tikvart, 1990). Other statistical measures have also been employed to compare the
concentration (or dose) values of "intended use," such as number of values within a factor of two,
linear least-square fits to scatter plots of observed and predicted values, and normalized mean-
squared errors of observed and predicted values (Fox, 1981). Implicit in such statistical comparisons
is an assumption that the predicted and observed distributions of concentration values are from the
same population, which may not be a well-founded assumption (Oreskes et al, 1994). Work is
underway to develop a new generation of evaluation metrics that takes into account the statistical
differences (in error distributions) between model predictions and observations. As a result, a shift
in philosophy is occurring as to how models of environmental processes can be acceptably
evaluated. Most models provide estimates of the first moment of conditions to be expected for each
ensemble (e.g., average time-space variation of the meteorological conditions, average time-space
variation of the surface-level concentration values). The key to the next-generation evaluation
metrics is that they will no longer assume that the modeled and observed values come from the same
statistical population of values. They will assume that they "share" certain fundamental properties,
but are inherently different.
To fill a part of this void, the U.S. Environmental Protection Agency (EPA) has
participated within the American Society for Testing and Materials (ASTM) to develop a
consensus on a philosophical basis that could be used in developing new statistical methods for
model evaluation. In doing this, a statistical procedure was drafted and documented within the
Annex to Standard Guide D 6589 for Statistical Evaluation of Atmospheric Dispersion Model
Performance (ASTM, 2000). This procedure implements the idea that the distributions of model
predictions and observations "share" certain fundamental properties, but are inherently different.
The procedure assesses the performance of transport and diffusion models to simulate the
average "centerline" concentration values from a point source release.
The EPA has developed a FORTRAN program called "Design 2" that implements the
-1-
-------
procedure described in the Annex to D 6589. This report describes the input files and formats of
these input files for running Design 2. The report describes the information provided by the
various output files created by Design 2. The report does not describe the actual statistical
procedures of Standard Guide D 6589, as these are documented in detail within this Guide's
Annex.
As noted in the ASTM standard, many practical problems must be resolved before the
software can be run (see Section XI .4), including how to group the data into regimes. If we
consider each observation period as an experiment and groups of monitors/receptors (nearly)
equidistant downwind as an arc, then we can define an experiment-arc as a single experiment
and single arc combination. An experiment-arc pair can then be included in a regime and the
software can connect observed data to modeled data. Some regime definition strategies are
mentioned in Section 4.
The purpose of this document is to provide guidance on the procedures to prepare the
data and run the evaluation software. Figure 1.1 shows a flow chart of the process. The first
input file required is the observed concentration data. The structure of this file is described in
Section 2. The second file is the modeled concentration data. Since this software determines a
best performing model, results from at least two models should appear in this file. The structure
of this file is described in Section 3. The third file defines ranges of conditions, or regimes,
based on one or more parameters that subdivide the results. In defining regimes, the behavior of
the dispersion should be similar within each subdivision. A more complete discussion of regimes
and the structure of this file is described in Section 4. For the software to function, a control file
is required that has the names of the input and output file, file formats, and processing options.
This file is short (less than 20 lines) and the structure of this file is described in Section 5. Once
the software is run, several output files are produced: 1) the primary file with the information on
the best performing model (the ASTM analysis), 2) a file of the bootstrap results, and 3) a file of
results by regime. An optional output file is the fitted observed data. The analysis and structure
of these files is discussed in Section 5. In Section 6, the results from the evaluation software are
presented.
The software was coded in FORTRAN. Although a FORTRAN 95 compiler was used to
create the executable, attempts were made to keep the code backward compatible with
FORTRAN 77 standards. In several places, the user is required to provide a format for the data
to be read by the program. These formats must follow FORTRAN conventions and MUST be
enclosed in parentheses; for example, (IX, 15,15, 3F12.0). The characters in the FORTRAN
statement can be upper or lower case; the spaces after the commas are not necessary and are
shown here only to provide better legibility.
The example used throughout this documentation is from the 1985 Indianapolis field
experiment (Murray and Bowne, 1988). The field experiment was performed in a flat, urban to
suburban environment. A tracer, SF6, was injected into the buoyant gases exiting from an 86-
meter point source. Data were collected for approximately 170 hours over a five-week period.
Monitors were located in arcs from about 250 meters to 12 kilometers downwind.
-2-
-------
Meteorological data were collected from a 94-meter and 10-meter tower in the urban
environment, a 10-meter tower in a suburban location, and 10-meter tower in a rural location.
National Weather Service meteorological data were from Indianapolis, IN (hourly weather
observations) and Dayton, OH (upper air data). For the dispersion modeling, receptors were
placed along a single line due east of the release point. To obtain centerline concentrations the
wind directions were all modified to be blowing from the west (270°).
INPUT
Observed Concentration
Data (2)
Modeled Concentration
Data (3)
OUTPUT
Fitted Observed Data
(optional)
Regime Definition
(4)
DESIGN 2
ASTM Analysis
(6)
Control File
(5)
Bootstrap Results
(6)
Regime Results
(6)
Figure 1.1 File I/O for the ASTM analysis software, DESIGN2. The numbers in
parentheses indicate the section in which the input/output is described.
-3-
-------
SECTION 2
OBSERVED CONCENTRATION DATA
The observed concentration data are required to be in a specific format for the analysis
software to correctly read and process the data. The general structure is shown in Table 2.1.
The first record in the file is Header 1 and contains a title that can be up to 40 characters long
and only appears once. Note that this title must be enclosed in single quotation marks. Headers
2 and 3 appear before each arc of observations. These arc header records are followed by the x-
and y-coordinates and observed concentration, for the number of receptors on the arc (the first
field on Header 3). The coordinates and concentration must be specified as x-coordinate, y-
coordinate and then concentration value. The FORTRAN format for this information must
remain fixed for all values in the file. This FORTRAN format is defined by the user as part of
the control file information (see Section 5). Currently, the number of observed concentration
records per experiment-arc is limited to 250 records. An example of the observed concentration
file is shown in Figure 2.1.
Header records 2 and 3 are read 'free' format, i.e., without a format statement. This
means that a comma or at least one space (or a combination of the two, as seen in Figure 2.1)
must separate each entry on each record. Since the date and time are the only character variables
in these two records, they must be enclosed in single quotation marks.
The header 2 records provide general information on the field experiment period and arc
combination. The experiment and arc (or traverse) pair identify an observation period and arc of
data. The pair must be unique. Usually, the experiment numbers are unique (possibly a simple
sequential numbering), but the arc numbers likely repeat from field experiment period to field
experiment period. The important thing to remember is that these two parameters are used to
connect the observed data with the modeled concentration data and the grouping into regimes, so
the pair must be unique. The date, start and stop time, and nominal distance to the arc are not
used in the program, but provide additional identifying information within the archive.
The header 3 records contain information on how to process this experiment-arc
combination. The first three values define the total number of receptors on the arc and which
receptors to use in the analysis. Receptors can be included or excluded from the analysis by
varying the starting and ending point for the analysis.
The location of the receptors must be entered in the same coordinate system used in
defining the receptor locations, namely Cartesian (x,y) or polar coordinates (r,0). The type of
coordinate system in use is defined by the value of IXYARC, which is the fourth parameter of
-4-
-------
Header 3. If the receptor locations are entered as polar coordinates, then the program converts
them to Cartesian coordinates internally. The program works with coordinates expressed in
meters, so if the units are other than meters, a conversion factor from the user's units to meters
must be applied. This factor is provided as the fifth parameter on header 3 records.
The sixth and seventh parameters identify the location of the source. If the x- and y-
coordinates are entered as polar coordinates, then the source is likely to be located at an origin,
i.e., (0,0). If the x- and y-coordinates are entered as Cartesian coordinates (e.g., UTM
coordinates), then the source is likely to be located at something other than (0,0). The software
translates the source and receptors to a (0,0)-based system internally to perform its analysis.
The tracer release rate is specified next and is used to normalize the fitted observed
concentrations. The tracer height and the height of the receptors are not used in this version of
the software, but may provide useful information possibly interpreting the results, and serve as a
check on the source and receptor heights specified in the dispersion modeling.
The last parameter on this record converts the tracer concentration from user's units to
micrograms per cubic meter (ng/m3). Since this value is applied to every tracer concentration
without regard to meteorology, careful consideration should be used as to the appropriate value
to enter. For example, converting from parts per trillion to ng/m3 requires atmospheric pressure
and ambient temperature.
The header records are followed by the location and observed concentration for the
number of receptors on the arc (defined by the first value on header type 3). At a minimum there
are three values on these records. However, since the format is specified in the control file (see
Section 5), any amount of information can be included and skipped with the format statement the
user provides. For example, if the coordinate location is in polar coordinates, Cartesian
coordinates can be included on the record but skipped with the appropriate format statement (as
is done in Figure 2.1). During the fitting process, the concentrations are divided by the emission
release rate (eighth parameter in Header 3).
-5-
-------
TABLE 2.1 STRUCTURE OF THE OBSERVED CONCENTRATION DATA FILE.
Record
Type
Header 1
Header 2
Header 3
Observa-
tion
Data
Title (up to 40 characters)
Experiment number
Traverse number
Date
Time (Start & Stop)
Nominal distance to arc
Number of receptors on the arc
Starting data point for integration
Ending data point for integration
How x- and y-coordinates are entered:
polar or Cartesian (IXYARC)
Conversion from user units to meters
X-coordinate of source
Y-coordinate of source
Tracer release rate
Tracer release height
Altitude of airborne traverse
Constant multiplier to convert tracer
concentration from user's units to |ig/m3
X-coordinate
Y-coordinate
Observed Concentration
Data
Type
C
I
I
C
C
R
I
I
I
I
R
R
R
R
R
R
R
R
R
R
Description
Appears only once, as the first record in the
file, in single quotation marks
Field experiment period
Identifier for the arc
10 characters, in single quotes
' MM-DD-YY '
10 characters, in single quotes
' HHMM-HHMM'
(not used)
Total number of receptors that will be read by
the program
Receptors prior to this point are omitted from
analysis
Receptors after this point are omitted from
analysis
<= 0, cartesian
> 0, polar
Converts user units to meters, e.g., for
kilometers to meters, enter 1000.
Coordinate in base system
Coordinate in base system
Release rate in grams/second
Height of release in meters (not used)
Height of receptors in meters (not used)
Enter 1.0 if no conversion is needed, otherwise
enter the multiplier that will be applied to all
observed concentrations
If IXYARC <= 0, x is in meters
If IXYARC > 0, x is in degrees
If IXYARC <= 0, y is in meters
If IXYARC > 0, y is radial distance to arc
Concentration in user units
Data Types: C=character, I=integer, R=real
-------
'IND SF61
1, 1, '
1, 1,
24.71
1, 2, '
8, 1,
5.10
21.13
34.63
57.90
73.93
309.61
325.45
347.88
1, 3, '
4, 1,
38.25
332.79
338.05
356.90
1, 4,'
6, 1,
250.85
290.40
298.74
323.61
335.41
344.65
1, 5, '
11, 1,
6.07
15.99
257.66
263.77
269.59
280.49
287.25
293.21
302.81
341.93
358.25
1, 6,'
17, 1,
10.43
27.36
44.34
51.53
68.32
250.99
257.43
268.08
274.44
282.63
290.17
297.51
305.40
313.61
323.38
332.62
340.45
09-16-85 ',' 1100-1200',
1, 1 , 1000.000, 0.0,
.31 .13 .28
09-16-85 ' , ' 1100-1200' ,
8, 1 , 1000.000, 0.0,
.47 .04 .47
.47 .17 .44
.40 .23 .33
.45 .38 .24
.43 .42 .12
.50 -.39 .32
.55 -.31 .45
.48 -.10 .47
09-16-85 ',' 1100-1200',
4, 1 , 1000.000, 0.0,
.66 .41 .52
.71 -.32 .63
.68 -.25 .63
.72 -.04 .72
09-16-85 ',' 1100-1200',
6, 1 , 1000.000, 0.0,
1.43 -1.35 -.47
1.03 -.97 .36
.94 -.82 .45
1.01 -.60 .81
1.01 -.42 .92
1.02 -.27 .98
09-16-85 ',' 1100-1200',
11, 1 , 1000.000, 0.0,
1.41 .15 1.40
1.48 .41 1.42
. 1.50 -1.46 -.32
1.47 -1.46 -.16
1.38 -1.38 -.01
1.81 -1.78 .33
1.52 -1.45 .45
1.55 -1.42 .61
1.77 -1.49 .96
1.47 -.46 1.40
1.47 -.05 1.47
09-16-85 ',' 1100-1200',
17, 1 , 1000.000, 0.0,
1.97 .36 1.94
1.93 .88 1.71
1.92 1.34 1.37
1.95 1.52 1.21
1.92 1.79 .71
2.06 -1.94 -.67
1.97 -1.93 -.43
1.79 -1.79 -.06
2.20 -2.19 .17
1.97 -1.92 .43
2.03 -1.91 .70
2.01 -1.79 .93
2.07 -1.69 1.20
2.04 -1.48 1.41
2.13 -1.27 1.71
1.95 -.90 1.73
1.96 -.66 1.85
0.2
0.0, 4.94000, 83.80, 1.50,
7.0
0.5
0.0, 4.94000, 83.80, 1.50,
155.0
9.0
.0
.0
.0
.0
18.0
143.0
0.7
0.0, 4.94000, 83.80, 1.50,
.0
38.0
25.0
84.0
1.0
0.0, 4.94000, 83.80, 1.50,
.0
.0
.0
6.0
116.0
206.0
1.5
0.0, 4.94000, 83.80, 1.50,
225.0
59.0
.0
.0
9.0
.0
.0
.0
.0
79.0
444.0
2.0
0.0, 4.94000, 83.80, 1.50,
55.0
.0
.0
.0
.0
.0
.0
.0
.0
.0
.0
.0
.0
.0
6.0
15.0
40.0
1.000
1.000
1.000
1.000
1.000
1.000
Figure 2.1. Sample of the observed concentration file.
-7-
-------
SECTION 3
MODELED CONCENTRATION DATA
The second component required to perform the analysis is a file of modeled concentration
data. This file contains the concentration estimates from two or more dispersion models.
Results from as many model runs can be included (within DOS or compiler limitations) in this
file since the user must specify in the control file the number of model runs to include in the
analysis and the format to read the data. However, only results from up to 15 models can be
included in a single analysis. The structure of the file is shown in Table 3.1 and an example is
shown in Figure 3.1.
The format of these data is contained in the control file (described in Section 5).
Specifying the format in this manner allows the user to identify which model estimates to include
in the analysis.
It should be mentioned that the units of the modeled concentration estimates MUST
match those in the fitted observed data, otherwise the results will be completely unreliable. It
may be necessary to develop an external program that converts the modeled concentrations to the
units in the observed concentration file, as was done for the example used in this documentation.
TABLE 3.1 STRUCTURE OF MODELED CONCENTRATION DATA FILE.
Record
Description
Header record; can be any descriptive language; the information in this record
is not used in the program
Header record; can be any descriptive language; the information in this record
is not used in the program
3...#of
experiment-
arc periods
Modeled concentration data; multiple columns of data; repeated for as many
arc-experiment periods to be included in the analysis
Required data on each record for N model runs (N is specified in the control
file):
Experiment period, Arc number, Concentration 1,..., Concentration N
-------
(3x,2i5,fll.3,2x,i6.6,i4.
EXP TRAV
1
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
1
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
1
2
3
4
5
6
7
8
9
10
11
12
X
0.200
0.500
0.700
1.000
1.500
2.000
3.000
4.000
6.000
8.000
10.000
12.000
0.200
0.500
0.700
1.000
1.500
2.000
3.000
4.000
6.000
8.000
10.000
12.000
0.200
0.500
0.700
1.000
1.500
2.000
3.000
4.000
6.000
8.000
10.000
12.000
0.200
0.500
0.700
1.000
1.500
2.000
3.000
4.000
6.000
8.000
10.000
12.000
4,5fl2.5)
DATETIME
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161100
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161200
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161300
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
8509161400
13
124
129
106
72
52
31
21
13
9
7
5
9
116
122
101
68
48
28
19
11
8
6
5
6
110
120
99
66
46
26
18
10
7
5
4
6
109.
122.
102.
67.
47.
26.
17.
9.
6.
5.
4.
ADMS
.52654
.19214
.02158
.83546
.80358
18434
09959
46810
23311
56996
39425
92507
67782
65285
54763
43091
40762
55746
52670
32401
43619
15481
31754
09892
93544
24163
36591
47187
36179
57782
95085
02947
29495
16038
51101
45420
02123
91783
56912
79445
80977
12925
90252
84781
99287
80607
18173
17275
AER02161
15.88650
62.27761
62.43041
51.13265
35.57238
27.28770
18.67307
14.77599
11.21865
9.23732
7.90674
6.94058
17.32178
67.36076
67.21519
53.96894
35.32141
24.98616
15.65525
11.41748
8.05074
6.53249
5.56385
4.87275
16.03329
69.93465
70.52279
56.41741
36.37708
24.99461
14.90788
10.54413
7.00644
5.59702
4.74401
4.14563
0.01478
50.51303
89.26309
84.29244
57.40505
39.04949
20.83178
12.90603
6.44580
3.97508
2.83168
2.22595
20
120
115
88
55
40
27
21
16
13
11
10
16
117
112
85
52
34
20
15
11
9
7
6
11
102
103
81
51
34
19
13
9
7
6.
5.
3.
76.
92.
82.
56.
39.
21.
14.
8.
6.
5.
5.
HPDM
58027
69160
09613
37265
99589
92309
57795
56590
25815
34078
56248
34749
85009
75447
84499
90490
63741
97790
92225
39928
10693
14303
83048
94412
07608
03912
26080
77794
53038
29897
38106
86609
48170
80348
67414
88584
08309
51933
48806
91390
95367
30663
43528
23599
91191
91433
94636
25265
ISCST3
0
11
51
76
68
57
42
34
26
22
19
17
0
10
48
70
54
41
30
25
19
16
14
12
0
11
48
68
50
37
26
21
16
13
12
10
0
21
64
75
50
34
23
18
14
12
10
9
.00000
.44278
.72357
.59892
.48630
.06482
.60660
.89255
.69948
29126
47426
48786
00000
78590
39931
07381
24477
89233
92515
32321
37537
17562
13122
68949
00000
46303
85902
63754
91323
08468
71908
87267
73271
96850
20238
95750
00000
24328
44592
67673
77602
06233
24096
99567
52074
11810
58417
50299
AMODNoObsZi
22.98427
67.62427
65.56213
51.24015
32.48832
22.16314
13.06510
9.14368
5.99775
4.77061
4.03602
3.52151
19.70494
67.53592
66.77747
52.49763
33.22327
22.42547
12.81057
8.82312
5.54923
4.33742
3.65312
3.18173
17.17133
68.71230
69.20486
54.65300
34.47306
23.10219
12.85674
8.71999
5.30995
4.06644
3.40525
2.95946
0.01169
48.07773
86.99314
83.30546
56.88217
38.64830
20.55332
12.70320
6.32857
3.90839
2.79318
2.20017
Figure 3.1. Example of modeled concentration file
The first two records can contain any information the user wants to include since neither
record is used by the analysis software. In this example, the FORTRAN format of the data
appears on the first record. The second record identifies the fields. In this example, there are
two model runs with the AERMOD dispersion model: the first with the column header
AER02161 and a second with the column header AMODNoObsZi. The former run utilized
observed mixing heights, and the latter allowed AERMET to calculate the mixing heights.
-9-
-------
SECTION 4
REGIMES
The third file required to perform the ASTM analysis is a file of 'regime' definitions.
This file groups the experiment-arc combinations into categories that the user feels are
appropriate to answer the question(s) under investigation. Comparing separately averaged
observed and modeled concentrations within grouped data "provides an empirical estimate of the
combined deterministic error associated with input uncertainty and formulation errors" (ASTM,
2000). An extended discussion of establishing the regimes is provided in the ASTM standard
guide (ASTM, 2000).
Regimes can be defined based on almost any property, such as atmospheric stability, type
of air mass, or wind speed. The definitions of the regimes appearing in the example in this
document were based on z/L, where zi is the mixing height and L is the Monin-Obukhov length.
Two additional restrictions were placed on the data to insure there were sufficient data to
perform the analysis. The first was that a minimum of five nonzero observed concentrations
must appear on an arc to be included in a regime and that there must be a minimum of five
experiment-arcs per regime. Regimes, including the one presented here, are not hard and fast
definitions, and in fact, may require modification if the initial results are ambiguous or the
meteorology changes. Additionally, which model's meteorology (or other model-specific data)
to use to define the regimes must be considered when developing the regimes.
The structure of the regime definition file is shown in Table 4.1. There are three record
types, with the first type defining the number of regimes (and appearing only once, at the
beginning of the file), and the second type defining the number of experiment-arc combinations
in a regime. The third record type consists of the experiment-arc pairs within the regime. All
required data are read as free format integers.
An example of the regime definition file is shown in Figure 4.1. Note that in the
example, additional information follows the required data on each record. Only the first field is
read on record types 1 and 2, so any descriptive language can follow these values. For example,
the criteria for the data grouping can appear after the first field. In record type three, only the
first two fields are required and are read free format. Additional information can appear after
these two fields that may be useful in reviewing and understanding the results. In this particular
example, z /L is used as the criterion to group the data. The first regime is defined for z/L <
-50.0. The values that follow the experiment and arc number on each of the type 3 records is the
number of nonzero observed concentrations.
-10-
-------
TABLE 4.1 STRUCTURE OF THE REGIME DEFINITION FILE
Record Type
1
2
3
Description
Number of regimes to use; appears only once as the first record
Number of experiments-arc combinations in the regime; appears before
grouping of experiment-arcs
each
Experiment period and arc to include in this regime
-11-
-------
2
3
19
20
21
28
32
33
34
113
114
116
12
2
3
19
20
21
28
32
33
34
113
114
116
29 Number of Regimes
12 Number of Exps in this regime
6
Cases where Zi/L is less than: -50.0
7
6
6
9
8
13
10
5
Number of Exps in this regime
6
6
7
5
9
7
13
15
9
than:
22
23
24
25
29
30
31
35
55
14
22
Number of
-25.0
Exps in this regime Cases where Zi/L is greater than: -50.0 and less
9
9
7
8
7
8
10
5
5
Number of
4 8
Exps in this regime
Figure 4.1. Example of the regime definition file. The required fields and information are
shown in bold and the information that is not required is shown in a lighter font.
-12-
-------
SECTION 5
CONTROL FILE
Once the data files are in place, the remaining file required to run the ASTM evaluation
software is the control file. This file contains file names, data formats, and processing options.
Table 5.1 shows the structure of the control file and Figure 5.1 shows an example. The output
from the analysis software will be presented in Section 6.
The first six records in the control file pertain to the fitting of the observed concentration
data. The fourth record is needed only if the results of the fit are to be retained in a file.
Otherwise, the data are retained in memory for the remainder of the analysis and lost when the
program terminates. The fifth record (IPHIY) indicates how oy is to be retained, either in user
units or in degrees. The sixth record defines the minimum number of nonzero concentration
values required to attempt a Gaussian fit of the observed data.
The remaining controls are relevant to the analysis of the modeled data and the actual
evaluation analysis. Records 7-9 apply to the sampling scheme (NPAIR) and method (NWIDE),
and the number of bootstrap samples (NBOOT). NPAIR defines whether to sample the data as
pairs or individually; NWIDE defines whether to sample modeled and observed concentrations
simultaneously or independently.
Record 10 defines the number of models or models runs (NMODEL) that are to be
analyzed with this run. The analysis can be performed with up to 15 models in any single run.
Note too that different runs of the same model or versions of a model can be included in an
analysis. This record is followed by the names of the NMODEL models. The names are only
used to identify the models in the output files. It is recommended to keep the identifying names
to 10 characters or less. If the name is longer than 10 characters, only the first 10 characters will
be used. The order of the names must match the order the modeling results are read in the
modeled concentration file (Section 3).
The record following the model names is the random number seed and should be at least
five digits. This value is used during the bootstrapping process. The seed value is followed by
the filtering option. It is an integer value greater than or equal to zero and controls the maximum
number of near-centerline concentrations to be considered that satisfy -0.67 < y/0y < 0.67.
Entering a value of 0 selects all near-centerline concentrations that satisfy this criterion.
The last six records (shown in Table 5.1 as records 13+NMODEL through
18+NMODEL) in the control file are input and output file names and the format of the modeled
concentration file. The names should follow the standard personal computer (PC) naming
-13-
-------
conventions. The FORTRAN code and executable version accompanying this documentation
accommodates filenames as long as 70 characters in length. The format specified on the fifth
record from the end can be as long as 30 charcters in length, must follow FORTRAN format
statement rules for the data to be read correctly, and must include the initial and closing
parentheses.
-14-
-------
TABLE 5.1 STRUCTURE OF THE INPUT CONTROL FILE FOR THE ASTM
EVALUATION SOFTWARE.
Record
1
2
3
4
5
6
7
8
9
10
11
10+NMODEL
11+NMODEL
12+NMODEL
13+NMODEL
14+NMODEL
15+NMODEL
16+NMODEL
17+NMODEL
18+NMODEL
Control*
filename
FRMXYC
ISPLUS
output filename
IPHIY
MINNOK
NPAIR
NWIDE
NBOOT
NMODEL
model name
model name
ISEED
NFILTER
input filename
data format
input filename
output filename
output filename
output filename
Description
File where arcs of observed data are stored
Format to read the location (x- and y-coordinates) and observed concentration
(can be up to 30 characters in length, including parenthetses).
Option to save results from the PLTFIT routines in a file:
= 0, do not save results to file
# 0, save results to file
File with fitted observed data - an optional record based on the value given on
the 3rd record
oy output units:
= 0, list output ay in user units,
# 0, list oy in degrees
Minimum number of values required to attempt Gaussian fit
Sampling scheme:
= 2, sample by pairs
* 2, do not sample by pairs
Sampling method
= 2, sample simultaneously observation & modeled values
* 2, do not sample simultaneously
Number of bootstrap samples
Number of models for which values will be provided
First model name
Last model name
Random number seed value (make sure it is 5 or more digits)
Filtering option (if zero, then no limit; otherwise this is the max number of
values to be selected)
File of modeled centerline concentration values for each model
Format of the data of modeled centerline concentration values
File defining which experiment numbers and arcs are in each Regime
Name of output file for listing results - the primary listing for evaluating
model results with the ASTM standard
File with bootstrap results
File with regime results
* Controls in capital letters show the variable name used in the source code, otherwise a generic 'name' is used.
-15-
-------
INDARCS.DAT
(lx,F8.0,F8.0,16x,F8.0)
1
INDSPLUS.DAT
2
3
2
2
500
5
ADMS
AMOD02161
HPDM
ISCST3
AMODNoObZi
12345
1
INDMODEL.DAT
(3x,2i5,13x,10x,5fl2.0)
INDREGM.DAT
INDASTM.OUT
INDBOOT.PLT
INDREGIM.OUT
Figure 5.1 Sample control file.
-16-
-------
SECTION 6
RESULTS
In the earlier sections, portions of the various input files were shown. In this section, the
results from the analysis software are presented based on the example files presented in the
previous sections. Due to the size of the files, not all sections of the output files will be shown.
A brief description of the general layout of the files will be provided.
The output filenames are specified by the user, as shown in Table 5.1 and Figure 5.1.
The last two files in the control file are results from the bootstrap method and information by
regime. The third file from the end (identified as 16+NMODEL in Table 5.1, INDASTM.OUT
in Figure 6.1) contains the listing of results used to apply the ASTM standard. The first part of
this file has the following structure:
1) Run summary of the information provided in the control file;
2) Results for each regime, including the number of nonzero values for each
experimental period and the number of centerline values that satisfy the following -0.67
< y/ay < 0.67, where y is the receptor distance from centerline and sy is the standard
deviation of the distances for all nonzero concentration estimates;
3) Average and standard deviation of Ci, and the geometric average and standard
deviation of Ci/Cavg, where the near-centerline concentrations and averages use only the
values resulting from the filtering process (i.e., limit on number of near-centerline
concentrations).
Items 2) and 3) are repeated for all regimes. Figure 6.1 shows an example of these initial records
in the output file.
-17-
-------
ASTM D22.ll Boot. for Draft Z6849Z VERSION:
SETUP INFORMATION:
Iseed (random number seed value) :
Npair (0=individual, 2=pair sampling):
Nwide (0=individual, 2=concurrent) :
063002
12345
2
2
Nboots (number of bootstrap samples) : 500
Nmodel (number of models) :
Model 1 ADMS
Model 2 AMOD02161
Model 3 HPDM
Model 4 ISCST3
Model 5 AMODNoObZi
Iphiy (0=m, l=degrees) :
Nfilter (Limit on number selected) :
Model file:
Output file:
Regime Definition File:
Results for Group: 1
5
1
1
indmodel
indASTM.
.dat
out
indregm.dat
N Exp Arc NumValues NumCenterline
123 13
233 14
3 19 3 17
4 20 3 19
5 21 3 18
6 28 3 15
7 32 3 18
8 33 3 18
9 34 3 18
10 113 3 10
11 114 3 10
12 116 3 10
Total number of values: 180
Computed Group Sy: 19.10 (deg)
Avg Std
Num Ci Ci
— OBS 12 5.86E+01 4.49E+01
ADMS 12 1.28E+02 4.63E+01
AMOD02161 12 7.41E+01 1.77E+01
HPDM 12 1.20E+02 3.64E+01
ISCST3 12 7.17E+01 3.94E+01
AMODNoObZi 12 6.09E+01 3.73E+01
1
1
1
1
1
1
1
1
1
1
1
1
Num
>0
11
12
12
12
10
12
NumModels
5
5
5
5
5
5
5
5
5
5
5
5
GeoAvg
Ci/Cavg
6.30E-01
7.41E-01
9.63E-01
9.26E-01
1.14E+00
4.17E-01
GeoStd
Ci/Cavg
3.31E+00
3.37E+00
1.34E+00
1.58E+00
1.37E+00
9.58E+00
Figure 6.1. The initial records in the output file used to determine the best performing model
(associated with record NMODEL+16 in the control file).
-18-
-------
These statistics by regime are followed by the statistical measures over all regimes for
each model and maximum observed value. Currently, there are 12 statistical measures calculated
in the software: fractional bias, absolute fractional bias, normalized mean squared error (NMSE),
mean squared error (MSB), slope, intercept, r2, (unsystematic MSE)/MSE, (systematic
MSE)/MSE, Willmott-d, average observed value, and average modeled value. For each
statistical measure, an abreviated description is provided of the distribution of values generated
during the bootstrap: average, standard deviation, minimum, low hinge (25-th percentile),
median, high hinge (75-th percentile), and maximum value. It is these statistics that are used to
determine the "best performing model." An example of the summary information provided for
each statistical measure is shown in Figure 6.2.
For each statistical measure, the "best performing model" is indicated by an asterisk or an
excalamation point after the model name (and before the column labeled Avg). An asterisk is
used for those measures for which the smallest (absolute) average value identifies the best
performing model. The exclamation point is used for those measures for which a value closest to
1.0 is appropriate, such as the correlation coefficient and the slope. Note that the first three and
the last three statistical measures are shown in the figure; there are another six measures not
shown.
-19-
-------
Summary Over All Groups
(No Inverse Variance Weighting
Employed)
Fractional Bias Results
Model Avg
MAX 0.11905
ADMS * 0.00008
AMOD02161 -0.29222
HPDM -0.03237
ISCST3 0.03411
AMODNoObZi -0.22308
t-value compared to the base
2 AMOD02161 5.7244
3 HPDM 0.8565
4 ISCST3 0.8028
5 AMODNoObZi 5.5705
Degrees of freedom (DF) : 28
Std
0.02346
0.06137
0.06333
0.06574
0.05951
0.06185
Nboots
500
500
500
500
500
500
Min
0.06918
-0.18474
-0.52664
-0.22245
-0.17623
-0.40081
HL
0.10212
-0.04431
-0.33212
-0.08129
-0.00250
-0.26444
Med
0
-0
-0
-0
0
11892
00080
29712
03377
03322
-0.22674
0
0
-0
0
0
-0
HH
.13289
.04321
.25111
.01350
.07034
.17864
Max
0
0
-0
0
0
-0
20091
22366 *
09162
17498
21014
02809
model : ADMS
(at the 90% confidence level n
significant difference in the
metrics
between the base model and the model
being
tested)
Absolute Fractional Bias Results
Model Avg
MAX 0.11905
ADMS 0.55608
AMOD02161 0.60549
HPDM * 0.46838
ISCST3 0.48538
AMODNoObZi 0.54344
t-value compared to the base
1 ADMS 2.0452
2 AMOD02161 3.0677
4 ISCST3 0.3725
5 AMODNoObZi 1.6021
Degrees of freedom (DF) : 28
(at the 90% confidence level
Std
0.02346
0.04422
0.04837
0.04687
0.05049
0.04826
Nboots
500
500
500
500
500
500
Min
0.06918
0.43668
0.46769
0.30258
0.32799
0.40345
HL
0.10212
0.52596
0.57155
0.43679
0.45442
0.51097
Med
0
0
0
0
0
0
11892
55828
60523
47103
48618
54504
0
0
0
0
0
0
HH
.13289
.58757
.63789
.50124
.51749
.57555
Max
0
0
0
0
0
0
20091
68484
74218
62682 *
66683
67938
model: HPDM
a
significant difference in the
Normalized Mean Squared Error
Model Avg
MAX 0.02203
ADMS 0.51363
AMOD02161 0.45289
HPDM * 0.34060
ISCST3 0.42333
AMODNoObZi 0.42698
t-value compared to the base
1 ADMS 2.7531
2 AMOD02161 1.4852
4 ISCST3 0.7391
5 AMODNoObZi 1.0565
Degrees of freedom (DF) : 28
(at the 90% confidence level
value >
metrics
1.701 indie.
between the
ites that
there is a
base model and the model
being
tested)
Sesults
Std
0.00956
0.06581
0.09301
0.07401
0.11733
0.09067
Nboots
500
500
500
500
500
500
Min
0.00478
0.33369
0.24901
0.10480
0.18697
0.22267
HL
0.01491
0.46615
0.38521
0.29104
0.34874
0.36160
Me
0
0
0
0
0
0
•d
02013
51186
44552
33899
40906
42129
0
0
0
0
0
0
HH
.02714
.55525
.51077
.38767
.48065
.48415
Me
0
0
0
0
1
0
X
06065
72839
84992
63089 *
04803
79317
model: HPDM
a
significant difference in the
Willmott - d
Model Avg
MAX 0.97037
ADMS 0.59000
AMOD02161 0.61483
HPDM ! 0.64662
ISCST3 0.59026
AMODNoObZi 0.60632
t-value compared to the base
1 ADMS 1.1011
2 AMOD02161 0.5605
4 ISCST3 0.7755
5 AMODNoObZi 0.6283
Degrees of freedom (DF) : 28
(at the 90% confidence level
value >
metrics
1.701 indie
between the
ates that
there is a
base model and the model
being
tested)
Results
Std
0.01428
0.05730
0.06993
0.07320
0.08255
0.07067
Nboots
500
500
500
500
500
500
Min
0.91508
0.41442
0.42199
0.46643
0.33910
0.42730
HL
0.96186
0.55072
0.56497
0.59056
0.53556
0.55701
Med
0
0
97311
59026
0.61813
0
0
0
64853
59158
60583
0
0
0
0
0
0
HH
.98161
.62879
.66311
.69900
.65071
.65867
Max
0
0
99403
76611
0.80535
0.89654 !
0
78874
0.82291
model: HPDM
a
significant difference in the
Average of Observed Values
Model Avg
MAX 49.65137
ADMS 49.65137
AMOD02161 49.65137
HPDM 49.65137
ISCST3 49.65137
AMODNoObZi 49.65137
value >
metrics
1.701 indie
between the
ites that
there is a
base model and the model
being
tested)
Results
Std
2.83214
2.83214
2.83214
2.83214
2.83214
2.83214
Nboots
500
500
500
500
500
500
Min
42.61565
42.61565
42.61565
42.61565
42.61565
42.61565
HL
47.59573
47.59573
47.59573
47.59573
47.59573
47.59573
Med
49
49
49
49
49
49
72298
72298
72298
72298
72298
72298
51
51
51
51
51
51
HH
.62337
.62337
.62337
.62337
.62337
.62337
Max
59
59
59
59
59
59
85370
85370
85370
85370
85370
85370
Figure 6.2. A portion of the output file with the statistical measures for the ASTM analysis.
The "best performing model" is indicated by an asterisk or exclamation point for
each statistical measure.
-20-
-------
To assess whether or not there is a significant difference in the statistical measure
between the "best performing model" and the other models, the bootstrap results for the
statistical measure are used to compute the t-value(s). The null hypothesis that the average of the
model bootstrap differences is greater than 0 can be tested for a given confidence level (ASTM,
2000). The t-values are shown immediately below the statistical measures in Figure 6.2.
Examining the Normalized Mean Square Error in Figure 6.2, HPDM is the "best performing
model," but there is no statistical difference at the 90% confidence level between HPDM and
both versions of AERMOD and ISCST3. However, there is a significant difference between
HPDM and ADMS.
The other two output files are a file of bootstrap results and results by regime. Each
contains different information that may be useful in interpreting the results. The data used to
compute the t-value(s) come from the bootstrap file.
The file of bootstrap results is divided into five parts:
1) Run summary of the information provided in the control file;
2) Averages and standard deviations of observed and modeled concentrations by regime;
3) Averages and standard deviations of the fractional bias and absolute fractional bias for
each regime. The definitions of fractional bias can be found in the discussion of the
ASTM standard (ASTM, 2000);
4) Averages computed for each boot and regime;
5) Comparison statistics for each boot; for each statistical measure.
In this example, there are nearly 21,000 records in the output file. Due to the length of
the initial records and the number of records in the file, a sample is not shown here.
Figure 6.3 shows a sample of the information by regime. The standard deviation of the
concentration values along the arc is listed first. Then for each experiment-arc combination an
analysis is provided of the observed concentration values satisfying the criterion of being within
-0.67 < y/0y < 0.67. Ci is a concentration value for this arc that satisfies the requirements of
NFILTER and Cavg is the average of all N values with zero values excluded for this regime.
The asterisk next to a Ci value indicates this value is the maximum observed concerntation found
anywhere along the arc. There can be multiple occurrences for an experiment-arc pair depending
on the value of NFILTER and the number values that are within ±0.67y/0y. Some additional
statistics appear below this grouping and are based on the nonzero values. These statistics are
followed by similar results where NFILTER was applied in the modeling. In this example,
NFILTER was set to one, so only one near-centerline concentration is selected for analysis.
-21-
-------
Process Regime :
Sy(deg) : 19.10201
i of N Exp Arc
1 27 2 3
2 27 2 3
3 27 2 3
4 27 3 3
5 27 19 3
6 27 19 3
7 27 19 3
8 27 20 3
9 27 20 3
10 27 20 3
11 27 21 3
12 27 21 3
13 27 28 3
14 27 28 3
15 27 28 3
16 27 32 3
17 27 32 3
18 27 33 3
19 27 33 3
20 27 33 3
21 27 34 3
22 27 34 3
23 27 113 3
24 27 113 3
25 27 114 3
26 27 116 3
27 27 116 3
Avg Std
Ci Ci
54.0696 40.2394
1
Ci
48.5830
92.7125*
35.4251
78.5425*
34.5494*
26.8240
30.2575
53 .8627
108.7983*
72.9614
148.1799*
134.4754
0.0000
0.0000
30.5376*
60.6452*
56.7742
72.4731
109.6774*
68.3871
21.7204
13.7634
5.5794
5.7940
3.2189
81.3305*
64.8069
GeoAvg
Ci/Cavg
0.7105
Cavg
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
58.3952
GeoAvg
Ci/Cavg
2.6809
Ci/Cavg
0.8320
1.5877
0.6066
1.3450
0.5916
0.4594
0.5182
0.9224
1.8631
1.2494
2.5375
2.3029
0.0000
0.0000
0.5229
1.0385
0.9722
1.2411
1.8782
1.1711
0.3720
0.2357
0.0955
0.0992
0.0551
1.3928
1.1098
Results from Subroutine Filter
i of N Exp Arc
1 12 2 3
2 12 3 3
3 12 19 3
4 12 20 3
5 12 21 3
6 12 28 3
7 12 32 3
8 12 33 3
9 12 34 3
10 12 113 3
11 12 114 3
12 12 -116 3
Avg Std
Ci Ci
58.5942 44.8805
Process Regime :
Sy(deg) : 19.13367
i of N Exp Arc
1 30 2 4
2 30 2 4
Ci
92.7125*
78.5425*
26.8240
108.7983*
134.4754
0.0000
56.7742
109.6774*
21.7204
5.5794
3 .2189
64.8069
GeoAvg
Ci/Cavg
0.6298
2
Ci
66.1943*
41.9028
Cavg
63.9209
63 .9209
63 .9209
63.9209
63 .9209
63 .9209
63 .9209
63 .9209
63 .9209
63.9209
63 .9209
63.9209
GeoAvg
Ci/Cavg
3.3111
Cavg
54.5058
54.5058
Ci/Cavg
1.4504
1.2287
0.4196
1.7021
2.1038
0.0000
0.8882
1.7158
0.3398
0.0873
0.0504
1.0139
Ci/Cavg
1.2144
0.7688
Figure 6.3. Sample of the output by regime.
-22-
-------
SECTION 7
REFERENCES
ASTM, 2000: Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model
Performance. D 6589-00. American Society for Testing and Materials, West Conshohocken,
PA.
Cox, W. M. and J. A. Tikvart, 1990: A Statistical Procedure for Determining the Best Performing
Air Quality Model, Atmospheric Environment, 24A, 2387-2395.
Fox, D. G., 1981: Judging Air Quality Model Performance, Bulletin of the American
Meteorological Society, 62, 599-609.
Murray, D. R. and N. E. Bowne, 1988: Urban Power Plant Plume Studies. EPPJ Report No. EA-
5468, Research Project 2736-1, Electric Power Research Institute, Palo Alto, CA.
Oreskes, N., K. Shrader-Frechette and K. Belitz (1994): Verification, validation, and
confirmation of numerical models in the earth sciences. Science, 263: 641-646.
-23-
-------
APPENDIX
DIRECTORY STRUCTURE OF DISTRIBUTION CD
ASTM Root Directory
Indianapolis
ASTM Evaluation
Dispersion Modeling
ADMS
AERMOD
AERMOD_NoObZi
HPDM
ISCST3
Meteorology
ADMS
AERMOD
AERMOD_NoObZi
HPDM
ISCST3
Kincaid
ASTM Evaluation
Dispersion Modeling
ADMS
AERMOD
AERMODJTurb
HPDM
ISCST3
Meteorology
ADMS
AERMOD
AERMODJTurb
HPDM
ISCST3
MixHts
-24-
-------
PrairieGrass
ASTM Evaluation
Dispersion Modeling
ADMS
AERMOD
HPDM
ISCST3
Meteorology
ADMS
AERMOD
HPDM
ISCST3
MixHts
Documentation
Software
-25-
-------
TECHNICAL REPORT DATA
(Please read Instructions on reverse before completing)
1. REPORT NO.
EPA-454/R-03-006
3. RECIPIENTS ACCESSION NO.
4. TITLE AND SUBTITLE
A Program for the Statistical Evaluation
Of Point Source Dispersion Models
Using ASTM D 6589
5. REPORT DATE
August 2003
6. PERFORMING ORGANIZATION CODE
7. AUTHOR(S)
8. PERFORMING ORGANIZATION REPORT NO.
9. PERFORMING ORGANIZATION NAME AND ADDRESS
Pacific Environmental Services, Inc, a MACTEC
Company
Research Triangle Park, NC
10. PROGRAM ELEMENT NO.
11. CONTRACT/GRANT NO.
12. SPONSORING AGENCY NAME AND ADDRESS
Office of Air Quality Planning and Standards
Emissions, Monitoring, and Analysis Division
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
13. TYPE OF REPORT AND PERIOD COVERED
Technical Report
14. SPONSORING AGENCY CODE
15. SUPPLEMENTARY NOTES
16. ABSTRACT
The ASTM published a Standard Guide D 6589 for Statistical Evaluation of Atmospheric
Dispersion Model Performance, which provides a procedure for assessing performance of
atmospheric transport and diffusion models to predict the "average centerline"concentration
values of a chemical released from a point source. The EPA has developed a FORTRAN
program entitled "Design 2" that implements this procedure. This report describes the input
files and formats of these input files for running Design 2. The report describes the
information provided by the various output files created by Design 2. The report does not
describe the actual statistical procedures of Standard Guide D 6589, as these are documented
in detail within this Guide's Annex.
17.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b. IDENTIFIERS/OPEN ENDED TERMS
c. COSATI Field/Group
Meteorological Data
Air Dispersion Models
Statistical Model Evaluation
18. DISTRIBUTION STATEMENT
Release Unlimited
19. SECURITY CLASS (Report)
Unclassified
21. NO. OF PAGES
34
20. SECURITY CLASS (Page)
Unclassified
22. PRICE
EPA Form 2220-1 (Rev. 4-77) PREVIOUS EDITION IS OBSOLETE
-------
United States Office of Air Quality Planning and Standards Publication No. EPA-454/R-03-006
Environmental Protection Planning and Standards August 2003
Agency Research Triangle Park, NC 27711
------- |