OCR error (C:\Conversion\JobRoot\00000CUZ\tiff\20015OJB.tif): Unspecified error
-------
RESEARCH REPORTING SERIES
Research reports of the Office of Research and Development, U S Environmental
Protection Agency, have been grouped into nine series These nine broad cate-
gories were established to facilitate further development and application of en-
vironmental technology Elimination of traditional grouping was consciously
planned to foster technology transfer and a maximum interface in related fields.
The nine series are
1 Environmental Health Effects Research
2 Environmental Protection Technology
3 Ecological Research
4 Environmental Monitoring
5 Socioeconomic Environmental Studies
6 Scientific and Technical Assessment Reports (STAR)
7 Interagency Energy-Environment Research and Development
8 ' Special" Reports
9 Miscellaneous Reports
This report has been assigned to the ENVIRONMENTAL MONITORING series
This series describes research conducted to develop new or improved methods
and instrumentation for the identification and quantification of environmental
pollutants at the lowest conceivably significant concentrations It also includes
studies to determine the ambient concentrations of pollutants in the environment
and/or the variance of pollutants as a function of time or meteorological factors
This document is available to the public through the National Technical Informa-
tion Service, Springfield, Virginia 22161
-------
EPA-600/4-80-013d
February 1980
EVALUATION OF THE REAL-TIME
AIR-QUALITY MODEL
USING THE RAPS DATA BASE
Volume 4. Evaluation Guide
by
Ronald E. Ruff
Atmospheric Science Center
SRI International
Menlo Park, California 94025
Contract No. 68-02-2770
Project Officer
John S. Irwin
Meteorology and Assessment Division
Environmental Sciences Research Laboratory
Research Triangle Park, North Carolina 27711
ENVIRONMENTAL SCIENCES RESEARCH LABORATORY
OFFICE OF RESEARCH AND DEVELOPMENT
U.S. ENVIRONMENTAL PROTECTION AGENCY
RESEARCH TRIANGLE PARK, NORTH CAROLINA 27711
-------
DISCLAIMER
This report has been reviewed by the Environmental Sciences Research
Laboratory, U.S. Environmental Protection Agency, and approved for publica-
tion. Approval does not signify that the contents necessarily reflect the
views and policies of the U.S. Environmental Protection Agency, nor does men-
tion of trade names or commercial products constitute endorsement or recom-
mendation for use.
ii
-------
ABSTRACT
The theory and programming of statistical tests for evaluating the
Real-Time Air-Quality Model (RAM) using the Regional Air Pollution Study
(RAPS) data base are fully documented in four volumes. Moreover, the
tests are generally applicable to other model evaluation problems. Vol-
ume 4 discusses the application and interpretation of the statistical
programs, particularly with regard to use on the RAM. In general, there
is no set procedure for evaluating an air-quality model because of the
different reasons for evaluating models and many subjective decisions
to be made during the process. However, guidelines are presented to
cover a wide variety of evaluation needs, with attention to data prepara-
tion, classification, analysis, .selection and application of tests, and
interpretation of results. Several methods of diagnosing causes of poor
model performance are discussed and some sample program outputs are also
provided. This report was submitted in fulfillment of Contract No.
68-02-2770 by SRI International under the sponsorship of the U.S. Envi-
ronmental Protection Agency. This report covers a period from 1
October 1977 to 1 April 1979, and work was completed as of 1 April 1979.
iii
-------
CONTENTS
Abstract iii
Figures vi
Tables vi
1. Overview 1
Introduction 1
General procedures 1
2. The Model Evaluation Process 3
Discussion .....3
Data preparation 4
Data analysis 12
Selection and application of the
statistical tests 14
Interpretation of results 16
Diagnosing the causes of poor
model performance 25
Combinations of poor performance indicators .... 33
References 35
Appendices
A. SPSS multiple regression output
for test data base 36
B. Partial sample of output for SPSSSORT program 45
-------
FIGURES
Number Page
1 General procedure for evaluating models 2
2 RAM evaluation data bases 11
3 General logic to compute multihour averages 13
4 Frequency distribution for test data base 17
5 Residual frequency distribution for test data base 17
6 Actual versus ideal fit for test data base 19
7 Scattergram with bounds for the test data base 19
8 Autocorrelation function of test data base 21
9 Normalized cumulative periodogram for test data base .... 21
10 SPSS summary output for test data base 24
11 Types of bias 27
12 Temporal bias example 31
TABLES
Number Page
1 Wind-Speed Classes 6
2 Mixing-Depth Classes 6
3 Temperature Classes 7
4 Modified Test Data Base Hourly Format 9
5 Description of RAM Evaluation Format and Parameters 10
VI
-------
SECTION 1
OVERVIEW
INTRODUCTION
In the previous volumes, a distinction was made between final-
evaluation statistics and intermediate-evaluation statistics. Final-
evaluation tests compute accuracy scores, which are used to compare a
model's performance against some standard, such as a competing model or
a different version of the same model. These accuracy scores do not pro-
vide any insight as to why a model is performing poorly (or well).
Instead, they provide a bottom-line figure (score) which sums up the
performance.
Intermediate evaluation statistics are used to diagnose the reasons
for a model's poor performance, either in general or for certain meteoro-
logical or emission conditions. In all, there are five tests that can
be used for intermediate evaluation: residual time series, chi-square
goodness-of-fit, bivariate regression and correlation, interstation error
correlation, and multiple regression of error residuals. Graph displays
are incorporated into most of the programs to facilitate interpretation.
GENERAL PROCEDURES
As will be discussed in the following pages, the general method of
applying the programs remains essentially the same regardless of the
specific air-quality model being tested. To adapt the programs to the
RAM evaluation, only minor software modifications must be made. These
include: (1) scaling the graphical displays, (2) annotating the output,
(3) adjusting DIMENSION statements to accommodate the proper number of
observations and stations, and (4) formatting the input data. The docu-
mentation on the above items is contained in Volume 3 of this report,
"Program User's Guide."
-------
As will be discussed in the following sections, the recommended model
evaluation procedure is an iterative process whereby results from one
test are used to decide which test to apply next. Before delving into
the specifics, consider the generalized procedure shown in Figure 1.
Our first step is to analyze the data base in a number of ways. After we
find out a little about the data base, we can make a well-founded decision
about which statistical test to apply. We then apply that test and ana-
lyze the results. Then we can opt to (1) end our evaluation, (2) conduct
another test, or (3) modify the model and proceed through the evaluation
process again.
PREPARE AND ANALYZE
DATA BASE
SELECT THE
STATISTICAL TEST
APPLY TEST
MODIFY
MODEL
L^-OPTION 3-
ANALYZE RESULTS
-OPTION 2
-J
Figure 1. General procedure for evaluating models.
The flow chart in Figure 1 depicts a human-decision process. (It
is not a flow chart for a computer program.) Because subjective human
decisions are involved, there are many possible variations in the speci-
fic choices. Guidelines for these choices are presented in the following
sections.
-------
SECTION 2
THE MODEL EVALUATION PROCESS
DISCUSSION
As we discussed in the previous section, there will be many varia-
tions possible in our testing strategy for the RAM or any other model.
The particular strategy depends on (1) the user-defined evaluation cri-
teria, (2) the interpretation of intermediate test results, and (3) a
subjective evaluation of the best course of action. Item (1) above recog-
nizes that the most important evaluation criteria can vary among applica-
tions. For instance, one user might be most interested in evaluating
performance for a 24-hour averaging period while another may select a
1-hour averaging period. The interpretation of the results of a particu-
lar test, item (2), will determine which test will be conducted next and
how to stratify the data base accordingly. Finally, for item (3), we
recognize that there are many "gray" areas in selecting a future course
of action. After evaluation of some intermediate test results, one user
may choose to run additional tests while another may decide to modify
the model and then rerun the same test.
There are probably numerous reasonable approaches to the model-
evaluation process, each of which could produce the same successful
results. Therefore, our discussion by necessity must dwell on a general
testing procedure, with some assumptions on how the RAM test would pro-
ceed. We also attempt to discuss the various alternatives along the way.
For evaluating the RAM, we are assuming that the primary objective
is to improve the model for the one-hour averaging time. Secondary objec-
tives would be to compare the RAM with other models; assess its accuracy
against some user-defined standards; or evaluate it for other averaging
times, such as 3-hour, 24-hour, monthly, seasonal, and annual. For these
secondary objectives, the final evaluation statistics are recommended.
-------
As we will be noting in the following sections, some intermediate evalua-
tion statistics may also be helpful, but only to complement the final-
evaluation statistics.
In the remainder of this section we discuss the steps in the model-
evaluation process, starting with data preparation and continuing through
the last computation of the final-evaluation statistics.
DATA PREPARATION
In this volume we have assumed that the primary objective is to eval-
uate and improve the RAM by testing it with the RAPS data base. As a
start, the test data base consists of hourly values covering a one-year
period for (1) observed and predicted concentrations at 13 locations;
(2) wind measurements at these locations; (3) derived wind speed and
direction, atmospheric stability class, mixing height, and temperature
parameters representing the St. Louis area (as a whole); and (4) an
emission inventory. The 13 locations are for the monitoring stations in
the RAPS network. The observed concentrations are averaged from the S09
monitoring instruments; the predicted concentrations result from applying
the RAM to the input data from (3) and (4) above.
Classification of Parameters
Except for atmospheric stability all the RAM input data are numerical
values which are directly averaged from the instrument readings. To stra-
tify test results according to meteorological and/or emission conditions,
the data are divided into classes for each relevant RAM input parameter.
The particular parameters and recommended number of classes are:
Parameter Number of Classes
Atmospheric stability 7
Wind speed 6
Wind direction 16
Mixing height 7
Temperature 5
Emissions 16
-------
If we examine the number of combinations from the above classes, we see
that there are 376,320 possible categories. Further, we note that there
will be only 8,784 hourly SO concentrations for the yearly (1976) test
data base. Therefore, we will have to combine the parameter categories
in a reasonable manner so that within each category there is a statisti-
*
cally valid sample (e.g., 10 cases ). For our purposes, it is better not
to limit the number of categories at this time. This is best done after
the RAPS data base has been reviewed, as will be discussed later.
Our classification structure, at this point, is taken mostly from
the literature. As stated earlier, stability is classified in one of the
seven atmospheric stability categories for an urban area, as calculated
in the RAM preprocessor. Wind directions are divided into 16 sectors of
o t
22.5 each, with the first sector centered in north. Wind speed is
divided into six categories according to the scheme used by the Climato-
2*
logical Dispersion Model. (See Table 1.) Mixing-depth classes corres-
pond to those used in the APRAC urban diffusion model.2 (See Table 2.)
Ambient temperature measurements are used both explicitly by RAM,
in the plume-rise calculation, and implicitly, in the emissions inven-
tory. (S0? emissions from space-heating sources are computed when ambi-
ent temperatures are below 69 F, or 20°C.) Our temperature categorization
o
scheme uses the space-heating cutoff temperature, 20 C, as a dividing
point between the third and fourth categories. Table 3 describes the
temperature categories.
For emission categories we suggest using the actual RAPS emission
rates. Prior to analyzing the data base, we tentatively suggest dividing
the inventory into 4 categories of area-source emissions and 4 categories
*
For statistical significance, the required number of cases depends on
the standard deviation of the sample
In our terminology, this corresponds to the direction from which the
wind is blowing.
*
References are listed at the end of this report.
-------
TABLE 1. WIND-SPEED CLASSES
Wind-Speed Class
1
2
3
4
5
6
Speed Interval
(knots)
0-3
4-6
7-10
11-16
17-21
>21
Central Wind Speed
(m/sec)
1.50
2.46
4.47
6.93
9.61
12.51
TABLE 2. MIXING-DEPTH CLASSES
*
Mixing-Depth Class
1
2
3
4
5
6
7
Mixing-Depth Interval
(meters)
iioo1'
100-200
200-400
400-800
800-1,600
1,600-3,200
>3,200
Geometric Mean
of Depth
70.0
141
283
566
1,131
2,262
4,525*
For classes 1 and 7, the geometric mean was calculated as
though the classes were bounded.
For the RAPS data base, mixing depths are a minimum of
100 meters.
-------
TABLE 3. TEMPERATURE CLASSES
Temperature Class
Temperature Interval
1
2
3
4
5
0-10
10-20
20-30
>30
of point-source emissions. Alternatively one could use the four seasons
of the year, a weekend-weekday separation, and a day-night separation.
(This results in 16 possible emissions categories: 4 seasons X 2 weekend-
weekday x 2 day-night.) In addition, as mentioned earlier, ambient tem-
perature could also be used in conjunction with the time period to fur-
ther categorize emissions.
Formats
The format used for our test data base, described in Volume 3,
must be modified somewhat for evaluating the RAM. The difference is that
for RAM we have derived wind temperature, stability, and mixing-height
data, which are used for model input. In addition we have wind measure-
ments averaged hourly for each station, which we also recommend incor-
porating into the format. (However, this is optional.) Therefore, the
recommended data base consists of values averaged hourly for the
following:
• Interpolated wind-speed class
• Interpolated wind-direction class
• Mixing-height class
• Temperature class
• Site-specific wind speed (optional)
-------
• Site-specific wind direction (optional)
• RAM-predicted concentration for each site
• RAPS-observed concentration for each site.
For the last four items, there will be 13 observations for each time
period. (This corresponds to the number of sites for which SO monitoring
data are available.) For the first four items there is just one observa-
tion per time period. We also suggest adding the following logistic
parameters:
• Date (one per time period)
• Time (one per time period)
• Site identification (13 per time period).
Therefore, for a full (leap) year (8,784 hours), the size of our data
base is:
8,784 X 6 + 8,784 X 5 X 13 = 623,664*
The manner in which these are ordered is considered next.
The general format of the test data base was given in Volume 3.
To process the RAM data base efficiently, we suggest a slight modifica-
tion in the hourly average format as indicated in Table 4. The parameters
are defined in Table 5. Note that for each hourly time period there are
NS + 1 lines. (We allow that in a mass storage file line numbers are
irrelevant. However, it simplifies our discussion here.) In general, the
first line of the group contains the categorized equivalent to the RAM
input data. The next 13 lines contain the observed and predicted concen-
trations and the wind measurements for each of the 13 stations.
*
If we eliminate the site-specific wind speed and direction the size
reduces to 394,500 words.
-------
TABLE 4. MODIFIED TEST DATA BASE HOURLY FORMAT
Line
Data
1
2
3
NS + 1
14
15
16
17
NS + 2
28
29
1,OC1, PC1,WS11,WD11
2,OC2,PC2,WS21,WD21
NS ,
,WDNS
D2,T2,ASC2,WSC2,WDC2
NS '°C2 -NS >PC2 -NS 'WSNS2 'WDNS2
D3 ,T3 ,ASC3 ,WSC3
-------
m
d
o
•H
4-1
CX
•r-l
S-i
o
CO
0)
P
*
1 1
cfl
g
O
0)
2 B
II
cfl
PM
cu ,,
J_l
0) _g
6 p
CO |
a*
*u
^_i
£ji
^ §
c ^
•H ^
•-1
f-x,
1
CM r-l
0
0
r-l CO
VO CO
1 • Cfl
i— 1
•> O
bQ ^
4-1
0) -H
r-l
•i-l
^•x XI
t>» CT* cfl
co m 4J
TJ CO CO
1 CM 1
O 1 CJ
B o M-I
1 0 S-l
S-i O 0)
>•> O JZ
*-s P-
•> CO
0) Q) O
•M B B
Cfl -S 4-1
O H <
VO r-l VO
0) CO i-l
S-l CO •> 1
•r-l CO CO r-l
13 r-l CO
1 0 Cfl "
13 r-l CO
d 4-> 0 CO
•H x; cfl
£ 00 0) r-l
•H S-l CJ
0) CD 3
4-1 x: 4J d
•r-l I cfl O
CO bO S-i T-I
O d Cl) CO
ex -H ex co
6 X g T-I
O -H Q) B
u s H w
CM r-l r-l vO
M M M M
U O
§ g H W
in vo i^- oo
m
CM
r-l
1
I-l
o
r-l
•*
S-i
OJ
Xl
d
d
o
•H
4J
cfl
4-1
CO
60
d
•H
S_i
o
4-1
•r-l
d
0
S
CO
M
to
r-l
•
oo
CM
vO
*"*
„
.
r-l
4J
O
U)
S-i
•I-l
13
4-1
O
d
cu
S-l
CO
^
cu
x;
4-1
CU
CQ
3
CO
CJ
0)
XI
13
CU
4J
4J
•I-l
B
6
cu
XI
>
a.
.
CQ
d
o
• !-(
4-1
CO
r-l
3
O
r-l
Cfl
U
TJ
C
•1-1
*
cu
4-1
•r4.
CQ
O
§
O
cu
x;
4J
14-1
O
r*1!
CJ
Cfl
3
cr
cu
13
CO
0)
JC
4J
bo
d
•r4
CO
CO
Q)
CQ
CO
Cfl
, S-i
o
10
-------
For the RAM evaluation, data exist in three forms, as illustrated
in Figure 2. These data will be supplied in three segments, as shown
on the left-hand side of the figure. The data will be processed into a
single data base in the format described in the previous paragraph. Fur-
thermore, we recommend an abbreviated data base consisting only of the
observed and predicted concentrations covering the time period of interest
for the 13 stations (e.g., for the year). Because most of the statistical
tests use only the observed and predicted concentrations, the abbreviated
data bases are generated to reduce processing costs. For the year, the
maximum size of this test data base is 228,384 words (8,784 hrs X 2 con-
centrations X 13 sites). However, some of these data bases will be segre-
gated by given meteorological and emissions conditions and thus could be
much smaller in size. The SPSS software described in Volume 3 can be
used to segregate the data.
Calculation of Nonhourly Averages
The statistical package can be used to evaluate the RAM for non-
hourly averages. Typical averaging periods of interest would include
3-hour, 24-hour, monthly, seasonal, and annual. To form the data base
for these periods, a simple FORTRAN program that converts the hourly
averaged data into the desired form is recommended.
RAM INPUT
DATA
RAPS MONITORING
DATA
RAM PREDICTIONS
RAM EVALUATION
DATA
ABBREVIATED
DATA BASES
Figure 2. RAM evaluation data bases.
11
-------
Figure 3 illustrates a suggested logic for the FORTRAN program,
which assumes that the data are time-sequential for each group of 13
sites. The date and time are also needed because the time index is com-
puted from the time, for the 3-hour average, and from the date, for the
24-hour, monthly, seasonal, and annual averages. (For instance, if we
assume time ranges from 0000 to 2300, the 3-hour average time index can
be computed from the following: 1 + ITIME/3. Since the second term
will truncate to an integer, we note that the index will range between 1
and 8.) The time index is needed only to indicate when to compute the
specified average. After the time index is computed, the program deter-
mines if the index has changed. If it has not, the observed (OC) and
predicted concentrations (PC) are accumulated, along with the count. If
the time index has changed, the observed and predicted concentrations are
averaged (if valid) and written on the output file. This process conti-
nues until an end-of-file is encountered.
Checks are needed to separate out bad data (denoted by a "-99") so
they are not counted in the multihour average. In addition, after the
time index changes, the number of data points comprising the average must
be checked. Do we need three for the hourly average, or will two suffice?
Such criteria must be established for each average of interest. For the
RAPS data base, these criteria will be fairly rigid, provided we can get
a valid statistical sample.
DATA ANALYSIS
Prior to conducting statistical tests, we recommend first analyzing
the data in an elementary manner to gain a better persepective on the
distribution of the observed and predicted concentrations and the cate-
gorized RAM input parameters. The EPA program, stored in SRI.FREQ (as
documented in Volume 3, p. 13), produces distributions on logarithm-
frequency axes. The SPSS3 procedure, FREQUENCIES, can be used to obtain
line-printer histograms for the RAM input parameters (see Volume 3,
p. 10).
12
-------
SPECIFY
AVERAGING TIME
READ
IDATE, 1TIME, ISITE
OC, PC
COMPUTE
TIME INDEX
HAS
TIME - INDEX
CHANGED OR
END - OF - FILE BEEN
ENCOUNTERED
7
YES
YES
N(ISITE) = N(ISITE) + 1
O(ISITE) = O(ISITE) + OC
P(ISITE) = P(ISITE) + PC
YES
ARE
THERE
ENOUGH
VALUES
7
0 ISITE = O ISITE /N ISITE
P(ISITE) = PIISITE /N(ISITE)
WRITE
ISITE, O(ISITE), P(ISITE)
CHANGE
TIME INDEX
NO
Figure 3. General logic to compute multihour averages.
13
-------
The distribution of concentrations, observed and predicted, will
show whether it is worthwhile to proceed further with out analysis. If
the RAPS data base is to prove useful, the distribution must cover a
reasonable range of observed concentrations (preferably from zero through
levels above the air-quality standards) and a statistically sound sample
size of these concentrations. This distribution should be plotted for
all 13 sites to provide insight into the usefulness of the data base for
each site.
The line-printer histograms of the RAM input parameters indicate how
representative our parameter classification scheme is. (The analysis may
suggest that a different scheme might be more appropriate. For instance,
if 90 percent of the temperatures are contained in two categories we may
choose to assign narrower ranges to the categories.) The histograms will
also be useful at a later stage in our analysis when we are relating
residuals (difference in observed and predicted concentrations) to input-
parameter categories.
SELECTION AND APPLICATION OF THE STATISTICAL TESTS
The basic order of events in evaluating and improving the RAM is
• Apply the final-evaluation statistics
• Run the intermediate-evaluation statistics
• Interpret the results
• Run tests on subgroups
• Modify the model (RAM)
• Apply the final-evaluation statistics to the modified model
• Repeat the above process until the desired improvement is
achieved.
In the following subsections, we examine each of the above steps in more
detail.
14
-------
Application of the Intermediate-Evaluation Statistics
The final-evaluation statistics are the gauge by which we judge a
model's overall performance. Because our goal is to improve the RAM,
its current performance becomes the baseline from which to compare future
"improved" versions. Hence, the final-evaluation statistics should be
applied for each version of the model.
There are eight unique tests in our final-evaluation package.
Because the incremental (computer) cost per test is small compared to the
input-output costs, we suggest application of most tests with exceptions
as noted below. The following parameter values are suggested (Refer to
Appendix B of Volume 2):
• Test 1 - No user-specified parameters required.
• Test 2 - No user-specified parameters required.
• Test 3 - The absolute-error threshold, which is 6 = 10 ppb (or
26.1 ug/m3); this error threshold is consistent with instrument
accuracy determined in the quality-assurance program.
• Test 4 - The percentage-error threshold, which is g = 0.20.
Using the percentage error can be misleading if the data base is
biased toward lower concentrations, which is usually the case.
Therefore, after analyzing the RAPS data base, we may suggest
omitting this test.
3
• Test 5 - As a start, set the cutoff = 500 ppb (or 1,307.5 ug/m ),
which corresponds to the Federal Air Quality Standard averaged
over 3 hours. This may have to be modified downward if there are
not enough concentrations at or above 500 ppb (say, a minimum of
200 hourly averages per station).
• Test 6 - The same criteria (Test 5) for the cutoff value applies
here; arbitrarily we suggest a value of 1 for j^ and 2 for #2-
However, this test is very subjective and may not be too useful
in the RAM evaluation.
• Test 7 - Again, this is a very subjective test requiring the entry
of a 5 x 5 loss matrix. Arbitrarily, we recommend the symmetric
loss matrix shown in Appendix B of Volume 2.
• Test 8 - This loss function depends on the distance between the
maximum-observed and maximum-predicted locations. Therefore, we
recommend a 13 X 13 loss matrix consisting of the distances
between the RAPS monitoring stations.
15
-------
In summary, we suggest conducting the final-evaluation tests for
each version of the RAM. For reasons stated above, Tests 4, 6, and 7
should be less valuable than the others.
Application of the Final Evaluation Statistics j
Each of the intermediate evaluation statistics can tell us something
different about the RAM's performance. Hence, at least for the first
application of intermediate statistics, it is a good idea to run all the
tests. In conjunction with the displays described on page 12, the
user now has a complete record of statistical figures-of-merit, associated
graphic displays and auxiliary displays.
The actual mechanics of running each test are documented in Volume
3. We now focus our attention on how to interpret and categorize the
results, prior to conducting more tests and subsequently modifying the
RAM.
INTERPRETATION OF RESULTS
There is some redundancy in our statistics package, which means that
conclusions about RAM performance may be arrived at in a number of dif-
ferent manners. When evaluating poor performance, we are looking for
one or more types of poor-performance indicators. For each such indica-
tor, there are one or more statistical tests that will tell us something
about the RAM performance. These indicators of poor performance include:
• General bias—if the residuals between observed and predicted
concentrations are mostly of the same magnitude and sign, then
we have a general bias. General bias will be graphically dis-
played by the following:
- Logarithm-frequency plots of residuals or observed and pre-
dicted concentrations on the same plots. These plots for the
test data base are shown in Figures 4 and 5, respectively.
From Figure 4 we note, for the 10-percent point, that the dif-
ference is less than a factor of 2. Thus, for over 90 percent
of the time the agreement between observation and prediction
is within a factor of 2. Figure 5 presents another view of
the above comparison by displaying the distribution of the
residual concentration (magnitude and sign).
16
-------
FREQUENCY DISTRIBUTION FOR OBSERVED AND PREDICTED CONCENTRATION
*__
mf m*9 m* •».« ttj *J m
Q
BSI
REI
RVI
1C
EO
CUMULATIVE FREQUENCY IX)
Figure 4. Frequency distribution for test data base.
FKEQt.
*
t-
§
z **
o ""
g ^
5 -
a
i ^
IEN
».
CY I
• M.
)IS
• w-
TRIt
-
IUT10
m
N FOI
t RES
^*«t
«
.. ..
OUR
•s.
L II
f
""
IBS
M_
1
1
:RV
/
«.t •".* l.» I4 ».Q 10.* M.O ».••*. 0*0.* M.
CUHULRT1VE FREQUENCY 1X1
ED-I
>
• i*.
•RED
1
,rf*"
r
ICTED
: _j!
^
) C0^
•
M
BO
4CENT
• "
UNDE
R«T
0 1
"
«ts
ION
• i
—5
in
1 0.
tltl
• M.I m.a m.am-a».» t».i
11 '•»'?•
ii>«*trr
t 0
i
Figure 5. Residual frequency distribution for test data base.
17
-------
- Chi-square test. The graphic output of this test closely par-
allels that of the log-frequency plots above, but is not as
effective in displaying bias.
- Bivariate regression and correlation test. Two of the plots
resulting from application to the test data base are presented
in Figures 6 and 7. In Figure 6, a comparison between the
least-squares regression line and "ideal" agreement line pro-
vides perhaps the most valuable indicator of bias. Bias is
indicated if the difference is large between the two. The 95-
percent confidence level shown in Figure 7 should be analyzed
prior to drawing any conclusions.
• Spatial bias—is similar to general bias, but differs in that it
focuses on the spatial distribution. Mainly, we wish to deter-
mine if there is a spatial pattern to bias. While clever appli-
cation of other tests could serve the purpose, the interstation
error correlation test specifically addresses spatial bias. Also,
the graphical output feature from the accuracy score tests pro-
vides a visual correlation for "spatial" performance.
• Temporal bias—the residual time series test is the only test
specifically designed to provide a direct indicator of temporal
(or cyclic) bias.
• Randomness—the -bivariate regression and correlation test provides
the best indicators. A low Pearson's correlation coefficient
summarizes "randomness", and the scatterplot provides a visual
indication.
• Input-parameter dependence—the multiple regression of error
residuals test specifically assesses the effect of each input
parameter using an SSPS procedure.3 In addition, it ranks each
parameter's relative contribution to the error residual. Other
statistical tests will also be used for this purpose. However,
these other tests must first be applied to subsets of the data
base that are categorized by input parameters; then comparisons
among categorized results are made. We discuss this further
later in this section.
The above breakdown summarizes what is initially looked for in performance
indicators.
Up to now we have been concentrating on application of the tests to
the entire data base. The preceding paragraph discusses the general types
of interpretation that lend themselves to our model evaluation process.
In the next subsection, we describe what actions can be taken after this
interpretation phase. First, however, we will look at each test individ-
ually and discuss how to interpret the results. (Refer to Volume 2 for
a more comprehensive description.)
18
-------
,.:riE*P REGRESSION ANALYSIS, C«SE 2
— UNITS (ie»*-6 >»'PU«-a J
•LOTTtO
Z
O
a
*
I
o
o
o
660
500
400
300
aee
100
100 800 300 400
OBSERVED CONCENTRATION
500
600
Figure 6. Actual versus ideal fit for test data base.
REiPESSlON HNHLVSIS, CnSE 6
(CHIUU/0) -- UNITS >'14»*-6U(IU*-2)
PLOTTtD
eee
see
300
o
o
100
• .n
I • «T.
• • .IT
100 200 300 400
OBSERVED CONCENTRATION
500
... . IIITV ...
U IMM - M* OQMTtKMCf MU
It M« - liMT MHU \M
Figure 7. Scattergram with bounds for the test data base.
19
-------
Accuracy
The output of each accuracy score test is a number that becomes
lower as agreement between observed and predicted concentrations becomes
better. A confidence interval is also computed. A narrow range about the
accuracy score is desired; the narrower the range, the more confidence we
have in the accuracy score computation.
Residual Time Series
Sample outputs for the residual time series test are given in Fig-
ures 8 and 9. The autocorrelation plot is examined to see if there is a
large cyclic component in the error residual. The normalized cumulative
periodogram yields much the same information but provides a cumulative
display. The autocorrelation function is somewhat easier to interpret.
Here one looks for the highest "spike" and, if it is above the 95-percent
confidence line, one can assume that the model performance is significantly
poorer for that time lag. In our example (Figure 8), we see that the
autocorrelation is greatest for a time lag of 1.
Chi-Square Goodness-of-Fit
The chi-square goodness-of-fit compares the distribution of observed
and predicted concentrations (example outputs are given in Volume 3).
As noted before, the logarithm-frequency plots provide the same graphical
2
information in much the same form. The X statistic summarizes the over-
all agreement of the two distributions in a single figure of merit. Like
o
the accuracy score tests, a lower X corresponds to a better agreement.
Bivariate Regression and Correlati^on
The bivariate regression and correlation test is perhaps one of the
most useful of our intermediate evaluation statistics. The output plots
contain many useful visual displays to go along with useful numeric indi-
cators. Individual features are described below:
• Pearson's correlation coefficient is an indicator of the linear-
ity between predicted and observed concentrations. As such, it
20
-------
TIHE SEPIES
1 .0
0.5
0.0
RUTOCORRELflriO
i
» «
» In
9.
, c«
-
-
—
ee
SE 1. STATION
1 1
1 1
2.00 4.00
TIHE LA
3
6.
G
00
8.
1
1
00 10.
00
1
1
12.00
1 .
1
14.00 16.
•LOTTED
l«'«S/71
HHJill
00
Figure 8. Autocorrelation function of test data base.
J'E SERIES, CftSE £, STrtTIOH 1
1.0
PlOTTtD
1»'«S/7
1BI23I1I
.- DftSHED LINES: 95X CONFIDENCE LIPIITS
I I I I I I I I I I I 1 I I I I I '"l I I I I I
0.0
0.0
0.1 0.2
FREQUENCY (F)
0.3
0.4
0.S
Figure 9. Normalized cumulative periodogram for test data base.
21
-------
becomes an invaluable detector of randomness. The highest score,
unity, basically rules out a random lack of agreement, while the
low score, zero, indicates a completely random lack of agreement.
The scatterplot is a visual display of the point-by-point compar-
ison of predicted versus observed concentrations. It is best
used with an auxiliary regression analysis.
Regression analysis is another indicator of general bias. The
slope and intercept of the least-squares line numerically quanti-
fies the bias. As shown in Figure 6, the agreement between the
least-squares (bias) line and the ideal (no bias) line can be
visually compared on the scatterplot. The 90-percent confidence
bounds (Figure 7) can also be displayed. These bounds tell the
viewer that the real least-squares line will fall within the
designated bounds at a statistically significant level. The 90-
percent probability bounds tell the viewer that there are 9 chan-
ces out of 10 that a new point will fall within those designated
bounds.
User-supplied sensitivity bounds can also be entered. These sen-
sitivity bounds essentially establish a dead zone on the scatter-
plot. In the subsequent calculations, the regression coefficients
are based solely on the points falling outside the zone defined
by dashed lines on the scatterplot. Thus, we can determine a
trend in the so-called "outlying" comparison points.
Interstation Error Correlation
Pearson's correlation coefficient is computed for the residual con-
centrations between each pair of monitoring stations. If all correlations
are about the same, we can probably conclude that the model is erroneously
biasing the results in a uniform manner for all sites. If there is a
large variation in the correlation, then we should examine the results
for trends, such as differences between urban and rural sites, proximity
to known local meteorological or emission factors, and so forth.
Multiple Regression of Error Residuals
The multiple regression of error residuals test is the only device
to explicitly evaluate model performance as a function of the input
parameters. (The other tests can be used for the same purpose but the
data base must first be subdivided by parameter.) For the multiple
regression test, essentially all model input parameters become independent
22
-------
variables in the linear multiple regression equation; the concentration
error residual is the dependent variable. The SPSS program computes the
coefficients in the regression equation, the multiple correlation coeffi-
cient, and the contribution to this coefficient by each parameter. Hence,
we have explicit indicators of which parameters contribute most to a
model's poor performance.
Let us consider our example, in which the multiple regression test
is applied to our test data base. The SPSS summary output for the test
data base is given in Figure 10. There are three independent variables:
• Atmospheric-stability class (ASC)
• Wind-direction class (WDC)
• Wind-speed class (WSC).
The summary table illustrates the multiple correlation coefficient
(MULTIPLE R) in the first column. The SPSS program arranged the param-
eters in the order of their contribution to MULTIPLE R. By far the most
substantial contribution comes from the ASC parameter, with a value of
0.62443. Going down the list, the contribution from each of the other
parameters is negligible because the MULTIPLE R increases only to
0.62586. For another way of looking at the results, note that the R
SQUARE column shows that the first parameter, ASC, explains about 39
percent of the variance in the residuals; the other two, WDC and WSC, do
not substantially account for more of the variance. The third and fourth
columns, RSQ CHANGE and SIMPLE R, give the incremental change to the
MULTIPLE R and the bivariate correlation coefficient, respectively.
The fifth column, B, gives the regression coefficients. Hence, the
linear equation becomes:
RESIDUAL = 487.89 - 125.27ASC + 5.21WDC + 2.23WDC + ^
where £.. is the residual or the number required to equate both sides of
the equation. However, the table shows that the contribution from the
23
-------
u z
-i S
•M
* in in
•> ft «4
« • N
i« a o
M88
o
u>
C
» 9> in
u « w m
_J f»J U) ^J
& w> KI r»
u
o *- r^ ^>
Z 9< • "I
5 Ssi
(0
13
U
a
a
§ «
wi n J J
LJ * J1 "3
to
E
to
d
£
O)
Z
LJ
a
z
u
o.
w
o
a!
«
n
or
u u u
^ Q V\ 1
24
-------
last two parameters is negligible. Therefore, as a good approximation,
our equation simplifies to:
RESIDUAL = 487.89 - 125.27ASC + €2
where £_ differs from €,.
Basically, we are being told that improvement in the ASC parameter
alone offers the potential for decreasing the variance in concentration
residuals. Therefore, our approach would be to improve our scheme in
estimating atmospheric stability (i.e., improve our model). On the other
hand, it appears that our parameterization of wind speed and wind direc-
tion is all right. Therefore, little is to be gained from improving the
model interface for wind speed and direction.
The summary table is only one part of the output resulting from the
multiple regression test. The remaining output and further description
of the SPSS results for the test data are given in Appendix A. In par-
ticular, we discuss statistical significance (the F statistic).
DIAGNOSING THE CAUSES OF POOR MODEL PERFORMANCE
The example of the previous subsection illustrates results that were
quite easy to interpret—namely, that atmospheric stability parameteriza-
tion was a primary contributor to poor model performance. The results
from the multiple regression tests will not always be so easy to inter-
pret. Instead, they will be used to augment results from other tests to
obtain a clear definition of the causes of poor model performance.
Our recommended approach in evaluating RAM is first to conduct any
test that will provide useful insight into the problem prior to attempt-
ing any improvements to the RAM. The rationale is simple—in contrast to
the model modification process, the statistical tests are inexpensive to
run. After any model modification, it is necessary to generate a new
test data base by running RAM using RAPS data for one year. This is a
very expensive process which should take place only after we have obtained
the maximum information from our diagnostic tests.
25
-------
Diagnosis of the causes of poof model performance will normally be
an iterative process, as shown in Figure 1 (Section 1). The iterative
diagnostic process is illustrated as OPTION 3. As stated earlier in Sec-
tion 2, we recommend application of all six tests to the entire data base
as a first step in the diagnostic process. The results are then analyzed
for our indicators of poor model performance: general bias, spatial bias,
temporal bias, and randomness (see Section 2). Next, we must use our
tests to relate these indicators to the causes of poor performance in
terms of RAM parameters. There is no predefined prescription of how to
proceed with our diagnosis. However, if we look at each of the indicators
separately, there are certain recommendations that can be made on how to
apply and interpret further tests.
In the following subsections, we address possible actions to explain
and diagnose indicators of poor model performance.
General Bias
General bias exists when predicted and observed concentrations differ
by a uniform amount (high or low) throughout the data base. This type of
bias can easily be diagnosed by several of our tests, as described ear-
lier. Using assumed displays from the bivariate regression and correla-
tion test, let us consider various ways bias can manifest itself by com-
paring the least-squares regression line and the ideal fit. Figure 11
shows three different types of overproduction bias. Of the three, the
second (diverging) is generally the most common in model evaluation.
(This happens because the general wind direction is usually known and
leads to a low concentration prediction to correspond with conditions when
the monitor was upwind of the source area and recording low concentra-
tions. For downwind conditions, the full diffusion equation comes into
play and small errors in formulation or input parameters lead to large
errors in concentration prediction.) The problem is, once we have
detected a bias, what action do we take to explain it?
26
-------
(a) CONSTANT OFFSET
(c) CONVERGING
(b) DIVERGING
Figure 11. Types of bias.
At this point we must go into more detail. The first question con-
cerns whether we are dealing with general bias or spatial bias, so the
next step is to run the bivariate regression and analysis test for each
monitoring site. If the comparisons between the ideal and least-squares
lines are essentially the same for each station, then we have general
bias. Next, we have to try to isolate the cause.
Results from other tests can help in diagnosing the causes of general
bias. In particular, the correlation of error residuals among monitoring
sites (interstation error correlation) can provide useful information.
If all correlations are high, we are being told that the overprediction
(or underprediction) generally exists for all stations at the same time.
This not only confirms our diagnosis of general bias but also tells us
that the causes of mispredictions are probably the same for all locations.
Next, we examine the results of the multiple regression of error residuals
test. This will tell us which RAM input parameter is most highly corre-
lated with the error residuals. For example, we might find that the error
residuals are inversely correlated to wind-speed class (WSC) or, as in
our test data base, atmospheric stability class (ASC).
27
-------
Unexplainable Biases--
At this point, let us depart from the above example and recognize
one limitation of our diagnostic tests for finding correlations between
RAM input parameters and error residuals. If a RAM input parameter
causes a high bias throughout the data base, none of our tests may help
us explain this. For instance, consider the relationship between pre-
dicted concentration (X) and emissions (Q) as follows:
X = kQ f(M)
where f(M) is the meteorological function and k is a constant. If Q is
biased high, then X will always be high if all other representations are
accurate. If Q is 25 percent high, then a constant 25-percent offset
would exist in our error residuals. Our tests are geared to correlate
error residual changes (variance) to input parameters. For a constant
25-percent offset in the error residuals, there would be no change and
hence no correlation.
It is unlikely that the above condition will exist in reality; emis-
sions will most likely be biased differently according to the type of
source. Hence, certain stations would exhibit a different bias from
others because they would be influenced by different sources.
Second Iteration--
After the detection of general bias and analysis of other results,
we will want to take a closer look at the dependence of error residuals
on RAM input parameters. As stated, the multiple regression test pro-
vides us with some indication of input parameter dependence. For general
bias, we will want to stratify the data base by classes of that parameter
which influences the error residual most (as determined by the multiple
regression of error residuals test).
Stratification of the data base can be accomplished by the
STAT06*SPSSSORT program (see Volume 3). In our previous example, where
the multiple regression test showed a high correlation between error
28
-------
residuals and wind-speed class, we might choose to conduct our evaluations
with the data base divided into low and high wind-speed categories. We
would then apply our SPSSSORT program twice and generate two files, one
for wind-speed classes of 1, 2, and 3, and one for classes 4, 5, and 6.
We can then apply our bivariate regression and correlation tests to each
file separately. If we find that one file shows good agreement while the
other shows poor performance, we have identified problems in the formula-
tion. Depending on how clear-cut the results are, we may wish to conduct
further tests or proceed to consider modifications.
It must be recognized that there is no preset prescription for diag-
nosing model performance, and that our description here is for illustra-
tive purposes only.
Spatial Bias
Probably the best initial test for spatial bias is the interstation
error correlation test. As stated in Section 2, the output correla-
tion matrix indicates the existence of a general bias if the correlation
coefficients among all stations are high (say, 0.5 or above). Conversely,
low correlation coefficients suggest the possibility of spatial bias.
However, further test results are needed to confirm our suspicion because
low correlation coefficients also result when any of the other indicators
(e.g., randomness) of poor model performance exist.
The map display of the accuracy score test can also lend insight
into spatial bias. If accuracy scores vary greatly from station to sta-
tion, we again suspect spatial bias, but need to confirm our suspicion.
Perhaps the best indicators for spatial bias are the bivariate correlation
and regression displays applied to each station. The test for site-
specific bias is then exactly the same as it was for general bias (see
Section 2).
For the RAM evaluation, we must establish the degree of spatial bias
for 13 stations. Next, our results must be categorized—namely, which
stations can be thought of as being similar in performance? Referring
-------
to our least-squares test versus the ideal fit (Figures 6 and 7), we can
broadly categorize stations according to actual and ideal agreement.
Results from the interstation error correlation test provide indicators
of which stations behave similarly. Hence, we have the means for esta-
blishing reasonable categorizations of stations with similar performance.
Using the resulting categorizations, we look for rather obvious dif-
ferences among locations, such as:
• Urban versus rural sites
• Eastern versus western sites
• Local influences (anomalous locations)
• Distribution in relation to large point-sources.
Furthermore, if we can group by similar stations, the next step is to use
our sorting program to establish data bases for these similar stations.
Then we apply the multiple regression of error residuals tests to these
smaller data bases and proceed as we described in Section 2 (General
Bias). It must be recognized, however, that each categorized data base
may lead to different conclusions and subsequent remedial actions.
Before concluding our description, we caution that it is possible to
have a situation in which certain stations exhibit spatial bias while
others do not; other sites, for instance, could exhibit completely random
but unbiased behavior.
Temporal Bias
The residual time series test is the only one capable of directly
detecting temporal bias. Either the autocorrelation function or the nor-
malized cumulative periodogram contains the same diagnostic information.
Consider the example for the autocorrelation function A(k) in Figure 12.
Temporal bias manifests itself when A(k) exceeds the 95-percent confidence
limits. In Figure 12, temporal bias is shown to occur every 5 hours.
30
-------
AUTOCORRELATION, A(K) ±
b
95-perce
nt limit
I
1 2 3
9&-nercent limit
45 678
TIME LAG (HOURS)
-1.0 |-
Figure 12. Temporal bias example.
Once detected, the cause of temporal bias must be diagnosed. We can
look for the obvious, such as weekend versus weekday (lags of 6 or 7
days), day versus night (lags of 12 hours) or daily differences (lags of
1 day). When obvious conclusions cannot be reached, we suggest conducting
tests for randomness.
Randomness
When randomness exists in the absence of bias (general or spatial),
we would expect large error residuals but fairly good agreement between
observed and predicted concentration distributions. Likewise, the least-
squares regression line could compare quite favorably with the ideal line;
however, the scatter by visual inspection would appear quite high and the
correlation coefficient would be low (about 0.6 or less).
After randomness is detected, we suggest analyzing those point out-
side the 90-percent probability bounds (or subjectively selected sensi-
tivity bounds) in conjunction with the results from our multiple regres-
sion of error residuals test, and with the results from the previous
analysis of our data base (Section 2). Basically, we wish to determine
which model input conditions exhibit the most influence on poor perfor-
mance. Therefore, by comparing the input parameter distributions for
31
-------
points outside the probability (or sensitivity) bounds with the input
parameter distributions for the entire data base, we can determine which
conditions most often lead to poor performance.
Let us consider the specific, step-by-step procedure along with an
example from our test data base to illustrate the recommendations in the
preceding paragraph:
Step 1: Conduct the bivariate regression and correlation test, plotting
either probability or sensitivity bounds on the scatter-plot.
For our example, consider those points outside the probability
bounds (see Figure 7).
Step 2: Determine some logical criteria for specifying the majority of
points falling outside the probability bounds. Referring to
Figure 7, we see that most of the outlying points occur when
predicted concentrations exceed 400 units and observed concen-
trations are below 300 units.
Step 3: Modify the SPSSSORT program so that it displays the input
parameters for those cases that meet the criteria established
in Step 2. For application to our test data base, we added the
following statements to our SPSSSORT program:
*SELECT IF (PC GT 4CO AND OC LT 300)
LIST CASES VARIABLES=ALL
FREQUENCIES INTEGER=WSC(1,6)
The first statement perfo*ins the sorting of the data base, given
our Step 2 criteria. LIST CASES then lists all the parameters
in the data base for those cases satisfying our criteria. The
FREQUENCIES statement plots a histogram of the desired input
parameter for the criteria case. A complete listing of program
and output is given in Appendix B.
Step 4: Analyze the Step 3 results and compare the frequency distribu-
tions with those obtained earlier for the whole data base.
Referring to our examples in Appendix B, we see that four cases
met our Step 2 criteria. All of these were for an atmospheric-
stability class (ASC) of 5--as shown on page 3 (Appendix B) of
output. Further, on page 4, we see that all 4 cases were in
the low wind-speed category. (Generally, we will be working
with frequency distributions for other input parameters. How-
ever, for 4 cases we can easily review the case-by-case
For our test data base, units are m * 10 ; for RAM, units will be
either |_ig m~^ or ppb.
32
-------
listings.) If we examine the atmospheric stability categories
(ASC) for the entire data base (as shown in Volume 3), we see
that the most frequent ASC is 4. Therefore, our results suggest
that our prediction mechanism is giving us the greatest error
when an ASC of 5 is input.
Step 5: Refine the analysis for the input parameter to which the model
is most sensitive. Basically, we would use the SPSSSORT pro-
gram to stratify our data base by that parameter. For our test
example, we would form a small data base for each ASC class.
Then we would apply our bivariate regression and correlation
test to each of these small data bases.
Step 6: Identify other sensitive input parameters using the above
steps. We wish to determine if our model is mispredicting
because of poor parameterizations for more than one input
parameter.
The above procedure is by no means unique and will not answer all our
questions about why the model is doing poorly in certain situations.
However, in concert with other tests, we have enough diagnostic tools to
quantify conditions leading to poor performance.
Input-Parameter Dependence
As stated above, the multiple regression of error residuals is the
only test that directly addresses input-parameter dependence. However,
as shown in the previous four subsections, other tests in conjunction
with SPSSSORT-stratified data bases can address the problem. The multi-
ple regression test itself can be applied to the stratified data bases.
The net result would then be a clearer indication of the second most
important parameter contributing to poor performance, i.e., high residuals.
COMBINATIONS OF POOR PERFORMANCE INDICATORS
Our previous discussion very neatly separated poor performance indi-
cators into categories. Now, it must be recognized that there may be
several indicators that must be treated. For instance, we could very
easily have high general bias and randomness at the same time. As a
further complication, the underlying causes may be the same or different.
33
-------
No matter how complex the problem appears to be, our tests can be
applied in the manner described in this text. As .stated before, there
is no set formula for applying a specific test at a certain point. Some-
times highly subjective decisions will enter into the process. Neverthe-
less, the tools of analysis remain the same and provide sufficient means
for assisting in the diagnosis of poor model performance.
34
-------
REFERENCES
1. A. D. Busse, and J. R. Zimmerman, "User's Guide to the Climatologi-
cal Dispersion Model," Publication EPA-R4-73-024, Environmental
Protection Agency, Research Triangle Park, North Carolina (1973).
2. F. L. Ludwig et al., "A Practical Multipurpose Urban Diffusion
Model for Carbon Monoxide," Final Report, SRI Project 7874, SRI
International, Menlo Park, California (1970).
3. N. H. Nie, C. H. Hull, J. G. Jenkins, K. Steinbrenner, D. H. Bent,
Statistical Package for the Social Sciences (McGraw-Hill Book Co.,
New York, New York, 1975).
4. F. Smith and R. Strong, "Independent Performance Audit #4," Interim
Report, EPA Contract 68-02-2407, Research Triangle Institute Project
43U-1291-1, Research Triangle Park, North Carolina (1977).
35
-------
APPENDIX A
SPSS MULTIPLE REGRESSION OUTPUT FOR TEST DATA BASE
SOURCE LISTING
Specification of the SPSS REGRESSION procedure is described in Vol-
ume III of this report. Referring to the first page of the program out-
put (located at the end of this appendix, the source listing, note in
particular the "(1)" in the field of the REGRESSION specification. This
makes the program order the input parameters as a function of their sig-
nificance or correlation with the error residual. Hence, the parameter
entered first (by the program) is the most highly correlated variable;
the second is the second most highly correlated variable, and so forth.
UNIVARIATE STATISTICS
The program when "ALL" statistics are specified automatically com-
putes some auxiliary statistics. This includes the mean, standard devi-
ation, and number of cases, as illustrated on the second page of our test
example.
BIVARIATE CORRELATIONS
Bivariate correlation coefficients are also printed out. See the
third page of the output listing.
STEP-BY-STEP PARAMETER ENTRIES
The step-by-step entry of variables and associated statistics is
illustrated on the fourth and firth pages of the sample output. In Table
A-l, definitions c-f a number of the output statistics are presented. Of
particular importance are the MULTIPLE R (as described in the test) and
36
-------
TABLE A-l. DEFINITION OF SELECTED REGRESSION STATISTICS'
Name
Description
MULTIPLE R
R SQUARE
ADJUSTED R SQUARE
STANDARD ERROR
DF
RESIDUAL
F
B
BETA
CONSTANT
Multiple correlation coefficient
Square of the correlation coefficient, which is
normally equated to the fraction of variance
of the dependent variable explained by the
independent variables
R SQUARE corrected by an SPSS procedure
according to the number of variables and cases
Standard error of estimate for the prediction
of the residual concentration
Degrees of freedom (or the number of independent
variables)
Number of cases - DF - 1
2
Significance indicator, which is the same as t
of student's t test. A table must be used to
equate the F value to "statistical significance"
or confidence levels. More formally,
F =
R SQUARE/DF
(1 - R SQUARE)/RESIDUAL
Coefficient of the independent variable in the
linear regression coefficient
Same as B except normalized to a unity value
for the dependent variable
Constant in the linear regression equation
* 3
Refer to Section 20 of the SPSS Manual for comprehensive
definitions of all parameters.
37
-------
F, which is a measure of statistical significance. The reader is referred
to the SPSS manual for a comprehensive description of all the statistics.
Examining the sample output, one can see that atmospheric stability
class (ASC) is entered in the first step and has a 0.62443 correlation
coefficient and a highly significant F value of 63.3. In step 2, wind-
direction class (WDC) is entered but fails to improve the correlation
coefficient appreciably while decreasing the significance (F = 31.5).
On the third step (fifth page), wind-speed class (WSC) is entered with
results even less significant than in the previous step.
SUMMARY TABLE
A summary table is printed on the last page of the output listing.
The input parameters are listed in the order of their importance.
38
-------
:
:
IA
X Ul
Ul M Ul OX *
s s i i
oo o £r:~ 2
-m -m 3 QC O »4 M
,™ • Q — Q
o ui in u 3 u 3
§ S 5 |5g d
* ** *" « m ui o * H- z
01 * I T •
M Ul M M O
51-33: •*
M O.
> IA Q QI IA
a . M R x
u IA N ui 4VtOO! O
O.QZ« •* O * W M M 3
Z (^ O M J f. »-IAZU _lu>L1
( O Z • ^- T£ * «UI JOtAlAi i4
• M o Li O •* J O -I * Ul UH >-4
z »- wuqo»-"» «"^ rf ^?, X. "*
a «* ».m-iakibi IAU, < x ij o _t
* u *_i _» ui a, a-n »- rfuiuij
o o x*aejh-u» u>3 * >aeo:*
n a j, Te»ci * "» a
M h. _* O • < Ul Ui Ul U>
i i/» »» u. w» a: ui juicui ui
it ui utcnx o ; H x * a. oe
3 Ul Z 3<»- « u. >- Z «A M
* d Ui -TOM O. Z IA 3
o u « a "til 16 (A o u 9
r.u,xa.!-x;''«UIM MMU.
f. 5>'«XJ *«: xzzz3u ui^-aJ
U « *- M OC O QC IA 3
ceo .j x x x i- oc
M -I -J 3 « * * J n
«ZX,Z3 *•
5 * O £ u O
K-Lll-C Ot»* • • • • H
a.o »
-------
LJ
O
U
o
U)
o
LJ o c a o o o
l/» *4 tH t-l «-l •-! r*
«r
U
u
o
«•
O
II
Ul
t-
«r
o
> kO U} <£
U to f 9-
O M
cr
«
o
cr>
tc
a-
u
09
Z M 3 (M "O i-l ^
a- O <£ to en en
u r~ CJ f- 10 to «
z: ui c; —< i-i c. ^~
^^ • • • • *
f N w M iv en
f H
C i
O
2:
tt
H
U1
Q-
CO
m
«r
M
K
u o ifl
U1 Q U
3 ai or
-------
Ul
o
to
o
M (M
en Q
o
H
V\
ia i- * M to a
u> M «M «M in e
N a u> ro to o
s
o> i-i «• (M O en
tr n t-i N o en
co O fw O o co
co » N CM c m
in a to f< a K!
I r* I
CO
3-
f>
C1
U rt O 3-
a M N
C» - M I
u> 3 a » u> '
• • • • •
l •-« i
•*.
3
e
z
o
Ul
K
tt
CL
v\
ui
0%
H Z
c z
• u
C> (J
O U. U.
H ou.
•r bl O
K
O
U
» C M «•! i-« (X
«o s m » *o r-
if> — •-« M a t-
M O "1 M » N
o a oc c c.
C 3- a- O CO (M
6aj in * a- h-
Lfj fl r-> CO u>
ui c N >4 r>
£ O C U> r4 tf> IM
O
o
Ul Ul
K I-
M H
I- O O
U t*
CO Q Ul
an * ee
41
-------
a-
Ul
co
^
Q,
CO
^»
g
w
a
r-l
.
O
z
z
^
cc
r-
l/l
Ul
««
CO
f-
^_
U
o
sr
a
II
Ul
o
1^1
5
•
*
ft
•
«
•
«
•
ft
*
•
•
•
z
0
M
Ul
in
UJ
CC
CO
Ul
cc
Ul
~*
a.
M
r-
_J
3
£
w
ft 1^
*T
ft 1
if
ft 3
O
ft M •
LI *
ft Ul -1
CC
4|
cc
ft Ul
CO
ft X
3
• a z
H
ft l/> Q.
Lw Ul
« CC T-
Li
z
• 0
* • o
UJ C
ft -J CC
CD Ul
• «r r~
M Z
* CC Ul
ft > —
10
• r— ••
Z Ul
• U J
o co
• z *
Ul H
ft 0. CC
Ul <
ft O >
1
3 1
U. "-I 1
r- 1
'** 1
• 1
•*> 1
ui !
1
1
a
c
Ul r*
5 H
•r in to «
3 (SI S S
a •— * c
I/I en i-l u
_ KI r-
z • • u
«r a- i- 3
u o en P
r M a-
r-l fO 2
(M I-
CSI
r-
c
3
u
* L
ui in u>
cc (si en c
«• r~ c c
3 en co t-
O» rO »•• Q
1.1 •• «
a- M a
L- C I-
0 K) (M
r-l U}
z INI a
3 M f)
LI
• •
u. M en
o en
Ul
z"
«r
CC
•r
>
U.
0 Z
o
I/I M -J
t-4 t/t «r
Ul I/I 3
Sfe-S
«f O H
Z UJ Ui
«r CC c:
i
1-
»•
•
t
L
(.
t
*
IO i-! IO O I
a- 01 f- ar I
tf Cn "! r-4
ig o? eo a (
_J ~1 "1 r-l .
' "E i
Ul
a
^
3 cr
a g
l/> or
a
cc cc LJ
ui ui a a
_i c. ui re
6- i f- «r
H 3 LO D
t- to 3 Z
_l Lfl -J <
3 Q I-
E cr «» L^
00
N
P«
U. •
B a
: Z r-l
) -1 »-
< K r*
» Ul ft
; =i
5 *•
j
: -i to
•f 1-1
M CD
: i- to
< 5 °
0.
>
1
J
j z a-
3 H t-<
r N
H * M
; r- °
• UJ •
» CO
u
t-
M
CC U
«r in
> ->
.
•-i
U, 1-
CM
.
IO
IX)
CO
CC «— i
o c«
CC f~
a N-
Ul ft
: «
3 Q SI
4 t- «-«
I/)
r
3
"5
u
J — ' to
C r- »
ui a-
CO «e —
*
•
*
«
*
*
«
*
•
*
«
•
n
*
•
«
•
ft
t
•
*
ft
*
*
•
*
«
•
•
#
•
*
»
*
*
ft
*
*
«
*
»
*
*
•
•
•
•
•
•
«
«
•
•
•
•
ft
•
*
*
•
«
»
ft
•
u
0
3
.
t
M
CC
Ul
CO
i:
z
£_
Ul
^-v
10
z
o
s
K
Ul
r-
Z
Ul
M
(/I
^
U
CO
^
M
K
>
i
S 1
**• S 1
in i
• 1
1
i
1
1
z
0
Ul M
CC (-
«r ^
3 - 3
o en o p
v/i LO en C
z". 1 UJ
«r a- in i
ui in (M r-
sc a m
fl f> Z
•-< H
t-
O
Z
tl
I/I UJ
LJ r- » _l
CC IO CO CO
*r en co «r
3 ro M H
o -1 ID «
LI • • «r
en co >
u c 10
o H a- i
(M in 1
r tv a-
3 N M
1/1 1
1
1
• « t
U. M CO 1
o en i
Ul
u
z
*
cc
«* 1
> 1
1
u.
0 Z
o
LO M _l
W l/» «r |
1/11/13 1
>• u) Q 1
-J CC M 1
«r (!) H 1
Z Ul Ul 1
«f cc a
i
i
i
r
c
i-
«
n
^
u
2
1-
7.
^-
uJ n «-i «-i f
in ro en fo L
lA r^ CO 'O „
CM cn f- h- 5
• I • t r-
ffi S
s
Ul 1
K
&
3 Ct
rt o
u> cc
n:
K or ui
u ui ° o
j o* w a
O- -<• r- «
H 3 LI O
t- C5 3 Z
3 a »-
_c ec «t i/i
(0
in
o
_. •
y "•
z en
«f H
cc r-
UJ •
d
i-
-j m
~ Ul
M 8-
1- N
5 •?
a.
z c-
H in
M
tf (SI
r- 0
Ul •
m
UJ
J
•r
S CJ
•f 1.1
> *
r- oo
u» in rvj
tM (M
• t
CM
a-
m
CC. fO -O
O eH CO
C. tt 3
pr ^ -^
Lj P~ *£
t •
O O"t CM
^- rH r-l
fl
J •» wC r-l
; (_ K) r-l
LJ sr CO
ea m a*
us 2
4 • •
1
1
J
4 f-
1 (SI
r m r- r-i
4 co :n co
m P M •
tC.c.
cn LO un
C>J
tH
1
1
•*
i ^-
Ul Z
, -1 •»
03 t-
•r LO
HI Z
CC O L) O
•t LI O O
> < -c *-
42
-------
.J 0
SB
«r ce
> 10
UI
a
1 U
z z
o *
H a
H- UI
3 d
UI
u
« in cn
ui a in
x .-i in
w 2
H- O.
«
r-
z
o
ui
£'
a- i-
«r (M Ifl
3 K) O-
cs r» »
i/i • •
r- c
U- t-l U*
o t*> Is*
CM UO
•c M a-
3 M KI
ui
13
m M
H •»
5 S
> m
UI
UI
UI
Ce
O
UI
1I-
cn
r-t
.
o
z
z
3
ce
^.
i/i
UI
^^
CO
r-
u
o
*
o
II
UI
K
u.
o z
o
t/1 M-l
MI/I*
in i/> 3
>- IU Q
_l a M
«r O UI
Z LJ UI
•r ce ce
i
ifl
H to oo
cn in in
•4 <-l O
t • t
01
fM
Q eg M
cr m r-
» 01 (M
CO C «M
H r-l (SI
• • •
PO *o cn
(M •-*
a> h- r*
* in u>
P" 1/1
ui in O
tf O PO
M (M
•*
u
K
a.
UI
tA
X
3
Z
M
X
^
X
43
-------
«4 N
•>> » <*
«o o e
• • •
I
o» ^ in
in in a
at •
"• to
o
o
« K) 01 N
» o» in
U # CO W
_| M IP (SI
fib IA ro 10
z
o
ca •- N r-
z w » w
01 «4 O
5 ™
K
a i/i o,
44
-------
APPENDIX B
PARTIAL SAMPLE OF OUTPUT FOR SPSSSORT PROGRAM
45
-------
t>
VI
VI
»
u
o
CL
O
a
o
Ci
en
in
o
a
o
1/1
a
K
O
o o
in O
w
_l
m
w
i-
o
(M
•
X
»>-
•
O
•
c
o
ct
Ul
CL
I/I
z
o
$
ui
a
o
z
o
I-
I-
01
01
K
3
(£
a
o
u
o
t/l
Ul
Ul
oc
U
u
•r
(E
LA o *
Ul O U
I- 3 -J O
?*« d
CE > O *
O •
but x ui
ox u
Z O I- «r
« O H O.
£^S !S
oc
b. U. U. O
000 »
«
o a.
U
oc
H U>
* u
O -J
Ul W
to u.
a. a
i/i p
U. 1^1
O UJ
(SI U
• Q.
O •
— « o
» * o
u. »
5* ~ °
UJ O Q
• 2
O UJ .
» « N to
Q.
V>
LJ Ul
X »-
3 U
OC CD
K Ik U
3 M UJ
a. •
H »- O
SO
W Ul
5
CL
Ul
Ul
u
a.
in
U.
U
O
ta
1/1
1/1 i/i
a. in
v\ o,
a n
46
-------
UJ
ca ui o
ui a. ui M M
i Ul
i a
O 3 O VI O Ul
'1 Ul -J —
* Ul Ul
o oc z «• ^J ce ui
HI ui o z * o u
o 3 a ae » « vi
o o • a ib bt ui vi
-J i" z u, ui x z « r> a> on c
S2 5 tf - &
«a Q •• W
a o
47
-------
u
o
•*
a.
00
h«
t-
^
_l
i
3
O
a
u
t-
i/i
3
->
a
*
u.
-»
a
*
>
u
z
Ul
i
Ul
ae
Ul
u
ac
Ul
0.
^
—
*-
Z
Ul
u
a
u
a.
•
in
N
o
*
in
(M
t
a
a
.-*
o
•
in
r-
ui >• —
Q
p>
o
Z
*
Ul
ae
u
I
«
K
^g
a
*-
*
d
et
u
z
Ul
3
O
Ul
a
u.
Ul
t-
3
J
O
v»
a
»-
Ul
u
cc.
Ul
Q.
««
>-
U
z
Ul
3
O
Ul
o
. •
in
(M
•4
a
•
in
h-
ro
l
• a
i •
f o
3
1
1
1 *
1
1
I/I
a.
1/1
i
ui
-e
ui
a
3
O
O
ul
48
-------
UJ
oc
u
a
irt
1/1
a.
1/1
ID
z
M M
O
ou.
>
o
z
a
o
u
u)
f,
u
a
49
-------
oo ON g m or*
O » O ut OL> OO
01 N ON .4** ••«
u
O Ul Ul Ul UJ
« x r x x
& MO MO MO HU
HO HO HO HO
CO •• •• ••
c- m M mm in i»>
«4 * r4
> e o o
O o* f4 M
•x. * » »
r» N r-
o
ui u u ut
HO HU HU HO
•r a * a «o * a
as ax a a a 3
a • a* a* o •
ON ON ~~ '
ON- ON QN O
§ § § §
H H H H
uio So ui
-------
TECHNICAL REPORT DATA
(Please read Instructions on the reverse before completing)
1 REPORT NO.
EPA-600/4-80-013d
2.
3. RECIPIENT'S ACCESSION-NO.
4. TITLE AND SUBTITLE
EVALUATION OF THE REAL-TIME AIR-QUALITY MODEL USING
THE RAPS DATA BASE
Volume 4. Evaluation Guide
5. REPORT DATE
6. PERFORMING ORGANIZATION CODE
February 1980
7 AUTHOR(S)
Ronald E. Ruff
8. PERFORMING ORGANIZATION REPORT NO
Final Report
SRI Project 6868
9. PERFORMING ORGANIZATION NAME AND ADDRESS
SRI International
333 Ravenswood Avenue
Menlo Park, California 94025
10. PROGRAM ELEMENT NO.
1AA603 AA-26 (FY-77)
11. CONTRACT/GRANT NO.
68-02-2770
12. SPONSORING AGENCY NAME AND ADDRESS
Environmental Sciences Research Laboratory — RTF, NC
Office of Research and Development
U.S. Environmental Protection Agency
Research Triangle Park, North Carolina 27711
13. TYPE OF REPORT AND PERIOD COVERED
FINAL 8/77-4/79
14. SPONSORING AGENCY CODE
EPA/600/09
15. SUPPLEMENTARY NOTES
16. ABSTRACT
The theory and programming of statistical tests for evaluating the Real-Time Air-Quality Model (RAM) using the
Regional Air Pollution Study (RAPS) data base are fully documented in four volumes. Moreover, the tests are
generally applicable to other model evaluation problems. Volume 4 discusses the application and interpretation
of the statistical programs, particularly with regard to use on the RAM. In general, there is no set procedure for
evaluating an air-quality model because of the different reasons for evaluating models and many subjective
decisions to be made during the process. However, guidelines are presented to cover a wide variety of evaluation
needs, with attention to data preparation, classification, analysis, selection and application of tests, and interpre-
tation of results. Several methods of diagnosing causes of poor model performance are discussed and some
sample program outputs are also provided.
17.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b. IDENTIFIERS/OPEN ENDED TERMS C. COSATI Field/Group
* Air Pollution
* Mathematical models
* Evaluation
* Tests
* Computer systems programs
-* Statistical tests
Real-Time Air-Quality Model
Regional Air Pollution Study
Data Base
13B
12A
14B
09B
13. DISTRIBUTION STATEMENT
RELEASE TO PUBLIC
19. SECURITY CLASS {ThisReport)
UNCLASSIFIED
21. NO. OF PAGES
57
20. SECURITY CLASS (Thispage)
UNCLASSIFIED
22. PRICE
EPA Form 2220-1 (9-73)
51
------- |