Method for Calculating Dispersion Modeling Uncertainty Applied to the Regulation of an Emission Source

Final Report

A METHOD FOR CALCULATING DISPERSION
MODELTNS UNCERTAINTY APPLIED TO THE
REGULATION OF AN EMISSION SOURCE

SYSAPP-85/007

8 January 19S5
Prepared for

v,S, Environmental Protect ton Agency
Offict of Air Quality Planning and S*.a«tar
-------
ACKNQM.EOGKENTS
tit wish to acknowledge the valuable contributions of Nr. Joseph TUvart.
and Hi*. Bill Cox of the Source Receptor Analysis Branch, Monitoring and
Data Analysis Division, Office of Air Quality Planning and Standards* U.S
Environmental Prelection ftgerscy who were very helpful in formulating tha
hypothetical example and the contents of our report, lie also wisn to
thank Marianne Qudik, Mfthra Moezzi, Paul us Irpan, and Bellt
of %sten$ Applications for their nelp in creating th*
-------
DISCLAIMER
Although this report has been funded by tilt U.S. Environmental Protection
Agency through Contract No. 68-02*3870, it nas not been subject to the
agency's peer and administrative review. Therefore, it does not
necessarily reflect the views of the agency, and r,o official endorsement
should be Inferred.
-------
ABSTRAH
A method for quantifying the uncertainty in 'iisponsion model predictions
If used to address three issues concerning node!-based demon s t rat-.i".* of
the attainment of national Ambient Air Qiality Standards (NAAQS): (1)
operational model performance, (i) the probability of tiAAQS atta4nr«nt»
and (3) setting emission limits for an emission sowrct. To illustrate the
method, the use of a dispersion model to demonstrate attainment of the
24-hour SD NAAQS near i 1300 NW coal-fired power plant is considered.
iv
-------
CONTENTS

Acknowl edqment s. i i
Disclaimer lii
Abstract iv
List of Illustrations. vi
List of Tables. vii
1 ~ JDUCTION. 1
2 CALCULATION OF UNCERTAINTY ESTIHATES (CUE) 3
3 AN EXAMPLE.. 6
Model Evaluation Results 6
CUE'S Use Of Model Evaluation Results 9
Characterizing Model Performance...... 11
Estimating the Prob*oilitv of NAAQS Af-ai«s*nt
Associated with a 6*»en Emission Rate.... 12
Estimating the ProbaMmy of Attainment Associated
With Current Practice Emission Limits 18
Estimating the Emission Rate Associated With a
Desired Probability of NAA0S Attainment 23
The Effects of Mcdel Bias and Imprecision............ 27
4 SUMMARY AND CONCLUSIONS 30
Summary of Results, 30
Conclusions....... 32
References 34
Appendix A: DETAILED DESCRIPTION OF CUE 35
Appendix 8: APPLICATION OF CUE TO BIASED OR
IMPRECISE MODELS.. 60
-------
ILLUSTRATIONS

1 Illustration of Calculation cf Uncertainty Estimates (CUE) ...... 4

2 Clifty Creek 24-hour average model evaluation results... ........ 10

3 Cumulative distribution of 8» th<» ratio of monitorea to
predicted design values... ................. ....... ..... ... ...... 13

4 Cumulative probability distributions for 1973-1977 mortitoriny
design values, assunino a 15,625 g/s annual average rate of
emission... .............. ....... .............. . ...... . ......... . 14

5 Box plots depicting prooability distributions for monitoring
design values, assuming a 15,625 g/s annual average rate of
*............... ....... . ................... ... ...... .. 16
6 Cuwrtttivt probability distributions for IS73-137? monitoring
design values, assuring the average rate of emission in eaen
year is equal to the emission lisrit set by current practice..... 19

7 Comparison of emission limits obtained via current practice
•si Si! sasitslsi* llsits required ts sefeiev^ 55 SR£ SG perccfst
probabilities of NAAQS attainment 25
-------
TABLES
1 Eirlssicn Units set by current practice.
2 Probabilities of NAAQS attainment associated with the given
emission rate and with emission Units set by current practice.. 21

3 Comparison of eoission limits set by current practice and those
calculated by CUE to yield a desired probability of NAAQS
attainment 24

4 \973 eraission limits set by CUE and designed to achieve a 30
percent probability of NAAQS attainment for different scenarios
of node! bias and precision....*....... 28
-------
INTRODUCTION
contract to the U,S» Environmental Protection Agency, Systems Appli-
cations, Inc. has been developing methods for quantifying uncertainty in
dispersion model results. During the past year, a setiKxJ for the Calcula-
tion of Uncertainty Estimates (CUE) was developed. CUE utilizes compari-
sons of monitored (i.e., observed) concentrations with corresponding model
predictions, together with a new statistical technique known as the
bootstrap method (Efron, 1982), to calculate uncertainty in model
predictions.

In particular, CUE is used to estimate the degree of uncertainty in tnoaet-
predictec design-value concentrations** In applications demonstrating
attainment of national Aabient Air Quality Standards (NAAQS), ground-level
concentrations a*c calculated at selected points (called receptors} in the
vicinity of emitting sources. The design-value concentration is the maxi-
cun over til ftctptors of tfie second-highest concentrations occurring
during a year. This quantity is often referred to as the "highest second
high" concentration. When the highest second high concentration does not
exceed the level prescribed by the KMQS, the area is considered to be in
attainment of the standard, When design values are available for more
than one year, the largest of the available values is compared to the
NAAOS.f
The term "design-value concentrations" refers to concentrations used
in the design of control strategies or derivation of emission limits.
A different procedure is used fv establishing attainment of the ozone
standard, or i*en backgroum, cwcent rat ions are significant.
-------
In this report we illustrate the application of the CUE method to a
typical -NAAQS attainment demonstration for a conventional ste;.n power
plant. The effects of uncertainty 1" model predictions on the attainment/
nonattainment decision 1s discussed quantitatively. Techniques are
presented for Incorporating model uncertainty Into model-based attainment
decision making. In the course of Illustrating the CUE method we explore
three Issues frequently raisec ay parties Interested in tne use of
dispersion models to demonstrate NAAQS attainment:

(1) Oaerati onal code! serf oraance : Mnat is the llitslinocd that tne
value predicted fcy the roiel Is higner or lower tnan tne
sonlteriaj design value, I.e., tne ceslgn value wa
otherwise ostain fror; s»«$ure^ents?
(2} a-opaS-llity of xa.^s attainnertt: WJtet is the liselinooe t.tit
the uR«tRO«ir poftitcrlrs; design val-8 is htcrter ;r Iswer rr.ar tne
RAAQS?

(S) Setting tni_sstcn. 'Islts: Wiat IsveT of eaissiORS should be
ptmltted froB a source to atn*$* the Hs< of violating tne
CS »*f1e ivs"idi!*s overt>* strt^cert trstssioR iiasi tat i acts?*
Here ** only jwwisie a procesure for calculating an emission lisit fc?
a prescribed acceptafele rt$n of nonattainnertt. i*e do not discuss or
rfwt such an acctptaele ris* migrtt be.
-------
2 CALCULATION OF UNCERTAINTY ESTIMATES (CUE)
The term "modeling uncert?*nty* can be defined ir. tnaiw ways. He define
•odellng uncertainty as the discrepancy between design-value concentra-
tions predicted by a dispersion node! and the corresponding design-value
concentrations obtained from ground-level ambient monitoring
instruments,* Under this definition, tne estimation of modeling uncer-
ttint> is equivalent to the determination of the distribution or likely
range for the unknown monitoring design value, i.e., the design-value
concentration that would have been observed !ta
-------
Meteorological and
emissions data
Precieted ground-level
concentrations ever a dense
network of receptors (moril-
toring data unavailable)
Known discrepancies
(e.?., from past node!
•valuxfion studiss)
between the «onitore
-------
design values predicted b> the dispersion model, fir example application
of the CUE method Is described in Section 3. We use this example to
address the t*iree issues raised in the introduction.
-------
AN EXAMPLE
MODEL EVALUATION RESULTS

To Illustrate the application of CUE we consider the use of a dispersion
model to demonstrate attainment of the 24-hour S02 NAAQS at the Ciifty
"reek power plant near Hedison, Indiana (see Stoeckenius et .»!., 1983,
1SB4). The Clifty Creek plant operates at an annual average emission rate
of about 8400 g/s (see Appendix A for further details), out for the pur-
poses of this report we assume a much higher hypothetical rate (15,625
g/s) to provide an example if a pe»ter plant that produces peak 24-nour
average impact levels close to the S^ primary HAAQS (365 ug/m3}.

Meteorological data collected at Cincinnati (surface) and Dayton (upper
tlr) for 1973 through 1977 were used in exercizing the MFFER dispersion
•ode! (Pi»rce and Turner, 19%). A polar grid of 130 receptors located at
10* Intervals along five concentric rings centered on the source at radial
distances of 1, 3, 6, 12, and 20 te was specified. The predicted 24-hour
«»ci iff «j< uunu—iCYti iinivc"kt ewTwtia "ti t uacu tu uuiatn tnc
second-high concentration (design value) 1n each year. A comparison of
these values with the 24-hour S02 NAAQS shows that the plant with this
enission rate would b* in violation of the standard in one of these five
years (1976). Based on the design value for each Individual year, or the
five-year Mximw design value, an emission limit could be calculated by
current procedures (see Table 1).

Since the nodel predictions are not perfectly accurate, i.e., they contain
uncertainties, a decision Baker sight wonder wheth«r it is really neces-
sary to designate the area as nenattainnent and revise the plant's
emission limit to pliwinate possible violations o"" the ambient standard.
-------
TABLE 1. Emission Units set by current practice. (See note}
Year
1973
197*
1S75
1976
1977
1973-77
Given Emission
Rate (g/s)
15,625
15,625
15,625
15,625
15,625
15,625
Highest
Second-High
Concentration
357
313
337
367
235
367*
24-Hour S02
NAAQS (wg/a3)
3€i
3b~
365
36S
355
355
Calculated
Emission
Limit
15,975
18.2E1
16,923
15,498
15,941
15.49B
Note
0 , where

calculated emission lisrit
Q0 * given mission rtte, f.t.» tfte hypothetic*! emissfon rite of
15,625 9/s used In the example
QV * highest seconO-hich S% concentration (ag/m3) predicted by the
dispersion M»tft« {taonn *s the 'design value"), I.e., the maximum
ftf the second-highest SQg ctjacentrttfots predietea «t
receptor. (For purposes of illustration, background SQ2
concentrations are assjeted to be
For the 1973-77 period, the design value concentration is the maximum of the
five annual design value concentrations.
-------
He might consider dismissing the 1976 model prediction as an aberration
and consider the plant to be in "attainment." To resolve this dilemma the
decision maker may wish to have an estimate of what the observed highest
second-high concentration would have been in each year if monitoring data
had been available at the 180 receptor sites. This is the issue of
operational model performance, i.e., the agreement between predicted and
monitored design values.

Mere monitoring data available at each of the 180 receptor sites and were
the monitoring design value less than the SAAQS reference concentration of
365 ug/m^ *n each of the five years, then the decision maker would have a
f*rm basis for tfeterstinirg the NAAQS to have been attained and emission
reductions to be unnecessary. In the absence of this dense network of
monitors, CUE provides estimates of the monitoring design value. We can
al»o use CUE to estimate the probability that the unknown design value
would have exceeded the NAAQS reference concent ration of 365 u9/ni3 in each
of the five years.

In applying CUE to this situation, we use a model evaluation study carried
out at Clifty Creek during 1975 (Wills et a1.» l§80; cf. Londergan et al.,
1983). In this Study, monitoring data and NPTER model results at six
locations near the plant were used to obtain monitored and predicted
highest second-high concentrations for the year. These values are mucn
network (Table 1) because they were obtained under the plant's actual
average 19*5 emission rate of §/s. The actu.tl average emission rate
is usea so that a useful comparison with monitored concentrations can be
made. Since ground-' avsl concentrations can te reliably assumed to be
linearly proportional to emission rates, model performance as measured by
the rat^o of monltared-to-predieted concentrations is unaffected. Thus,
model evaluation results fcr the 24-hour average concentrations obtained
In this example we a.*e intentionally excluding treatment of the
stochastic (variable) nature of the sulfur content of coal. This
exclusion is a temporary sisjplificatioft and is not a limitation of CUE.
-------
in 1975 under the lower, actual average emission rate can be reliably
applied to model predictions obtained for the larger emission rate since
we are considering only the ratio of monitored-to-predicted 24-hour
average concentrations.*

Talcing the ratio of the monitored highest second-high concentration and
corresponding predicted highest second-high concentration, we see from
Figure 2 that the model prediction was 28 percent lower than the monitored
design value. However, if the model evaluation experiaert had been
repeated for • different year, we may well have obtains^ a different ratio
since meteorological conditions would have changed and since monitoring
equipment is not perfectly precise. What would fee post useful are data
for many additional ywaii, ."raw «.:.tc^. «£ c-^Ti f:~ 2 ifstrlt-jtlc?1 ?f
ratios that would represent the range of lisely values. Altnough addi-
tional data are not available, CUE can pro«--jue us with such a distribution
using the bootstrap technique. IP tne following paragraphs, we briefly
describe the calculations carried out by the CUE method. (See Appendix A
far additional details,}
CUE'S USE OF MODEL EVALUATION RESULTS

The CV.t Method uses information obtained front a model evaluation study to
produc* i range of adjustments to the predicted design value, thereby
producing a ^'stribution for the unknown monitoring design value, i.e.,
the highest second-Mgh concentration that night be monitored over a field
Of receptor sites. The steps Involved in the CUE method can be summarized
as follows:
* Model performance, as characterized by these ratios, would be improved
were the wxiel to use the actual, varying rates of emission rather than
a constant, annual average rate.
-------
pretj MX 2nd highest = 133.54
200 -

100 ISO
PresJi ctstf
200
:crr Coef : 0,42
Q »«e«ptor i
O •wwptar 2
A **c*ptor 9
« *«e*pt«r 4
X t«««pt*r S
m M*C«»ter ft
observed >«x
2n
-------
(1) Obtain model evaluation results at actual monitor locations
(six in our example) consisting of ^airs of monitored and pre-
dicted concentrations for each tine interval (24-hour period
in our example) at each location.

(2) Using the bootstrap technique, simulate 1000 sample years of
model evaluation results. For each sample year determine the
six-station monitored arse! predicted design values, as well as
their ratio, R. Using the distribution of R, determine the
probability with which the monitored design value exceeds the
predicted design value (i.e., the probability with which R
exceeds unity).

(3) Determine the design value predicted by the model for a large
array of receptors (the number of receptors is typically 180,
as in our example} based on a year of meteorological data.

(4) Multiply this 180-recepter predicted design value by the 1000
ratios, R, to produce a distribution for the 180-receptor
monitoring design value. Using this distribution, find the
probability that the monitoring* design value does not exceed
the ambient standard.

(5) Perform steps 3 MO 4 for a total of five years of meteorologi-
cal data to produce OR uncertainty distribution for the 180-
receptor monitoring design value in each year. From these five
uncertainty distributions, calculate the distribution of the
five-year maximum of the monitoring design values.
* Throughout this report "monitored" design value refers to that obtained
for the six stations, whereas "monitoring* design value refers to the
value that would be obtained from monitoring at 180 receptors.

11
-------
CHARACTERIZING MODEL PERFORMANCE

The 1000 ratios (R) of monitored-to-predicted design values generated by
CUE from the model evaluation results cnaract»rl23 tne operational per-
formance of the model (the first issue raised in the Introduction). That
Is, good operational performance by the model would be characterized by
R's tightly distributed about unity. From Figure 3 we see that the model
is unbiased 1n our example, since the 30th percent lie of R is close to
unity. The distribution is also fairly tight since R is between 0.8 and
1.2 {i.e., the predicted design value is within 20 percent of the moni-
tored design value) about 80 percent of the time.
ESTIMATING THE PROBABILITY OF NAA0.S ATTAIlfHENT
ASSOCIATED WITH A GIVEN EMISSION RATE

Upon multiplying the 180-receptor predicted design value by the IOOU
values of R, we obtain the likely range for the unknown monitoring design
value (see Appendix A for details). These probability (or uncertainty)
distributions arc plotted in cumulative fora for each year in Figure 4.
F1gur* 4 also shows the probability distribution of the five-year maximum
of the monitoring desian values.

For each year the figure indicates the predicted design value, the NAAQS
concentration of 365 ng/w-, aSo tne probability that tne monitoring design
value does not exceed the NAAQS concentration. Thus, for an average emis-
sion rate of 15,625 g/s in each year, there is an estimated 55 percent
chance that the 24-hour NAAQS would have been attained in 1973, whereas
there is a 9* percent chance that it would have been attained in 1977.
Hie chance that the standard would have been attained in all five years
during the 1S73-7? period is estimated to be 15 percent.

An alternative, nore compact, way of displaying the information contained
IR Figure 4 is to use a box plot (Figure 5). Box plots allow sketches of
12
-------
» -1.8 t.e t.i t.t i.s 1,9 i.t .1.2us. i.* ,1.5 1,8 .
**•*( ' i i j I 1 » 'i'11 I " ! I I "i | i J I ' i^— •• , i "I1 " 1..
I.W
••• i.S I.f I.t U2 US 1.4 1.5 1.1
Ut-hr
FISURC 3. CuwUtivc ths ratio of wonitortdl to

predicted design values.
13
-------
5 = :S*2S e.'s
* *.
ity C»st.r:sut>cns *0r 1S73 - 197"
c a IS»S25 5/5 a-"»rvj.a' ave.-afS fats o
conc*^"
*»„ (Sets tfte aoee^-srsStrtec concentration and tie NAAQS
NAA3S
14
-------
290,
300.
Ran 'ta
400. 500.
v*sue ru§/»*T . 0 = 15525 o/s
i - - a, *<
V.
io.ic

1
/
^<
toe.
It?? W6T.«w

t *
- 1 T ! « » 1 « 1 ' 1 ! 1 ill
3B>. -83, SOD,
;PS 5ss:^ **1we tus/**!. C it t562S 5/s

5-^ —L
e.is
i—1_
58S.
32C,
*B3,
S08,
= 1SSS5 f/*
4.
IS
-------
5BB

SS8

SCO
OlfTT CTEEK - t2*-HR BVGJ
sac
X
j^»
c—u
IBCr-
l«

j.
SSIS
197*
1975
lifts
IS??
1973-137?
•ILE
U
— HTM
FIGURE S, lox plots (d«p(ietin5 probability distributions for nonitoring
design values, *ssuKin§ * ",5»KS §/s annual tvtrage rate of emission.
Hit iistrtbutittn for tach /tar frwr, 1973 to 197? is depicted, along with
tltt distribution far the wxiwrn o*' the ft** annual design values.
IS
-------
value distributions from more than one year to be displayed con-
veniently on a single page. As in the cumulative distribution plots of
Figure 4» the arrows in Figure § indicate the design values predicted in
each year.

The 1973-77 design value Is represented in Figures 4 and 5 because !n many
cases decision makers are concerned with the largest design value taken
over a five-year period. As noted earlier (Table 1), the largest predic-
ted design value just exceeds the NAAQS concentration. Decision makers
might wish to know the probability with which the largest of the five
monitoring design values woula have been greater tnan (or less tnan} th*
NAAQS concentration. Figure * shows that there is 3 15 percent chance
that the largest design value taker, aver the five years would be less than
the NAAQS concentration. This probability is im-ch lower tnan the
probability in individual years because we are focusing attention on the
nwjtiamn of five design values drawn from tne first five distributions in
Figure 4, any one of wsieh erf gut be larger than the standard.

Solely on the oasis of the original, conventionally interpreted, model
results, « decision Biker sight conclude that the CHfty Creek plant,
operating at «n average emission rate of 15,825 g/s» was in violation of
the standard in one of the five ye*rs, namely, 1976. The decision maker
night then designate the are* *non«tt»fFwieRt* and in so doing seek to have
the mission Unit of the plant reduced to bring the area into "attain-
ment.* However. Figure 4 shows that there was a 41 percent chance of
attaining the HAAQS in lt?6, i.e., £
-------
ESTIMATING TOE PROBABILITY OF ATTAIWIENT ASSOCIATED
WITH CURRENT PRACTICE EMISSION UNITS

The above emission rate of 15.625 g/s was chosen to illustrate the use of
CUE when attainment status is difficult to determine reliably by conven-
fonal wethods. The Cut raethod quantifies the risk of making an incorrect
decision. -In a similar vein, we can use CUE to obtain the probability of
NAAQS attainment for an average emission rate equal to the emission limit
•s currently derived from the dispersion model (see Table 1).

For each individual year from 1973 tnrough 1ST? and for the five-year
period 1973-??, Figure 5 shows the probability distribution of the »oni-
toring design value for an average esission rate equal to the emisslsn
limit prescribed oy the dispersion model, Recall that the emission limit
is the emission rate at which the model predicts a design value concentra-
tion equal to the KAAQS concentratios of 365 u9/»". The corresponding
~\fipnitori«s design value concentration, i.e., that likely to result from an
emission rate equal to tl» etrfssion Haft, can therefore be approximated
by multiplying a aonitored-to-predicted ratio of design values by the
hAAQS concentration.

Performing this aulttplication for the 1000 ratios represented in Figure
3, we obtain the probability distribution for individual years shown in
?*SSf* S. Sstw tftst \Jss selssfsR listt *srlss f?wS y€»r ts jfssr fest tse-
probability distribution remains the sane* with the NAAQS concentration
OtCttrriftf at the 4tth percentite corresponding to the percentile at whicti
unity occurs in the ttistribution of ratios.

Figure 6 also shows the distribution of the 1973-7; maximum monitoring
dtsi^n ¥*"!«»« for tn eBtsston rate equal to the minimum of the five annual
emission Halts, Use of the Host restrictive emission limit confonas to
current regulatory practice (see Tsbl* I).

Table 2 swBsariies. ts« attaiiw»nt pro^aeilities associated with the hypo-
thetical e»itslo*» rate of 1S»S25 5/s and with the emission limits set tiy
-------
O.M
' -'
I » I »
s
• i 'I ' .,-1-
1 l__JL
530. .....
1973
380. 400,
§ 0«*»fB Value t
SOD,
= 15?7S
tl t 1 I t r I t
soo.
5. 8 »
f B.OD
le.aftf-
I
1
290.
ItTS
388,
*SQ.

J. 0
500,
6, Cumulative erofetfeiltty distributions fof 1973 - If?? monitoring
,j-- v
-------
±0.80
9.M.*-
!*-4
i«.»L
« t i _•_
203. 300.
It?" N«n«torttti} »«»
400. 580.
**»«« £«s/»*S. B * I»S*l s/s
i
f 4,80
it. ««
fo.
io
i.J?
-.1.:.-:^ S-
MO.
3SS3.
*00.
(us/r»s), 8
SOO,
Fig 6. concluded.
20
-------
TABLE 2. Probabilities of NAAQS attainment associated with the given
emission rate and with emission Units set by current practice.
fear
Siven Emission Rate
Emission
Associated
Probabi ity (S)
Calculated Emission Limit
Emission Associated
Rate (q/s) Probability (%}
1973
1974
1975
1976
1977
1973-77
15,625
15,625
15,625
15,625
15,625
15,625
55
85
70
48
94
15*
15,975
18,221
16,923
15.498
19,941
15.4B
49
49
49
49
49
17*
1$ the probability that th* 1973-77 Mxim* iBomtorins design
value concentration resulting frcn the indicated 1973-77 emission rate
would not exceed the HMQS concentration of 365
-------
current practice. As a consequence of the operational unbiasedness of the
model, i.e., the fact that the monitoring design value falls above or
below the predicted design value with nearly equal probability, the
attainment probability associated with the current practice emission limit
is «i percent for each year. Using the five-year artniaysi emission limit
results in a 17 percent probability that the standard would be attained in
each cf the five years, i.e., an 83 percent probability that the standard
would not be attained in at least one of the five years.

It is tempting to conclude free the 83 percent probability of non-
attaimsnt that the emission limits set by current practice are too
Itnitnt. Such a conclusion is not warranted at this staye, however, for
several reasons. First, the wgnitasle of the probability is overstated
because 1t was derived from ratios in Figure 3 that characterize a model
that ass uses e constant rate of emission over the one- or five-year
period. This assumption exaggerates the uncertainty or imprecision of
model predictions. Second, as we discuss in subsequent sections, trie
adequacy of the emission limits derived from a model by current practice
should be judged not by the corresponding probability of attainment, but
by the probability that ca«i be achieved with only a slight alteration of
the emission limit. Thus, an unbiased model would produce an annual
emission Unit having an associated S3 percent probability of attainment,
but if the model is sufficiently precise the Mission limit corresponding
to a 90 percent (or perhaps even « 99 percett) prcbaoility of attainment
would be enly slightly more stringent. Pl*mll>, addttiorr'i mode.
evaluation studies and settorelogical records snculd be examined before
general 1 zing the results presented he^e,
we have cautioned ageiast generalizations about rode! perfo mance
or the apparent leniency of current practice, the foregoing discussion
dononstrates how CUE can be used in a given setting to assess the effects
of modeling wcertainty on attainsent/nonattaiflnent decisions. We thus
see hot* the CUi method can be used to answer the second question raised in
the introduction. Me now turn to the third question: what level of
emissions should be permitted from a source in order to bring the risk of
22.
-------
nonattainment to an acceptably low level while avoiding unnecessarily-
stringent emission limits?
ESTIMATIN6 WE EHISSI0M RATE ASSOCIATED WITH
A OESIREO raOASlUTY OF HAAQS ATTAIIfCMT

Since Me can as>une that concentrations are linearly proportional tc emis-
sion rates* we can choose an «rission rate for each yea** from 1973 through
1S77, or for the 1973-77 period, such that the resulting probability of
NAAQS attainment is soae desire value. Table 3 shows emission limits
corresponding to attainment probabilities of SO, 90, 55, and 99 percent,

lie see that the 1S73-77 emission Unit based on current practice would be
1S,498 8/$, slifhtly less (< 1 percent) than the hypothetical value of
15,625 g/s. Me also see that, to achieve a SO percent probability of
attaining the standard fn all five ytars» the enission limit would be set
at 14,092 g/s» or reduced by about 9 percent from current practice.
Alternatively, if a 90 percent presabtlity were desired, the retired
emission rate Mould be 12,098 §/s, or 22 percent lower than current
practice.

The Mission li»t determined by current practice for 1S76, the year in
Mhich the naximra predicted tJ?sign value occurs, is by, definition the
(nil3siun llHif obi-airrcQ for the 1573-7? perioa. ThfS IS ROt the Case,
however,, for eaission liwits determined by CUE since the maximum design
value my occur in arty of the five years (albeit with different proba-
bilities). If, for exaaple, the desired probability af attainment is SO
percent, then the emission limit obtained by CUE is 15,442 9/5 for 19?b,
whereas ft Is l*,0it g/* for the 1973-77 period.
The results In Table 3 are sw*aHzed in Figure 7, which illustrates emis-
sion Units corresponding to current practice and two probabilities of
attainment: $0 and SO percent.
23
-------
TABLE 3. Comp«r1$on of mti*fon ItmUs set by current practice and those calculated by UJt
to yield a desired probability of NAAQS attainment.
Highest Associated
Sccond-M^i Emission Probability of
Concent radon Llfftt NMQS Attainment
Vwr (W/»3 («j/»J (I)
1973
1974
1975
1976
1977
1973-77**
357
313
337
360
m6
368
15,975
18,221
16,923
15,498
19,941
If ,498
49
49
49
49
49
il
Emission limit (g/s)
(or Desired Probability
of MAAQ& Attainment
bU%
15,901
18,150
16.83B
15,442
19,824
14,092
9(11
13,207
15,075
13,985
12,825
16,466
12,098
95*
12,331
14,075
13.057
11,974
15,373
11.684
99*
11.294
12,891
11,959
10,967
14,080
10,886
»*
Computed using CUE »f 1n Table 2,

This row refers to the probability tuat the W !'! niaxtmum design value concentration
resulting from the Indicated 1973-77 Mission rate woul) not exceed the
concentration of 365
-------
C*i3tiOn MBit 1x1832 g/Sl
S IJ II 12 1^ 14 15 IS 17 IS SS 2p
1973
IStt
tun
?«78
187?
///S//////////////A
»tt»ir»cnt
of
?. CunptHsofi of tirtsslOR limts crtsttfnei vis currtnt prtctict
wisslw ti*1ts reottiitd m tchieve SO and §0 percent probabilities
Of NAAQS «tt»':m»nt.
25
-------
All the emission limits reported here conform to current practice In one
respect. They are based on model evaluation results and attainment
demonstrations 1n which the dispersion model uses a constant rate of emis-
sion (I.e., they do not allow for the varl >ility of actual rates of emis-
sion}. Had the model usrd fluctuating rather than constant, peak
emissions then the current-practice emission limits in Table 3 would be
somewhat higher (or less stringent). The magnitude of this change depends
on the magnitude of the fluctuation in emission rates relative to the
average emission rate, 10-30 percent being typical for 24-hour average
rates of emission from large coal-ftred boilers.

Additionally, the model's use of actual fluctuating emissions would most
likely improve the performance of the mode!, as characterized by design
value ratios {Figure 3). Moreover, if the CUE procedure were modified to
include the randomly varying fuel-sulfur content (i.e., incorporating an
ExEx-like methodology into C41E), then the resulting emission limits from
the modified CUE procedure would also be altered. In this case, however,
it is not possible to estimate the change in stringency because the shift
Is strongly influenced by policy choices about statistical interpretations
Of the NAAQS and the setting and enforcement of emission limits derived
form such Interpretations. Suffice it to say here that there is enough
latit'jde in these policy options to allow for any such shift.

Thus, by recognizing the variability of emission rates, as recorded in the
model evaluation study and as projected due to natural, random variations
In coal-sulfur content, the estimated risk of nonattainment may be
substantially altered without su&stantta! rel«x«tien of current practice
emission limits.
«-3 2 26
-------
THE EFFECTS OF MODEL BIAS AND IMPRECISION

The foregoing results represent a hypothetical situation In which there is
little direct evidence of model bias (i.e., the tendency of the model to
overestimate or underestimate monitored design values). In this example,
therefore, the principal source of the uncertainty in providing design
value estimates is model imprecision. In Appendix B we examine the
effects of model bias and imprecision en NAAQS attainment probabilities
and emission limits. The results can be summarized by discussing emission
limits set by CUE under several cases of model bias and imprecision.

Me illustrate the separate and combined effects of different levels of
bias and imprecision by making hypothetical but reasonable adjustments to
the ratios characterizing operational aodel performance given in Figure
3. He consider three levels of bias (one of which is no bias, cf. Figure
3) and three levels of precision (one of which is unaltered from the
results in Figure 3), resulting in nine cases, eight of wnicn are new.

The levels of bias chosen were ±30 percent. Bias mas introduced by shift-
ing the overall distribution of ratios of monitored to predicted design
values (cf. Figure 3) by a factor of 1.3 (underprediction) and by a factor
of 1/1.3 (overprediction), thus altering the geometric mean ratio by these
factors. The operational precision of the model was altered by,
respectively, squaring or taking the square root of the geometric standard
deviation of til* ratios. In a certain sense, stated in Appendix B. the
operational imprecision of the model was doubled or halved, respectively,
by these alterations.

The effects ef changes in overall model accuracy or* emission limits
obtained from CUE are summarized in Table 4 for the year 1973. The
entries In Table * represent allowable emissions for a 90 percent proba-
bility of KfcAQS attainment. The entry in the center of the table, for the
case of unaltered precision and no bias, is identical to the allowable
19?3 eoisslons given In Table 3. Considering first changes in precision
only (no bias), we see that when model precision is enhanced, the emission
-------
TABLE 4, 1973 emission Units set by CUE and designed
to achieve a SO percent probability of HAAQS attainment
for different scenarios of model bias and imprecision.

Unde rpred i ct 1 on Unbiasedness Qverprediction
_ (e < o) _ u • o) __ (s > o)

Enhanced 14,528 14,528 14,528
Precision
U - 1/2)

Unalttrtd 13,237 13,20? 13,207
Precision
(» « D*
Precision
C* - V*
* See Appendix 8.
-------
limit Increases approximately 10 percent, from 13,207 to 14,528 g/s;
higher emissions ear: be permitted because of the greater reliability of
the dispersion model. Similarly, when precision Is diminished, the
emission Halt decreases 17 percent, from 13,20i to 10,915 g/s. The lower
allowable emissions result from the need to compensate for greater
uncertainty.

Turning to the results for cases of »as!el bias, we see that the CUE methoa
provides values for emission Halts that are Identical to those obtained
for the unbiased results; that Is, In the 30 percent umierprediction case
CUE provides the sane results (10315, 13,207, and 14,528 g/s) as obtained
for an unbiased model. This, of course, is as tt should be. These
results (cases of laodel bias} nay be contpareci with what would be obtained
from current practice for, say, the case of 30 percent underprediction.
In this situation current practice wsuls give an emission limit 30 percent
higher (viz., 20,768 * 1.3 x 15,975, cf. Table 3), which CUE would adjust
downward to account for the bias.

The effect of wnlel imprecision on enfssfon limits will be particularly
pronounced for emission Halts based: on the five-year maximum design value
(see Table 3). Here «* •fain note that gains My be made both In model
precision and in the realism of the regulatory use of models by
incorporating the actual, varying rates of emission in model evaluation
result* and then incorporating these varying rates in the linkage provided
by CUE between jn annual average emission rate and the corresponding
.y of NAAQS attainment.
Thus, by incorporating variable rates of emission, CUE will provide more
realistic emission limits while maintaining an abjective means of
incorporating modeling uncertainty into regulatory decisions.
r2 2 29
-------
4 SUMMARY AND COKCLUSIONS
SUHRAR* OF RESULTS
This report has described a hypothetical regulatory application in which a
dispersion ac.de) was used to estimate the sir quality impact of an
electric stew power plant. Sodel results, stated in terms of design
values, showea the plant to be in violation of the 24-hoyr NAAQS for SU2
in one of the five soUeled .years, while predicted design values in tne
other four years were less than the standard, given only this informa-
tion, a decision eaker Bight conclude tftat the plant is in violation at
the NAAQS and that Missions reductions are needed (Table 1). However,
model results are known to contain uncertainties and therefore a decision
•iker pright want to know what the chance is that the plant would cause a
violation. By this we aean that a decision maker might wish to have some
information regarding the likely Monitoring design values, i.e., the
values tint would have been obtained from monitoring at each of the 180
•odel receptors for each of the five years.
The CUE method uses infernal ion obtained from a given model
study to "correct* the predicted design values and thereby to estimate the
highest second-high concentration that night sa measured ov*tr a field of
receptor sites. The steps involved Is the CUE setftcd srs sususarizea in
Section 3 and are further discussed in Appendix A.

In our hypothetical example, the probability of NAAQS attainment in each
year fni» 1S73 to 1ST? ranged from 48 to 54 percent. However, the
probability of attaining the standard in all five years was only 15
percent (Table 2). Thus, a regulatory policy that focuses on the maximum
tfeslp vtlae taktn over a five-year period is much mor* likely to result
30
-------
1n a finding that the plant 1s In vloiaiScr, t^n one that looks only at
the design value for an individual year. It should be notes shot tM*
Increased stringency results primarily because the plant (operating at its
hypothetical emission rate) Is producing peak 24 -hour average
concentrations very close to the HAAQS In each of the five years, larger
year-to-year differences in the design values, coupled with decreases in
Modeling uncertainty (greater agreement between monitored and modeled
design values), would result In less dramatic differences in the effects
of these two alternative policies than is the case In tne present example.

Are emissions reductions required at this plant to avoid an excessively
high risk of NAAQS nonattaiRaent? If we suppose that a 9S perctnt pros-
ability of attainment, I.e., a 1 percent probability 3? violation,
represents an acceptable level of risk for decision makers, then a
reduced emission H«it is required, since Taa^e 2 snows attainment
probabilities t*elow 99 percent in all esses. Alternatively, if i SO
percent level of risk were acceptable, then the need for a reduced
•Mission Han't cannot be determined without considering which attainment
period policy Is preferred, »it*» the five»ya«r or individual year
attainment period. If the former were chosen, tnen a reduced emission
Unit would be required; if the tarter acre chosen, then a reduced
•mission Unit would not be required. Hie amount that e-issicn limits
would have to be reduced for any alternative probabilitv-of-NA vis-attain-
ment policy can be calculated as shown in Table 3.

At the cad of Section 3, we briefly considered the effects of model bias
and imprecision on the emission limits, derived: by CUE, required to
achieve a desired probability of KAAQS attainment. Me illustrated how
CUE, unlike current practice, compensates for model bias. We also illus-
trated how CU£-deriv*d emission Units would increase with improvements in
•edtl precision and decrease with degradation in ntodel precision.

Ml 8*1 1«v* th* exanple presented here is encouraging because it represents
a frame wort and a method for incorporating modeling uncertainty into
attainment-demonstration decision making. However, we strongly caution
3,
-------
against generalization of the results and policy implications from this
one hypothetical example. Our understanding of how results depend on
source characteristics, source location, results from different model
evaluation studies, explicit treatment of fuel-sulfur variability and
pcllcy options is, as yet, incomplete.
CONCLUSIONS

CUE can be «sed to address the three issues raised in the introduction,
namely,

Characterizing operational Boaej.. performance in the form of a distri-
bution of ratios of notiitored-to-predicted design values (Figure 3),
a distribution centered about artity indicating "operational unbiased-
ness," and the steepness of the slope in the cumulative distribution
Indicating "operational precision";

Estimating the probability of KAAQS attainment for an annual average
emission rate, either given or prescribed by the dipsersian model
(see Table 2};

Estimating the annualaverage mission rate rteedecto achieve a
desired probability of MMQS attainment ($t* T*ble 3).

With respect to the emission limits derived bjr CUE, « have seen that

(Milike current practice, CUE compensates for model bias

CUE provides hiftter fless stringent) emission limits as the model
becones wore precise a«ot provides lower (more stringent) emission
Hirtts for Imprecise models to compensate for the greater uncer-
tainty.
32
-------
Finally, model precision is likely to be increased when we provide the
model with the actual, varying rates of emission. Moreover, once model
evaluation results of this kind are available, it is possible to modify
CUE so that this greater model precision is incorporated in the calcula-
tion of more realistic emission limits. CUE thus offers an objective
method for deriving emission limits that compensates for a model's
operational bias, uses a model's precision, and incorporates alternative
policies for managing the risk attendant to these uncertainties.
33
-------
REFERENCES
Efron, B. 1982. The Jackknife, the Bootstrap and Other
Plans. Society for Industrial and Applied Hathanatics, Philadelphia,
Pennsylvania.

EPA. 1977. User's Manual for Single-Source (CRSTER) Model. U.S.
Environmental Protection Agency, Research Triangle Part?, North
Carolina.

Londergan, R. j,, 0. H. Ht*nctt» D. J. Mackter, T. H. Kincaid, and D. M.
Bonitata. 19S2. "E*al«tt1oo of Rural Air Quality Simulation
ftodels." U.S. cnvironaeRtal Protection Ayency, Research Triangle
Park, Jtortl? Carol int (EPA-450/4-83-003).

Hills, K. T.» R» Caitzza, 0. B. Hergert, and A. Lynn. 1981). "Evaluation
of Point Source Dispersion Models. " Teknekron Research, Inc.,
We It ham, «fass»chusetts.

Pierce, HIQMS E.» ani 0. 8. Turner. 1980. User's Suide for HPTER: A
Multiple Point Smssian S1sp«rsl{» AlseHthm *tith ftittonal Terrain
Wjustnent. U»S» |nvirctiiie«t«1 Protection Agency, Research Triangle
Park, 1C (EWU6QQ/8-80~fJl«J,
RiO, K,» and L. Sitterfieltf. ISS2. HPTER-QS: The MPTER Model
U»S. Invironiwntal
-
Triangle Park, North Carolina.

Stoeckenius, T. l»» S. Thrall, and C. S. Stirtsan. 1983. "Evaluating the
Effect of Mdeling Uocertaint^ on the Determination of Emission
Units Using the "lootstrap* Technitpte." Sjystems Applications, Inc.,
San Rafael, California.

Steeckenius, T. E.» 0, Thrall, and C. S. Burton. Ii84. "Incorporating
EsfiiMtes of Modeling Uncertainty into Dispersion wsdel Results."
IppHcations, Inc., San Rafael, California.
8* it* i* ? 34
-------
Appendix A

DETAILED SI*CKlPTIOH OF Oil
35
-------
Appendix A

DETAILED DESCRIPTION OF CUE
MODEL EVALUATION RESULTS

In our example of the apalication of CUE we consider an attainment
demonstration of a topical regulatory situation Involving a large point
source of SO? emissions. As in our previous reports (Stoeckenius pt al.,
1983, 1984). **e -se the CTiftv Creek potter plant near Nae$tson» Indiana as
the basis of our example. Relevant plant pa~a«eters are listed in "sale
A-l. Althoy§h the Cliftj Creek plant operates at an annual average
•mission rate of about 8400 g/s, for the purposes of this report, we
assiwe * Much higher hypothetical rate (15,625 g/s) to provide an example
of • power plant that produces peak 24-hour average impact levels close to
the S0| primary JtAJUS {365 uQ/n3}.

Meteorological data collected at Cincinnati (surfacp^ anct Da^tcm (upper
•if) for li?3 throufh 19?7 were ased in exercising the WTER dispersion
•will (Pierce and Turner, 1980). A polar gria of 180 receptors located at
10s intervals aiong five concentric rings, centered OR the source at
racial distances of 1, 3, 6> 12, end 20 te, was specified; ttte resulting
f average ground-level concentrations Mere used to calculate the
s»conc Hlfitest concentration (NZH^gQ) in eacn year. TaSlt A-Z
shows K2H|gQ for each fear along with a Modified design value wfticn we
call MDVli80 (iUVligQ is «$e4 1n siwtlttions, wfticn we sftall describe
shortly).
A comparison of the waximw second hfgftest concentration with the 24-hour
S02 NAAQS shows that the plant with this emission rate would be in
-------
TABLE A>1. Clifty Creek stack parameters.
(Source: Kills et al. 1980).*
Scack 6as
Stack Height Diameter Temperature Stack Gas
Exit Velocity
1
2
3
207.8?
207.8?
207.8)
4. S3
4.63
4.63
445.37
445.37
445.37
49.9
49.9
49.9
Total plant capacity = 1300 MW; total plant emissions
(Hypothetical) - 15,625 §/s-
TABLE A-2. 24-ho«»r average desigr -values
predicted toy the NPTIR dispsrsfon «xtel
for a network of ISO receptors ysls§ a
hypothetical emissions rate &f 15,62= g/s.

|1|xt|iu|| •
Second Highest Maximum D«l
Concentration *
Year ug/B3 (PP»)*
Ifll
1S74
»?5
lS?f
1977
3S?
313
337
36?
286
36U
365
36S
348
313
DV1 Is the concentration value exceeded
exactly once p*r year as determined front
* tail exponential fit to th* distri-
bution of daily concentrations.
37
*'
-------
violation of the standard in only one of these five years (1976). By
current procedures, we can derive an emission limit based on each year's
design value by multiplying the 15,625 g/s emission rate by the ratio of
XNAAQS Ith* 365 sS/1"3 1eve1 of the current 24-hour S02 NAAQS) by the
calculated maximum second highest concentration, N2Hi80 (see Table A-3).

Since the model predictions are not perfectly accurate (i.e., they contain
uncertainties), a decision maker might wonder whether it is really
necessary to designate the area as nonattainaent and oegin the process of
modifying the plants emission limit to ensure that the plant does not
cause violatiors of the ambient standard, or whether the 1976 model
prediction can be dismissed as an aberration. What, the decision maker may
wish to know is what the observed saxintw second highest concentration
would have been in each year if Monitoring data had been available at the
180 receptor site, ye can denote these values by «2H18Q. If H2H180 is
less than the NAAQS in all S years, then the decision maker could assume
that the plant is in attainment of the HAAQS and that no reduction in
Mission rate is required. CUE allows us to estimate what N2H180 would
have been for each year.

In applying CUE to this situation, we use a model evaluation study carried
out at Clifty Cretk during 1875 {Mils et al,» 1S80). In this study,
Monitoring data and MPTER «odel results at six locations near the plant
Mere used to obtain observed and predicted maximum second highest concen-
tr*ft*SB* (Matg «!£ "BS* respectively) ?er tfes
A-l. These values are ouch lower than those calculated from the
NPTER results for the ISO-receptor network (Table A-2) because they were
obtained under the plant's actual 1975 emission rate of 3420 g/s.
However, since ground-level concentrations can be reliably assumed to be
linearly proportional to eeission rates, model performance as measured by
the ratio of otserwt-te-predicted concentrations is unaffected. Thus,
model valtdatioa results for the 24-hottr average concentrations obtained
in 1875 ander the actual lower emission rate can be reliably applied to
-------
TABLE A-3. Emission limits set by current practice. {See note)
Year
1973
1974
W7S
ir%
1177
1973-77
Given Emission
Sate (g/s)
15,629
15,625
15,625
15,625
15,525
15,£?S
Maximuir
Second Highest
Concentration
(n9/"i )
357
313
337
367
285
367
24-Hour S02
NAAQS (jig/m3)
365
365
365
365
365
3§5
Calculated
Emission
Limit
15,975
18.2Z1
16,9E3
15,498
19,941
15,438
Note:
OIL- T*

= calculated emission livit

. f^^OUT $$% NAAQS (36$

QQ * 9iven emission rate. I.e., the nypotnetical aaission rate of
15,625 g/s used in the example analysis

0V * Mxlauo second highest concentration (ug/m3) calculated by
the d1sf»rston mMtl (known as the "Mesifn *ttae* due to its
role in determining eaission limits)
39
-------
model predictions obtained for the hypothetical emission rate as long as
we art considering only the ratio of observed-to-predlcted 24-hour average
concentrations.
NOW CUE USES MODSl EVALUATION RESULTS

Taking the ratio of M2Hg/K2Hg from the values given in Figure A-l, we see
that the node! underpredictec the design value by a factor of 1.4.
However, 1f the model validation experiment had been repeated for a
different .year, we my well have obtained a different ratio since meteoro-
logical conditions would have changed and since centering equipment is
not perfectly precise. What would be atost useful are data for many
additional years. fro* which w* could fora a distribution of ratios that
would represent tfte range of lifcely H2Hg/feH6 v«1aes. Although additional
data are not available, CUE can provide us with such a distribution usiny
the bootstrap technique. In the following paragraphs, we provide a brief
description of the calculations carried out in the CUE method.

The first step Involves the construction of a large noB**r (say, 1000) of
years. Each sample year is nwcV ap of 36f day?. *r«wn at rancora
tt» 38S days in calendar year t9?S» tey C-'VSK aty may at selected
•ore than once for inclusion In the sane sample yeer. For each day drawn
•t random for inclusion in a sanple year, the cf> responding observed and
pftsSlsts^ css^sstratlsstS «t s*c»i wf tfee sis rtcr *csrs US»T in tne ^tiSel
validation study *""* recorded. Sets of observed ar.d p.c'^-ea concentr^-
tions obtained in this way for a complete sample year f.-' :nen used to
calculate a pair of observed and predicted design va*uee for that ye&r.
figure JU2 shows MM the 1000 pairs of design values an? gv.eratec.

Because of the May the bootstrap technique wonts, the maximum second
highest concentration is not a design value suitable for this procedure
{Stoeckenius et a1.» 1983). He use instead a related design value known
as NOV1. which is calculated as follows: For a given sample year et each
receptor, the value txcte«fed exactly once per year U determined from a
•m%%r *
40
-------
pretf 2na
= 120.24
200
100 ISO
Predicted
a Receptor 1
Corr Coef : 0.30

OM 2nd Mgtt**t * 187.CZ
200 250
too -
t i J
100 ISO
Preen ct«ct
200
UMl
CD Receptor 2
Corr Caef : O.S3
- 117.04
250
A-l. Clifty Creek. 24-hour average insdel validation results.
41
-------
200-
Receptor 3
Corr Coef : 0.33
2nd
157.57
so
100 ISO
- Predicted
200 250
» t3S.i*
200
SO

XMt NifhMt
100 150
Prtdtctect
Receptor 4
Corr Coef : 0.58
2nd
= 102.4}
200 250
A-l continued

42
-------
pred 2nd
« 7S.06
200
50 100 ISO 200 250
100 ISO
Presti cted
ttt*
*nd
X Receptor 5
Corr Coef : 0.35
- 060 2nd highest = 54.15
Receptor 6
Corr Coef : 0.24
ob« 2nd
- 130. SI
200 250
FISURE A-i continued

43
-------
200 -
Ob*«rved MX
2nd r>lgh«»t a 187.02
100 ISO
rr*sitei«I
200
250
Corp Ceef s 0.42
O
© tac«f»tor S
3
4
X t*c*ptar S
K l«c«f>tor ft
PISORE A-l concluded
44
-------
=>. «
I f
u

-1 9
f! *'!
Il |i«-'!
ii ii III
o
o
•
t
at
«c

I
o
3

til
O
O
S
r
r-i
'

T
§
i
-------
till exponential fit (Breiman tt tl.» 1978) to the upper 10 percent of the
365 observed concentrations. The maximum of these values taken across all
6 receptors is the design value far the sample year, We refer to this
value as WVlfi. Hie predicted design value (MOYlg) is calculated in a
similar manner. HDV1 values are generally comparable to M2t^ values and
tile smoothing Imparted by the ta 1 exponential fit sakes then suitable for
use in the bootstrap method. A set of 1000 («3V16, «0¥lg) tr,as generated
by the bootstrap technique is shown in Figure A-3.
CHARACTERIZING N
Tne next step in CUE is to use the 1000 (HOVlg, WVlg) pairs shown in
Figure a-i to caicuivte iuuu ratios ?.» wfeere K - fWig/«u*i.» A cianyla-
tive distribution of these ratios is shew* in figure A-4. In the final
step, the 1000 values of 8 are used to "correct* or adjust tne M2H design
values predicted by the MPTBt model for e«cR of S years at the 1BC-
receptor network (H2H180)» thus providing «n estimate of the monitoriny
design value,
WD8ABILITT OF FWAQ5 ATmiM€NT
ASSOCIATE WITH AN {MISSION RATE

Mhen each of the 1000 values of R is substitued into Equation 1, 1000
H2M180 values are produced, forming a distribution describing the likely
range of the monitoring design value, i.e., the design value that would
have been observed had monitoring data been available over the ISO-
receptor network. The »odele
-------
ize
178
220
272
s*
i

1
V-, » , .
w» * * '
* «. * »*
f m
»_•«.
»• „ >• « » »»
*m * ** ** m*m J» 1 " %i«* * *

*(t» * » * & ^ 1L • *»*« * » *
228
27B
Matf»t-t«s«a
•278
-J22B
A-3. Scatter plot of o&$er¥«t (non1tor>t»sed} and predicted
HW1 design *a1i«s ftnertttci by the bootstrap method.
-------
AS •«• 1.7 >.« t.t !.• i.J 1.2 _ l.S U< 1.5 1.6 . _
i I ' I * 1 » I « I ' I ' i ' i i i^»~P' | . ) .'«••
*••-$•
«.*
t-.->-_---*.-• '• t ' ' • I

-a. ao
-9.99
-a.sa
-*.«•
-i.3B
-B.2B
-a. ia
».» 1>V [.I l.Z i.S I.« S.S J.« "'
Ctiffcy t»«Mt V17S
A-«* Cuwlatiws eltstHbuticm of R * NTOg/MDVl, obtained from
bootstrap results 1o Ffpre A-3, ° °
48
-------
293.
1973
300.
«CO. 500.

{«$/»*>„ 0 - 15S2S s/s
200.
^ PT X

t t

JLT||L I .. |.. I _fe_ _ _f tl.lt t .. >

300, *SO,
t ^
500.
. . .
197-4 want Wing Ois-jr V*Fu* !ug/»'j, C = 15625 9/«
A-5. Cumulative prcbabilfty dtstrtbutions for 1973 - 1977
oesiin values, assustng a 1S.S25 s/s annual averdie rate of
» «Sottt the i»o4sl-presjTct*cl ccncentratior and the NAAOS
of 3S5 ug/is are indicated. The probability of NAAQS
attatnaent is also shown.)
-------
A-5.
SO
-------
estimated probability that M2H180 is less than or equal to the NAAQS
concentration of 365 ug/m3. Thus, the probability of NAAQS attainment Is
shown for each year.

In many cases, decision makers are concerned with the largest design value
taken over a five-year period. As noted earlier (Table A-2), the largest
M2H,gQ just exceeds the HAAQS. Decision makers might wish to know with
what probability the largest of the Monitoring design values would have
been greater than (or less than) the NAAQS. Let's call this largest
design value the 1973-7? aonitoring design value, denoted as Xj.g (the
largest value out of a set of five). Me can find the distribution of X|.g
by assuming that the design value in each year is a random value, whose
distribution is shown in Figure A-5» independent of the value in any other
year. uenoting the aeaion *a>ue l-.t f*+r I Lj «* > -c J;svs S. '
, X^, X*. X^)- T*»" for artv concentration XQ
Equation 2 defines the cumulative distribution of X|;5, which is also
shewn in Figure JUS. Of particular interest is the result obtained from
Equation 2» where XQ * 365 p§/«3» the 2«-hr S02 NAAQS. As shown in Figure
A-S, there is * IS percent chance that the largest design value taken over
the five years Mould be less than or equal to the standard. This
probability is Much lower than the proofioiTlty in individual years because
Me are focusing attention on the maximum cf five design values, any one of
wtelch wight be larger than the 5ta*wf»nts drawn from the five annual
distributions in Figure A-S.

A» alternative. «ore cospact, way of displaying the information contained
1n Figure A-5 is to use a box plot. As shown schematically 1i Figure A-6,
i box plot is constructed by simply noting the concentrations
corresponding to various percentiles of the distribution with special
syftbols. Box plots allow information from more than one year to be
-------
twgi
FISISI A-6. Ex«ap1* of the derivtttioit of a tjcsx plot from a
distrltetion.
-------
displayed conveniently and clearly on a single page (Figure A-?}. As In
tin cumulative distribution plots (Figure A-5), the arrows in Figure A- 7
Indicate the location of feH.gQ in each year. Figure A-7 shows that there
1s a chance that the NAAQS would not have been exceeded by an observed
design value 1n each year. These probabilities are listed in Table A-4,
which Illustrates NAAQS attainment probabilities ranging from 48 percent
for it?f to §* percent for 19??. Usin§ Equation 2, and th« pooling
information across the 5 years, we see that the chance that 5 -year seximwn
design value 1$ below 36S wg/sn^ is only 15 percer.,, As we shall discuss
shortly. Table A-4 also shows probabilities of NAAQS attainment
corresponding to emission rates equal tc the emission limits prescribed ay
current sractiee.
en the basis of the original, conventionally interpreted model
results (KZHjan values), a decision swicgr might conclude that "he Clifty
Creek plant MS In violation of the standard in one (1976) of the five
years. Pro* our CUE calculations of modeling uncertainty, however, we
estimate that there is a IS percent chance that this would be an incorrect
conclusion based on the worst-in-five year design value and that there is
a 48 percent chance that this would be an incorrect conclusion based on
just the 1976 design value. Alternatively, ff a decision maker concluded,
based on conventional use of sodel results (i.e.* all five years), that
the plant did net demonstrate attainment , then such a decision maker could
bt 85 percent confident of being cornet. However t if a dtcisifjn «§ker
concluded that the plant did not demonstrate attainment based on one year
(1976), an cstteat* of th* probability tfttt such a decision is correct is
52 percent. To reiterate, the 15 and 4t percent probability estimates
represent the probabilities th«t «o srrission reduction is needed.
we can assuat that concentrations are linearly proportional to
emission rates, the results showtt 1n Figure A-7 can be easily rescaled to
SIMM whit would happen if the Clifty Crtek plant were to operate at a
different tsrissien rate. In particular, we can rescale the results to
stew the probability of NAAQS attaino*nt that would result if the plant
nere to operate at an emission limit, Q|-jm» calculated strictly jn the
basis of the fetf* walues listed in Table A-Z:
13
-------
559
«se
glSB

law

|zse
u
Urn
CUFTT Og£K -
BVSj
ET
T
r i

r
1873
197S
_
T E
1973-1577

-95TM I^*C£N'":^E
•4- aim TOKEN- -.s
4- I«TM
-5TH PtSCSNTILE
lECC

•55:.

j sea

•«B
*
J150

hes
3

150

j.
FISIRE A-?. Box plots depicting probability distributions for raenitorina
d*sign v*1u«s, assuaing a 1S.S2S §/$ annual average ratt of emission,
The distribution for each year from 1873 te 187? is depicted, tlong with
the distribution for the wxiat» of the five annual design values.
54
-------
TABLE A-4. Probabilities of NAAQS attainment associated with the given
Mrtsslon rate and with emission Units set by current practice.
Given Emission Rate
Year
1173
1174
1975
1976
117?
1973-77
Mission
Rate C»/*>
15,525
15,621
IS.S2S
15,625
15,625
IS .§25
Associated
Probability (1)
55
85
70
48
94
IS*
Calculated
Emission
Rate (g/s)
1S.97S
1S.2 I
16323
li,4S8
19*941
15,498
Emission Unit
Associated
Probability (X)
49
49
49
49
49
17*
Tfct probability that the 1973-77 maxim;*! *dnitorl»$ dasfgn
concentration resulting fro« the indlcateU 1973-77 mission rate
would not exceed the NAAQS concentration of 3i§
-------
M2H180

where XKAAQS " the 24-hour NAAQS for S02 (365 ws/«3). and Q0 » the
original emission rate used in constructing Figure A-? (15,625 g/s).

Values of 0^,, correspond to current EPA practice for setting emission
limits. These values, along with the probabilities of attainment thit
would result froa their ispleaentation, are listed in Table A-4.
ESTIMATING THE EMISSION RATE ASSOCIATES MITH
A DESIRED PROBABILITY OF NAAQS ATTAIWOlT

It is also possible to calculate an emission rate, y , sucft that any one
9
of the Six uncertainty distributions in Figure A-5 we choose to focus
attention on vill be altered to Indicate an a percent probability of
attalnnent. lit can calculate Q fn»
where 0¥w » the design value corresponding to an a percent probability of
attainment as determined fro* Figure A-4 for the distribution of interest.

Table A-S provides values of QjlB and 0 » the latter corresponding to
a « SO, 90, 95, and 99 percent probabilities of attainment determined from
Equation 3, The various eafsslon Hints are shown for each year and the
worst~in-f1vc years, from Table A-S we see that the emission limit based
on current practice would be 15,498 g/s, Ue also see that considering
nodel uncertainty and accepting a SO percent probability of NAAQS
attainment based OR a worst~in~fi*e year policy, the emission limit would
56
-------
in
TA8LI A-5. GonptrlMA of *mii»t»n Hmlt* ttt by current prdctUe antl those calculated fey CUt
to yield • desired probability of JMAQS fttatnwnt.
Highest Associated Emission Limit (tj/s)
Second-High EnHsloo Probability of fur Uestreo Probability
CwicttttrattW) Limit MMQS Altai (went of MM(|S Attainment
Year {ug/mj (9/1) (%)
1973
1974
1975
1976
1977
1973-77**
357
313
337
360
286
368
15,975
18,221
16,923
16,498
19,941
15,498
49
49
49
49
49
17
WJ*
15.9U1
IB. 150
16,a3l)
15,442
I '.1,8*1
14,092
90*
13,101
16,071
13,905
12,825
16,466
12,09(1
9b»
12,331
14,075
13,0b?
11,974
1S.373
11,684
9»»
11,294
12,891
11,9*9
10,967
14,080
10,886
Computed using CUE *$ In Table 2*

This row refers to the probability that the 1973~77 nwxluium deslijn value concentration
resulting from the Indicated 1973-77 emission rate would not exceed the NAAqs
concentration of 368
-------
be set at 14,092 g/s» er reduced by about 9 percent from current prac-
tice. Alternatively, i* an acceptable probability of attainment Is
established at 90 percent for a worst-in-five year policy, the resulting
emission Unit would be 12,098 g/s, or 22 percent lower than current
practice. It Is also of Interest to note that If the estimate of NAAQS
attainment probability is based on a single year, say 1976 because It
produces titf highest design value estimates, and an acceptable probability
1s established to be 50 percent, then the resulting emission limit would
be 15.442 o./s, which 1s virtually Identical to the emission limit that
results from current practices (15,498 f/s),

Many other comparison can also be made between alternative emission liarit
setting and aodtl uncertainty risk management policies* To facilitate
such conparisons, Fvgure A-8 Illustrates emission limits obtained from
current practice and two probability of attainment levels: 50 sr.d 90
percent.
-------
E*i»i0n
ui eee
JS73
197*
23"
^XS\NNX\\V

'S///?////7S///S/SS/A
!^so«S'»
-------
Appendix B

TIC APPUCATlQJt OF CUE TO MASS OR IMPRECISE MODELS
»*i**r * 60
-------
Appendix §

THE APPLICATIOH OF CUE TO SIASID OR IMPRECISE MOTELS
When we used Clifty Creek 1975 data and CUE to exercise the Gaussian
dispersion 'node! fWTEi, it exhibited almost no. bias in design value and
was also fairly precise. But It would be useful to see applications of
CUE to a variety of settings Is which eodel performance, is rsarkedly better
or worse thin that of SPtli for CUfty Creek. In consiaeration of
available resources, we have had to simulate these other settings by
artiflctlly altering the deslpi values derived from our study of WPTER
with the Clifty Creek data.

There art many ways 1n which design values could be altered for the
purpose of altering • Model's bias or imprecision, we decided to utilize
observation-prediction ratios since (1) they p1*jf * key role in our work,
and- (2) their characterization of •odel performance Is familiar to the
•QBtlfiti cowounity. For exiaple, suppose that a model 1s "within % factor
of t (of Monitored values) Half the tine,* Specifleally» suppose that the
25tn percentfie of observed/predicted ratios Is equal to one-heIf, and
that the 75th percentile Is equal to 2; or, taking logarithms, that the
ISth percentile of the lofptf ratio 1s equal to - In Z, and the ?itn
percentile Is equal to + In 2. Then, In a sense, the nodel is unbiased
because the distribution of Lotted ratte* is centered i&oat zero, but 1s
also rather 1«prec!se» since the distribution of lo§5*s< ratios is fairly
broad.
-------
This point of view suggests that we alter our bootstrapped design values
by altering the distribution of logged ratios. That 1s, we can bias the
Model by shifting the distribution away from zero, or we cm make the
model more or less precise by multiplying legged ratios by =» constant that
Is less than or greater than unity.

What we have in mind, then, Is to alter the ratio (R} of monitored design
values («) L? predicted design values (p) to produce a new ratio
(If) such that

(- In W) « 6 + • « (- In R).

(The reason for the sinus sign Mill be made clear in a moment.) If we
agree to l*»»e the ssnltwretf design value unaltered, i.e., to alter bias
and imped si on by altering the predicted design value (p) alone, we
obtain:
where P is the altered version of p.

Tables S-! and B-2 shew thet the alters model will have a greater
tendency to underpredict (or less of a tendency to overpredict) if s is
negative, and will have a greater tendency to overpredict (or less of a
teftieitcy to airterpredfictj if s . To
Introduce an approximate 30 percent bias, we selected three values for
62
-------
TABLE 8-1. The balwwior of I for certain combinations of * and y.
8 < 0
& * 0
6 > 0
0 < v < 1

0-1

• > 1
- In 8 « a < 0

- In I < - In R

-In I = 0
fin R*i < | 1ft a j
. }n R « - In R
(In it > (in !lj
- In R » e > 0

- In R > . *,n H
1
TABLE B-2. The Behavior of p for certain cojnb.nttiens of „ ami y.
o < 0
B > 0
9*0

0 < « < i

^ * 1

* > i
P » C - B < »

S
p - 5 • P * P

p = S>
P closer than p to a
**w>
J* * p
p farther than p to m
- t»
D = e • m > m

~ i
p « e' • p > p

63
-------
Underprediction

8 < 0
(Jnplastdness

8 * 0-
Overpredfcticn
B > 0
Enhanced

Precision *;

* * 1 ft-
* - In 1.3 ,
15 •-
!! ~
8 * « o

| «.w
n
p
Isii
jai
Unaltered

Prtcision

»* 1
!»i:
> In 1.3 ,
p *'«
fe »• I
m
t - + In J.3 ;

* " I ;
Diminished
* « t
?!:
i!?:
•
&t
;i
+1
M i. •--.,
-i o +1
FI8«E B-l.
of -Is R for certain combinations of 8 and *.
-------
Underpredletion
g < 0
Unbiased
8-0
Over-prediction
Enhanced
Precision
* cO
I » 0
* - itz
*» i
Umlttrtd -
Precision
5 * 0 -
g - * If 1,2
•a:;-2t~.
Precision
/ • • • »R 1.3
*« 0
*-1
/ t . * If, 1.3

* « ?
?J~~&L>i
rf • A • i

B-I. seatttr dtttfiK of alter** s»de!-b*$ed design values
ftoHientil axis) wrsus »oniter-l»seci desffn values (vertical §x1s)
for c**t»in GOB&inations cf s »nd t-
6S
-------
6, namely, zero and ± It* 1.3. A change in precision by a factor of 2 also
seemed fie be • natural choice sc -*e have assigned $ the values of 1/2,
1, and 2. Swinery statistics for the resulting distributions of predicted
design values and design value differences are shown, respectively, in
Tables B-3 and 8-4.

To carry our distussions « step furtner, it is interesting to calculate
the effect of altering the bias and precision of a model on ratios, design
values, and eafssion limits. Table 8-5 shews these effects using the
notation of Section 3.

Since ratios are of observed divided bv predicted design values, an
increased tendency of tne nodel to overestimate raonitared design values
leads to smaller ratios and. conversely, an increased tendency to
underestimate leads to larfer ratios. Tn«s, if mode, predictions art
altered by a factor of es» these ratios are altered' by a factor of
«**. Also, ttw distribution of the altered ratios will be narrower than
the
-------
TWLI B-3. SuBBiry statistics for predicted design values (MDV1PR) for
certain combinations of e and *.
1/2
*- I
I « - 1ft 1.3
8-0
8 » * in 1.3
Standard
Mean Deviation
263.10 20.89S
263,3? 31,305
2SS.S? 71.225
Standard
Msan Deviation
342.03 27.1S3
342,31 40.701
test 97 «J5,??*>
1
Stasdard
Heart Deviation
«*4.6S 3S.320
445,10 52.911
4S4.0S 120. 54
TWlt 8-4, Suanary statistics for design value iifftrences (TOV1PR-MOV108)
for certain combinations of • and f.
6 » - IT 1.3
g » 0
s * * m 1.3
*« 2
Standard
Nean Be vi at ion
. »*Si
- W.32 *6.4§
- ?i.0t MK2t
Standard
Mean Qeviatiae
- t.SS
- l»3i 53.04
+ 5.S8 I 10.40
Standars
Mfcin Deviation
* tw»M IS.5S |
* 101.41 62.65
* 113.37 137.18
-------
TABLE ft-f,, Th« *ff*et of «1tcrtfl|! tn« bi«$ and precision of a node)
an ra».1o», tJestgrv valuer,, ami million Mmlti,

ftltfos of ohttrvttd-
vtlut*
Otiffn y«fu*f

OeVcfVp'tlofi Original Model
ath quMitfl* of Ma
bootstrap dHtrt-
butfon
A
Crt.««£ . gn
-------
Putting these results together we obtain
or « R1 . or
ft 0
R* -
$0 that the distribution of the unknown Monitoring design value generates
by CUE 1t not affected by «ode1 Mas. Hurt is, CUE compensates far bias
•net produces the distribution that would be derived for an unbiased
•ode!. We tee, however, that the distribution is affeetea by tne alter-
ation of nodel precision. As the model Increases in precision (as $
approaches t»ro)» the distribution of R narrows «6sat «nitj, am! tne
distribution of the unknown worn tor Ing aeslgn value narrows aboat the
original, unaltered, predicted design value, S¥,
Mt aay also write
0¥* » R* • B*
9 w

« t**1 » R
tt *
R *• • 0V
a a
to obtain
i-* • «t
-------
Again, we see that the alteration of model bias daes not alter the
emission Units produced by CUE, However, as the model becomes more
precise (as * approaches zero), the emission limit, Q', needed to achieve
a desired probability, a, of NAAQS attainment converges to the emission
Unit obtained from current practice.

Hie emission linrit derived by current practice is. of course, affected by
the alteration of model bias, since
Thus, the more the model tends to owerpredict design values, the smaller
(less lenient) the emission limit becomes; conversely, the more the model
tends to underpredict, the greater (Bore lenient} the emission limit
becoews.

Thus, an important feature of CUE is that its emission limits compensate
for model bias, whereas those obtained by current practice do not.

Table S-6 Illustrates these effects for emission limits obtained from 1973
£11 ft* Cre*k data under the various alterations of model bias and
precision*
-------
TA81E t-i. 1973 Mission Units Stt by current policy and
policies designed to acMave SO and 90 percent probabilities
of MMQS attainment for different scenarios of model bias ana*
Imprecision.

Enhanced
Prtclifwi
C* - 1/2)
Unaltered
Precision
{*- 1)
01 ntnt shed
Precision
f* - I)
Underpredictlon
(8 <0)
20»?«8?
IS.9416
M.Sit6
20.768
IS.SQ1
13,207
20.763
15,821
in.iis
Urbiasedness
{S -0)
15»§75
15.S41
14,528
1S.975
15,901
13,207
15»S?5
15,821
10.915
Over predict ion
tS>0)
12,289
1S.M1
14,523
12.2W
15,901
13,207
12.289
1S.821
10,915
* Current policy
b SO percent policy
c 90 percwii policy
n
-------