United States Office of Air Quality EPA 450/4-84-006
Environmental Protection Planning and Standards February 1984
Agency Research Triangle Park NC 27711
__
oEPA Special Report,
Issues Concerning
The Use of Precision
And Accuracy Data
-------
EPA-450/4-84-006
February 1984
SPECIAL REPORT
ISSUES CONCERNING THE USE OF
PRECISION AND ACCURACY DATA
by
A. D. Thrall and C. S. Burton
Systems Applications, Inc.
101 Lucas Valley Road
San Rafael, CA 94903
In conjunction with the
Work Group on the Utilization of Precision and Accuracy Data:
U.S. Environmental Protection Agency
Office of Air and Radiation
Office of Research and Development
Office of Policy, Planning and Evaluation
Region 4
Region 5
Prepared for
U.S. ENVIRONMENTAL PROTECTION AGENCY
Office of Air and Radiation
Office of Air Quality Planning and Standards
Research Triangle Park, NC 27711
February 1984
U.S. Environmental Protection Agency
Region 5, Library (PL-12J)
77 West Jackson Bouievarjl 12th Floor
Chicago, II 60604-3590
-------
DISCLAIMER
This report has been reviewed by the Office of Air Quality Planning
and Standards, U. S. Environmental Protection Agency, and approved for
publication as received from Systems Applications, Inc. Approval does
not signify that the contents necessarily reflect the views and policies
of the U. S. Environmental Protection Agency, nor does mention of trade
names or commercial products constitute endorsement or recommendation for
use.
-------
PREFACE
This report represents the efforts of the Work Group on the
Utilization of Precision and*Accuracy (P&A) data, which was formed to
determine how we should utilize the P&A data relative to the regulatory
decision-making process. The P&A data is reported by the National
Aerometric Data Bank (NADB), along with the associated air quality.
The P&A data bank is maintained by the Environmental Monitoring Systems
Laboratory (EMSL) and contains the P&A data reported by the States to
the Regional Offices and EMSL.
The Work Group was formed in December 1982 and is composed of people
representing the Regional Offices, the Office of Research and Development,
the Office of Policy and Resource Management, the Office of Air Quality
Planning and Standards, and EPA consultants. The following individuals
are members of this Work Group:
N. Frank, MDAD, OAQPS (Chairman)
D. Brittain, Region 4
S. Goranson, Region 5
0. Puzak, EMSL, ORD
R. Rhodes, EMSL, ORD
H. Sauls, EMSL, ORD
W. Nelson, HERL, ORD
T. Matzke, OPRM
J. Warren, 0PM
D. Stonefield, CPDD, OAQPS
W. Cox, MDAD, OAQPS
S. Sleva, MDAD, OAQPS
J. Summers, MDAD, OAQPS
W. Hunt, MDAD, OAQPS
F. Smith, RTI (ORD contractor)
S. Burton, SAI
T. Thrall, SAI
The report would not have been possible without their technical
assistance and overall guidance.
-------
CONTENTS
Introduction 1
Description of Precision and Accuracy Calculations 3
What Precision and Accuracy are Currently Being
Attained Nationwide? 7
What Precision and Accuracy Data, and What Data Quality,
are Necessary for Trends or Attainment Determinations? 16
How Should Precision and Accuracy Data be Used for
(1) Quality Assurance and (2) Reported Statistics? 19
Are Presently Prescribed Precision and Accuracy Check
Frequencies Adequate to Assess Data Quality? 20
How Should Precision and Accuracy Measurements be Used to Screen
Out Anomalous Data Values, e.g., Gross Measurement Errors? 21
Should Measurements be Corrected in Accordance with the
Precision and Accuracy Results Before Being Reported? 23
Which of the Following Precision and Accuracy Data Should be
Required of Reporting Organizations and Stored in the EPA
Computer Fi 1 e? 24
Where Should Precision and Accuracy Results be Reported? 26
What Criteria can be Used to Determine the Cost Effectiveness
of Generating Precision and Accuracy Information? 27
What are the Components of Measurement Error or Measurement
Variation? Which of These Components are Reflected in the
Preci sion and Accuracy Data? 28
References 30
-------
INTRODUCTION
A cardinal element in the U.S. Environmental Protection Agency's
administration, review, refinement, and revision of its air quality
management, policies, and regulations is the record of ambient air
pollutant concentrations monitored and reported by state and local
agencies (referred to as reporting organizations). The data reside in the
National Aerometric Data Bank (NADB); they are used by the EPA and others
to identify trends in air quality, determine the attainment status of
geographical areas, assess the efficacy of possible revisions to ambient
standards, and for many other purposes.
Because of the importance of the reliability of these data, the state
and local agencies are required, as part of regulation 40 CFR 58 promul-
gated on 10 May 1979, to maintain a quality assurance program that entails
both quality control and quality assessment activities. Reporting
organizations assess data quality by conducting precision checks and
accuracy audits of the monitoring instruments (referred to as analy-
zers). Precision and accuracy summary statistics are stored in files
maintained by the EPA's Environmental Monitoring Systems Laboratory
(EMSL). In this report we discuss several issues concerning the utiliza-
tion of the precision and accuracy data described in 40 CFR 58 Appendix A,
"Quality Assurance Requirements for State and Local Air Monitoring
Stations (SLAMS)."
There seem to be two areas of activity in which knowledge of the
precision and accuracy of measured ambient concentrations, if used, could
be very important. First, in determining whether a site is or is not in
attainment of a National Ambient Air Quality Standard (NAAQS), it may be
of use to decision makers to know the extent to which a concentration
reported to be either above or below the standard is likely to be a result
830'+or 2 i
-------
of measurement error. Second, in setting NAAQS, it is of interest to
policy makers to judge the protection likely to be afforded by existing
and possibly revised ambient standards on either a national or regional
basis. Such a judgement may be influenced by measurement uncertainties.
The two activities are complementary: In the first activity a decision
maker may wish to avoid unjustly penalizing an organization for measure-
ment error, whereas in the second activity a policy maker is required by
the Clean Air Act to provide adequate protection by incorporating a margin
of safety to compensate for uncertainties, including those due to measure-
ments.
Our use of the terms "precision" and "accuracy" in this report should
be explained. In usual quality assurance parlance, the precision of a
measurement process or device refers to the repeatability or variability
of measurements under prescribed conditions, usually quantified as the
sample variance (or its square root, the sample standard deviation) of the
measurements. Accuracy refers to the closeness of the measurements to
some reference standard. Often a reference material, whose measure is
known, is used to challenge the measuring device or process. Measurement
accuracy is usually quantified as the sample bias of the measurements,
i.e., the difference between the average of the measurements and the known
correct value.
In this report, however, "precision" and "accuracy" refer to the
precision checks and accuracy audits, respectively, required under federal
regulations. We describe these requirements in the report, but the point
here is that the sample bias and variance are relevant quantities that are
computed for both the precision checks and the accuracy audits. Thus, to
avoid confusion, we have reserved the term "precision and accuracy data"
to mean the data collected under the program of precision checks and
accuracy audits.
In the next section we describe the precision and accuracy (P&A)
calculations and reported results. We then discuss several P&A issues.
Each issue is presented in the form of a question.
-------
DESCRIPTION OF PRECISION AND ACCURACY CALCULATIONS
Precision and accuracy checks currently required for State or Local
Air Monitoring Stations (SLAMS) and consequently for National Air Monitor-
ing Stations (NAMS) are described in Federal Register 44, 92: 27574-
27577. These checks are intended to serve as a basis for assessing, and
hence improving, the quality of monitoring data.
Table 1 depicts the concentration points at which biweekly precision
checks and annual accuracy audits are made. Table 2 shows the precision
and accuracy calculations that are made: p .j^ and Sjjjp) are the average
and standard deviation of the percentage differences p^ for the i^
precision check made during the j quarter on the k*" instrument
(analyzer); p ^ and si (p) are the average and standard deviation
J J *
obtained by pooling the K instruments within a reporting organization.
Similarly, a>m and sm (a) are the average and standard deviation obtained
from auditing the K instruments at the m* concentration level.
Typically, there are n = 6 or 7 biweekly precision checks within a
quarter. When this number varies across the instruments within a report-
ing organization, the pooled average and standard deviation are obtained
from weighted averages. For TSP, the flow rate, rather than the concen-
tration, is the audited quantity; each high-volume sampler is audited once
per year.
An exception to the precision calculations presented in Table 2
occurs when precision is estimated from a pair of collocated instruments,
rather than from challenges to a single instrument by a reference material
of known concentration. This procedure, referred to as a manual method,
is followed for instruments that intermittently monitor particulate
matter, lead, or other air pollutants. (The other procedure, referred to
as an automated method, is used for continuous analyzers.) In this case,
83040T 2 3
-------
TABLE 1. Concentration points (in parts per million,
ppm) for precision checks and accuracy audits.
S02, N02, 03CO
One-point precision 0.80 - 0.10 8-10
check once every two weeks
Four-point accuracy 0.03-0.08 3-8
audit once per year 0.15 - 0.20 15 - 20
0.40 - 0.45 40 - 45
0.80 - 0.90 80 - 90
8301+0 3
-------
TABLE 2. Precision and accuracy calculations.
Indexes
i - itn precision check
j - jt" quarter
k - k instrument (analyzer)
m - mth concentration level
Measured and Reference Variables
Y = Concentration (or flow rate in the case of TSP) as measured by
the instrument
X = Known reference concentration (or flow rate)
Precision Calculations
pi,k - 100 x (vijk - xijk)/xijk
p.jk = l & pijk sjk(p) =
K
P.J. = ]< 2 P.jk Sj.(p) =
" i " 2'
-rrr 2 (p.jk-p.j.)2"
1 1/2
ll/2
Estimated 95% lower (L) and upper (U) probability limits for precision:
Lj{p) = P-j-" U
Uj(p) = P-j-* 1>
Accuracy Calculations
akm
a.m
= 100 x (Y. - X. )/X,
4 I\ -IX A
= ^ I a, s (a) =
K ^ km m^
1 V /a , \t
v r / v3i_ - a }
N - 1 i/1--! icm .m
11/2
Estimated 95% lower (L) and upper (U) probability limits for accuracy:
Lm(a)*ail,-1.96sm(a),
-------
one of the collocated instruments is designated as the ambient monitor
(symbolized by Y in Table 2), and the other is designated as the reference
(X). The calculations are modified by dividing the standard deviation
s. (p) by / 2, to allow for the fact that the observed imprecision is due
J
not only to the ambient measurement but also, in part, to imprecision in
the reference concentration.
Several aspects of the reported P&A data are worth noting. The only
P&A data that are required to be reported to the EPA are summary
statistics consisting of the number of precision checks (n), accuracy
audits (K), and the upper and lower 95% probability limits for precision
and accuracy given in Table 2. Also, the reported probability limits
pertain to the reporting organization rather than a specific instrument
within a reporting organization. Thus, if we were to apply the probabil-
ity limits to a specific monitor, we must assume that measurement errors
are similar among the monitors within a reporting organization.
Generally, reporting organizations have been formed from a collection of
sites whose monitoring practices are reasonably homogeneous. There are,
however, possible exceptions. In contrast to, for example, the State of
New York, which constitutes one reporting organization made up of a
collection of operating organizations, the State of Florida consists of 13
fairly homogeneous reporting organizations each consisting of one
operating organization. Finally, the calculation of the probability
limits follows from an assumption that measurement error, expressed as a
percentage of the actual value (concentration or flow rate) is normally
distributed.
-------
ISSUE 1
Issue. What precision and accuracy are currently being attained
nationwide?
Discussion: From Figures 1-8, we see that few reporting organizations
have measurement errors (95 percent probability limits) of more than 25
percent for automated (continuous) ambient monitors. Somewhat greater
error limits are reported for manual (integrated) monitors. In addition,
the accuracy audits show that percentage errors tend to decrease as the
audited concentration increases.
Figures 1 through 8 illustrate the precision and accuracy attained by
monitors of sulfur dioxide (Figures 1 and 2), ozone (Figure 3), carbon
monoxide (Figure 4), nitrogen dioxide (Figures 5 and 6), total suspended
particulate (Figure 7), and lead (Figure 8). Each figure consists of
parts (a) through (d), which in the notation of Table 2 shows the fre-
quency distribution across reporting organizations of L.(p), U.(p),
Lm(a), and Um(a), respectively.
-------
J3
30.
25.
20.
15.
10.
o
t 5.
01
c 0>
0)
£ -5.
CD
"-10.
-15.
-20.
-25.
-30.
-^c;
i i i i i | i | i i i i i | i | i i i
-
|-
r
-
>-
-
- X
-
-
i i i i i i i I i i i i i I i i i i i
J.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
concentration (ppm)
(a) precision - lower prob. limit.
30.
25.
20.
15.
10.
o
!: 5.
c 0-
o
if -5.
a
"-10.
-15.
-20.
-25.
-30.
T
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
concentration (ppml
(b) precision - upper prob. limit
33 .
30.
25.
20.
15.
10.
0
t 5.
01
-* 0.
c
c_ -5 .
"-10.
-15.
-20.
-25.
-30.
i i | i | i | i | i | i | i | i -
-
-
-
-
-
-
_ ^_
|
J-
X
*,
X X I
x 1
-
-
-
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 _l 1
.0 0.1 0.2
0.3 8.4 0.5 0.6 0.7 0.8 0.9 1.0
concentration (ppm)
35.
30.
25.
20.
15.
10.
5.
0.
' -5.
L-10.
-15.
-20.
-25.
-30,
-35.,
I r
(c) accuracy - lower prob. limit
.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
concentration (ppm)
(d) accuracy - upper prob. limit
FIGURE 1. Distribution of sulfur dioxide precision and accuracy (percentage error,
automated analyzer) attained by reporting organizations, 1981.
-------
1 PI 01
1 Kl 10 t
90.
80.
70.
60.
50.
10.
30.
20.
10.
0.
-10.
-20.
-30.
-10.
-50.
-60.
-70.
-80.
-90.
_ t affl
-
-
-
-
-
-
_
-
_
_
-
_
-
-
r
I
X
KEY
I "
- 5( percent lt«
V:r.
1 ffld . .
-
-
-
-
-
-
_
_
-
_
-
-
-
-
I
90.
80.
70.
60.
50.
40.
30.
o 20.
t-
o 10-
-* 0.
c
5 -10'-
o -20.
a.
-30.
-10.
-50.
-60.
-70.
-80.
-90.
-10IR
-
-
-
X
-
-
-
-
-
_
-
-
-
-
- .
for concentrations of 0.015 ppm and above
for concentrations of 0.015 ppm and above
(a) precision - lower prob. limit
(b) precision - upper prob. limit
30.
25.
20.
15.
O
*" R
QJ
^ 0.
C
O)
t~ ~5 ,
QB
a-l0.
-15.
-20.
-25.
-30.
i
-
-
-
-
_
-
_
-
_
-
i
X
1
1 1
X
1 '
1 1
1
X
1
1
1 1 '1 1
-
-
-
-
.
-
_
-
-
-
1 1 1 1
JO.
30.
25.
20.
15.
O
(_ b .
01
-» 0.
c
at
" -5.
0)
-15.
-20.
-25.
-30.
-I < | i i i | 1-
-
-
-
i-
x
-° X
1 I
-
_ _
-
_ _
-
1 I 1 1 1 1 1 1
.00 0.02 0.01 0.06 0.08
concentration (ppm)
(c) accuracy - lower prob, limit
.00 0.02 0.01 0.06
concentration (ppm)
0.08
(d) accuracy - upper prob, limit
FIGURE 2. Distribution of sulfur dioxide precision and accuracy (percentage error,
manual method) attained by reporting organizations, 1981.
-------
KET
, - 90
rr7'
- 50 ptrcinlll*
Lpl. 90
30.
25.
20.
15.
10.
S 5.
t.
a>
percent
i i
m a ui is
-20.
-25.
i | r- | i | - i - | i ] i | i | i | i | i
-
-
- 1
X
1-1
1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 , 1 ! 1 1
30.
25.
20.
15.
10.
£ 5.
OJ
£ 0.'
06
5-5-
a.
-10.
-15.
-20.
-25.
-qn
1 - 10
i 1 i 1 i 1 i 1 i i i i i i i i i t i
-
- 1
L X
" 1
-
-
-
i i i i i t I i I i i i i i i , i i
i.0 0.1
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
concentration (ppm)
0.0 0.1 0.2 0.3 0.1 0.5 0.6 0.7
concentration (ppm)
0.8 0.9 1.0
a) precision - lower prob. limit
(b) precision - upper prob. limit
25.
20.
10.
0.
-5.
-10.
-15.
-20.
-25.
qn
_
-
-
i
(
1
1
r" T
X
T
i
1 1 1
j 1 1
X
1 | 'I | 1 | 1 -|
1 1,1 1 1 1 ... 1 1
'
X
1
1
1 '
-
-
~
_
-
-
-
-
1 1
25.
20.
10.
C_
-« 0.
c
Q>
(J
u -5 .
a.
-10.
-15.
-20.
-25.
i I i ' i i i i i i i i ' i i i i
-
-
X
1
-
-
-
-
-
l 1,1. i l , l i l i i , i ,
concentration (ppm!
concGntration (ppm)
(c) accuracy - lower prob. limit
(d) accuracy - upper prob. limit
FIGURE 3. Distribution of ozone precision and accuracy (percentage error,
automated analyzer) attained by reporting organizations, 1981.
-------
30.
25.
20.
15.
10.
o
L- c
01
c 0-
Q}
" -5.
GJ
-10.
-15.
-20.
-25.
-30.
(
30.
25.
20.
15.
10.
C_
D
t 5.
C
Q)
°--i0
-20.
-25.
-30.
B
1 i 1 i 1 i 1 i 1 > 1 > 1 i" 1 i | i i i
-
- X
1 1 1 1 I 1 1 1 1 1 1 1 1 1 1 1 1
10. 20. 30. 10. 50. 60. 70. 80. 90. 10
concantrat ion (ppm)
a) precision - lower prob. limit
-
~
1 ,-L,
X X
1
1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
10. 20. 30. 40. 50. 60. 70. 80. 90. 10
30.
25.
20.
15.
10.
0
t, 3
o
-g 0.
Q)
U £-
0}
-15.
-20.
-25.
-30.
0. 35la
30.
25.
20.
15.
10.
t_
O
t 5.
C
U c
C. '->
Q>
°"-10
-20.
-25.
-30.
0. "35T3
KEY
1 ' M
».l . , 7«
T^ M
1 - 18
i i i i i i i i i | i | i | i | i
:1
X
~ 1
_
ill 1 1 1 1 1 1 1 1 1 1 I 1 1 1 1
10. 20. 30. 40. 50. 60. 70. 80. 90. 10
concentration tppm)
(b) precision - upper prob. limit
i i i ' ' i > > i i ; ' i >
1
-x 1
X
l X
X
r V _
i i i i i i i i i i i i i.i
10. 20. 30. 40. 50. 60. 70. 80. 90. 10C
concentration (ppm!
(c) accuracy - lower prob. limit
(d) accuracy - upper prob. limit
FIGURE 4. Distribution of carbon monoxide precision and accuracy (percentage error,
automated analyzer) attained by reporting organizations, 1981.
-------
OKJ .
80.
70.
60.
50.
40.
30.
o 20.
-p 0.
S-i0.
£-20.
-30.
-40.
-50.
-60.
-70.
-80.
-an
i 1 l i l i l i 1 i l i i i I i 1 i
-
-
-
-
-
-
: . :
'-i
-
r
-
-
-
",,,,,,, i , i ,,,,,,, i ,"
1.0-
I
i-
KEY
- 98
- 78
- SB ptrctntll*
- 38
- 18
"0.0 0.1
0.2 0.3 0.4 0.5 0.6 0.7
concentration (ppm)
SB .
80.
70.
60.
50.
40.
30.
o 20.
U
m 10.
-*> 0.
S-i0.
£-20.
-30.
-50.
-60.
-70.
-80.
i 1 i 1 i i | i 1 i 1 i 1 i 1 i 1 i
-
_
-
L-
-
- 1
- B
-
-
-
~
-
-
, i , i i i i i , i i i i i i i i i i
0.8 0.9 1.0
(a) precision - lower prob. limit
(b) precision - upper prob. limit
3».
80.
70.
60.
50.
40.
30.
S 20.
£ 10-
-£ 0.
S-10.
S.-20.
-30.
-40.
-50.
-50.
-70.
-80.
-aa
I I | I | -I | i" r i ' 1 -r | r T I-T~"
-
-
-
_
"ri. rS ^ * "
M LXJ
~ x I 1
_ _
-
-
-
i i i i i i i i i i i i i i i i i i
3B.
80.
70.
60.
50.
40.
30.
S 20.
I IB-
a 0.
S-10.
£-20.
-30.
-40.
-50.
-60.
-70.
-80.
Qd
1 i t i | i - [ i | i | i | i | i | i
-
-
-
-
X 1
v . \-
"T -1-
-
-
_
III I 1 1. 1 1 ..1 1 1 L . 1 1 1 i 1 1
concentration (ppm)
concentration (ppm)
(c) accuracy - lower prob. limit
(d) accuracy - upper prob. limit
FIGURE 5. Distribution of nitrogen dioxide precision and accuracy (percentage error,
automated analyzer) attained by reporting organizations, 1981.
-------
O
l_
QJ
-P
C
CD
U
Q>
d
1WB.
90.
pa
O 10 .
70.
60.
50.
40.
30.
20.
-
-
-
-
-
.-
-
10. j-
0.
-10.
-20.
-30.
-40.
-50.
-60.
-70.
-80.
_
X ~
-
-
-
-
-90. i-
i ma I
for concentrations of 0.016 ppm and above
(a) precision - lower prob. limit
100.
90.
80.
70.
60.
50.
40.
30.
20.
10.
-20.
-30.
-40.
-50.
-60.
-70.
-80.
-90.
-100.
KEY
- 98
- 7«
- SB ptrctnillt
- 38
- IB
for concentrations of 0.016 ppm and above
(b) precision - upper prob. limit
20.
15. -
10.
-p 0. -
-10.
-15.
. 00
1 |
I
i.02
0.04 0.06
concentration (ppm)
(c) accuracy - lower prob. limit
20.
15.
10.
5.
0.
-10.
-15. -
-20,.
'. 00
T
1 1
0.02 0.04 0.06
concentration (ppm)
B.i
(d) accuracy - upper prob. limit
FIGURE 6. Distribution of nitrogen dioxide precision and accuracy (percentage error,
manual method) attained by reporting organizations, 1981.
-------
30. .
25.
20.
15.
10.
5.
0.
-5.
1 n
ID.
15.
28.
25.
qa
-
-
_
_
-
KET
I 9B
rn 7<
r
"~*~ ~ 39 percent ))
MM
1 " "
1 - 10
_ . i n
-
X
-
-
3U .
25.
20.
t *\
i 3 .
1 01
I U .
\ 5-
Q)
-^ 0.
(J
Q.
- 1 0,
-15.
-20.
-25.
3a
-
-
X
-
~
-
-
-
-
JU . " ' ~ ' ~ ~ ' ~ ~ ~ ' u U . ' ' '
for concentrations of 20 ug per cubic meter and above for concentrations of 20 ug per cubic meter and abov
(a) precision - lower prob. limit
(b) precision - upper prob. limit
o
c_
i_
0>
c
Q>
U
0)
a
OB.
25.
20.
15.
10.
5.
0.
-5.
-10.
-15.
-20.
-25.
-3%
l 1 l i i i i i i
-
-
L-
-
-
-
1
-
T
-
-
i i i i i i i i i
30
25
20
15
10
t_
E 5
-** 0
c
OJ
£ -5
0.
-10
-15
-20
-25
10. 20. 30. 40. 50. 60. T0- 80. 90. 100. -30
f 1 oy rate (cf m)
1 r
(c) accuracy - lower prob. limit
19. 10. 20. 30. 10. 50. 60. 70. 80. 90. 100
f I ow rate (cfm)
(d) accuracy - upper prob. limit
FIGURE 7. Distribution of total suspended particulate precision and accuracy (percental
error, manual method) attained by reporting organizations, 1981.
-------
100.
90.
80.
70.
60.
50.
30.
o 20.
^ 10.
£ 0.
o " 1 o .
o -20.
a.
-30.
-40.
-50.
-60.
-70.
-80.
-90.
-100.
fo
25.
20.
15.
10.
o 5.
at
-^ 0 .
c
CD
CJ
1_
Cl
-10.
-15.
-20.
-
I 1 I
:
concentrations of 0.15 ug per cubic meter and ab
(a) precision - lower prob. limit
-
X
X
1
, I 1 1 1 1 1 1
100.
90.
80.
70.
60.
50.
40.
30.
o 20.
£ 10-
^ 0.
S -10.
co -Z0.
a.
-30.
-40.
-50.
-60.
-70.
-80.
-90.
-100.
3V8 f 0
25.
20.
15.
10.
o 5.
CO
£ 0.
CD
0
co -5.
a.
-10.
-15.
-20.
KET
_L " 9"
ri " 7i
V'3'
1 - IB
-
-
-
concQntrat ions of 0.15 ug per cubic meter and aba
(b) precision - upper prob. limit
-
X
1
1
1 1 1 1 1 1 1 !
.0 1.0 2.0 3.0 4.0
concentration (ug per cubic meter!
(c) accuracy - lower prob. limit
5.0 " 0.0 1.0 2.0 3.0 4.0
concentration (ug per cubic meter!
(d) accuracy - upper prob. limit
FIGURE 8. Distribution of lead precision and accuracy (percentage error, manual method)
attained by reporting organizations, 1981.
-------
ISSUE 2
Issue. What P&A data, and what data quality, are necessary for
trends or attainment determinations?
Recommendation: At a minimum, an estimate of the average and standard
deviation of instrument-specific precision checks are needed to support
the accuracy of attainment determinations. Provided the measurement error
for an instrument is within a range of ±20 percent, measurement impre-
cision is not currently believed to adversely influence attainment
determinations. In contrast, the currently available 95 percent probable
error range for reporting organizations may be adequate for trends
determination, and acceptable measurement error may here be as much as
100%.
It is recommended that organization-wide measurement error, reflected
by the upper and lower 95 percent probability limits for precision and
accuracy, be held within the range of ± 25 percent for automated analy-
zers. Additionally, it is recommended that NAAQS attainment be determined
directly from recorded ambient concentrations, without adjustment for
measurement error. These recommendations are subject to review by the
Standing Air Monitoring Work Group (SAMWG).
Discussion; In a simulation study, Curran, Stiegerwald, and Burton (1981)
show that measurement imprecision of 20 percent or less may cause measured
annual second-highest concentrations to be biased upward by as much as 5
percent, while more robust air quality indicators used in trends
determinations are more resistant to measurement error (seasonal averages
were shown to be practically unaffected by as much as 100 percent measure-
ment imprecision). Moreover, instrument-specific information is needed to
-------
interpret concentration extremes relevant to short-term standards, whereas
trends are typically averaged over monitors in a region, so that instru-
ment-specific information may not be necessary.
Thus, the overriding requirements for both the type of P&A data and
the quality of ambient data are for attainment determinations. We note
that reporting organizations are currently required to maintain instru-
ment-specific P&A records for three years, and they are encouraged to keep
permanent instrument-specific records. As discussed in Issue 1, holding
measurement error of automated analyzers to ± 25 percent is an attainable
goal.
There has been considerable discussion of whether and how to use P&A
data in determining NAAQS attainment. One possibility, for example, would
be to construct 95 percent confidence limits for the actual annual ambient
concentration, and to judge at site to be in attainment of the annual
standard unless the entire confidence interval lay above the annual NAAQS
concentration. The confidence interval would be computed as
where C is the computed annual average concentration, U and L are the
upper and lower 95 percent probability limits based on precision checks,
and n is the number of terms used to compute the annual average concentra-
tion. The interpretation of this interval as a 95 percent probability
interval is based on the assumption that the annual average is approxi-
mately normally distributed, an assumption that can be justified without
assuming that individual measurement errors are normally distributed. In
the case of short-term standards, however, it is essential that the
distribution of individual measurement errors be known, at least approxi-
mately, in order for us to determine the appropriate use of P&A data in
the assessment of short-term NAAQS attainment.
An additional difficulty in using P&A data in NAAQS attainment
determinations is that such policies may result inadvertently either in
more lenient determinations for the reporting organizations having the
830fOT 2 17
-------
largest measurement errors, or in unjustly stringent determinations for
all reporting organizations. If, however, no P&A adjustment is made in
assessing attainment, reporting organizations have a natural incentive, in
addition to explicit requirements, for maintaining data reliability, for,
as Curran, Steigerwald, and Burton (1981) show, measurement imprecision
tends to result in the highest measured concentrations being higher than
the actual ambient concentration.
As pointed out in the 1983 EPA document Guideline on the Meaning and
Use of Precision and Accuracy Data Required by 40CFR58? Appendices A and
JJ, the precision checks and accuracy audits conducted by reporting
organizations serve as one component of the quality assurance programs.
At a local level, P&A data enable reporting organizations to identify
aspects of their quality assurance programs that may need strengthening.
The P&A data also enable the EPA to determine the steps that may need to
be taken to improve the quality of ambient data, such as additional
research on measurement procedures, increased quality control for certain
types of measurements, or technical assistance to some areas of the
country needing improved quality control. The checks and audits have not
been designed, however, to yield data directly applicable to the
determination of the attainment of ambient standards. For this purpose
appropriate averaging times, spatial representativeness, and other
components of error or variation would have to be considered.
saotor 2 18
-------
ISSUE 3
Issue. How should P&A data be used for (1) quality assurance and (2)
reported statistics?
Recommendation: The primary purpose for the P&A program is to assist in
the quality control of concentration measurements. The program enables
reporting organizations to identify occurrences of unacceptably large
measurement errors (e.g., more than 25 percent for automated analyzers)
and to tighten up quality control procedures as necessary. Additionally,
the P&A data roughly indicate the reliability of the data that are used
for the analysis of trends or alternative regulatory policies. The P&A
data are not, and should not be, used to alter concentration values stored
in the NADB, but the NADB maintains special reader comment files where
changes in measurement procedures may be reported.
Discussion: The precision and accuracy (P&A) activities supplement the
quality control (QC) activities of reporting organizations. The P&A and
QC components comprise the quality assurance (QA) program. As part of the
quality control activities, instruments are regularly calibrated to
minimize measurement error. The P&A phase of the program serves to inform
all concerned as to the data quality being achieved.
Additionally, the user of ambient concentration data stored in the
NADB has access to the corresponding P&A data maintained by the EMSL, so
that informed judgments may be made as to the reliability of ambient air
quality data. Note however that the P&A activities are not designed for
this additional purpose: P&A data describe only a portion of the measure-
ment error associated with ambient air quality data.
-------
ISSUE 4
Issue. Are presently prescribed P&A check frequencies adequate to assess
data quality?
Recommendation; The current practice of conducting precision checks once
every two weeks and accuracy audits once per year appears to be ade-
quate. Note that reporting organizations having fewer than four analyzers
for a given pollutant are required to audit a randomly selected analyzer
each calendar quarter (so that each analyzer is audited once or more per
year).
Discussion: There are essentially two considerations in deciding how
frequently to conduct the precision checks and accuracy audits, namely,
cost and quality assurance. The frequency of precision checks necessary
to assure data quality has been determined by the Environmental Monitoring
Systems Laboratory (EMSL) to be once every two weeks, based on knowledge
of instrument drift (how rapidly an instrument goes out of calibration)
and other factors. The frequency of the more costly accuracy audits have
similarly been determined to be once per year per instrument (and a
minimum of one audit per quarter) based on both cost and quality assurance
considerations.
-------
ISSUE b
Issue. How should P&A measurements be used to screen out anomalous data
values, e.g., gross measurement errors?
Recommendation: Current EPA screening procedures appear to be an adequate
safeguard against gross outliers. P&A data at the reporting organization
level can be used to screen out groups of values that resulted from
unacceptably imprecise or inaccurate instruments.
Discussion: Methods for the detection of gross outliers and other data
anomalies are presented in the 1978 EPA guideline document Screening
Procedures for Ambient Air Quality Data. These automated procedures are
most valuable, but may fail to detect biased, yet self-consistent,
measurements that could be checked at the reporting organization level.
In flagging suspect measurements, measurement error should be kept in
mind. If a monitored concentration is suspiciously low or high, but the
P&A data show that the anomaly may be an instance of extreme measurement
error, the question turns from the validity of a single reported concen-
tration to the validity of a set of reported concentrations that were
subject to large measurement error. If, on the other hand, the monitored
concentration cannot be explained by measurement error, based on P&A data,
the value may be excluded from future calculations, the grounds for
exclusion being a probable error in something other than the measurement
process, e.g., a data transcription error.
Consider, for example, three colocated instruments whose precision
checks reveal that the instrument-specific averages agree, but that one
instrument is much more variable than the other two. An implausibly large
concentration from the two less variable instruments would suggest that a
21
-------
transcription or keypunch error may have occurred, whereas the same
concentration from the highly variable instrument might be in the realm of
measurement error. Thus in the case of a highly variable instrument, it
may be more appropriate to question the validity of the set of readings
from that instrument, rather than to focus on a single high value. Under
current practice, comments about data reliability are recorded in reader
comment files maintained by the EMSL.
Finally, it should be emphasized that the primary purpose of the
precision checks and data audits is not to assess the validity of ambient
measurements, but to assess and guide quality control procedures.
22
-------
ISSUE 6
Issue. Should measurements be corrected in accordance with the P&A
results before being reported?
Recommendation: Measurements should never to be corrected on the basis of
P&A data.
Discussion: In principle NADB data should agree with data from the state
from which they were obtained. As an illustration, the EPA and the State
of California agreed in revising recorded ozone concentrations downward
and in not revising nitrogen dioxide concentrations. In both instances,
the state noted biases in the calibration procedure for ozone and nitrogen
dioxide monitoring instruments; however, the state revised only the ozone
concentrations (since records permitted a precise estimate of calibration
bias) and not the nitrogen dioxide concentrations.
If an instrument were subject only to a constant systematic bias and
not to imprecision, then the detection and quantification of this bias
would allow us to obtain actual concentrations, uncontaminated by
measurement error, by making a bias correction. Under these circumstances
it would seem that P&A results need not be reported once appropriate bias
corrections were made. Realistically, however, an instrument is subject
to some imprecision, as well as bias, and the estimation of an appropriate
bias correction is itself subject to error. So we may infer the likely
range of the actual concentration corresponding to a measurement only if
we know the likely range of measurement error. Since P&A results roughly
indicate the level of measurement error, they need to be available even if
state or local agencies make bias corrections.
23
8301+or 2
-------
ISSUE 7
Issue. Which of the following P&A data should be required of reporting
organizations and stored in the EPA computer file?
1. Raw data for each instrument
2. Number of measurements, average, standard deviation; for each
instrument
3. The currently reported 95 percent probability limits for each
reporting organization
Recommendation: Current practice (the third option) may be adequate,
provided that each reporting organization is comprised of a homogeneous
set of monitors such that the pooled P&A results provide reliable esti-
mates for each monitor.
Discussion: Note that the average (x) and standard deviation (s) for a
reporting organization may be calculated from the upper and lower 95
percent probability limits (U and L), and vice-versa, i.e.,
U = x + 1.96 s
L = x - 1.96 s
so that
x = (U + L)/2
and
s = (U - D/3.92
24
-------
Thus the choice of (U, L) versus (x, s) is not a major issue. Some
preference may be given to the (x, s) format because the interpretation of
U and L as 95 percent probability limits follows from the possibly
unwarranted assumption that measurement error is normally distributed.
What is at issue is the availability of instrument-specific P&A
results, so that, for example, more informed attainment determinations
pertaining to a specific monitor may be made. Currently, reporting
organizations are required to maintain instrument-specific P&A results for
three years, and they are encouraged to maintain permanent records.
The current organization-wide reporting may be sufficient provided
that there are no large differences between instrument-specific P&A
results within a reporting organization. It is possible, for example, for
there to be substantial systematic measurement errors (biases) for several
instruments within a reporting organization, but that these biases cancel
out in the calculation of organization-wide measurement bias. Such
inhomogeneity would inflate the organization-wide standard deviation,
thereby making the attainment of the ±25 percent P&A goal difficult. This
situation could lead to an erroneous interpretation of concentrations
recorded by a specific instrument.
Thus, current practice demands a degree of homogeneity among the
instruments within a reporting organization. The alternative is to
require instrument-specific P&A reporting (the second option).
25
-------
ISSUE 8
Issue* Where should P&A results be reported?
Recommendation: P&A summary statistics currently reside in files main-
tained by the Environmental Monitoring Systems Laboratory (EMSL), as
appropriate. In EPA documents concerning monitored concentration data,
the average and standard deviation of the percentage errors obtained from
precision checks could be reported parenthetically if, in the judgment of
the data user, measurement error were pertinent to the discussion and
adequately described by the P&A results.
Discussion; One possibility under consideration is to automatically
supply P&A results and the reader comment files to those people requesting
ambient concentration data. This would allow the greatest flexibility in
the treatment of measurement error when interpreting calculations based on
monitored concentrations.
26
-------
ISSUE 9
Issue. What criteria can be used to determine the cost effectiveness of
generating P&A information?
Recommendation: The benefit of quality assurance programs (which include
the collection of P&A information used to guide quality control activi-
ties) is greatest for those sites in potential nonattainment of NAAQS.
Considering possible economic consequences to an urban area due to an
erroneous determination of NAAQS nonattainment based on faulty data, the
benefit of reliable data may be on the order of tens to hundreds of
millions of dollars, whereas the cost is on the order of thousands to tens
of thousands of dollars.
Discussion: A detailed formulation of the marginal cost and benefit of a
quality assurance program as a function of expected levels of ambient
concentration and potential consequences of NAAQS nonattainment is beyond
the scope of the present report. Nevertheless, it is clear from the work
of Curran, Steigerwald, and Burton (1981) that those sites near the NAAQS
limit deserve the most quality assurance (including P&A) effort, if
limited funding requires that some sites be given more attention than
others. At such sites, each additional increment of imprecision leads to
an increased chance that the area would be decreed to be not in attainment
when in fact it is.
830HOr 2
-------
ISSUE 10
Issue. What are the components of measurement error or measurement
variation? Which of these components are reflected in the P&A data?
Discussion: To a limited extent the P&A data describe measurement error,
but they do not describe spatial and temporal variation in actual air
pollutant concentrations. Presumably, the latter factors comprise most of
the variation in concentration measurements, but further investigation
would be needed to verify and quantify this. The P&A data do not account
for errors in sample collection, i.e., "scrubbing" by sampling lines and
leaks in sampling lines.
The P&A data can be used to gauge measurement error for a single
instrument and to gauge the variation of instrument accuracy within a
reporting organization. The P&A data do not reflect the spatial repre-
sentativeness of a site measurement, nor do they indicate temporal
variation in concentration levels at a site. Also the P&A data do not
reflect sample collection and sample handling errors. These errors are
caused by adsorption and absorption of acid gases on sample tube walls and
sample tube contaminants; they may also be caused by irreversible chemical
reactions of reactive gases on sample tube walls and sample tube con-
taminants.
Measurement error and concentration variation can be distinguished, for
example, in the components of variation about an annual average concentra-
tion representing a large region, obtained by averaging the calculated
annual averages at several monitors. Variation of individual concentra-
tion measurements about this spatial-temporal average are accounted for,
to some extent, by measurement error. The spatial and temporal variation
of actual concentrations probably account for most of this variation,
830401" 2 28
-------
however. Sample collection and sample handling errors can be estimated by
challenging instruments through the sampling line rather than bypassing
the sampling line.
?Q
83040T 2. "
-------
REFERENCES
Curran, T. C., B. J. Steigerwald, and C. S. Burton. 1981. "Establishing
Data Quality Acceptance Criteria for Air Pollution Data." Paper
presented at the 35th Annual Conference of the American Society for
Quality Control, San Francisco, California, May 1981.
EPA. 1978. "Screening Procedures for Ambient Air Quality Data." U.S.
Environmental Protection Agency. (EPA-450/2-78-037).
EPA. 1983. "Guideline on the Meaning and Use of Precision and Accuracy
Data Required by 40 CFR Part 58, Appendices A and B." U.S.
Environmental Protection Agency, Research Triangle Park, North
Caroli na. (EPA-600/4-83-023).
830J+OT 2 30
-------
TECHNICAL REPORT DATA
/Please read Instructions on the reverse bejort' completing}
1 REPORT NO.
EPA 450/4-84-006
3. RECIPIENT'S ACCESSION NO.
4. TITLE AND SUBTITLE
Special Report, Issues Concerning the Use of Precision
and Accuracy Data
6. PERFORMING ORGANIZATION CODE
5 REPORT DATE
February 1984
7. AUTHOR(S)
A. D. Thrall and C. S. Burton
8. PERFORMING ORGANIZATION REPORT NO
9. PERFORMING ORGANIZATION NAME AND ADDRESS
10. PROGRAM ELEMENT NO.
Systems Applications Inc
101 Lucas Valley Road
San Rafael, CA 94903
11. CONTRACT/GRANT NO.
12. SPONSORING AGENCY NAME AND ADDRESS
U. S. Environmental Protection Agency
Office of Air and Radiation
Office of Air Quality Planning and Standards
Triangle Park. NC 27/11
13. TYPE OF REPORT AND PERIOD COVERED
Work Group Report
14. SPONSORING AGENCY CODE
15. SUPPLEMENTARY NOTES
16. ABSTRACT
This reports represents the efforts of the Work Group on the Utilization of
Precision and Accuracy (P&A) data, which was formed to determine how P&A data
should be utilized relative to the EPA regulatory decision-making process. The
report discusses ten issues concerning the utilization of P&A data and summarizes
the calculations specified in 40 CFR 58 Appendix A, "Quality Assurance Requirements
for State and Local Air Monitoring Stations (SLAMS)." P&A data is reported by
the National Aerometric Data Bank (NADB), along with the associated air quality.
The P&A data bank is maintained by the Environmental Monitoring Systems Laboratory
(EMSL) and contains the P&A data reported by the States to the EPA Regional
offices and EMSL.
17.
KEY WORDS AND DOCUMENT ANALYSIS
DESCRIPTORS
b.lDENTIFIERS/OPEN ENDED TERMS Ic. COSATI Field/Group
Precision
Accuracy
Quality Assurance
Measurement Error
Statistics
Air Quality Data
18. DISTRIBUTION STATEMENT
Release Unlimited
19. SECURITY CLASS (ThisReportJ
Unclassified
21. NO. OF PAGES
31
20, SECURITY CLASS /This page}
Unclassified
22. PRICE
EPA Form 2220-1 (Rev. 4-77) PREVIOUS EDITION is OBSOLETE
31
-------
Protectfon
77w', ^J)
//West Jackson Boulevard TO». r-,
Chicago, JL 60604-3590 ' Floor
------- |