United States
Environmental Protection
Agency
Environmental Monitoring
Systems Laboratory
Research Triangle Park NC 27711
Research and Development
EPA/600/S4-87/003 Apr. 1987
v>EPA Project Summary
Precision and Accuracy
Assessments for State and
Local Air Monitoring Networks
1985
Raymond C. Rhodes and E. Gardner Evans
Precision and accuracy data obtained
from State and local agencies during
1985 are summarized and evaluated.
Some comparisons are made with the
results reported for prior years to deter-
mine any trends. Some trends indi-
cated continued improvement in the
completeness of reporting of precision
and accuracy data. The national sum-
maries indicate a further improvement
in the precision and accuracy assess-
ments of the pollutant monitoring data
collected. The annual results from each
reporting organization are given so that
comparisons may be made from year to
year.
A comparison is made of the preci-
sion and accuracy data from the Preci-
sion and Accuracy Reporting System
with those from the independent per-
formance audit program conducted by
the Environmental Monitoring Systems
Laboratory.
This Project Summary was devel-
oped by EPA's Environmental Monitor-
ing Systems Laboratory, Research Tri-
angle Park, NC, to announce key
findings of the research project that is
fully documented in a separate report
of the same title (see Project Report
ordering information at back).
Introduction
The purpose of this document is to
report the fifth year of data from the Pre-
cision and Accuracy Reporting System
(PARS). Federal regulations promul-
gated on May 10, 1979, require quality
assurance precision and accuracy (P
and A)* data to be collected. Collection
started January 1,1981, according to re-
quirements set forth in 40 CFR Part 58
Appendix A. These requirements pro-
vide for more uniform Quality Assur-
ance programs and specific precision
and accuracy assessment and reporting
requirements across all State and local
air monitoring agencies.
The major portion of this report con-
sists of summarizations and evaluations
of the P&A data obtained by the efforts
of the States and local agencies. In addi-
tion, comparisons have been made of
the accuracy data collected for PARS
with the results of the National Perform-
ance Audit Program (NPAP) which has
been an ongoing program conducted
by the Environmental Monitoring Sys-
tems Laboratory (EMSL) since the early
1970's.
These summarizations and evalua-
tions of precision and accuracy data
serve the following purposes:
1. Quantitative estimates of the preci-
sion and accuracy of their monitor-
ing data are available to State and
local agencies.
2. A comparison of the data from all
of the above agencies can indicate
the need to improve quality assur-
ance systems in specific reporting
organizations.
*When one speaks of precision and accuracy of
measurement data, one really means the preci-
sion and accuracy of the measurement process
from which the measurement data are obtained
Precision is a measure of the "repeatability of the
measurement process under specified condi-
tions " Accuracy is a measure of "closeness to the
truth."
-------
3. An evaluation of the results may
indicate a need for improvement in
monitoring methodology.
4. The assessments provide users of
data from the State and Local Air
Monitoring Stations (SLAMS) net-
work a quantitative estimate of the
precision and accuracy of the am-
bient air quality data.
Ambient air quality data, collected by
States and local agencies since 1957,
have been stored in the National Aero-
metric Data Bank (NADB). These data
are used in (1) planning the nation's air
pollution control strategy, (2) determin-
ing if the National Air Quality Standards
are being achieved, and (3) determining
long-term trends of air quality. Prior to
the EPA air monitoring regulations of
May 10, 1979, the procedures used in
selecting monitoring sites, operating
and controlling the equipment, and cal-
culatmg, validating and reportingvthe
data varied considerably among agen-
cies. Frequently, the procedures being
used were not well-documented. These
conditions made it difficult to intercom-
pare data from different sites and agen-
cies. Furthermore, little information was
available on the reliability of the moni-
toring data.
To help alleviate these problems,
EPA's air monitoring regulations im-
posed uniform criteria on network de-
sign, siting, quality assurance, monitor-
ing methods, and data reporting after
December 30, 1980. For example, only
EPA reference, equivalent, or other
EPA-approved air monitoring methods
were to be used. Also, calibration stand-
ards were to be traceable to the Na-
tional Bureau of Standards (NBS) or
other authoritative standards. Further,
the quality assurance systems of the
states were required to be documented
and approved by the EPA Regional Of-
fices. Finally, the reporting organiza-
tions must also follow specific proce-
dures when assessing the P and A of
their measurement systems and must
report quarterly the P&A data to EPA.
Starting January 1, 1981, these regula-
tions became effective for National Air
Monitoring Sites (NAMS), and begin-
ning January 1, 1983, for all State and
Local Air Monitoring Stations.
The precision assessments were de-
termined by performing repeated meas-
urements of ambient-level "calibration"
gases at two-week intervals for continu-
ous methods, or by obtaining duplicate
results from collocated samplers for
manual methods. The accuracy assess-
ments were generally determined by
analyzing blind audit materials trace-
able to NBS. During each calendar year,
each site or instrument must be audited
at least once. Details concerning the
specific procedures and computations
used to assess P and A are contained in
the regulations.
National Results
National Data Reporting
The fifth year of data collected by
State and local agencies for P&A has
been compiled and summarized. Con-
tinuing improvements in the network
operation have been made. Table 1
shows the percentage of data reporting
for the nation.
Improvement in reporting continues
for the continuous N02 method. How-
ever, the other continuous methods
have either remained the same as or
slightly decreased from 1984. Reporting
for TSP and the manual N02 method
remain at a high level, 99 and 100 per-
cent, respectively. Reporting for the Pb
method has improved to 96 percent. Re-
porting for the manual 862 method,
used in 1985 by only three reporting or-
ganizations, remains low at 75 percent.
7985 Results from the PARS
Program
The measures of precision and accu-
racy are required to be computed and
reported for each calendar quarter by
each Reporting Organization (a State or
local agency) as percentage deviation
values. For precision, the repeatability
for each check is measured as the devia-
tion from the expected value as a per-
centage of the expected value. For accu-
racy, the deviation of the audit value
from the true value is measured as a
percentage of the true value. For both
precision and accuracy, 95 percent
probability limits are computed from
the average and standard deviations of
the individual percentage values:
Upper Limit = D + 1.96 S
Lower Limit = D - 1.96 S
where D = the average of the individual
percent differences;
S = the standard deviation of the
individual percent differ-
ences;*
1.96 = the multiplication factor cor-
responding to 95% probabil-
ity.
*Note: For the precision of manual
methods obtained from paired
observations, the standard de-
viation, S, is divided by V2, to
obtain variability estimates that
apply to individual reported val-
ues.
It is these upper and lower 95% proba-
bility limits which are reported and dis-
cussed in this report.
Moreover, it should be noted that the
data and the evaluations presented in
this report include any outlier values
which may have been reported by the
States and local agencies. The presence
of outliers can influence such compari-
sons by having undue impact on aver-
age values for individual reporting or-
ganizations.
Table 2 exhibits the national probabil-
ity limits for each of the manual meth-
ods. The probability limits in Tables 2
and 3 reflect the total variability in the
nation in 1985. They are the limits which
would be obtained if the results of all
the individual precision (or accuracy)
checks in the nation were combined as
one sample. The national limits for this
report and the report for 1984 correctly
reflect the total variability in the data.
They are somewhat wider than the cor-
responding limits for reports for years
1983 and before due to a change in the
Table 1. Percent of Reporting Organizations Reporting Precision and Accuracy Data
Pollutant
measurement
CO
SO2
NO2
03
TSP
Pb
SO2 (manual)
NO2 (manual)
1981
77
82
56
83
94
—
—
—
1982
89
93
72
89
97
—
—
—
1983
99
96
88
99
99
93
75
86
1984
99
97
94
99
99
92
80
100
1985
96
97
96
95
99
96
75
100
-------
Tiethod of calculation beginning with
the 1984 data.
The precision limits reflect the re-
peatability of the methodology used in
the field to collect and analyze the sam-
ples at ambient levels. The spread of the
limits may be somewhat inflated due to
measurements at relatively low concen-
tration levels.
The accuracy of the manual methods
indicates the limits at predetermined
concentration levels for the chemical
analysis performed on the samples for
lead, sulfur dioxide, and nitrogen diox-
ide. For the TSP method, the accuracy
measurement is for the flow rate only.
The probability limits for manual accu-
racy are very good and reflect the qual-
ity of work done in the chemical labora-
tories for lead, sulfur dioxide, and
nitrogen dioxide analyses, and in the
field for flow rate measurement for the
TSP method. Because of the continual
replacement of the manual S02 and
N02 methods with continuous meth-
ods, further discussion of the manual
methods is limited. The detailed results
for each reporting organization are tab-
ulated in an appendix to the full report.
The number of reported precision
checks and accuracy audits have de-
creased from 1984, particularly for the
manual S02 and N02 method which are
being replaced by continuous instru-
ments.
The precision and accuracy limits for
automated methods are presented in
Table 3. The effort expended for the col-
lection of quality assurance precision
and accuracy data is appreciable, but it
is necessary to assess data quality. The
numbers of reported precision and ac-
curacy checks for the continuous meth-
ods have decreased from 1984, particu-
larly for the continuous S02 method.
The number of precision checks is 40
percent less, and the number of accu-
racy audits is 14 percent less in 1985
than for 1984. No explanation can be
given for these significant decreases,
since the number of sites have re-
mained essentially the same. Details of
the results are discussed in the analysis
section.
National Precision Results
Comparison
Figure 1 shows the national probabil-
ity limits for precision for the various
methods. With data from the four most
recent years, some minor trends are ev-
ident. Some slight improvement, as
measured by a reduction in the spread
of the limits, is noted for TSP and the
continuous methods. The slight but per-
sistent negative bias for the continuous
S02 method indicates that on the aver-
age there is some negative instrument
drift from the most recent calibration or
Table 2.
National Precision and Accuracy Probability Limit Values for Manual Methods for
1985
Precision
Pollutant
TSP
Lead
Sulfur
dioxide
Nitrogen
dioxide
Number of
valid
collocated
data pairs
16,462
3,308
185
469
Probability
limits (%)
Lower Upper
-15
-18
-42
-27
+ 17
+ 19
+ 35
+29
Accuracy
Probability limits <%)
No. of
audits
6,770
1,616
174
161
Level 1
Lower Upper
-12
-33
-7
+ 10
+ 17
+8
Level 2
Lower Upper
-8
-10
-18
_ 7
+ 8
+ 8
+ 9
+ 5
Level 3
Lower Upper
-19
-3
+ 9
+ 5
instrument adjustment to the time of
the biweekly precision check.
Although the manual methods for Pb,
SO2, and N02 were not required to be
reported until 1983, a number of agen-
cies began reporting in 1981. The re-
sults for Pb show a continuing improve-
ment. The manual S02 and N02
methods are much more variable than
the continuous methods. And, although
they have shown considerable improve-
ment over the five-year period, the re-
sults for 1985 are somewhat worse than
for 1984.
National Accuracy Results
Comparison
Figures 2a and 2b show the national
probability limits for accuracy audits for
the manual and continuous methods,
respectively, for the four most recent
years, 1982-1985. Improvement for the
manual methods is evident for Pb and
N02. The variability for the TSP method
remains the same and the S02 method
has shown a definite increase. The re-
sults for the manual methods for S02
and N02 vary considerably from year-
to-year because the methods are used
in only 2 or 3 regions and are being re-
placed by the continuous methods.
Slight improvement is evident for all the
continuous methods. The continuous
methods for S02 and N02 show more
inaccuracy than all other methods.
However, the accuracy audits for the
manual methods check only a portion of
the measurement method.
Although the continuous N02 method
is more variable than the other meth-
ods, it has shown the greatest improve-
ment, particularly for the level 1 concen-
tration.
The general, and expected, pattern of
variability across levels is very evident,
with the greatest percentage variability
at the lowest concentration levels. The
slight negative bias for the continuous
SO2 method is consistent across all
three levels. A possible cause is that, on
Table 3. National Precision and Accuracy Probability Limit Values for Automated Analyzers for 1985
Precision Accuracy
CO
S02
No. of
precision
checks
14,465
22,863
7,695
18,822
Probability
limits (%)
Lower Upper
-9
-9
-12
-10
+9
+ 7
+ 12
+ 9
No. of audits
Total
1,159
1,481
573
1,620
Level
4
16
84
23
121
Level 1
Lower Upper
-14
-16
-20
-14
+ 13
+ 14
+21
+ 12
Probability
Level 2
Lower Upper
-8
-12
-13
-11
+8
+ 12
+ 12
+ 9
Limits (%)
Level 3
Lower Upper
-8
-13
-12
-10
+ 7
+ 12
+ 10
+ 8
Level 4
Lower Upper
-19
-8
-14
+ 14
+ 10
+ 5
+ 5
-------
80
National Values for Precision
1982-1985
-80
Figure 1. National precision probability limits for 1982 through 1985
the average, a negative drift occurs with
these analyzers from the time of last cal
ibration or instrument adjustment unti
the time of the accuracy audit.
Comparison of Results from
the PARS and the Performance
Audit Program
A general comparison between the
accuracy data of the PARS program and
the Performance Audit (PA) data is in-
cluded in this report. The Performance
Audit data are the results of an indepen-
dent check conducted by the Quality As-
surance Division (QAD) of the EMSL un-
der the National Performance Audit
Program (NPAP).
In the NPAP, specially prepared audit
samples or devices are sent from QAD
to the participating ambient air monitor-
ing agencies. The samples or devices
are carefully and accurately assessed by
EMSL utilizing NBS Standard Reference
Materials (SRM's) or standards. The
monitoring agencies analyze or meas-
ure the samples or devices as un-
knowns or blinds and report their re-
sults to QAD for evaluation. Audit
programs are conducted for the follow-
ing pollutant measurements using the
materials indicated:
40
30
a 20
-10 -I
-20
-30
-40
National Values for Accuracy
1982-1985
Manual Methods
•V
4?
$
s -V
•£•
National values for Accuracy
1982-1985
Figure 2. National accuracy probability limits for 1982 through 1985.
4
-------
Measurement
Audit materials
Portion of measurement
system audited
SO2 (manual)
N02 (manual)
Pb
TSP
CO
S02
Freeze-dried sodium-sulfite
Aqueous sodium nitrite
Filter strip with lead nitrate
Reference flow device
Cylinders containing CO gas
Cylinder containing SO2 gas
Chemical analytical
Chemical analytical
Chemical analytical
Flow
Continuous instrument
Continuous instrument
The audit materials or devices are
prepared at three to six different con-
centrations or flow levels. Separate re-
ports on the evaluation of the PA data
are published by EMSL.
As indicated above, the NPAP does
not yet include an audit for the ozone or
continuous N02 methods. Therefore, no
comparisons of the NPAP or PA data
with the PARS data are possible for
these pollutants.
Since precision assessments are not
made in the PA program, only accuracy
can be compared across the PARS and
the PA programs. For the purpose of
this report, the results from PARS and
the PA system are compared at approx-
imately the same levels by matching
laboratories and reporting organiza-
tions. Since the PARS data are pre-
sented with outliers, the same approach
was taken with the audit data. Knowl-
edge of the historical audit data reports,
however, indicates that the presence of
outliers may make a significant differ-
ence in the audit results for some agen-
cies.
Comparisons of the national values of
the probability limits (Table 4) exhibit
fairly good agreement between the re-
sults of the two programs. However,
there is considerable variation between
the results of the two programs when
comparisons are made on Regional and
reporting organization bases. Lack of
better agreement results from several
factors. First, the inclusion of outlier val-
ues in both data sets appears to have
introduced some excessive distortion of
general trends. Second, the concentra-
tion levels for the two systems do not
coincide exactly at each of the audit lev-
els. Third, the PA data are the results of
independent external audits, while the
PARS accuracy data are based on the
results of independent internal audits.
The expected effects of the last-
mentioned factor would cause the
spread of the limits for the PA to be
wider than that for the PARS. Examina-
tion of the results (see Table 4) confirm
these expectations for all pollutants ex-
cept the manual NO2. The manual N02
limits are wider for PARS than for PA,
perhaps because of small sample vari-
ability or that more care was taken with
the external audits.
Conclusions and
Recommendations
The results of PARS data for 1985 in-
dicate some further improvement over
the data for previous years. However,
considerable differences exist among
Regions and individual reporting orga-
nizations for most measurement meth-
ods. Investigations should be made by
the Regions and the states to determine
the causes of these significant differ-
ences. Elimination of the causes of the
excessive deviations will further im-
prove the quality assessments of the
measurement systems used for routine
monitoring in the nation.
Comparisons of PARS and PA data
show more variability of the PA data
than for PARS except for CO, S02 and
manual N02 methods. These differ-
ences are presumably due to the fact
that the external PA accuracy audits are
more completely independent than the
internal PARS accuracy audits. These
differences have been consistent for
past years.
Further improvement in the data
quality assessments, which are meas-
ures of the monitoring data quality, can
be achieved only through continuing ef-
forts of State and local agency person-
nel involved (first-hand) with the opera-
tion and quality control of their
measurement systems. Regional QA
Coordinators can also assist through
their review of the operations and qual-
ity control practices across the States in
their Regions.
Each Regional QA Coordinator should
evaluate the PARS data from all the re-
porting organizations within his Region
to identify those organizations having
excessively large variations of probabil-
ity limits. Investigation should be made
to determine the causes and correct
them to preclude future excessive devi-
ations. Similarly, Regional QA Coordi-
nators should review the operations of
the reporting organizations having sig-
nificantly better precision and accuracy
results in order to identify specific pro-
cedures which should be uniformly
used throughout the Region and the Na-
tion to further improve the reliability of
the monitoring data in the National
Aerometric Data Base.
Table 4. Summary Comparison of EMSL Performance Audits (PA) vs. PARS Accuracy Audit
Data for Year 1985
National values
95% probability limits (%)
Pollutant
CO
PA
PARS
SO2
PA
PARS
TSP
PA
PARS
Pb
PA
PARS
SO2 (manual)
PA
PARS
NO2 (manual)
PA
PARS
Audits
388
667
756
1326
3772
5928
432
777
15
55
15
40
Level 1
Lower Upper
-14
-13
-20
-17
-16
-12
-24
-21
0
-20
10
13
26
14
16
9
25
20
3
20
Level 2
Lower Upper
-10
-8
-18
-13
-10
-8
-15
-9
-9
-10
0
-10
9
8
22
12
12
8
10
7
22
12
7
15
Level 3
Lower Upper
-8
-8
-17
-13
*
-10
-2
-14
9
7
20
12
9
11
4
19
-------
The EPA authors. Raymond C. Rhodes (also the EPA Project Officer, see below)
and £. Gardner Evans, are with Environmental Monitoring Systems
Laboratory, Research Triangle Park, NC 27711.
The complete report, entitled "Precision and Accuracy Assessments for State
and Local Air Monitoring Networks 1985," (Order No. PB 87-145 447/AS;
Cost: $24.95, subject to change) will be available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at:
Environmental Monitoring Systems Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
Official Business
Penalty for Private Use S300
EPA/600/S4-87/003
0000329 PS
U S ENVIR PROTECTION AGENCY
REGION 5 LIBRARY
230 S DEARBORN STREET
CHICAGO IL 60604
------- |