United States
 Environmental Protection
 Agency
 Environmental Monitoring
 Systems Laboratory
 Research Triangle Park NC 27711
 Research and Development
EPA/600/S4-86/031 Jan. 1987
 Project  Summary
Precision  and  Accuracy
Assessments  for  State  and
Local   Air  Monitoring
Networks  1984

Raymond C. Rhodes and E. Gardner Evans
  Precision and accuracy data obtained
from state and local agencies during 1984
are summarized and evaluated. Some
comparisons are made with the results
previously reported for 1981, 1982, and
1983 to determine any trends. Some
trends indicated continued improvement
in the completeness of reporting of preci-
sion and accuracy data. The national sum-
maries indicate a further improvement in
the precision and accuracy assessments
of the pollutant monitoring data collected.
The annual results from each reporting
organization are given so that comparisons
may be made from 1981 to 1984 and also
with other reporting organizations.
  A comparison of the precision and accu-
racy data from the Precision and Accuracy
Reporting System with those from the in-
dependent  performance audit  program
conducted by the Environmental  Monitor-
ing Systems Laboratory is given.
  This Project Summary was developed
by  EPA's  Environmental  Monitoring
Systems Laboratory,  Research  Triangle
Park, NC, to announce key findings of the
research project that is fully documented
in a separate report of the same title (see
Project Report ordering information  at
back).

Introduction
 The purpose of the full document is to
report the third year of data from the Preci-
sion and  Accuracy Reporting  System
(PARS). Federal regulations promulgated
on May 10, 1979, require quality assur-
ance precision and accuracy (P and A)*
data to be collected. Collection started
January 1, 1981, according to require-
ments set forth in 40 CFR Part 58 Appen-
dix  A.  These requirements  provide for
more uniform Quality Assurance programs
and specific precision and accuracy as-
sessment and  reporting requirements
across all State and local air monitoring
agencies.
  The major portion of the report consists
of summaries and evaluations of the P and
A data obtained by  the efforts of the
states and local agencies. In addition,
comparisons have been  made of the ac-
curacy data collected for PARS with the
results of the National Performance Audit
Program (NPAP), which has been an ongo-
ing program conducted  by the  Environ-
mental  Monitoring Systems Laboratory
(EMSL) since the early 1970's.
  These summaries and evaluations serve
the following purposes:
  1. Quantitative estimates of the preci-
    sion and accuracy of their monitor-
    ing data are available to state and
    local agencies.
  2. A comparison of the data from all the
    agencies can indicate the need to im-
"When one speaks of precision and accuracy of
 measurement data, one really means the precision
 and accuracy of the measurement process from
 which the data are obtained. Precision is a measure
 of the "repeatability of the measurement process
 under specified conditions." Accuracy is a measure
 of "closeness to the truth."

-------
     prove quality assurance systems in
     specific reporting organizations.
  3.  An evaluation of the results may indi-
     cate a need for improvement in mon-
     itoring methodology.
  4.  The assessments provide users of
     data from the State and Local Air
     Monitoring Stations (SLAMS) net-
     work a quantitative estimate  of the
     precision and accuracy of  the am-
     bient air quality data.
  Ambient air quality data, collected by
states and local agencies since 1957, have
been stored in the National Aerometric
Data Bank (NADB). These data are used
in (1) planning the nation's air pollution
control  strategy, (2) determining  if the
National Air Quality Standards are being
achieved, and (3) determining long-term
trends of air quality. Prior to the EPA air
monitoring regulations of May 10, 1979,
the procedures used in selecting monitor-
ing sites,  operating and  controlling the
equipment, and calculating, validating and
reporting  the  data varied  considerably
among  agencies.  Frequently the  proce-
dures being used  were not  well docu-
mented. These conditions made it difficult
to intercompare data from different sites
and agencies. Furthermore, little informa-
tion was available on the reliability of the
monitoring data.
  To help alleviate these problems, EPA's
air  monitoring regulations imposed
uniform criteria on network design, siting,
quality  assurance, monitoring methods,
and  data  reporting after December 30,
1980. For example, only EPA reference,
equivalent, or other  EPA-approved air
monitoring methods were  to be used.
Also, calibration standards  were to be
traceable to the National Bureau of Stand-
ards (NBS) or other authoritative stand-
ards. Further, the  quality  assurance
systems of the states were required to be
documented  and approved  by  the EPA
Regional  Offices.  Finally, the reporting
organizations must also follow specific
procedures when assessing the P and A
of their measurement systems and must
report the P and A data to EPA quarterly.
Starting January 1,1981, these regulations
became effective for National Air Monitor-
ing Sites (NAMS), and beginning January
1,  1983, for  all  State and Local  Air
Monitoring Stations.
   The precision assessments were deter-
mined by performing  repeated measure-
ments of ambient-level "calibration" gases
at  two-week  intervals for  continuous
methods, or by obtaining duplicate results
from collocated samplers  for  manual
methods. The accuracy assessments were
generally determined by analyzing blind
audit materials traceable to NBS. During
each calendar year, each site or instrument
must be audited at least once. Details con-
cerning the specific procedures and com-
putations used to assess P and A are con-
tained  in the regulations.

National Results

National Data Reporting
  The  fourth year of data collected by
state and local agencies for P and A has
been compiled and summarized. The net-
work operation has  been continually im-
proved. Table 1 shows the improvement in
data reporting for the nation.
  Improvement continues for the contin-
uous NO2 method; however, the percent-
age still lags behind that for continuous
CO, S02 and 03 methods. Reporting for
the manual methods for Pb, SO2 and NO2
was required by the regulations beginning
January 1,  1983.  Reporting  for  Pb  is
negligibly different  from  1983 to  1984.
Reportings for the  manual methods for
S02 and N02 have significantly improved
from 1983 to 1984.

 1984 Results From The Pars
Program
  The  measures of precision and accuracy
are required to be computed and reported
for each calendar quarter by each report-
ing organization (a state or local agency)
as percentage deviation values. For preci-
sion, the repeatability for each check is
measured as the deviation from the ex-
pected value as a percentage of the ex-
pected valua For accuracy, the deviation
of the audit value from the true value is
measured  as a  percentage of the true
value.  For both precision and accuracy, 95
percent probability limits are computed for
the percentage  values from the average
 and standard deviations of the individual
 percentage values:

              D   1.96  S

   where D  = the average of the individual
              percent differences;
         S  = the standard  deviation of
              the  individual percent
              differences;*
       1.96  = the  multiplication  factor
              corresponding  to 95%
              probability.
Table 1.    Percent of Reporting
          Organizations Reporting
          Precision and Accuracy Data
 Pollutant
measurement   1981   1982   1983   198'
CO
SO2
NO2
03
TSP
Pb
SO2 (manual)
N02 (manual)
77
82
56
83
94



89
93
72
89
97



99
96
88
99
99
93
75
86
95
91
94
9S
9&
92
8C
IOC
 "Note: For the precision of manual methods obtain-
       ed from paired observations, the standard
       deviation,  S, is divided by  \f2, to obtain
       variability estimates that apply to individual
       reported values.
It is these upper and lower 95% probabilit
limits which are reported and discussed i
the full report.
  Moreover, it should be noted that th
data and the evaluations presented in th
report include any outlier values whic
may have been reported by the states an
local  agencies.  The presence of outlier
can influence such comparisons by havin
undue impact on average values for ind
vidual reporting organizations.
  Table 2 exhibits the national probabilit
limits for each of the manual methods. Th
probability limits in Tables 2 and 3 are cor
solidated and weighted limits of all th
reported limits  for 1984. They are th
limits that would be obtained if the result
of all the individual precision (or accurac
checks in the nation were combined i
one sample. The national limits for th
report  more correctly reflect  the  tot
variability in the data and are somewhi
wider than  the corresponding  limits f(
previous reports due to a change in th
computation of these  limits.
   The precision limits  reflect the repea
ability of the methodology used in the fie
to collect and analyze the samples at an
bient levels. The spread of the limits me
be somewhat  inflated due to  measur
ments at relatively  low concentratic
levels.
   The accuracy of the manual methods i
dicates the limits at predetermined co
centration levels for the chemical analys
 performed in the samples for lead, sulf
dioxide, and nitrogen dioxide. For the TJ
 method, the accuracy measurement is f
the flow rate only. The probability limits f
 manual accuracy are very good and refle
 the quality  of work done in the chemic
 laboratories for lead, sulfur dioxide, ai
 nitrogen dioxide analyses, and in the fie
 for  flow  rate measurement for the T!
 method. Because of the continual replac
 ment  of  the  manual  SO2  and  Nl
 methods with continuous methods, fi
 ther discussion of the  manual methods

-------
Table 2.    National Precision and Accuracy Probability Limit Values for
          Manual Methods for 1984
                    Precision                             Accuracy

         Number of                                 Probability limits (%)
         valid col-      Probability            ~~~  "          
         located       limits (%)    No. of    Level 1       Level 2       Level 3
Pollutant  data pairs   Lower    Upper audits  Lower  Upper Lower  Upper Lower  Upper
TSP
Lead
Sulfur
dioxide
Nitrogen
dioxide
17,152
3,937

297

691
-16
-18

-33

-27
+ 17
+ 20

+ 31

+ 27
7,436
1,657

203

175

- 17

-20

-8

+ 15

+ 9

+ 10
-8
- 11

-14

-7
+ 8
+ 10

+ 7

+ 8



-12

-6



+ 7

+ 7
 5?
 Q
  CO
 8
                             National Values for Precision
                                    1981-1984
Figure 1.    National precision probability limits for 1981 through 1984.
limited. The detailed  results  for  each
reporting organization are tabulated in an
appendix to the full report.
  The precision and accuracy limits for
automated methods are presented in Table
3. The effort expended for the collection
of quality assurance precision and accur-
acy data is appreciable, but it is necessary
to assess data  quality.

National Precision Results
Comparison
  Figure 1 shows the national probability
limits for precision for the various meth-
ods. With data from four years, some
minor  trends are  evident.  Some slight
improvement, as measured by a reduction
in the spread of the limits, is noted for TSP
and the continuous methods, except for
NO2.  The slight but  persistent negative
bias for the continuous SO2 method indi-
cates that on the average there is some
negative instrument drift from the  most
recent calibration  or instrument adjust-
ment to the time of the biweekly precision
check.
  Although the manual methods for Pb,
S02, and N02 were not required to be re-
ported until  1983,  a number of agencies
began reporting  in 1981. The results for Pb
show a decided  improvement. The manual
S02 and N02 methods are  much  more
variable than the  continuous  methods.
However,  they  do show  considerable
improvement over  the four-year period.


National Accuracy Results
Comparison
  Figures 2a and  2b show the national
probability limits for accuracy audits for
the continuous and manual methods,
respectively. Improvement for the manual
methods is not evident except perhaps for
TSP and SO2. The variability for the Pb
method is  increased and for the NO2
method has  shown  no definite trend.
Slight improvement is evident for all the
continuous  methods.  The  continuous
methods for S02 and N02 show more in-
accuracy than all other methods. However,
 Table 3.    National Precision and Accuracy Probability Limit Values for Automated Analyzers for 1984
                   Precision                                                          Accuracy

CO
SO2
NO2
03
No. of
precision
checks
14,692
38,312
8.653
20,031
Probability
limits 1%)
Lower Upper
-9
-12
-14
-12
+ 8
+ 11
+ 13
+ 10
No. of audits
Total
1,288
1,666
613
1,773
Level
4
23
166
24
144
Probability
Level 1
Lower Upper
- 14
-16
-21
-16
+ 13
+ 14
+ 20
+ 14
Level 2
Lower Upper
-9
-12
-13
-12
+ 8
+ 11
+ 12
+ 10
limits (%)
Level 3
Lower Upper
_ g
-12
-13
- 11
+ 8
+ 11
+ 10
+ 10
Level 4
Lower Upper
-10
-13
-18
-6
+ 9
+ 12
+ 14
+ 5

-------
                                  (a)
National Values for Accuracy
1981-1984
Continuous Methods
30 -
^ 20 -
to
1 1-
1
- -10 -

-20-
-30 -





Mill
:  j ^*



'It


^


*u -* ( \ \ \ \




IE




\







_
' ' Bl
I  j-

"

81 82 8384
II BD
, .

f


IP
1





!

3


!,
cP c& &   c sa sa s t* \*a v*
                                                                                    National Values for Accuracy
                                                                                           1981-1984
                                                                                        Manual Methods
Figure 2.    National accuracy probability limits for 1981 through 1984.
in the accuracy audits for the manual
methods, only a portion of the measure-
ment method is checked.
  Although the continuous N02 method
is more variable than the other methods, it
has shown the greatest improvement, par-
ticularly for the level  1 contentration.
  The general and expected pattern  of
variability  across levels is very  evident,
with the greatest percentage variability at
the lowest concentration levels. The slight
negative  bias for  the continuous SO2
method is consistent across  all  three
levels. A possible cause is that, on the
average, a negative drift occurs with these
analyzers from the time of last calibration
or instrument adjustment until the time of
the accuracy audit.

Comparison  of  Results  from the
PARS and the Performance Audit
Program
  A general comparison between the ac-
curacy data of the PARS program and the
Performance Audit (PA) data is included in
the full report. The Performance Audit data
are the  results of an indpendent check
conducted by the Quality Asssurance Divi-
sion (QAD) of the EMSL under the Na-
tional Performance Audit Program (NPAP).
  In the NPAP, specially  prepared audit
samples or devices are sent from QAD to
the participating ambient air monitoring
agencies. The samples or devices are care-
fully and accurately assessed by  EMSL
utilizing NBS Standard Reference Materi-
als (SRM's) or standards. The monitoring
agencies analyze or measure the samples
or devices  as  unknowns or blinds and
report their results to QAD for evaluation.
Audit programs  are conducted  for the
following pollutant measurements using
the materials indicated:
  Since precision assessments are nc
made in the PA program, only accurac
can be compared across the PARS and th
PA programs. For the purpose of the fu
report, the results from PARS and the P/
system are compared at approximately th
same levels by matching laboratories an
reporting organizations. Since the PAR!
data are presented with outliers, the sam
approach was taken with the audit date
Knowledge of the historical audit dat

                   Portion of measure
Measurement
S02 (manual)
N02 (manual)
Pb
TSP
CO
S02
Audit materials
Freeze-dried sodium sulfite
Aqueous sodium nitrite
Filter strip with lead nitrate
Reference flow device
Cylinders containing CO gas
Cylinder containing S02 gas
ment system audite
Chemical analytical
Chemical analytical
Chemical analytical
Flow
Continuous instrument
Continuous instrument
  The audit materials or devices are pre-
pared at three to six different concentra-
tions or flow  levels. Separate reports on
the evaluation of the PA data are published
by EMSL.
  As indicated above, the NPAP does not
yet include an audit for the ozone or con-
tinuous NO2 methods. Therefore, no com-
parisons of the NPAP or PA data with the
PARS  data  are  possible  for  these
pollutants.
reports, however, indicates that the pres
ence of outliers may make a significant di
ference in the audit  results for som
agencies.
  Comparisons of the national values c
the probability limits (Table 4) exhibit fair!
good agreement between  the results c
the two programs. However, there is cor
siderable variation between the results c
the two programs when comparisons ar
made on Regional and reporting organize
                                    4

-------
Table 4.    Summary Comparison of EMSL Performance Audits (PA) vs.
           PARS Accuracy Audit Data for Year  1984
                                          National values
                                       probability limits (%)
                     Level 1          Level 2          Level 3          Level 4
Pollutant    Audits Lower   Upper  Lower    Upper  Lower    Upper  Lower    Upper
CO
PA
PARS
SO2
PA
PARS
TSP
PA
PARS
Pb
PA
PARS
SO2 (manual)
PA
PARS
NO2 (manual)
PA
PARS

771
974

357
819

2447
6559

723
1259

30
190

30
139

- 9
-14

-23
- 13




-35
-17


- 18

- 5
- 6

12
13

19
11




30
15


8

	 *
8

-20
- 8

-16
- 12

- 15
- 6

- 17
-11

-15
- 12

_ j
- 6

21
8

14
11

18
7

11
10

6
6

- 2
7

- 7
- 8

-17
-12




-22


-18
-12

- 3
- 4

8
7 -10

14 -22
10 -11




14


15 -14
6

4-7
5


8

20
9







16


- 3

tion bases. Lack of better agreement re-
sults from several factors. First, the inclu-
sion of outlier values in the PA data ap-
pears to have introduced some excessive
distortion of general trends. Second, even
though the PARS averages in Table 4 are
weighted by the number of audits, varia-
tions due to many sources of error for  both
data sets are averaged together to obtain
the national values, thereby masking any
correlations which may have existed for
the results of individual agencies. Third,
the concentration levels for the two  sys-
tems  do  not coincide exactly at each of
the audit  levels. Fourth, the PA data are the
results of independent external audits,
while the PARS accuracy data  are based
on the results of independent internal
audits. The expected effects of the  last-
mentioned factor would cause the spread
of the limits for the PA to be wider than
that for  the PARS. Examination  of the
results  (see Table  4)  confirm  these
expectations.

Conclusions and
Recommendations
  The results of PARS  data for 1984 in-
dicate some general improvement over the
data for previous years. However, consid-
erable differences  exist among Regions
and individual reporting organizations for
most  measurement methods.  Investiga-
tions should be made by the Regions and
the states  to  determine the  causes of
these significant differences.
  Comparison of PARS and PA data show
more  variability of the  PA data than for
PARS except for CO. These differences are
presumably due to the fact that the exter-
nal'PA accuracy audits are more complete-
ly independent than  the internal PARS
accuracy audits. These differences have
been consistent for past years.
  Further improvement in the data quality
assessments, which are measures of the
monitoring data quality,  can be achieved
only through continuing efforts of state
and local agency personnel involved first-
hand with the operation and quality con-
trol of their measurement systems. Re-
gional  QA Coordinators can  also assist
through their review of the operations and
quality control practices across the states
in their Regions.
  Each Regional QA Coordinator should
evaluate the PARS data from all the report-
ing organizations within his Region to
identify those organizations having exces-
sively large variations of probability limits.
Investigation should be made to determine
the causes and correct them to preclude
future   excessive deviations.  Similarly,
Regional QA Coordinators should review
the operations of the  reporting organiza-
tions having significantly better precision
and accuracy results in order to  identify
specific procedures that should  be uni-
formly used throughout the Region and
the nation to further improve the reliability
of the monitoring data in the National
Aerometric Data Base.

-------
    The EPA authors Raymond C. Rhodes (also the EPA Project Officer, see below)
     and E.  Gardner  Evans are with  the Environmental Mentoring Systems
     Laboratory, Research Triangle Park, NC 27711.
    The complete report, entitled "Precision and Accuracy Assessments for State
     and Local Air Monitoring Networks 1984," (Order No. PB 87-111 720/AS;
     Cost: $18.95. subject to change) will be available only from:
           National Technical Information Service
           5285 Port Royal Road
           Springfield, VA 221611
           Telephone: 703-487-4650
    The EPA Project Officer can be contacted at:
           Environmental Monitoring Systems Laboratory
           U.S. Environmental Protection Agency
           Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
Official Business
Penalty for Private Use S300

EPA/600/S4-86/031
                                               1   VhB "-'8-'

                                                N   ,   .-.
U.S.POSUGt

I  0 .3 ?. r:
        0000329    PS

-------