United States
Environmental Protection
Agency
Environmental Monitoring
Systems Laboratory
Research Triangle Park NC 27711
Research and Development
EPA/600/S4-85/031 Aug. 1985
ŁEPA Project Summary
Summary of Precision and
Accuracy Assessments for the
State and Local Air Monitoring
Networks 1982
E. Gardner Evans, Raymond C. Rhodes,
William J. Mitchell and John C. Puzak
Precision and accuracy data obtained
from State and local agencies during
1982 are summarized and evaluated.
Some comparisons are made with the
results previously reported for 1981 to
determine the indication of any trends.
Some trends indicating improvement in
the precision and accuracy of monitor-
ing data are given on a national and
regional basis- The annual average
results from each reporting organiza-
tion are given so that comparisons may
be made from 1981 to 1982 and also
with other reporting organizations.
A comparison of the precision and
accuracy from the Precision and Accu-
racy Reporting System and that from
the independent performance audit
program conducted by the Environ-
mental Monitoring Systems Laboratory
is given.
This Project Summary was devel-
oped by EPA's Environmental Monitor-
ing Systems Laboratory, Research
Triangle Park, NC, to announce key
findings of the research project that is
fully documented in a separate report of
the same title (see Project Report order-
ing information at back).
Introduction
The purpose of this document is to
report the second year of data from the
Precision and Accuracy Reporting Sys-
tem (PARS). Federal regulations promul-
gated on May 10, 1979, require quality
assurance precision and accuracy (P&A)*
data to be collected. Collection started
January 1, 1981, according to require-
ments set forth in 40 CFR Part 58 Appen-
dix A. These requirements provide for
more uniform Quality Assurance pro-
grams and specific precision and accu-
racy reporting requirements across all
State and local air monitoring agencies.
The major portion of this report con-
sists of summarizations and evaluations
of the P & Adata obtained by the efforts of
the States and local agencies. In addition,
comparisons have been made of the
accuracy data collected for PARS with
the results of the National Performance
Audit Program (NPAP) which has been an
ongoing program conducted by the Envi-
ronmental Monitoring Systems Labora-
tory (EMSL) since the early 1970's.
These summarizations and evalua-
tions of precision and accuracy data
serve the following purposes:
1. Quantitative evaluations of the
quality of their monitoring data are
available to State and local
agencies.
2. A comparison of the data from all
the agencies can indicate the need
to improve quality assurance sys-
tems in specific reporting
organizations.
3. An evaluation of the results may
indicate a need for improvement in
monitoring methodology.
*When one speaks o1 precision and accuracy of mea -
surement data', one really means the precision and
accuracy of the measurement process from which
the measurement data are obtained. Precision is a
measure of the "repeatability of the measurement
process under specified conditions." Accuracy is a
measure of "closeness to the truth."
-------
4. The assessments provide users of
data from the State and Local Air
Monitoring Stations (SLAMS) net-
work a quantitative estimate of the
precision and accuracy of the
ambient air quality data.
Ambient air quality data, collected by
States and local agencies since 1957,
have been stored in the National Aero-
metric Data Bank (NADB). These data are
used in (1) planning the nation's air pollu-
tion control strategy, (2) determining if
the National Air Quality Standards are
being achieved, and (3)determining long-
term trends of air quality. Prior to the EPA
air monitoring regulations of May 10,
1979, the procedures used in selecting
monitoring sites, operating and control-
ling the equipment, and calculating, vali-
dating and reporting the data varied
considerably among agencies. Fre-
quently the procedures being used were
not well-documented. These conditions
made it difficult to intercompare data
from different sites and agencies. Furth-
ermore, little information was available
on the reliability of the monitoring data.
To help alleviate these problems, EPA's
air monitoring regulations imposed uni-
form criteria on network design, siting,
quality assurance, monitoring methods,
and data reporting after December 30,
1980. For example, only EPA reference,
equivalent, or other EPA-approved air
monitoring methods were to be used.
Also, calibration standards were to be
traceable to the National Bureau of
Standards (NBS) or other authoritative
standards. Further, the quality assurance
systems of the states were required to be
documented and approved by the EPA
Regional Offices. Finally, the reporting
organizations must also follow specific
procedures when assessing the P&A of
their measurement systems and must
report the P&A data to EPA quarterly.
Starting January 1, 1981, these regula-
tions became effective for National Air
Monitoring Sites (NAMS), and beginning
January 1, 1983, for all State and Local
Air Monitoring Stations.
The precision assessments were
determined by performing repeated mea-
surements on ambient-level "calibra-
tion" gases at two-week intervals for
continuous methods, or by obtaining
duplicate results from collocated
samplers for manual methods. The accu-
racy assessments were generally deter-
mined by analyzing blind audit materials
traceable to NBS. During each calendar
year, each site or instrument must be
audited at least once. Details concerning
the specific procedures and computa-
tions used to assess P&A are contained
in the regulations.
National Results
National Data Capture
The second year of data collected by
State and local agencies for P&A has
been compiled and summarized. Obvious
improvements in the network operation
have been made. Table 1 shows the
improvement in data capture for the
nation.
Table 1. National Percent Data Capture
for Required Precision and
Accuracy
Pollutant 1981 1982 Relative
< Change
CO
SO2
NO2
03
rsp
77
82
56
83
94
89
93
72
89
97
16
13
29
7
3
The automated NO2 analyzers which
tend to break down quite often had the
worst percent data capture for the first
two years. However, these analyzers had
the largest increase in data capture from
1981 to 1982 of 29%, which indicates a
substantial improvement. CO had the
next largest percent change, an increase
from 77% to 89%. SO increased from
82% to 93%. TSP had the lowest %
change between years, but it had very
little room for improvement in collection
of the precision and accuracy flow data.
1982 Results from the PARS
Program
The measures of precision and accu-
racy are required to be computed and
reported by the States and local agencies
as percentage values. For precision, the
repeatability for each check is measured
as the deviation from expected values as
a percentage of the expected value. For
accuracy, the deviation of the audit value
from the true value is measured as a per-
centage of the true value. For both preci-
sion and accuracy, 95 percent probability
limits are computed for the percentage
values from the average and standard
deviations of the individual percentage
values:
D+ 1.96 S
where D = the average of the individ-
ual percent differences;
S = the standard deviation of t
the individual percent {
differences;*
1.96= the multiplication factor
corresponding to 95%
probability.
It is these upper and lower 95% probabil-
ity limits which are reported and dis-
cussed in this report.
Moreover, it should be noted that the
data and the evaluations presented in
this report include any outlier values
which may have been reported by the
States and local agencies. It is possible
that the presence of outliers might influ-
ence such comparisons by having undue
impact on average values.
Table 2 exhibits the national averages
for each of the manual pollutants. By exa-
mining the numbers of valid collocated
data pairs (16,233) and the number of
audits (6461) performed for TSP, one can
estimate the amount of effort being spent
in this country to obtain these data qual-
ity assessments.
The precision limits reflect the repeata-
bility of the methodology used in the field
to collect and analyze the samples at
ambient levels. The spread of the limits
may be somewhat inflated due to mea-
surements at relatively low concentra-
tion levels. |
The accuracy of the manual methods
indicates the limits at predetermined
concentration levels for the chemical
analysis performed in the samples for
lead, sulfur dioxide, and nitrogen dioxide.
For the paniculate matter, the accuracy
measurement is for the flow rate only.
The probability limits for manual accu-
racy are very good and reflect the quality
of work done in the chemical laboratories
for lead, sulfur dioxide, and nitrogen diox-
ide analyses, and in the field for flow rate
measurement for particulate matter.
Because of the continual replacement of
the manual SO2 and NO, methods with
continuous methods, further discussion
of the manual methods is limited. The
detailed results, however, are tabulated
in Appendix B for each reporting
organization.
The precision and accuracy limits for
automated methods are presented in
Table 3. Apparent from the number of
precision checks, for example 23,144 for
S02, the effort expended for the collec-
tion of quality assurance precision and
accuracy data is appreciable, but neces-
*For the precision of manual methods obtained from
paired observations, the standard deviation, S, is
divided by V 2, to obtain variability estimates that .
apply to individual reported values t
-------
Table 2. National Precision and Accuracy Probability Limit Averages lor Manual Methods
Precision
Accuracy
Number of
Probability Limits C/o)
Pollutant
TSP
Lead
Sulfur
Dioxide
Nitrogen
Dioxide
Table
Valid Col- Probability
located Limits C/o)
Data Pairs Lower Upper
16.233
1.669
706
1.168
3. National
Analyzers
Precision
-12
-15
-38
-29
Precision and
+13
+16
+42
+34
Accuracy
No. of Level 1
Audits Lower Upper
6.461 — —
692 -11 +08
551 -13 +08
583 -07 +08
Probability Limit Averages for Automated
Accuracy
Level 2
Lower Upper
-07 +07
-07 +04
-O9 +07
-07 +06
Level 3
Lower Upper
-08 +05
-O5 +06
both S02 and CO. NO 2 improved
atically for level 1 measurements,
the range of the limits for Levels 2
dram-
while
and 3
sq
°3
CO
NO2
No. of
Precision
Checks
23, 144
18,964
13,089
6.876
Probability
Limits (%>
Lower Upper
-14 +09
-JO +09
-09 +07
-13 +13
Probability Limits (%)
No. of
Audits
1,367
1.524
1,208
479
Level 1
Lower Upper
-16 +10
-12 +10
-11 +10
-21 +17
Level 2
Lower Upper
-12 +09
-09 +08
-06 +06
-14 +10
Level 3
Lower Upper
-12 +09
-09 +08
-06 +05
-12 +07
sary to assess data quality. Details of the
results are discussed in the analysis
section.
National Precision
Results Comparison
While this report represents the
second year of precision and accuracy
data, it is too early to determine reliably
any trends analysis. However, some ten-
tative observations can be made. As can
be seen in Figure 1, some minor changes
have occurred since the start-up in 1981.
O3 and CO showed the most overall
change in precision with a decrease in
the limit spread for both upper and lower
limits. For TSP and SO2 the upper limits
stayed the same while the lower limit
increased from -13 to -12, decreasing
overall variability only slightly. The
spread or variability for NO2 precision
probability limits remained the same but
did exhibit an upward shift, possibly elim-
inating a slight average bias which
existed the first year.
National Accuracy
Results Comparison
Accuracy for TSP, which is determined
from an air flow measurement on the
high volume sampler, deteriorated
slightly from 1981 to 1982. Figure 2 dis-
plays the national accuracy average for
TSP and each of the three levels for con-
tinuous 03, CO, S02, and N02.
For the continuous analyzers, it is
obvious from Figure 2 that the first level
of each pollutant has a wider range than
the other levels. This is to be expected
with low concentration values. Levels 2
and 3 are expected to be similar to each
other and show less variability than level
1, as expected.
Examination of Figure 2 demonstrates
the improvements that have been made
across the country in the accuracy mea-
surements. For each level of ozone, the
range of the limits have decreased from
1981 to 1982. The same trend occurs for
Comparison of Results
from the PARS and the
Performance Audit
Program
A general comparison between the
accuracy data of the PARS program and
the Performance Audit (PA) data is
included in this report. The audit data are
the results of an independent check, the
National Ambient Air Audit Program,
conducted by the Quality Assurance Div-
ision (QAD) of the EMSL. Blind samples
are sent to laboratories that perform the
State and local agencies' analyses. The
samples are analyzed and results are
sent to QAD where they are evaluated.
Since precision assessments are not
made in the PA program, only accuracy
can be compared across the PARS and
the PA programs. For the purpose of this
report, the results from PARS and the PA
system are compared at approximately
National Averages for Precision
1981-1982
Figure 1. National precision averages for 1981 to 1982.
-------
the same levels by matching laboratories
and reporting organizations. Since the
PARS data are presented with outliers,
the same approach was taken with the
audit data. Knowledge of the historical
audit data reports, however, indicates
that the presence of outliers may make a
significant difference in the average
audit results for the PA.
Comparisons of the national averages
of the probability limits (Table 4) exhibit
good agreement between the results of
the two programs. However, there is con-
siderable variation between the results of
the two programs when comparisons are
made on Regional and reporting organi-
zation bases. Lack of better agreement
results from several factors. First, the
inclusion of outlier values in the PA data
appears to have introduced some exces-
sive distortion of general trends. Second,
even though the PARS averages in Table
4 are weighted by the number of audits,
variations due to many sources of error
for both data sets are averaged together
to obtain the national averages, thereby
masking any correlations which may
have existed for the results of individual
agencies. Third, the concentration levels
for the two systems do not coincide
exactly at each of the audit levels. Fourth,
the PA data are the results of independ-
ent external audits, while the PARS accu-
racy data are based on the results of
independent internal audits. The
expected effects of the last-mentioned
factor would cause the spread of the lim-
its for the PA to be wider tha n that for the
PARS. Examination of the results (see
Table 4) confirm these expectations.
40-
30-
20-
70-
0
•8
-10-
-20-
-30-
-40-
National Averages for Accuracy
1981-1982
~I I 1 1 1 1 1 1 1 1 1 1 T
2123123123123
Level
Figure 2. National Accuracy Averages for 1981 to 1982.
Table 4. Summary Comparison of EMSL Performance Audits (PA) vs
PARS Accuracy Audit Data for Year 1982
I
1981
1982
National Averages
Probability Limits (%)
Pollutant &
Method Code
CO
PA
PARS
N02
PA
PARS
SOz
PA
PARS
LEAD
PA
PARS
HIV
PA
PARS
SOz (Cont)
PA
PARS
Audits
1704
(1122)
127
(526)
130
(445)
377
(529)
2860
(5475)
363
(656)
Level 1
Lower Upper
-17 +12
(-15) MS)
-24 +21
(-11) (+12)
-32 +22
(-18)1+11)
-37 +35
(-16) (+11)
-27 +21
(-19) (+14)
Level 2
Lower Upper
-7 +8
(-7) (+8)
-20+15
(-8) (+9)
-21 +20
(-12)(+ 8)
-24 +20
(-11K+ 8)
-13 +12
(- 7) (+7)
-25 +21
(-17) (+13)
Level 3 -
Lower Upper
-7 +7
(-7) (+7)
-25 +19
(-7)1+7)
-15 +2
(-10) (+ 7)
-22 +14
-25 +21
(-16) (+13)
Level 4
Lower Upper
(-3) (+3)
-23 +16
-13 + 9
-22 +20
(-16)1+12)
. S. GOVERNMENT PRINTING OFFICE: 1985/559-111/20627
-------
-------
E. Gardner Evans. Raymond C. Rhodes (also the EPA Project Officer, see below),
William J. Mitchell, and John C. Puzak are with Environmental Monitoring
Systems Laboratory. Research Triangle Park, NC 27711.
The complete report, entitled "Summary of Precision and Accuracy Assessments
fortheState and Local Air Monitoring Networks 1982," (Order No. PB 85-208
171/AS; Cost: $16.00, subject to change) will be available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at:
Environmental Monitoring Systems Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45^68
BULK RATE
POSTAGE & FEES PAI
EPA
PERMIT No. G-35
Official Business
Penalty for Private Use $300
EPA/600/S4-85/031
------- |