United States
Environmental Protection
Agency
Environmental Monitoring
Systems Laboratory
Research Triangle Park, NC 27711
Research and Development
EPA/600/S4-88/037 Jan. 1989
&EPA Project Summary
Precision and Accuracy
Assessments for State and
Local Air Monitoring Networks
1981-1986: Supplement to
EPA/600/4-88/007
Raymond C. Rhodes
Precision and accuracy
assessments of air quality data
obtained from state and local air
monitoring agencies from 1981
through 1986 are summarized in
graphical form for each reporting
organization. For most reporting
organizations, the graphs show an
improvement in data quality during
the six-year period. Comparisons
across reporting organizations for
the same pollutant measurements
reveal persistent, significant
differences.
The data quality assessments are
also summarized and shown
graphically for each Region and for
the nation. Persistent, significant
differences are evident among the
Regions. Investigation to identify the
causes of these differences and
implementation of appropriate
corrective actions are recommended
to further improve data quality. The
importance of the originating
agencies' plotting of quality control
charts for the individual site results
is emphasized. Examples of the
charts included in Florida's quarterly
quality assurance report are
presented and discussed.
The work covered by the full report
was completed as of July 1988. The
report is a supplement to EPA/600/4-
88/007.
This Project Summary was
developed by EPA's Environmental
Monitoring Systems Laboratory,
Research Triangle Park, NC, to
announce key findings of the research
project that is fully documented in a
separate report of the same title (see
Project Report ordering information at
back).
Introduction
Federal regulations promulgated on
May 10, 1979 require the state and local
air monitoring agencies to perform
special checks to assess the accuracy
and precision of their ambient air
measurement systems for the criteria
pollutants. The special checks have been
required since January 1, 1981, and a
summarized form of the results is
reported to the EPA, Environmental
Monitoring Systems Laboratory (EMSL),
Research Triangle Park (RTP), North
Carolina for further summarization and
evaluation.
Details concerning the performance of
the special checks and the reporting of
the results have been presented in
annual reports for the years 1981 through
1986. The annual results for each
reporting organization were tabulated in
the reports, and graphical presentations
were made to show time trends from
year to year for the EPA regions and the
nation.
-------
The purpose of the full report is to
present graphs of the results for each
reporting organization for the six-year
period, 1981-1986, so that time trends
and other relationships can be shown.
Revised regulations of March 19, 1986
require that the special checks for
accuracy and precision be continued and
that all of the raw data rather than
summarized results be reported to
EMSL/RTP for all special checks made
after December 31, 1986. Because the
more detailed reporting of data for 1987
and beyond will enable a more detailed
summarization, analysis, and evaluation
to be made, it was deemed appropriate
to provide this six-year graphical
summary for each reporting organization
to the states and EPA Regional Offices.
Also included in the full report is some
discussion of the need for the originating
agencies to maintain quality control
charts for each site. Examples of charts
that the State of Florida includes in its
quarterly quality assurance report are
presented and discussed.
Results by Reporting
Organization
Air monitoring data quality
assessments, as 95 percent probability
limits, are shown graphically for each
reporting organization by year.
Figures 1 and 2 illustrate the form and
arrangement of the graphs for each
reporting organization. The example,
Kentucky, Jefferson County Air Pollution
Control District, was selected because it
shows the reporting for all possible
measurement methods. The first page
shows the graphs for Continuous
Methods, while the second shows the
graphs for Manual Methods The
Continuous Methods are presented in the
following order, from top to bottom: CO,
S02, N02, and 03. The Manual methods
are presented in the following order, from
top to bottom: Total Suspended
Particulates (TSP), S02, NOs, and Pb.
The scales for all corresponding graphs
are the same throughout to enable visual
comparisons to be made from reporting
organization to reporting organization.
The upper and lower extremes of the
vertical bars shown for each year
represent the upper and lower 95
percent probability limits, respectively,
for all of the precision and accuracy
checks made during the year m the
reporting organization. It is expected that
5 percent of the results of the individual
precision or accuracy checks would
exceed these limits. The mid-points of
the vertical bars are connected to better
show trends and biases. In all cases the
vertical scales on each chart are the
percent deviation of an observed
precision and accuracy result from a test
or reference value.
For more detail concerning the actual
precision and accuracy checks and the
computations of the 95 percent
probability limits, the reader is referred to
the previously issued annual reports.
As can be seen in Figures 1 and 2,
the scale ranges (%) are as follows:
Continuous
Methods
CO
SO2
N02
03
Precision
-40 to +40
-40 to + 40
-40 (0 +40
-40 to +40
Accuracy
All Levels
-40 to +40
-50 to + 50
-70 to + 70
-40 to + 40
Manual
Methods
TSP
S02
NO2
Pb
-40 to + 40
-100 to +100
-80 to + 80
-80 to + 80
-30 to + 30
-30 to + 30
-20 to + 40
-40 to +40
Figures 1 and 2 also show that although
the graphs for the continuous SOa and
NOa and the manual S02 and N02 are
located in the same positions on the
page for comparison purposes, the
scales are different. Always shown at the
top of each page is the EPA Region
number, the state number and name, and
the reporting organization number and
name.
Order of Presentation
The graphs in the report are
presented in the following order:
1. By Region, in numerical order;
2. Within Region by state, in
alphabetical and numerical order.
3. Within State by reporting
organization, in numerical order.
In addition to the graphs for the 169
reporting organizations in the nation,
summary graphs are included for each
the EPA Regions and for the nation.
Observations from the
Graphical Presentations
A number of trends or effects can
readily perceived by a visual review
the graphs. Examples of some of the
trends or effects are given below. T
discussions of the various exampl
shown in the figures are not intended
be complete. A critical review
particular graphs may reveal oth
significant patterns.
Negative or Positive Bias. A slig
negative bias for the continuous Si
method has been noted from t
Regional graphs of all of the anni
reports, 1981-1986 (see Figure :
Possible explanations for these negati
biases, given in the annual reports, ha
been the negative instrument drift of t
analyzers or degradation of tl
precision/accuracy check materials, i
example of these negative biases 1
reporting organizations is given belo
The negative bias for precision ai
accuracy is about 10 percent. A fe
reporting organizations show a positi
bias.
Time Trends. For many of the reportii
organizations, improvement is ve
evident, as shown by decreasing widt
of the probability limits (Figure '
However, some reporting organizatio
show either no improvement or
worsening.
Wide Limits, Narrow Limits. Son
reporting organizations have considerat
wider probability limits than othei
indicating much poorer precision
accuracy. Such reporting organizatio
should perform a thorough review of thi
measurement systems to determine t
causes of the excessive variability and
take appropriate corrective actions
improve their results. Something m^
also be gained from discussions wi
those reporting organizations that see
to do much better than most. The grap
for Figure 5 show consistently wide
narrow limits.
Similar Patterns of Variability. Normally,
biases occur at one level of accurac
they also occur at the other levels
well. These conditions would exist whe
all of the limits are too wide, too narro
show the same trend over time, el
Several examples of such patterns a
shown in Figure 6. In other case
however, the patterns from level to lev
may not be uniform or smooth but m,
-------
35% Probability Limits 81 -86
Continuous Methods
Reg: 04
State: 18 Kentucky
Reporg: 002 KY, Jefferson Co. Air Poll. Control Dist
Precision
Accuracy
Level 2
+40 -r
Level3
+40 T
-40^81 82 83 848586 -40-*-81 82 83 84 85 86 -40-*-81 82 83 84 85 86 -*0 -1- 81 82 83 84 85 86
S02
+40 -r +50 T +50 T +50 -r
-40 * a 1 82 838485 86 -50-*-81 82 83 84 85 86 50 81 82 83 84 85 86 -so ^ 81 82 83 84 85 86
+70 -r +70 -r +70 T
-40-*- 81 82 83 84 85 86 -70-'si 82 83 84 85 86 -70^81 82 83 84 85 86 -70-1-81 82 83 84 85 86
03
+40-r +40 T +40 T +40-r
-40 * 81 82 83 84 85 86 -40-*-81 82 83 84 85 86 ~40 81 82 83 84 85 86 ~40 L 81 82 83 84 85 86
Figure 1. Example graphs for a reporting organization, continuous method results.
vary considerably in a similar pattern
from year to year.
Trends Across Accuracy Levels. The
probability limits, which are percentages,
are generally wider for level 1 than level
2 and wider for level 2 than level 3. In
other words, the length of the vertical
bars decreases with increasing level
This is the usual relationship for most
measurement methods.
The decreasing limits with increasing
level is particularly evident in Figure 7.
In many of the other charts, the
above relationship may not be evident
visually but would be shown from the
actual numerical values This relationship
is more evident from the regional and
national charts that appear in Appendices
H and I of the full report, as these result
from a much larger amount of data.
Zero Limits. Some of the charts show the
upper and lower probability limits as
zeros (Figure 8). Such instances
represent years when no data were
reported. The charts for the EPA Regions
and the nation show similar patterns and
-------
95% Probability Limits 81-86
Continuous Methods
Reg: 04
State: 18 Kentucky
Reporg: 002 KY, Jefferson Co. Air Poll. Control Dist
Precision
T
0-
-40-
+100-
0-
-700-
A
+SO-
0-
-SO-
P
+80-
0-
-80-
SP
8
0,
LS;
IO2
-H
/ 82 t
8.
t
N,
81 t
I
1
Accuracy
Level 1 Level 2 Level 3
+30-
"TT" \ °-
(3 84 85 86 -30
+30 -]
283848586 -3°
2 \
B
1
81 82 8
+40-,
3848586 -40
+40-i
u-i T-T 0
3 84 Rfi RK -40
+30-
"Tb ;
r \
' /
81 82 83 84 85 86 ~30
1
r +40i
:U r:
r i
' S/ 82 83 848586 -40
+40-
Tj "
81 8^ R.1 R4 RK Rfi -40-
- I
- 81 82 83 84 85 86
+30-
"M
/J-l 0
L S/ 82 83 84 85 86 ~30
- * +40-
ill c'
M
81 82 83 84 85 86 ~40
- -r
. /
81 82 83 84 85 86
1
T TT
.IT*
81 82 83 84 85 86
H4^i^
81 82 83 84 85 86
Figure 2. Example graphs for a reporting organization, manual method results.
trends as discussed above for reporting
organizations.
Control Charts for Precision
and Accuracy Data
The importance of the originating
agencies preparing and maintaining
control charts for each site in a timely
manner is emphasized. Reference is
made to EPA/600/4-83/023, Guideline
on the Meaning and Use of Precision
and Accuracy Data Required by the 40
CFR Part 58, Appendices A and B, June
1983, which provided instructions on the
preparation of the control charts.
Examples of such charts are included in
the State of Florida's quarterly quality
assurance report. Sample copies of
Florida's report can be obtained or
written request to:
Mr. Don Stuart, Administrator
Quality Assurance Section
Bureau of Air Quality Management
Twin Towers Office Building
2600 Blair Stone Road
Tallahassee, Florida 32399-2400
-------
Precision
Level 1
Accuracy
Level 2
Levels
+40-r
+50-r
-m
+50-T-
+50-.-
-40-L 81 82 83 84 85 86 -50 J- 81 82 83 84 85 86 -50-L 81 82 83 84 85 86
Figure 3. Example of negative bias reported for SOi continuous method.
-5QJ-81 8283848586
Precision Accuracy
Level 1 Level 2
03
+40-
0-
-40-
r- +40 -
- a
ITTT ;
1 82 83 84 85 86 -40 -
+40-,
"llllii r"
1 1 n
- 81 8283848586 -40-
_
- £
-111
tiu
1 82 838
Level3
+40-1
T T 0
J. 1 u
4 85 86 -40
'T-rTT
- 81 82 83 84 85 86
Figure 4. Example of improvement in precision and accuracy over time.
Precision
Level 1
Accuracy
Level 2
CO
+40 T
+40-r
-40 -1- 81 82 83 84 85 86 -40 ±- 81 82 83 84 85 86 -40-L 81 82 g3 84 85 gg -40 -«- gj 82 83 84 gs 86
+40 -r- +4
PB
+80-r
-H-H-i
±
-80 -L 81 82 33 84 85 86 -40-*- gj g2 83 S4 85 86 -40 -^ 81 82 83 84 85 86
Figure 5. Examples of consistently wide and narrow probability limits.
-------
Precision
Accuracy
Level 1 Level 2
CO
+40-
O-
-40-
111
n]
+40-,
ET T^*i n
1 { -I °
818283848586 -40-
S02
+40-1
0-
-An-
T T
.H
ai a-> a
r +40~\
'klTjJj "
-818283848586 ~40
'Ki
T
i- \
l^
^
Level 3
+40-,
. 0:
- 87 82 83 84 85 86 -40 -
+50-
-**'"- ^1
\| °
7 OA DK f»K -50
r +50-
*. _ n
h
fl> C9 O1 DA BK OK ~50
- a
M 14
I 1
87 82 83 84 85 86
+50-i
n _
K i
i ay a1} DJI OK ac -50
_ -^vl
- 01 eo 01 DA OK ac
Figure 6. Examples of graphs showing results that vary considerably in a similar pattern from year to year.
Precision
CO
+40 -,-
Level 1
+40 -r-
Accuracy
Level 2
Level3
+40-1-
+40 -r
I I I I I
-40 J. 81 82 83 84 8586 ~40 *- 81 82 83 84 85 86 -40-i-81 82 83 84 85 86 -40 -L g / 82 83 84 85 86
Figure 7. Example graphs showing decreasing limits with increasing accuracy levels.
Precision
S02
+100-.-
Level 1
+30 -,-
Accuracy
Level 2
Level 3
+30-r
+30-,-
-/OO J- 81 82 83 84 85 86 -30 -L 81 82 83 84 85 86 -30-L 81 82 83 84 85 86 -30 -*- 81 82 83 84 85 86
Figure 8. Example graphs for years when no data were reported.
-------
Several examples are given below
Figure 9 shows a definite bias between
the two collocated TSP samplers during
the first and second quarters of 1987
Figure 10 reveals a significant shift in the
bias from the first to the second quarter
of 1987.
A definite improvement in the uniformity
of the precision check results for the
ozone results is evident from Figure 11.
Conclusions and
Recommendations
The charts in the appendices of the
full report show in graphical form the
yearly results of precision checks and
accuracy audits for each of the years
1981-1986 for each reporting
organization in the nation and reveal a
number of persistent and significant
trends and relationships. A general
improvement in monitoring data quality
during these years is indicated However.
significant differences exist among
reporting organizations and EPA
Regions. These differences warrant
continued investigation by the Regional
Offices to further improve and make
more uniform the quality of data obtained
by the ambient air monitoring networks
of the nation.
Recommendations are again made
that quality control charts, as described
and recommended in Guideline on the
Meaning and Use of Precision and
Accuracy Data Required by 40 CFR Part
58, Appendices A and B, EPA/600/4-
83/023, June 1983, be prepared and
maintained in a timely manner for each
monitoring site to detect as soon as
possible significant trends, biases, and
out-of-control conditions Corrective
actions resulting from the appropriate
use of control charts for the precision
and accuracy data will further improve
data quality.
Examples of charts maintained by
the State of Florida (included in their
quality assurance report) are provided to
illustrate how these charts are made and
used to detect significant trends and
biases that should be investigated and
corrected It is recommended that all the
states prepare and issue a quarterly or
an annual quality assurance report
similar to that issued by the State of
Florida.
»" r"**in «"««
.....r. pT»<-<«if-i »» III! I 120Q-OQS-FM »M«««tn TSP
/t
Offici*
fici
1? SK &
_>£
. i
I .irr.j,
-k
>?
2-7.S-.
.irsT
-3
'
£*
-3
Jbci
A
M-Ml
r«
.7i!(^
ft
" tgu-t»T_-» y»
(2
.'1
V
St.
& X>. *
Si
-J
?;
Figure 9. Results of collocated TSP samplers at a site.
PtSnOCT
W-IIT «SSW« KTMI
Pr-Cilim »-> nil I ;H-tH»3-r02
I tin Hi \tol
Officll
&U
J3»3B
Is.
.1
.'
A
if.
«?.
?/ M.
life
r'^3?
. ti
3
tt
tt. 2,
>4
Z.%
SkSt
w,
&-
Figure 10. Results of collocated TSP samplers at a second site.
-------
W.ITT KSMHO. KfKt
mm 1987
> Mil I «140-002-r01
Ozone
Known
1UMI.1
t >irr.«.
-//
«7krt
«»
v.
»!
ssfcstktiiiffcfil
ba
-fr
L*U
re?.
oskoj
22
hL
ȣi
.ft.
A!k2T
f£«»J
SE
sntel
II IT II H H II 11 H 1' 11 II " 1'
F
. -j-
Figure 11. Results of precision checks for an ozone analyzer.
The EPA author Raymond C. Rhodes (also the EPA Protect Officer/ is with the
Environmental Monitoring Systems Laboratory, Research Triangle Park,
NC 27711.
The complete report, entitled "Precision and Accuracy Assessments for State
and Local Air Monitoring Networks, 1981-1986: Supplement to
EPA/600/4-88/007," (Order No. PB 89-126 726 AS: Cost: S36.95,
subject to change) will be available only from:
National Technical Information Service
5285 Port Royal Road
Springfield, VA22161
Telephone: 703-487-4650
The EPA Project Officer can be contacted at:
Environmental Monitoring Systems Laboratory
U.S. Environmental Protection Agency
Research Triangle Park, NC 27711
United States
Environmental Protection
Agency
Center for Environmental Research
Information
Cincinnati OH 45268
Official Business
Penalty for Private Use $300
EPA/600/S4-88/037
0000329 PS
------- |