o
1
/
Guideline on the Meaning and
The Use of Precision and Bias
Data Required by 40 CFR Part 58
Appendix A
FINAL DRAFT
-------
EPA-454/B-07-001
January, 2007
Guideline on the Meaning and the Use of Precision and Bias Data
Required by 40 CFR Part 58 Appendix A
By:
Louise Camalier
Shelly Eberly
Jonathan Miller
Michael Papp
FINAL DRAFT
U.S. Environmental Protection Agency
Office of Air Quality Planning and Standards
Air Quality Assessment Division
Air Quality Analysis Group
Research Triangle Park, North Carolina
11
-------
Foreword
The EPA's Ambient Air Quality Monitoring Program is implemented under the authority of the
Clean Air Act to provide air quality data:
1. Provide air pollution data to the general public in a timely manner.
2. Support compliance with air quality standards and emissions strategy development
3. Support air pollution research studies.
EPA recognizes the importance of collecting data across the nation that one can be assured that it was
of acceptable and consistent quality. The ambient air monitoring regulations were revised in 1979
and at that time two Appendices were added:
• Appendix A- Quality Assurance Requirements for State and Local Monitoring Stations
• Appendix B-Quality Assurance Requirements for Prevention of Significant Deterioration
Monitoring
A 1983 guidance document titled "Guideline on the Meaning and Use of Precision and Accuracy
Data Required by 40 CFR Part 58 Appendices A andB was developed as a companion document to
the Appendices to help explain the rational for the statistics and their use. On October 17, 2006 the
EPA Administrator signed the Ambient Air Monitoring Rule. This rule changed a number of
requirements in 40 CFR Appendix A. One important change was the statistical techniques use
estimate the precision and bias of the various quality control and performance evaluation checks
included in Appendix A.
The objective of this Guideline is to provide the monitoring organization with a description of the
ambient air monitoring quality system, the quality control techniques in the Appendix A regulations
and provide the guidance and spreadsheets necessary for to understand and implement these
statistics. This document is intended to the replace the 1983 Guideline.
The document is separated into two sections. Section 1 provides the background and rationale for the
statistics while Section 2 provides the guidance for the new statistics. Those just interested in how to
calculate the new statistics may want to proceed to Section 2.
The statements in this document, with the exception of referenced requirements, are intended solely
as guidance. This document is not intended, nor can it be relied upon, to create any rights
enforceable by any party in litigation with the United States. EPA may decide to follow the guidance
provided in this document, or to act at variance with the guidance based on its analysis of the specific
facts presented. This guidance may be revised without public notice to reflect changes in EPA's
approach to implementing 40 CFR Part 58, Appendix A.
This document is available on hardcopy as well as accessible as a PDF file on the Internet under the
Ambient Monitoring Technical Information Center (AMTIC) Homepage
(http://www.epa.gov/ttn/amtic/pmqa.html). The document can be read and printed using Adobe
Acrobat Reader software, which is freeware that is available from many Internet sites (including the
EPA web site).
in
-------
Table of Contents
Section Page
Foreward iii
Table of Contents iv
List of Figures iv
List of Tables v
List of Acronyms v
1.0 Introduction 1
1.1 Goal of the Guideline 1
1.2 Background 1
1998-2000 PM2.5 and the National Ambient Air Monitoring Strategy 2
1.3 Link between Data Quality Objectives, Data Quality Indicators and
Measurement Quality Objectives 3
Data Quality Objectives 3
Data Quality Indicators 4
Measurement Quality Objectives 5
Measurement Quality Data Aggregation-The Primary Quality Assurance
Organization 6
1.4 The Development of the New Statistics 8
2.0 The New Statistics: AFool-Proof Method and DASC Tool 12
2.1 Gaseous Precision and Bias Assessment 14
2.2 Precision Estimates from Collocated Samples 17
2.3 PM2.5 Bias Assessment 19
2.4 PM2.5, PMio-2.5 Absolutes Bias Assessment 21
2.5 One-Point Flow Rate Bias Assessment 23
2.6 Semi- Annual Flow Rate Audits 25
List of Figures
Figure Page
1 Effect of positive bias on the annual average estimate resulting in an incorrect
declaration on non-attainment 3
2 Effect of negative bias on the annual average estimate resulting in an incorrect
declaration on attainment 3
3 Relationship of data quality objectives to data quality indicators, measurement
quality objectives and data quality assessments 4
4 Precision and bias estimates for a hypothetical example 9
5 DASC Main Menu 12
6 Gaseous Precision and Bias DASC Worksheet 14
7 Collocated Precision DASC Worksheet 17
8 PM2.5 Bias DASC Worksheet 19
9 Absolute Bias DASC Worksheet 21
10 One-Point Flow Rate DASC Worksheet 23
11 Semi-Annual Flow Rate Audits DASC Worksheet 25
12 Lead Bias DASC Worksheet 27
iv
-------
List of Tables
Table
1
2
3
4
age
Ambient Air Monitoring Measurement Quality Samples
Reporting Organization (Old) and Primary Quality Assurance Organization (New)
Definitions in 40 CFR Part 5 8 Appendix A
Data Quality Indicators Calculated for Each Criteria Pollutant
Minimum Concentration Levels for Particulate Matter Precision Assessments
...7
.12
.17
Acronyms
AQS Air Quality System
AMTIC Ambient Monitoring Technology Information Center
CASAC Clean Air Scientific Advisory Committee
CFR Code of Federal Regulations
CV coeffi ci ent of vari ati on
DASC Data Assessment Statistical Calculator
DQA data quality assessment
DQI data quality indicator
DQO data quality objective
EDO environmental data operation
EPA U.S. Environmental Protection Agency
FW focus workgroup
MQO measurement quality objective
NAAQS National Ambient Air Quality Standards
NPAP National Performance Audit Program
OAQPS Office of Air Quality Planning and Standards
ORD Office of Research and Development
P&A precision and accuracy
PEP Performance Evaluation Program
PM2.5 particulate matter < 2.5 microns
PQAO primary quality assurance organization
QA/QC quality assurance/quality control
QA quality assurance
QAPP quality assurance project plan
SLAMS state and local monitoring stations
SOPs standard operating procedures
-------
Final Draft 01/19/07
Section 1: Introduction
1.1 Goal Of the Guideline
On October 17, 2006 the EPA amended its national air quality monitoring requirements. This
rule changed a number of requirements in 40 CFR Part 58 Appendix A, the section which
describes the planning, implementation, assessment and reporting of the ambient air monitoring
quality system. One important change was the statistical techniques used to estimate the
precision and bias of the various quality control and performance evaluation checks included in
Appendix A.
Prior to this revision, the statistics used to estimate precision and bias (then called accuracy)
where developed in the late 1970's. In 1983, the guidance document titled "Guideline on the
Meaning and Use of Precision and Accuracy Data Required by 40 CFR Part 58 Appendices A
andB"1 (hereafter referred to as "1983 Guideline") was developed as a companion to Appendix
A and B to help explain the rationale for the statistics and how they were used.
The objective of this new Guideline is to provide the data user with a brief history of the
establishment of the ambient air monitoring quality system, the quality control techniques that
have been in place up until the promulgation of the new monitoring rule, and to provide the
guidance and spreadsheets necessary to understand and implement these new statistics. This
document is intended to the replace the 1983 Guideline.
1.2 Background
The EPA's Ambient Air Quality Monitoring Program is implemented under the authority of the
Clean Air Act to provide air quality data for one or more of the three following objectives:
Provide air pollution data to the general public in a timely manner.
Support compliance with air quality standards and emissions strategy development.
Support air pollution research studies.
In order to support the objectives the monitoring networks are designed with a variety of
monitoring sites that generally fall into the following categories which are used to:
1. determine the highest concentrations expected to occur in the area covered by the
network;
2. determine typical concentrations in areas of high population density;
3. determine the impact on ambient pollution levels of significant sources or source
categories;
4. determine the general background concentration levels;
5. determine the extent of regional pollutant transport among populated areas, and in
support of secondary standards; and
http://www.epa.gov/ttn/amtic/cpreldoc.html
-------
Final Draft 01/19/07
6. measure air pollution impacts on visibility, vegetation damage, or other welfare- based
impacts.
These different objectives can potentially require information of varying quality. EPA
recognized the importance of collecting data of acceptable and consistent quality. In the late
1970's EPA started developing consistent techniques to identify the objectives that required the
highest quality data and then to develop a set of requirements to collect and assess this
measurement quality information. The EPA embarked on the process very similar to what is
now called the Data Quality Objectives (DQO) Process and determined that the comparison of
data to the National Ambient Air Quality Standards (NAAQS) was the highest priority objective
and that data would be collected in a manner that minimized the uncertainty in making
attainment decisions. The ambient air monitoring regulations were revised in 1979 and at that
time two Appendices were added:
• Appendix A- Quality Assurance Requirements for State and Local Monitoring Stations
(SLAMS)
• Appendix B-Quality Assurance Requirements for Prevention of Significant Deterioration
(PSD) Monitoring
These appendices established the development of a quality assurance program to be implemented
at the reporting organization level of aggregation. The appendices identified quality control,
audits and performance evaluation techniques that would be implemented internally as well as by
external organizations like the EPA Regions, ORD and OAQPS, and established the statistical
techniques to evaluate the data quality indicators. The primary data quality indicators for the
ambient air program were identified as precision and accuracy (P&A).
The 1983 Guideline provided a rationale for the use of the P&A data that was required to be
collected in the two appendices mentioned. As was written in the 1983 Guideline, "the P&A
statistics represented a compromise between (a) theoretical statistical exactness, and (b)
simplicity and uniformity in computational procedures". The P&A statistics were aggregated by
reporting organization over various time periods and combined into a probability limit estimate.
1998-2000 PM2.s and the National Ambient Air Monitoring Strategy
In 1998, with the promulgation of the PM2.5 NAAQS, EPA formally implemented the DQO
process and established acceptance criteria for precision and bias using statistics which were a
departure from the statistics in the 1983 Guideline. During this time period, OAQPS and the
monitoring organizations were cooperating to develop a new Monitoring Strategy2. OAQPS
formed a QA Strategy Workgroup that set out to perform a thorough review of the Appendix A
requirements and improve the quality system where appropriate. One outcome of this review
was the suggestion that EPA look at a way to provide a more consistent set of statistics for the
estimates of precision and bias. As part of this process, the Workgroup endorsed the use of the
DQO process and the measurement quality objectives (MQOs) that lead to attainment of the
DQOs.
! DRAFT National Ambient Air Monitory Strategy, December 2005 http://www.epa.gov/ttn/amtic/monstratdoc.html
-------
Final Draft 01/19/07
Before describing the new statistics, one needs to understand a little about DQOs, data quality
indicators (DQIs), and measurement quality objectives (MQOs).
1.3 Link between Data Quality Objectives, Data Quality Indicators and
Measurement Quality Objectives
Data Quality Objectives
In order to provide decision makers with data of acceptable quality, OAQPS uses the DQO
process3 to determine the data quality requirements for the ambient air criteria pollutants. Data
quality objectives (DQOs) are a full set of
performance constraints needed to design
an environmental data operation (EDO),
including a specification of the level of
uncertainty (error) that a decision maker
(data user) is willing to accept in the data to
which the decision will apply. Throughout
this document, the term decision maker is
used. This term represents individuals that
are the ultimate users of ambient air data
and therefore may be responsible for:
setting the NAAQS, developing a quality
system, evaluating the data, or comparing
\ „ • Biased (+15%), mean = 16.6
\
Concentration
Figure 1. Effect of positive bias on the annual average estimate
resulting in an incorrect declaration of non-attainment.
a 0.04--
Unbiased, mean = 16
* Biased (-15%), mean = 13.6
15 20 25
Concentration
Figure 2. Effect of negative bias on the annual average
resulting in an incorrect declaration of attainment
data to the NAAQS. The DQO will be
based on the data requirements of the
decision maker. Decision makers need to
feel confident that the data used to make
environmental decisions are of adequate
quality. The data used in these decisions
are never error free and always contain
some level of uncertainty. Because of these
uncertainties or errors, there is a possibility
that decision makers may declare an area
"nonattainment" when the area is actually
in "attainment" or "attainment" when
actually the area is in "nonattainment".
Figures 1 and 2 illustrate how errors can
affect a NAAQS attainment/nonattainment
decision based on an annual mean
concentration value of 15. There are
serious political, economic and health
consequences of making such decision
errors. Therefore, decision makers need to
understand and set limits on the probabilities
3 Guidance on the Systematic Planning Using the Data Quality Objectives Process EPA/240/B-06/001 Feb. 2006
http://www.epa.gov/quality/qa_docs.html
-------
Final Draft 01/19/07
of making incorrect decisions with these data.
In order to set probability limits on decision errors, one needs to understand and attempt to
control uncertainty. Uncertainty is used as a generic term to describe the sum of all sources of
error associated with an EDO. Uncertainty can be illustrated as follows:
S20=S2p+S2m
where:
S0= overall uncertainty
Sp= population uncertainty (spatial and temporal)
Sm= measurement uncertainty (data collection).
Uncertainty =
Population
Measurement
Data Quality Indicators
2.Precision
S.Bias
4. Completeness
5. Comparability
6. Detectability
Quality Systefl*
Figure 3. Relationship of data quality objectives to data quality
indicators, measurement quality objectives and data quality
assessments.
Figure 3 provides a description of the
relationship between uncertainty and the
DQO. The estimate of overall
uncertainty is an important component
in the DQO process. Both population
and measurement uncertainties must be
understood. The DQOs are assessed
through the use of data quality indicators
(DQIs) which are the quantitative
statistics and the qualitative descriptors
used to interpret the degree of
acceptability or utility of data to the
user. The DQIs can then be used to
establish the MQOs which will be
discussed below. Once the MQOs are
established and monitoring is implemented, data quality assessments (DQAs) are performed to
determine whether the DQOs were achieved. If not, the monitoring program should take steps to
identify the major sources of uncertainty and find ways to reduce these uncertainties to the
acceptable levels.
Data Quality Indicators
The data quality indicators are:
Representativeness - the degree in which data accurately and precisely represents a
characteristic of a population, parameter variation at a sampling point, a process
condition, or an environmental condition.
Precision - a measure of mutual agreement among individual measurements of the same
property usually under prescribed similar conditions. This is the random component of
error. Precision is estimated by various statistical techniques using some derivation of
the standard deviation.
Bias - the systematic or persistent distortion of a measurement process which causes error
in one direction. Bias will be determined by estimating the positive and negative
-------
Final Draft 01/19/07
deviation from the true value as a percentage of the true value.
Detectability - the determination of the low range critical value of a characteristic that a
method specific procedure can reliably discern.
Completeness- a measure of the amount of valid data obtained from a measurement
system compared to the amount that was expected to be obtained under correct, normal
conditions.
Comparability - a measure of confidence with which one data set can be compared to
another
Accuracy has been a term frequently used to represent closeness to "truth" and includes a
combination of precision and bias error components. This term had been used throughout the
CFR but has been replaced with bias when there is the ability to distinguish precision from bias.
The quality system for the ambient air monitoring program focuses on understanding and
controlling (as much as possible) measurement uncertainty and because of that, mainly focuses
on the data quality indicators of precision, bias, detectability completeness and comparability.
Representativeness is addressed through network designs and is not, per-se, something that the
quality system can control through better measurements.
Measurement Quality Objectives
For each DQI one must identify a level of uncertainty or error that is acceptable and will achieve
the DQO. MQOs are designed to evaluate and control various phases (sampling, preparation,
analysis) of the measurement process to ensure that total measurement uncertainty is within the
range prescribed by the DQOs. This finally gets us to CFR where the various quality control
checks, like the one point quality control check for the gaseous pollutants or the paniculate
matter collocated instruments, are established. These checks help quantify a data quality
indicator and their acceptance criteria are the MQOs. Table 1 provides a complete listing of the
required measurement quality checks and the MQOs as they are currently defined in Appendix
A.
EPA has not changed the types of samples it uses to assess precision and bias. Although the
2006 rule has changed some of the names and some of their sampling frequencies, the basic
checks are the same. Although the types of checks have not changed, EPA changed the statistics
used to evaluate precision and bias and in some cases how the measurement quality data are
aggregated.
-------
Final Draft 01/19/07
Table 1. Ambient Air Monitoring Measurement Quality Samples (Table A-2 in 40 CFR Appendix A)
Method
One-Point QC:
CFR Reference
Coverage (annual)
Automated Methods
Minimum frequency
MQOs*
O3 Precision 7%, Bias + 7%.
for SO2, NO2, O3, CO Section 3.2. 1
Annual performance
evaluation Section 3.2.2
for S02, N02, 03, CO
Flow rate verification Section 3.2.3
PMio,PM2.5, PMio-2.5
Semi-annual flow rate Section 3 2 4
audit
PMio, PM2.5, PMio.2.5
Section 3.2.5
Collocated sampling
PM25, PMio-2.5
Section 3.2.7
PM Performance
evaluation program
PM2.5,PMio-2.5
Each analyzer
Each analyzer
Each sampler
Each sampler
15%
1 . 5 valid audits for primary
QA orgs, with < 5 sites
2. 8 valid audits for primary
QA orgs, with > 5 sites
3. All samplers in 6 years
Once per 2 weeks
Once per year
Once every month
Once every 6 months
Every twelve days
over all 4 quarters
Precision 10% , Bias + 10%
< 1 5 % for each audit
concentration
< 4% of standard and 5% of
design value
< 4% of standard and 5% of
design value
PM25, - 10% precision
PMio-2.5- - 15% precision
PM25, - + 10% bias
PMio-2.5. - ±15% bias
Manual Methods
Collocated sampling
PMio, TSP, PMio-2.5, PM2.
Flow rate verification
PMio(lowVol),PMio-25,
PM25
Flow rate verification
PM10 (High-Vol), TSP
Semi-annual flow rate
audit
PMio(low Vol), PMio-2.5,
PM25
Semi-annual flow rate
audit
PMio (High-Vol), TSP
Manual Methods
Lead
Every 12 days
PSD -every 6 days
Once every month
Once every quarter
Once every 6 months
PMio, TSP, PM2.5, - 10%
precision
PMio-2.5- - 15% precision
< 4% of standard and 5% of
design value
< 10% of standard and
design value
<_ 4% of standard and 5% of
design value
Once every 6 months <_ 10% of standard and
design value
3.3.land 3.3.5 15%
332 Each sampler
3.3.2 Each sampler
333 Each sampler, all locations
3.3.3 Each sampler, all locations
334 1. Each sampler
2. Analytical (lead strips)
3.3.7 and 3.3.8 1.5 valid audits for primary
QA orgs, with < 5 sites
2. 8 valid audits for primary
QA orgs, with > 5 sites
3. All samplers in 6 years
* Some of the MQOs are found in CFR and others in the QA Handbook Vol II (Appendix 15) which is under revision during the
development of this guidance document.
Measurement Quality Data Aggregation -The Primary Quality Assurance Organization
In order to assess whether or not measurement quality data meet the established DQOs, the data
must be aggregated in an appropriate manner. Prior to the new rule, measurement quality data
was aggregated by "reporting organizations". The 1983 Guideline described the reporting
Performance evaluation
program
PM25 PM,o-25
1. Include with TSP
2. Each quarter
Over all 4 quarters
1. Same as for TSP.
2. - + 10% bias
PM25, ±10% bias
PMio-2.5-, +15% bias
-------
Final Draft 01/19/07
organization as "a State or subordinate organization within a State which is responsible for a set
of stations which monitor the same pollutant and for which precision and accuracy assessments
can be pooled... and can be expected to be reasonably homogeneous as a result of common
factors". The term has very important implications to quality assurance activities. Reporting
organizations, from a QA standpoint, serve at least two purposes: 1) it allows one to group a
fewer number of QC data points that might be variable at one level (site level) into a larger set
(reporting organization) for more meaningful assessments in shorter time periods, and 2) it
allows expensive assessments that could not afford be accomplished at every site (collocated
precision, PEP) to be aggregated at higher levels that are representative of the sites within that
reporting organization.
The 1983 Guideline also pointed out that "the definition of reporting organization does not relate
to which agency or organization reports routine monitoring or to which agency or organization
reports precision or accuracy data, but rather to the total operational system involved in
sampling, calibration, analysis, and reporting for routine monitoring for a specific pollutant."
Unfortunately, this guidance did not appear to be consistently followed. Over the years, more
and more monitoring organizations gained the experience in reporting data to the Air Quality
Subsystem (AQS) and it appeared that some organizations were using the term not as it was
defined, but to identify itself as the agency reporting data to AQS.
Therefore, EPA believed that the term "reporting organization" had two applications. To combat
this potential double meaning, in the 2006 Appendix A revision, the term "Reporting
Organization" is replaced with the term "Primary Quality Assurance Organization (PQAO)". The
2006 rule adds one additional common factor to the old definition, but essentially the definition
remains the same. Table 2 provides the comparison of the old and new rule. The changes in the
proposed rule are highlighted in blue and underlined.
Table 2. Reporting Organization (Old ) and Primary Quality Assurance Organization (New) Definitions in 40 CFR Part
58 Appendix A
Old Rule (before 10/17/06)
New Rule
3.0.3 Each reporting organization shall be defined such
that measurement uncertainty among all stations in the
organization can be expected to be reasonably
homogeneous, as a result of common factors.
(a) Common factors that should be considered by in
defining reporting organizations include:
(1) Operation by a common team of field operators
(2) Common calibration facilities.
(3) Oversight by a common quality assurance
organization.
(4) Support by a common laboratory or headquarters.
3.1.1 Each primary quality assurance organization shall be
defined such that measurement uncertainty among all stations
in the organization can be expected to be reasonably
homogeneous, as a result of common factors. Common
factors that should be considered by monitoring organizations
in defining primary quality assurance organizations include:
(a) Operation by a common team of field operators
according to a common set of procedures:
(b) Use of a common OAPP or standard operating
procedures;
(c) Common calibration facilities and standards;
(d) Oversight by a common quality assurance
organization; and
(e) Support by a common management, laboratory or
headquarters.
EPA believes that the 5 common factors listed are the key criteria to be used when an agency
decides the sites to be considered for aggregation to a PQAO. The requirement does not intend
that all 5 factors have to be fulfilled but that these factors are considered. However, common
procedures and a common QAPP should be strongly considered as key to making decisions to
consolidate monitoring sites into a PQAO.
-------
Final Draft 01/19/07
Some of the precision and bias statistics can be evaluated at the site or instrument level; others
must be evaluated at the PQAO level. In general, any measurement quality sample in Table 1
that has a coverage indicated as "each sampler/analyzer" can and will be evaluated at the
site/instrument level. This data can also be aggregated at the PQAO level and the precision and
bias statistics perform the appropriate evaluation at both site and PQAO level. Because only a
percentage of sites in any monitoring organization implement collocated sampling and the
Performance Evaluation Program (PEP) in any one year, the data must be aggregated and
evaluated at the PQAO level. Although this paniculate matter measurement quality data should
be used to evaluate the instruments from which the checks are made, the data aggregation to the
PQAO to assess the achievement of the DQO is of primary importance.
1.4 The Development of the New Statistics
As mentioned earlier in this document, a Focus Workgroup (FW), a subset of the QA Strategy
Workgroup, was formed to review and revise the precision and bias statistics. The FW proposed
that the MQOs be based on confidence intervals. That is, determining whether the bias and
precision variables meet the measurement quality objectives will be based on whether the
confidence intervals for these variables meet the measurement quality objectives. The intent of
this is two-fold. One reason for using confidence intervals is to be confident the measurement
quality objectives are being met. It is different to say the bias is 5% plus or minus 10%
compared to saying the bias is 5% plus or minus 1%. A second, and very practical, reason for
using confidence intervals is to allow organizations that show tight acceptable results the
flexibility in reducing the frequency of certain QC checks. For example, the site with a bias of
5% plus or minus 1% likely does not need as many QC checks as the site with the bias of 5%
plus or minus 10%. The acceptance criteria are based on the number of years of data that
coincide with the time frame of the ambient air quality standards. For example, since the 8-hour
ozone standard is based on 3 years of data, the acceptance criteria for bias and precision will also
be based on 3 years of data. Additionally, the acceptance criteria apply to each site operating an
automated method.
For the automated methods, estimates of both bias and precision are derived from the one-point
quality control checks and then double-checked with the annual performance evaluations,
independent State audits and the NPAP Program. To test the reasonableness of estimating bias
and precision from bi-weekly checks, the FW made up some actual/indicated pairs, assuming
different levels of bias and precision, and tested a couple of proposed statistics. The FW
simulated 3 years of data and provided summary statistics at the quarterly, annual, and 3-year
level. For each scenario, the data was summarized by the three methods below
1. CFR Probability Interval. For these statistics, EPA reviewed what was currently in
CFR, namely the overall percent difference in the actual and indicated concentrations and
an associated probability interval that shows where 95% of all the percent differences
should fall. Note that this does not provide separate estimates of bias and precision.
2. Signed Bias & Precision (CV). For this case, EPA estimate bias and precision
separately and also estimated confidence intervals for bias and confidence intervals for
precision. A comment on this approach is that if there is trend in bias, such as +10% one
-------
Final Draft 01/19/07
year, 0% the next year, and -10% the third year, then, from a 3-year perspective, you
may say the system is unbiased but very variable. This is how these statistics treat the
trend in bias. Thus the bias tends to be small and the precision large, in general.
3. Absolute Bias & Precision (CV). As with the signed case above, EPA estimated bias
and precision separately and also estimated confidence intervals for bias and confidence
intervals for precision. However, since the absolute value for bias is used, if there is
trend in bias, such as +10% one year, 0% the next year, and -10% the third year, then,
from a 3-year perspective, one may say the system has a great potential for bias but is
precise. This is how these statistics treat the trend in bias. Thus the bias tends to be large
and the precision small, in general.
Figure 4 shows the results for one of the hypothetical cases studied. It provides an example of
the various precision and bias estimates for a 3 year data set where the true measurement
imprecision is 5% and the true bias is 15% for year 1, 0% for year 2 and -10% for year 3. There
are 5 sections to the Figure.
Data and 3-Year DQO IZstinutles
Input:
CFR
Rob.
Interval
Figure 4. Precision and Bias Estimates for a Hypothetical Example
• The left-most area of Figure 4 shows the spread of the relative differences of the
biweekly checks for each of the years and for all the years combined. These box and
whisker plots show that the bias varies from year to year and that it is decreasing (the
center of boxes shifts from around 15% to 0% to -10%) but that the imprecision is small
(the boxes are small and whiskers short). On the other hand, the 3-year box and whisker
plot shows no bias (the box is centered about 0) but large variation (the box is wide and
the whiskers are long).
• The next section of Figure 4 shows the true bias (represented as *'s) and imprecision
(represented as a line from 0 to the amount of imprecision, 5% in this case).
• The next section shows the results based on the statistics currently in CFR. The center of
the interval is represented by the "*." The interval indicates where 95% of the past,
present, and future relative differences from the biweekly checks are expected to be.
-------
Final Draft 01/19/07
• The next section shows the bias and precision estimates and their respective confidence
intervals for the "Signed" case described above. Precision is represented by the term
"CV". The estimates are represented by a "*." Thus bias is nearly 0 and imprecision is
around 10%. The confidence interval for the signed bias is always centered on the "*."
In this example, the 90% confidence interval for bias is about -3% to 5%. The
confidence interval for the precision estimate always extends from 0 to the upper
confidence limit. That is, the lower confidence limit for precision is not shown since it
will always be between 0 and the estimate for precision. The upper confidence limit for
precision is about 12%. So based on "Signed" estimates, one would say that this site is
operating with a bias that is somewhere between -3 and +5% with an imprecision that
may be as large as 12%.
• The last section shows the bias and precision estimates and their respective confidence
intervals for the "Absolute" case described above. Again, the estimates are represented
by a "*" so for this case the bias is about 8% and the imprecision (CV) is around 5%.
The confidence interval for the absolute bias is always centered on 0. So for this
example, the bias has the potential to be as large as +12% or as small as -12% . And as
above, the confidence interval for the precision estimate always extends from 0 to the
upper confidence limit, which is about 6% in this case. So for the "Absolute" estimates,
this site is operating with a potential bias as small as -12% or as large as 12% with an
imprecision that may be as large as 6%.
The FW also reviewed the statistics against a number of routine monitoring sites. As the FW
reviewed the various cases and discussed the information they came to the following
conclusions:
• Use the absolute bias confidence interval and the signed precision (CV) confidence
interval as the statistics for setting the acceptable measurement quality objectives for the
data quality indicators of precision and bias. This provides a conservative approach but it
also allows flexibility in implementing quality control activities.
• Develop measurement quality objectives at the site level of data aggregation. Since
every site performs the biweekly checks at an acceptable frequency there is enough
information to assess and control data quality at the site level. Data will still be presented
by reporting organization because several QA decisions and QA implementation occur at
this level.
This FW provided the results of the study back to the larger QA Strategy Workgroup who
endorsed the conclusions and developed a paper4 proposing the new approach that was discussed
at Monitoring Strategy Steering committee meeting as well as being presented at the July, 20045
Clean Air Scientific Advisory Committee (CASAC) meeting where the statistics were endorsed.
As the QA Strategy Workgroup moved forward in its review of other facets of the ambient air
quality system, EPA pursued using similar statistics for the particulate matter parameters. In
4 Proposal: A New Method for Estimating Precision and Bias for Gaseous Automated Methods for the Ambient Air
Monitoring Program (http://www.epa.gov/ttn/amtic/parslist.html)
5 http://www.epa.gov/sab/fisclrpt.htm
10
-------
Final Draft 01/19/07
general, and as the following information will show, EPA has followed a similar path for the
majority of the parti culate matter measurement quality samples. One exception must be noted.
Since DQOs and there accompanying statistics had been developed for PM2.5 shortly before the
FW efforts to develop the new statistical techniques, and because EPA was preparing the
monitoring rule proposal at a time when PM2.5 design values were being compared to the
NAAQS, EPA did not want to modify the bias statistics to the new absolute bias confidence limit
technique. Therefore, although the precision statistic for PM2.5 has been changed to be consistent
with the gaseous pollutants, the bias estimate for PM2.5 has been maintained as written prior to
the 10/17/06 rule. However, for convenience, the companion software for this document will
provide an assessment of PM2 5 bias by both statistical methods.
11
-------
Final Draft 01/19/07
Section 2: The New Statistics: A Fool-Proof Method & DASC Tool
Section 1 provided a background on the Ambient Air Monitoring program, the use of the
precision and accuracy statistics prior to the promulgation of the new statistics in 2006, and an
explanation of how the new statistics in the 2006 rule were developed. This section and the
companion Data Assessment Statistical Calculator (DASC) tool has been produced specifically
for the data user community in an effort to provide an easy way to explain and calculate the new
data assessment statistics in CFR. Each equation explained in this document is numbered and
matches the numbering convention in CFR!
Given measurement quality data for a particular pollutant and site, this section provides a step-
by-step way to: (1) know what to calculate and (2) calculate it.
The DASC tool can be found under its filename, "P & B DASC", in the Precision and Accuracy
Reporting System within the Quality Assurance section of AMTIC6 and uses data that you input
as the basis to perform all calculations outlined in this document. The DASC contains eight (8)
different worksheets; one for each of the seven different categories of statistics that need to be
calculated, and the eighth being a menu selection tool to help you find the appropriate worksheet.
Table 3 illustrates what data quality indicators are applicable for each of the monitored
pollutants.
Table 3 Data Quality Indicators Calculated for each Measured Pollutant
What to Calculate
Pick a Statistic
Precision Estimate
Bias Estimate
Absolute Bias Estimate
Semi-Annual Flow Rate
One Point Flow Rate
Pollutants
03
V^
V
S02
V'
V'
N02
V^
V
CO
V'
V'
PM2.5
v^
V
V
V
V
PM10
V'
V
V
PMiQ-2.5
,/-
^S
W'
^
Lead
V^
V
DASC (Data Assessment Statistical Calculator)
Site : {Enter Site ID or Name Here}
Step 1
Pick a Pollutant
Automated Methods
r 302
'•' N02
r 03
r CO
'" PM 2.5
<"" PM10
r PM 10-2.5
Manual Methods
r PM2.5
•"" PM 10
<~ PM 10-2.5
<~ Lead
Step 2
Pick a Statistic to Calculate
f* Precision Estimate
•" Bias Estimate
Step 3
Go To Worksheet
The titles under the "Pick a Statistic" label correspond to
the titles of the worksheets in the DASC. Figure 5 is the
"Menu" worksheet.
At the menu you can select: 1) the pollutant, and 2) the data
quality indicator you want to calculate. Selecting the "Go
to Worksheet" button takes you to the worksheet. In the
case shown in Figure 5 the user would be taken to the
gaseous pollutant precision worksheet for NO2.
Figure 5. DASC main menu
'http://www.epa.gov/ttn/amtic/parslist.html
12
-------
Final Draft 1/19/07
All measurement quality checks start with a comparison of the concentration/value measured by
the analyzer (measured value) and the audit concentration/value (audit value) and all use
percent or relative percent difference as the comparison statistic. All other calculations are based
on these two "starting" statistics. To create a measurement quality spreadsheet using the DASC
tool, put the measured value data in Column A and the corresponding audit value data in
Column B. Remember to start the data in Row 4 so that Rows 1-3 are reserved for labels. All
subsequent calculations will be automatically generated by the spreadsheet. For those who have
used the AQS precision and accuracy transaction files, the measured value equates to the
"indicated value" and the audit value equates to the "actual value"
The spreadsheet has been created with a pre-defined set of 13 audit pairs to provide an example.
These values will need to be replaced by your sample data set. All the formulas in columns
C, E, F, and G, from row 4 through row 503 have been preset to calculate the necessary statistics.
If you plan to add more than 500 data pairs you will have to revise the excel spreadsheet. Each
worksheet allows you to print the data and graphs for that worksheet by using the "print
worksheet" button. The pages will be automatically set based on the number of values input. If
you would like to print out the entire workbook head back to the main menu.
Please Note
The DASC tool contains macros that need to be enabled in order for the spreadsheet to work
properly. When you first open the document, you should see a message that looks like this:
Security Warning
"C:\amqg\LC_Guidance.xls" contains macros.
Macros may contain viruses. It is usually safe to disable macros, but if the
macros are legitimate, you might lose some functionality,
DisableMacros
Enable Macros
More Info
Click on "Enable Macros" and the worksheet will open. If you do not get this message, make sure
that you have the macro security level setting set to "Medium". The macro security level setting can
be found in "Security" tab from the "Tools" -> "Options..." menu in Excel
Tools
Protection
Macro
Add-Ins.
Options,.
Lookup..
13
-------
Final Draft 01/19/07
2.1 Gaseous Precision and Bias Assessments
Applies to: CO, O3, NO2, SO2
40 CFR Part 58 Appendix A References:
• 4.1.1 Percent Difference
• 4.1.2 Precision Estimate
• 4.1.3 Bias Estimate
• 4.1.3.1 Assigning a sign (positive /negative) to the bias estimate.
• 4.1.3.2 Calculate the 25th and 75th percentiles of the percent differences for each site.
• 4.1.4 Validation of Bias Using the one-point QC Checks
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
13
13
20
21
22
23
24
75
26
27
23
23
A | B
C
D
E I F
G
H
I I
CO Assessments
Site ID: {Enter Site ID} [Pollutant type: CO ] CVut (%)
Meas Val (Y) Audit Val (X)
262 25
266 25
266 25
2.56 2 5
2 65 2 5
268 25
256 25
265 25
256 25
261 25
2.7 25
2.68 25
269 25
d (Eqn. 1) 25th Percentile
4300r 4400
6 400 75th Percentile
6.400' 7200
2.400
6000
7200
2400
6 000
2400
4400
8.000
7200
7600
d2
23040
40.960
40 960
5.760
36.000
51 840
5 760
36.000
5 760
19.360
64.000
51 840
57760
|d|
4800
6400
6400
2.400
6000
7200
2400
6000
2400
4.400
3.000
7.200
7600
[tf
23.040
40960
40360
5.760
36000
51 840
5 760
36.000
5760
19.360
64000
51.840
57760
n Sd
13 2022
n-1 Sd
12 71 200
CV (%) (Eqn 2)|
279|
J K
L
Bias (°i)
Sd! Z|d|
20.274 71 200
Zd2 Z|d|!
439 040 439 040
Bias (°i) (Eqn 3)
643
Signed Bias (%}
+648
"AB" (Eqn 4)
5.477
"AS" (Eqn 5)
2022
Both Signs Positive
TRUE
Both Signs Negative
FALSE
Upper Probability Limit Lower Probability Limit
9 44 ' 151
Return to M air
10000
o.occ , —
-E.OOO -
Menu Print
Percent Differences
— *. *~-+^ +. s* — • — *
"~-t^ ~~"*-^ """*--""*'
Figure 6. Gaseous precision and bias DASC worksheet example
Percent Difference
Equations from this section come from CFRPt. 58, App. A, Section 4, "Calculations for Data
Quality Assessment". For each single point check, calculate the percent difference, dt, as follows:
Equation 1
meas- audit
audit
100
where meas is the concentration indicated by the monitoring organization's instrument and audit
is the audit concentration of the standard used in the QC check being measured.
Precision Estimate
The precision estimate is used to assess the one-point QC checks for gaseous pollutants
described in section 3.2.1 of CFR Part 58, Appendix A. The precision estimator is the coefficient
of variation upper bound and is calculated using Equation 2 as follows:
14
-------
Final Draft 01/19/07
Equation 2
where $ o.i,n-i is the 10th percentile of a chi-squared distribution with n-1 degrees of freedom.
Bias Estimate
The bias estimate is calculated using the one point QC checks for 862, NC>2, 63, or CO described
in CFR, section 3.2.1. The bias estimator is an upper bound on the mean absolute value of the
percent differences as described in Equation 3 as follows:
Equation 3
where n is the number of single point checks being aggregated; to.95,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom; the quantity AB is the mean of the absolute values of
the dj 's (calculated by Equation 1) and is expressed as Equation 4 as follows:
Equation 4
and the quantity AS is the standard deviation of the absolute value of the dt's and is calculated
using Equation 5 as follows:
Equation 5
AS=]
Since the bias statistic as calculated in Equation 3 of this document uses absolute values, it does
not have a tendency (negative or positive bias) associated with it. A sign will be designated by
rank ordering the percent differences (4's) of the QC check samples from a given site for a
particular assessment interval. Calculate the 25th and 75th percentiles of the percent differences
for each site. The absolute bias upper bound should be flagged as positive if both percentiles are
positive and negative if both percentiles are negative. The absolute bias upper bound would not
be flagged if the 25th and 75th percentiles are of different signs (i.e. straddling zero).
15
-------
Final Draft 01/19/07
The final precision value should be found in cell 113 and the final bias estimate should be
found in cell K13 in the spreadsheet. See the corresponding spreadsheet ('P&B DASC.xls').
Validation of Bias Using the one-point QC Checks
The annual performance evaluations for SC>2, NC>2, Os, or CO are used to verify the results
obtained from the one-point QC checks and to validate those results across a range of
concentration levels. To quantify this annually at the site level and at the 3-year primary quality
assurance organization level, probability limits will be calculated from the one-point QC checks
using equations 6 and 7:
Equation 6
Upper Probability Limit = m + 1.96 • S
Equation 7
Lower Probability Limit = m - 1.96 • S
where, m is the mean (equation 8):
Equation 8
1 k
m= — Yd.
k tf
where, k is the total number of one point QC checks for the interval being evaluated and S> is the
standard deviation of the percent differences (equation 9) as follows:
Equation 9
k(k-l)
Please Note
P&B DASC.xls only calculates the upper and lower confidence limits on a per
site basis. A similar procedure would need to take place to calculate the upper
and lower confidence limits across a Primary Quality Assurance Organization.
16
-------
Final Draft 01/19/07
2.2 Precision Estimates from Collocated Samples
Applies to: PM2.s, PM10, PM10.2.s, Lead
40 CFR Part 58 Appendix A References:
• 4.2.1 Precision Estimate from Collocated Samplers
• 4.3.1 Precision Estimate(PM2 s &PM10_2 5)
• 4.4.1 Precision Estimate (Lead)
1
2
3
4
-:
6
7
B
9
10
11
12
13
14
15
15
17
18
19
20
£1
22
23
2-
25
25
27
2B
2t!
M
31
-J^
A | B
C
D
E
F
G
H
I
j
K
L
Precision Estimate (From Collocated Samples)
'Site ID: {Enter S te ID}
MeasVaim Audit Val (X)
5.05 5.55
12.35 13
14.67 1435
9.59 7.68
10.51 11.3
12.44 13.45
3.73 451
18.56 16.99
12.56 13.32
7.68 7.89
14.56 15.32
17.45 16.76
12.34 13.44
Pollutant type:
d(EqnlO) 25th Percentile cl'
-9.434' -7.802 89.000
-5.128 75th Percentile 26.298
2.205' 2.205 4.864
22.119
-7.244
-7 802
-17512
8.833
-5.873
-2.597
-5.037
4.034
-8.534
489.263
52.481
60.875
310.168
78.015
34.495
7.275
25.878
16.272
72.82;
CVUO (%)
HI'
9.434
5.128
2.205
22.119
7.244
7.802
17.612
8.833
5.873
2.697
5.087
4.034
8.534
lull'
89.000
26.298
4.864
489.263
52.481
60.876
310.168
78.015
34.495
7.275
25.878
16.272
72.825
n Z|cl| Z|cl|'
13 106.603 1267.710
n-1 zcl Sd"
12 -32.221 1267.710
CV(:')(Ec|n11!
9.71
Return
to Main Menu
M
tW^sheet
20.000 -
10 000
-20.000 -
-30.000 -
H
Percent Differences
*
/\
y \ A, ^
I-*'"" ' ' \ ' ^ ' / ' "V-1-"*— W \^
N/
Figure 7. Collocated precision DASC worksheet example
Precision is estimated for manual instrumentation via duplicate measurements from collocated
samplers at a minimum concentration (see table below for minimum concentration levels).
Table 4. Minimum Concentration Levels for Particulate Matter Precision Assessments
Pollutant
PM2.5
PMiQ-2.5
Lo-Vol PM2.5
Hi-Vol PM2.5
Lead
Minimum Concentration Level
(in ug/m3)
3
3
3
15
0.15
Precision is aggregated at the primary quality assurance organization (PQAO) level quarterly,
annually, and at the 3-year level. For each collocated data pair, the relative percent difference,
df, is calculated by Equation 4.
Equation 10
--100
17
-------
Final Draft 01/19/07
where Xt is the concentration of the primary sampler and Yt is the concentration value from the
audit sampler.
The precision upper bound statistic, CVub, is a standard deviation on d; with a 90 percent upper
confidence limit (Equation 11).
Equation 11
, .. -IS",
CV ub =
2n(n-l)
where, n is the number of valid data pairs being aggregated, and %o.i,n-i is the 10th percentile of a
chi-squared distribution with n-1 degrees of freedom. The factor of 2 in the denominator adjusts
for the fact that each d;is calculated from two values with error.
18
-------
Final Draft 01/19/07
2.3 PM2.s Bias Assessment
Applies to: PM2.s
40 CFR Part 58 Appendix A Reference:
• 4.3.2 Bias Estimate (PM25)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
75
26
27
?K
A
B
C D
E
F
G
H
PM2.5 Bias
'Site ID: {Enter Site ID}
Meas Val (Y)
5.05
12.35
1467
959
10 51
1244
378
1856
12.56
763
1456
1745
1234
Audit Val (X)
5.55
13
1435
768
11 3
1345
451
1699
1332
789
15.32
1676
1344
Pollutant type: PM2 :| Bias (?'.)
d (Eqn 1) d2 |d| |d|2
-9.009 81 162 9009 81 162
-5.000 25.000 5.000 25.000
2 230 4 973 2 230 4.973
24870 61850724.370 618.507
-6 991 48.876' 6.991 48 376
-7.509 56 389 7.509 56 389
-16186 261 995'16 136 261 995
9241 85391 9241 85391
-5706 32555 5706 32555
-2 662 7 084 2 662 7 084
4961 24.610 4.961 24.610
4117 16949 4.117 16949
-8.185 66986 8185 66.986
n
13 -1
n-1
12 1C
i
D
981
Sd
326
BiasUCL(%)(Eqn#13)
3 12
Bias LCL (%) (Eqn. #14)
-7 09
Return to Main Menu
J
K
Print Worksheet
30 000 -i
20 000 -
0 000 -
Percent Differences
A
/ \
7 \ A,
L
,^*\
-^ x
Figure 8. PM2.5 bias DASC worksheet example
The bias estimate is calculated using the Performance Evaluation Program (PEP) audits
described in CFR, section 4.1.3 of Part 58, Appendix A. The bias estimator is based on upper and
lower probability limits on the mean percent differences (Equation 1). The mean percent
difference, D, is calculated by Equation 12 below.
Equation 12
Confidence intervals can be constructed for these average bias estimates in Equation 12 of this
document using equations 13 and 14 below:
Lower 90% Confidence Interval = D -1
fi.95.tli'
Equation 13
Upper 90% Confidence Interval = D +10 95 df • -^=
V^
Equation 14
19
-------
Final Draft 01/19/07
Where, to.95,df'^ the 95th quantile of a t-distribution with degrees of freedom df=rij-l and Sd is an
estimate of the variability of the average bias and is calculated using Equation 15 below:
Equation 15
20
-------
Final Draft 01/19/07
2.4 PM 10-2.5 and PM2.s Absolute Bias Assessment
Applies to: PM10.2.s andPM2.s (for estimation purposes only)
40 CFR Part 58 Appendix A Reference:
• 4.1.3 Bias Estimate (PMj0.2.5)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2U
21
22
23
24
?S
26
27
28
29
3D
31
A I B
C
D I E
F
G
H
I
PM2.s Absolute Bias
'Site ID: {Enter Site ID} [Pollutant type: PM; = (Absolute Bias)
Meas Val (Y)
5.05
12.35
14.67
9.59
10.51
12.44
3.78
18.56
12.55
7.68
14.56
17.45
12.34
Audit Val (X)
13
14.35
7.68
11.3
13.45
4.51
16.99
13.32
7.89
15.32
16.76
13.44
d(Eqn1) 25th Percentile
-9.009 ' -7.509
-5.000 75th Percentile
2.230 ' 2.230
24.870
-6.991
-7.509
-15.186
9.241
-5.706
-2.562
-4.961
4.117
-8.185
d:
81.152
25.000
4.973
618.507
48.876
56.389
261.995
85.391
32.555
7.084
24.510
J
K
L
Bias(::}
[d|
9.009
5.000
2.230
24.870
6.991
7.509
15.186
9.241
]d|>
81.162
25.000
4.973
618.507
48.876
56.389
n
13
£|d|
106.66
i
n-1 zd
12 -25.751
adi'
1330.478
Sd'
1330.478
261.995
85.391
5.705 32.5S5
2.662 7.084
4.961 24.610
16.9491 4.1171 16.949
66.986I 8.185] 66.986
Bias (=-t) (Eqn 3)
11.25
Signed Bias (::l
+•'-11.25
"AB" (Eqn 41
8.205
"AS" (Eqn 5)
6.160
Both Signs Positive
FALSE
Both signs Negative
FALSE
Return to Main Menu Print Worksheet
M
20.0011 -
to. ooo •
\
Percent Differences
A
/ \
V \ A, ..A.
ir~-**' \ ' ), / >-J~ ~-*'' - »
"V
Figure 9. PM2.5 absolute bias DASC worksheet example (PM10_2.s is virtually the same)
The bias estimate is calculated using the Performance Evaluation Program (PEP) audits
described in CFR, section 4.1.3 of Part 58, Appendix A. The bias estimator is an upper bound on
the mean absolute value of the percent differences (Equation 1), as described in Equation 3 as
follows:
Equation 3
^n_\ •-=
V»
where n is the number of PEP audits being aggregated; to.9s,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom; the quantity AB is the mean of the absolute values of
the dj 's (calculated by Equation 1) and is expressed as Equation 4 as follows:
Equation 4
and the quantity AS is the standard deviation of the absolute value of the 4's (Equation 1) and is
calculated using Equation 5 as follows:
21
-------
Final Draft 01/19/07
Equation 5
In i n
"•ZKP-ZKI
AS=V 1=l
Since the bias statistic as calculated in Equations 3 and 6 of this document uses absolute values,
it does not have a sign direction (negative or positive bias) associated with it. A sign will be
designated by rank ordering the percent differences of the QC check samples from a given site
for a particular assessment interval. Calculate the 25th and 75th percentiles of the percent
differences for each site. The absolute bias upper bound should be flagged as positive if both
percentiles are positive and negative if both percentiles are negative. The absolute bias upper
bound would not be flagged if the 25th and 75th percentiles are of different signs (i.e. straddling
zero).
22
-------
Final Draft 01/19/07
2.5 One-Point Flow Rate Bias Estimate
Applies to: PM10, PM25, PM10-2.5
40 CFR Part 58 Appendix A References:
• 4.2.2 Bias Estimate Using One-Point Flow Rate Verifications (PMla
• 4.3.2 Bias Estimate (PM10-2 5)
• Assigning a sign (positive /negative) to the bias estimate.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
2£
26
27
28
29
30
31
T?
A
B | C
D | E
F
G
H
I
One-Point Flow Rate Bias Estimate
'Site ID: {Enter Site ID} [Pollutant type:
Meas Val IY1 Audit Val (X)
16.52 16.67
16.71 16.67
16.5 16.67
1721 16.67
15.67 16.67
15.92 16.67
16.4 16.67
17.2 16.67
1732 16.67
16.67 16.67
15.89 16.67
16.05 16.67
16.46 16.67
I
d|Eqn. 1) 25th Percentile
-0.900' -3.719
0.240 75th Percentile
-1.020' 0.240
3.239
-5.999
Jt.499
-1.620
3.179
3.899
0.000
-4.679
-3.719
-1.260
-------
Final Draft 01/19/07
and the quantity AS is the standard deviation of the absolute value of the dt's (Equation 4) and is
calculated using Equation 5 as follows:
Equation 5
AS =
Since the bias statistic as calculated in Equation 3 of this document uses absolute values, it does
not have a sign direction (negative or positive bias) associated with it. A sign will be designated
by rank ordering the percent differences of the QC check samples from a given site for a
particular assessment interval. Calculate the 25th and 75th percentiles of the percent differences
for each site. The absolute bias upper bound should be flagged as positive if both percentiles are
positive and negative if both percentiles are negative. The absolute bias upper bound would not
be flagged if the 25th and 75th percentiles are of different signs (i.e. straddling zero).
24
-------
Final Draft 01/19/07
2.6 Semi-Annual Flow Rate Audits
Applies to: PMW TSP, PM2S, PM10_2S
40 CFR Part 58 Appendix A References:
• 4.2.3 Assessment Semi-Annual Flow Rate Audits
• 4.2.4 Percent Differences
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
A | B
C
D
E
F
G
H
Semi-Annual Flow Rate Audits
PQAO: | Pollutant type:
Meas Val (Y)
16 52
16.71
165
17.21
1567
15.67
16.4
172
1732
1667
15.89
1605
16.46
Audit Val (X)
1667
16 67
1667
1667
1667
1667
1667
16.67
16 67
1667
1667
1667
1667
d (Eqn. 1) d!
-0900 0.810
0 240 0.058
-1 020 1.040
3239 10.493
-5.999 35 936
-5.999 35.986
-1.620" 2.623
3.179 10103
3.899 15204
0 000 0 000
-4 679 21 394
-3719 13833
-1 260
1.587
Bias (%)
|dl
0900
0240
1 020
3239
5.999
5.999
1 620
3 179
3.899
0000
4679
3719
1 260
|dl!
0 810
0 053
1 040
10493
35 936
35.986
2623
10 108
15204
0000
21 894
13833
1.587
n
13
n-1
12
S|d|
35.753
S|d|2
149621
Upper Probability Limit
293
Lower Probability Limit
-5 18
I
"AB" (Eqn 4)
-1 126
"AS" (Eqn 5)
2067
Return to Main Menu
F
J
K
rint Worksheet
L
/~ "N
Percent Differences
5 000
0 000 t-r + -i-.^'lV • • •'f • • "* •
\ •'/*" "\. ^>
Figure 11. Semi-annual flow rate audit DASC worksheet example
The flow rate audits are used to assess the results obtained from the one-point flow rate
verifications and to provide an estimate of flow rate acceptability. For each flow rate audit,
calculate the percent difference in volume using equation 1 of this document where meas is the
value indicated by the sampler's volume measurement and audit is the actual volume indicated
by the auditing flow meter.
Equation 1
meas-audU
audit
To quantify this annually at the site level and at the 3-year primary quality assurance
organization level, probability limits are calculated from the percent differences using equations
6 and 7 of this document where m is the mean described in equation 8 of this document and k is
the total number of one-point flow rate verifications for the year
Equation 6
Upper Probability Limit = m + 1.96 • S
25
-------
Final Draft 01/1 9/07
Equation 7
Lower Probability Limit = m - 1.96 • S
where, m is the mean (equation 8):
Equation 8
m
= —Yd,
where, k is the total number of one point QC checks for the interval being evaluated and Si is the
standard deviation of the percent differences (equation 9) as follows:
Equation 9
s = ,, i=i V,=i
k(k-l)
26
-------
2.7 Lead Bias Assessments
Final Draft 01/19/07
Applies to: Lead
40 CFR Part 58 Appendix A References:
• 4.4.2 Bias Estimate (Lead)
o 4.4.2.1 Flow Rate Audit ("volume bias") Calculations
o 4.4.2.2 Lead Strip ("mass bias") Calculations
o 4.4.2.3 Final Bias Calculations
1
2
3
4
5
6
7
3
9
10
11
12
13
14
15
16
17
18
13
20
22
23
24
25
26
27
23
30
31
3?
33
35
A
B
C
D
E
F
G
H
1
J
K
L
M
Pb Bias
rSite ID: (Enter Site ID} \ Pollutant tipe: Pb|
Flow Audit Values
Meas Val (Y)
16.52
16.71
16.5
17.21
Audit Val (X) d (Eqn. 1)
16.67 -0.300
16.67 0.240
16.67 -1.020
16.67 3.239
d1
0.810
0.058
1.040
10.493
0.900
0.240
1.020
3.239
Ml'
0.810
0.058
1.040
10.493
Pb Strip Audit Values
Meas Val (Y) Audit Val (X) d [Eqn. 1)
5.05 5.55 -3.003
12-35 13 -5.000
14.67 14.35 2.230
9.59 7.68 24.870
10.51 11.3 -6.991
12.44 13.45 -7.50S
3.73 4.51 -16.186
13.56 16.39 3.241
12-56 13.32 -5.706
7.68 7.39 -2.662
14.56 15.32 -4.961
17.45 16.76 4.117
d1 |d| Idl1
31.162 3.003 81.162
25.000 5.000 25.000
4-373 2.230 +.373
618.507 24.370 616.507
48.876 6.991 48.876
56.388 7.503 56.389
261.935 16.186 261.335
35.331 3.241 35.391
32-555 5.706 32.555
7.034 2.662 7.034
24.810 4.961 24.810
16.949 4.117 16.949
N 0 P Q
Flc-v Avail f'vc-iumf Aia^ '}
n
4
S, S,, Zldl -AB-(Eqn4]
1.383 4.947 5.399 1.350
n-l Zd I Zd1 I Zd|J 'AS- (Eqn 5)
3 1.560 12.401 12.401 1.306
R
Bias &) (E
2.89
f*A ffrap Aadff fntass A/ss '}
n Sj S« ZJdl "AB- (Eqn 4)
12
10.603 175.876 33.431 3.207
n-1 Zd Zd1 Zd|z "AS" (Eqn 5)
11 -17.566 1263.491 1263.491 6.433
25th Percenlile
(bias) {'/.} (Equation 18) T -7.121
14 .86 75th Percentile
Bias {Y.} (E
11.54
Both Signs Fc
FALSE
Both Signs Ne
Signed Bias: W-14.66 2.702| FALSE
RatLiT to h' an Menu Print Worksheet
Percent Differences
0.000 -
-10.000 -
-20000 -
V
A
? \
_,_ >'^* '", A-. ^
---w
J
Figure 12 - Lead Bias DASC worksheet example
In order to estimate bias, the information from the flow rate audits and the Pb strip audits needs
to be combined as described below. To be consistent with the formulas for the gases, the
recommended procedures are to work with relative errors of the lead measurements. The relative
error in the concentration is related to the relative error in the volume and the relative error in the
mass measurements using Equation 16 of this appendix:
rel. error =
Equation 16
_ (measured concentration - audit concentration)
audit concentration
1
1 + rel. error
(rel. mass error - rel. volume error)
As with the gases, an upper bound for the absolute bias is desired. Using Equation 16 above, the
absolute value of the relative (concentration) error is bounded by equation 17 of this appendix:
27
-------
Final Draft 01/19/07
rel.error
Equation 17
I relative mass error I + relative volume error
1 - relative volume error
The quality indicator data collected are then used to bound each part of Equation 17 separately.
"Flow Audit" calculations.
For each flow rate audit, calculate the percent difference, d^m volume by Equation 1 of this
appendix where meas is the value indicated by the sampler's volume measurement and audit is
the actual volume indicated by the auditing flow meter.
Equation 1
audit
The absolute "volume bias" upper bound is then calculated using Equation 3 of this appendix
where n is the number of flow rate audits being aggregated; to.95,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom; the quantity AB is the mean of the absolute values of
the <^'s and is calculated using Equation 4, and the quantity AS in Equation 3 of this appendix is
the standard deviation of the absolute values of the ^'s and is calculated using Equation 5 of this
appendix.
Equation 3
\bias\=AB+t095^_l •-=
v»
where n is the number of flow audits being aggregated; to.95,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom; the quantity AB is the mean of the absolute values of
the dj 's (calculated by Equation 4) and is expressed as Equation 4 as follows:
Equation 4
and the quantity AS is the standard deviation of the absolute value of the 4's (Equation 4) and is
calculated using Equation 5 as follows:
Equation 5
AS=
-------
Final Draft 01/19/07
"Pb Strip Audif'calculations.
Similarly for each lead strip audit, calculate the percent difference,^ in mass by Equation 1
where meas is the value indicated by the mass measurement and audit is the actual lead mass on
the audit strip.
Equation 1
meas- audit
audit
The absolute "mass bias" upper bound is then calculated using Equation 3 of this appendix
where n is the number of lead strip audits being aggregated; t0.9s,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom;
Equation 3
where n is the number of flow audits being aggregated; t0.95,n-i is the 95th quantile of a t-
distribution with n-1 degrees of freedom; the quantity AB is the mean of the absolute values of
the df 's (calculated by Equation 4) and is expressed as Equation 4 as follows:
Equation 4
and the quantity AS is the standard deviation of the absolute value of the df's (Equation 4) and is
calculated using Equation 5 as follows:
Equation 5
Final |Bias| Calculation.
Finally, the absolute bias upper bound is given by combining the absolute bias estimates of the
flow rate and Pb strips using Equation 18 of this appendix:
29
-------
Final Draft 01/19/07
Equation 18
Imassbias +1 vol. bias
bias = J ;—! 100
100-1 vol. bias
where mass bias is the bias calculated for the Pb strips, and vol is the bias calculated for the flow
rate audits. The numerator and denominator have been multiplied by 100 for expression in
percent.
Since the bias statistic as calculated in Equation 3 of this document uses absolute values, it does
not have a sign direction (negative or positive bias) associated with it. A sign will be designated
by rank ordering the percent differences of the QC check samples from a given site for a
particular assessment interval. Calculate the 25th and 75th percentiles of the percent differences
for each site. The absolute bias upper bound should be flagged as positive if both percentiles are
positive and negative if both percentiles are negative. The absolute bias upper bound would not
be flagged if the 25th and 75th percentiles are of different signs (i.e. straddling zero).
Time Period for Audits.
The statistics in this section assume that the mass and flow rate audits represent the same time
period. Since the two types of audits are not performed at the same time, the audits need to be
grouped by common time periods. Consequently, the absolute bias estimates should be done on
annual and 3-year levels. The flow rate audits are site-specific, so the absolute bias upper bound
estimate can be done and treated as a site-level statistic.
30
-------
United States
Environmental Protection
Agency
Office of Air Quality Planning and Standards
Air Quality Strategies and Standards Division
Research Triangle Park, NC
Publication No. EPA-454/B-07-001
January, 2007
Postal information in this section where appropriate.
------- |