GUIDELINES FOR ASSESSING AND REPORTING
DATA QUALITY FOR ENVIRONMENTAL MEASUREMENTS
Environmental Monitoring and Support Laboratory - Cincinnati
Environmental Monitoring Systems Laboratory - Research Triangle Park
Environmental Monitoring Systems Laboratory - Las Vegas
Office of Research and Development
U.S. Environmental Protection Agency
January 14, 1983
-------
CONTENTS
Section Page
- • Ji—
1.0 INTRODUCTION 1
2.0 PURPOSE, SCOPE AND APPLICATION 3
2.1 Purpose 3
2.2 Scope and Application 3
3.0 ASSESSMENT OF PRECISION 5
3.1 Definitions 5
3.2 Measurement of Precision 6
3.3 Reporting Precision 11
3.4 Continual Precision Assessments Using Duplicate Measurements ... 12
4.0 ASSESSMENT OF ACCURACY 14
4.1 Definitions 14
4.2 Measurement of Accuracy 15
4.3 Reporting Accuracy 19
4.4 Continual Accuracy Assessment 19
5.0 METHOD DETECTION LIMIT (MDL) 22
5.1 Definition 22
5.2 Measurement of MDL 22
5.3 Reporting MDL and Values Near MDL 23
6.0 COMPLETENESS 25
6.1 Definition .... 25
6.2 Calculation of Completeness 25
6.3 Reporting of Completeness 26
7.0 INCORPORATION OF ASSESSMENTS INTO ENVIRONMENTAL DATA BASES 27
7.1 General Discussion 27
7.2 Critical Elements and Formats 27
8.0 SOURCES OF ADDITIONAL INFORMATION 30
8.1 Study Planning 30
8.2 Sampling 30
8.3 Assessment of Precision 30
8.4 Assessment of Accuracy 31
8.5 Use of Control Charts 32
8.6 Method Detection Limits 34
APPENDIX A - EXAMPLE CALCULATIONS 35
-------
- ACKNOWLEDGEMENTS
This document has been prepared by James E. Lpngbottom and other staff
members of the Environmental Monitoring and Support Laboratory in
Cincinnati, Ohio, in cooperation with the staffs of the Environmental
Monitoring Systems Laboratory in Research Triangle Park, North Carolina and
the Environmental Monitoring Systems Laboratory in Las Vegas, Nevada. The
extensive technical contributions of Raymond C. Rhodes of EMSL- Research
Triangle Park and the assistance of the Agency's Quality Assurance Officers
in reviewing the document and providing comments during its generation are
gratefully acknowledged.
-------
1.0 INTRODUCTION
U.S. Environmental Protection Agency's (USEPA) policy regarding quality
assurance requires participation by all Agency regional offices, program
offices, and laboratories, as well as the States, in a centrally managed
program, as stated in the Administrator's memorandum of May 30, 1979. This
requirement applies to all environmental monitoring and measurement efforts
mandated or supported by the Agency through regulations, grants, contracts,
or other formalized means not currently covered by regulation. The respon-
sibility for developing, coordinating, and directing the implementation of
this program has been delegated to the Office of Research and Development
(ORD), which has established the Quality Assurance Management Staff (QAMS)
for this purpose.
The importance of the mandatory Quality Assurance program to the Agency was
reaffirmed by the Administrator on November 2, 1981 when she stated:
"One of the major concerns of this administration and myself is that we
support all of our actions and decisions with statistically represen-
tative and scientifically valid measurement of environmental quality.
To meet this objective, it is essential that each of you continue to
support and implement the Agency's mandatory Quality Assurance program
which is being implemented by the Office of Research and Development.
It is especially essential that you assure that the appropriate data
quality requirements are included in all your extramural and intramural
environmental monitoring activities."
-------
Each Agency office or laboratory generating data has the minimum responsi-
bility to implement procedures which assure that the quality of its data is
known and reported. To ensure that this responsibility is met uniformly
across the Agency, each office or laboratory must implement the guidelines
contained herein for each environmentally related measurement activity
within its purview.
-------
2.0 PURPOSE, SCOPE AND APPLICATION
2.1 Purpose
This document provides guidelines for the assessment and reporting
of data quality for any environmentally related measurements and
for the incorporation of such assessments into major environmental
data bases. It is to be used with QAMS-005/80 (Section 8.1,
Reference 1) in the development of quality assurance project plans
for all USEPA environmentally related measurements.
2.2 Scope and Application
This document provides procedures to calculate and report statis-
tical assessments of data quality. Such assessments are valuable
to environmental data users for defining the overall quality of a
set of environmental data. They are also useful in judging the
suitability of a measurement system for an intended purpose. Data
quality assessments of precision, accuracy, and the MDL are based
on special measurements (e.g., analyses of replicate samples,
analyses of spiked samples, and blank determinations) made during
the period of operation of the measurement system being used to
routinely generate the measurement data. Application of the data
quality assessments to the complete measurement data set relies on
the assumption that the special measurements are made under
typical, "in control" conditions and, therefore, are representa-
-------
tive of the complete data set. This assumption is based upon large
sample population theory, and its validity will generally be
proportional to the number of data quality measurements performed.
Procedures are provided for both end-of-project data quality
assessment of terminated data collection projects and for continual
data assessment of ongoing data collection programs. Example
calculations for determining data quality assessments are given in
Appendix A.
These guidelines are designed to be directly applicable to discrete,
normally-distributed chemical measurements on most environmental
matrices. Some adaptation will be required to apply the guidelines
to continuous measurement systems, certain biological and chemical
measurement systems, and certain solid or extremely non-homogenous
matrix types. Sources of additional information are provided in
Section 8 that can be used in the adaptation of these guidelines as
necessary.
-------
3.0 ASSESSMENT OF PRECISION
3.1 Definitions
Precision - a measure of agreement among individual measurements
of the same property, under prescribed similar conditions.
Precision is expressed in terms of concentration units (range of
duplicates or standard deviation) or as it is related to the mean
concentration (relative range or relative standard deviation).
Replicate Measurements - individual test results for two or more
samples that are considered representative subsamples of the same
environment. The samples are processed, normally, through the
entire measurement system. Replicate measurements are used to
assess the precision of the environmental data and ideally should
include the variability caused by sampling, preservation and
analysis.
Basic Precision Statistics
Range of Duplicates (R) - a basic statistic indicating the
agreement between duplicate measurements of the same
property. The range of duplicates is used to express
precision in place of the standard deviation when only two
replicate measurements are performed. The average range (R")
of a large number of duplicate measurements taken from a
sample population can be used to estimate the standard
deviation (s) of the population as follows:
R = 1.128(s). Eq. 1
(See Section 8.5, Reference 3)
5
-------
Standard Deviation (s) - a basic statistic indicating the
dispersion of two or more replicate measurements about the
mean value.
Operational Precision Statistics:
Relative Range of Duplicates (RR) - an operational statistic
indicating the dispersion of duplicate measurements as a
percentage of the mean value. When derived from duplicate
measurements from a representative portion of a sample lot of
similar character, the average relative range (RR") can be used
to estimate the range of duplicate measurements at any
individual concentration within that sample lot.
Relative Standard Deviation (RSD) - an operational statistic,
also called the coefficient of variation, indicating the
dispersion of a set of replicate measurements as a percentage
of the mean value. When derived using replicate measurements
from a representative portion of a sample lot of similar
character, the average relative standard deviation (RSO) can
be used to estimate the standard deviation at any individual
concentration within that sample lot.
3.2 Measurement of Precision
Guidelines -- The precision assessment should represent the
-------
variability of the environmental measurement data. Therefore, field
replicate samples are analyzed, where possible, to incorporate the
variability of sampling, sample handling, preservation and storage
into the data quality assessment along with the variability of the
actual measurement process. If the nature of the matrix type,
sampling procedure or measurement system prevents the assessment of
the entire measurement system, the replicate measurements used to
assess precision should be selected to incorporate as much of the
measurement system as possible.
For methods used to analyze discrete samples, precision assessments
are based upon the results of replicate measurements made at concen-
tration levels representative of the entire range observed in
routine samples. In general, the precision assessment should
represent measurements performed by the same method and by the same
laboratory. The frequency of replicate measurements will depend
upon the data quality needs of the program, the precision of the
measurement system, the size of the sample lot and other
considerations. For large, sample lots, a fixed frequency for
duplicate measurements (such as one sample in ten or twenty) is
recommended. For small sample lots the frequency of replication
should be much higher and may require the analyses of three or more
replicates of some samples to insure that sufficient data is avail-
able to assess precision. Alternately, multiple sample lots of a
common matrix analyzed by the same measurement system can be com-
bined as discussed under continual precision assessments (Section
3.4).
7
-------
If the environmental measurements normally produce a high
percentage of results below the MDL (Section 5), samples for
replicate measurement should be selected from those containing
measurable levels of analyte. Where this is impractical, such as
with complex multi -analyte methods, sample replicates may be
spiked at an appropriate concentration level to ensure that
sufficient data will be available to assess precision.
Calculation of Basic Statistics - The basic precision statistics,
range of duplicates and standard deviation, are used with the
average concentration (T. ) to develop the operational
statistics from a portion of the population. For each set of
replicate measurements, calculate the average concentration of (n)
number of measurements ^s follows:
n Eq. 2
If duplicate measurements are used to assess precision, calculate
the range of each duplicate pair (R. ):
Ri = X] - X2 Eq. 3a
where X^ represents the larger and X2 the smaller of the two
random observations.
NOTE: For certain applications, X-j and X2 are assigned to
specific observations, so that signed differences may be
computed. For example, individual measurements for co-located
samplers are assigned so that possible significant differences
between the samplers may be detected. (See 40 Code of Federal
Regulation, Part 58, Appendix A.)
-------
If the standard deviation or relative standard deviation is used
to assess precision, calculate the standard deviation (s^) at
each concentration level from (n) replicates as follows:
n
z
1=1
*H
n-l
f n \ 2
E X. *
1=1 1
n
Calculation of Operational Statistic - The operational statistic
is an assessment of the precision of a measurement system for a
project or time period which may be used to estimate a basic
precision statistic associated with any individual concentration
contained in that sample lot. In certain cases, the operational
statistic may provide the basis for a continual assessment of
precision in subsequent small sample lots. The operational
statistic is developed from the basic statistics gathered
throughout the project or time period represented. .Because the
precision of environmental measurement systems is often a function
of concentration (e.g., as concentration decreases, relative
standard deviation increases), evaluate this relationship before
selecting the most appropriate form of the operational statistic.
Using the basic statistics gathered in the project or time period,
calculate the relative range (RR) of duplicates or the relative
standard deviation (RSD) for larger numbers of replicates at each
concentration level (i) as:
-------
RR,. = -z- (100) Eq. 4a
RSD. *-^- (100) £q. 4b
Xi
Inspect the individual results or perform appropriate statistical
tests for a dependency on concentration. If necessary, rank, group
or plot the entries by concentration to discern any such
relationship. See Appendix A (Example 3) for an example data set.
If a relationship between RR^ , or RSD. and concentration level
is not clearly evident, calculate the average relative range of
duplicates (RR") from (k) sets of duplicate measurements as:
— 1 k
RR = -r- s RR, j- -
K j _ i i • Eq. 5a
Where three or more replicates of each sample were used to assess
precision, calculate the "average" or pooled relative standard
deviation (RSO) as:
RSD = * n2RSD|
• W lX
where n is the number of replicates in a set and k is the number of
sets.
If a relationship between RR. or RSD. and concentration is
clearly evident, it is necessary to use a more complex approach,
such as a linear regression equation, to describe this
10
-------
relationship. A linear regression of the basic statistics (R.. or
s.j) versus concentration results in two coefficients, a slope and
an intercept, which are used to assess the precision of the data
set. An example data set appropriate for a linear regression
analysis is illustrated in Appendix A, Example 3.
Where no relationship between RR. or RSD^ and concentration has
been established, the expected range of duplicates (R1) or expected
standard deviation (s1) of any concentration (X) found in the
sample lot is estimated as:
R1 = (RR/100)(X) Eq. 6a
S1 = (RSD/100)(X) Eq. 6b
If a concentration dependency has been established, this relation-
ship is used to estimate the range of duplicates or standard
deviation. For example, for a linear regression equation, these
basic statistics are estimated as either:
R1 = a X + b; or Eq. 7a
s1 * a X + b Eq. 7b
where a is the slope and b is the intercept of the regression
equation.
3.3 Reporting Precision
Each environmental measurement must be reported with an assessment
of precision. Because each data user must determine the confidence
limits required for his application, the data reporter must provide
a range of duplicates (R1) or standard deviation (s1) for each
measurement.
11
-------
The data user should be provided with a narrative statement along
with tabulated values for R1 or s1. The statement might be
presented in either of the following forms:
"The estimated range of duplicates (R1) is provided for each
concentration found. This value can be used to establish a
probability limit for the agreement between duplicate
measurements at each concentration. For example, the upper
95% probability limit for the range between duplicates is
2.46 R1. If the estimated range is 5 yg/L, 95% of the time
duplicate measurements should agree within 2.46 x 5 or 12.3
yg/L."
"The estimated standard deviation (s1) is provided for each
concentration found. This value can be used to estimate a
probability interval for the sample concentration (X). For
example, the 95% probability interval is represented by X ±
1.96 s1. If the concentration value is 10 ug/L and the
standard deviation is 2 yg/L, then the 95% probability
interval is 6.08 to 13.92 yg/L."
3.4 Continual Precision Assessments Using Duplicate Measurements
For laboratories in which small sample lots are routinely analyzed
and data are reported on a frequent basis, the basic precision
statistics from multiple small lots of a given sample matrix may be
combined to develop a single assessment for the combined sample
set. This assessment can also be extended to include subsequent
small sample lots, unless test results for these new lots indicate
that method precision is significantly different. This combining
12
-------
of data permits the laboratory to provide a precision assessment
derived from a statistically significant data base rather than from
limited data produced in a small study. It also can provide the
basis for a convenient quality control check for the measurement
system. The procedure is based upon the availability of a
precision assessment (normally developed from prior performance of
the system), the use of control limits, and routine duplicate
measurements.
Historical data must first be combined as necessary to develop an
assessment of precision that defines the expected range of
duplicates (R1) as a function of concentration in the form of
Equation 6a or 7a. For each duplicate set in the new sample lot,
the range observed (R..) is compared to an upper control limit for
the expected range, R', calculated for the observed average sample
concentration (x!j). If R^ £3.27 R', the established precision
assessment can be applied to the individual members of the new
sample lot. If R. > 3.27 R1, either the established precision
assessment is not applicable to the new data set, or the
measurement system is out of control. A typical calculation is
provided in Appendix A, Example 4. For further information on the
use of control limits, including the rationale for the 3.27
constant, see Section 8.5, Reference 1.
At least annually, and preferably after the accumulation of 30-50
new sets of duplicates, new assessments for precision must be
calculated to reflect the current precision of the measurement
system. This may be done by either expansion or replacement of the
historical data base with the most current data.
13
-------
4.0 ASSESSMENT OF ACCURACY
4.1 Definitions
Accuracy - a measure of the closeness of an individual measurement
or an average of a number of measurements to the true value.
Accuracy includes both precision and recovery and can be expressed
as a percent recovery or percent bias interval.
Reference Material - a material of known or established concen-
tration used to assess the accuracy of a measurement system.
Depending on requirements, reference materials may be used as
prepared or diluted with inert matrix as a blind environmental
sample.
Spiking Material - a material of known or established concentration
used to spike environmental samples to assess the accuracy of
environmental measurements.
Percent Recovery (P) - a basic accuracy statistic indicating the
observed increase in measured value for a sample that has been
spiked as a percentage of the increase expected, resulting from the
addition of a known amount of analyte. For the analysis of refer-
ence materials, the definition reduces to the measured value as a
percentage of the true value. Accuracy assessments for automated
air methods reported to the Storage and Retrieval of Aerometric
Data (SAROAD) data system are expressed as percent bias. Percent
bias is equal to (percent recovery - 100).
14
-------
Percent Recovery Interval - an operational statistic to indicate
the variability of a set of accuracy measurements. The percent
recovery interval is expressed as a 95% probability interval for
the individual percent recovery values.
4.2 Measurement of Accuracy
Guidelines - Accuracy assessments for environmental measurements
should be made using spiking materials or reference materials as
independent as possible from similar materials used for calibration
of the measurement system. The entire assessment for accuracy
should be as independent as possible from the routine calibration
process.
Spiking materials or reference materials should ideally be
introduced in the field so that the accuracy assessment includes
any losses caused by sample handling and storage. If the matrix
type (e.g. solids) or the measurement system prevents such
practices, accuracy assessments must be made for as large a portion
of the measurement system as possible. For example, for manual
methods for air (e.g., SCk and NO?), introducing analytes of
known concentration is not practical at the field site. Therefore,
the accuracy of the flow measurement and the accuracy of the
analytical portion of the method are assessed separately.
Where possible, accuracy assessments are based upon spiked samples
rather than the analysis of reference materials so that the effect
15
-------
of the matrix on recovery 1s incorporated into the assessment. For
methods used to analyze discrete samples, a representative portion
of the sample lot is selected for spiking.
For example, the equivalent of one spiked sample in ten could be
used to assess accuracy. The spiking frequency will depend upon
the data quality needs of the program, the accuracy and precision
of the measurement system, the size of the sample lot and other
considerations. To properly assess the accuracy for small sample
lots a relatively high percentage of samples should be spiked.
However, where the method performance for multiple sample lots of
similar matrix type is expected to be equivalent, small sample lots
may be combined to lower the necessary spiking frequency.
Spikes should be added at different concentration levels to cover
the range of expected sample concentrations. For some measurement
systems (e.g., continuous field measurements for ambient air), the
spiking of samples is not practical, and assessments must be made
using audit gases or other reference materials.
It will rarely be possible to establish a statistically significant
relationship between average recovery and concentration using the
spiking program described above. The variability of recovery as a
function of concentration is more frequently discernible, however,
and may need to be addressed using the same approaches as are used
to assess precision as a function of concentration. Accuracy
assessments reported to SAROAO for continuous air analyzers, for
16
-------
example, are calculated from measurements performed within four
different concentration ranges. The confidence intervals for
accuracy are determined separately at each concentration level.
Except for very large studies, however, the relationship of the
confidence interval for recovery to concentration is difficult to
capture and therefore not discussed in detail below.
For certain multianalyte methods, such as EPA Method 608 for organo-
chlorine pesticides and PCBs in water, accuracy assessments are
compounded by mutual interference between certain analytes that
prevent all of the analytes being spiked into a single sample. For
such methods, lower spiking frequencies can be employed for analytes
that are seldom, or never, found. The use of spiked surrogate
compounds for multianalyte GC/MS procedures is considered a quality
control practice and not an assessment of accuracy. It is used, for
example, to evaluate the applicability of methodology and,
indirectly, data quality assessments to individual members of a
sample lot. Such practices do not preclude the need to assess
accuracy by spiking with the analytes being measured or reported.
Calculation of Accuracy Statistics - A portion of the samples in the
sample lot, or the equivalent, is spiked at multiple concentration
levels to determine individual measurements of percent recovery.
These recoveries are used to calculate an operational statistic for
the entire sample lot. The operational statistic is used to
estimate the percent recovery for each individual measurement in the
lot. For each sample spike (i), calculate the percent recovery
(Pi), where:
17
-------
P( - A1 • B1 (100) Eq. 3
"
where: A. = the analytical result from the spiked sample,
B. - the background level determined by a separate
analysis of the unspiked sample,
T. = the known true value of the spike.
Dilution of the sample by the addition of the spike should not
exceed 10* and must be considered in the calculation of recovery.
If reference materials are analyzed in lieu of spiked samples to
assess accuracy, percent recovery is calculated using Equation 8
with B. equal to zero.
Upon completion of the project or time period, the accuracy
assessment for the data set of environmental measurements is
calculated from the individual percent recoveries (P.) observed
through the project period. Unless sufficient data is available to
establish a relationship between the variability of recovery to
concentration, all recovery measurements are combined
for the accuracy assessment. Calculate the average percent
recovery, "P, and the standard deviation of the percent recovery
(s ) as in Equations 2 and 3b. The meaning of the value for s
H H
is considerably different from the precision assessment. For spiked
samples it includes the variability of the background measurement
plus the variability of the final measurement. In addition, the
individual recoveries are usually gathered over an extended time
period, rather than over short time intervals normally used for
18
-------
replicate measurements and therefore may reflect the presence of
many other variables.
4.3 Reporting Accuracy
Each environmental measurement must be reported with an assessment
of accuracy. Accuracy should be expressed to the data user as a
percent recovery interval from F - 2s_ to P" + 2s . Where
reference materials are used as a matrix-free check on laboratory
performance as a supplement to sample spiking, only the results of
the sample spikes should be submitted to an environmental data base.
The data user should be provided with a narrative statement along
with tabulated percent recovery intervals. The statement might
read:
"Accuracy is expressed as a 95% probability interval around
the mean percent recovery. A percent recovery interval of
91-107, for example, means that 95% of the time the recovery
of the measured material was found to be between 91 and 107%,
with a mean recovery of 99£."
4.4 Continual Accuracy Assessment
As with precision assessments, laboratories in which small sample
lots are routinely analyzed and data are reported on a frequent
19
-------
basis may combine the basic accuracy statistics of multiple small
sample lots of a given matrix into a single accuracy assessment for
the combined sample set. This assessment can also be extended to
include subsequent small sample lots, unless test results for these
new lots indicate that method accuracy is significantly different.
Combining data in this manner permits the laboratory to provide an
accuracy assessment derived from a statistically significant data
base rather than from limited data produced in a small study. It
also can provide the basis for a convenient quality control check
for the laboratory.
Historical data must first be combined as necessary to develop an
assessment of accuracy which includes the determination of average
percent recovery (F) and the standard deviation of the percent
recovery (sp). They are used to develop control limits for
subsequent measurements as F ± 3 s . Each recovery measurement,
P.., in the new lot must be compared with the control limits. If
each value for P^ falls within the control limits, the accuracy
assessment can be applied to all individual measurements of the new
sample lot. If P.. falls outside the control limits, either the
historical precision assessment is not applicable to the new data
set or the laboratory operation is out of control. A typical
calculation is provided in Appendix A, Example 6.
At least annually, and preferably after no more than 30 to 50 new
recovery measurements have been taken, the control limits must be
20
-------
recalculated to reflect current accuracy of the measurement
system. This may be done by either expansion or replacement of the
historical data base to include the most current data.
-------
5.0 METHOD DETECTION LIMIT (MDL)
5.1 Definition
Method Detection Limit - the minimum concentration of a substance
that can be measured and reported with 99% confidence that the true
value, corresponding to a single measurement, is above zero.
5.2 Measurement of MDL
Each laboratory should establish and periodically reevaluate its
own MDL for each sample matrix type and for each environmental
measurement method. The MDL is determined for discrete measurement
systems by the analyses of seven or more replicates at or near zero
concentration. As with precision and accuracy, the assessment of
MDL should be based upon the performance of the entire measurement
system, including the measurement of a response. For measurement
systems where background zero is nulled and cannot be expressed in
quantitative terms, the variability of zero is estimated from
replicate measurements at.a concentration near but above zero. The
standard deviation of the response (s ), in concentration units,
is determined as in Equation 2. MOL is calculated as:
MDL = sm(t
-------
t gg = "Student's t value" appropriate for a one-tailed test
at the 99% confidence level and a standard deviation estimate
with n-1 degrees of freedom.
For example, if the MDL is determined using seven replicates of an
appropriate sample, t gg = 3.14 (six degrees of freedom). If the
determination yielded a standard deviation of 0.15 concentration
units, the MDL is calculated (Equation 9) to be (3.14)(0.15) * 0.47
concentration units.
5.3 Reporting MDL and Values Below Detection
Environmental measurements reported for inclusion in an environ-
mental data base must be accompanied with an assessment of MDL.
Any individual measurement taken at a concentration of MDL or less
and reported directly to a data user must be flagged and reported
with the MDL. For example, the value for a measurement of 5 from a
measurement system with an MDL of 7 must be identified in the
report as a measurement below MDL. Reporting conventions for
environmental data bases may not require a flag on individual
entries at MDL or below, if MDL can be stored separately.
Since each environmental measurement will be associated with an
assessment of accuracy, precision and detection limit, qualitative
measurements ("presumptive," "present but less than this concen-
tration," "estimated concentration") should be unnecessary for
measurement systems that produce a number continuum.
-------
It is the responsibility of the data user, not the data producer,
to censor data near zero. The data producer has the responsibility
of establishing a reference point (MDL) for the data user to employ
according to his needs.
24
-------
6.0 COMPLETENESS
6.1 Definition
Completeness - a measure of the valid data obtained from a measure-
ment system expressed as a percentage of the amount of data that
should have been collected. Completeness 1s of particular
importance to multiyear intensive monitoring programs.
6.2 Calculation of Completeness
At the end of a project or specified time period, calculate
completeness as:
r««,«!«*-««<.«. f = Number of Valid Data Acquired „ /inm c« in
Completeness, % = total Number of Values Planned x (100) Eq* 10
For example, most manual methods for ambient air monitoring programs
require sampling every six days, or 15 days per calendar quarter. If
valid data for only 13 days are acquired, the completeness is 13/15 =
87%.
For continuous measurement methods, results are usually reported is
hourly averages. Most calendar quarters contain 91 days, or 2,134
hours. Thus, 2,184 could be used as the denominator. If for
example, a total of 2,000 valid hourly values are acquired for a
particular continuous analyzer during a 91-day quarter, the
percentage completeness would be 92%.
25
-------
6.3 Reporting Completeness
Report completeness as whole percentage numbers from 0 to 100, and
specify the base for the percentage completeness. For the continuous
measurement method example used above, 2,184 hours was used as the
denominator. However, during time used for calibration, quality
control checks, and preventative maintenance, no monitoring data are
required. The denominator could be adjusted for these periods.
26
-------
7.0 INCORPORATION OF ASSESSMENTS INTO ENVIRONMENTAL DATA BASES
7.1 General Discussion
The ultimate user of environmental measurements must have access to
data quality assessments. All major environmental data bases must
be capable of accepting, storing and retrieving these assessments
with each measurement. For most data bases, these assessments will
be compressed into a directory that would be automatically accessed
during data retrieval. While it is beyond the scope of this
document to discuss the design of such a system, the elements that
are considered essential to assessing data quality and the formats
used in this document are summarized below.
The formats are applicable for the incorporation of data quality
assessments into reports of environmentally related measurements
not intended for data bases, including all Agency-sponsored
research activities.
7.2 Critical Elements and Formats
Originator - The laboratory acquiring measurements must be
identified by name and complete address so that data users can
obtain detailed information not available in data summaries, or
computerized data bases.
Sample Matrix - The sample matrix type must be characterized in
sufficient detail as required by the projected data users. The
assessments of data quality, however, can incorporate more than one
matrix type into each assessment if the measurement system
27
-------
critical for the user to know whether a hardness measurement was
performed on surface water or a finished water sample, but a single
data quality assessment could be developed in the laboratory that is
comprehensive for hardness in both matrices.
Analytical Method - The analytical method used must be identified irr
sufficient detail to be completely understood by the scientific
community. An alpha numeric code can be developed to identify the
method. For example, the designation ASTM 03223-79, which identifies
a specific method for mercury in water, should be used instead of a
broad method code that would include all methods that determine
mercury by cold vapor atomic absorption.
Precision - Each environmental measurement must be reported with an
assessment of the precision of the measurement. To incorporate a
precision assessment into a directory system, data bases should allow
entry of as many as three terms. While single-term assessments, such
as RSO can be accommodated with a single entry, more complex
relationships between concentration and precision require entry of
two numerical coefficients (for example, slope and intercept for a
linear regression equation) and a third entry to reference the
mathematical function to be used to produce estimates of precision
for any reported measurement concentration.
Accuracy - Each environmental measurement must be reported with an
assessment of the accuracy of the measurement. As a minimum, data
bases should allow entry of an average percent recovery and a
23
-------
standard deviation of the percent recovery. Since both terms may be
expressed as a function of concentration, as many as three entries
may be required to report each term.
Method Detection Limit (MDL) - Each environmental measurement must be
reported with an MDL. MDLs can be entered into major environmental
data bases in the same format as used for the analytical result (two
significant figures for floating point systems). In addition, any
environmental measurement reported at or below the MDL must be so
identified to any future user of the data, and the user should have
the option to censor such data.
Completeness - Depending upon the program, environmental measurements
may be reported with an assessment of completeness. Data bases
should allow entry of the percent completeness as whole numbers from
0 to 100.
29
-------
8.0 SOURCES OF ADDITIONAL INFORMATION
8.1 Study Planning
1. "Interim Guidelines and Specifications for Preparing Quality
Assurance Project Plans,"QAMS-005/80, U.S. EPA, Office of
Research and Development, Washington, D.C. 20560, December, 1980.
2. Natrella, M.G., Experimental Statistics. NBS Handbook 91, U.S.
Department of Commerce, National Bureau of Standards, 1966.
3. Davies, O.L., The Design and Analysis of Industrial Experiments.
2nd edition, Hafner Publishing Co., New York, 1956.
4. Cox, D.R., Planning of Experiments. Wiley, New york, 1958.
5. Box, G.E.P., W.G. Hunter and J.S. Hunter, Statistics for
Experimenters, Wiley, New York, 1978.
6. Youden, W.J., "Statistical Aspects of Analytical Determina-
tions," Journal of Quality Technology. 4(1), 1972, pp. 45-49.
7. Elder, R.S., "Choosing Cost-Effective QA/QC Programs for
Chemical Analysis," EPA Contract No. 68-03-2995, Radian
Corporation, Austin, Texas, 1981 (draft).
8.2 Sampling
1. Environmental Monitoring and Support Laboratory, Handbook for
Sampling and Sample Preservation of Water and Wastewater,
EPA-600/4/82-029, U.S. EPA, Office of Research and Development,
Cincinnati, 1982.
2. Brumbaugh, M.A., "Principles of Sampling in the Chemical
Field," Industrial Quality Control, January 1954, pp. 6-14. .
3. Kratochvil, B. and J.K. Taylor, "Sampling for Chemical
Analysis," Analytical Chemistry, 53(8), 1981, pp. 928A-938A.
4. Currie, L.A. and J.R. DeVoe, "Systematic Error in Chemical
Anaysis," In: Validation of the Measurement Process, ACS
Symposium Series 63, American Chemical Society, Washigton, O.C.,
1977, pp. 114-139.
8.3 Assessment of Precision
1. Bennett, C.A. and N.L. Franklin, Statistical Analysis in
Chemistry and the Chemical Industry, Wiley, New York, 1954.
2. Rhodes, R.C., "Components of Variation in Chemical Analysis."
In: Validation of the Measurement Process, ACS Symposium Series
No. 63, American Chemical Society, Washington, O.C. 1977, pp.
176-198.
30
-------
3. Wilson, A.L., "The Performance Characteristics of Analytical
Methods-II," Talanta. 17, 1970, pp. 31-44.
4. Sicking, C.A., "Precision in the Routine Performance of
Standard Tests," ASTM Standardization News. January 1979, pp.
12-14.
5. Merten, 0., L.A. Currie, J. Mandel, 0. Suschny and G.
Wernimont, "Intercomparison, Quality Control and Statistics."
In: Standard Reference Materials and Meaningful Measurements,
NBS Special Publication 408, U.S. Department of Commerce,
National Bureau of Standards, 1975, p. 805.
6. Janardan, K.G.. and D. J. Schaeffer, "Propagation of Random
Error in Estimating the Levels of Trace Organics in
Environmental Sources," Analytical Chemisty, 51(7), 1979, pp.
1024-1026.
7. Bicking, C.A., "Inter-Laboratory Round Robins for Determination
of Routine Precision of Methods." In: Testing Laboratory
Performance. NBS Special Publication 591, U.S. Department of
Commerce, National Bureau of Standards, 1980, pp. 31-34.
8. Wernimont, G., "Use of Control Charts in the Analytical
Laboratory," Industrial and Engineering Chemistry, 18(10),
1946, pp. 587^55?!'
9. Frazier, R.P., et al., "Establishing a Quality Control Program
for a State Environmental Laboratory," Water and Sewage Works.
121(5), 1974, pp. 54-57.
10. Dorsey, N.E. and C. Eisenhart, "On Absolute Measurement." In:
Precision Measurement and Calibration, NBS Special Publication
300, U.S. Department of Commerce, National Bureau of Standards,
1969, pp. 49-55.
11. Suschny, 0. and D.M. Richman, "The Analytical Quality Control
Programme of the International Atomic Energy Agency." In:
Standard Reference Materials and Meaningful Measurements, NBS
Special Publication 408, U.S. Department of Commerce, National
Bureau of Standards, 1975, pp. 75-102.
8.4 Assessment of Accuracy
1. Uriano, G.A. and C.C. Gravatt, "The Role of Reference Materials
and Reference Methods in Chemical Analysis," CRC Critical
Reviews in Analytical Chemistry, 6(4), 1977, pp. 361-411.
2. Uriano, G.A. and J.P. Cali, "Role of Reference Materials and
Reference Methods in the Measurement Process." In: Validation
of the Measurement Process, ACS Symposium Series No. 63,
American Chemical Society, Washington, D.C., 1977, pp. 140-161.
31
-------
3. Skogerboe, R.K. and S.R. Koirtyohann, "Accuracy Assurance in
the Analysis of Environmental Samples." In: Accuracy in Trace
Analysis. Vol. 1, NBS Special Publication 422, U.S. Department
of Commerce, National Bureau of Standards, 1976, pp. 199-210.
4. Watts, R.R., "Proficiency Testing and Other Aspects of a
Comprehensive Quality Assurance Program." In: Optimizing
Chemical Laboratory Performance through the Application of
uality Assurance Principles. Association of Official
nalytical Chemists, Arlington, VA, 1980, pp. 87-115.
5. Horwitz, W.L., R. Kamps and K.W. Boyer, "Quality Assurance in
the Analysis of Foods for Trace Constituents," Journal of the
Association of Official Analytical Chemists. 63(6), 1980, pp.
1344-1354.
6. Colby, B.N., "Development of Acceptance Criteria for the
Determination of Organic Pollutants at Medium Concentrations in
Soil, Sediments, and Water Samples," EPA Contract No.
68-02-3656, Systems Science and Software, LaJolla, CA, 1981.
7. Bicking, C., S. 01 in and P. King, Procedures for the Evaluation
of Environmental Monitoring Laboratories, Tracer Jitco, Inc.,
EPA-600/4-78-017, U.S. EPA, Office of Research and Development,
Environmental Monitoring and Support Laboratory, Cincinnati,
1978.
8. U.S. Department of the Army, "Quality Assurance Program for
U.S. Army Toxic and Hazardous Materials Agency," Aberdeen
Proving Ground, MD., August 1980 (draft).
9. Freeberg, F.E., "Meaningful Quality Assurance Program for the
Chemical Laboratory." In: Optimizing Chemical Laboratory
Performance Through the Application of Quality Assurance
Principles, Association of Official Analytical Chemists,
Arlington, VA, 1980, pp. 13-23.
10. American Society for Testing and Materials, "Standard Practice
for Determination of Precision and Bias of Methods of Committee
D-19 on Water," ASTM Designation: 02777-77. In: 1977 Annual
Book of ASTM Standards, Part 31. pp. 7-19.
11. Frazier, R.P., et al., "Establishing a Quality Control Program
for a State Environmental Laboratory," Water and Sewage Works.
121(5). 1974, pp. 54-57.
8.5 Use of Control Charts
1. Shewhart, W.A., Economic Control of Manufacture Products. Van
Nostrand, New York, 1931.
32
-------
2. McCully, K.A. and J.G. Lee, "Quality Assurance of Sample Analysis
in the Chemical Laboratory." In: Optimizing Chemical Laboratory
Performance through the Application of Quality Assurance
Principles, Association of Official Analytical Chemists,
Arlington, VA, 1980, pp.57-86.
3. Duncan, A.J., Quality Control and Industrial Statistics. 3rd
edition, Richard D. Irwin, Inc., Homewood, IL, 1968.
4. Grant, E.L. and R.S. Leavenworth, Statistical Quality Control. 4th
edition, McGraw-Hill, New York, 197TI
5. Environmental Monitoring and Support Laboratory, Handbook for
Analytical Quality Control in Water and Wastewater Laboratories,
EPA-600/4-79-019, U.S. EPA, Office of Research and Development,
Cincinnati, 1979.
6. Wernimont, G., "Use of Control charts in the Analytical Laboratory,
Industrial and Engineering Chemistry, 18(10), 1946, pp.587-592.
7. Bennett, C.A. and N.L. Franklin, Statistical Analysis in Chemistry
and the Chemical Industry, Wiley, New York, 1954.
8. Eisenhart, C., "Realistic Evalution of the Precision and Accuracy
of Instrument Calibration Systems." In: Precision Measurement
and Calibration, NBS Special Publication 300, U.S. Department of
Commerce, National Bureau of Standards, 1969, pp.21-47.
9. Wernimont, G., "Statistical Control of the Measurement Process."
In: Validation of the Measurement Process, ACS Washington, D.C.,
1977, pp.1-29.
10. Moore, P.G., "Normality in Quality Control Charts," App11ed
Statistics, 6(3), 1957, pp.171-179.
11. Morrison, J., "The Lognormal Distribution in Quality Control,"
Applied Statistics, 7(3), 1958, pp.160-172.
12. Iglewicz, B. and R.H. Myers, "Comparison of Approximations to the
Percentage Points of the Sample Coefficient of Variation,"
Technometrics, 12(1), 1970, pp.166-170.
13. Environmental Monitoring and Support Laboratory, Quality Assurance
Handbook for Air Pollution Measurement Systems, Volume I -
Principles, EPA-60Q/9-76-QQ5, U.S. EPA, Office of Research and
Development, Research Triangle Park, NC, 1976, p.22 (Appendix H).
14. Grubbs, F.E. "The Difference Control Chart with an Example of Its
Use," Industrial Quality Control, July, 1946, pp.22-25.
15. Page, E.S., "Cumulative Sum Charts," Technometrics, 3(1), 1961,
pp.1-9.
33
-------
16. Jackson, J.E., "Quality Control Methods for Several Related
Variables," Technometrics. 1(4), 1959, pp. 359-377.
17. Jackson, J.E. and R. H. Morris, "An Application of Multivariate
Quality Control to Photographic Processing," Journal of the
American Statistical Association. 52, 1957, pp. 186-199.
18. Montgomery, O.C. and H.M. Wadsworth, "Some Techniques for
Multivariate Quality Control Applications," ASQC Technical
Conference Transactions. 1972.
19..Frazier, R.P., J.A. Miller, J.F. Murray, M.P. Mauzy, D.J.
Schaeffer and A.F. Westerhold, "Establishing a Quality Control
Program for a State Environmental Laboratory," Water and Sewage
Works. 121(5), 1974, pp. 54-57.
20. Hillier, F.S., "X and R-Chart Control Limits Based on a Small
Number of Subgroups,"Journal of Quality Technology. 1(1), 1969,
pp. 17-26.
8.6 Method Detection Limits
1. Glaser, J.A., D.L. Foerst, G.,D. McKee, S.A. Quave, W.L. Budde,
"Trace Analysis for Wastewaters," Environmental Science and
Technology, 15, 1981, pp. 1426-143^
2. Hubaux, A. and G. Vos, "Decision and Detection Limits for Linear
Calibration Curves," Analytical Chemistry. 42, 1970, pp. 849-855.
3. "Guidelines for Data Acquisition and Data Quality Evaluation in
Environmental Chemistry," Analytical Chemistry, 52, 1980, pp.
3342-2249.
4. Currie, L.A., "Limits for Qualitative Detection and Quantitative
Determination - Application to Radiochemistry," Analytical
Chemistry. 40, 1968, pp. 586-594
5. Ramirez-Munoz, J., "Qualitative and Quantitative Sensitivity in
Flame Photometry." Talanta. 13, 1966, pp. 87-101.
6. Parsons, M.L., "The Definition of Detection Limits," Journal of
Chemical Education. 46, 1969, pp. 290-292.
7. Ingle, J.D., Jr., "Sensitivity and Limit of Detection in
Quantitative Spectrometric Methods," Journal of Chemical
Education, 51, 1974, pp. 100-105.
8. Wilson, A.L., "The Performance Characteristics of Analytical
Methods - III," Talanta. 20, 1973, pp. 725-732.
9. Kaiser, H., "Guiding Concepts Relating to Trace Analysis," Pure
and Applied Chemistry, 34, 1973, pp. 35-61.
34
-------
APPENDIX A
EXAMPLE CALCULATIONS
I. PRECISION ASSESSMENTS
A. Precision assessments based upon three or more replicate measurements
Example 1; One sample In a small 5-sample lot was analyzed four
times to assess precision. The results were 48, 55, 50 and 45
concentration units (CU). Using Equations 2, 3b, arid 45:
J. - 49.5 CU
s1 * 4.2 CU
RSO. = - (100) = 8.5%
The standard deviation of individual measurements in this sample lot
is estimated at concentration X to be 0.085X.
B. Precision assessments based upon duplicate measurements
Example 2: For a 100-sample study, 10 samples were analyzed in
duplicate. The results for each set of duplicates (X-j and X£) are
tabulated below, along with values for "X. , R., and RR.
calculated using Equations 2, 3a, and 4a.
xl
1.5
1.7
2.0
2.4
2.7
3.4
3.9
5.0
4.8
5.2
x2
1.7
1.6
2.1
2.1
2.4
3.5
4.3
4.5
4.9
4.7
Xi
' 1.6
1.65
2.05
2.25
2.55
3.45
4.1-
4.75
4.85
4.95
Ri
0.2
0.1
0.1
0.3
0.3
0.1
0.4
0.5
0.1
0.5
Sum
RR.
12.5
6.1
4.9
13.3
11.8
2.9
9.8
10.5
2.1
10.1
3470"
An inspection of the tabulation reveals no clear relationship between
the relative range and average concentration. Therefore,
35
-------
the average relative range (RR) is calculated using Equation 5a:
RR = 84.0S/10 = 8.4%
The precision of individual measurements in Example 2, expressed as
average range of duplicates at any concentration, X, is estimated to
be 0.084X.
Example 3; For a 100-sample study, 10 samples were analyzed in
duplicate. The results for each set of duplicates are tabulated
below along with values for X^., R^ and RR^:
xl
5.33
10.1
19.5
18.6
32.8
108.5
132
186
501
3517
x2
6.37
8.65
17.6
20.5
36.1
102
124
197
527
3341
Xi
5.85
9.38
18.55
19.55
34.45
105.2
128
191.5
514
3429
Ri
1.04
1.45
1.9
1.9
3.3
6.5
8.0
11
26
176
RRf
17.8
15.5
10.2
9.8
9.6
6.2
6.2
5.7
5.1
5.1
In this example, the tabulation shows a clear decrease in relative
range with increasing concentration. A least-squares linear
regression analysis of R-} as a function of X"j for the data
above yields a regression line:
R = 0.051 X~ + 0.987
The regression equation is used to represent the precision of the
measurement system and is used to calculate the estimate range of
duplicates for all members of the sample lot. The individual
measurements in Example 3 are estimated to have, at any
concentration, X, an average range of duplicates of 0.051X + 0.99
36
-------
concentration units.
C. Continual precision assessments
Example 4; A laboratory is analyzing and reporting samples on a
continual basis. An examination of historical data for duplicate
results has established that R' = 0.051 7+ 0.99 concentration units.
A sample is analyzed in duplicate, yielding 18.6 and 20.5 CU. The *•
expected range for the duplicate pair (R1) is calculated for the
average concentration of 19.55 CU to be R' = 0.051(19.55) +0.99 =
1.99 CU. The control limit for the range of the duplicate pair is
calculated to be 3.27(1.99) CU = 6.52 CU. Since the observed
difference (1.9 CU) is less than the 6.52 CU control limit, the
result is within
expectations and the precision assessment can be considered valid for
the sample. A graphical presentation of R1 vs. X" may be convenient
for use in a laboratory analyzing large number of samples.
II. ACCURACY ASSESSMENTS
A. Summary Accuracy Assessments
Example 5: For a 100-sample study, 10 sample aliquots were spiked
and analyzed along with unspiked aliquots. No volume correction was
required. The results of the analyses are tabulated below:
Background Spike Result Recovery Percent
Bi
4.0
7.9
4.5
1.3
17.3
26.3
5.7
5.0
62.5
34.5
Ti
20.0
20.0
20.0
20.0
50.0
100.0
20.0
20.0
200.0
100.0
Ai
24.8
26.2
25.4
21.2
66.7
128.0
24.8
24.8
260.5
135.3
20.8
18.3
20.9
19.9
49.4
101.7
19.1
19.8
197.8
100.8
Recovery.P^
104.0
91.5
104.5
99.5
94.8
101.7
95.5
99.0
98.9
100.8
Sum 99U77
37
-------
Individual percent recoveries were calculated using Equation 8. The
average percent recovery, P, and the standard deviation of. the
percent recovery, s , are calculated as in Equation's 2 and 3b:
P~ = 99.0
sp.4.1
The first term of the 95% confidence interval Is 99 - 2(4.1) = 91 and
the second term is 99 + 2(4.1) = 107. Each individual measurement in
Example 5 is estimated to have an accuracy, expressed as the 95%
recovery interval, of 91-107%.
B. Continual Accuracy Assessments
Example 6: A laboratory is analyzing and reporting samples on a
continual basis. Historical data for the analysis of spiked samples
established that P = 99.0%, and sp = 4.1; the control limits are
"P ± 3 s , or 86.7-111.3%. A sample with a [measured background
level (B.) of 22.0 concentration units was spiked with the
equivalent of 30.0 concentration units (T. ) without affecting the
sample volume. The result for the analysis of the spiked sample
(A.) was 49.2 concentration units. Using Equation 8:
Control Limits = 86.7 - 111.3%
Because Pj falls within the control limits, the accuracy
assessment can be considered valid for the sample. A graphical
presentation of P" ± 3s versus test number may be convenient for
use in a high volume laboratory.
38
------- |