EPA REGION VII IRC
097720
EPA/620/R-96/001
February 1996
ENVIRONMENTAL MONITORING AND ASSESSMENT PROGRAM (EMAP)
CHEMICAL METHOD EVALUATION GUIDANCE
prepared by
Gary Collins
Aquatic Research Division
National Exposure Research Laboratory
U.S. Environmental Protection Agency
Cincinnati, Ohio 45268
edited by
Shannon Fink
DynCorp/TAI
Cincinnati, Ohio 45268
EPA Contract No. 68-C1-0022
Project Officer
Linda Ransick
Aquatic Research Division
National Exposure Research Laboratory
U.S. Environmental Protection Agency
Cincinnati, Ohio 45268
Aquatic Research Division
National Exposure Research Laboratory
U.S. Environmental Protection Agency
Cincinnati, Ohio 45268 ^•S;T'™™'™^1 Protection Agency
' ''
lMlo;-:-n'.ion T^mirce Center
-------
DISCLAIMER
The U.S. Environmental Protection Agency through its Aquatic Research Division funded
and managed the preparation of the guidance described here under Contract No. 68-C1-0022. It has
been reviewed by management of the Environmental Monitoring and Assessment Program (EMAP)
and the National Exposure Research Laboratory (NERL), U.S. Environmental Protection Agency
(USEPA). It has further been subjected to the Agency's peer and administrative review process and
has been approved for publication. The mention of trade names or commercial products does not
constitute endorsement or recommendation for use.
11
-------
FOREWORD
The U.S. Environmental Protection Agency (USEPA) is developing the Environmental
Monitoring and Assessment Program (EMAP) to determine the current status and trends in the
condition of our nation's ecological resources on regional and national scales. EMAP's goal is to
monitor the condition of the nation's ecological resources, thereby contributing to decisions on
environmental protection and management. EMAP's monitoring efforts will operate on regional
scales over periods of years to decades and will involve collecting data from all ecosystems. EMAP
data will enable policy makers, scientists, and the public to evaluate the success of current policies
and programs and to identify emerging problems before they become widespread or irreversible.
EMAP's ecological status and trends data will allow decision makers to assess objectively whether
or not the nation's ecological resources are responding positively, negatively, or not at all to
regulatory programs.
This document has been prepared to assist scientists and managers hi selecting and evaluating
chemical laboratory measurement methods for producing EMAP data. It is the first in a series of
documents intended to ensure that the methods used to generate EMAP data will provide the quality
required of the task. Subsequent documents in this series will provide guidance for use when
evaluating biological laboratory methods, field measurement methods, and field collection methods.
Each evaluation guidance document will provide guidance for use in determining if: 1) method
documentation is adequate to meet the needs of EMAP, and 2) a method produces data of sufficient
quality to meet the needs of EMAP.
The "EMAP Chemical Method Evaluation Guidance" has been prepared by Gary Collins,
the EMAP-Methods Technical Coordinator, and edited and formatted through Contract #68-Cl-
0022 with DynCorp/TAI. The suggested citation for this document is:
Collins, Gary. 1996. Environmental Monitoring and Assessment Program: Chemical
Methods Evaluation Guidance, EPA 620/R-96/001. Environmental Monitoring and
Assessment Program: U.S. Environmental Protection Agency, Cincinnati, Ohio.
in
-------
TABLE OF CONTENTS
Page
DISCLAIMER ii
FOREWORD iii
ACKNOWLEDGEMENTS vi
SECTION 1 - INTRODUCTION 1
1.1 Background 1
1.2 Chemical Laboratory Method Evaluation Guidance 2
SECTION 2 - METHOD PERFORMANCE EVALUATION 4
2.1 Summary 4
2.2 Method Approval for Previously Used Methods 5
2.3 Method Approval for New Candidate Methods 5
SECTION 3 - METHOD PERFORMANCE CRITERIA 9
3.1 Summary 9
3.2 Bias and Percent Recovery 9
3.3 Precision 10
3.4 Method Detection Limit 11
3.5 Selectivity 11
SECTION 4 - METHOD CONTENT 12
4.1 Summary 12
4.2 EMMC Method Format 12
4.3 Evaluation of Method Documentation 16
SECTION 5 - EMAP METHOD VALIDATION 18
5.1 Summary 18
5.2 EMAP Method Validation 18
5.3 Required Documentation 20
5.4 Checklist 22
5.5 Certification Statement 25
5.6 Evaluation of Method Performance 25
IV
-------
TABLE OF CONTENTS (cont.)
Page
SECTION 6 - EMAP DEMONSTRATION OF METHOD COMPARABILITY 28
6.1 Summary 28
6.2 EMAP Demonstration of Method Comparability 28
6.3 Required Documentation 30
6.4 Checklist 31
6.5 Certification Statement 35
6.6 Evaluation of Method Performance 35
SECTION 7 - REFERENCES 37
APPENDICES 38
Appendix A: EMAP Method Validation Checklist for Chemical Methods 39
Appendix B: Method Range and Method Detection Limit 42
Appendix C: Matrix Validation 46
Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals 52
Appendix E: Statistical Example 67
Appendix F: EMAP Demonstration of Method Comparability Checklist for
Chemical Methods 79
-------
ACKNOWLEDGEMENTS
This document has resulted from several stages of technical review occurring from
December, 1993 to October, 1995. The assistance of EMAP-Center in the preparation of draft
documents, especially that of Joanie Currin, is gratefully acknowledged. Helpful comments are
acknowledged from reviewers at each stage including Barbara Conkling, Susan Diaz-Ramos, Sidney
Draggan, Florence Fulk, Tammy Goyke, Robert Graves, Laura Jackson, Lora Johnson, Linda
Kirkland, John Martinson, Mike McGinnis, Dennis McMullen, Barbara Metzger, Tony Olsen, Jack
Pfaff, David Peck, Robert Slater, D.L. Stevens, Jr., and Russell Wright.
The EMAP Methods Coordination Group (EMAP-Methods) would also like to acknowledge
the contributions of DynCorp/TAI in the preparation of the statistical sections of this document.
Laura Gast prepared Appendix D: Statistics for Determining Outliers, Significance, and Confidence
Intervals, and Cecily Shimp prepared Appendix E: Statistical Example.
VI
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 1 of 81
SECTION 1
INTRODUCTION
1.1 Background
1.1.1 As originally conceived, EMAP consisted of seven Coordination Groups: Quality Assurance,
Indicators, Information Management, Design and Statistics, Assessment and Reporting, Landscape
Characterization, and Methods. These groups coordinated the collection of data from seven
Resource Groups: Rangelands, Agroecosystems, Forests, Inland Surface Waters, the Great Lakes,
Estuaries, and Landscapes. This organizational structure is currently undergoing change, but the
guidance provided here will serve any long term environmental monitoring effort. As one of several
coordination activities being undertaken by EMAP-Methods, this document provides guidance on
chemical method evaluation in support of the larger EMAP goals of developing indicators that
produce comparable data at regional and national scales. The coordination activities being
undertaken by EMAP-Methods include providing method format guidance, identifying and
organizing EMAP methods in use, and providing guidance for evaluating EMAP methods. EMAP-
Methods has the following objectives:
• To provide guidance for preparing methods in a standard method format that includes written
specifications for the style and content of EMAP methods and method manuals;
• To establish a database on the EPA Wide Area Information Server (WAIS) and Web Server
containing the full text of all EMAP methods;
• To establish a relational database in the EMAP Information Management System containing
full text and abstracted versions of all EMAP methods to be directly tied to analytical data sets;
and
• To develop a method evaluation protocol suited to the requirements of EMAP that ensures that
methods used by the Resource Groups are consistent and produce comparable data.
1.1.2 Method manuals produced and used by the Resource Groups are in different stages of
development. Field and laboratory method manuals have been preprared for pilot, demonstration,
and full implementation stages of EMAP. Resource Groups have prepared these methods and
manuals in the style most familiar to them. EMAP-Methods has prepared the document entitled
"EMAP Methods Format Guidance" (USEPA, 1995) to meet the objective of providing
standardization for method and method manual preparation within EMAP. The method format
presented in that document, and required for use with EMAP methods, is the Environmental
Monitoring Methods Council (EMMC) format. Specifically, this format is required for use with
field collection, field measurement, and biological and chemical laboratory methods. In addition
to describing the required method format, the "EMAP Methods Format Guidance" (USEPA, 1995)
provides specifications and guidance for preparing hardcopy and electronic versions of methods and
method manuals. This document is available over the Internet at the following address:
http://www.epa.gov/docs/emap/coordgrp/methods.
1.1.3 One of the databases under development will become a part of the EPA WAIS and Web
Servers and will provide public access to the full text of methods and related guidance documents.
1
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 2 of 81
The other database will form the EMAP-Methods relational database, which will provide authorized
users with on-line access to the full text and abstracted versions of methods. The relational database
will link analytical data to methods information to support compliance with EMAP's "20 year rule,"
which states that all EMAP data must be sufficiently documented to be fully understandable in 20
years. The databases will provide method users with a rapid way of determining if an existing
method for a given parameter has been used within EMAP. The databases will facilitate the
coordination of methods issues across the program by providing a means of tracking the introduction
of new methods into the program, as well as method changes through time. Standard documentation
is essential for the entry and retrieval of methods into and from the databases and; therefore, certain
format specifications provided in the "EMAP Methods Format Guidance" (USEPA, 1995) were
defined specifically to support their use.
1.1.4 EMAP-Methods is responsible for developing a method evaluation protocol that will support
EMAP's Performance-Based Method System (PBMS). Methods used within EMAP are typically
selected from scientific literature or are newly developed. This method evaluation protocol provides
guidance on the validation and acceptance of methods obtained from various sources. The protocol
is intended to assist method evaluators in determining if the methods used within EMAP are
adequately documented and whether the methods meet the needs of the program. EMAP-Methods
will provide a set of four evaluation guidance documents, one for each of the following types of
methods: field collection, field measurement, biological laboratory, and chemical laboratory
methods. Each of these documents will provide the standard format for EMAP methods with
specific information pertinent to the targeted discipline. The documents will also provide
instructions for performing and evaluating method validation activities and method comparability
studies. These instructions are presented to assist scientists and managers in determining if
candidate method performance is adequate to meet the needs of EMAP.
1.2 Chemical Laboratory Method Evaluation Guidance
1.2.1 This document has been written to provide practical guidance for chemical laboratory method
evaluation and method approval. It will enable scientists and managers to evaluate whether a
candidate method is fully documented and if it will provide data that meet EMAP MQOs. This
document provides a standardized procedure for establishing any candidate method as an approved
EMAP method.
1.2.2 To apply the guidance on method evaluation provided in this document, the following must
be true.
• A parameter to be measured has been identified;
• A candidate method has been used or proposed to measure the parameter; and
• Measurement Quality Objectives (MQOs) for the measurement have been defined by the
program and are available to method assessors.
1.2.3 Any candidate method will be accepted for EMAP use once the method has been shown to
be completely documented and capable of meeting EMAP quality objectives. A previously used
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 3 of 81
or new candidate method may become an approved EMAP method if the method has been
completely documented in the EMMC method format and:
• Has been validated and shown through validation to meet MQOs; or
• Has been compared to a validated method and shown to meet or exceed the performance
specifications of that method.
1.2.4 Outline of Guidance: This document provides guidance to method users for characterizing
and documenting method performance, and to method assessors for evaluating method performance
and accepting or rejecting methods based on performance data. The following paragraphs describe
the framework of the guidance provided.
Section 2 summarizes the procedures required to characterize, document, and evaluate method
performance for approval. This section provides an overview of the guidance contained in
subsequent sections of this document.
Section 3 provides a discussion of the criteria that are used to evaluate the performance of
candidate chemical methods during the approval process.
Section 4 provides a description of the EMMC method format to assist in preparing written
chemical methods and evaluating the completeness of method documentation. The "EMAP
Methods Format Guidance" (USEPA, 1995) contains detailed instructions for preparing
methods in the EMMC format and should be used to obtain document formatting specifications.
To become accepted for future use in performing EMAP measurements, methods must be
documented in the EMMC format.
Section 5 describes the procedures required to document and evaluate a method using the
EMAP Method Validation, and assists method evaluators in determining if a candidate method
produces data that meet quality objectives. The EMAP Method Validation is required to
approve a new candidate method as an EMAP method, when the candidate method has not
previously been validated, and when no approved EMAP method is available for comparison.
The required elements of the EMAP Method Validation are performed to characterize the
method's performance in terms of precision, percent recovery, method range, and method
detection limit (MDL). This section of the guidance also introduces ruggedness testing, which
may be used to characterize the method's performance in producing acceptable data over a
reasonable range of analytical conditions.
Section 6 describes the procedures required to document and evaluate a method using the
EMAP Demonstration of Method Comparability. This section is intended to assist method
evaluators in determining if a candidate method produces data that meet or exceed the
performance specifications of an approved EMAP method. The procedures described will only
be used when previously approved EMAP methods are available for comparison.
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 4 of 81
SECTION 2
METHOD PERFORMANCE EVALUATION
2.1 Summary
2.1.1 Selecting appropriate measurement methods is a critical task for EMAP, A method evaluation
protocol for EMAP should ensure that the methods used are reliable in producing data of sufficient
quality to meet the needs of EMAP. The EPA publishes validated test methods that are designated
as appropriate for complying with various regulations. These test methods are fully documented and
have been extensively tested and characterized to assure that they meet specific measurement
requirements under a variety of conditions likely to be encountered in their intended application.
The process for approving EPA validated methods for EMAP use consists of determining: 1) if the
method is written in the EMMC format; and 2) if the method's performance specifications meet
MQOs.
2.1.2 To allow for technological improvements and to enable laboratories utilizing varying
methodologies to participate in EMAP, the EMAP method evaluation protocol addresses the
approval of alternate methods for use within the program. For EPA, approval of alternate methods
has required extensive alternative test procedures to show equivalence to validated methods (i.e.,
the "Requirements for Approval of Alternate Test Procedures for Inorganic and Organic Analytes
in National Pollutant Discharge Elimination System Monitoring" (USEPA, 1993)). Recently, the
PBMS used by other Federal agencies has been approved for use by EPA. This approach provides
method performance specifications and permits the use of validated methods, as well as alternate
or adapted methods that meet the specifications provided.
2.1.3 EMAP has chosen to apply a PBMS to its method evaluation and approval process. This
approach will be used to allow laboratories participating in EMAP the flexibility to utilize
measurement methods other than those supplied by EPA. Furthermore, the approach will enable
laboratories to provide methods to EMAP when EPA validated methods are not available for a
parameter of interest. This document provides a formal approach to method approval for EMAP.
This method evaluation guidance has been developed to support the desired PBMS and to control
the propagation of multiple methods. To achieve both objectives, the requirements for method
approval consist of a combination of elements of the "Single-Laboratory Method Validation
Protocol" (USEPA, 1985) and elements of the PBMS to ensure that:
• the methods used for EMAP are adequately documented;
• the methods used for EMAP perform reliably in meeting MQOs;
• method selection is not restricted to EPA validated methods only; and
• the use of multiple methods for measuring the same analytes or properties will not result in a
loss of quality or comparability among the data.
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 5 of 81
2.2 Method Approval For Previously Used Methods
2.2.1 Candidate methods may be methods that have already been used in pilot and/or demonstration
stages of EMAP, or they may be new to the program. Because no formal review and approval
procedures were set forth prior to this guidance, the methods already in use have been used without
having undergone the procedures detailed here. These methods will be "grandfathered" only to the
extent that they will be entered into the methods database regardless of their approval status to
permit traceability to the data that has already been produced. For data collection or generation
subsequent to this guidance, however, formally approved methods must be used for collecting and/or
generating the data.
2.2.2 Previously used, but non-approved methods must be taken through the evaluation procedures
issued in this guidance for approval for continued or future use. To reduce the burden of validating
these methods at this time, the guidance permits the use of an abbreviated evaluation and approval
process. The flowchart presented in Figure 1: Evaluation and Approval Procedures for Previously
Used Methods (p.7) summarizes the requirements for approving methods that have been previously
used to generate EMAP data. These methods become approved EMAP methods when they have
been completely documented. As used here, "completely documented" means that the method is
written in the EMMC format, including all method performance specifications such as precision,
percent recovery, method range, and MDL. All candidate methods must meet the complete
documentation specification.
2.2.3 Section 4 - Method Content, provides guidance for preparing the method in the EMMC
format. To meet the requirement for documenting method performance, data must be available in
a level of detail at least as rigorous as that described in Section 5 - EMAP Method Validation. If this
information is not included in the written method in sufficient detail, the method must be treated as
a new method. All new methods must undergo one of the procedures described subsection 2.3 -
Method Approval for New Candidate Methods.
2.3 Method Approval For New Candidate Methods
2.3.1 For new candidate methods, the evaluation and approval process will depend upon whether
the method has undergone a method validation and whether an approved EMAP method is available
for the parameter to be measured. Throughout this guidance, the "Single-Laboratory Method
Validation Protocol" (USEPA, 1985) is cited as an acceptable alternate to the EMAP Method
Validation. Furthermore, wherever the "Single-Laboratory Method Validation Protocol" (USEPA,
1985) is cited, any validation protocol may be substituted, provided that the substituted protocol is
at least as rigorous as the "Single-Laboratory Method Validation Protocol" (USEPA, 1985) The
evaluation and approval procedures for new candidate methods are summarized in Figure 2:
Evaluation and Approval Procedures for New Candidate Methods (p. 8). If the method is written
in the EMMC format and is an EPA validated method, it will be approved for EMAP use if the
method's performance specifications are evaluated and determined to meet defined MQOs. If the
method is not an EPA validated method, it must undergo one of the procedures described in
subsections 2.3.4 and 2.3.5, as appropriate.
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 6 of 81
2.3.2 The guidance for evaluating and approving new methods presented in this document has been
compiled from existing information for the convenience of EMAP method users and assessors. The
EMAP Method Validation procedures have been adapted from the "Single-Laboratory Method
Validation Protocol" (USEPA, 1985). The EMAP Demonstration of Method Comparability
procedures have been adapted from the "Guidance on the Evaluation of Safe Drinking Water Act
Compliance Monitoring Results from Performance-Based Methods" (USEPA, 1994). Sections of
these two documents have been selected and modified to define the minimum requirements for
characterizing and documenting the performance of candidate EMAP methods.
2.3.3 While this guidance provides two procedures for obtaining method approval, the
requirements for method approval have been designed to ensure that all approved EMAP methods
are traceable to a formally validated method. All previously used methods must be evaluated for
adherence to the requirements presented here before they are used to generate additional data and
before they may be used as the basis of method comparisons.
2.3.4 For new candidate methods that have not been validated according to the "Single-Laboratory
Method Validation Protocol" (USEPA, 1985) or according to another validation procedure which
is at least as rigorous, the evaluation process will depend upon whether an approved EMAP method
is available for the parameter to be measured. When no approved EMAP method is available for
comparison to a new candidate method, the EMAP Method Validation must be conducted and
evaluated to assess the method's performance. The procedures outlined in Section 5 - EMAP Method
Validation constitute the minimum requirements for characterizing, documenting, and evaluating
the performance and reliability of candidate methods when method comparability cannot be
assessed. A method will be approved for EMAP use when an evaluation of validation results
determines that the method meets MQOs.
2.3.5 Once a method has been written in the EMMC method format and method validation has
shown that the method meets MQOs, the method may be used as the approved EMAP method to
which a new candidate method's performance is compared. Using the approved EMAP method, the
EMAP Demonstration of Method Comparability is conducted to assess the candidate method's
performance. If the performance characteristics of the candidate method are determined to meet or
exceed the performance characteristics of the approved method, the candidate method will be
approved for use.
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 7 of 81
I Approve method for \
I EMAP use. I
Figure 1: Evaluation and Approval Procedures for Previously Used Methods
-------
EMAP Chemical Method Evaluation Guidance, Section 1, February 1996, Page 8 of 81
Summarize
validation results on
checklist and submit
all required
documentation.
* Throughout this guidance, "EPA validated" means validated according to the "Single-Laboratory Method Validation Protocol"
(USEPA, 1985).
Figure 2: Evaluation and Approval Procedures for New Candidate Methods
-------
EMAP Chemical Method Evaluation Guidance, Section 3, February 1996, Page 9 of 81
SECTION 3
METHOD PERFORMANCE CRITERIA
3.1 Summary
3.1.1 Performance criteria or data quality indicators are quantitative and qualitative descriptors that
are used to interpret the degree of acceptability of data to the user. Data quality indicator
information is calculated and kept to support the scientific integrity of EMAP data. The principal
data quality indicators addressed in this guidance are bias, percent recovery, precision, Method
Detection Limit (MDL), and selectivity. This section of the guidance provides working definitions
of these indicators.
3.1.2 The most important factor in evaluating a candidate method for approval for EMAP use is
determining whether the candidate method produces data that meet the quality objectives of the
EMAP Resource Group. MQOs are defined by the Resource Group based upon the intended use(s)
of the data. MQOs place numerical requirements on the value of data quality indicators to ensure
that the quality of EMAP data is sufficient to support EMAP's goals. Project planners will prepare
the project's Quality Assurance Project Plan (QAPP) containing the MQOs. The QAPP should
contain the MQOs with a discussion of how they are determined and how data will be evaluated
relative to them. For environmental data to be useful to EMAP, the methods used to generate the
data must meet these MQOs. This document's focus, therefore, is placed on evaluating the
performance of chemical laboratory methods by analyzing data quality indicators and comparing
analytical results to MQOs. Method users and assessors must retrieve the MQOs from the
appropriate QAPPs to evaluate whether a candidate method produces data of adequate quality.
3.2 Bias and Percent Recovery
3.2.1 Bias: In chemical measurements, bias is the difference between a measured value and the true
value of the quantity measured. Bias represents the systematic error of the measurement due to
sampling and analytical systems. Bias is determined from repeat analysis of reference (certified)
materials or samples fortified with known amounts of analytes of interest. For repeated
measurements of samples with known composition, bias may be calculated in absolute or relative
terms with the following equations:
Bias, B = x - T
-or-
x 100
where x equals the mean value for a set of measurements and T equals the theoretical true value of
the evaluation sample.
3.2.2 Percent Recovery: In chemical measurements, percent recovery is the ratio of the measured
value and the true value of a quantity measured. Percent recovery is estimated through the analysis
-------
EMAP Chemical Method Evaluation Guidance, Section 3, February 1996, Page 10 of 81
of quality control matrix samples or reagent blanks that are spiked with known amounts of target
analytes. When reagent blank samples are used, the percent recovery of the spiked analyte is
determined with the following equation:
Percent Recovery = — x 100
where x equals the mean value for a set of measurements and T equals the theoretical true value of
the evaluation sample.
Alternatively, environmental samples or standard matrix samples may be used for analysis. When
these samples are used, unspiked samples must be analyzed to determine if background
concentrations of target analytes are present in the sample. Percent recovery in a spiked sample
matrix that contains a background concentration is calculated using the following equation:
Percent Recovery = * ~ B x 100
where Jc equals the mean value for a set of measurements, T equals the theoretical true value of the
evaluation sample, and B equals the mean background concentration in the unspiked sample.
3.2.3 Because bias is not commonly used to quantify method performance, the EMAP
Demonstration of Method Comparability and the EMAP Method Validation require determinations
of percent recovery for the assessment of candidate method performance. For the EMAP
Demonstration of Method Comparability, the percent recovery of the candidate method will be
evaluated by comparison to the percent recovery of an approved EMAP method for the same
measurement. For the EMAP Method Validation, the percent recovery of the candidate method will
be evaluated by comparison to MQOs for percent recovery obtained from the appropriate QAPP.
3.3 Precision
3.3.1 Precision is the degree to which a set of replicate measurements agree. It may be expressed
as standard deviation, variance, or range. Precision represents the random error of the measurement
due to sampling and analytical systems. The error due to analytical systems is estimated from
repeated measurements of split samples or spikes in the laboratory. Precision may be expressed in
absolute terms as the standard deviation, s, or in relative terms as the relative standard deviation,
RSD. Precision is calculated with the following equations:
s = .£i^
(n- 1)
-or-
10
-------
EMAP Chemical Method Evaluation Guidance, Section 3, February 1996, Page 11 of 81
RSD = - x 100
where xt is an individual measurement, x is the mean value of the set of measurements, and n is the
number of measurements in the set.
3.3.2 The EMAP Demonstration of Method Comparability and the EMAP Method Validation
require replicate sample analyses to determine the precision of a candidate chemical laboratory
method. Respectively, the determined precision will be evaluated by comparison to the precision
of an approved EMAP method for the same measurement or by comparison to MQOs obtained from
the appropriate QAPP.
3.4 Method Detection Limit
3.4.1 The MDL is the lowest level at which the method can be expected to reliably measure the
analyte or property of interest. The MDL is defined as the minimum concentration of a substance
that can be measured and reported with 99% confidence that the analyte concentration is greater than
zero. MDLs are determined by analyzing at least seven replicate samples, spiked at a concentration
at or near an estimated limit of detection. MDLs are analyzed over time and/or across sample
batches to include long-term variation in the estimate. They are calculated from the standard
deviation between these replicate measurements as follows:
MDL = '.-1,0.01 X »
where tn_I001 equals the upper first percentile point of the ^-distribution with n-1 degrees of freedom
and s equals the standard deviation of the replicate measurements (see equation in subsection 3.3.1).
3.4.2 Both the EMAP Demonstration of Method Comparability and the EMAP Method Validation
require that the MDL be determined, documented, and evaluated for method approval.
3.5 Selectivity
3.5.1 Selectivity refers to the method's ability to selectively identify and measure the constituents
of interest in the presence of interferences and, in the case of multiple analyte methods, in the
presence of all other constituents of interest. Interferences are determined through the analysis of
reagent or standard matrix blank samples. Comparisons of blank analyses to those of spiked
standard matrices or sample matrices will allow method users to assess the method's selectivity.
3.5.2 The EMAP Demonstration of Method Comparability and the EMAP Method Validation
require that method interferences be identified. Whenever known interferences reduce the method's
selectivity, the method may be deemed unacceptable for the specific application. However, if the
method employs procedures that effectively minimize or eliminate the effects of the interferences,
the method may be approved for use.
11
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 12 of 81
SECTION 4
METHOD CONTENT
4.1 Summary
4.1.1 As noted in the introduction to this document, methods used to produce EMAP data are
required to be completely documented and properly formatted. EMAP-Methods stresses the
importance of consistent method preparation and documentation across the program for two reasons.
First, standardized documentation is necessary to ensure that all written EMAP methods contain
information that is appropriately detailed to meet EMAP needs. Second, standardized
documentation will be essential to storing and retrieving methods from the methods database, which
will enable the tying of data sets to methods and the tracking of methods through time.
4.1.2 To address the issue of standardization, the "EMAP Methods Format Guidance" (USEPA,
1995) was prepared. The guidance requires the use of the EMMC method format for all EMAP
measurement methods. The following subsections of this guidance document provide the sections
of the EMMC method format with content requirements and suggestions for preparing the method
sections for application to chemical laboratory methods. These requirements and suggestions are
presented to assist authors in preparing methods and to aid evaluators in assessing methods' content.
4.2 EMMC Method Format
4.2.1 All chemical laboratory methods used to produce EMAP data are required to be written in the
EMMC method format, containing the following sections, numbered as shown:
1. Scope and Application
2. Summary of Method
3. Definitions
4. Interferences
5. Personnel Health and Safety
6. Equipment and Supplies
7. Reagents and Standards
8. Sample Collection, Preservation, and Storage
9. Quality Control
10. Calibration and Standardization
11. Procedure
12. Data Analysis and Calculations
13. Method Performance
14. Pollution Prevention
15. Waste Management
16. References
17. Tables, Diagrams, Flowcharts, and Validation Data (optional final section)
12
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 13 of 81
4.2.2 Each of method sections one through 16 must be addressed in the written method. If any of
these sections do not apply to the method, the section heading must be listed as shown, with "Not
Applicable" or "N/A" appearing below. If Section 17. Tables, Diagrams, Flowcharts, and
Validation Data is not needed, the heading for this section is not required to be included in the
written method.
1. Scope and Application
Describe the purpose of the method (i.e., what chemical analyte or attribute is being measured).
Include target analyte lists for the method and Chemical Abstract Service Registry Numbers for
chemical analytes. Identify all matrices for which the method applies.
2. Summary of Method
Provide a brief summary of the method's major steps. The purpose of the summary is to
provide a succinct overview of the technique to aid the reviewer or data user in evaluating the
method and the data. List sample volumes, extraction, digestion, concentration, and other
preparation steps employed, analytical instrumentation and detection systems, and qualitative
and quantitative data analysis techniques used.
3. Definitions
List definitions of terms relevant to the method, or terms with which the reader may be
unfamiliar. This section is a useful location for defining quality control samples and acronyms.
For extensive lists of definitions, a glossary may be attached to the end of the method.
4. Interferences
Describe any known or potential problems, such as sample or equipment contamination,
instrument noise, etc. that may be encountered during the performance of the method. Describe
the effects of any known or potential interferences on method performance. Describe any
procedures employed by the laboratory to prevent or minimize interferences or contamination
problems.
5. Personnel Health and Safety
Describe special precautions needed to ensure personnel safety during the performance of the
method. Safety issues discussed here should be limited to those that are method-specific and
that are beyond the scope of routine laboratory practices. Include information regarding
specific toxicity of target analytes or reagents. List personal protective equipment required for
personnel performing the method.
6. Equipment and Supplies
List all nonconsumable supplies or instruments needed to perform the method. Include material
specifications (glass, plastic, stainless steel, etc.) when needed. Use generic language whenever
13
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 14 of 81
possible; however, when specific equipment is necessary, this should be stated clearly. When
computational or data analysis software is used, include user-generated instructions for using
the software here.
7. Reagents and Standards
List all reagents and standards required to perform the method, along with necessary grades or
quality. Provide detailed instructions for preparation of reagents and standards. Reference
specific suppliers when necessary.
8. Sample Collection, Preservation, and Storage
Provide requirements and instructions for sample collection, preservation, shipment, and
storage conditions. For laboratory methods, extensive detail about sample collection,
preservation, and shipment are typically not required. These aspects should be listed here, but
need only be addressed as necessary for performance of the laboratory method. This section
should describe any actions required in response to problems with collection, preservation, etc..
For example, provide details regarding actions that are required to handle samples when they
have been received with a loss of integrity (i.e., without appropriate preservation or in broken
sample containers). When holding times are known and their effects have been studied, specify
requirements in this section, along with instructions for actions to be taken if holding times are
exceeded.
9. Quality Control
Provide all Quality Control (QC) requirements of the method. Describe the types of QC
samples analyzed, and the frequency of their use. Describe the analysis of QC samples,
including calculations such as percent recovery. List data acceptance criteria, along with
procedures for dealing with data that do not conform to requirements. This section may be used
to describe data review procedures, documentation requirements, etc..
10. Calibration and Standardization
Describe the initial method or instrument calibration. Indicate calibration frequency
requirements, the range of calibration standards required, the number of calibration standards
required, performance specifications, and corrective actions that are taken when performance
criteria are not met. Verification or continuing calibration procedures may be described here
or in 11. Procedure.
11. Procedure
Present the complete instructions for using the method, describing all sample processing,
instrumental analysis, or physical analysis steps in detail.
14
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 15 of 81
12. Data Analysis and Calculations
Provide instructions for qualitative data analysis procedures, such as those employed for
chemical analyte identification. Provide instructions for quantitative data analysis, including
calculations and definitions of constants. When analytical and/or computational software is
used to automate analysis, refer to any proprietary instructional documentation here and provide
user-generated instructions for use of the software.
13. Method Performance
Present method performance data, including precision, percent recovery, MDLs, and limitations
of the method. All information necessary to evaluate whether the method will meet MQOs
should be summarized in this section.
14. Pollution Prevention
Cite practices employed to minimize or prevent pollution that may be attributable to the
method. The following guidelines for the content of this section have been taken from Method
350.1 (EPA/600/R-93/100).
Pollution prevention encompasses any technique that reduces or eliminates the quantity or
toxicity of waste at the point of generation. Numerous opportunities for pollution prevention
exist in laboratory operation. EPA has established a preferred hierarchy of environmental
management techniques that places pollution prevention as the management option of first
choice. Whenever feasible, laboratory personnel should use pollution prevention techniques
to address waste generation. When wastes cannot be feasibly reduced at the source, EPA
recommends recycling as the next best option.
The quantity of a chemical or reagent purchased should be based on expected usage during its
shelf life and on the disposal costs of unused material. Actual reagent preparation volumes
should reflect anticipated usage and reagent stability.
For information about pollution prevention that may be applicable to laboratories and research
institutions, consult the document "Less is Better: Laboratory Chemical Management for Waste
Reduction", available from the American Chemical Society's Department of Government
Regulations and Science Policy, 1155 16th Street N.W., Washington D.C. 20036, (202) 872-
4477.
15. Waste Management
Describe proper disposal of waste and samples specific to the method. The following
guidelines for the content of this section have been taken from Method 350.1 (EPA/600/R-
93/100).
15
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 16 of 81
EPA requires that laboratory waste management practices be conducted consistent with all
applicable rules and regulations.
Excess reagents, samples and method process wastes should be characterized and disposed of
in an acceptable manner. EPA urges laboratories to protect the air, water, and land by
minimizing and controlling all releases from hoods and bench operations, complying with the
letter and spirit of any waste discharge permit and regulations, and complying with all solid and
hazardous waste regulations, particularly the hazardous waste identification rules and land
disposal restrictions.
For further information on waste management consult the "Waste Management Manual for
Laboratory Personnel", available from the American Chemical Society at the address listed in
14. Pollution Prevention.
16. References
List source documents or publications. Each method must be a free-standing document, in
which all information necessary for the method user to perform the method may be found.
Procedural steps or instructions may not be referenced as being found elsewhere and must be
included in full within the written method.
17. Tables, Diagrams, Flowcharts, and Validation Data
Tables, diagrams, flowcharts, and validation data may be included in the written method as a
final section, as an attachment to the method, or dispersed throughout the method, as
appropriate.
4.3 Evaluation of Method Documentation
4.3.1 Method documentation should be evaluated with regard to conformance to the EMMC format,
and with regard to completeness. The following questions should be asked when evaluating method
documentation.
• Is the method written in the EMMC format?
• Are all EMMC method sections represented in the method?
• Is "Not Applicable or N/A" used in place of needed information?
4.3.2 If the method is written in the EMMC format, with all sections represented, and containing
all needed information, the method meets the minimum documentation requirements. A method that
meets minimum documentation requirements should undergo further evaluation to determine the
method's suitability for its intended use. Otherwise, the method should be revised to meet these
requirements before additional evaluation takes place.
16
-------
EMAP Chemical Method Evaluation Guidance, Section 4, February 1996, Page 17 of 81
4.3.3 Methods that have been used for pilot and demonstration stages of EMAP and that are
completely documented as described in this section, including all required method performance data,
will be considered approved. In the event that a method is determined to conform to the EMMC
format, but some or all of the method performance data is missing or incomplete, the organization
using the method must produce the missing or incomplete data. This data must be produced by
completing the procedures described in Section 5 - EMAP Method Validation or those described in
Section 6 - EMAP Demonstration of Method Comparability. Once method performance data is
available, it shall be compared to MQOs to determine the method's capability of meeting EMAP
needs.
17
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 18 of 81
SECTION 5
EMAP METHOD VALIDATION
5.1 Summary
5.1.1 Any method intended to be used for EMAP measurements must be determined to be valid or
to perform comparably to a valid method. A method that has been taken through the validation
procedures described in this section or through the more exhaustive "Single-Laboratory Method
Validation Protocol" (USEPA, 1985) and has been shown to meet MQOs will be considered a valid
and approved EMAP method. Once a method has become an approved EMAP method, new
candidate methods may be compared to that method for approval, as described in Section 6 - EMAP
Demonstration of Method Comparability. All method comparisons must stem from an initial
validated method. Because the EMAP Method Validation is intended to provide a solid foundation
for later method comparisons, its requirements are rigorous.
5.1.2 The EMAP Method Validation is used to determine a method's precision, percent recovery,
method range, and MDL. Ruggedness testing is recommended, but not required, to characterize the
method's performance in producing acceptable data over a reasonable range of analytical conditions.
The data generated during validation are compared to MQOs to determine the method's suitability
for measuring the parameter of interest. A completely documented method that is determined to
meet MQOs, through evaluation of the data obtained from the EMAP Method Validation, will be
approved as an EMAP method. Alternatively, the "Single-Laboratory Method Validation Protocol"
(USEPA, 1985) may be substituted for the EMAP Method Validation to characterize the method's
performance.
5.1.3 The EMAP Method Validation must take place when a new method is proposed for use and
no approved method is available for comparison. Furthermore, any method used to perform EMAP
measurements, regardless of its prior use within the program, must contain the method performance
data described in this section. If such data is unavailable, the method must undergo the EMAP
Method Validation. After the validation steps outlined in this section have been completed, the
information and statistics generated should be included in the EMMC method format to complete
the document.
5.2 EMAP Method Validation
5.2.1 Two procedures are required to be performed for the EMAP Method Validation. These
procedures have been derived from the "Single-Laboratory Method Validation Protocol" (USEPA,
1985) and are described in detail in Appendix B: Method Range and Method Detection Limit and
Appendix C: Matrix Validation.
This section provides a brief description of the method range and method detection limit
determinations, of the matrix validation, and of ruggedness testing, which is highly recommended
although not required as part of the EMAP Method Validation. Figure 3: The EMAP Method
Validation (p.21) contains a flowchart that summarizes the sample analyses required of the EMAP
Method Validation. The three aspects of performance testing outlined in this figure may be
performed in any sequence desired. Appendix D: Statistics for Determining Outliers, Significance,
18
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 19 of 81
and Confidence Intervals provides instructions for calculating the statistics required by these
analyses. Appendix E: Statistical Example provides examples of the calculations to further aid
method users and evaluators.
5.2.2 Method Range and Method Detection Limit (MDL): Method validations must include a
determination of the method's range, which is the sample concentration range over which the method
is valid. The method range is determined by analyzing replicate standard matrix samples spiked at
multiple concentration levels spread over the instrument range. The method range is bounded by
the highest and lowest concentrations of analytes that produced acceptable results for precision and
percent recovery when processed through the entire analytical procedure. The MDL is determined
by analyzing at least seven replicate samples, spiked at a concentration at or near an estimated limit
of detection. MDLs are analyzed over time and/or across sample batches to include long-term
variation in the estimate. Appendix B: Method Range and Method Detection Limit contains detailed
instructions for performing determinations of method range and MDL.
5.2.3 Matrix Validation: Method precision, percent recovery, and range must be determined for
all matrices for which the use of the method is intended. Validation for a single matrix consists of
a determination of analyte background levels from the analysis of unspiked matrix samples, the
analysis of replicate spiked matrix samples, and a comparison of the results obtained for precision
and percent recovery to the performance requirements of the method. Replicate samples must be
analyzed at a minimum of two concentration levels, the high and low end of the determined method
performance range. If acceptable results are not obtained, intermediate concentrations are tested.
The range for the matrix is then defined from the lowest to the highest concentration that produced
acceptable results for precision and percent recovery when processed through the entire analytical
procedure. Matrix validation procedures are described in detail in Appendix C: Matrix Validation.
5.2.4 Ruggedness Testing (optional): Ruggedness testing is performed to determine method
sensitivity or insensitivity to deviations from method-specified conditions such as extraction
temperatures, solvent ratios, sorbent hydration, flow rates, etc. A ruggedness test is conducted to
show that reasonable and allowable differences in these conditions will not cause any important
change in analytical results. Methods which show extreme sensitivity to slight condition variations
may be unsuitable if such conditions are not easily controlled.
Whenever a method has the potential to be sensitive to changes in specified conditions, method
performance testing should be conducted under varied conditions. Specifically, the method range
and MDL, and matrix validations described previously should be reevaluated under the varied
conditions. Samples should be analyzed under varied conditions within a range of conditions that
might be encountered in normal application of the method. For example, if a sample extraction
procedure requires heating a sample at a specified temperature with a device of known variability,
the method's sensitivity to changes in temperature may be assessed by performing the extraction at
two temperatures - the high and low extremes of the heating device's known variation around the
desired extraction temperature. If method performance is shown to meet MQOs at both extremes,
the method is considered insensitive to expected variation in extraction temperature.
19
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 20 of 81
Some methods may require a more detailed approach to ruggedness testing. If the method has the
potential to show sensitivity to several method conditions, a carefully designed ruggedness test is
advised to examine the effects of variations of multiple conditions and combinations of conditions
simultaneously. A designed experiment will enable this simultaneous examination of several
combinations of conditions, while minimizing the number of required analyses. When a designed
experiment is needed, consult "Chemometrics: A Textbook" (Vandeginste, 1988) or the "Statistical
Manual of the A.O.A.C." (Youden, 1975) for proper procedures.
When method validation includes ruggedness testing, a method that is approved as an EMAP
method will be approved for use under all conditions where ruggedness testing results met MQOs.
When ruggedness testing is not performed, a method that is approved as an EMAP method will be
approved for use only under the exact conditions of the method validation.
5.3 Required Documentation
5.3.1 Method validations must be supported by data sufficient to allow an independent reviewer to
verify every measurement and calculation generated by the providing laboratory. The data must be
clearly labeled, compiled, and maintained in an organized manner.
5.3.2 The documentation required to support the EMAP Method Validation consists of the
following items:
• The EMAP Method Validation Checklist for Chemical Methods and any required attachments
(see Appendix A: EMAP Method Validation Checklist for Chemical Methods and subsection 5.4
- Checklist).
• Raw data that will allow an independent reviewer to verify each determination and calculation
performed by the laboratory. The documentation must support the tracing of data from sample
preparation and spiking through the final results reported. Because different methods will
employ different instrumentation or equipment to measure the parameter of interest, the specific
format of raw and calculated data may differ by method.
• The Certification Statement (see subsection 5.5 - Certification Statement).
5.3.3 In developing the PBMS, EPA has designed a checklist to aid the facility attempting to
employ a performance-based method in assessing and documenting specific aspects of the
performance-based method. EMAP-Methods has adapted this checklist for use in documenting and
evaluating candidate method performance as part of its method approval system. The checklist is
presented in Appendix A: EMAP Method Validation Checklist for Chemical Methods. The first page
of the checklist summarizes information about the conditions under which the validation was
conducted. The second page summarizes the analytical results obtained during validation. In
addition to the checklist, method performance data generated during a method validation must be
included in the written method in Section 13. Method Performance and in Section 17. Tables,
Diagrams, Flowcharts, and Validation Data when needed (see subsection 4.2 - EMMC Method
Format for descriptions of these method sections).
20
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 21 of 81
EMAP Method Validation
Method Range in a
Standard Matrix
(Appendix B)
1. Analyze 24 spiked
standard matrix samples
- eight at each of three
concentrations.
2. Determine
background level in
sample matrix by
analyzing three replicates
and comparing results to
standard matrix results.
3. If background
concentration is present
in sample matrix,
analyze five additional
replicates.
Figure3: The EMAP Method Validation
21
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 22 of 81
5.4 Checklist
5.4.1 Header information: The header information required on each page of the EMAP Method
Validation Checklist for Chemical Methods includes the following items:
• Date: The "date" entered on each page should be the date of submission of the information to
the Resource Group evaluating the method's performance.
• Facility Name: The "facility name" should be the name of the facility proposing the use of the
candidate method.
• Resource Group/Program: For EMAP's purposes, the "Resource Group/Program" is the
name of the Resource Group to whom the results will be reported.
• Matrix Type: The "matrix type" refers to the specific environmental sample matrices to which
the candidate method has been applied and for which approval is being sought. Because most
evaluations of a method will involve matrix-specific performance measures, a separate checklist
should be prepared for each matrix tested.
• Analyte or Class of Analytes: The last header field is used to indicate the analyte or class of
analytes to which the candidate method applies. When a method applies to a large number of
analytes, a class of analytes may be entered in this field. If such classification is used, a
separate list of analytes and their respective Chemical Abstract Service Registry Numbers must
be attached to the checklist.
5.4.2 With the exception of the certification statement, each page of the checklist also contains the
field "Page # of #" for consecutive pagination of the checklist.
5.4.3 Checklist Items: The two pages of the EMAP Method Validation Checklist for Chemical
Methods are used to summarize the conditions of validation testing and the results obtained. The
items on the checklist are consecutively numbered, as shown in the following descriptions, for easy
reference.
1. Written candidate method (EMMC format) attached?: The first item on the checklist
is a simple "yes/no" question regarding the required method write-up. The details of the
method must be written in the EMMC format as described in Section 4 - Method Content.
Conformance to the required method format is essential to EMAP's goal of achieving program
consistency.
2. Title and date of QAPP containing MQOs: This field identifies the QAPP that contains
the MQOs to which the performance specifications of the candidate method are being
compared.
22
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 23 of 81
3. Method range tested (w/units): This range should reflect the actual range of sample
concentrations that were tested and must include the concentration units. These concentration
units are typically expressed in terms of mass/mass or mass/volume. The range should not be
expressed in terms of the concentrations of the standards tested.
4. Linear working range (w/units): The linear working range is the range of concentrations
over which the analytical system exhibits a linear relationship between the amount of material
introduced into the instrument and the instrument's response. Linearity is typically measured
as either a linear regression for inorganic analytes or as the RSD of the response factors or
calibration factors for organic analytes. Traditional performance specifications consider any
regression line with a correlation coefficient (r) of 0.995 or greater as linear. For organic
analytes, an RSD of 25% or less is typically considered linear. Using these limits for linearity
as guidance, enter the method's linear working range in this field.
5. Concentrations of calibration standards w/units corresponding to final sample
concentration: The concentrations of the calibration standards are entered in the next field.
The number of calibration standards may vary for different methods.
6. Calibration curve attached?: The calibration curve is the graphical representation of the
instrument response versus the concentration of the calibration standards. For methods that use
linear regression or another statistical curve, the calibration curve must be included with any
data submission. For methods that use the RSD of the calibration results as a measure of
linearity, the curve is not needed if the RSD specifications are met. However, calibration
curves must be attached for methods where the instrument response is not sufficiently linear to
pass the RSD specifications.
7. Slope of calibration regression line: If a regression line is calculated, its equation should
be entered in this field.
8. Relative Standard Deviation (RSD) of calibration factors: If the RSD calculation is used
to determine a calibration's linearity, the result should be entered in this field.
9. Sample preservatives and holding times: If holding times have been evaluated, include
the conclusions of the evaluation in this field.
10. Interferences: In this field, enter information on any known or suspected interferences
with the candidate method. While interferences may be difficult to predict, they may be
indicated by unacceptable spike recoveries, especially when such interferences are not noted
when testing a clean matrix such as reagent water.
11. Qualitative identification criteria used: Qualitative identification criteria are crucial to
the accurate assessment of contaminants and will vary depending on the analytical technique
employed. Enter all relevant criteria used for target analyte identification. These criteria should
include items such as retention time, spectral wavelengths, co-elution of peaks for specific
m/z's, ion abundance ratio requirements, etc. If the list of criteria is lengthy, attach it to the
checklist on a separate sheet and enter "Attached" in this field.
23
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 24 of 81
12. Surrogates used: If surrogate compounds are added to samples prior to preparation,
which is typically done for organic analyses, indicate the specific surrogates used in the
candidate method.
13. Surrogate recovery limits established: Summarize the surrogate recovery limits that
have been established for the candidate method. For methods with multiple surrogates, include
a range of recovery limits, and attach a detailed list to the checklist. These recovery limits may
be derived from specific performance tests conducted during the EMAP Method Validation or
based on a scientifically defensible consensus of expected performance.
14. Method Range Determination
Method range in a standard matrix: Enter the range of concentrations in a standard
matrix for which the method was determined to perform reliably during method validation.
Standard matrix used: Enter the standard matrix used to determine the method range.
Spike levels: Enter the theoretical concentrations of analytes in the standard matrix after
spiking. The concentrations should be expressed in the same units in which the sample
results will be expressed, not in the concentration units of the spiking solution.
Source of spiking material: If reference materials obtained from commercial sources are
used, indicate the sources. If externally prepared materials are not available, spiking
materials must be prepared in the laboratory and this should be indicated on the checklist,
along with the source of all neat materials used.
Number of replicates: In this field, indicate the number of replicate sample analyses that
were used. The minimum number of replicates required for the method range determination
is sixteen - eight replicates at the high concentration and eight replicates at the low
concentration of the method range determined.
15. Matrix Validation
Sample matrix used: Enter the matrix used to determine the valid method range.
Spike levels: Enter the theoretical concentrations of analytes in the sample matrix after
spiking. The concentrations should be expressed in the same units in which the sample
results will be expressed, not in the concentration units of the spiking solution.
Source of spiking material: If reference materials obtained from commercial sources are
used, indicate the sources. If externally prepared materials are not available, spiking
materials must be prepared in the laboratory and this should be indicated on the checklist,
along with the source of all neat materials used.
24
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 25 of 81
Number of replicates: In this field, indicate the number of replicate sample analyses that
were used. The minimum number of replicates required for the matrix validation is five.
16. Performance Characteristics: These items summarize the method range, MDL,
precision, and percent recovery results for the candidate method. For multiple analyte methods,
it would be impractical to enter the results for each analyte in these fields. Therefore, for
multiple analyte methods, enter the range of results in the precision and percent recovery fields
and attach an analyte-specific summary to the checklist. In the left-hand column for each of
these criteria, enter the MQO obtained from the QAPP indicated on page 1 of the checklist. In
the right-hand column for each of these criteria, enter the results of validation testing for the
candidate method.
Method range determined: Enter the method range determined during matrix validation.
For multiple analyte methods, enter "Attached" and attach a full listing of results to the
checklist. If approval is being sought for more than one sample matrix, attach a full listing
of results for each selected matrix.
Method Detection Limit (MDL): Enter the MDL determined during validation, including
the number of replicates analyzed.
Precision: The precision statistics determined during matrix validation, at both the high and
low concentrations that define the method range, are entered in this field.
Percent Recovery: The percent recovery estimates determined during matrix validation, at
both the high and low concentrations which define the method range, are entered in this field.
5.5 Certification Statement
Appendix A: EMAP Method Validation Checklist for Chemical Methods contains the certification
statement required to be documented with every EMAP Method Validation. This statement is
intended to ensure that the laboratory or organization that provided and tested the candidate method
assumes full responsibility for method performance and for the data provided on the checklist.
5.6 Evaluation of Method Performance
5.6.1 Evaluation of the EMAP Method Validation will consist of ensuring that the required
documentation is in place and complete and that the candidate method meets or exceeds the MQOs
specified in the appropriate EMAP QAPP. Laboratories proposing candidate methods for approval
must perform all required performance testing, document the results, and submit the documentation
to the appropriate Resource Group. The Resource Group is then responsible for evaluating the
documentation and determining the approval status of the proposed candidate method.
5.6.2 Incomplete documentation will not be evaluated for approval. Whenever required information
is missing or incomplete, the providing laboratory or organization will be required to complete the
information and resubmit the completed documentation for approval. Once the documentation has
25
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 26 of 81
been determined to be complete, method evaluators will assess the method's suitability for EMAP's
needs.
5.6.3 With documentation requirements met, method assessors may evaluate the method by using
the documentation to answer the following questions.
Is the method specific (or selective) for the analyte? Evaluation of the analysis of spikes
and standards, along with noted interferences, will enable the method assessor to determine the
method's selectivity. If the method is not specific for the analyte, it will not be approved in its
current form. The method may be revised and retested if desired. Otherwise, the method is not
approved for EMAP use.
Do the method's precision and percent recovery results meet or exceed MQOs? The
results of the EMAP Method Validation are available for this evaluation. If the precision and
percent recovery estimates determined during performance testing do not meet or exceed
MQOs, the method will not be approved in its current form. The method may be revised and
retested if desired. Otherwise, the method is not approved for EMAP use.
Does the MDL meet or exceed MQOs? If the MDL of the candidate method does not meet
or exceed the MDL requirement specified in the QAPP, the method will not be approved in its
current form. The method may be revised and retested if desired. Otherwise, the method is not
approved for EMAP use.
Is the method valid over the range of conditions expected in normal use? As noted in
subsection 5.2 - EMAP Method Validation, conducting ruggedness testing is an optional aspect
of the EMAP Method Validation. If ruggedness testing is performed and all other requirements
for method approval have been met, the method will be approved for use under all conditions
for which the method was tested and determined to perform acceptably. If ruggedness testing
is not performed, the method will be approved for use only for the conditions under which the
validation was conducted.
5.6.4 Multi-Analyte Methods: For methods that measure more than one parameter, method
approval will be more complicated than with single parameter methods. For example, an organic
method that measures pesticides will measure a number of compounds in a single analysis. The
EMAP Method Validation and the EMAP Demonstration of Method Comparability will involve
characterizing the method's performance for each compound measured. Likewise, method
evaluation will involve evaluating the method's performance for each compound measured. In
multiple parameter methods, the possibility exists that the performance statistics of some of the
method's analytes will meet requirements, while others do not.
Whenever the performance statistics of some fraction of a method's analytes fail to meet MQOs, any
of three potential decisions may be made regarding approval of the method for EMAP use. The
method may be approved for use in measuring only those parameters whose performance statistics
met requirements; the method may be approved "as is"; or the method may be rejected. Criteria for
making this determination are not addressed in this guidance. Method acceptance criteria fall within
26
-------
EMAP Chemical Method Evaluation Guidance, Section 5, February 1996, Page 27 of 81
the responsibilities of the Resource Groups. Along with MQOs, specific method acceptance criteria
should be defined by the Resource Group and included in the appropriate QAPP.
27
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 28 of 81
SECTION 6
EMAP DEMONSTRATION OF METHOD COMPARABILITY
6.1 Summary
6.1.1 For most measurements needed for EMAP, methods are available that have either been
validated by EPA or have already been widely used within EMAP. As described in Section 4 -
Method Content, when these methods are completely documented and evaluated with respect to
MQOs, they will become approved EMAP methods. With approved methods available for
performing EMAP measurements, the Resource Groups will be able to recommend their use when
seeking laboratories to perform chemical analyses. Utilizing a PBMS, the laboratories may then
adopt the recommended method or may alternatively propose to use a modified version of the
method or an entirely new method to perform the needed measurement.
6.1.2 When an alternate or modified method is proposed for use in performing EMAP
measurements, the laboratory proposing the new candidate method must demonstrate and document
the candidate method's performance using one of two procedures. The laboratory may choose to
determine the method's precision, percent recovery, method range, and MDL by performing the
EMAP Method Validation as described in Section 5 - EMAP Method Validation. Alternatively, the
laboratory may perform the EMAP Demonstration of Method Comparability to characterize method
performance.
6.1.3 The EMAP Demonstration of Method Comparability is performed to compare the
performance statistics of a new candidate method to the performance statistics of an approved
EMAP method. The EMAP Demonstration of Method Comparability is adequate for method
assessment only when an approved EMAP method is available for measuring the parameter of
interest. For an alternate method to gain approval, the candidate method must be written in the
EMMC method format and the EMAP Demonstration of Method Comparability must be
successfully completed.
6.2 EMAP Demonstration of Method Comparability
6.2.1 The procedures specified for the EMAP Demonstration of Method Comparability have been
adapted from the "Guidance on the Evaluation of Safe Drinking Water Act Compliance Monitoring
Results from Performance-Based Methods" (USEPA, 1994). Much of the first chapter of that
document is repeated here, slightly modified for EMAP's use. The analyses required for the EMAP
Demonstration of Method Comparability include replicate spike analyses and MDL determination.
Figure 4: The EMAP Demonstration of Method Comparability (p.30) contains a flowchart which
summarizes the sample analyses required of these procedures. Appendix C: Matrix Validation
provides detailed instructions for spiking and analyzing replicate matrix spike samples. Appendix
B: Method Range and Method Detection Limit describes the MDL determination. Method users
should refer to these appendices when performing the EMAP Demonstration of Method
Comparability.
6.2.2 Replicate Spike Analyses: To demonstrate comparability, at least five aliquots of a spiked
sample matrix are processed through the entire analytical procedure. The concentration of each
28
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 29 of 81
analyte is measured and the results are used to evaluate the method's precision and percent recovery.
• Matrix: Replicate analyses are required to be performed in at least two matrices: a reference
matrix and in each sample matrix for which the method is intended. The reference matrix will
vary depending upon the analysis performed and the sample matrix. A variety of "clean"
matrices such as reagent water, clean sand, sediment, or soil may be used during performance
testing. The results of the analyses of clean matrices, when compared to those of sample
matrices, may yield important information about method interferences. The sample matrix
evaluated must be the same matrix as the samples to which the method will routinely be
applied.
• Spike levels: For most analytical techniques, the precision and percent recovery of the
measurement technique are directly related to concentration of the analyte. As the
concentration of the analyte increases, the absolute error of the measured concentration will
increase, while the relative error will decrease. Therefore, the choice of the concentration at
which to demonstrate performance is critical. Use the approved EMAP method as a guide in
choosing the concentration at which to spike the replicate aliquots for the comparability study.
The spiking level must be no higher than the spiking level in the approved method. If the
approved method does not specify a concentration level for spiking, spike the sample matrix
at one to five times the MDL for each analyte of interest. If the sample matrix is known to
contain the analytes of interest at appropriate levels (one to five times the MDL), spiking may
not be required.
• Source of spiking material: Whenever possible, it is preferable to use a spiking solution or
material prepared by or obtained from an external source. Reference materials for various
analytes are available from a number of commercial sources. If externally prepared materials
are not available, spiking materials must be prepared in the laboratory.
• Number of replicates: The minimum number of replicate aliquots required for the EMAP
Demonstration of Method Comparability is five.
• Precision: The precision is calculated as the RSD of the replicate measurements, using («-/)
degrees of freedom.
• Percent recovery: The percent recovery is calculated as the ratio of the measured value to the
true value of the spiked analyte, multiplied by 100.
6.2.3 Method Detection Limit: The MDL is determined by analyzing at least seven replicate
samples, spiked at a concentration at or near an estimated limit of detection. MDLs are analyzed
over time and/or across sample batches to include long-term variation in the estimate. Appendix B:
Method Range and Method Detection Limit contains detailed instructions for determining the MDL.
29
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 30 of 81
EMAP Demonstration of Method Comparability
Method Detection
Limit
(Appendix B)
1. Calculate the MDL
using the standard deviation
of recoveries determined at
the low concentration
defining the method range
2. If calculated MDL is
>10 times lower than the
concentration analyzed,
analyze new samples at a
lower spiked
concentration.
Matrix Validation
(Appendix C)
1. Analyze two standard
matrix replicates to
determine the background
concentration of the
analytical system.
2. Determine background
concentration in sample
matrix by analyzing three
sample matrix replicates
and comparing results to
standard matrix results.
3. If background
concentration is present in
sample matrix, analyze five
additional replicates.
4. Spike and analyze five
sample matrix replicates at
a concentration of one to
five times the MDL.
5. Calculate precision and
percent recovery statistics
and compare to
performance statistics of
approved EMAP method.
Figure 4: The EMAP Demonstration of Method Comparability
6.3 Required Documentation
6.3.1 Demonstrations of method comparability must be supported by data sufficient to allow an
independent reviewer to verify every measurement and calculation generated by the laboratory. In
30
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 31 of 81
a PBMS, the data must be clearly labeled, compiled, and maintained in an organized manner and
available for inspection at the facility.
6.3.2 The documentation required to support the EMAP Demonstration of Method Comparability
consists of the following items:
• The EMAP Demonstration of Method Comparability Checklist for Chemical Methods and any
required attachments (see Appendix F: EMAP Demonstration of Method Comparability
Checklist for Chemical Methods and subsection 6.4 - Checklist).
• A summary of all quality control results required by the approved EMAP method and the
candidate method.
• Raw data that will allow an independent reviewer to verify each determination and calculation
performed by the laboratory. The documentation must support the tracing of data from sample
preparation and spiking through the final results reported. Because different methods will
employ different instrumentation or equipment to measure the parameter of interest, the specific
format of raw and calculated data may differ by method.
• The Certification Statement (see subsection 6.5 - Certification Statement}.
6.3.3 The evaluation checklist developed for documenting the EMAP Demonstration of Method
Comparability is very similar to the list adapted for use with the EMAP Method Validation. As with
the latter, the former checklist is intended to aid method evaluators and other users in assessing
method performance and its documentation. The checklist for the EMAP Demonstration of Method
Comparability is presented in Appendix F: EMAP Demonstration of Method Comparability
Checklist for Chemical Methods. The first page of the checklist summarizes information about the
conditions under which the comparability testing was performed. The second page of the checklist
summarizes the analytical results obtained during performance testing.
6.4 Checklist
6.4.1 Header information: The header information required on each page of the EMAP
Demonstration of Method Comparability Checklist for Chemical Methods includes the following:
• Date: The "date" entered on each page should be the date of submission of the information to
the Resource Group evaluating the method's performance.
• Facility Name: The "facility name" should be the name of the facility proposing the use of the
candidate method.
• Resource Group/Program: For EMAP's purposes, the "Resource Group/Program" is the
name of the Resource Group to whom the results will be reported.
• Matrix Type: The "matrix type" refers to the specific environmental sample matrices to which
the candidate method has been applied and for which approval is being sought. Because most
31
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 32 of 81
evaluations of a method will involve matrix-specific performance measures, a separate checklist
should be prepared for each matrix tested.
• Analyte or Class of Analytes: The last header field is used to indicate the analyte or class of
analytes to which the candidate method applies. When a method applies to a large number of
analytes, a class of analytes may be entered in this field. If such classification is used, a
separate list of analytes and their respective Chemical Abstract Service Registry Numbers must
be attached to the checklist.
6.4.2 With the exception of the certification statement, each page of the checklist also contains the
field "Page # of #" for consecutive pagination of the checklist for submission.
6.4.3 Checklist Items: The two pages of the EMAP Demonstration of Method Comparability
Checklist for Chemical Methods are used to summarize the condition of comparability testing and
the analytical results obtained. The items on the checklist are consecutively numbered, as shown
in the following descriptions, for easy reference.
1. Written candidate method (EMMC format) attached?: The first item on the checklist
is a simple "yes/no" question regarding the required method write-up. The details of the
method must be written in the EMMC format as described in Section 4 - Method Content.
Conformance to the required method format is essential to EMAP's goal of achieving program
consistency.
2. Title and date of approved EMAP method: This field identifies the accepted EMAP
method to which the performance-based method is being compared.
3. Copy of approved EMAP method maintained on-site?: This item is another "yes/no"
question regarding required paperwork. Although a copy of the approved EMAP method is not
required to be attached to the checklist submitted for approval, a copy should be maintained at
the laboratory.
4. Listing of differences between candidate method and approved EMAP method
attached?: The submission for evaluation must include a side-by-side listing of the differences
between the candidate method and the approved EMAP method. The listing should contain
each section of the EMMC method format of the approved EMAP method and the candidate
method. The left-hand column will list pertinent aspects of the approved EMAP method, while
the right-hand column will list pertinent aspects of the candidate method. The comparison
should focus on differences between the two procedures in technique, not minor variations in
the glassware used or the concentrations of specific reagents. The listing should highlight any
differences between method performance data. All method performance information must be
included in the comparison in the method sections for Scope and Application, Interferences,
Quality Control, and Method Performance.
5. Performance range tested (w/units): This range should reflect the actual range of sample
concentrations that were tested and must include the concentration units. These concentration
units are typically expressed in terms of mass/mass or mass/volume. The range should not be
expressed in terms of the concentrations of the standards tested.
32
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 33 of 81
6. Linear working range (w/units): The linear working range is the range of concentrations
over which the analytical system exhibits a linear relationship between the amount of material
introduced into the instrument and the instrument's response. Linearity is typically measured
as either a linear regression for inorganic analytes or as the RSD of the response factors or
calibration factors for organic analytes. Traditional performance specifications consider any
regression line with a correlation coefficient (r) of 0.995 or greater as linear. For organic
analytes, an RSD of 25% or less is typically considered linear. Using these limits for linearity
as guidance, enter the method's linear working range in this field.
7. Concentrations of calibration standards w/units corresponding to final sample
concentration: The concentrations of the calibration standards are entered in the next field.
As noted on the checklist, the concentration of the lowest standard must be one to four times
the MDL. The number of calibration standards may vary for different methods. Under the
PBMS, the concentrations of the calibration standards must bracket the sample concentrations.
No sample result may be reported that is above the calibration range of the method unless that
sample or extract has been diluted for analysis. Similarly, no result may be reported that is
below the calibration range specified in the method.
8. Calibration curve attached?: The calibration curve is the graphical representation of the
instrument response versus the concentration of the calibration standards. For methods that use
linear regression or another statistical curve, the calibration curve must be included with any
data submission. For methods that use the RSD of the calibration results as a measure of
linearity, the curve is not needed if the RSD specifications are met. However, calibration
curves must be attached for methods where the instrument response is not sufficiently linear to
pass the RSD specifications.
9. Slope of calibration regression line: If a regression line is calculated, its equation should
be entered in this field.
10. Relative Standard Deviation (RSD) of calibration factors: If the RSD calculation is
used to determine a calibration's linearity, the result should be entered in this field.
11. Sample preservatives and holding times: If holding times have been evaluated, include
the conclusions of the evaluation in this field.
12. Interferences: In this field, enter information on any known or suspected interferences
with the candidate method. While interferences may be difficult to predict, they may be
indicated by unacceptable spike recoveries, especially when such interferences are not noted
when testing a clean matrix such as reagent water. It may also be useful to indicate whether
known interferences with the approved EMAP method are not interferences with the candidate
method, and vice versa.
13. Qualitative identification criteria used: Qualitative identification criteria are crucial to
the accurate assessment of contaminants and will vary depending on the analytical technique
employed. Enter all relevant criteria used for target analyte identification. These criteria should
include items such as retention time, spectral wavelengths, co-elution of peaks for specific
m/z's, ion abundance ratio requirements, etc. The accepted EMAP method should be used as
33
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 34 of 81
guidance when specifying identification criteria if the instrumental techniques are similar. If
the list of criteria is lengthy, attach it to the checklist on a separate sheet and enter "Attached"
in this field.
14. Surrogates used: If surrogate compounds are added to samples prior to preparation,
which is typically done for organic analyses, indicate the specific surrogates used in the
candidate method.
15. Surrogate recovery limits established: Summarize the surrogate recovery limits that
have been established for the candidate method. For methods with multiple surrogates, include
a range of recovery limits, and attach a detailed list to the checklist. These recovery limits may
be derived from specific performance tests performed during the EMAP Demonstration of
Method Comparability, adopted from the approved EMAP method, or based on a scientifically
defensible consensus of expected performance.
16. Testing conditions:
Matrix used: Enter the matrix used to perform comparability testing.
Spike levels: When the standard matrix aliquots are spiked, enter the theoretical
concentrations of analytes after spiking. The concentrations should be expressed in the same
units in which the sample results will be expressed, not in the concentration units of the
spiking solution. If spiking is not required, enter that information in this field. Include the
background concentrations of the analytes found in the standard matrix.
Source of spiking material: If reference materials obtained from commercial sources are
used, indicate the sources. If externally prepared materials are not available, spiking
materials must be prepared in the laboratory and this should be indicated on the checklist,
along with the source of all neat materials used.
Number of replicates: In this field, indicate the number of replicates that were used. The
minimum number of replicates required is five.
17. Performance Characteristics: The last three items on the checklist summarize the
precision, percent recovery, and MDL results for the candidate method. For multiple analyte
methods, it would be impractical to enter the results for each analyte in these fields. Therefore,
for multiple analyte methods, enter the range of results in the precision and percent recovery
fields, and attach an analyte-specific summary to the checklist. In the left-hand column for each
of these criteria, enter the performance specification from the accepted EMAP method. In the
right-hand column, enter the results of performance testing for the candidate method.
Precision: The precision statistics determined from the analysis of replicate aliquots are
entered in this field.
Percent Recovery: The percent recovery estimates determined from the analysis of
replicate aliquots are entered in this field.
34
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 35 of 81
Method Detection Limit (MDL): The MDL determined for each method analyte is entered
in this field.
6.5 Certification Statement
Appendix F: EMAP Demonstration of Method Comparability Checklist for Chemical Methods
contains the certification statement required to be documented with every EMAP Demonstration of
Method Comparability. This statement is intended to ensure that the laboratory or organization that
provided and tested the candidate method assumes full responsibility for method performance and
for the data provided on the checklist.
6.6 Evaluation of Method Performance
6.6.1 Evaluation of the EMAP Demonstration of Method Comparability will consist of ensuring that
the required documentation is in place and complete and that the candidate method meets or exceeds
the performance specifications of the approved EMAP method. Laboratories proposing candidate
methods for approval must perform all required performance testing, document the results, and
submit the documentation to the appropriate Resource Group. The Resource Group is then
responsible for evaluating the documentation and determining the approval status of the proposed
candidate method.
6.6.2 Incomplete documentation will not be evaluated for approval. Whenever required information
is missing or incomplete, the providing laboratory or organization will be required to complete the
information and resubmit the completed documentation for approval. Once the documentation has
been determined to be complete, method evaluators will assess the method's suitability for EMAP's
needs.
6.6.3 With documentation requirements met, method assessors may evaluate the method by using
the documentation to answer the following three questions.
Is the method specific (or selective) for the analyte? Evaluation of the analysis of spikes
and standards, along with noted interferences, will enable the method assessor to determine the
method's selectivity. If the method is not specific for the analyte, it will not be approved in its
current form. The method may be revised and retested if desired. Otherwise, the method is not
approved for EMAP use.
Do the method's precision and percent recovery results meet or exceed those of the
approved EMAP method? The results of the EMAP Demonstration of Comparability are
available for this evaluation. If the precision and percent recovery estimates determined during
performance testing do not meet or exceed those of the approved EMAP method, the method
will not be approved in its current form. The method may be revised and retested if desired.
Otherwise, the method is not approved for EMAP use.
Does the MDL meet or exceed the MDL specifications of the approved EMAP method?
If the MDL of the candidate method does not meet or exceed the MDL specifications of the
35
-------
EMAP Chemical Method Evaluation Guidance, Section 6, February 1996, Page 36 of 81
approved EMAP method, the method will not be approved in its current form. The method may
be revised and retested if desired. Otherwise, the method is not approved for EMAP use.
6.6.4 At the discretion of the Resource Group, method assessors may find it beneficial to consider
a number of additional Quality Control (QC) checks when evaluating candidate methods. The
following list provides examples of the types of checks that may be useful:
Verification of linear working range
Calibration verification
External QC samples
Performance evaluation studies
Surrogate recoveries
6.6.5 Multi-Analyte Methods: As noted in subsection 5.6.4, for methods that measure more than
one parameter, method approval will be more complicated than with single parameter methods. The
EMAP Method Validation and the EMAP Demonstration of Method Comparability will involve
characterizing the method's performance for each compound measured. Likewise, method
evaluation will involve evaluating the method's performance for each compound measured. In
multiple parameter methodds, the possibility exists that the performance statistics of some of the
method's analytes will meet requirements, while others do not.
Whenever the performance statistics of some fraction of a method's analytes fail to meet
performance objectives, any of three potential decisions may be made regarding approval of the
method for EMAP use. The method may be approved for use in measuring only those parameters
whose performance statistics met requirements; the method may be approved "as is"; or the method
may be rejected. Criteria for making this determination are not addressed in this guidance. Method
acceptance criteria fall within the responsibilities of the Resource Groups. Along with MQOs,
specific method acceptance criteria should be defined by the Resource Group and included in the
appropriate QAPP.
36
-------
EMAP Chemical Method Evaluation Guidance, Section 7, February 1996, Page 37 of 81
SECTION 7
REFERENCES
Chaloud, D.J. and Peck, D.V. (eds.), 1994. Environmental Monitoring and Assessment Program:
Integrated Quality Assurance Program Plan for the Surface Waters Resource Group (1994
Activities). EPA 600/X-91/080 (Revision 2.00). U.S. Environmental Protection Agency, Las
Vegas, Nevada.
Massart, D.L., Vandeginste, B.G.M., Deming, S.N., Michotte, Y., and Kaufman, L., 1988.
Chemometrics: A Textbook. Elseveir, New York.
Taylor, J.J., 1987. Quality Assurance for Chemical Measurements. Lewis Publishers, Inc., Chelsea,
Michigan.
U.S. EPA, 1985. Single-Laboratory Method Validation Protocol. Contract Number 68-03-3224.
U.S. Environmental Protection Agency, Environmental Monitoring Systems Laboratory,
Cincinnati, Ohio (This document was not published.).
U.S. EPA, 1987. Data Standards for the Electronic Transmission of Laboratory Measurement
Results. EPA/IRM-87/21802. U.S. Environmental Protection Agency, Environmental
Monitoring Systems Laboratory, Cincinnati, Ohio.
U.S. EPA, 1993. Requirements for Approval of Alternate Test Procedures for Inorganic and
Organic Analytes in National Pollutant Discharge Elimination System Monitoring. Revision
1.2. U.S. Environmental Protection Agency, Environmental Monitoring Systems Laboratory,
Cincinnati, Ohio.
U.S. EPA, 1994. Guidance on the Evaluation of Safe Drinking Water Act Compliance Monitoring
Results from Performance-Based Methods. EMMC Performance-Based Methods Work Group,
U.S. Environmental Protection Agency, Environmental Monitoring Systems Laboratory,
Cincinnati, Ohio (This document is a work in progress.).
U.S. EPA, 1995. Environmental Monitoring and Assessment Program (EMAP) Methods Format
Guidance. EPA/620/R-95/001. U.S. Environmental Protection Agency, Environmental
Monitoring Systems Laboratory, Cincinnati, Ohio.
Youden, W.J., and Steiner, E.H., 1975. Statistical Manual of the A.O.A.C.. Assocation of Official
Analytical Chemists, Washington, D.C.
37
-------
APPENDICES
Page
Appendix A: EMAP Method Validation Checklist for Chemical Methods 39
Appendix B: Method Range and Method Detection Limit 42
Appendix C: Matrix Validation 46
Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals 52
Appendix E: Statistical Example 67
Appendix F: EMAP Demonstration of Method Comparability Checklist for
Chemical Methods 79
38
-------
EMAP Chemical Method Evaluation Guidance, Appendix A, February 1996, Page 39 of 81
Appendix A: EMAP Method Validation Checklist for Chemical Methods
Date:
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
Page _ of _
Conditions
1 . Written candidate method (EMMC format)
attached?
2. Title and date of QAPP containing MQOs
3. Method range tested (w/units)
4. Linear working range (w/units)
5. Concentrations of calibration standards w/units
corresponding to final sample concentration (lowest
standard must be one to four times MDL)
6. Calibration curve attached?
7. Slope of calibration regression line (if applicable)
8. Relative Standard Deviation (RSD) of calibration
factors (if applicable)
9. Sample preservatives and holding times
10. Interferences
1 1 . Qualitative identification criteria used
12. Surrogates used (if applicable)
13. Surrogate recovery limits established
Measurement Quality
Objectives (MQOs)
—
Summary
of Results
Y/N
Y/N
Y/N
39
-------
EMAP Chemical Method Evaluation Guidance, Appendix A, February 1996, Page 40 of 81
EMAP Method Validation Checklist for Chemical Methods
Date:
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
Page _ of _
Analytical Results*
14. Method Range Determination
Method range in a standard matrix
Standard matrix used
Spike levels (w/units corresponding to final
sample concentration)
Source of spiking material
Number of replicates (minimum n=8)
15. Matrix Validation
Sample matrix used
Spike levels (w/units corresponding to final
sample concentration)
Source of spiking material
Number of replicates (minimum n=5)
16. Performance Characteristics"
Method range determined (in the sample
matrix)
Method detection limit (MDL) (w/units and
number of replicate samples analyzed)
Precision (RSD [n-1] of replicate
measurements)
Percent Recovery
Measurement Quality
Objectives (MQOs)
Validation Results
Complete one page for each matrix tested.
For multiple analyte methods, enter the range of results obtained, and attach a full listing of
results to the checklist.
40
-------
EMAP Chemical Method Evaluation Guidance, Appendix A, February 1996, Page 41 of 81
EMAP Method Validation
Certification Statement*
Date: Page _ of _
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
We, the undersigned, CERTIFY that:
1. The candidate method provided for EMAP use has undergone the EMAP Method Validation (or
the "Single-Laboratory Method Validation Protocol" (USEPA, 1985)) and has met the MQOs of the
approved QAPP.
2. A copy of the candidate method, written in the EMMC format, is attached to this checklist and
is available on-site.
3. The data associated with the EMAP Method Validation are complete (including the mandatory
copy of this checklist), all raw data necessary to reconstruct and verify the analyses have been
retained, and the associated information is available for review.
Name and Title Signature Date
Name and Title Signature Date
Name and Title Signature Date
Name and Title Signature Date
'This certification form must be completed when the EMAP Method Validation is performed to
approve a candidate method for EMAP use.
41
-------
EMAP Chemical Method Evaluation Guidance, Appendix B, February 1996, Page 42 of 81
Appendix B: Method Range and Method Detection Limit
The method range is the concentration range over which the method is valid. In actual application
of the method, the method range is used to determine when sample dilution is required or when a
smaller sample size should be used for analysis. The Method Detection Limit (MDL) is defined as
the minimum concentration of a substance that can be identified, measured, and reported with 99
percent confidence that the analyte concentration is greater than zero. The MDL is used to judge
the significance of a single measurement in a future sample.
To determine the method range, eight standard matrix samples, spiked at each of three
concentrations spread over the instrumentation range, are processed through the entire analytical
procedure. The results are compared to the precision and percent recovery performance
requirements to determine the concentration range over which the method is valid. For each method
analyte, both an upper limit and a lower limit for the method range are determined. The lower limits
are then used to calculate the MDL for each analyte.
1. Procedure
1.1 Determine the three spike concentrations for each method analyte.
1.1.1 The low spike concentration (L) should be one of the following, whichever is higher:
(a) The concentration value that corresponds to the lower limit of the instrumentation range.
(b) The concentration value that corresponds to a signal-to-noise ratio in the range of 2.5 to 5.
If the criteria for qualitative identification of the analyte are based upon pattern recognition
techniques, the least abundant signal necessary to achieve identification must be considered in
making the estimate.
(c) The concentration value that corresponds to three times the standard deviation of replicate
instrumental measurements for the analyte in reagent water.
(d) The concentration value that corresponds to the region of the standard curve where there
is a significant change in sensitivity at low analyte concentration (i.e., a break in the slope of
the standard curve).
1.1.2 The high spike concentration (H) should be the concentration that defines the upper limit of
the instrument range.
1.1.3 The middle spike concentration (M) should be the geometric mean of the low and high spike
concentrations as calculated from the following equation:
M = (H x L)
1/2
42
-------
EMAP Chemical Method Evaluation Guidance, Appendix B, February 1996, Page 43 of 81
1.2 Spike eight standard matrix samples with each method analyte at each of the three spike
concentrations.
1.3 Process the spiked samples through the entire analytical procedure in random order. Make all
calibrations and computations according to the procedure and express the concentrations found in
the procedure reporting units.
2. Evaluation of Results
2.1 Test the concentrations found for outliers.
2.1.1 For each analyte, test the concentrations found at each of the three spike concentrations for
outliers at the 1 percent significance level (see Appendix D: Statistics for Determining Outliers,
Significance, and Confidence Intervals). Check each data point declared an outlier for calculation
and/or recording errors. Eliminate any outlying data point.
2.1.2 If the results from step 2.1.1 produce less than six data points for any analyte-concentration
combination, analyze new standards as needed to assure a minimum of six data points for each
analyte at each concentration.
2.1.3 If new standards were analyzed in step 2.1.2, return to step 2.1.1. Otherwise, proceed to step
2.2.
NOTE: The completion of step 2.1 will result in a minimum of six data points for each analyte-
concentration combination. Because the outlier detection procedure eliminates data on an analyte-
by-analyte basis and the analysis of new standards produces data for all analytes, different numbers
of data points may be obtained for the three concentrations for some analytes.
2.2 Calculate standard deviation statistics and compare them to the precision performance
requirement for the method.
2.2.1 Calculate the standard deviation (s) and the RSD of the method for each analyte-concentration
combination.
2.2.2 Calculate a 100 (1-.01/WJ percent confidence interval for the RSD of each analyte-
concentration combination, where Na is the number of analytes for which the method is being
validated (Appendix D: Statistics for Determining Outliers, Significance, and Confidence Intervals).
If any of the lower limits of the confidence intervals calculated are greater than the precision
performance requirement, the method has failed to meet the precision performance requirement for
that analyte-concentration combination.
2.3 Calculate percent recovery statistics and compare them to the performance requirements.
2.3.1 Calculate the percent recovery of the method for each analyte-concentration combination.
2.3.2 For each analyte-concentration combination, calculate a 100 (1-.01/AT) percent confidence
interval for the true percent recovery of the method where Na is the number of analytes for which
43
-------
EMAP Chemical Method Evaluation Guidance, Appendix B, February 1996, Page 44 of 81
the method is being validated (Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals). If any of the upper limits of the confidence intervals are less than the lower
performance requirement or if any of the lower limits of the confidence intervals are greater than
the upper performance requirement, the method has failed to meet the percent recovery performance
requirement of the corresponding analyte-concentration combination.
2.4 Determine the range of the method for each method analyte. The upper limit of the method
range for an analyte is the highest analyte concentration that met both the precision and percent
recovery performance requirements. The lower limit of the method range for an analyte is the
lowest analyte concentration tested such that all concentrations included in the method range passed
both the precision and percent recovery performance requirements. If the upper and lower limits
are equal for a single analyte, then the method has failed for that analyte and the analyte should be
eliminated from the method.
2.5 Determine the MDL for each analyte.
2.5.1 Compute the MDL as follows:
MDL = tn.1>0.01(s)
where tn.1001 = the upper first percentile point of the Student's t distribution with n-1 degrees of
freedom (see Table B-l: Upper First Percentile Points of the Student's t Distribution), s — the
standard deviation of the concentrations found for the analyte at the spike concentration that defines
the lower limit of the method range as determined in step 2.4. n = the number of replicate analyses
from which the standard deviation is calculated.
2.5.2 If 10 times the calculated MDL is less than the spike concentration from which the standard
deviation for the MDL calculation was obtained, the calculated MDL is not reasonable. In this case,
a lower spike concentration should be selected for analysis. The procedures outlined in steps 1.1
through 1.3 should be repeated using the new spike concentration.
2.5.3 Compute the 95% confidence limits for the MDL obtained as shown in Appendix D: Statistics
for Determining Outliers, Significance, and Confidence Intervals.
2.5.4 Modify the written method to include the calculated MDL values. The standard matrix used
to determine the MDL must be identified with the MDL value. In addition, report the analyte
concentration used to calculate the MDL and the percent recovery.
44
-------
EMAP Chemical Method Evaluation Guidance, Appendix B, February 1996, Page 45 of 81
Table B-l: Upper First Percentile Points of the Student's t Distribution
Degrees of
Freedom tn.10M
(n-1)
5 3.365
6 3.143
7 2.998
8 2.896
9 2.821
10 2.764
11 2.718
12 2.681
13 2.650
14 2.624
45
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 46 of 81
Appendix C: Matrix Validation
Matrix validation consists of a determination of the method precision, percent recovery, and range
for a minimum of two matrices - the standard matrix and a representative sample matrix. The
number of matrices to be tested will depend upon the end use of the method. The method must be
validated for a standard matrix and for each sample matrix for which it will be used. Although it
is not possible to list appropriate matrix types for all methods for which validation may be required,
some representative sample matrices are provided in Table C-l: Representative Sample Matrices
for Matrix Validation along with the relevant standard matrix. In all cases, the matrices selected for
validation are assumed to be homogenous.
A validation for a single sample matrix consists of a determination of analyte background levels and
the analysis of five matrix samples at each of two spike concentrations. The results obtained are
statistically compared to the results obtained for the standard matrix during method range
determination. Finally, the results obtained are compared to the precision and percent recovery
performance requirements for the method.
Spike concentrations to be used for matrix validation are the high and low concentrations which
define the method range for the standard matrix. If acceptable results are not obtained at these
concentrations, an intermediate spike concentration is tested. The method range for a matrix is then
defined to be the range from the lowest spike concentration that produced acceptable results to the
highest spike concentration that produced acceptable results for precision and percent recovery.
Table C-l: Representative Sample Matrices for Matrix Validation
Method
Application
Standard
Matrix
Appropriate Representative
Sample Matrix
Drinking water
Wastewater
Sewage sludge
Reagent water
Reagent water
Wet (H2O) diatomaceous
earth
Drinking water
POTW aqueous outfall
Industrial wastewater
Surface water (lake, river)
POTW sludge of >2 suppliers
Sediments
Organic Liquids
Solid Waste
Toluene, hexane, MeCl,
Wet (H2O or organic solvent)
diatomaceous earth
Organic waste stream
Waste solvents
Soil
Stillbottoms from manufacturer
46
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 47 of 81
1. Procedure
1.1 Select the sample matrices to be used in the method validation. Obtain samples of each selected
matrix.
1.2 For each sample matrix, determine if a background level of any method analyte exists.
1.2.1 Analyze two method blank samples consisting of the standard matrix that have been processed
through the entire analytical procedure to establish the background level of the analytical system.
1.2.2 Process three replicates of the sample matrix through the entire analytical procedure. Any
response above the response obtained for the method blank samples for any method analyte
indicates that a background level of that analyte is present in the sample matrix.
NOTE: The analyses conducted to determine the background concentration of method analytes will
indicate other instances when a matrix is inappropriate for the validation. For example, a matrix that
interferes with the detection device employed in the method or that will not allow the
chromatographic system to return to baseline conditions within a reasonable length of time would
be considered inappropriate. The experience of the analyst is important in determining the
appropriateness of a matrix.
1.2.3 If a sample matrix contains a background level of any method analyte, process five additional
samples of that matrix through the entire analytical procedure and proceed to step 1.3. Otherwise,
proceed to step 1.5.
1.3 Test the concentrations found for outliers.
1.3.1 For each matrix-analyte combination in which a background level was detected, test the eight
concentrations found for outliers at the 1 percent significance level (Appendix D: Statistics for
Determining Outliers, Significance, and Confidence Intervals). Check each data point declared an
outlier for calculation and/or recording errors. Correct any errors and retest for outliers. Eliminate
any outlying data points.
1.3.2 If step 1.3.1 results in less than five data points for any matrix-analyte combination in which
a background level was detected, analyze new matrix samples as needed to assure a minimum of
five data points for each matrix-analyte combination.
1.3.3 If new samples were analyzed in step 1.3.2, return to step 1.3.1. Otherwise, proceed to step
1.4.
1.4 For each sample matrix, determine the mean background concentration of each analyte.
1.4.1 If no response was found for an analyte in step 1.2.2, the background concentration for that
matrix-analyte combination is assumed to be zero.
47
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 48 of 81
1.4.2 For matrix-analyte combinations in which a background level was detected in step 1.2.1, the
mean background concentration (B) is the average of the data points for that matrix-analyte
combination after step 1.3.
1.4.3 If the background concentration of any method analyte is higher than the middle spike
concentration (M) used to determine the method range, the matrix is inappropriate for the validation
and should be eliminated.
1.5 Choose spike concentrations for the matrix validation. A spike concentration is the final
concentration of a matrix sample after spiking, and is equal to the background concentration of an
analyte plus the amount of analyte spiked into the matrix (T).
1.5.1 The low spike concentration for an analyte should be the lowest concentration determined by
the procedures outlined in Appendix B: Method Range and Method Detection Limit that is at least
five times higher than the background level of that analyte.
1.5.2 The high spike concentration for an analyte should be the concentration that defines the upper
limit of the method range for that analyte.
1.6 Prepare a spiking solution containing each method analyte in a solvent compatible with the
matrix being evaluated. The concentration of analytes in the spiking solution should be such that
the volume of spiking solution added to a matrix sample to produce the desired spike concentration
does not exceed 1 percent of the volume of the matrix sample. This limitation does not apply to
spiking solutions where the solvent used to prepare the spiking solution is identical to the bulk of
the matrix.
1.7 For each sample matrix, prepare five samples at each spike concentration using the spiking
solution. Record the amount of each analyte added to the samples.
1.8 For each sample matrix, process all samples through the entire analytical procedure in random
order. Make all calibrations and computations according to the method and express the
concentrations found in the method reporting units.
2. Evaluation of Results
2.1 Test the concentrations found for outliers.
2.1.1 For each matrix-analyte-concentration combination, test the concentrations found for outliers
at the 1 percent significance level (Appendix D: Statistics for Determining Outliers, Significance,
and Confidence Intervals). Check each data point declared an outlier for calculation and/or
recording errors. Correct any errors and retest for outliers. Eliminate any outlying data point for
which no explanation is found.
2.1.2 If step 2.1.1 results in less than five data points for any matrix-analyte-concentration
combination, analyze new samples as needed to assure a minimum of five data points for each
matrix-analyte-concentration combination.
48
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 49 of 81
2.1.3 If new samples were analyzed in step 2.1.2, return to step 2.1.1. Otherwise, proceed to step
2.2.
2.2 For each matrix, calculate mean percent recovery statistics and compare them to the
corresponding statistics for the standard matrix and to performance requirements.
2.2.1 For each matrix, calculate the mean percent recovery for each analyte at both the high and low
spike concentration, correcting for any background level of the analyte determined by the procedure
outlined in Appendix B: Method Range and Method Detection Limit.
2.2.2 For each matrix-analyte-concentration combination, calculate a 100 (1-,025/Af,) percent
confidence interval for the difference between the mean percent recovery in the sample matrix and
the percent recovery at the corresponding concentration in the standard matrix (Appendix B: Method
Range and Method Detection Limit), where Na is the number of analytes for which the method is
being validated (Appendix D: Statistics for Determining Outliers, Significance, and Confidence
Intervals). If any of the confidence intervals do not contain the value zero, the bias in the sample
matrix is significantly different from the percent recovery in the standard matrix for the
corresponding matrix-analyte-concentration combination.
2.2.3 For each matrix-analyte-concentration combination, calculate a 100 (l-.Q25/Na) percent
confidence interval for the mean percent recovery, where Na is the number of analytes for which the
method is being validated (Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals). If the upper limit of any of the confidence intervals is less than the lower
performance requirement, or if the lower limit of any of the confidence intervals is greater than the
upper performance requirement, the method has failed to meet the percent recovery performance
requirements for the corresponding matrix-analyte-concentration combination.
2.2.4 If the method fails to meet the bias requirement and the bias in the sample matrix is
significantly different from the bias in the standard matrix, the method is declared to have failed the
matrix validation for that matrix-analyte-concentration combination.
2.3 For each matrix, calculate standard deviation statistics and compare them to the corresponding
statistics for the standard matrix and to the precision performance requirement.
2.3.1 For each matrix, calculate the standard deviation for each analyte and the RSD for each
analyte at both the high and low concentration.
2.3.2 For each matrix-analyte-concentration combination, calculate a 100 (1-.025/7V0) percent
confidence interval for the ratio of the standard deviation in the sample matrix to the standard
deviation in the standard matrix at the corresponding concentration, where Na is the number of
analytes for which the method is being validated (Appendix D: Statistics for Determining Outliers,
Significance, and Confidence Intervals). If any of the confidence intervals do not contain the value
1, the standard deviation in the sample matrix is significantly different from the standard deviation
in the standard matrix for the corresponding matrix-analyte-concentration combination.
2.3.3 For each matrix-analyte-concentration combination, calculate a 100 (1-.025/7VJ percent
confidence interval for the true RSD of the method, where Na is the number of analyte for which the
49
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 50 of 81
method is being validated (Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals}. If the lower limit of any of the confidence intervals is greater than the
precision performance requirement, the method has failed to meet the precision performance
requirement for the corresponding matrix-analyte-concentration combination.
2.3.4 If the method fails to meet the precision performance requirement and the standard deviation
in the sample matrix is significantly different from the standard deviation in the standard matrix, the
method is declared to have failed the matrix validation for the corresponding matrix-analyte-
concentration combination.
2.4 If the method has not failed for any matrix-analyte-concentration combination in steps 2.2.4 and
2.3.4, proceed to step 2.6. If a failure has occurred in either step, determine whether an intermediate
spike concentration should be tested.
2.4.1 If the failure occurred at both the high and low spike concentrations, no intermediate spike
concentration should be tested.
2.4.2 If the failure occurred at only the high or low spike concentration, but no spike concentration
for which standard matrix samples were analyzed (Appendix B: Method Range and Method
Detection Limit) falls between the high and low spike concentrations, no intermediate spike
concentration should be tested.
2.4.3 If the failure occurred at only the high or low spike concentration, and at least one
intermediate spike concentration is available, an intermediate spike concentration should be tested
according to step 2.5.
2.5 For each failed matrix-analyte-concentration combination identified in step 2.4.3, choose and
test an intermediate spike concentration.
2.5.1 The intermediate spike concentration to test for an analyte is the spike concentration used to
determine the method range that falls between the high and low spike concentrations. If more than
one intermediate spike concentration is available, the one nearest the spike concentration that caused
failure should be used.
2.5.2 Prepare five samples of a matrix to contain each failed analyte at the intermediate spike
concentration using the spiking solution. Record the amount of each analyte added to the samples.
2.5.3 Process the five samples through the entire analytical procedure.
2.5.4 For each additional set of analyses performed in step 2.5.3, perform the tests, calculations, and
comparisons in steps 2.1 through 2.4. For these steps, the intermediate spike concentration should
be considered the high or low concentration, whichever caused the failure for the matrix-analyte-
concentration combination.
2.6 For each matrix, determine the method validity and range.
50
-------
EMAP Chemical Method Evaluation Guidance, Appendix C, February 1996, Page 51 of 81
2.6.1 A method is valid for an analyte in a particular matrix if two spike concentrations investigated
for that matrix-analyte combination do not result in failure. These two spike concentrations may be
the original high and low concentrations or the high and low concentrations combined with a
subsequently investigated intermediate concentration.
2.6.2 A method is not valid for a matrix-analyte combination that caused failure at both the initial
high and low concentrations.
2.6.3 A method is not valid for a matrix-analyte combination that caused a failure at either the initial
high or low concentration and all intermediate spike concentrations.
2.6.4 If a failure occurs for a single analyte in all matrices tested, the method is not valid for that
analyte and the analyte should be deleted from the method. If a method fails for the majority of
analytes in all matrices tested, the method is not valid.
2.6.5 The method range for each valid matrix-analyte combination extends from the lowest to the
highest spike concentration that did not result in failure.
51
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 52 of 81
Appendix D: Statistics for Determining Outliers,
Significance, and Confidence Intervals
This appendix explains how to perform the statistical calculations and analyses required of the
performance tests described in previous sections of this document. The primary reference for these
techniques is "Statistical Methods" (Snedecor and Cochran, 1980). This text and additional
references are cited within this appendix where appropriate. Full citations are provided in Section
8 - Statistical References at the end of the appendix. See Appendix E: Statistical Example for
examples of the calculations contained here.
1. Using Statistical Tables
1.1 This section describes methods for using reference tables to determine the distributional deviates
needed in the statistical calculations contained in this document. The particular tables referenced
in this section can be found in "Statistical Methods" (Snedecor and Cochran, 1980). Similar tables
may be found in nearly any general statistics text. Note that reference tables for a given distribution
may take many forms. Therefore, it is important to read the description of the table in order to
understand what the table provides and how to find the particular value needed in a calculation. The
required values may also be obtained using certain statistical packages. Examples are provided for
obtaining the distributional deviates using the SAS System.
1.2 The Standard Normal Distribution
1.2.1 This section describes methods for obtaining required distributional deviates from the standard
normal distribution. The distributional deviates take the following form: zs is the deviate from the
standard normal distribution such that:
P(z > z5) = 5
1.2.2 To find this upper-tail deviate in a table that provides P(0 < z < zs), the desired d must be
subtracted from 0.5 to obtain the appropriate value for use with the standard normal distribution
table.
1.2.3 Scan the standard normal distribution table to find the value of 0.5 - d. The first portion of
standard normal deviate associated with this value can be found in the leftmost column of the row
in which the value of 0.5 - <5was found. The hundredths place for the deviate value is found in the
top row of the column in which the value was found. Since standard normal tables generally
provide areas expressed to four decimal places, the value of d should be rounded to four decimal
places.
1.2.4 To illustrate, consider calculating 100 (l-.01/Wa)% confidence intervals for some parameter
where Na = 2. In this case, 6 = .01/(2#) = 0.0025 and 0.5 - S = 0.4975. Scanning the standard
normal table, it can be seen that this area corresponds to z = 2.81.
1.2.5 If the area in question does not appear in the table, there are two methods to determine the
appropriate value of z. The simplest method is to take the larger of the two z-scores whose
associated probabilities bracket the value in question, giving a conservative estimate. The other
52
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 53 of 81
method is to interpolate between the two z-scores whose associated probabilities bracket the value
in question. See subsection 1.3.6 of this appendix for instructions on performing linear
interpolation.
1.2.6 If the probability being sought is associated with more than one value of z, the conservative
estimate is to choose the highest z-score associated with the value. For example, if Na = 5, then d
= .017(2*5) = 0.001 and 0.5 - 6= 0.4990. Scanning the table reveals that 0.4990 is associated with
three values of z: 3.08, 3.09, and 3.10. The largest of these values, 3.10, would be selected for use
in calculations.
1.2.7 The required standard normal deviate may be obtained using statistical software. In the SAS
System, the PROBIT function returns deviates from the standard normal distribution for a given
probability. The form of the function is as follows:
value = PROBIT (1 - 5);
Because the PROBIT function returns values associated with quantiles of the standard normal
distribution, (i.e., z such that P(Z tdff) ). The
first row of the table indicates the level of the two-tailed probability. Locate the appropriate column
for e and find the value of t in the row corresponding to the appropriate number of degrees of
freedom. In this case, e= 2-6.
1.3.3 Because in these sections the calculation of <5 usually involves division by Na, a column for
the exact level of <5 required may not appear in the table. Because the distribution of t is different
for each value of df, it would be too cumbersome to provide a complete table (similar to the standard
normal distribution table) for each number of degrees of freedom. Rather, values corresponding to
certain commonly chosen levels of <5 are provided for the different degrees of freedom. Linear
interpolation between tabular values is preferable to the conservative estimate of using the tabular
value for the probability value closest to, but less than 6. Because the difference between this
conservative estimate and the actual value required can be relatively large, linear interpolation
between adjacent tabular values is recommended to enhance calculation results.
1.3.4 It is more common for tables of the Student's t distribution to provide one-tailed probabilities
rather than two-tailed probabilities. In this case, the value of d is as defined in subsection 1.3.1, one
53
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 54 of 81
half of the value that would be used in a two-tailed table. The remainder of the procedure is the
same as noted in the previous subsection.
1.3.5 To illustrate, consider calculating 100 (l-.01/7Va)% confidence intervals for some parameter
where Na = 4. To use a two-tailed t table, set 6= .01/(JVB) = 0.0025. If n = 10, then df= d - 1 = 9.
Using the row of the two-tailed table corresponding to 9 degrees of freedom, the deviate
corresponding to d— 0.0025 is bracketed by t9 005 = 3.690 and t9 m = 4.781.
1.3.6 Linear interpolation between the two adjacent tabular values is performed as follows:
& 0.005 0.0025 0.001
tb: 3.690 ? 4.781
First calculate:
0.0025 - 0.001
0.005 - 0.001
= 0.375
Rearranging the above equation gives:
0.0025 = 0.001 + 0.375 (0.005 - 0.001)
The desired value of the t distribution can be found by replacing the 6 values with the corresponding
values of tb in the above equation as shown:
t6 = 4.781 + 0.375 (3.690 - 4.781) = 4.372
1.3.7 To use a one-tailed t table for the example above, set d - .Gl/(2Na) = 0.00125 and proceed as
above. Some tables will provide values associated with quantiles of the Student's t distribution, (i.e.,
t such that P(tdf < tdf£) = e). When using a table of this form, use 1 - 6 in place of 6.
1.3.8 The required Student's t distribution deviate may be obtained using statistical software. In the
SAS System, the TINV function returns deviates from the Student's t distribution for a given
probability. The form of the function is as follows:
value = TINV (1 - 6, df);
Because the TINV function returns values associated with quantiles of the Student's t distribution,
(i.e. t such that P(% < fdf£) = e), 1 - <5is the appropriate parameter for the function. Note that other
statistical software may not perform in the same manner. Consult the software user's guide to
determine the proper function and syntax. The value obtained using the computer software will
probably not be the same as the result of the linear interpolation performed above. The computer-
produced values are usually more accurate than the results found by simple linear interpolation.
54
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 55 of 81
1.4 The F Distribution
1.4.1 This section describes methods for obtaining required distributional deviates from the F
distribution. The distributional deviates take the following form: Fdfn_df^s is the deviate from the F
distribution with dfn degrees of freedom in the numerator andj df degrees of freedom in the
denominator such that:
1.4.2 This value can be found in tables such as those contained in "Statistical Methods" (Snedecor
and Cochran, 1980), which provides upper-tail critical values (i.e., ~P(Fdf,dfd > Fdf,dfi^. In most tables,
numerator degrees of freedom are listed across the top of the table (columns) and the denominator
degrees of freedom are listed down the left side of the table (rows). Either individual tables are
provided for various values of 6 or rows are provided for the values of 6 within each level ofdfd.
Locate the appropriate position in the table for dfn and dfd.
1.4.3 As with the Student's t distribution only certain commonly chosen levels of 6 are provided in
the table and the exact value of <5 required may not appear in the table. Linear interpolation between
tabular values is preferable to the conservative estimate of using the tabular value for the probability
value closest to, but less than 6. Because the difference between this conservative estimate and the
actual value required can be relatively large, linear interpolation between adjacent tabular values is
recommended to enhance calculation results. Tables of the F distribution may not contain the
particular combination of numerator and denominator degrees of freedom required by the
calculation. In this case, linear interpolation between available degrees of freedom should be
performed as well.
1.4.4 In general, only upper-tail values are provided in tables of the F distribution. Although the
F distribution is not symmetric, the following relationship locates lower-tail critical values using
upper- tail critical values:
1
The lower-tail critical value is simply the reciprocal of the corresponding upper-tail critical value
with the numerator and denominator degrees of freedom reversed. Some tables will provide values
associated with quantiles of the F distribution, (i.e., F such that P(Fdf,dfl < Fdf,4fd^ = e). When using
a table of this form, use 1 - d in place of 6. The smallest value of 6 represented in tables of the F
distribution from "Statistical Methods" (Snedecor and Cochran, 1980) is d - 0.005. Deviates
corresponding to values of d less than 0.005 will need to be obtained from other tables or by using
statistical software.
1.4.5 To illustrate, consider calculating 100 (l-.01/7V~a)% confidence intervals for a ratio of standard
deviations where Ng = 2. For this problem, 6= .Q25/(2Na) = 0.00625. If n, = 8 and «, = 10, then
dfn = n, -1 = 1 and dfd = n2 -1 = 9. In this case, d= 0.00625 is bracketed by values of <5= 0.01 and
6= 0.005. The use of two tables in "Statistical Methods" (Snedecor and Cochran, 1980) is required
55
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 56 of 81
for this problem. In the appropriate tables, using the columns for 7 numerator degrees of freedom
and the rows for 9 denominator degrees of freedom, F79i01 = 5.62 and F7i9iM5 = 6.88.
1.4.6 Linear interpolation between the available tabular values is performed as follows:
d: 0.01 0.00625 0.005
K: 5.62 ? 6.88
First calculate:
0.00625 - 0.005
0.01 - 0.005
= 0.25
Rearranging the above equation gives:
0.00625 = 0.005 + 0.25(0.01 - 0.005)
The desired value of the F distribution can be found by replacing the 6 values with the
corresponding values of F8 in the above equation as shown:
F6 = 6.88 + 0.25(5.62 - 6.88) = 6.565
1.4.7 The lower-tail deviate is obtained in a similar fashion. As above, tabular values associated
with the values of d that bracket the desired level are obtained using the following relationship:
1
'7,9,1 -.00625
^9,7,.00625
From the table, F9 7 Ol = 6.71 and F9 7 005 = 8.51. Linear interpolation is then performed, as above.
S: 0.01 0.00625 0.005
F8: 6.71 ? 8.51
The first two calculations will be the same as the linear interpolation step performed above. Only
the step replacing the values of S with the corresponding .F8 values in the equation above needs to
be redone.
8.51 + 0.25(6.71 - 8.51) = 8.06
Therefore,
F - ^ - ^ - 0 124
r7,9,1-.00625 c one
/"9,7,.00625 °'Ub
56
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 57 of 81
1.4.8 The required F distribution deviates may be obtained using statistical software. In the SAS
System, the FINV function returns deviates from the F distribution for a given probability. The form
of the function is as follows:
value = F//W(1 -5,dfn,dfd);
Because the FINV function returns values associated with quantiles of the F distribution, (i.e., F
such that f(Fdf,tdf < Fdfidfi£) = e), 1 - 6 is the appropriate parameter for the function. Note that other
statistical software may not perform in the same manner. Consult the software user's guide to
determine the proper function and syntax. The values obtained using the computer software will
probably not be the same as the results of the linear interpolation performed above. The computer-
produced values are usually more accurate than the results found by simple linear interpolation.
1.5 The x2 Distribution
1.5.1 This section describes methods for obtaining required distributional deviates from the x2
distribution. The distributional deviates take the following form: y?dfi8 is the deviate from the X7
distribution with df degrees of freedom such that:
= 5
1.5.2 This critical value can be found in a table such as the one contained in "Statistical Methods"
(Snedecor and Cochran, 1980). Because the x2 distribution is not symmetric and there is no
relationship to obtain lower-tail values using upper-tail values, tables of the x2 distribution must
provide critical values for both tails of the distribution. The x2 tables are usually organized with the
probability values (d) across the top of the table (columns) and the degrees of freedom down the left
side of the table (rows), similar to most tables of the Student's t distribution. Locate the appropriate
column for <5and find the value of x2 in the row corresponding to the appropriate number of degrees
of freedom.
1.5.3 As with the Student's t and F distributions, because the calculation of 6 usually involves
division by Ng, a column for the exact level of 6 required may not appear in the table, making linear
interpolation necessary. Linear interpolation between available values is preferable to the
conservative estimate of using the tabular value for the column in the table with the probability value
closest to, but less than 6. Because the difference between the nearest available deviate and the
actual value required can be relatively large, linear interpolation between adjacent tabular values is
recommended to enhance calculation results. The format of some tables may provide values
associated with quantiles of the x2 distribution, (i.e., x2 such that P(X^/ ^ X#e) = e)- When using a
table of this form, use 1 - 8 in place of d. The smallest value of 6 represented in tables of the x2
distribution from "Statistical Methods" (Snedecor and Cochran, 1980) is 8 = 0.005. Deviates
corresponding to values of d less than 0.005 will need to be obtained from other tables or by using
statistical software.
1.5.4 To illustrate, consider calculating a 95% confidence interval for the MDL of a particular
analytical method. In this case, d= .05/2 = 0.025. If n = 10, then df= n-l=9. Using the row of
57
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 58 of 81
the x2 table corresponding to 9 degrees of freedom and the column for d= 0.025, x\ 025 = 19.02.
Similarly, xl,.915 = 2.70.
1.5.5 If the particular value of 6 is not represented in the x2 table, linear interpolation to determine
the desired value may be performed (see subsection 1.3.5).
1.5.6 The required x2 distribution deviates may be obtained using statistical software. In the SAS
System, the CINV function returns deviates from the x2 distribution for a given probability. The
form of the function is as follows:
value = CINV (1 -5,off);
Because the CINV function returns values associated with quantiles of the x2 distribution, (i.e., x2
such that P(xi// ^ JC//«r) = £), 1 - <5 is the appropriate parameter for the function. Note that other
statistical software may not perform in the same manner. Consult the software user's guide to
determine the proper function and syntax. For values of 6 not represented in a table of the x2
distribution, values obtained using the computer software will probably not be the same as the
results of a linear interpolation calculation. The computer-produced values are usually more
accurate than the results found by simple linear interpolation.
2. Testing for Outliers in a Single Set of Data
2.1 This section describes the maximum normed residual outlier test to detect outliers in a set of
data, xt is used to denote the ith data point and n is the number of data points in the data set. This
procedure assumes that the data are a random sample from a normal distribution.
2.2 If n is four or less, it is not possible to perform an outlier test.
2.3 If n is greater than four, compute the average of the sample observations:
n
EX,
X = -^i-
n
2.4 For each data point jc,-, compute the absolute deviation from the mean:
d,= |x,-x|
2.5 The largest of the d, values and the sum of the squared 4 are used to compute the Maximum
Normed Residual (MNR) statistic as follows:
max Id,}
MNR =
n
\-^ j2
58
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 59 of 81
2.6 Compare the MNR statistic to the critical value corresponding to the sample size n. Critical
values of the MNR statistic are provided in Table D-1: Critical Values for the MNR Outlier Test at
the 1% Level of Significance.
2.6.1 If the MNR statistic is greater than the critical value, the data point associated with the
maximum d: value is declared an outlier and this data point is eliminated from the data set.
2.6.1.1 If more than four data points remain in the data set, return to subsection 1.3 and perform the
test calculations again using the reduced data set. The sample size, «, will be reduced by one.
2.6.1.2 If only four data points remain, terminate the outlier test.
2.6.2 If the MNR statistic is less than or equal to the critical value, conclude that there are no
outliers in the data set and terminate the outlier test.
TABLE D-1: CRITICAL VALUES FOR THE MNR OUTLIER TEST
AT THE 1% LEVEL OF SIGNIFICANCE
n
5
6
7
8
9
10
11
12
13
14
15
Critical
Value
0.882
0.882
0.873
0.860
0.844
0.827
0.811
0.795
0.779
0.764
0.750
3. The Percent Relative Standard Deviation (RSD) and Approximate Confidence Intervals
3.1 This section describes the procedures for calculating approximate confidence intervals for the
RSD also known as the percent Coefficient of Variation (CV) of an analytical method. The notation
and formulas that follow assume that the distribution of the sampled population is normal. The
sampled population should not be assumed to be normal if the RSD is too large. Generally, if the
sample RSD is greater than 33%, the sampled population should not be assumed to be normal
(Johnson and Welch, 1940). Additional notation will be introduced for the RSD when it is not
expressed as a percentage, which is to be used in calculating limits.
59
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 60 of 81
3.2 Compute the average of the sample observations for each analyte-concentration combination:
EX,
X =
n
3.3 Compute the standard deviation of the sample observations for each analyte-concentration
combination:
s =
E (x, - x)2
n - 1
3.4 Compute the percent RSD for each analyte-concentration combination:
RSD = 1x100
x
The notation RSD' will be used to denote the relative standard deviation not expressed as a
percentage. RSD' is calculated as follows:
RSD' = -
3.5 Compute an approximate 100 (l-a)% confidence interval for the RSD' of each analyte-
concentration combination (Johnson and Welch, 1940), where a= the significance level (.Ql/Na,
.Q25/Na, etc.) as follows:
LOWER LIMIT:
100 -Jn
fi
RSD'
7
-£L
2
N
1
2(n-
n
1)(RSD')2
L/PPER L/M/T:
100
V" _ 7Q
RSD' TM
1 +-
/ \2
2(n-1)(RSD/)
where « is the number of sample observations in the analyte-concentration combination and zs is the
standard normal deviate such that P[Z > zj = d.
60
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 61 of 81
4. Calculating Confidence Intervals for the Mean Percent Recovery of an Analytical Method
4.1 This section describes the procedures for calculating confidence intervals for the mean percent
recovery of an analytical method with and without a background level of analyte present. The
calculations in this section assume the independence of x and B. Note that if x and B are not
independent, the estimates of the mean percent recovery and associated confidence intervals will be
over or under estimated.
4.2 If a background level of analyte is present in the matrix, compute the percent recovery as
follows:
PR = * " B-10Q
where x the average of the concentrations found, B = the average of the background concentrations
found, and T= the amount of analyte spiked into the replicate samples.
4.3 Compute upper and lower 100 (\-a)% confidence limits for the percent recovery (Moser and
Stevens, 1992), where a = the significance level (.01/JVa, .Q25/Na, etc.), as follows:
LOWER LIMIT:
T "-7
UPPER LIMIT:
s2 s2
_ + _
n, n2
V,—
s2 s2
where n, is the number of replicate samples at the concentration being considered, s, is the standard
deviation of the n, concentrations found, n2 is the number of replicate samples analyzed to determine
the background level, s2 is the standard deviation of the n2 background concentrations found, and
tvS is the deviate from the Student's t distribution with v degrees of freedom such that P[tv > tvd\ =
6. This value can be found in a table of the Student's t distribution (see subsection 1.3). The
following procedure is used to determine the degrees of freedom, v:
61
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 62 of 81
Q,2
n2 -
s2 s2
wnere Q1 = — and Q2 = —
n2
The result of the preceding calculation is usually not an integer. Common practice is to round down
to the nearest integer.
4.4 If no background level of analyte is present in the matrix, compute the percent recovery as
follows:
PR = --100
where x the average of the concentrations found and T = the amount of analyte spiked into the
replicate samples.
4.5 Compute upper and lower 100 (l-a)% confidence limits for the percent recovery, where a= the
significance level (.01/A^, .025/Na, etc.), as follows:
UPPER LIMIT :
T n^-~J^
LOWER LIMIT
where n, is the number of replicate samples at the concentration being considered, s, is the standard
deviation of the n, concentrations found, and tn,_isis the deviate from the Student's t distribution with
n,-l degrees of freedom such that
* = 6
This value can be found in a table of the Student's t distribution (see subsection 1.3).
62
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 63 of 81
5. Calculating Confidence Intervals for the Method Detection Limit for an Analyte
5.1 This section describes the procedures for calculating confidence intervals for the method
detection limit (MDL) for an analyte for a given analytical method.
5.2 Calculate the MDL as follows:
MDL = fn_1i0.01(s)
where n is the number of replicate samples at the spike concentration that defines the lower limit of
the method range, 5 is the standard deviation of the n concentrations found, and tn_, s is the deviate
from the Student's t distribution with n-1 degrees of freedom such that P[^_/ > tn_, s] = 6. This value
can be found in a table of the Student's t distribution (see subsection 1.3).
5.3 Compute upper and lower 100 (\-a)% confidence limits for the MDL, where a = the
significance level (generally 0.05) as follows:
LOWER LIMIT :
MDL
\
<-a
2
UPPER LIMIT :
MDL
n
2
X
n-
-1
1,1--
2
where n is the number of replicate samples at the spike concentration that defines the lower limit of
the method range and Xn-i,s i§ tne deviate from the x2 distribution such that P[xi/ ^ X«-/
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 64 of 81
6.2 If there is a background level of analyte present, compute upper and lower 100 (\-a)%
confidence limits for the difference in percent recovery, PR, - PR2, where a = the significance level
(.Ql/Na, .Q25/Na, etc.), as follows:
LOWER LIMIT :
(PR,-PR,) - 100z^
T,2",
T22n2
T2 n
UPPER LIMIT:
S1
. + .
s|
+ .
^
2\
T2n2
where n, is the number of replicate samples in the first set of concentrations found, T, is the amount
of analyte spiked into the first set of samples, s, is the standard deviation of the H, concentrations
found in the first set of samples, n2 is the number of replicate samples in the second set of
concentrations found, T2 is the amount of analyte spiked into the second set of samples, $ is the
standard deviation of the n2 concentrations found in the second set of samples, n3 is the number of
samples analyzed to determine the background level, s} is the standard deviation of the n
background concentrations found, and z8 is the deviate from the standard normal distribution such
that P[z > zs] = d. This value can be found in a table of the standard normal distribution (see
subsection 1.2).
6.3 If no background level of analyte is present in the second sample, compute the percent recovery
for the second set of samples as follows:
PR9 = —-100
2
where x2 the average of the concentrations found in the second set of samples.
6.4 Compute the upper and lower 100 (l-a)% confidence limits for the difference in percent
recovery, PR, - PR2, where a= the significance level (.01/AT,, .025/Na, etc.), as follows:
LOWER LIMIT :
(PR,-PR2) - 100 Za
2\\
Tfn,
T|n2
64
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 65 of 81
UPPER LIMIT :
(PR,-PR2) + 100 Za
2\|
where n, is the number of replicate samples in the first set of concentrations found, T, is the amount
of analyte spiked into the first set of samples, s, is the standard deviation of the n, concentrations
found in the first set of samples, n2 is the number of replicate samples in the second set of
concentrations found, T2 is the amount of analyte spiked into the second set of samples, s2 is the
standard deviation of the n2 concentrations found in the second set of samples, and zd is the deviate
from the standard normal distribution such that P[z > zs] - d. This value can be found in a table of
the standard normal curve or by using computer software capable of providing exact values (see
subsection 1.2).
7. Calculating Confidence Intervals for the Ratio of Two Standard Deviations
7.1 This section describes the procedures for calculating 100 (\-a)% confidence intervals for the
ratio of the population standard deviations (Oj/ff2) associated with two sets of concentrations,
where a= the significance level (.01/A^, .Q25/Na, etc.).
7.2 The upper and lower 100 (l-ar)% confidence limits for the ratio of two standard deviations,
o,/o2, are computed as follows:
LOWER LIMIT :
1
"l-1,n2-1,-f
UPPER LIMIT :
SH
3i
1
-
2
where n, is the number of replicate samples in the first set of concentrations found, s, is the standard
deviation of the n, concentrations found in the first set of samples, q is the number of replicate
samples in the second set of concentrations found, s2 is the standard deviation of the2 n
concentrations found in the second set of samples, and Fn,_, _n,_, s is the deviate from the F distribution
with n,-l numerator degrees of freedom and n2-l denominator degrees of freedom, such that:
This value can be found in a table of the F distribution (see subsection 1.4).
65
-------
EMAP Chemical Method Evaluation Guidance, Appendix D, February 1996, Page 66 of 81
8. Statistical References
Snedecor, George W. and Cochran, William G. 1980. Statistical Methods, Seventh Edition. The
Iowa State University Press, pp 279-280.
Johnson, N. L. and Welch, B. L. 1940. Applications of the Non-Central ^-Distribution. Biometrika,
31,362.
Moser. Barry C. and Stevens, Gary R. 1992. Homogeneity of Variance in the Two-Sample Means
Test. The American Statistician, 46, 19.
66
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 67 of 81
Appendix E: Statistical Example
This appendix contains examples of statistical calculations and analyses described in Appendix D:
Statistics for Determining Outliers, Significance, and Confidence Intervals.
1. Example of Testing for Outliers in a Single Set of Data
1.1 This section contains an example of using the maximum normed residual outlier test (Snedecor
and Cochran, 1980) to detect outliers in a set of data. If the sample size, n, is 4 or less, it is not
possible to perform the outlier test. Statistical data adapted from a data set produced by Technology
Applications Inc. for Environmental Monitoring Systems Laboratory, USEPA, Cincinnati, Ohio
using cadmium as the analyte is found in Table E-l: Data for Outliers Test. The values of the
cadmium are in jAg/1.
Table E-l: Data for Outliers Test
n
1
2
3
4
5
6
7
8
xi
1.91
2.02
2.30
2.20
2.11
3.15
1.81
2.14
1.2 Compute the average of the sample observation.
n
X -
n
where xt is the ith observation and n is the number of data points in the data set.
For the data in this example, n = 8 and 23c,-= 17.64. The calculated mean is:
- _ 17.64
x =
8
= 2.205
67
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 68 of 81
1.3 Compute the absolute deviation from the mean for each data point xt.
rl = I Y — Y I
U, \ X, X \
The calculated deviations for this example are listed in Table E-2: Absolute Deviations from the
Mean.
1.4 Compute the Maximum Normed Residual statistic (MNR).
max id)
MNR =
For the data in this example,
MNR = °'945 - 0.865
A/1.1926
Table E-2: Absolute Deviations from the Mean
X;
1
2
3
4
5
6
7
8
1.91
2.02
2.30
2.20
2.11
3.15
1.81
2.14
0.295
0.185
0.095
0.005
0.095
0.945
0.395
0.065
0.087025
0.034225
0.009025
0.000025
0.009025
0.893025
0.156025
0.004225
1.5 The decision rule for this test is to compare the MNR as computed in subsection 1.4 to the
critical value corresponding to the sample size, n, found in Table D-l: Critical Values for the MNR
Outlier Test at the 1% Level of Significance in Appendix D: Statistics for Determining Outliers,
Significance, and Confidence Intervals. If the computed MNR is greater than the critical value,
conclude that the data point associated with the maximum dt value is an outlier and should be
eliminated from the data set. Because MNR = 0.865 is greater than the critical value, 0.860,
eliminate the data point 3.15.
1.6 If more than four data points remain in the dataset, return to subsection 1.2 and perform the
calculations again using the reduced data set. The sample size, n, is reduced by one.
1.7 The data for this calculation of MNR is outlined in Table E-3: Reduced Data Set. For the data
in this example, n = l, &t = 14.49, and x - 2.07. The calculated MNR is:
68
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 69 of 81
max d,
MNR = -
E42
MNR = °'26 = 0.627
Table E-3: Reduced Data Set
x,
1
2
3
4
5
6
7
1.91
2.02
2.30
2.20
2.11
1.81
2.14
0.16
0.05
0.23
0.13
0.04
0.26
0.07
0.0256
0.0025
0.0529
0.0169
0.0016
0.0676
0.0049
1.8 Compare the MNR statistic of the reduced data set to the critical value corresponding to the
reduced sample size, n, found in Table D-l: Critical Values for the MNR Outlier Test at the 1%
Level of Significance in Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals. The new MNR statistic, 0.627, is less than the critical value, 0.873. There
are no remaining outliers in the data set. End the outlier test and use the reduced data set for all
future calculations.
2. Example of Calculating the Percent Relative Standard Deviation (RSD) and Approximate
Confidence Intervals
2.1 This section contains an example of calculating approximate confidence intervals for the RSD.
The distribution of the sampled population is assumed to be normal. The reduced data set found in
Table E-3: Reduced Data Set is used to illustrate the calculations.
2.2 Compute the average of the sample observations for each analyte-concentration combination:
n
X=>-^
n
For the data in this example, n = 7, Sxt = 14.49. The computed mean, x, is 2.07.
69
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 70 of 81
2.3 Compute the standard deviation of the sample observation for each analyte-concentration
combination:
s =
E (x, -
n - 1
For the data in this example:
0.172
7-1
=0.169
2.4 Compute the RSD for each analyte-concentration combination:
RSD = 1 x 100
x
For the data in this example:
RSD = - x 100 = 8.2
2.070
2.5 Compute the RSD' statistic that denotes the RSD not as a percentage.
RSD' = - = - - 0.082
x 2.070
2.6 Compute the 100 (!-#)% confidence interval for the RSD of each analyte-concentration
combination, using the RSD' statistic, as follows:
LOWER LIMIT:
RSD
- Z «
n
2(n-1)(RSD/)2
UPPER LIMIT:
-^- - Zc.
RSD' T^
n
/ \2
2(A7-1)(RSD7 )
70
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 71 of 81
Where a is the the significance level (.01 INa, .Q25/Na, etc), n is the number of sample observations
in the analyte-concentration combination, and zs is the standard normal deviate such that P[Z > zd]
= 6. This value can be found in a table of the Standard Normal distribution (see subsection 1.2 in
Appendix D: Statistics for Determining Outliers, Significance, and Confidence Intervals).
2.7 For the data in this example, at a= .05, the confidence limits are:
LOWER LIMIT :
0.082 1'—\| (2)(6)(0.082)2
= 0.052
UPPER LIMIT:
v/7
0.082
"Z-\
2 \
1 +
7
(2)(6)(0.082)2
- 0.190
The 95% confidence interval for the percent RSD is (0.052, 0.190).
3. Example of Calculating Confidence Intervals for the Mean Percent Recovery of an
Analytical Method
3.1 This section contains an example of the calculation of confidence intervals for a mean percent
recovery of an analytical method with a background level present. B and x are assumed to be
independent. The data for this example is found in Table E-4: Data with a Background Level
Present.
71
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 72 of 81
Table E-4: Data with a Background Level Present
n
1
2
3
4
5
6
7
*,
1.91
2.02
2.30
2.20
2.11
1.81
2.14
Background
Level
0.00
0.03
0.19
0.15
0.07
0.01
0.09
3.2 Compute the percent recovery as follows:
PR - • 100
r
where x is the average of the concentration measurements, B is the average of the background
concentration measurements, and Tis the amount of analyte spiked into the replicate samples.
3.3 For the data in this example:
PR = 2-070 - °-077 • 100 = 99.650
3.4 Compute the upper and lower 100 (1-0)% confidence limits for the percent recovery, as follows:
LOWER LIMIT:
PR -
TV, — \
2 \
2
fl -
"1
2
H -fi.
"2
* i , £_
t a +
UPPER LIMIT:
100
PR
t «
7V,— \
2 \
"
where tfis the significance level (.OlA^, .Q25Na, etc), n, is the number of replicate samples at the
concentration being considered, s, is the standard deviation of the n, measurements found, n2 is the
number of replicate samples analyzed to determine the background level, s2 is the standard deviation
of the n2 background measurements found, and tv8 is the deviate from the Student's t distribution
with v degrees of freedom such that P[tv > fa] = 6. This value can be found in a table of the
72
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 73 of 81
Student's / distribution (see subsection 1.3 in Appendix D: Statistics for Determining Outliers,
Significance, and Confidence Intervals).
The following equations are used to compute the degrees of freedom, v:
(Qi + Q2)2
V =
n1 - 1 n2 -
s2 s2
where Q1 = — and Q2 = —
3.5 For the data in this example, the degrees of freedom are:
Q _ (0.169)2 _ 108.10-3 Q (0.073)2
1 7 2 7
[(4.080-10"3)+(7.613-1Q-4)]2 _ ,
= 7.613.10-4
1Q-5 5.796-10
-7
7-1
7-1
Rounding v down to the nearest integer, gives v = 8. The confidence limits are:
LOWER LIMIT:
99.650 - — t M_.
2 ' •)
(0.169)2 (0.073)2
= 91.627
UPPER LIMIT:
99.650 + ™t.
(0.169)2 + (0.073)2
= 107.672
The 95% confidence interval for the mean percent recovery is (91.27, 107.672).
4. Example of Calculating Confidence Intervals for the Method Detection Limit of Analytes
73
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 74 of 81
4.1 This section contains an example of the calculation of confidence intervals for an MDL for an
analyte of a given analytical method. The data for this example is found in Table E-3: Reduced
Data Set.
4.2 Compute the MDL as follows:
MDL = t,
n-1,0.01
(s)
where n is the number of replicate samples at the spike level that defines the lower limit of the
method range, 5 is the standard deviation of the n measurements found, and t^ 5 is the deviate from
the Student's t distribution with n-1 degrees of freedom such that P[tn_j > tn_, d] = 6. This value can
be found in a table of the Student's t distribution (see subsection 1.3 in Appendix D: Statistics for
Determining Outliers, Significance, and Confidence Intervals).
4.3 For the data in this example:
MDL - f7_1001(0.169)
= (3.143) (0.169)
= 0.531
4.4 Compute upper and lower 100 (!-#)% confidence limits for the MDL, where a — the
significance level as follows:
LOWER LIMIT:
MDL
\
n-1
UPPER LIMIT:
MDL-
n-1
£1 1-
a
7
where n is the number of replicate samples at the spike level that defines the lower limit of the
method range, s is the standard deviation of the n measurements found, tn_, s is the deviate from the
Student's t distribution with n-1 degrees of freedom such that P|X_, 2 tn_, s] = <5, and X«-/,,? is the
deviate from the Student's t distribution such that P[X2n-/ ^ xV/.J = £
74
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 75 of 81
The tabular values can be found in tables of the Student's t distribution and the x2 distribution (see
subsections 1.3 and 1.5 in Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals).
4.5 For the data in this example:
LOWER LIMIT: 0.531
7-1
= 0.342
UPPER LIMIT: 0.531 •
= 1.168
7-1
X7-1,1--°?.
2
The 95% confidence interval for this example is (0.342, 1.168).
5. Example of Calculating Confidence Intervals for the Difference in Two Percent Recovery
Means
5.1 This section contains an example of calculating confidence intervals for the difference in two
percent recovery means including an adjustment for a background level of the analyte present. B,
x,, and x2 are assumed to be independent. The statistical data are found in Table E-5: Data with a
Background Level Present. The spike level for the first concentration is 2 and the spike level for
the second concentration is 3.
75
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 76 of 81
Table E-5: Data with a Background Level Present
Concentration
Background Level
1
2
3
4
5
6
7
1.91
2.02
2.30
2.20
2.11
1.81
2.14
0.00
0.03
0.19
0.15
0.07
0.01
0.09
1
2
3
4
5
6
2.98
3.04
3.12
3.07
2.94
2.97
0.02
0.01
0.13
0.10
0.01
0.03
5.2 Compute the upper and lower 100 (l-a)% confidence limits for the difference in percent
recovery PR, - PR2 as follows:
LOWER LIMIT:
(PR, - PR2) - 100 Za
722n2
UPPER LIMIT:
(PR^ - PR2) + 100 Za_
2\|
72n2
r2n3
Where ais the significance level (.01/7Vfl, .025/Na, etc.), n; is the number of replicate samples in the
first set of measurements found, T, is the amount of analyte spiked into the first set of samples, st
is the standard deviation of the n1 measurements found in the first set of samples, n2 is the number
of replicate samples in the second set of measurements found, T2 is the amount of analyte spiked into
76
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 77 of 81
the second set of samples, s2 is the standard deviation of the «2 measurements found in the second
set of samples, n3 is the number of replicate samples use to determine the background level, s3 is the
standard deviation of the n3 background measurements found, and z^is the deviate from the standard
normal deviation such that P[z > zs] = 6. This value can be found in a table of the standard normal
distribution (see subsection 1.2 in Appendix D: Statistics for Determining Outliers, Significance, and
Confidence Intervals.
5.3 For the data in this example:
LOWER LIMIT:
(99.650 - 99.000) - 100 z
0.1 692
0.0682
0.0662
227
326
3213
= -5.976
UPPER LIMIT:
(99.650 - 99.000) + 100 z.c
0.
0.0682
326
.0662
3213
= 6.626
The 95% confidence interval for the difference in the two percent recoveries is (5.976, 6.626).
6. Example of Calculating Confidence Intervals for the Ratio of Two Standard Deviations
6.1 This section includes an example of calculating 100 (\-d)% confidence intervals for the ratio
of the population standard deviations (cr/tr2) associated with two sets of measurements. The data
for this example is found in Table E-5: Data with a Background Level Present.
6.2 Compute the upper and lower 100 (l-ar)% confidence limits for the ratio of the standard
deviations (o~,/a2y.
LOWER LIMIT:
UPF
S2\
si
SZN
1
Fn _^n_^a_
"1 ."2 • 2
>ER LIMIT:
1
- °1 C
Fn --\n -M-- S2^ "2"1'ni"1'f
77
-------
EMAP Chemical Method Evaluation, Appendix E, February 1996, Page 78 of 81
where or is the significance level (.Ol/Na, .Q25/Na, etc.), n, is the number of replicate samples in the
first set of measurements found, Sj is the standard deviation of the n, measurements found in the first
set of samples, n2 is the number of replicate samples in the second set of measurements found, s2 is
the standard deviation of the n2 measurements found in the second set of samples, and Fnl_ln2.liS is
the <5 deviate from the F distribution with n,-l numerator degrees of freedom and n2-l denominator
degrees of freedom, such that:
This value can be found in a table of the F distribution (see subsection 1.4 in Appendix D: Statistics
for Determining Outliers, Significance, and Confidence Intervals).
6.3 For the data in this example:
LOWER LIMIT:
0.169
0.068
1
~7-1,6-1,-°5
2
= 0.941
UPPER LIMIT:
0.169 r^
0.068 \| 6-1,7-1,-^
= 5.716
The 95% confidence interval for the ratio of two standard deviations is (0.941, 5.716).
78
-------
EMAP Chemical Method Evaluation Guidance, Appendix F, February 1996, Page 79 of 81
Appendix F: EMAP Demonstration of Method Comparability Checklist
for Chemical Methods
Date:
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
Page _ of _
Conditions
1 . Written candidate method (EMMC format)
attached?
2. Title and date of approved EMAP method
3. Copy of approved EMAP method maintained on-
site?
4. Listing of differences between candidate method and
approved EMAP method attached?
5. Performance range tested (w/units)
6. Linear working range (w/units)
7. Concentrations of calibration standards w/units
corresponding to final sample concentration (lowest
standard must be one to four times MDL)
8. Calibration curve attached?
9. Slope of calibration regression line (if applicable)
10. Relative Standard Deviation (RSD) of calibration
factors (if applicable)
1 1 . Sample preservatives and holding times
12. Interferences
13. Qualitative identification criteria used
14. Surrogates used (if applicable)
15. Surrogate recovery limits established
Approved EMAP
Method
Performance
Specification
—
—
—
Summary
of Results
Y/N
Y/N
Y/N
Y/N
Y/N
79
-------
EMAP Chemical Method Evaluation Guidance, Appendix F, February 1996, Page 80 of 81
EMAP Demonstration of Method Comparability Checklist
for Chemical Methods
Date:
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
Page of_
Analytical Results*
16. Testing Conditions
Matrix used (reagent water, drinking water, etc.)
Spike levels (w/units corresponding to final sample
concentration)
Source of spiking material (using external source whenever
possible)
Number of replicates (minimum n=5)
17. Performance Characteristics
Precision (RSD [n-1] of replicate measurements - for multiple
analyte methods, give range of results and attach full listing)
Percent recovery - for multiple analyte methods, give range of
results and attach full listing)
Method Detection Limit (MDL) (w/units and number of
replicates - for multiple analyte methods, give range of results
and attach full listing)
Approved EMAP
Method
Performance
Specification
Candidate Method
Results
* Complete one page for each matrix tested.
80
-------
EMAP Chemical Method Evaluation Guidance, Appendix F, February 1996, Page 81 of 81
EMAP Demonstration of Method Comparability
for Chemical Methods Certification Statement*
Date: Page _ of _
Facility Name:
Resource Group/Program:
Matrix Type:
Analyte or Class of Analytes:
We, the undersigned, CERTIFY that:
1. The candidate method proposed for EMAP use has undergone the EMAP Demonstration of
Method Comparability, as specified under the EMAP PBMS, and has met the performance
specifications of the approved EMAP method.
2. A copy of the candidate method, written in the EMMC format, and copies of the approved EMAP
method and laboratory-specific Standard Operating Procedures (SOPs) are attached to this checklist
and are available on-site.
3. The data associated with the EMAP Demonstration of Method Comparability are complete
(including the mandatory copy of this checklist), all raw data necessary to reconstruct and verify the
analyses have been retained, and the associated information is available for review.
Name and Title Signature Date
Name and Title Signature Date
Name and Title Signature Date
Name and Title Signature Date
'This certification form must be completed when the EMAP Demonstration of Method
Comparability is performed to approve a candidate method for EMAP use.
81
-------
-------
|