August 2003
Environmental Technology
Verification Report
Industrial Test Systems, Inc.
Quick™ II Test Kit
Prepared by
Battel le
W
Baneiie
. . . Pulling Technology To Work
Under a cooperative agreement with
*** U.S. Environmental Protection Agency
-------
August 2003
Environmental Technology Verification
Report
ETV Advanced Monitoring Systems Center
Industrial Test Systems, Inc.
Quick™ II Test Kit
by
Tim Kaufman
Patty White
Amy Dindal
Zachary Willenberg
Karen Riggs
Battelle
Columbus, Ohio 43201
-------
Notice
The U.S. Environmental Protection Agency (EPA), through its Office of Research and
Development, has financially supported and collaborated in the extramural program described
here. This document has been peer reviewed by the Agency and recommended for public release.
Mention of trade names or commercial products does not constitute endorsement or
recommendation by the EPA for use.
11
-------
Foreword
The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the
nation's air, water, and land resources. Under a mandate of national environmental laws, the
Agency strives to formulate and implement actions leading to a compatible balance between
human activities and the ability of natural systems to support and nurture life. To meet this
mandate, the EPA's Office of Research and Development provides data and science support that
can be used to solve environmental problems and to build the scientific knowledge base needed
to manage our ecological resources wisely, to understand how pollutants affect our health, and to
prevent or reduce environmental risks.
The Environmental Technology Verification (ETV) Program has been established by the EPA to
verify the performance characteristics of innovative environmental technology across all media
and to report this objective information to permitters, buyers, and users of the technology, thus
substantially accelerating the entrance of new environmental technologies into the marketplace.
Verification organizations oversee and report verification activities based on testing and quality
assurance protocols developed with input from major stakeholders and customer groups
associated with the technology area. ETV consists of seven environmental technology centers.
Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/.
Effective verifications of monitoring technologies are needed to assess environmental quality
and to supply cost and performance data to select the most appropriate technology for that
assessment. In 1997, through a competitive cooperative agreement, Battelle was awarded EPA
funding and support to plan, coordinate, and conduct such verification tests for "Advanced
Monitoring Systems for Air, Water, and Soil" and report the results to the community at large.
Information concerning this specific environmental technology area can be found on the Internet
at http://www.epa.gov/etv/centers/centerl .html.
111
-------
Acknowledgments
The authors wish to acknowledge the support of all those who helped plan and conduct the
verification test, analyze the data, and prepare this report. In particular we would like to thank
Rosanna Buhl, Adam Abbgy, and Bea Weaver of Battelle, and Mike Madigan and Rick Linde of
the Ayer, Massachusetts Department of Public Works Water Division. We also acknowledge the
assistance of Jeff Adams of EPA and AMS Center stakeholders Vito Minei and Marty Link, who
reviewed the verification reports.
iv
-------
Contents
Page
Notice ii
Foreword iii
Acknowledgments iv
List of Abbreviations viii
1 Background 1
2 Technology Description 2
3 Test Design and Procedures 4
3.1 Introduction 4
3.2 Test Design 4
3.3 Test Samples 5
3.3.1 QC Samples 5
3.3.2 PT Samples 7
3.3.3 Environmental Samples 7
3.4 Reference Analysis 8
3.5 Verification Schedule 8
4 Quality Assurance/Quality Control 9
4.1 Laboratory QC for Reference Method 9
4.2 Audits 11
4.2.1 Performance Evaluation Audit 12
4.2.2 Technical Systems Audit 12
4.2.3 Data Quality Audit 13
4.3 QA/QC Reporting 13
4.4 Data Review 13
5 Statistical Methods 15
5.1 Accuracy 15
5.2 Precision 16
5.3 Linearity 16
5.4 Method Detection Limit 16
5.5 Matrix Interference Effects 17
5.6 Operator Bias 17
5.7 Inter-Unit Reproducibility 17
5.8 Rate of False Positives/False Negatives 17
6 Test Results 18
6.1 QC Samples 18
6.2 PT and Environmental Samples 24
6.2.1 Accuracy 25
6.2.2 Precision 25
6.2.3 Linearity 31
6.2.4 Method Detection Limit 34
v
-------
6.2.5 Matrix Interference Effects 35
6.2.6 Operator Bias 35
6.2.7 Inter-Unit Reproducibility 36
6.2.8 Rate of False Positives/False Negatives 37
6.3 Other F actors 37
6.3.1 Ease of Use 37
6.3.2 Sample Analysis Time 41
6.3.3 Reliability 41
6.3.4 Waste Material 41
6.3.5 Cost 41
7 Performance Summary 42
8 References 45
Figures
Figure 2-1. Industrial Test Systems, Inc., Quick™ II Arsenic Test Kit 2
Figure 2-2. Quick™ II Color Chart 3
Figure 6-1. Linearity of Quick™ II Color Chart Results 33
Figure 6-2. Linearity of Quick™ Arsenic Scan Results 33
Figure 6-3. Linearity of Compu-Scan Results 34
Figure 6-4. Comparison of Quick™ II Test Results for Technical and Non-Technical
Operators 35
Figure 6-5. Inter-Unit Reproducibility for the Quick™ Arsenic Scan and
Compu-Scan Units 36
Tables
Table 3-1. Test Samples for Verification of the Quick™ II Test Kit 6
Table 3-2. Schedule of Verification Test Days 8
Table 4-1. Reference Method QCS Analysis Results 10
Table 4-2. Reference Method LFM Sample Results 11
Table 4-3. Reference Method Duplicate Analysis Results 12
Table 4-4. Reference Method PE Audit Results 12
Table 4-5. Summary of Data Recording Process 14
vi
-------
Table 6-la. RB Sample Results for the Technical Operator 19
Table 6-lb. RB Sample Results for the Non-Technical Operator 20
Table 6-2a. QCS Results for the Technical Operator 21
Table 6-2b. QCS Results for the Non-Technical Operator 22
Table 6-3 a. LFM Sample Results for the Technical Operator 23
Table 6-3b. LFM Sample Results for the Non-Technical Operator 24
Table 6-4. Quick™ II Test Kit and Reference Sample Results 26
Table 6-5. Quantitative Evaluation of Accuracy for Quick™ II Test Kits 29
Table 6-6. Qualitative Evaluation of Agreement for Quick™ II Test Kits 30
Table 6-7. Precision Results for Quick™ II Test Kits 32
Table 6-8. Detection Limit Results for Quick™ II Test Kit 34
Table 6-9. Rate of False Positives for Quick™ II Test Kits 38
Table 6-10. Rate of False Negatives for Quick™ II Test Kits 40
Table 7-1. Summary of Linear Regression Equations for Test Kit and Reference Results 43
vii
-------
List of Abbreviations
AMS
Advanced Monitoring Systems
EPA
U.S. Environmental Protection Agency
ETV
Environmental Technology Verification
HDPE
high-density polyethylene
ICPMS
inductively coupled plasma mass spectrometry
LFM
laboratory-fortified matrix
MDL
method detection limit
MSDS
Material Safety Data Sheet
NIST
National Institute of Standards and Technology
ppb
parts per billion
ppm
parts per million
PE
performance evaluation
PT
performance test
QA
quality assurance
QA/QC
quality assurance/quality control
QC
quality control
QCS
quality control standard
QMP
Quality Management Plan
RB
reagent blank
RPD
relative percent difference
RSD
relative standard deviation
TSA
technical systems audit
viii
-------
Chapter 1
Background
The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology
Verification (ETV) Program to facilitate the deployment of innovative environmental tech-
nologies through performance verification and dissemination of information. The goal of the
ETV Program is to further environmental protection by accelerating the acceptance and use of
improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-
quality, peer-reviewed data on technology performance to those involved in the design,
distribution, financing, permitting, purchase, and use of environmental technologies.
ETV works in partnership with recognized testing organizations; with stakeholder groups
consisting of buyers, vendor organizations, and permitters; and with the full participation of
individual technology developers. The program evaluates the performance of innovative
technologies by developing test plans that are responsive to the needs of stakeholders, con-
ducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing
peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality
assurance (QA) protocols to ensure that data of known and adequate quality are generated and
that the results are defensible.
The EPA's National Exposure Research Laboratory and its verification organization partner,
Battelle, operate the Advanced Monitoring Systems (AMS) Center under ETV. The AMS Center
recently evaluated the performance of the Industrial Test Systems, Inc., Quick™ II test kit for
measuring arsenic in water.
1
-------
Chapter 2
Technology Description
The objective of the ETV AMS Center is to verify the performance characteri stics of
environmental monitoring technologies for air, water, and soil. This verification report provides
results for the verification testing of the Quick™ II test kit for arsenic in water (Figure 2-1).
Following is a description of the test kit, based on information provided by the vendor. The
information provided below was not verified in this test.
Figure 2-1. Industrial Test Systems, Inc.,
Quick™ II Arsenic Test Kit
To perform arsenic analyses with the Quick™
II test kit, the water sample to be tested is
mixed in the supplied reaction vessel with
reagent #1 (tartaric acid with rate enhancers) to
acidify the water sample. Reagent #2, an
oxidizer (potassium peroxymonosulfate), is
added to remove hydrogen sulfide interference.
The test tolerates up to 2 parts per million
(ppm) hydrogen sulfide without interference.
Zinc powder, reagent #3, is added to reduce
inorganic arsenic compounds (As 3 and As"3)
to arsine gas. As arsine gas is generated and
comes in contact with the test strip, the
mercuric bromide indicator on the test strip
changes color from white to shades of yellow
or brown.
Material Safety Data Sheets (MSDSs) for all reagents and test strips are provided with each test
kit. The MSDSs include information on how to safely handle the reagents and test strips,
including instructions for exposure controls and personal protection.
Once the reaction is completed, the test strip is removed and visually compared to a color chart
to obtain a semi-quantitative measure of the arsenic concentration in the tested sample.
The color chart consists of a series of color blocks (Figure 2-2). The color blocks correspond to
concentrations ranging from 2 parts per billion (ppb) to >150 ppb. If the color on the test strip is
between two color blocks, then the operator may estimate the concentration as between the two
values associated with the color blocks on either side. The test strip may also be read with the
Quick™ Arsenic Scan hand-held instrument, which operates on the same principle as a
colorimeter and provides a quantitative result. The Quick™ Arsenic Scan is calibrated weekly
2
-------
(ig/L 2 ppb 2.5 3 3.5 4 5 6 7 10
(ig/l 12 ppb 14 20 30 >30 >50 >70 >100 >150
Quick™ II Calibration Valid 24° to 30°C bdl=Ben™ Detect u*ei 01/16/2003
Figure 2-2. Quick™ II Color Chart
using a calibration card provided by the manufacturer. Quantitative results may also be obtained
from the test strip with a portable Compu-Scan scanner and laptop computer system. The
scanned test strip image is converted to an arsenic concentration using the Home Port Computer
System Arsenic Program Revision 5b software. The scanner is calibrated by the manufacturer.
The Quick™ Arsenic Scan and Compu-Scan are not provided with the Quick™ II test kit as a
standard feature. The standard test kit with the color chart was the subject of the verification test;
however, results for the Quick™ Arsenic Scan and Compu-Scan are also provided.
The optimal detection range for the Quick™ II test kit is below 10 ppb. Dilution instructions are
provided for samples with arsenic levels above 15 ppb. The recommended temperature range for
sample analysis is 24°C to 30°C. A modified testing protocol that specifies longer reaction times
(up to 30 minutes longer for samples between 5° and 15°C) is available for sample temperatures
below this range.
The Quick™ litest kits are available in sets of 50 tests. The typical shelf life of the kits is
24 months.
3
-------
Chapter 3
Test Design and Procedures
3.1 Introduction
This verification test was conducted according to procedures specified in the Test/QA Plan for
Verification of Portable Analyzers^ The verification was based on comparing the arsenic
results from the Quick™ II test kit to those from a laboratory-based reference method. The
reference method for arsenic analysis was inductively coupled plasma mass spectrometry
(ICPMS) performed according to EPA Method 200.8(2) The Quick™ II test kit relies on
comparisons to a color chart provided with the test kit to allow semi-quantitative measurements
of arsenic concentrations. Quantitative results were also obtained from a Quick™ Arsenic Scan
and a Compu-Scan system. The test kit performance was verified by analyzing laboratory-
prepared performance test samples, treated and untreated drinking water, and fresh surface water.
All samples were tested using both the test kit and the reference method. Both semi-quantitative
and quantitative analyses were performed by the technical and non-technical operators. The test
design and procedures are described further below.
3.2 Test Design
The Quick™ II test kit was verified by evaluating the following parameters:
¦ Accuracy
¦ Precision
¦ Linearity
¦ Method detection limit (MDL)
¦ Matrix interference effects
¦ Operator bias
¦ Inter-unit reproducibility
¦ Rate of false positives/false negatives.
All sample preparation and analyses were performed according to the manufacturer's
recommended procedures. All samples were warmed to 24°C prior to analysis using a hot water
bath, which is at the lower end of the optimal temperature range listed in the test kit instructions.
Color chart, Quick™ Arsenic Scan and Compu-Scan results were recorded manually. The results
from the Quick™ II test kits were compared to those from the reference method to assess
accuracy and linearity. Multiple aliquots of performance test samples, drinking water samples,
and surface water samples were analyzed to assess precision. Multiple aliquots of a low-level
4
-------
performance test sample were analyzed to assess the detection limit of the method. Potential
matrix interference effects were assessed by challenging the test kit with performance test
samples of known arsenic concentrations that contained both low levels and high levels of
interfering substances.
Identical sets of samples were analyzed independently by a technical and a non-technical
operator. The technical operator was a technician at Battelle with three years of field and
laboratory experience and a B.A. degree. The non-technical operator was a part-time temporary
helper enrolled in undergraduate studies. Because the reagents of the Quick™ II test kits were
consumed in use, it was not feasible for the two operators to use the same kits; however, each
operator used multiple kits in order to analyze all the samples and it is assumed that kit-to-kit
variability was similar for both operators. Results of all analyses were statistically compared to
evaluate operator bias. The technical operator analyzed all samples using two Quick™ Arsenic
Scan units and two Compu-Scan systems to assess inter-unit reproducibility.
The rate of false positive and false negative results were evaluated relative to the 10-ppb
maximum contaminant level for arsenic in drinking water.(4) Other factors that were qualitatively
assessed during the test included time required for sample analysis, ease of use, and reliability.
3.3 Test Samples
Three types of samples were analyzed in the verification test, as shown in Table 3-1: quality
control (QC) samples, performance test (PT) samples, and environmental water samples. The QC
and PT samples were prepared from National Institute of Standards and Technology (NIST)
traceable standards purchased from a commercial supplier and subject only to dilution as
appropriate. Under the Safe Drinking Water Act, the EPA lowered the maximum contaminant
level for arsenic from 50 ppb to 10 ppb in January 2001; public water systems must comply with
this standard by January 2006.(4) Therefore, the QC sample concentrations targeted the 10 ppb
arsenic level. The PT samples ranged from 10% to 1,000% of the 10 ppb level (i.e., from 1 ppb
to 100 ppb). The environmental water samples were collected from various drinking water and
surface freshwater sources.
Each sample was assigned a unique sample identification number when prepared in the
laboratory or collected in the field. The PT and environmental samples were submitted blind to
the technical and non-technical operators and were analyzed randomly to the degree possible.
3.3.1 QC Samples
QC samples included laboratory reagent blank (RB) samples, quality control samples (QCS), and
laboratory-fortified matrix (LFM) samples (Table 3-1). The RB samples consisted of the same
ASTM Type I water used to prepare all other samples and were subjected to the same handling
and analysis procedures as the other samples. The RB samples were used to verify that no
arsenic contamination was introduced during sample handling and analysis. RB samples were
analyzed at a frequency of 10%.
5
-------
Table 3-1. Test Samples for Verification of the Quick™ II Test Kit
Type of
Arsenic
No. of
Sample
Sample Characteristics
Concentration^
Replicates
Quality
Control
Reagent Blank (RB)
~ 0 ppb
10% of all
Quality Control Sample (QCS)
10 ppb
10% of all
Laboratory Fortified Matrix (LFM)
10 ppb above
native level
1 per site
Performance
Prepared arsenic solution
1 ppb
4
Test
Prepared arsenic solution
3 ppb
4
Prepared arsenic solution
10 ppb
4
Prepared arsenic solution
30 ppb
4
Prepared arsenic solution
100 ppb
4
Prepared arsenic solution for detection limit
15 ppb
7
determination
Prepared arsenic solution spiked with low levels of
interfering substances
10 ppb
4
Prepared arsenic solution spiked spiked with high
levels of interfering substances
10 ppb
4
Environmental
Battelle drinking water
<0.5 ppb
4
Ayer untreated water
64.8 ppb
4
Ayer treated water
1.39 ppb
4
Falmouth Pond water
<0.5 ppb
4
Taunton River water
1.31 ppb
4
/a\
Performance test sample concentrations are target concentrations; environmental sample concentrations are actual
(average of four replicate measurements).
The QCS consisted of ASTM Type I water spiked in the lab to a concentration of 10 ppb arsenic
with a NIST-traceable standard. QCS were used as calibration checks to verify that the Quick™
II test kit was operating properly. QCS were analyzed at the beginning and end of each testing
period, as well as after every tenth sample. Because the test kit utilized a color chart that could
not be calibrated, no performance criteria were specified for the QCS.
The LFM samples consisted of aliquots of environmental samples that were spiked in the field to
increase the arsenic concentration by 10 ppb. The spike solution used for the LFM samples was
prepared in the laboratory and brought to the field site. One LFM sample was prepared from
each environmental sample.
6
-------
3.3.2 PT Samples
Three types of PT samples used in this verification test (Table 3-1): spiked samples ranging
from 1 ppb to 100 ppb arsenic, a low-level spiked sample for evaluation of the test kit's detection
limit, and matrix interference samples that were spiked with potential interfering substances. All
PT samples were prepared in the laboratory using ASTM Type I water and NIST-traceable
standards.
Five PT samples containing arsenic at concentrations from 1 ppb to 100 ppb were prepared to
evaluate Quick™ II test kit accuracy and linearity. Four aliquots of each of these samples were
analyzed to assess precision.
To determine the detection limit of the Quick™ II test kit, a PT sample was prepared with an
arsenic concentration approximately five times the manufacturer's estimated detection level.
Seven non-consecutive replicates of this 15 ppb arsenic sample were analyzed to provide
precision data with which to estimate the method detection limit (MDL).
The matrix interference samples were spiked with 10 ppb arsenic as well as potentially
interfering substances commonly found in natural water samples. One sample contained low
levels of interfering substances that consisted of 1 ppm iron, 3 ppm sodium chloride, and
0.1 ppm sulfide. The second sample contained high levels of interfering compounds at the
following concentrations: 10 ppm iron, 30 ppm sodium chloride, and 1.0 ppm sulfide. Four
replicates of each of these samples were analyzed.
3.3.3 Environmental Samples
The environmental samples listed in Table 3-1 included three drinking water samples and two
surface water samples. All environmental samples were collected in 20-L high density
polyethylene (HDPE) carboys. The Battelle drinking water sample was collected directly from a
tap without purging. Untreated and treated groundwater samples from the Ayer, Massachusetts
Department of Public Works Water Treatment Plant were collected directly from spigots, also
without purging. Four aliquots of each sample were analyzed using the Quick™ II test kit in the
Battelle laboratory as soon as possible after collection. One aliquot of each sample was preserved
with nitric acid and submitted to the reference laboratory for reference analysis.
One surface water sample was collected from a pond in Falmouth, Massachusetts and another
was collected from the Taunton River near Bridgewater, Massachusetts. These samples were
collected near the shoreline by submerging a 2-L HDPE sample container no more than one inch
below the surface of the water, and decanting the water into a 20-L HDPE carboy until full. Each
water body was sampled at one accessible location. These samples could not be analyzed at the
field location as planned because of persistent, severe winter weather conditions. Therefore, the
samples were returned to a storage shed at the Battelle laboratory, which was heated but not
serviced by running water. The storage shed was intended to simulate realistic field conditions
under which the test kits might be used. Four aliquots of each surface water sample were
analyzed in the storage shed as soon as possible after collection. One aliquot of each sample was
preserved with nitric acid and submitted to the reference laboratory for reference analysis.
7
-------
3.4 Reference Analysis
The reference arsenic analyses were performed in a Battelle laboratory using a Perkin Elmer
Sciex Elan 6000 ICPMS according to EPA Method 200.8, Revision 5.5.(2) The sample was
introduced through a peristaltic pump by pneumatic nebulization into a radiofrequency plasma
where energy transfer processes caused desolvation, atomization, and ionization. The ions were
extracted from the plasma through a pumped vacuum interface and separated on the basis of their
mass-to-charge ratio by a quadrupole mass spectrometer. The ions transmitted through the
quadrupole were registered by a continuous dynode electron multiplier, and the ion information
was processed by a data handling system.
The ICPMS was tuned, optimized, and calibrated daily. The calibration was performed using a
minimum of five calibration standards at concentrations ranging between 0.5 and 250 ppb, and a
required correlation coefficient of a minimum of 0.999. Internal standards were used to correct
for instrument drift and physical interferences. These standards were introduced in line through
the peristaltic pump and analyzed with all blanks, standards, and samples.
3.5 Verification Schedule
The verification test took place from January 28 through February 24, 2003. Table 3-2 shows the
daily activities that were conducted during this period. The reference analyses were performed
on March 7 and March 14, 2003, five to six weeks after sample collection.
Table 3-2. Schedule of Verification Test Days
Sample Analysis
Sample
Date
Collection
Tech.
Non-tech.
Testing
Date
Op.
Op.
Location
Activity
1/28/03-
1/28/03-
1/29/03-
Battelle
Preparation and analysis of PT and
2/10/03
2/10/03
2/10/03
Laboratory
associated QC samples.
2/12/03
2/13/03
2/13/03
Battelle
Collection and analysis of Ayer untreated
Laboratory
and treated water and associated QC
samples.
2/17/03
2/17/03
2/19/03
Battelle
Collection and analysis of Battelle drinking
Laboratory
water and associated QC samples.
2/21/03
2/21/03
2/21/03
Battelle
Collection and analysis of Falmouth Pond
Storage Shed
water and associated QC samples.
2/23/03
2/24/03
2/24/03
Battelle
Collection and analysis of Taunton River
Storage Shed
water and associated QC samples.
8
-------
Chapter 4
Quality Assurance/Quality Control
Quality assurance/quality control (QA/QC) procedures were performed in accordance with the
quality management plan (QMP) for the AMS Center(3) and the test/QA plan for this verification
test.(1) QA/QC procedures and results are described below.
4.1 Laboratory QC for Reference Method
Reference analyses were conducted on March 7 and March 14, 2003. Laboratory QC for the
reference method included the analysis of RB, QCS, LFM and analytical duplicate samples.
Laboratory RB samples were analyzed to ensure that no contamination was introduced by the
sample preparation and analysis process. The test/QA plan stated that if arsenic was detected in a
RB sample above the MDL for the reference instrument, then the contamination source would be
identified and removed and proper blank readings achieved before proceeding with the reference
analyses. All of the laboratory RB samples analyzed were below the reporting limit for arsenic
(i.e., below the concentration of the lowest calibration standard) except for several blanks that
were analyzed at the end of the day on March 7. Three of the six test samples that were
associated with these RB samples were re-analyzed on March 14, with acceptable RB sample
results. The other three test samples had arsenic concentrations that were approximately twenty
times higher than the RB sample concentrations; therefore, no action was taken.
On March 7, the instrument used for the reference method was calibrated using nine calibration
standards, with concentrations ranging from 0.5 to 250 ppb arsenic. On March 14, it was
calibrated using eight standards ranging in concentration from 0.1 to 25 ppb arsenic for more
accurate analysis of low level samples. The accuracy of the calibration was verified after the
analysis of every ten samples by analyzing a QCS of a known concentration. The percent
recovery of the QCS was calculated from the following equation:
R = — xlOO ^
s
where Cs is the measured concentration of the QCS and 5 is the spike concentration. If the QCS
analysis differed by more than 10% from the true value of the standard, the instrument was
recalibrated before continuing the test. As shown in Table 4-1, all QCS analyses were within the
required range.
9
-------
Table 4-1. Reference Method QCS Analysis Results
Measured
Actual
Percent
Sample ID
Analysis Date
(ppb)
(ppb)
Recovery
CCV25
3/7/2003
24.96
25.00
100%
QCS 25
3/7/2003
26.81
25.00
107%
CCV25
3/7/2003
24.50
25.00
98%
CCV25
3/7/2003
25.39
25.00
102%
CCV25
3/7/2003
25.73
25.00
103%
CCV25
3/7/2003
25.81
25.00
103%
CCV25
3/7/2003
25.64
25.00
103%
CCV25
3/7/2003
25.30
25.00
101%
CCV 25
3/7/2003
24.90
25.00
100%
CCV 25
3/7/2003
22.67
25.00
91%
QCS 25
3/14/2003
24.90
25.00
100%
CCV 2.5
3/14/2003
2.74
2.50
110%
QCS 2.5
3/14/2003
2.70
2.50
108%
CCV 2.5
3/14/2003
2.58
2.50
103%
CCV 2.5
3/14/2003
2.65
2.50
106%
CCV 2.5
3/14/2003
2.66
2.50
106%
CCV 2.5
3/14/2003
2.61
2.50
104%
CCV 2.5
3/14/2003
2.60
2.50
104%
LFM samples were analyzed to assess whether matrix effects influenced the reference method
results. The LFM percent recovery (R) was calculated from the following equation:
R = °s ~C xlOO (2)
s
where Cs is the measured concentration of the spiked sample, C is the measured concentration of
the unspiked sample, and 5 is the spike concentration. If the percent recovery of an LFM fell
outside the range from 85% to 115%, a matrix effect was suspected. As shown in Table 4-2, all
of the LFM sample results were within this range.
Duplicate samples were analyzed to assess the precision of the reference analysis. The relative
percent difference (RPD) of the duplicate sample analysis was calculated from the following
equation:
RPD= (°~C^ xlOO (3)
(C + Cd)/2
10
-------
Table 4-2. Reference Method LFM Sample Results
Sample ID
Matrix
Analysis
Date
Unspiked
(PPb)
Spiked
(ppb)
Amount
Spiked
(ppb)
Percent
Recovery
ASTM Type I
CAA-22
water
3/7/2003
11.02
37.20
25.00
105%
ASTM Type I
CAA-25 R4
water
3/7/2003
0.95
22.76
25.00
87%
ASTM Type I
CAA-28 R2
water
3/7/2003
3.45
30.64
25.00
109%
ASTM Type I
CAA-29 R4
water
3/7/2003
34.98
60.37
25.00
102%
CAA-37 R4
Drinking water
3/7/2003
0.52
28.20
25.00
111%
CAA-41 R4
Drinking water
3/7/2003
1.24
28.88
25.00
111%
CAA-48
Surface water
3/7/2003
12.26
39.40
25.00
109%
CAA-47 R4
Surface water
3/7/2003
1.07
28.41
25.00
109%
ASTM Type I
CAA-27 R1
water
3/14/2003
2.56
4.73
2.50
87%
CAA-37 R3
Drinking water
3/14/2003
0.45
3.11
2.50
107%
CAA-47 R1
Surface water
3/14/2003
1.36
4.16
2.50
112%
CAA-88 R3
Drinking water
3/14/2003
0.43
3.16
2.50
109%
CAA-88 R4
Drinking water
3/14/2003
0.42
3.18
2.50
111%
Where C is the concentration of the sample analysis, and Cd is the concentration of the duplicate
sample analysis. If the RPD was greater than 10%, the instrument was recalibrated before con-
tinuing the test. As shown in Table 4-3, the RPDs for the duplicate analysis were all less than
10%. The RPD for one duplicate pair was 9.5%; however, the reported concentrations were
below the reporting limit for the reference method (i.e., below the concentration of the lowest
calibration standard).
4.2 Audits
Three types of audits were performed during the verification test: a performance evaluation (PE)
audit of the reference method, a technical systems audit of the verification test performance, and
a data quality audit. Audit procedures are described further below.
4.2.1 Performance Evaluation Audit
A PE audit was conducted to assess the quality of the reference measurements made in this
verification test. For the PE audit, an independent, NIST-traceable, reference material was
obtained from a different commercial supplier than the calibration standards and the standard
used to prepare the PT and field QCS samples. Accuracy of the reference method was verified by
comparing the arsenic concentration measured using the calibration standards to that obtained
11
-------
Table 4-3. Reference Method Duplicate Analysis Results
Sample
Duplicate
Relative
Sample
Analysis
Concentration
Concentration
Percent
ID
Date
(ppb)
(ppb)
Difference
CAA-4
3/7/2003
9.33
9.20
1.4%
CAA-70
3/7/2003
10.93
10.82
1.0%
CAA-26 R1
3/7/2003
1.14
1.13
1.4%
CAA-28 R3
3/7/2003
3.49
3.45
1.1%
CAA-31 R1
3/7/2003
111.89
112.20
0.3%
CAA-38
3/7/2003
11.96
11.90
0.5%
CAA-42
3/7/2003
13.02
13.06
0.3%
CAA-4 8
3/7/2003
12.26
12.22
0.4%
CAA-23
3/14/2003
3.03
2.99
1.3%
CAA-27 R2
3/14/2003
2.64
2.61
0.9%
CAA-37 R4
3/14/2003
0.44
0.43
2.3%
CAA-47 R2
3/14/2003
1.31
1.32
0.2%
CAA-88 R4
3/14/2003
0.42
0.38
9.5%
using the independently-certified PE standard. Relative percent difference as calculated by
Equation 3 was used to quantify the accuracy of the results. Agreement of the standard within
10% was required for the measurements to be considered acceptable. As shown in Table 4-4, the
PE sample analysis was within the required range.
Table 4-4. Reference Method PE Audit Results
Measured Arsenic
Actual Arsenic
Date of
Concentration
Concentration
Percent
Sample ID
Analysis
(PPb)
(PPb)
Difference
PE-1
3/24/03
9.63
10.0
4
4.2.2 Technical Systems Audit
An independent Battelle Quality staff conducted a technical systems audit (TSA) on February 6
to ensure that the verification test was being conducted in accordance with the test/QA plan(1)
and the AMS Center QMP.(3) A TSA of the reference method performance was conducted by the
Battelle Quality Manager on March 5, 2003, when the reference analyses were initiated. As part
of the TSA, test procedures were compared to those specified in the test/QA plan, data
acquisition and handling procedures were reviewed, and the reference standards and method
were reviewed. Observations and findings from the TSA were documented and submitted to the
Battelle Verification Test Coordinator for response. None of the findings of the TSA required
corrective action. TSA records are permanently stored with the Battelle Quality Manager.
12
-------
4.2.3 Data Quality Audit
At least 10% of the data acquired during the verification test were audited. Battelle's Quality
Manager traced the data from the initial acquisition, through reduction and statistical analysis, to
final reporting to ensure the integrity of the reported results. All calculations performed on the
data undergoing the audit were checked.
4.3 QA/QC Reporting
Each audit was documented in accordance with Sections 3.3.4 and 3.3.5 of the QMP for the ETV
AMS Center.(3) Once the audit reports were prepared, the Battelle Verification Test Coordinator
ensured that a response was provided for each adverse finding or potential problem and imple-
mented any necessary follow-up corrective action. The Battelle Quality Manager ensured that
follow-up corrective action was taken. The results of the TSA and the data quality audit were
submitted to the EPA.
4.4 Data Review
Records generated in the verification test received a one-over-one review before these records
were used to calculate, evaluate, or report verification results. Table 4-5 summarizes the types of
data recorded and reviewed. All data were recorded by Battelle staff. Data were reviewed by a
Battelle technical staff member involved in the verification test, but not the staff member that
originally generated the record. The person performing the review added his/her initials and the
date to a hard copy of the record being reviewed. Review of some of the test data sheets occurred
outside of the two week period specified in the test/QA plan.
13
-------
Table 4-5. Summary of Data Recording Process
Data to be Recorded Where Recorded How Often Recorded Disposition of Data(a)
Dates, times of test
events
ETV field data
sheets
Start/end of test event
Used to organize/check test
results; manually
incorporated in data
spreadsheets as necessary
Test parameters
(temperature, analyte/
interferant identities,
and all Quick™ II
test kit results for
color chart, Quick™
Arsenic Scan and
Compu-Scan
ETV field data
sheets
When set or changed, or
as needed to document
test
Used to organize/check test
results, manually
incorporated in data
spreadsheets as necessary
Reference method
sample analysis,
chain of custody, and
results
Laboratory record
books, data
sheets, or data
acquisition
system, as
appropriate
Throughout sample
handling and analysis
process
Transferred to spreadsheets
(a) All activities subsequent to data recording were carried out by Battelle.
14
-------
Chapter 5
Statistical Methods
The statistical methods used to evaluate the performance factors listed in Section 3.2 are
presented in this chapter. Qualitative observations were also used to evaluate verification test
data.
5.1 Accuracy
All samples were analyzed by both the Quick™ II test kit and reference methods. For each
sample, accuracy was expressed in terms of a relative bias (B) as calculated from the following
equation:
B = =x\00 (4)
where d is the average difference between the reading from the Quick™ II test kit and those
from the reference method, and CR is the average of the reference measurements. An additional
assessment of accuracy was conducted for the color chart results because of the semi-quantitative
nature of the visual comparisons. Each color in the chart represents a concentration range.
Performance was assessed by determining whether the result falls within the expected
concentration range as measured by the reference analysis. Overall agreement was assessed by
calculating the percent of results that fell within the correct range, calculated from the following
equation:
A = — x 100 (5)
n
where A is the percent of measurements in agreement, 7is the number of measurements within
the expected color range, and n is the total number of measurements. Readings below the
vendor-stated detection limit of the test kit (e.g., <2 ppb) were judged to be in agreement with the
reference result if the reference value was in the specified "less than" range.
15
-------
5.2 Precision
When possible, the standard deviation (S) of the results for the replicate samples was calculated
and used as a measure of Quick™ II test kit precision at each concentration. Standard deviation
was calculated from the following equation:
S =
1
n -
t2X
1 k=1
-cy
(6)
where n is the number of replicate samples, C/, is the concentration measured for the kth sample,
and C is the average concentration of the replicate samples. Precision was reported in terms of
the relative standard deviation (RSD) as follows:
RSD =
S_
c
x 100
(7)
5.3 Linearity
Linearity was assessed by performing a linear regression of Quick™ II test kit results against the
reference results, with linearity characterized by the slope, intercept, and correlation coefficient
(R). Linearity was tested using the five PT samples over the range 1 to 100 ppb arsenic. Samples
with results below the vendor-stated detection limit of the test kit were not included in the
analysis. Color chart, Quick™ Arsenic Scan, and Compu-Scan results were plotted against the
corresponding reference concentrations and separate regressions were performed.
5.4 Method Detection Limit
The MDL for the Quick™ II test kit was assessed using results from all three detection methods
(color chart, Quick™ Arsenic Scan, and Compu-Scan) for seven replicate analyses of a sample
spiked with approximately 15 ppb arsenic. The standard deviation of the seven replicate samples
was calculated using Equation 6. The MDL was calculated using the following equation:
MDL = txS (8)
where t is the Student's t value for a 99% confidence level and S is the standard deviation of the
seven replicate samples.
16
-------
5.5 Matrix Interference Effects
The potential effect of interfering substances on the sensitivity of the Quick™ II test kit was
evaluated by the calculating accuracy (expressed as bias) using Equation 4. These results were
qualitatively compared with accuracy results for PT samples containing only arsenic to assess
whether there was a positive or negative effect due to matrix interferences.
5.6 Operator Bias
Potential operator bias for the Quick™ II test kit was assessed by performing a linear regression
of sample results above the detection limit generated by the technical and non-technical operator.
Color chart, Quick™ Arsenic Scan, and Compu-Scan results were evaluated. The slope,
intercept, and correlation coefficient were used to evaluate the degree of operator bias. A paired
t-test was also conducted to evaluate whether the two sets of sample results were significantly
different at a 95% confidence level.
5.7 Inter-Unit Reproducibility
Inter-unit reproducibility for the Quick™ II test kit with the Quick™ Arsenic Scan and the
Compu-Scan was assessed by performing a linear regression of sample results generated by the
two units of each device used by the technical operator. The slope, intercept, and correlation
coefficient were used to evaluate the degree of inter-unit reproducibility. A paired t-test was also
conducted to evaluate whether the two sets of sample results were significantly different at a
95% confidence level.
5.8 Rate of False Positives/False Negatives
The rates of false positives and false negatives produced by the Quick™ II test kit were assessed
relative to the 10-ppb target arsenic level. A false positive result is defined as any result reported
to be greater than the guidance level (10 ppb) and greater than 125% of the reference value,
when the reference value is less than or equal to the guidance level. Similarly, a false negative
result is defined as any result reported below the guidance level and less than 75% of the
reference value, when the reference value is equal to or greater than the guidance level. The rates
of false positives and false negatives were expressed as a percentage of total samples analyzed
for each type of sample.
17
-------
Chapter 6
Test Results
The results of the verification test of the Quick™ II test kits are presented in this section.
6.1 QC Samples
As described in Section 3.3.1, the QC samples analyzed with the Quick™ II test kit included RB
samples, QCS, and LFM samples (these QC samples were different than those analyzed in
conjunction with the reference method). The RB samples were analyzed at a frequency of 10%
and results were used to verify that no arsenic contamination was introduced during sample
handling and analysis. QCS were analyzed at the beginning and end of each test period, and after
every tenth sample. The QCS results were used to verify that the test kit was operating properly.
One LFM sample was prepared from each environmental sample to evaluate potential matrix
interferences. Acceptance criteria for test kit QC samples were not specified in the test/QA plan
because modifications to the technology would not be made during testing.
RB sample results for the technical and non-technical operators are presented in Tables 6-la and
6-lb, respectively. Unique sample identification codes were assigned to each container of ASTM
Type I water that was used. The RB samples were analyzed at the required frequency. The
technical and non-technical operators reported all RB samples as below the detection limit for
the color chart and Quick™ Arsenic Scan. The Compu-Scan units almost always returned a
detected value for the RBs because a detection limit was not provided by the manufacturer.
Because all color chart and Quick™ Arsenic Scan results for the RB samples were below
detection, it appeared that arsenic contamination resulting from sample handling and analysis did
not occur.
QCS results for the technical and non-technical operators are presented in Tables 6-2a and 6-2b,
respectively. The QCS were analyzed at the required frequency except on the first day of testing,
when the technical operator inadvertently omitted these samples. The percent recovery of the
QCS was calculated using Equation 1 (Section 4.1). The QCS percent recoveries for the
technical operator ranged from 35% to 100% for the color chart, from 31% to 101% for the
Quick™ Arsenic Scan, and from 9% to 112% for the Compu-Scan. The QCS percent recoveries
for the non-technical operator ranged from 25% to 100% for the color chart, from 10% to 101%
for the Quick™ Arsenic Scan, and from 16% to 124%> for the Compu-Scan. On average, percent
recoveries were more than 20% lower than the true value, which was lower than expected.
18
-------
Table 6-la. RB Sample Results for the Technical Operator
Color
Quick™ Arsenic
Quick™ Arsenic
Compu-Scan
Compu-Scan
Chart
Scan #1
Scan #2
#1
#2
Sample ID
Replicate
Analysis Date
(ppb)
(ppb)
(ppb)
(ppb)(a)
(ppb)
CAA-3
1
1/28/2003
<2
<1
<1
-
0.7
CAA-8
1
1/31/2003
<2
<1
<1
1.4
2.9
CAA-11
1
2/3/2003
<2
<1
<1
1.5
4.3
CAA-13
1
2/5/2003
<2
<1
<1
1.4
2.8
CAA-13
2
2/5/2003
<2
<1
<1
1
2
CAA-50
1
2/7/2003
<2
<1
<1
1.9
2.3
CAA-52
1
2/10/2003
<2
<1
<1
0.7
0.9
CAA-53
1
2/13/2003
<2
<1
<1
0.9
1.2
CAA-54
1
2/13/2003
<2
<1
<1
0.8
0.8
CAA-56
1
2/17/2003
<2
<1
<1
0.9
4.9
CAA-58
1
2/21/2003
<2
<1
<1
1.7
1.2
CAA-59
1
2/24/2003
<2
<1
<1
1.4
1
(a) Software failed to operate.
-------
Table 6-lb. RB Sample Results for the Non-Technical Operator
Quick™ Arsenic
Arsenic
Color Chart
Scan #1
Compu-Scan #1
Sample ID
Replicate
Analysis Date
(ppb)
(ppb)
(ppb)
CAA-3
1
1/29/2003
<2
<1
1.4
CAA-8
1
1/31/2003
<2
<1
0.0
CAA-11
1
2/3/2003
<2
<1
3
CAA-13
1
2/5/2003
<2
<1
1.7
CAA-13
2
2/5/2003
<2
<1
1.8
CAA-50
1
2/7/2003
<2
<1
3
CAA-52
1
2/10/2003
<2
<1
1.1
CAA-53
1
2/13/2003
<2
<1
0.9
CAA-54
1
2/13/2003
<2
<1
1.2
CAA-57
1
2/19/2003
<2
<1
1.2
CAA-58
1
2/21/2003
<2
<1
2
CAA-59
1
2/24/2003
<2
<1
1.6
-------
Table 6-2a. QCS Results for the Technical Operator(a)
Sample
ID
Replicate
Analysis
Date
Color
Chart
(PPb)
Quick™
Arsenic
Scan #1
(PPb)
Quick™
Arsenic
Scan #2
(PPb)
Compu-
Scan #1
(PPb)
Compu-
Scan #2
(PPb)
Amount
Spiked
(PPb)
Percent
Recovery
Color
Chart
Percent
Recovery
Quick™
Arsenic
Scan #1
Percent
Recovery
Quick™
Arsenic
Scan #2
Percent
Recovery
Compu-
Scan #1
Percent
Recovery
Compu-
Scan #2
CAA-4
1
1/29/2003
10
9.4
8.4
7.6
12.9
10.0
100%
94%
84%
76%
129%
CAA-15
1
1/31/2003
7
5.6
5.4
7.7
5.3
10.0
70%
56%
54%
77%
53%
CAA-15
2
1/31/2003
10
9.7
9.2
6.6
9.8
10.0
100%
97%
92%
66%
98%
CAA-16
1
2/3/2003
7
4.4
4.2
5.1
7
10.0
70%
44%
42%
51%
70%
CAA-16
2
2/3/2003
6
3.7
4.2
7.2
7.3
10.0
60%
37%
42%
72%
73%
CAA-17
1
2/5/2003
8
5.2
4.9
5.3
6.3
10.0
80%
52%
49%
53%
63%
CAA-17
2
2/5/2003
10
10.1
9.7
8.1
12.2
10.0
100%
101%
97%
81%
122%
CAA-17
3
2/7/2003
4
3.5
3.3
2.5
5
10.0
40%
35%
33%
25%
50%
CAA-18
1
2/7/2003
6
4.4
4.4
2.3
3.6
10.0
60%
44%
44%
23%
36%
CAA-19
1
2/10/2003
9
5.6
5.8
5.4
8.2
10.0
90%
56%
58%
54%
82%
CAA-19
2
2/10/2003
10
10.4
10.7
10.3
17.2
10.0
100%
104%
107%
103%
172%
CAA-20
1
2/13/2003
5
3.7
3.7
3.6
5.4
10.0
50%
37%
37%
36%
54%
CAA-20
2
2/13/2003
10
7.3
7.5
8
10
10.0
100%
73%
75%
80%
100%
CAA-21
1
2/17/2003
7
5.2
5.2
0.9
4.4
10.0
70%
52%
52%
9%
44%
CAA-21
2
2/17/2003
4
3.3
3.3
2.1
2.8
10.0
40%
33%
33%
21%
28%
CAA-70
1
2/21/2003
6
4.2
4.9
5.2
5.2
10.0
60%
42%
49%
52%
52%
CAA-70
2
2/21/2003
5
3.1
3.5
2.4
2.6
10.0
50%
31%
35%
24%
26%
CAA-72
1
2/24/2003
3.5
3.1
3.3
3.5
3.4
10.0
35%
31%
33%
35%
34%
CAA-72
2
2/24/2003
10
8.9
8.9
11
11.8
10.0
100%
89%
89%
110%
118%
¦7T) ;
The technical operator inadvertently omitted QCS samples on the first day of testing.
-------
Table 6-2b. QCS Results for the Non-Technical Operator
Sample ID
Replicate
Analysis
Date
Color
Chart
(ppb)
Quick™
Arsenic
Scan #1
(ppb)
Compu-
Scan #1
(ppb)
Amount
Spiked
(ppb)
Percent
Recovery
Color
Chart
Percent
Recovery
Quick™
Arsenic
Scan #1
Percent
Recovery
Compu-
Scan #1
CAA-4
1
1/29/2003
10
8.4
7.9
10.0
100%
84%
79%
CAA-4
2
1/29/2003
10
8.7
8.2
10.0
100%
87%
82%
CAA-15
1
1/31/2003
10
8.9
8.1
10.0
100%
89%
81%
CAA-15
2
1/31/2003
10
9.7
7.7
10.0
100%
97%
77%
CAA-16
1
2/3/2003
6
5.4
6.2
10.0
60%
54%
62%
CAA-16
2
2/3/2003
7
8.2
6.9
10.0
70%
82%
69%
CAA-17
1
2/5/2003
7
5.8
4.3
10.0
70%
58%
43%
CAA-17
2
2/5/2003
10
7.8
12.3
10.0
100%
78%
123%
CAA-17
3
2/7/2003
6
5.6
8.2
10.0
60%
56%
82%
CAA-18
1
2/7/2003
6
6
3.8
10.0
60%
60%
38%
CAA-19
1
2/10/2003
2.5
1
1.6
10.0
25%
10%
16%
CAA-19
2
2/10/2003
7
7.5
7.4
10.0
70%
75%
74%
CAA-20
1
2/13/2003
6
4.9
5.3
10.0
60%
49%
53%
CAA-20
2
2/13/2003
10
5.4
9.3
10.0
100%
54%
93%
CAA-22
1
2/19/2003
7
2.9
3.7
10.0
70%
29%
37%
CAA-22
2
2/19/2003
7
4.9
8.3
10.0
70%
49%
83%
CAA-70
1
2/21/2003
10
5.8
6
10.0
100%
58%
60%
CAA-70
2
2/21/2003
10
6
6.1
10.0
100%
60%
61%
CAA-72
1
2/24/2003
5
4.9
6.8
10.0
50%
49%
68%
CAA-72
2
2/24/2003
10
10.1
12.4
10.0
100%
101%
124%
-------
The LFM results for the technical and non-technical operators are presented in Tables 6-3 a and
6-3b. The percent recovery associated with each LFM sample was calculated using Equation 2
(Section 4.1). Reference method results are also provided for comparison.
Table 6-3a. LFM Sample Results for the Technical Operator
Unspiked(a)
Spiked
Amount Spiked
Percent
Description
(ppb)
(ppb)
(ppb)
Recovery
Battelle drinking water LFM
Color Chart
<2
6
10
60%
Quick™ Arsenic Scan # 1
<1
4.9
10
49%
Quick™ Arsenic Scan #2
<1
5.2
10
52%
Compu-Scan # 1
1.1
4
10
29%
Compu-Scan #2
1.3
5
10
37%
Reference
<0.5
11.96
10
120%
Ayer untreated water LFM
Color Chart
25
35
10
100%
Quick™ Arsenic Scan # 1
14.5
23.5
10
90%
Quick™ Arsenic Scan #2
16.25
26
10
98%
Compu-Scan # 1
18.9
35.5
10
166%
Compu-Scan #2
30.5
42.5
10
120%
Reference
64.82
69.74
10
49%
Ayer treated water LFM
Color Chart
<2
6
10
60%
Quick™ Arsenic Scan # 1
<1
4.4
10
44%
Quick™ Arsenic Scan #2
<1
4.4
10
44%
Compu-Scan # 1
1.45
4.6
10
32%
Compu-Scan #2
1.55
11
10
95%
Reference
1.39
13.02
10
116%
Falmouth Pond water LFM
Color Chart
<2
5
10
50%
Quick™ Arsenic Scan # 1
<1
4.2
10
42%
Quick™ Arsenic Scan #2
<1
3.9
10
39%
Compu-Scan # 1
1.9
3.9
10
20%
Compu-Scan #2
2.75
5.4
10
27%
Reference
<0.5
11.50
10
115%
Taunton River water LFM
Color Chart
<2
6
10
60%
Quick™ Arsenic Scan # 1
<1
4.7
10
47%
Quick™ Arsenic Scan #2
<1
4.9
10
49%
Compu-Scan # 1
2.25
6.9
10
47%
Compu-Scan #2
2.6
11
10
84%
Reference
1.31
12.26
10
109%
(a) Average of four replicates. Non-detects were assigned a value of zero.
23
-------
Table 6-3b. LFM Sample Results for the Non-Technical Operator
Unspiked(a)
Spiked
Amount Spiked
Description
(ppb)
(ppb)
(ppb)
Percent Recovery
Battelle drinking water LFM
Color Chart
<2
7
10
70%
Quick™ Arsenic Scan # 1
<1
6.8
10
68%
Compu-Scan # 1
1.6
6.1
10
45%
Reference
<0.5
11.96
10
120%
Ayer untreated water LFM
Color Chart
20.3
20
10
-3%
Quick™ Arsenic Scan # 1
18.6
15.5
10
-31%
Compu-Scan # 1
24.9
23.5
10
-14%
Reference
64.82
69.74
10
49%
Ayer treated water LFM
Color Chart
<2
3
10
30%
Quick™ Arsenic Scan # 1
<1
2
10
20%
Compu-Scan # 1
1.7
2.3
10
6%
Reference
1.39
13.02
10
116%
Falmouth Pond water LFM
Color Chart
<2
7
10
70%
Quick™ Arsenic Scan # 1
<1
6.6
10
66%
Compu-Scan # 1
2.5
16.2
10
137%
Reference
<0.5
11.50
10
115%
Taunton River water LFM
Color Chart
<2
6
10
60%
Quick™ Arsenic Scan # 1
<1
4.7
10
47%
Compu-Scan # 1
2.7
7
10
43%
Reference
1.31
12.26
10
109%
(a) Average of four replicates. Non-detects were assigned a value of zero.
No evidence of matrix interferences is clearly indicated by these results; although particularly
low recoveries were measured by the non-technical operator for the Ayer untreated water LFM
sample, these results were not confirmed by the technical operator.
6.2 PT and Environmental Samples
Table 6-4 presents the sample results for the PT and environmental samples. The table includes
the Quick™ II test kit results and the reference method results. The Quick™ II test kit results are
shown for both the technical and non-technical operators, the Quick™ Arsenic Scan Units #1
and #2, and the Compu-Scan Units #1 and #2. Some Quick™ II test kit results were below the
detection limit and were assigned a value of <2 ppb for the color chart and <1 ppb for the
Quick™ Arsenic Scan. No reporting limit was assigned by the manufacturer for the Compu-Scan
system. The reporting limit for the reference analyses was 0.5 ppb, which corresponds to the
lowest calibration standard used. Results for each performance factor are presented below.
24
-------
6.2.1 Accuracy
Table 6-5 presents the accuracy results for the Quick™ II test kit, expressed as percent bias as
calculated by Equation 4 (Section 5.1). Percent bias was not calculated for results below the
detection limit. The four replicate analyses for each sample were averaged in the calculation of
bias. The relative bias for the color chart ranged from -61% to 10% for the technical operator and
from -77% to 96% for the non-technical operator. The relative bias for the Quick™ Arsenic Scan
ranged from -78% to -4% for the technical operator and from -85% to -22% for the non-technical
operator. The relative bias for the Compu-Scan ranged from -71% to 96% for the technical
operator and from -82% to 108%) for the non-technical operator. The low arsenic concentration
in the Taunton River water sample resulted in high positive biases for the Compu-Scan results.
Table 6-6 presents accuracy results for each PT and environmental replicate sample according to
whether the color chart result agreed with the reference value for that sample. Each color block
on the color chart represents a range of concentrations. The reference sample result was assigned
to the correct corresponding color block. A test kit result was considered to agree with the
reference method result if it fell within the range of plus or minus one color block (i.e., the
concentration range spanning three adjacent color blocks). If the color chart test result for a given
sample was within this range, then a "Y" was reported in Table 6-6. If the color chart result was
outside this range, then an "N" was reported. Overall agreement was determined by calculating
the total percent of results in agreement for the technical and non-technical operators. The total
percent agreement using this method was 68% for the technical operator and 72% for the non-
technical operator.
6.2.2 Precision
Precision results for the Quick™ II test kit are presented in Table 6-7. The RSD was determined
according to Equation 7 (Section 5.2). The RSD was not calculated if any of the results for a set
of replicates were below the detection limit (i.e., <2 ppb for the color chart or <1 ppb for the
Quick™ Arsenic Scan). For the technical operator, RSDs ranged from 16% to 24% for the color
chart, lP/o to 44% for the Quick™ Arsenic Scan, and 10% to 58% for the Compu-Scan. For the
non-technical operator, RSDs ranged from 0% to 38% for the color chart, 13% to 38% for the
Quick™ Arsenic Scan, and 16% to 108%> for the Compu-Scan. For the reference measurements,
RSDs were a maximum of 4%.
25
-------
Table 6-4. Quick™ II Test Kit and Reference Sample Results
Technical
Technical
Non-Technical
Non-
Technical
Operator
Operator
Technical
Technical
Non-
Operator
Technical
Operator
Color
Quick™
Arsenic Scan
Quick™
Arsenic Scan
Operator
Compu-
Operator
Compu-Scan
Technical
Operator
Quick™
Arsenic Scan
Operator
Compu-
Sample
Chart
#1
#2
Scan #1
#2
Color Chart
#1
Scan #1
Reference
Description
ID
Replicate
(ppb)
(ppb)
(ppb)
(ppb)
(ppb)
(ppb)
(ppb)
(ppb)
(ppb)
PT - 1 ppb As
CAA-26
1
<2
<1
<1
0.4
1.2
2.50
<1
1.5
0.91
CAA-26
2
<2
<1
<1
1
2
2.50
<1
1.2
0.86
CAA-26
3
<2
<1
<1
0.4
1.2
2
<1
0.9
0.90
CAA-26
4
<2
<1
<1
0.7
1
2
1
1.2
0.86
PT - 3 ppb As
CAA-28
1
2.50
1.7
1.3
1.9
2.3
4
2.3
1.2
2.56
CAA-28
2
2.50
<1
2.9
1.3
2.5
4
2
2.7
2.64
CAA-28
3
3.50
2.9
3.1
2.1
2.9
4
1.7
0.9
2.50
CAA-28
4
3
2.9
2.6
1.9
2.8
4
3.1
1.9
2.71
PT - 10 ppb As
CAA-1
1
10
9.4
9.4
8.3
15.2
10
3.5
7.8
9.09
CAA-1
2
10
9.4
9.2
12.5
13
10
6.6
10.2
8.95
CAA-1
3
7
7.8
7.3
6.5
11
10
8.2
7.1
8.83
CAA-1
4
7
7.8
7.5
6.3
11.6
10
6
7.7
8.99
PT - 30 ppb As
CAA-29
1
30 ®
25 (a)
29 (a)
30(a)
44.5 (a)
12 (a)
11 ®
15®
33.96
CAA-29
2
20 ®
10 00
15.5 (a)
8 (a)
17 (a)
20 (a)
18 (a)
19®
34.39
CAA-29
3
17.50 (a)
11.5 (a)
11.5 (a)
15.5 (a)
16(a)
30 (a)
22 (a)
18.5 (a)
34.51
CAA-29
4
25 W
14.5 (a)
16.5 (a)
12.5 (a)
21.5 (a)
30 (a)
26 (a)
31.5 (a)
34.98
PT - 100 ppb As
CAA-31
1
100 (b)
OO
to
78 (b)
107 (b)
OO
70 (a)
74.5 (a)
99 (a)
111.89
CAA-31
2
80®
56®
54®
54 (b)
75 ®
60 (a)
57 (a)
78 ®
115.57
CAA-31
3
120 ®
117(h)
97 0>)
131 ®
199®
60 (a)
64 (a)
91 ®
114.65
CAA-31
4
100®
84 ®
75 ®
96 (b)
213 ®
60 (a)
74.5 (a)
68 ®
113.83
Detection limit
CAA-24
1
8
6.8
7
7.5
9
7
4.9
15.2
CAA-24
2
8
6.3
6.3
5.2
12.3
10
10.4
14.6
CAA-24
3
12
8.2
8.4
8.6
24.4
10
11
11.7
CAA-24
4
6
4.4
4.4
8.2
8.5
10
9.2
8.4
14.18
CAA-24
5
7
6.6
6.3
7.8
10.4
10
9.2
12.1
CAA-24
6
7
6.8
7.3
6.6
11.6
10
9.2
9.1
CAA-24
7
5
4.2
3.9
6.5
7.2
10
9.2
9.1
-------
Table 6-4. Quick™ II Test Kit and Reference Sample Results (continued)
Non-
Technical
Technical
Non-
Technical
Non-
Technical
Operator
Operator
Technical
Technical
Technical
Operator
Technical
Operator
Color
Quick™
Arsenic
Quick™
Arsenic
Operator
Compu-
Operator
Compu-
Operator
Color
Quick™
Arsenic
Operator
Compu-
Sample
Chart
Scan #1
Scan #2
Scan #1
Scan #2
Chart
Scan #1
Scan #1
Reference
Description
ID
Replicate
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
PT- lOppbAs
CAA-33
1
6
4.4
4.4
4.2
4.4
7
6.6
4.6
+ low level
CAA-33
2
8
6.3
6
9.8
11.5
7
5.8
5.9
9.90
interferents
CAA-33
3
7
5.8
6
7.9
9.6
7
5.6
2.8
CAA-33
4
7
4.2
4.2
3.5
6
7
6.3
5.4
PT - lOppbAs
CAA-35
1
13
8.4
8.7
8.2
14.9
7
6
8.8
+ high level
interferents
CAA-35
CAA-35
2
3
13
15
9.4
10.1
9.7
10.1
11.5
8.5
12.9
11.4
10
10
7.8
12.8
10
20.3
11.59
CAA-35
4
10
8.2
8.7
6.5
10.2
10
9.4
12
Battelle
CAA-37
1
<2
<1
<1
0.8
1.2
<2
<1
1.9
<0.5
drinking water
CAA-37
2
<2
<1
<1
1.5
1.2
<2
<1
1.5
<0.5
CAA-37
3
<2
<1
<1
1
1.7
<2
<1
1.9
<0.5
CAA-37
4
<2
<1
<1
1.2
1
<2
<1
1.2
<0.5
Battelle
drinking water
CAA-38
1
6
4.9
5.2
4
5
7
6.8
6.1
11.96
LFM
Ayer untreated
CAA-39
1
30 (a)
17.5 (a)
18.5 (a)
23.1 (a)
34(a)
17.5 (a)
10(a)
11.5 (a)
65.61
water
CAA-39
2
20 (a)
10(a)
13 (a)
10.5 (a)
18 (a)
17.5 (a)
15.5 (a)
12.5 (a)
62.73
CAA-39
3
30 (a)
22 (a)
23.5 (a)
29 (a)
52 (a)
25 (a)
24.5 (a)
65 (a)
67.47
CAA-39
4
20 (a)
8.5 (a)
10(a)
13 (a)
18 (a)
21 (a)
24.5 (a)
10.5 (a)
63.48
Ayer untreated
water LFM
CAA-40
1
35 (a)
23.5 (a)
26 (a)
35.5 (a)
42.5 (a)
20 (a)
15.5 (a)
23.5 (a)
69.74
Ayer treated
CAA-41
1
<2
<1
<1
1.5
1.4
<2
<1
1.9
1.36
water
CAA-41
2
<2
<1
<1
1.3
1.3
<2
<1
1.2
1.45
CAA-41
3
<2
<1
<1
1.7
2.5
<2
<1
1.8
1.44
CAA-41
4
<2
<1
<1
1.3
1
<2
<1
1.8
1.32
Ayer treated
water LFM
CAA-42
1
6
4.4
4.4
4.6
11
3
2
2.3
13.02
Falmouth Pond
CAA-43
1
<2
<1
<1
1.9
3.3
<2
1
2.8
<0.5
water
CAA-43
2
<2
<1
<1
2.3
2.7
<2
1.7
1.8
<0.5
CAA-43
3
<2
<1
<1
2.3
2.8
<2
<1
2.3
<0.5
CAA-43
4
<2
<1
<1
1.2
2.2
<2
<1
3
<0.5
-------
Table 6-4. Quick™ II Test Kit and Reference Sample Results (continued)
Non-
Technical
Technical
Non-
Technical
Non-
Technical
Operator
Operator
Technical
Technical
Technical
Operator
Technical
Operator
Quick™
Quick™
Operator
Operator
Operator
Quick™
Operator
Color
Arsenic
Arsenic
Compu-
Compu-
Color
Arsenic
Compu-
Sample
Chart
Scan #1
Scan #2
Scan #1
Scan #2
Chart
Scan #1
Scan #1
Reference
Description
ID
Replicate
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
(PPb)
Falmouth Pond
water LFM
CAA-46
1
5
4.2
3.9
3.9
5.4
7
6.6
16.2
11.50
Taunton River
CAA-47
1
<2
<1
<1
1.9
2.7
<2
1.3
2.3
1.36
water
CAA-47
2
<2
<1
<1
2.4
2.2
<2
<1
3.2
1.31
CAA-47
3
<2
<1
<1
2.5
2.9
<2
<1
2.4
1.31
CAA-47
4
<2
<1
<1
2.2
2.5
<2
<1
3
1.26
Taunton River
water LFM
CAA-48
1
6
4.7
4.9
6.9
11
6
4.7
7
12.26
^ = 1:5 dilution; '-b> = 1:10 dilution.
to
00
-------
Table 6-5. Quantitative Evaluation of Accuracy for Quick™ II Test Kits
Bias
Non-
Technical
Technical
Non-
Technical
Non-
Technical
Operator
Operator
Technical
Technical
Technical
Operator
Technical
Operator
Quick™
Quick™
Operator
Operator
Operator
Quick™
Operator
Description
Color
Arsenic
Arsenic
Compu-
Compu-
Color
Arsenic
Compu-
Chart
Scan #1
Scan #2
Scan #1
Scan #2
Chart
Scan #1
Scan #1
Performance Test Samples
1 ppb As
NA
NA
NA
-45%
18%
96%
NA
5%
3 ppb As
-20%
NA
-31%
-50%
-27%
11%
-37%
-54%
10 ppb As
-5%
-4%
-7%
-6%
42%
12%
-32%
-9%
30 ppb As
-33%
-56%
-47%
-52%
-28%
-33%
-44%
-39%
100 ppb As
-12%
-26%
-33%
-15%
39%
-45%
-41%
-26%
10 ppb As + low level interferents
-29%
-48%
-48%
-36%
-20%
-29%
-39%
-53%
10 ppb As + high level interferents
10%
-22%
-20%
-25%
7%
-20%
-22%
10%
Environmental Samples
Battelle drinking water
NA
NA
NA
NA
NA
NA
NA
NA
Battelle drinking water LFM
-50%
-59%
-57%
-67%
-58%
-41%
-43%
-49%
Ayer untreated water
-61%
-78%
-75%
-71%
-53%
-69%
-71%
-62%
Ayer untreated water LFM
-50%
-66%
-63%
-49%
-39%
-71%
-78%
-66%
Ayer treated water
NA
NA
NA
4%
11%
NA
NA
20%
Ayer treated water LFM
-54%
-66%
-66%
-65%
-15%
-77%
-85%
-82%
Falmouth Pond water
NA
NA
NA
NA
NA
NA
NA
NA
Falmouth Pond water LFM
-57%
-63%
-66%
-66%
-53%
-39%
-43%
41%
Taunton River water
NA
NA
NA
71%
96%
NA
NA
108%
Taunton River water LFM
-51%
-62%
-60%
-44%
-10%
-51%
-62%
-43%
Percent bias calculated according to Equation 4, Section 5.1.
NA: one or more replicates below detection limit
-------
Table 6-6. Qualitative Evaluation of Agreement for Quick™ II Test Kits
Within Range
(Y/N)
Technical
Within Range
(Y/N)
Non-Technical
Description
Sample ID
Replicate
Operator
Color Chart
Operator Color
Chart
Performance Test Samples
1 ppb As
CAA-26
1
Y
Y
CAA-26
2
Y
Y
CAA-26
3
Y
Y
CAA-26
4
Y
Y
3 ppb As
CAA-28
1
N
Y
CAA-28
2
N
Y
CAA-28
3
Y
Y
CAA-28
4
Y
Y
10 ppb As
CAA-1
1
Y
Y
CAA-1
2
Y
Y
CAA-1
3
Y
Y
CAA-1
4
Y
Y
30 ppb As
CAA-29
1
Y
N
CAA-29
2
N
Y
CAA-29
3
N
Y
CAA-29
4
N
Y
100 ppb As
CAA-31
1
Y
Y
CAA-31
2
N
N
CAA-31
3
Y
N
CAA-31
4
Y
N
10 ppb As +
CAA-33
1
N
Y
low level
CAA-33
2
Y
Y
interferents
CAA-33
3
Y
Y
CAA-33
4
Y
Y
10 ppb As +
CAA-35
1
Y
N
high level
CAA-35
2
Y
Y
interferents
CAA-35
3
N
Y
CAA-35
4
Y
Y
Environmental Samples
CAA-37
1
Y
Y
Battelle drinking water
CAA-37
CAA-37
2
3
Y
Y
Y
Y
CAA-37
4
Y
Y
Battelle drinking water LFM
CAA-38
1
N
N
CAA-39
1
N
N
Ayer untreated water
CAA-39
CAA-39
2
3
N
N
N
N
CAA-39
4
N
N
Ayer untreated water LFM
CAA-40
1
N
N
30
-------
Table 6-6. Qualitative Evaluation of Accuracy for Quick™ II Test Kits (continued)
Within
Within
Range (Y/N)
Technical
Range (Y/N)
Non-
Technical
Description
Sample ID
Replicate
Operator
Color Chart
Operator
Color Chart
CAA-41
1
Y
Y
Ayer treated water
CAA-41
CAA-41
2
3
Y
Y
Y
Y
CAA-41
4
Y
Y
Ayer treated water LFM
CAA-42
1
N
N
CAA-43
1
Y
Y
Falmouth Pond water
CAA-43
CAA-43
2
3
Y
Y
Y
Y
CAA-43
4
Y
Y
Falmouth Pond water LFM
CAA-46
1
N
N
CAA-47
1
Y
Y
Taunton River water
CAA-47
CAA-47
2
3
Y
Y
Y
Y
CAA-47
4
Y
Y
Taunton River water LFM
CAA-48
1
N
N
Percent Agreement
68%
72%
6.2.3 Linearity
The linearity of the Quick™ II test kit readings was assessed by performing a linear regression of
the test kit results against the reference method results for the five PT samples ranging from
1 ppb to 100 ppb arsenic. In these regressions, the 1 ppb PT sample results were not used
because they were reported as below the detection limit. Figures 6-1, 6-2 and 6-3 present the
results of the linear regressions for the color chart, Quick™ Arsenic Scan and the Compu-Scan
results, respectively. The slope, intercept and correlation coefficient for each regression equation
are shown on the charts. For all three detection methods (color chart, Quick™ Arsenic Scan and
Compu-Scan), the results for the non-technical operator were more linear than the results for the
technical operator, although the results for the technical operator corresponded more closely to
the reference method results.
31
-------
Table 6-7. Precision Results for Quick™ II Test Kits
RSD
Non-
Technical
Technical
Non-
Technical
Non-
Technical
Operator
Operator
Technical
Technical
Technical
Operator
Technical
Operator
Quick™
Quick™
Operator
Operator
Operator
Quick™
Operator
Description
Color
Arsenic
Arsenic
Compu-
Compu-
Color
Arsenic
Compu-
Reference
Chart
Scan #1
Scan #2
Scan #1
Scan #2
Chart
Scan #1
Scan #1
Method
Performance Test Samples
1 ppb As
NA
NA
NA
46%
33%
13%
NA
20%
3%
3 ppb As
17%
28%
33%
19%
10%
0%
26%
48%
4%
10 ppb As
20%
11%
13%
34%
15%
0%
32%
17%
1%
30 ppb As
24%
44%
42%
58%
54%
38%
33%
34%
1%
100 ppb As
16%
29%
23%
33%
39%
8%
13%
16%
1%
Environmental Samples
Battelle drinking
water
NA
NA
NA
27%
23%
NA
NA
21%
NA
Ayer untreated
water
23%
44%
37%
46%
53%
18%
38%
108%
3%
Ayer treated
water
NA
NA
NA
13%
42%
NA
NA
19%
4%
Falmouth Pond
water
NA
NA
NA
27%
16%
NA
37%
22%
NA
Taunton River
water
NA
NA
NA
12%
12%
NA
NA
16%
3%
NA: one or more replicates below detection limit.
-------
y = 0.88x -1.82
R =
40 60 80
Reference concentration (ppb)
~ Technical Operator
¦ Non-technical Operator
- - - Linear (Technical Operator)
— — Linear (Non-technical Operator)
Figure 6-1. Linearity of Quick™ II Color Chart Results
Reference concentration (ppb)
Figure 6-2. Linearity of Quick™ Arsenic Scan Results
~ Technical Operator Unit #1
~ Technical Operator Unit #2
¦ Non-technical Operator Unit #1
- - - Linear (Technical Operator Unit #1)
Linear (Technical Operator Unit #2)
— — Linear (Non-technical Operator Unit #1)
33
-------
Reference concentration (ppb)
Figure 6-3. Linearity of Compu-Scan Results
~ TechnicalOp erator Un it # 1
~ TechnicalOp erator Un it #2
¦ Non-technical Operator Unit #1
- - - Linear (Technical Operator Unit #1)
Linear (Technical Operator Unit #2)
~~ — Linear (Non-technical Operator Unit #1)
6.2.4 Method Detection Limit
The MDL was assessed by analyzing seven replicates of a sample spiked at approximately
15 ppb arsenic. Table 6-8 provides the standard deviation for the seven replicate samples for the
color chart, Quick ™ Arsenic Scan and Compu-Scan results, and the calculated MDLs.
Table 6-8. Detection Limit Results for Quick™ II Test Kit
Sample ID
Technical
Operator
Color
Chart
(ppb)
Technical
Operator
Quick™
Arsenic
Scan #1
(ppb)
Technical
Operator
Quick™
Arsenic
Scan #2
(ppb)
Technical
Operator
Compu-
Scan #1
(ppb)
Technical
Operator
Arsenic
Compu-
Scan #2
(ppb)
Non-
Technical
Operator
Color
Chart
(ppb)
Non-
Technical
Operator
Quick™
Arsenic
Scan #1
(ppb)
Non-
Technical
Operator
Compu-
Scan #1
(ppb)
CAA-24 Rep 1
8
6.8
7
7.5
9
7
4.9
15.2
CAA-24 Rep 2
8
6.3
6.3
5.2
12.3
10
10.4
14.6
CAA-24 Rep 3
12
8.2
8.4
8.6
24.4
10
11
11.7
CAA-24 Rep 4
6
4.4
4.4
8.2
8.5
10
9.2
8.4
CAA-24 Rep 5
7
6.6
6.3
7.8
10.4
10
9.2
12.1
CAA-24 Rep 6
7
6.8
7.3
6.6
11.6
10
9.2
9.1
CAA-24 Rep 7
5
4.2
3.9
6.5
7.2
10
9.2
9.1
Standard
Deviation
2.23
1.42
1.59
1.17
5.79
1.13
1.95
2.73
Method Detection
Limit (ppb)
7.0
4.5
5.0
3.7
18.2
3.6
6.1
8.6
34
-------
6.2.5 Matrix Interference Effects
Matrix interference effects were assessed by comparing the calculated bias for the samples
containing low-level and high-level concentrations of interfering substances with the bias
reported for the other PT samples (Table 6-5). For both operators, bias data for the low- and
high- interferent samples were consistent with the bias data for PT samples containing arsenic
only, indicating that the presence of interfering substances did not affect the recovery of arsenic.
6.2.6 Operator Bias
Operator bias was evaluated by comparing the color chart, Quick™ Arsenic Scan Unit #1, and
Compu-Scan Unit #1 results above the detection limit for all PT and environmental samples
produced by the technical and non-technical operators (the non-technical operator did not use the
Quick™ Arsenic Scan Unit #2 or Compu-Scan Unit #2). Linear regression results for the three
sets of data are shown in Figure 6-4. The plots indicate that the color chart, Quick™ Arsenic
Scan, and Compu-Scan results tended to be higher for the technical operator than for the non-
technical operator. Paired t-tests of the three sets of data indicated that the results were not
significantly different at a 5% significance level for the Quick™ Arsenic Scan and Compu-Scan.
Color chart results for the technical and non-technical operators were significantly different at a
5% significance level.
A
Color Chart
¦
Quick Arsenic Scan Unit #1
~
Compu-Scan Unit #1
- "
- Linear (Color Chart)
— Linear (Quick Arsenic Scan Unit #1)
— Linear (Compu-Scan Unit #1)
Figure 6-4. Comparison of Quick™ II Test Results for Technical and Non-Technical
Operators
60 80 100 120 140
Technical Operator
35
-------
6.2.7 Inter-Unit Reproducibility
Inter-unit reproducibility was evaluated by comparing the data for the two Quick™ Arsenic Scan
units and the two Compu-Scan units used by the technical operator. Only results above the
detection limit were included in the analysis. Linear regressions of the two sets of data for each
instrument are shown in Figure 6-5. The regression line for the Quick™ Arsenic Scan almost
exactly corresponded to the 1:1 line, indicating that the performance of the two units was very
similar. The data for the Compu-Scan units showed more scatter, and the position of the regres-
sion line indicates that Unit #2 tended to return higher results than Unit #1. Paired t-tests of the
two sets of data indicated that the Quick™ Arsenic Scan results were not significantly different
at a 5% significance level. The results for the two Compu-Scan units were significantly different.
250
200
y = 1,62x - 0.77
R = 0.9751
150
1:1 line
100
50
y = 0.88x + 1.70
R = 0.9946
0
20
40
60
80
100
120
140
Unit #1
~ Quick Arsenic Scan
¦ Compu-Scan
- - - Linear (Quick Arsenic Scan)
— "Linear (Compu-Scan)
Figure 6-5. Inter-Unit Reproducibility for the Quick™ Arsenic Scan and
Compu-Scan Units
36
-------
6.2.8 Rate of False Positives/False Negatives
Tables 6-9 and 6-10 show the data and results for the rates of false positives and false negatives,
respectively, obtained from the Quick™ II test kit. All PT and environmental samples were
included in this evaluation.
As shown in Table 6-9, 32 samples had an arsenic concentration below 10 ppb as measured by
the reference analysis. For these samples, none of the color chart or Quick™ Arsenic Scan
results were >10 ppb and greater than 125% of the reference measurement, yielding false
positive rates of 0% for the technical and non-technical operators. The rate of false positives for
the Compu-Scan was 3% and 9% for the technical operator (Units #1 and #2, respectively) and
0% for the non-technical operator (Unit #1).
Twenty-one samples had arsenic concentrations above 10 ppb as measured by the reference
analysis (Table 6-10). For these samples, the test kit color chart results were <10 ppb and less
than 75% of the reference measurement for four samples for the technical operator and five for
the non-technical operator, yielding false negative rates of 19% and 24%, respectively. The rates
of false negatives for the Quick™ Arsenic Scan units were 33% and 19% for the technical
operator (Units #1 and #2, respectively) and 29% for the non-technical operator (Unit #1). The
rates of false negatives for the Compu-Scan units were 38% and 10% for the technical operator
(Units #1 and #2, respectively) and 14% for the non-technical operator (Unit #1).
6.3 Other Factors
During testing activities, the technical and non-technical operators were instructed to keep a
record of their comments on ease of use, reliability, portability, and generation of waste
materials. This section summarizes these observations and other comments pertaining to any
problems encountered during testing. Cost information is also presented.
6.3.1 Ease of Use
The technical and non-technical operators both reported that the Quick™ II test kit was very easy
to use. The test kit instructions were clear and easy to follow. Although the manufacturer
provided instructions for diluting samples above the 15 ppb arsenic level, the non-technical
operator sometimes had difficulty successfully performing dilutions and correctly converting the
results to a final concentration. Dilution of samples with arsenic concentrations exceeding the
optimal detection range may be a source of error and reduce the accuracy and precision of the
associated results because of the difficulty in performing accurate dilution in a field setting. The
three scoops used to sequentially add reagents were color coordinated, which facilitated the
efficient operation of the test kit. The sample bottles were of moderate size and were relatively
easy to handle, with little spillage of reagents. Extra care had to be taken to ensure that the caps
to the reaction vessels were completely dry before proceeding with further analyses. The test kit
materials were readily transported to the Battelle storage shed where environmental samples
were tested.
37
-------
4>
H
'3
a
.©
3
«
H
False Positive (Y/N)
Non-
Technical
Operator
Compu-Scan
#1
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Non-
Technical
Operator
Quick™
Arsenic
Scan #1
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Non-
Technical
Operator
Color
Chart
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Technical
Operator
Compu-
Scan #2
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Technical
Operator
Compu-
Scan #1
£ £ £ £
£ £ £ £
£ ^ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Technical
Operator
Quick™
Arsenic
Scan #2
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Technical
Operator
Quick™
Arsenic
Scan #1
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Technical
Operator
Color Chart
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
£ £ £ £
Replicate
'—1 (N m
'—1 (N m
' *3"
¦—1 (N m *3-
¦—1 (N m *3-
Sample ID
CAA-25
CAA-25
CAA-25
CAA-25
CAA-27
CAA-27
CAA-27
CAA-27
CAA-1
CAA-1
CAA-1
CAA-1
CAA-37
CAA-37
CAA-37
CAA-37
CAA-41
CAA-41
CAA-41
CAA-41
CAA-43
CAA-43
CAA-43
CAA-43
Description
1 ppb As
3 ppb As
10 ppb As
Battelle
drinking water
Ayer treated
water
Falmouth Pond
water
38
-------
Table 6-9. Rate of False Positives for Quick™ II Test Kits (continued)
False Positive (Y/N)
Non-
Technical
Technical
Non-
Technical
Non-
Operator
Operator
Technical
Technical
Technical
Operator
Technical
Technical
Quick™
Quick™
Operator
Operator
Operator
Quick™
Operator
Operator
Arsenic
Arsenic
Compu-
Compu-
Color
Arsenic
Compu-Scan
Description
Sample ID
Replicate
Color Chart
Scan #1
Scan #2
Scan #1
Scan #2
Chart
Scan #1
#1
Taunton River
CAA-47
1
N
N
N
N
N
N
N
N
water
CAA-47
2
N
N
N
N
N
N
N
N
CAA-47
3
N
N
N
N
N
N
N
N
CAA-47
4
N
N
N
N
N
N
N
N
10 ppb As +
CAA-33
1
N
N
N
N
N
N
N
N
low level
CAA-33
2
N
N
N
N
N
N
N
N
interferents
CAA-33
3
N
N
N
N
N
N
N
N
CAA-33
4
N
N
N
N
N
N
N
N
Total number of samples
32
32
32
32
32
32
32
32
Total number of false positives
0
0
0
1
3
0
0
0
Percent false positives
0%
0%
0%
3%
9%
0%
0%
0%
U>
-------
Table 6-10. Rate of False Negatives for Quick™ II Test Kits
False Negative (Y/N)
Non-
Technical
Technical
Non-
Technical
Non-
Description
Sample ID
Replicate
Technical
Operator
Color
Chart
Operator
Quick™
Arsenic
Scan #1
Operator
Quick™
Arsenic
Scan #2
Technical
Operator
Compu-
Scan #1
Technical
Operator
Compu-
Scan #2
Technical
Operator
Color
Chart
Operator
Quick™
Arsenic
Scan #1
Technical
Operator
Compu-
Scan #1
30 ppb As
CAA-29
1
N
N
N
N
N
N
N
N
CAA-29
2
N
N
N
Y
N
N
N
N
CAA-29
3
N
N
N
N
N
N
N
N
CAA-29
4
N
N
N
N
N
N
N
N
100 ppb As
CAA-31
1
N
N
N
N
N
N
N
N
CAA-31
2
N
N
N
N
N
N
N
N
CAA-31
3
N
N
N
N
N
N
N
N
CAA-31
4
N
N
N
N
N
N
N
N
Battelle drinking
water LFM
CAA-38
1
Y
Y
Y
Y
Y
Y
Y
Y
Ayer untreated
CAA-39
1
N
N
N
N
N
N
N
N
water
CAA-39
2
N
N
N
N
N
N
N
N
CAA-39
3
N
N
N
N
N
N
N
N
CAA-39
4
N
Y
N
N
N
N
N
N
Ayer untreated
water LFM
CAA-40
1
N
N
N
N
N
N
N
N
Ayer treated water
LFM
CAA-42
1
Y
Y
Y
Y
N
Y
Y
Y
Falmouth Pond
CAA-46
1
N
water LFM
Y
Y
Y
Y
Y
Y
Y
Taunton River
CAA-48
1
Y
water LFM
Y
Y
Y
Y
N
Y
Y
10 ppb As +
high level
interferents
CAA-35
CAA-35
CAA-35
1
2
3
N
N
N
Y
N
N
N
N
N
Y
N
Y
N
N
N
Y
N
N
Y
Y
N
N
N
N
CAA-35
4
N
Y
N
Y
N
N
N
N
Total number of samples
21
21
21
21
21
21
21
21
Total number of false negatives
4
7
4
8
2
5
6
3
Percent false negatives
19%
33%
19%
38%
10%
24%
29%
14%
-------
6.3.2 Sample Analysis Time
The average total analysis time for a sample was about 15 minutes at a sample temperature of
24°C. The manufacturer provided a modified protocol that specified increased reaction times for
samples below 24°C. The test kit enabled two samples to be run concurrently without any
confusion.
6.3.3 Reliability
The Quick™ II test kits operated reliably throughout the period of the test.
6.3.4 Waste Material
The waste generated by the Quick™ II test kit was manageable. The vendor's instructions
provide a warning that hydrogen and arsine are generated during the test and recommend that
testing be conducted in a well-ventilated area away from open flames and other sources. MSDSs
should be reviewed before handling any chemicals. Instructions for the disposal of residual
materials were clear and complete. The residual liquid in the reaction vessel was allowed to settle
before disposal in order to let particulates accumulate on the bottom. A dilute hydrochloric acid
solution was used to clean the reaction vessel prior to subsequent analyses. Disposal of this waste
in an appropriate manner must be taken into consideration.
6.3.5 Cost
The listed price for a Quick™ II test kit for analysis of 50 samples is $219.99. Replacement
reagents and supplies are not available; kits are provided as a complete set because reagents, test
strips, and color charts are made to perform optimally with each other. The Quick™ Arsenic
Scan and Compu-Scan are available as options for an additional cost of $1,599.99 each.
41
-------
Chapter 7
Performance Summary
The Quick™ II test kit was verified by evaluating the following parameters:
¦ Accuracy
¦ Precision
¦ Linearity
¦ MDL
¦ Matrix interference effects
¦ Operator bias
¦ Inter-unit reproducibility
¦ Rate of false positives/false negatives.
The quantitative assessment of accuracy indicated that the relative bias for the color chart ranged
from -61% to 10% for the technical operator and from -77% to 96% for the non-technical
operator. The relative bias for the Quick™ Arsenic Scan ranged from -78% to -4% for the
technical operator and from -85% to -22% for the non-technical operator. The relative bias for
the Compu-Scan ranged from -71% to 96% for the technical operator and from -82% to 108%)
for the non-technical operator. The overall agreement for the color chart results based on an
assessment of whether the result was assigned to the correct color block indicated that the total
percent agreement was 68% for the technical operator and 72% for the non-technical operator.
Precision was assessed by analyzing four replicates of each sample. For the technical operator,
precision expressed as a RSD ranged from 16% to 24% for the color chart, 11% to 44% for the
Quick™ Arsenic Scan and 10% to 58% for the Compu-Scan. For the non-technical operator,
RSDs ranged from 0% to 38% for the color chart, 13% to 38% for the Quick™ Arsenic Scan and
16%) to 108%) for the Compu-Scan. These results exclude samples where one or more of the
replicate results was not detected by the test kit.
The linearity of response was evaluated by plotting the test kit results against the reference
analysis results for the PT samples. The equations for the linear regressions that were performed
to evaluate linearity are summarized in Table 7-1. The slope, y-intercept, and correlation
coefficient corresponding to a linear response that exactly matched reference concentrations
would be 1.0, 0, and 1.0, respectively.
42
-------
Table 7-1. Summary of Linear Regression Equations for Test Kit and Reference Results
Description
Slope
Intercept
Correlation
Coefficient
®
Color chart, technical operator
0.88
-1.82
0.9779
Color chart, non-technical operator
0.52
3.37
0.9822
Quick™ Arsenic Scan #1, technical operator
0.75
-2.42
0.9340
Quick™ Arsenic Scan #2, technical operator
0.66
-0.30
0.9565
Quick™ Arsenic Scan #1, non-technical operator
0.59
0.095
0.9828
Compu-Scan Unit # 1, technical operator
0.85
-2.67
0.9301
Compu-Scan Unit #2, technical operator
1.39
-5.12
0.9117
Compu-Scan Unit # 1, non-technical operator
0.73
-0.55
0.9787
The MDL was assessed by analyzing seven replicates of a sample spiked at a level approxi-
mately five times the manufacturer's estimated detection limit for the color chart (approximately
15 ppb). The MDLs calculated using the precision data from these replicates ranged from
3.6 ppb to 7 ppb for the color chart, 4.5 ppb to 6.1 ppb for the Quick™ Arsenic Scan, and
3.7 ppb to 18.2 ppb for the Compu-Scan.
Results for samples containing low and high levels of interfering substances indicated that the
presence of interfering substances did not affect the detection of arsenic.
An evaluation of Quick™ II test kit results for the technical and non-technical operators
suggested that measurements for the color chart, Quick™ Arsenic Scan, and Compu-Scan made
by the technical operator tended to be higher than for the non-technical operator. Paired t-tests of
the three sets of data indicated that the results were not significantly different at a 5% signifi-
cance level for the Quick™ Arsenic Scan and Compu-Scan results. Color chart results for the
technical and non-technical operators were significantly different. The regression equations were
as follows:
Color chart ^ = 0.57x + 4.56, R = 0.9538
Quick™ Arsenic Scan y = 0.70x + 4.14, R= 0.9270
Compu-Scan y = 0.79x + 3.21, R = 0.9173
where x is the technical operator and .y is the non-technical operator.
Inter-unit reproducibility was evaluated by comparing the data for the two Quick™ Arsenic Scan
units and two Compu-Scan systems used by the technical operator. The Quick™ Arsenic Scan
results almost exactly corresponded, indicating that the performance of the two units was very
similar. The data for the Compu-Scan units showed more scatter, and Unit #2 tended to return
higher results than Unit #1. Paired t-tests of the two sets of data indicated that the Quick™
43
-------
Arsenic Scan results were not significantly different at a 5% significance level whereas the
results for the two Compu-Scan units were significantly different. The regression equations were
as follows:
Quick™ Arsenic Scan
Compu-Scan
y = 0.88x + 1.70, R = 0.9946
1.62x-0.77, R= 0.9751
where x is Unit #1 and y is Unit #2.
A false positive was defined as a test kit result that was greater than 10 ppb and greater than
125% of the reference concentration, when the reference concentration is less than or equal to 10
ppb. The rates of false positives for the technical and non-technical operators using the color
charts and Quick™ Arsenic Scan units were all 0%. The rates of false positives for the Compu-
Scan units were 3% and 9% for the technical operator (Units #1 and #2) and 0% for the non-
technical operator. A false negative was defined as a test kit result that was below 10 ppb and
less than 75% of the reference concentration, when the reference concentration was greater than
or equal to 10 ppb. The false negative rates for the non-technical and technical operators using
the color charts were 19% and 24%, respectively. The rates of false negatives for the Quick™
Arsenic Scan units were 33% and 19% for the technical operator (Units #1 and #2, respectively)
and 29% for the non-technical operator (Unit #1). The rates of false negatives for the Compu-
Scan units were 38% and 10% for the technical operator (Units #1 and #2, respectively) and was
14%) for the non-technical operator (Unit #1).
The Quick™ II test kits were easy to use and readily transportable to the field. The time to
analyze one sample is approximately 15 minutes at a temperature range of 24°C to 30°C; longer
reaction times are required for samples below this range. Two samples can be run concurrently
without difficulty. The sample bottles were of moderate size and were relatively easy to handle.
The test kit components were reliable. Dilution of samples with arsenic concentrations exceeding
the optimal detection range may be a source of error and reduce the accuracy and precision of the
associated results because of the difficulty in performing accurate dilution in a field setting. The
cost for a 50-sample test kit with a color chart is listed as $219.99. Replacement reagents and
supplies are not available; kits are provided as a complete set because reagents, test strips, and
color charts are made to perform optimally with each other, according to the vendor. The
Quick™ Arsenic Scan and Compu-Scan are available as options for an additional cost of
$1,599.99.
44
-------
Chapter 8
References
1. Test/QA Plan for Verification of Portable Analyzers, Battelle, Columbus, Ohio, Version 2.0.
December 8, 2000.
2. U.S. EPA Method 200.8, Determination of Trace Elements in Waters and Wastes by
Inductively Coupled Plasma Mass Spectrometry, Revision 5.5, October 1999.
3. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Pilot, Version
4.0, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus, Ohio,
December, 2002.
4. U.S. Code of Federal Regulations, Title 40, Part 136, Appendix B.
45
------- |