August 2006
Environmental Technology
Verification Report
TRACEDETECT
SAFEGUARD TRACE METAL ANALYZER
Prepared by
Battelle
Baltelle
I "he Husiness *.*/ Innovation
Under a cooperative agreement with
^jf CrTr\ U.S. Environmental Protection Agency
ET1/ET1/ET1/
-------
-------
August 2006
Environmental Technology Verification
Report
ETV Advanced Monitoring Systems Center
Trace Detect
SafeGuard Trace Metal Analyzer
by
Anne Gregg
Tom Kelly
Zachary Willenberg
Amy Dindal
Karen Riggs
Battelle
Columbus, Ohio 43201
-------
Notice
The U.S. Environmental Protection Agency (EPA), through its Office of Research and
Development, has financially supported and collaborated in the extramural program described
here. This document has been peer reviewed by the Agency. Mention of trade names or
commercial products does not constitute endorsement or recommendation by the EPA for use.
11
-------
Foreword
The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the
nation's air, water, and land resources. Under a mandate of national environmental laws, the
Agency strives to formulate and implement actions leading to a compatible balance between
human activities and the ability of natural systems to support and nurture life. To meet this
mandate, the EPA's Office of Research and Development provides data and science support that
can be used to solve environmental problems and to build the scientific knowledge base needed
to manage our ecological resources wisely, to understand how pollutants affect our health, and to
prevent or reduce environmental risks.
The Environmental Technology Verification (ETV) Program has been established by the EPA to
verify the performance characteristics of innovative environmental technology across all media
and to report this objective information to permitters, buyers, and users of the technology, thus
substantially accelerating the entrance of new environmental technologies into the marketplace.
Verification organizations oversee and report verification activities based on testing and quality
assurance protocols developed with input from major stakeholders and customer groups
associated with the technology area. ETV consists of six environmental technology centers.
Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/.
Effective verifications of monitoring technologies are needed to assess environmental quality
and to supply cost and performance data to select the most appropriate technology for that
assessment. Under a cooperative agreement, Battelle has received EPA funding to plan,
coordinate, and conduct such verification tests for "Advanced Monitoring Systems for Air,
Water, and Soil" and report the results to the community at large. Information concerning this
specific environmental technology area can be found on the Internet at http://www.epa.gov/etv/
centers/center 1 .html.
in
-------
Acknowledgments
The authors wish to acknowledge the support of all those who helped plan and conduct the
verification test, analyze the data, and prepare this report. We would like to thank
Kenneth M. Hill and Paul J. Ponturo, Suffolk County Department of Health Services;
Marty Link, Water Quality Division, Nebraska Department of Environmental Quality; and
Jeff Adams, U.S. Environmental Protection Agency National Risk Management Research
Laboratory for their careful review of this verification report.
IV
-------
Contents
Notice ii
Foreword iii
Acknowledgments iv
Contents v
List of Abbreviations viii
List of Abbreviations viii
Chapter 1 Background 1
Chapter 2 Technology Description 2
Chapter 3 Test Design and Procedures 4
3.1 Introduction 4
3.2 Test Design 4
3.3 Test Samples 5
3.3.1 QC Samples 6
3.3.2 PT Samples 7
3.3.3 Environmental Samples 8
3.4 Reference Analysis 8
3.5 Verification Schedule 9
Chapter 4 Quality Assurance/Quality Control 10
4.1 Laboratory QC for Reference Method 10
4.2 Audits 13
4.2.1 Performance Evaluation Audit 13
4.2.2 Technical Systems Audit 13
4.2.3 Data Quality Audit 13
4.3 QA/QC Reporting 14
4.4 Data Review 14
Chapters Statistical Methods 15
5.1 Accuracy 15
5.2 Precision 15
5.3 Linearity 16
5.4 Method Detection Limit 16
5.5 Matrix Interference Effects 16
5.6 Operator Bias 16
-------
5.7 Inter-Unit Reproducibility 16
5.8 Rate of False Positives/False Negatives 17
Chapter 6 Test Results 18
6.1 QC Samples 18
6.2 PT and Environmental Samples 22
6.2.1 Accuracy 22
6.2.2 Precision 22
6.2.3 Linearity 27
6.2.4 Method Detection Limit 29
6.2.5 Matrix Interference Effects 29
6.2.6 Operator Bias 29
6.2.7 Inter-Unit Reproducibility 30
6.2.8 Rate of False Positives/False Negatives 31
6.3 Other Factors 37
6.3.1 Ease of Use 37
6.3.2 Analysis Time 38
6.3.3 Reliability 38
6.3.4 Waste Material 38
6.3.5 Cost and Consumables 38
Chapter 7 Performance Summary 40
Chapters References 42
Figures
Figure 2-1. TraceDetect SafeGuard 2
Figure 6-1 a. Linearity of SafeGuard Results for the Technical Operator 29
Figure 6-lb. Linearity of SafeGuard Results for the Non-Technical Operator 29
Figure 6-2. Comparison of SafeGuard Results for Technical and Non-Technical Operators 31
Figure 6-3. Comparison of SafeGuard Results for Units # 1 and # 2 32
Tables
Table 3-1. Test Samples for Verification of the SafeGuard 6
Table 3-1. Test Samples for Verification of the SafeGuard (continued) 7
VI
-------
Table 3-2. Schedule of Verification Test Days at Battelle Laboratory 9
Table 4-1. Reference Method QCS Analysis Results 11
Table 4-2. Reference Method Analytical Spike Results 12
Table 4-3. Reference Method Duplicate Analysis Results 12
Table 4-4. Reference Method PE Audit Results 13
Table 4-5. Summary of Data Recording Process 14
Table 6-la. RB Sample Results for the Technical Operator 19
Table 6-lb. RB Sample Results for the Non-Technical Operator 19
Table 6-2a. QCS Results for the Technical Operator 20
Table 6-2b. QCS Results for the Non-Technical Operator 20
Table 6-3a. LFM Sample Results for the Technical Operator 21
Table 6-3b. LFM Sample Results for the Non-Technical Operator 21
Table 6-4a. SafeGuard and Reference Sample Results for the Technical Operator 23
Table 6-4b. SafeGuard and Reference Sample Results for the Non-Technical Operator 25
Table 6-5. Quantitative Evaluation of Accuracy for the SafeGuard(a) 26
Table 6-6. Precision Results for the SafeGuard 27
Table 6-7. Summary of Linear Regression Equations for SafeGuard and
Reference Results 28
Table 6-8. Detection Limit Results for SafeGuard 29
Table 6-9. Summary of Linear Regression Equations for Assessing Operator Bias and Inter-
unit Reproducibility 31
Table 6-10a. Rates of False Positives for the Technical Operator 32
Table 6-10b. Rates of False Positives for the Non-Technical Operator 34
Table 6-1 la. Rates of False Negatives for the Technical Operator 36
Table 6-1 Ib. Rates of False Negatives for the Non-Technical Operator 37
vn
-------
List of Abbreviations
AMS
As
ASTM
ASV
EPA
ETV
HOPE
ICP-MS
L
LFM
MCL
MDL
mL
mm
ND
NIST
ppb
ppm
PE
PT
QA
QA/QC
QC
QCS
QMP
r2
RB
RPD
RSD
ISA
Advanced Monitoring Systems
arsenic
American Society for Testing and Materials
anodic stripping voltammetry
U.S. Environmental Protection Agency
Environmental Technology Verification
high-density polyethylene
inductively coupled plasma mass spectrometry
liter
laboratory-fortified matrix
maximum contaminant level
method detection limit
milliliter
millimeters
non-detect
National Institute of Standards and Technology
parts per billion
parts per million
performance evaluation
performance test
quality assurance
quality assurance/quality control
quality control
quality control sample
Quality Management Plan
coefficient of determination
reagent blank
relative percent difference
relative standard deviation
technical systems audit
Vlll
-------
Chapter 1
Background
The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology
Verification (ETV) Program to facilitate the deployment of innovative environmental
technologies through performance verification and dissemination of information. The goal of the
ETV Program is to further environmental protection by accelerating the acceptance and use of
improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-
quality, peer-reviewed data on technology performance to those involved in the design,
distribution, financing, permitting, purchase, and use of environmental technologies.
ETV works in partnership with recognized testing organizations; with stakeholder groups
consisting of buyers, vendor organizations, and permitters; and with the full participation of
individual technology developers. The program evaluates the performance of innovative
technologies by developing test plans that are responsive to the needs of stakeholders,
conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing
peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality
assurance (QA) protocols to ensure that data of known and adequate quality are generated and
that the results are defensible.
The EPA's National Exposure Research Laboratory and its verification organization partner,
Battelle, operate the Advanced Monitoring Systems (AMS) Center under ETV. The AMS Center
recently evaluated the performance of the TraceDetect SafeGuard trace metals analysis system.
SafeGuard was used to measure total arsenic in water in this verification test.
-------
Chapter 2
Technology Description
The objective of the ETV AMS Center is to verify the performance characteristics of
environmental monitoring technologies for air, water, and soil. This report provides results for
the verification testing of the TraceDetect SafeGuard trace metal analyzer. Following is a
description of the technology, based on information provided by the vendor. The information
provided below was not verified in this test.
TraceDetect's SafeGuard is designed to
automatically measure total arsenic (As)
concentrations in drinking water samples
(including raw water and treated water)
over a range from 1 part per billion (ppb)
to over 100 ppb. Once the operator has
introduced the sample vial and selected
"measure" on the control computer, all
calibrations, dilutions, reductions,
standard additions, and measurements are
performed by the SafeGuard with the
results displayed and logged in a data file.
The software program is designed for both
technical and non-technical operators, by
having a basic mode of operation and an
administrator mode of operation. This
level of operation produces analysis
diagrams and shows more detailed
information about performance. Control
over the communication and configuration of the instrument is also available when in
"administrator" mode.
Figure 2-1. TraceDetect SafeGuard
The SafeGuard consists of three main components: the expert system, the fluidics system, and
TraceDetect's patented NanoBand™ sensor and potentiostat. Each of these components has a
part in the measurement process from controlling the pumps, to adding chemicals, to making
measurements and interpreting the results.
-------
The SafeGuard uses anodic stripping voltammetry (ASV) and the method of standard addition to
make metals measurements. ASV is an electro-analytical method that detects ions in a solution
by the potential at which they oxidize and strip away from the surface of an electrode. The
SafeGuard is able to measure As (III) and reduce As (V) to As (III) to measure total arsenic. It
can also be configured to analyze copper, lead, zinc, cadmium, and mercury in water.
The SafeGuard stores data for every measurement and operation. The base of the SafeGuard is
15 inches by 28 inches (381 millimeters [mm] by 711 mm). It is 22 inches (559 mm) high and
requires a computer, mouse, monitor, and keyboard. The TraceDetect SafeGuard as configured
for measuring arsenic during this verification test was priced at $35,000, excluding options that
the customer may require for unique sample preparation (e.g., copper removal from samples,
filters for high turbidity samples).
-------
Chapter 3
Test Design and Procedures
3.1 Introduction
This verification test was conducted according to procedures specified in the Test/QA Plan for
Verification of Portable Analyzers^ as amended December 4, 2002; January 24, 2003; and
December 19, 2005. The verification was based on comparing the arsenic results from the
SafeGuard to those from a laboratory-based reference method. The reference method for arsenic
analysis was inductively coupled plasma mass spectrometry (ICP-MS) performed according to
EPA Method 200.8.(2) The SafeGuard performance was verified by analyzing laboratory-
prepared performance test (PT) samples, treated and untreated drinking water samples, and fresh
surface water samples. All samples were tested using both the SafeGuard and the reference
method. The test design and procedures are described further below.
3.2 Test Design
The SafeGuard was verified by evaluating the following parameters:
• Accuracy
• Precision
• Linearity
• Method detection limit (MDL)
• Matrix interference effects
• Operator bias
• Inter-unit reproducibility
• Rate of false positives/false negatives.
All sample preparation and analyses were performed according to the vendor's recommended
procedures and the test/QA plan. The results from the SafeGuard were compared to those from
the reference method to assess accuracy and linearity. Multiple aliquots of PT samples, drinking
water samples, and surface water samples were analyzed to assess precision. Multiple aliquots of
a low-level PT sample were analyzed to assess the detection limit of the SafeGuard. Potential
matrix interference effects were assessed by challenging the SafeGuard with PT samples of
known arsenic concentrations that also contained either low levels or high levels of potentially
interfering substances. All samples were analyzed using two identical SafeGuard units
-------
(designated Unit #1 and Unit #2). Results of analyses from the two units were statistically
compared to evaluate inter-unit reproducibility.
Operator bias was assessed by two Battelle staff members of differing technical expertise who
operated the SafeGuard. Both the relatively skilled (technical) and relatively unskilled (non-
technical) operators used both Units #1 and #2 to analyze identical sets of samples. The results
from the two operators were reported separately, and the two sets of results were compared to
determine operator bias.
The rates of false positive and false negative results were evaluated relative to the 10-ppb
maximum contaminant level (MCL) for arsenic in drinking water.(3) Other factors that were
qualitatively assessed during the test included ease of use, time required for sample analysis, and
reliability.
3.3 Test Samples
Three types of samples were analyzed in the verification test, as shown in Table 3-1: quality
control (QC) samples, PT samples, and environmental water samples. The QC and PT samples
were prepared from National Institute of Standards and Technology (NIST)-traceable standards
purchased from a commercial supplier and subject only to dilution as appropriate. Under the Safe
Drinking Water Act, the EPA lowered the MCL for arsenic from 50 ppb to 10 ppb in
January 2001; public water supply systems were required to comply with this standard by
January 2006.(3) Therefore, the QC sample concentrations targeted the 10-ppb arsenic level. The
PT samples ranged from 10% to 1,000% of the 10-ppb level (i.e., from 1 ppb to 100 ppb). The
seven MDL replicates were randomly distributed throughout the PT samples. The environmental
water samples were collected from various drinking water and surface freshwater sources.
According to the test/QA plan (Section 4.1), the QC and PT samples were to be prepared within
two days of analysis and stored at approximately 4°C until use. The QC and PT samples were
prepared for each operator to analyze on two instruments and as needed in 1-liter batches.
However, because of the length of time needed to acquire each measurement (30 to 50 minutes),
preparing solutions at the suggested rate would have meant that there would have been many
more samples for the reference analysis, as well as the extra labor to prepare and analyze these
extra samples. Extending the holding time to three weeks is acceptable because arsenic is a very
stable element, and the PT and QC samples were prepared in a clean matrix of ASTM Type 1
water. Also, PT and QC samples were stored at room temperature instead of 4°C, because the
vendor's recommended analysis conditions for the SafeGuard are at room temperature. Allowing
the samples to come to room temperature from 4°C every day would have substantially
decreased the average number of samples analyzed per day, extending the length of the test. The
environmental samples were stored at 4°C until analysis to minimize bacterial growth.
These deviations from the test/QA plan did not affect the integrity of the samples or the final
results of the test. The ICP-MS results of all samples were within 10% of the prepared
concentration, confirming the stability of the samples. The analysis was done under the vendor's
suggested conditions and is consistent with arsenic analysis in drinking water by ICP-MS. Fresh
-------
QC and PT samples were prepared for each operator. This kept the age of the samples under
3 weeks and the samples provided to the two operators as similar as possible.
Each sample was assigned a unique sample identification number when prepared in the
laboratory or collected in the field. Each replicate of a sample had the identification number and
a consecutive letter (i.e., a, b, c, or d). All SafeGuard samples were analyzed at room temperature
without preservative. All samples were analyzed without pretreatment except the drinking water
samples collected from plumbing (Battelle and residential well drinking waters). Following the
vendor's instructions, samples that traveled through pipes were filtered to remove possible
copper contamination from copper piping and brass fittings before analysis.
3.3.1 QC Samples
QC samples included laboratory reagent blanks (RB), quality control samples (QCS), and
laboratory-fortified matrix (LFM) samples (Table 3-1). The RB samples consisted of the same
American Society for Testing and Materials (ASTM) Type I water used to prepare all other
samples and were subjected to the same handling and analysis procedures as the other samples.
The RB samples were used to verify that no arsenic contamination was introduced during sample
handling and analysis. Ten percent of all samples analyzed were RB samples.
The QCS consisted of ASTM Type I water spiked in the laboratory to a concentration of 10 ppb
of arsenic with a NIST-traceable arsenic standard. QCSs were used to ensure the proper
calibration of the SafeGuard. The SafeGuard was factory calibrated so no additional calibration
was performed by the operators. However, QCSs were still analyzed (without defined
performance limits) by the SafeGuard to demonstrate its proper functioning to the operator.
QCSs were analyzed as the first and last samples, as well as after every tenth sample.
The LFM samples consisted of environmental samples that were spiked in the laboratory to
increase the arsenic concentration by 10 ppb. One LFM sample was prepared from each
environmental sample.
Table 3-1. Test Samples for Verification of the SafeGuard
Type of Sample
Quality Control(b)
Sample Description
Reagent blank (RB)
Quality control sample (QCS)
Laboratory-fortified matrix (LFM)
Arsenic
Concentration(a)
-Oppb
10 ppb
10 ppb above
native level
No. of
Replicates
10% of all
First, last, and
every 10th sample
1 per site
-------
Table 3-1. Test Samples for Verification of the SafeGuard (continued)
Type of Sample
Performance
Test(b)
Sample Description
Prepared arsenic solution
Prepared arsenic solution
Prepared arsenic solution
Prepared arsenic solution
Prepared arsenic solution
Prepared arsenic solution for MDL
Arsenic
Concentration^
Ippb
3ppb
lOppb
30 ppb
lOOppb
5ppb
No. of
Replicates
4
4
4
4
4
7
Prepared arsenic solution spiked
with low levels of interfering
substances
Prepared arsenic solution spiked
with high levels of interfering
substances
lOppb
lOppb
Environmental
Battelle drinking water (treated
drinking)
Olentangy River water (surface)
Residential well water (untreated
drinking)
Alum Creek reservoir water
(surface)
0.6 ppb
1 .4 ppb
1.0 ppb
1.5 ppb
4
4
4
4
(a) Target concentration for QCS and PT samples; measured native concentration for environmental samples
(average of four replicate reference measurements).
(b)
Prepared in ASTM Type I water
3.3.2 PT Samples
Three types of PT samples were used in this verification test (Table 3-1): spiked samples
ranging from 1 ppb to 100 ppb arsenic, a low-level spiked sample for evaluation of the
SafeGuard's MDL, and matrix interference samples that were spiked with potentially interfering
substances. All PT samples were prepared in the laboratory using ASTM Type I water and
NIST-traceable arsenic standards.
Five PT samples containing arsenic at concentrations from 1 ppb to 100 ppb were prepared to
evaluate the SafeGuard's accuracy and linearity. Four aliquots of each sample were analyzed to
assess precision.
-------
To determine the MDL of the SafeGuard, a PT sample was prepared with an arsenic
concentration approximately five times the vendor-stated detection limit (i.e., 1 ppb x 5 = 5 ppb).
Seven non-consecutive replicates of this 5-ppb arsenic sample were analyzed to provide
precision data with which to estimate the MDL.
The matrix interference samples were spiked with 10 ppb of arsenic, as well as potentially inter-
fering species commonly found in natural water samples. One sample contained relatively low
levels of interfering substances that consisted of 1 part per million (ppm) of iron, 0.1 ppm of
sodium sulfide, and 3 ppm of sodium chloride. The second sample contained relatively high
levels of interfering compounds at concentrations of 10 ppm of iron, 1.0 ppm of sodium sulfide,
and 30 ppm of sodium chloride. Four replicates of each sample were analyzed to assess potential
interferences.
3.3.3 Environmental Samples
The environmental samples listed in Table 3-1 included two drinking water samples (treated and
untreated) and two surface water samples collected in Columbus, Ohio. All environmental
samples were collected in 1-liter (L) high density polyethylene (HDPE) bottles and analyzed in
the Battelle laboratory. The Battelle drinking water sample was collected directly from a
drinking water fountain without purging. Residential well water was collected from the spigots
directly after the pressurized holding tank. Four aliquots of each sample were analyzed using the
SafeGuard in the Battelle laboratory as soon as possible after collection. One aliquot of
approximately 100 milliliters (mL) of each sample was preserved with nitric acid and submitted
to the reference laboratory for reference analysis.
One surface water sample was collected from Alum Creek Reservoir in Columbus, Ohio, and
another was collected from the Olentangy River in Columbus, Ohio. These samples were
collected near the shoreline by submerging 1-L FtDPE sample containers no more than one inch
below the surface of the water. Four such containers were filled at one accessible location from
each water source. The samples were transported to Battelle, and the four samples from each site
were combined into a single 4-L volumetric flask to ensure homogeneity. The 4-L samples were
then split into five samples, four for the replicate unspiked samples and one for the LFM, spiked
at 10 ppb. One aliquot of approximately 100 mL of each sample was preserved with nitric acid
and submitted for reference analysis.
3.4 Reference Analysis
The reference arsenic analyses were performed in a Battelle laboratory using a Perkin Elmer
Sciex Elan 6000 ICP-MS according to EPA Method 200.8, Revision 5.5.(2) The sample was
introduced through a peristaltic pump by pneumatic nebulization into a radio frequency plasma
where energy transfer processes caused desolvation, atomization, and ionization. The ions were
extracted from the plasma through a pumped vacuum interface and separated on the basis of their
mass-to-charge ratio by a quadrupole mass spectrometer. The ions transmitted through the
quadrupole were registered by a continuous dynode electron multiplier, and the ion information
was processed by a data handling system.
-------
The ICP-MS was tuned, optimized, and calibrated according to Method 200.8 requirements and
Battelle procedures. The calibration was performed using 11 calibration standards at concen-
trations ranging from 0.1 to 250 ppb, and a minimum coefficient of determination (r2) of 0.999
was required. Internal standards were used to correct for instrument drift and physical inter-
ferences. These standards were introduced in line through the peristaltic pump and analyzed with
all blanks, standards, and samples.
3.5 Verification Schedule
The verification test took place from March 2 through April 4, 2006. Table 3-2 shows the daily
activities that were conducted during this period by the two operators. The reference analysis
was performed on April 6, 2006, once all samples were analyzed by both operators. Subsamples
for reference method analysis were collected and preserved with nitric acid when analyzed by
SafeGuard.
Table 3-2. Schedule of Verification Test Days at Battelle Laboratory
Sample
Preparation/
Collection Date
Sample Analysis Date
Non-
Technical Technical
Operator Operator
Activity
3/2/06-4/4/06 3/21/06-
4/4/06
3/9/06
3/9/06
3/10/06
3/10/06
3/23/06
3/21/06
3/22/06
3/22/06
3/2/06 - Preparation and analysis of PT and associated QC
3/17/06 samples.
3/17/06 Collection and analysis of Battelle drinking water
and associated QC samples.
3/14/06 Collection and analysis of Olentangy River water
and associated QC samples.
3/16/06 Collection and analysis of residential well water and
associated QC samples.
3/15/06 Collection and analysis of Alum Creek reservoir
water and associated QC samples.
-------
Chapter 4
Quality Assurance/Quality Control
Quality assurance/quality control (QA/QC) procedures were performed in accordance with the
quality management plan (QMP) for the AMS Center(4) and the test/QA plan for this verification
test, except for the deviations noted in Section 3.3.(1) These deviations adapted the holding time
and temperature of PT and QC samples. QA/QC procedures and results are described below.
4.1 Laboratory QC for Reference Method
All reference analyses, including QC samples, were conducted on April 6, 2006. Laboratory QC
for the reference method included the analysis of KB, QCS, analytical spike samples, and
analytical duplicate samples. Laboratory KB samples were analyzed to ensure that no
contamination was introduced by the sample preparation and analysis process.
The accuracy of the ICP-MS calibration was verified after the analysis of every 10 samples by
analyzing a QCS at 25 ppb. The percent recovery of the QCS was calculated from the following
equation:
% Recovery = ^xlOO ^
s
where Cs is the measured concentration of the QCS and s is the spike concentration. If the QCS
analysis differed by more than 10% from the true value of the standard, the instrument was
recalibrated before continuing the test. As shown in Table 4-1, all reference QCS analyses were
within the required range.
10
-------
Table 4-1. Reference Method QCS Analysis Results
Sample ID
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
CCV25
Measured (ppb)
25.13
24.99
25.05
24.74
25.11
25.04
25.11
24.75
24.80
24.78
24.67
24.46
23.29
Actual (ppb)
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
25.00
% Recovery (R)
101
100
100
99
100
100
100
99
99
99
99
98
93
Spiked samples were analyzed to assess whether matrix effects influenced the reference method
results. There was an analytical spike every 10th sample in the sequence per EPA
Method 200.8.(3) The analytical spike percent recovery (R) was calculated from the following
equation:
R = Cs~C xlOO (2)
s
where Cs is the measured concentration of the spiked sample, C is the measured concentration of
the unspiked sample, and s is the spike concentration. If the percent recovery of an analytical
spike fell outside the range from 85% to 115%, a matrix effect was suspected. As shown in
Table 4-2, all of the LFM sample results were within this range.
Analytical duplicate samples were analyzed to assess the precision of the reference analysis.
There was an analytical duplicate sample every 10 samples in the sequence per EPA Method
200.8.(3) The relative percent difference (RPD) of the duplicate sample analysis was calculated
from the following equation:
RPD= "g x10Q (3)
(C + CD)/2
where C is the concentration of the sample analysis, and CD is the concentration of the duplicate
sample analysis. If the RPD was greater than 10%, the instrument was recalibrated before
continuing the test. As shown in Table 4-3, the RPDs for the duplicate analyses were all less than
10%; in two samples non-detects were seen in both duplicate analyses.
11
-------
Table 4-2. Reference Method Analytical Spike Results
Amount
Sample ID
51385-02-06-B
51385-11-10-D
51385-02-28-C
51385-13-06-A
51385-15-02-D
51385-18-05-A
51385-15-14-B
51385-21-07-A
51385-23-03-C
51385-23-10-D
51385-17-25
51385-17-13
Matrix
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
ASTM Type I water
Surface water
Drinking water
Unspiked
(ppb)
9.87
5.08
2.99
99.83
9.33
10.14
5.00
3.02
10.12
10.12
10.83
9.75
Table 4-3. Reference Method Duplicate Analysis
Sample Concentration
Sample ID
51385-02-02-A
51385-11-05-C
51385-11-10-E
51385-02-28-D
51385-13-06-B
51385-18-02-A
51385-18-05-B
51385-15-14-C
51385-21-07-B
51385-23-03-D
51385-16-15-A
51385-17-02-A
(ppb)
<0.10
10.33
5.24
3.01
100.8
<0.10
10.22
5.02
3.05
9.93
1.38
0.89
Spiked
(ppb)
36.65
31.09
29.67
128.5
35.19
36.90
32.11
29.42
36.30
36.40
38.34
38.48
Results
Duplicate
Concentration
(ppb)
<0.10
10.40
5.23
3.02
102.0
<0.10
10.15
5.05
3.05
10.16
1.45
0.83
Spiked
(ppb)
25
25
25
25
25
25
25
25
25
25
25
25
R
(%)
107
104
107
115
103
107
108
106
105
105
110
115
RPD
(%)
-
0.7
0.2
0.3
1.2
-
0.7
0.6
0.0
2.3
4.9
6.3
12
-------
4.2 Audits
Three types of audits were performed during the verification test: a performance evaluation (PE)
audit of the reference method, a technical systems audit (TSA) of the verification test
performance, and a data quality audit. Audit procedures are described further below.
4.2.1 Performance Evaluation Audit
A PE audit was conducted to assess the quality of the reference measurements made in this
verification test. For the PE audit, two independent NIST-traceable reference materials were
obtained from different commercial suppliers. One was used for the calibration standards in the
reference analysis and the other used to prepare the QCS, PT, LFM, and PE samples. Accuracy
of the reference method was verified by comparing the arsenic concentration measured based on
the calibration standards to those obtained using the independently certified PE standard. RPD as
calculated by Equation 3 was used to quantify the accuracy of the results. Agreement of the
standard within 10% was required for the measurements to be considered acceptable. As shown
in Table 4-4, the PE sample analysis was within the required range.
Table 4-4. Reference Method PE Audit Results
Measured Independent Standard
Arsenic Concentration Concentration RPD
Sample ID (ppb) (ppb)
PE 26.60 25.00 6.2
4.2.2 Technical Systems A udit
An independent Battelle Quality management staff member conducted a TSA to ensure that the
verification test was being conducted in accordance with the test/QA plan(1) and the AMS Center
QMP.(4) As part of the TSA, test procedures were compared to those specified in the test/QA
plan,(1) and data acquisition and handling procedures as well as the reference method procedures
were reviewed. Observations and findings from the TSA were documented and submitted to the
Battelle Verification Test Coordinator for response. None of the findings of the TSA required
corrective action. TSA records are permanently stored with the Battelle Quality Manager.
4.2.3 Data Quality Audit
At least 10% of the data acquired during the verification test were audited. Battelle's Quality
Manager traced the data from the initial acquisition, through reduction and statistical analysis, to
final reporting to ensure the integrity of the reported results. All calculations performed on the
data undergoing the audit were checked.
13
-------
4.3 QA/QC Reporting
Each audit was documented in accordance with Sections 3.3.4 and 3.3.5 of the QMP for the ETV
AMS Center.(4) Once the audit reports were prepared, the Battelle Verification Test Coordinator
ensured that a response was provided for each adverse finding or potential problem and imple-
mented any necessary follow-up corrective action. The Battelle Quality Manager ensured that
follow-up corrective action was taken. The results of the TSA were submitted to the EPA.
4.4 Data Review
Records generated in the verification test received a one-over-one review before these records
were used to calculate, evaluate, or report verification results. Table 4-5 summarizes the types of
data recorded and reviewed. Data were reviewed by a Battelle technical staff member involved
in the verification test. The person performing the review added his/her initials and the date to a
hard copy of the record being reviewed.
Table 4-5. Summary of Data Recording Process
Data to be Recorded Where Recorded How Often Recorded
Disposition of Data
Dates, times of test
events
Test parameters
(temperature, analyte/
interferent identities,
and all SafeGuard
portable analyzer
results)
Reference method
sample analysis,
chain of custody, and
results
ETV laboratory
record book and
data acquisition
system
ETV laboratory
record book and
data acquisition
system
Laboratory record
books or data
acquisition system,
as appropriate
Start/end of test event and
sample analysis
Throughout sample
handling and analysis
process
Throughout sample
handling and analysis
process
Used to organize/check
test results; transferred to
spreadsheets electronically
Used to organize/check
test results, transferred to
spreadsheets and manually
incorporated in data
spreadsheets as necessary
Transferred to
spreadsheets electronically
14
-------
Chapter 5
Statistical Methods
The statistical methods used to evaluate the performance factors listed in Section 3.2 are
presented in this chapter. Qualitative observations were also used to evaluate verification test
data.
5.1 Accuracy
All samples were analyzed by both the SafeGuard and reference methods. For each sample,
accuracy was expressed in terms of a relative bias (B) as calculated from the following equation:
CD
(4)
where d is the average difference between the SafeGuard results and the reference method
results, and CR is the average of the reference measurements.
5.2 Precision
When possible, the standard deviation (S) of the results for the replicate samples at each
concentration was calculated and used as a measure of SafeGuard precision. Standard deviation
was calculated from the following equation:
S =
1
n-
(5)
where n is the number of replicate samples, Ck is the concentration measured for the kth sample,
and C is the average concentration of the replicate samples. Precision was reported in terms of
the relative standard deviation (RSD) as follows:
RSD =
S_
C
xlOO
(6)
15
-------
5.3 Linearity
Linearity was assessed by performing a linear regression of SafeGuard results against the
reference results, with linearity characterized by the slope, intercept, and coefficient of
determination (r2). Linearity was tested using the four analyses of each of the five PT samples
over the range of 1 ppb to 100 ppb arsenic. Samples with results below the vendor-stated
detection limit were not included in the analysis. Results from both SafeGuard units were plotted
against the corresponding reference concentrations, and separate regressions were performed.
5.4 Method Detection Limit
The MDL for the SafeGuard was assessed using results from both units for seven replicate
analyses of a sample spiked with 5 ppb of arsenic. The standard deviation of the seven replicate
results was calculated using Equation 5. The MDL was then calculated using the following
equation:
MDL = txS (7)
where t is the Student's t value for a 99% confidence level, and S is the standard deviation of the
seven replicate results.
5.5 Matrix Interference Effects
The potential effect of interfering substances on the sensitivity of the SafeGuard was evaluated
by calculating accuracy (expressed as bias) using Equation 4. These results were compared with
accuracy results for PT samples containing only arsenic to assess whether there was a positive or
negative effect due to matrix interferences.
5.6 Operator Bias
The effect of operator skill level on the performance of the SafeGuard was assessed by
comparing results from the two operators for all samples producing results above the detection
limit. Two types of statistical evaluations were conducted. First, linear regression of SafeGuard
results against reference results was conducted for all analyses by each operator, and the two
regressions were compared to one another. Second, a paired t-test of the two data sets was
conducted to assess whether the means of the results from the two operators were significantly
different. This t-test was done separately for results from SafeGuard Unit #1 and Unit #2 and
would indicate a significant difference at the 0.05 level if the two means differed by more than
about 10%.
5.7 Inter-Unit Reproducibility
Inter-unit reproducibility was assessed by performing a linear regression of sample results
generated by the two units. The slope, intercept, and r2 were used to evaluate the degree of inter-
16
-------
unit reproducibility. A paired t-test was also conducted to evaluate whether the two sets of
sample results were significantly different at a 95% confidence level.
5.8 Rate of False Positives/False Negatives
The SafeGuard produced quantitative results over a range from 1 ppb to over 100 ppb. The
purpose of the false positive/negative evaluation was to assess whether the SafeGuard produced
comparable results to the reference value regarding the MCL level. The rates of false positives
and false negatives produced by the SafeGuard were assessed relative to the 10-ppb target
arsenic level for PT, QC, and environmental samples. A false positive result is defined as any
result reported to be greater than 10 ppb and greater than 125% of the reference value, when the
reference value is less than or equal to that guidance level. (The additional criterion to compare
the SafeGuard result to 125% of the reference value was used to account for analytical
uncertainty.) Similarly, a false negative result is defined as any result reported as below or equal
to 10 ppb and less than 75% of the reference value, when the reference value is greater than that
guidance level. The rates of false positives and false negatives were expressed as a percentage of
total samples analyzed for each type of sample.
The inverses of these rates are the specificity and sensitivity of the SafeGuard. These are given
as probabilities. The specificity of the SafeGuard is the probability of correctly identifying
arsenic levels above 10 ppb. The sensitivity of the instrument is the probability of correctly
identifying arsenic levels below 10 ppb.
17
-------
Chapter 6
Test Results
The SafeGuard is automated by a computer and a specific software program that displays the
arsenic measurement in ppb and prompts the user when regular maintenance is necessary. When
the measurement result was lower than the detection limit of 1 ppb, the SafeGuard reported the
measurement as "less than" a concentration, for example < 0.52 ppb. These readings are reported
here as < 1 ppb. Values denoted as non-detects (ND) are ones that had an error message that
read, "Overflow script text - error # 6 - Line # 1213 - Column # 8." This error originated when
the software system did not detect arsenic and consequently divided an equation by zero.
TraceDetect gave instruction that this error message should be recorded as a non-detect. One of
the method detection performance test sample results for the technical operator on unit #1 gave
this error message. The accuracy, precision, and MDL were calculated excluding this result.
6.1 QC Samples
As described in Section 3.3.1, the QC samples analyzed with the SafeGuard included KB, QCS,
and LFM samples. Ten percent of all samples analyzed were KB samples, and the results were
used to verify that no arsenic contamination was introduced during sample handling and analysis.
QCSs were analyzed first, last, and after every tenth sample. The QCS results were used to verify
that the system was operating properly; however, since the SafeGuard is not calibrated by the
operator, they were analyzed without defined performance limits. One LFM sample was
prepared from each environmental sample to evaluate potential matrix interferences.
KB sample results for the SafeGuard are presented in Tables 6-la and b for the technical and
non-technical operators, respectively. One replicate by the technical operator on Unit #1 was
reported at 1.11 ppb. This was just above the detection limit specified by the vendor. All other
KB results with both units and both operators, including an aliquot of the same KB before and
after this result were below the SafeGuard's detection limit. All KB samples were analyzed by
the reference method and were below the 1-ppb detection limit of the SafeGuard. It was
concluded that arsenic contamination resulting from sample handling did not occur.
QCS results for the technical and non-technical operators are presented in Tables 6-2a and 6-2b,
respectively. The QCSs were analyzed first, last, and after every tenth sample, as required,
except for one, when the non-technical operator inadvertently switched the last two samples of
the test. The QCS was prepared at 10 ppb for the percent recovery calculated using Equation 1
(Section 4.1). The QCS percent recovery for the technical operator ranged from 63% to 96%.
The QCS percent recovery for the non-technical operator ranged from 0% to 102%, because one
18
-------
QCS was not detected by SafeGuard Unit #1. Excluding this reading, the percent recovery for
the non-technical operator ranged from 51% to 102%.
One LFM sample was prepared from each environmental sample to evaluate potential matrix
interferences. The LFM sample results for the technical and non-technical operators are
presented in Tables 6-3a and 6-3b respectively. The R value associated with each LFM sample
was calculated using Equation 2 (Section 4.1). Reference method results are also provided for
comparison. One spiked sample of Battelle drinking water was not detected by Unit #1 when
analyzed by the technical operator. Except for that sample, the lowest recoveries for the
SafeGuard from both operators on both units (12% to 45%) were associated with the residential
well water LFM sample. This was because of the substantial level of arsenic measured in the
unspiked sample. However, arsenic was not detected above 1 ppb by the reference method in the
unspiked sample. This indicates that a matrix effect was exaggerating the level of arsenic in the
unspiked residential well water sample as reported by the SafeGuard. The other environmental
samples did not noticeably affect the instrument because good recoveries were observed for
those samples.
Table 6-la. RB Sample Results for the Technical Operator
Sample ID
51385-18-02-A
51385-18-02-B
51385-18-02-C
51385-18-02-D
51385-18-02-E
51385-18-02-F
51385-22-03-G
51385-22-03-H
51385-22-03-1
Replicate
1
2
3
4
5
6
1
2
3
Unit #1
Analysis Date (ppb)
3/20/2006 <1
3/21/2006 <1
3/22/2006 <1
3/23/2006 <1
3/27/2006 <1
3/30/2006 <1
4/3/2006 <1
4/4/2006 1.11
4/4/2006 <1
Unit #2
(ppb)
<1
<1
<1
<1
<1
<1
<1
<1
<1
Table 6-lb. RB Sample Results for the Non-Technical Operator
Sample ID Replicate
51385-02-02-A 1
51385-02-02-B 2
51385-11-02-C 1
51385-11-02-D 2
51385-1 1-02-E 3
51385-16-02-F 1
51385-16-02-G 2
51385-16-02-H 3
Unit #1
Analysis Date (ppb)
3/1/2006 <1
3/3/2006 <1
3/7/2006 <1
3/8/2006 <1
3/9/2006 <1
3/9/2006 <1
3/15/2006 ND
3/17/2006 <1
Unit #2
(ppb)
<1
<1
<1
<1
<1
<1
<1
<1
ND = non-detect
19
-------
Table 6-2a. QCS Results for the Technical Operator
Analysis Unit #1
Sample ID Replicate Date (ppb)
51385-18-05-A
51385-18-05-B
51385-18-05-C
51385-18-05-D
51385-18-05-E
51385-18-05-F
51385-22-08-G
51385-22-08-H
51385-22-08-1
Table 6-2b. QCS
1
2
3
4
5
6
1
2
O
Results
3/20/2006
3/21/2006
3/22/2006
3/23/2006
3/27/2006
3/30/2006
4/3/2006
4/4/2006
4/4/2006
8.38
9.47
7.34
8.04
9.60
7.68
8.90
9.64
9.22
for the Non-Technical
Amount
Unit #2 Spiked
(ppb)
8.16
9.11
7.22
6.84
6.86
6.26
8.71
8.35
8.96
Operator
Unit#l Unit #2
Sample ID Replicate
51385-02-06-A
51385-02-06-B
51385-11-05-C
51385-11-05-D
51385-11-05-E
51385-16-06-F
51385-16-06-G
51385-16-06-H
51385-16-06-1
1
2
1
2
3
1
4
5
6
Analysis Date
3/2/2006
3/3/2006
3/7/2006
3/8/2006
3/9/2006
3/9 &
3/ll/2006a
3/11/2006
3/15/2006
3/17/2006
(ppb)
8.83
8.46
9.17
8.10
10.2
8.36
ND
5.09
9.84
(ppb)
7.76
9.34
9.94
9.59
6.56
7.41
8.56
8.06
7.96
(ppb)
10
10
10
10
10
10
10
10
10
Amount
Spiked
(ppb)
10
10
10
10
10
10
10
10
10
%
Recovery
Unit #1
84
95
73
80
96
77
89
96
92
%
Recovery
Unit #1
88
85
92
81
102
84
0
51
98
%
Recovery
Unit #2
82
91
72
68
69
63
87
84
90
%
Recovery
Unit #2
78
93
99
96
66
74
86
81
80
ND = non-detect, reported error message by SafeGuard due to division by zero
a QCS was analyzed on Unit #1 (3/11/06) and Unit #2 (3/9/06) on different days
20
-------
Table 6-3a. LFM Sample Results for the Technical Operator
Unspiked(a)
Description (ppb)
Battelle drinking water LFM
Unit # 1
Unit # 2
Reference
Olentangy River water LFM
Unit # 1
Unit # 2
Reference
Residential well water LFM
Unit # 1
Unit # 2
Reference
Alum Creek Reservoir water LFM
Unit # 1
Unit # 2
Reference
ND
ND
0.62
<1
<1
1.38
6.31
7.11
0.98
<1
<1
1.48
Spiked Amount Spiked
(ppb) (ppb)
ND
9.40
9.75
9.63
7.63
10.6
8.36
8.27
10.7
9.30
7.90
10.8
10
10
10
10
10
10
10
10
10
10
10
10
R(b)
(%)
0
94
91
96
76
92
21
12
97
93
79
94
(a) Average of four replicates. Non-detects and <1 ppb results were assigned a value of zero for calculation of
average
(b) Non-detects and < 1 ppb results were assigned a value of zero for calculation of R
ND = non-detect, reported error message by SafeGuard due to division by zero
Table 6-3b. LFM Sample Results for the Non-Technical Operator
Description
Battelle drinking water LFM
Unit # 1
Unit # 2
Reference
Olentangy River water LFM
Unit # 1
Unit # 2
Reference
Residential well water LFM
Unit # 1
Unit # 2
Reference
Alum Creek Reservoir water LFM
Unit # 1
Unit # 2
Reference
Unspiked00
(ppb)
<1
<1
0.62
<1
<1
1.38
7.38
6.49
0.98
<1
<1
1.48
Spiked Amount Spiked
(ppb) (ppb)
7.04
8.00
9.75
7.84
7.78
10.6
11.9
10.9
10.7
8.96
7.83
10.8
10
10
10
10
10
10
10
10
10
10
10
10
R(b)
(%)
70
80
91
78
78
92
45
44
97
90
78
94
(a) Average of four replicates. < 1 ppb results were assigned a value of zero for calculation of average
(b) Non-detects and <1 ppb results were assigned a value of zero for calculation of R
21
-------
6.2 PT and Environmental Samples
Tables 6-4a and 6-4b present the results for the PT and environmental samples for the technical
and non-technical operators, respectively. Each table includes the SafeGuard results for both
units and the reference method results. The SafeGuard results below the detection limit were
assigned a value of <1 ppb. Results for each performance factor are presented below.
6.2.1 Accuracy
Table 6-5 presents the accuracy results for the SafeGuard, expressed as relative percent bias as
calculated by Equation 4 (Section 5.1). Percent bias was not calculated if any result for a set of
replicates was below the detection limit (<1 ppb). The bias ranged from -28% to 629% for the
technical operator and -28% to 657% for the non-technical operator. The high end of these
ranges is due to the residential well water. The high bias for this environmental sample confirms
the apparent matrix effect observed in the LFM sample with this matrix (see Section 6.2.5).
Excluding the residential well water sample, the bias ranged from -28% to 7% and -28% to 11%
for the technical and non-technical operators, respectively.
6.2.2 Precision
Precision results for the SafeGuard are presented in Table 6-6. The RSD was determined as a
percentage according to Equation 5 (Section 5.2). The RSD was not calculated if any of the
results for a set of replicates were below the detection limit. The RSDs ranged from 3% to 44%
for the technical operator and from 2% to 38% for the non-technical operator. The average RSD
of the PT samples for the technical operator was 10% and the average RSD for the non-technical
operator was 9%
22
-------
Table 6-4a. SafeGuard and Reference Sample Results for
Description
PT- 1 ppb As
PT- 3 ppb As
PT- 10 ppb As
PT- 30 ppb As
PT- 100 ppb As
Detection Limit
PT- 10 ppb As + low
level interferents
Sample ID
5 1385-2 1-02-A
51385-21-02-B
51385-21-02-C
51385-21-02-D
5 1385-2 1-07-A
51385-21-07-B
51385-21-07-C
5 1385-2 1-07-D
51385-21-12-A
51385-21-12-B
51385-21-12-C
51385-21-12-D
51385-13-02-A
51385-13-02-B
51385-13-02-C
51385-13-02-D
51385-13-06-A
51385-13-06-B
51385-13-06-C
51385-13-06-D
51385-15-14-A
51385-15-14-B
51385-15-14-C
51385-15-14-D
51385-22-13-E
51385-22-13-F
51385-22-13-G
51385-23-10-A
51385-23-10-B
51385-23-10-C
51385-23-10-D
Replicate
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
5
6
7
1
2
3
4
Analysis
Date
3/29/2006
3/29/2006
3/29/2006
3/29/2006
3/29/2006
3/29/2006
3/29/2006
3/30/2006
3/30/2006
3/30/2006
3/31/2006
3/31/2006
4/3/2006
4/3/2006
4/3/2006
4/3/2006
4/3/2006
4/3/2006
4/3/2006
4/3/2006
3/27/2006
3/29/2006
3/30/2006
3/31/2006
4/3/2006
4/3/2006
4/4/2006
4/4/2006
4/4/2006
4/4/2006
4/4/2006
the Technical Operator
Unit #1
(ppb)
<1
<1
<1
<1
2.48
2.87
1.77
2.50
8.52
8.70
9.32
8.65
25.0
24.1
27.8
27.8
93.1
105.0
105.0
95.9
<1
4.43
4.91
4.50
5.50
7.35
6.04
10.3
10.6
10.0
10.6
Unit #2
(ppb)
1.08
<1
1.11
<1
3.07
2.65
2.41
2.96
8.49
7.81
6.31
8.35
25.7
25.1
23.9
21.0
81.8
84.1
85.6
73.7
3.77
4.67
3.44
5.01
4.48
6.27
4.58
8.48
8.87
9.58
8.39
Reference
(ppb)
1.12
1.09
1.09
1.10
3.02
3.05
3.04
3.01
9.56
9.88
9.85
9.96
29.7
29.4
29.9
29.9
99.8
100.8
100.2
100.8
5.10
5.00
5.02
5.05
5.20
5.23
5.11
10.2
10.3
10.2
10.1
23
-------
Table 6-4a. SafeGuard and Reference Sample Results for the Technical
(continued)
Description
PT- 10 ppb As + high
level interferents
Battelle drinking water
Battelle drinking water
LFM
Olentangy River water
Olentangy River water
LFM
Residential well water
Residential well water
LFM
Alum Creek Reservoir
water
Alum Creek Reservoir
water LFM
Sample ID
51385-23-03-A
51385-23-03-B
51385-23-03-C
51385-23-03-D
51385-16-10-A
51385-16-10-B
51385-16-10-C
51385-16-10-D
51385-17-13
51385-16-15-A
51385-16-15-B
51385-16-15-C
51385-16-15-D
51385-17-17
51385-17-02-A
51385-17-02-B
51385-17-02-C
51385-17-02-D
51385-17-21
51385-17-08-A
51385-17-08-B
51385-17-08-C
51385-17-08-D
51385-17-25
Replicate
1
2
3
4
1
2
3
4
1
1
2
3
4
1
1
2
3
4
1
1
2
3
4
1
Analysis
Date
4/4/2006
4/4/2006
4/4/2006
4/4/2006
3/23/2006
3/23/2006
3/23/2006
3/23/2006
3/21/2006
3/21/2006
3/21/2006
3/21/2006
3/21/2006
3/20/2006
3/22/2006
3/22/2006
3/22/2006
3/22/2006
3/21/2006
3/22/2006
3/22/2006
3/22/2006
3/22/2006
3/20/2006
Unit #1
(ppb)
11.3
11.2
11.6
9.35
ND
ND
ND
ND
ND
<1
<1
1.16
<1
9.63
6.20
7.27
6.90
4.85
8.36
<1
<1
<1
<1
9.30
Operator
Unit #2
(ppb)
8.56
8.07
9.51
8.91
ND
ND
ND
ND
9.40
<1
2.92
<1
<1
7.63
7.97
8.58
2.50
9.39
8.27
<1
<1
<1
<1
7.90
Reference
(ppb)
10.3
10.1
10.1
9.93
0.56
0.66
0.63
0.61
9.75
1.38
1.41
1.41
1.33
10.6
0.89
1.12
0.95
0.94
10.7
1.50
1.48
1.42
1.50
10.8
ND = Non-detects, reported error message by SafeGuard due to division by zero.
24
-------
Table 6-4b. SafeGuard and Reference Sample Results for the Non-Technical Operator
Description
PT- 1 ppb As
PT- 3 ppb As
PT- 10 ppb As
PT- 30 ppb As
PT- 100 ppb As
Detection Limit
PT- 10 ppb As + low
level interferents
PT- 10 ppb As + high
level interferents
Battelle drinking water
Battelle drinking water
LFM
Sample ID
51385-02-23-A
51385-02-23-B
51385-02-23-C
51385-02-23-D
51385-02-28-A
51385-02-28-B
51385-02-28-C
51385-02-28-D
51385-11-14-A
51385-11-14-B
51385-11-14-C
51385-11-14-D
51385-13-02-A
51385-13-02-B
51385-13-02-C
51385-13-02-D
51385-13-06-A
51385-13-06-B
51385-13-06-C
51385-13-06-D
51385-02-15-A
51385-02-15-B
51385-02-15-C
51385-11-10-D
51385-11-10-E
51385-11-10-F
51385-15-14-G
51385-15-08-A
51385-15-08-B
51385-15-08-C
51385-15-08-D
51385-15-02-A
51385-15-02-B
51385-15-02-C
51385-15-02-D
51385-16-10-A
51385-16-10-B
51385-16-10-C
51385-16-10-D
51385-17-13-A
Replicate
1
2
3
4
1
2
o
3
4
1
2
3
4
1
2
3
4
1
2
o
3
4
1
2
o
3
4
5
6
7
1
2
3
4
1
2
o
3
4
1
5
7
9
1
Analysis
Date
3/2/2006
3/2/2006
3/2/2006
3/2/2006
3/3/2006
3/3/2006
3/3/2006
3/3/2006
3/6/2006
3/6/2006
3/6/2006
3/6/2006
3/7/2006
3/7/2006
3/7/2006
3/7/2006
3/7/2006
3/7/2006
3/7/2006
3/7/2006
3/2/2006
3/3/2006
3/3/2006
3/6/2006
3/7/2006
3/7/2006
3/8/2006
3/8/2006
3/8/2006
3/9/2006
3/9/2006
3/9/2006
3/9/2006
3/9/2006
3/9/2006
3/9/2006
3/17/2006
3/17/2006
3/17/2006
3/17/2006
Unit #1
(ppb)
1.14
1.16
1.09
1.05
3.52
3.30
2.94
3.30
6.70
9.93
8.58
9.98
26.8
25.9
24.4
26.0
83.2
95.2
85.9
87.7
6.06
4.39
3.90
5.71
6.14
4.90
3.50
4.90
6.37
8.72
11.8
8.01
7.24
8.67
8.41
ND
<1
ND
<1
7.04
Unit #2
(ppb)
1.12
<1
<1
1.04
2.81
3.00
3.04
3.29
8.79
9.73
9.64
9.79
26.2
25.2
24.0
24.7
75.8
81.4
82.9
82.6
4.36
4.03
5.08
5.52
5.46
5.78
5.35
8.76
8.93
8.77
9.15
7.58
7.34
7.40
7.57
1.08
<1
<1
<1
8.00
Reference
(ppb)
1.06
1.05
1.05
1.02
3.02
2.92
2.99
3.01
10.2
10.3
10.2
10.3
29.7
29.4
29.9
29.9
99.8
100.8
100.2
100.8
5.08
5.02
4.97
5.08
5.24
5.11
5.15
9.71
9.49
9.43
9.35
9.28
9.31
9.19
9.33
0.56
0.66
0.63
0.61
9.75
25
-------
Table 6-4b. SafeGuard
(continued)
Description
Olentangy River water
Olentangy River water LFM
Residential well water
Residential well water LFM
Alum Creek Reservoir water
Alum Creek Reservoir water
LFM
and Reference Sample Results for the Non-Technical Operator
Sample ID
51385-16-15-A
51385-16-15-B
51385-16-15-C
51385-16-15-D
51385-17-17-A
51385-17-02-A
51385-17-02-B
51385-17-02-C
51385-17-02-D
51385-17-21-A
51385-17-08-A
51385-17-08-B
51385-17-08-C
51385-17-08-D
51385-17-25-A
Replicate
1
2
3
4
1
1
6
7
8
1
1
2
3
4
1
Analysis
Date
3/14/2006
3/14/2006
3/14/2006
3/14/2006
3/14/2006
3/15/2006
3/16/2006
3/16/2006
3/16/2006
3/16/2006
3/14/2006
3/15/2006
3/15/2006
3/15/2006
3/15/2006
Unit #1
(ppb)
1.65
7.84
9.70
4.75
7.48
7.60
11.9
ND
8.96
Unit #2
(ppb)
J!
7.78
6.88
6.62
4.80
7.65
10.9
J!
7.83
Reference
(ppb)
1.38
1.41
1.41
1.33
10.6
0.89
1.12
0.95
0.94
10.7
1.50
1.48
1.42
1.50
10.8
ND = Non-detects, reported by SafeGuard as error message because of division by zero.
Table 6-5. Quantitative Evaluation of Accuracy for the SafeGuard
(a)
Bias
Description
PT Samples
PT- 1 ppb As
PT- 3 ppb As
PT- 10 ppb As
PT- 30 ppb As
PT- 100 ppb As
Detection limit
PT- 10 ppb As + low level
interferents
PT- 1 0 ppb As + high level
interferents
Environmental Samples
Battelle drinking water
Battelle drinking water LFM
Olentangy River water
Olentangy River water LFM
Residential well water
Residential well water LFM
Alum Creek Reservoir water
Alum Creek Reservoir water LFM
Technical Operator
Unit #1
NA
-21%
-10%
-12%
-1%
7%00
2%
7%
NA
NA
NA
-9%
547%
-22%
NA
-14%
Technical Operator
Unit #2
NA
-8%
-21%
-19%
-19%
-10%
-13%
-13%
NA
-4%
NA
-28%
629%
-23%
NA
-27%
Non-Technical
Operator Unit #1
6%
9%
-14%
-13%
-12%
-3%
-16%
-13%
NA
-28%
NA
-26%
657%
11%
NA
-17%
Non-Technical
Operator Unit #2
NA
2%
-7%
-16%
-20%
0%
-6%
-19%
NA
-18%
NA
-27%
565%
2%
NA
-28%
(a) Percent bias calculated according to Equation 4, Section 5.1.
(b) One replicate result = non-detect, calculated bias excluding this result.
NA = one or more replicates below the detection limit.
26
-------
Table 6-6. Precision Results for the SafeGuard
RSD
Description
FT Samples
PT- 1 ppb As
PT- 3 ppb As
PT- 10 ppb As
PT- 30 ppb As
PT- 100 ppb As
Detection limit
PT- 10 ppb As + low
level interferents
PT- 10 ppb As + high
level interferents
Average RSD for PT
samples with
interferents
Average RSD for PT
samples
Environmental
Samples
Battelle drinking water
Olentangy River water
Residential well water
Alum Creek Res water
Average RSD
Technical
Operator Unit #1
NA
19%
4%
7%
6%
20%(a)
3%
9%
6%
10%
ND
NA
17%
NA
13%
Technical
Operator Unit #2
NA
11%
13%
9%
7%
20%
6%
7%
ND
NA
44%
NA
Non-Technical
Operator Unit #1
4%
7%
18%
4%
6%
21%
38%
8%
Non-Technical
Operator Unit #2
NA
7%
5%
4%
4%
13%
2%
2%
12%
9%
NA(b)
NA
27%
NA(b)
NA
NA
19%
NA
11%
(a) One replicate result = non-detect, calculated RSD excluding this result.
(b) Includes values reported as ND.
NA = One or more replicates below detection limit.
ND = Non-detects, reported by SafeGuard as error message because of division by zero.
6.2.3 Linearity
The linearity of the SafeGuard measurements was assessed by performing a linear regression of
the SafeGuard results against the reference method results for the five PT samples ranging from
1 ppb to 100 ppb of arsenic. Figure 6-la presents the results of the linear regression for the two
SafeGuard units when operated by the technical operator and Figure 6-lb for the two units when
operated by the non-technical operator. In these regressions, results reported below the detection
limit (<1 ppb) by the SafeGuard or identified as ND due to error message were not included. The
slope, intercept, and coefficient of determination (r2) for each regression equation are shown on
the charts. Table 6-7 summarizes the equations for the linear regressions and presents the 95%
confidence interval for the slopes as ± error. All linear regressions compared to the reference
method results had coefficients of determination greater than 0.99. The 95% confidence
intervals for the slopes indicate that only the technical operator data for Unit #1 were consistent
with a slope of 1 and were not significantly different from the reference analysis results. The
95% confidence intervals for the y-axis intercept included zero for both operators on both units
indicating no significant difference from the reference analysis results.
27
-------
•fl
1 60
/ = 1.0045x-1.6184
Technical Operator Unit #1
Technical Operator Unit #2
- Linear (Technical Operator
UnitfH)
-Linear (Technical Operator
Unit #2)
60
Reference concentration (ppb)
Figure 6-la. Linearity of SafeGuard Results for the Technical Operator
60
40
O
1
R = 0.9961
i
Non-Technical Operator Unit
Non-Technical Operator Unit
#2
Linear (Non-Technical
Operator Unit #1)
Linear (Non-Technical
Operator Unit #2)
40 60 i
Reference concentration (ppb)
Figure 6-lb. Linearity of SafeGuard Results for the Non-Technical Operator
Table 6-7. Summary of Linear Regression Equations for SafeGuard and Reference
Results
Description
Safeguard Unit #1, technical operator
Safeguard Unit #2, technical operator
Safeguard Unit #1, non-technical operator
Safeguard Unit #2, non-technical operator
Slope
(± Error)
1.005 (0.044)
0.808 (0.034)
0.874 (0.027)
0.796(0.019)
Intercept
(± Error)
-1.618(2.32)
0.060(1.70)
0.155(1.27)
0.960 (0.96)
Coefficient of
Determination
0.9942
0.9936
0.9961
0.9979
28
-------
6.2.4 Method Detection Limit
The MDL was assessed by analyzing seven replicates of a sample spiked at approximately five
times the vendor-stated detection limit for the SafeGuard. Table 6-8 lists the replicate results,
provides the standard deviations for the replicate results for the SafeGuard results, and shows the
calculated MDLs. The calculated MDL values for the technical operator were 3.75 ppb and
2.87 ppb for Units #1 and #2 respectively. The MDL values for the non-technical operator
calculated were 3.33 ppb and 2.04 ppb for Units #1 and #2.
Table 6-8. Detection Limit Results for SafeGuard
Sample
Concentration
(ppb)
5
5
5
5
5
5
5
Standard Deviation
t (n=7)
MDL
Technical
Unit #1
(ppb)
ND(a)
4.43
4.91
4.50
5.50
7.35
6.04
1.11
3.37(b)
3.75
Operator Non-Technical
Unit #2
(ppb)
3.77
4.67
3.44
5.01
4.48
6.27
4.58
0.91
3.14
2.87
Unit #1
(ppb)
6.06
4.39
3.90
5.71
6.14
4.90
3.50
1.06
3.14
3.33
Operator
Unit #2
(ppb)
4.36
4.03
5.08
5.52
5.46
5.78
5.35
0.65
3.14
2.04
included in MDL calculation
(b) t(n = 6)
6.2.5 Matrix Interference Effects
Matrix interference effects were assessed by comparing the calculated percent bias for the
samples containing low-level and high-level concentrations of interfering substances with the
bias reported for the other PT samples (Table 6-5). The biases for the samples with low and high
levels of interfering compounds ranged from -19% to 7%, which is within the range of the PT
samples (-21% to 9%). As such, neither the low nor the high levels of interferences tested
appeared to have affected the arsenic levels measured by the SafeGuard.
As discussed in Section 6.2.1, residential well water clearly affected the SafeGuard
measurement, because the native (unspiked) replicates from both operators and both SafeGuard
units reported an arsenic concentration from 2.50 ppb to 9.70 ppb, whereas the reference method
reported this sample at 0.89 ppb to 1.12 ppb. Thus a positive unknown interference exists in the
residential well water sample. Battelle drinking water, Olentangy River water, and Alum Creek
reservoir water did not appear to have matrix interference effects.
6.2.6 Operator Bias
Operator bias was evaluated by comparing the SafeGuard results above the detection limit
produced by the technical and non-technical operators for all PT and environmental samples.
Linear regression results for the two sets of data are shown in Figure 6-2. The slopes of the
regressions show little difference between operators with Unit #2 (slope = 0.98), but slightly
29
-------
higher results overall from the technical operator with Unit #1 (slope = 0.87). The 95%
confidence intervals were calculated for the Unit #1 and Unit #2 regressions in Figure 6-2 and
are shown as ± error in Table 6-9 (Section 6.2.7). The 95% confidence interval includes a slope
of 1 for Unit #2, but the 95% confidence interval does not include a slope of 1 for Unit #1,
indicating a significant operator bias (technical operator results > non-technical operator results)
with that unit.
Paired t-tests of the two sets of data indicate that the SafeGuard results were not significantly
different at a 0.05 level of significance depending on the operator. The t-test finds a significant
difference if the means of the data sets from the two operators differ by more than about 10%.
The respective operator means for Unit #1 differed by 8% and those for Unit #2 differed by 2%.
Overall, these results indicate at most a small operator bias with one of the two SafeGuard units.
Technical Operator (ppb)
Figure 6-2. Comparison of SafeGuard Results for Technical and Non-Technical Operators
6.2.7 Inter-Unit Reproducibility
Inter-unit reproducibility was evaluated by comparing the data for the two SafeGuard units. All
detected results for the PT and environmental samples were included in the analysis. Linear
regressions of the data for each unit are shown in Figure 6-3 and show that Unit #2 readings were
lower than Unit # 1 readings with both operators, but more strongly with the technical operator.
The 95% confidence intervals were calculated for the technical and non-technical operator
regressions in Figure 6-3 and are shown as ± error in Table 6-9. Neither 95% confidence interval
includes a slope of 1, indicating a significant inter-unit bias that is more pronounced with the
technical operator than with the non-technical operator.
A paired t-test of the data indicated that the results from the two units with the technical operator
were significantly different at a 0.05 level of significance; however, the results from the two
units with the non-technical operator were not significantly different. The t-test finds a
significant difference if the means of the two SafeGuard units differ by more than about 10%.
30
-------
100
40
20
y=0.9066x+0.8642
y = 0.8024x+1.2084
R2 = 0.9897
Non-Technical Operator
Techical Operator
Linear (Non-Technical
Operator)
Linear (Techical Operator)
20
40
60
Unit # 1
100
Figure 6-3. Comparison of SafeGuard Test Results for Units # 1 and # 2
Table 6-9. Summary of Linear Regression Equations for Assessing Operator Bias and
Inter-unit Reproducibility
Description
Unit #1, operator bias
Unit #2, operator bias
Technical operator, Inter-unit reproducibility
Non-technical operator, Inter-unit reproducibility
Slope
(± Error)
0.872(0.033)
0.983 (0.033)
0.802 (0.029)
0.907 (0.023)
Intercept
(± Error)
0.957(1.18)
0.566 (0.98)
1.208(1.03)
0.864 (0.74)
Coefficient of
Determination
0.9886
0.9909
0.9897
0.9947
The means for Units #1 and #2 differed by 17% when used by the technical operator and by 7%
when used by the non-technical operator. Overall, these results show an inter-unit bias with the
technical operator, but minimal bias with the non-technical operator.
6.2.8 Rate of False Positives/False Negatives
Tables 6-10a and b show the false positives for the technical and non-technical operators
respectively, and Tables 6-1 la and b present the false negative data for the two operators. These
calculations included all PT, QC, and environmental results. The rates of false positives for the
SafeGuard were 0% for both units for the technical operator and 2% and 0% for the non-
technical operator (Units #1 and #2, respectively). The rates of false negatives for the SafeGuard
units were 4% and 22% for the technical operator (Units #1 and #2, respectively), and 18% for
both units for the non-technical operator. The false positive and negative rates of the technical
and non-technical operators were averaged to determine sensitivity and specificity. The results
indicate that the SafeGuard correctly identified arsenic concentrations below the federal drinking
water standard (<10 ppb) 99.5% of the time (0.995 = sensitivity) and identified arsenic
31
-------
concentrations that did not meet the federal standard (>10 ppb) 85.4% of the time (0.854 =
specificity).
Table 6-10a. Rates of False Positives for the Technical Operator
Description
PT- 1 ppb As
PT- 3 ppb As
PT- 10 ppb As
Detection limit
PT- 10 ppb As + high-level
interferents
Battelle drinking water
Battelle drinking water LFM
Olentangy River water
Residential well water
Unit #1
(ppb)
0.0
0.0
0.0
0.0
2.48
2.87
1.77
2.50
8.52
8.70
9.32
8.65
0.0
4.43
4.91
4.50
5.50
7.35
6.04
9.35
ND
ND
ND
ND
0.0
<1
<1
1.16
<1
6.20
7.27
6.90
4.85
False Positive
(Y/N) Unit #1
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Unit #2
(ppb)
1.08
0.0
1.11
0.0
3.07
2.65
2.41
2.96
8.49
7.81
6.31
8.35
3.77
4.67
3.44
5.01
4.48
6.27
4.58
8.91
ND
ND
ND
ND
9.40
<1
2.92
<1
<1
7.97
8.58
2.50
9.39
False Positive
(Y/N) Unit #2
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Reference
(ppb)
1.12
1.09
1.09
1.10
3.02
3.05
3.04
3.01
9.56
9.88
9.85
9.96
5.10
5.00
5.02
5.05
5.20
5.23
5.11
9.93
0.56
0.66
0.63
0.61
9.75
1.38
1.41
1.41
1.33
0.89
1.12
0.95
0.94
32
-------
Table 6-1 Oa. Rates of False Positives for the Technical Operator (continued)
Description
Alum Creek Reservoir water
RB
Total # samples <10 ppb by
reference method
Total # false positive (Y)
Percent false positive
False Positive
Unit #1 (Y/N) Unit #1
(ppb) Qualitative
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
1.11 N
<1 N
46
0
0%
False Positive
Unit #2 (Y/N) Unit #2
(ppb) Qualitative
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
46
0
0%
Reference
(ppb)
1.50
1.48
1.42
1.50
<0.
<0.
<0.
<0.
<0.
<0.
<0.
<0.
<0.
33
-------
Table 6-10b. Rates of False Positives for the Non-Technical Operator
Description
PT- 1 ppb As
PT- 3 ppb As
Detection limit
PT- 10 ppb As + low level
interferents
PT- 10 ppb As + high level
interferents
Battelle drinking water
Battelle drinking water LFM
Olentangy River water
Residential well water
Unit #1
(ppb)
1.14
1.16
1.09
1.05
3.52
3.30
2.94
3.30
6.06
4.39
3.90
5.71
6.14
4.90
3.50
4.90
6.37
8.72
11.8
8.01
7.24
8.67
8.41
ND
<1
ND
<1
7.04
1.65
<1
<1
<1
9.70
4.75
7.48
7.60
False Positive
(Y/N) Unit #1
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Y
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Unit #2
(ppb)
1.12
0.0
0.0
1.04
2.81
3.00
3.04
3.29
4.36
4.03
5.08
5.52
5.46
5.78
5.35
8.76
8.93
8.77
9.15
7.58
7.34
7.40
7.57
1.08
<1
<1
<1
8.00
<1
<1
<1
<1
6.88
6.62
4.80
7.65
False Positive
(Y/N) Unit #2
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Reference
(ppb)
1.06
1.05
1.05
1.02
3.02
2.92
2.99
3.01
5.08
5.02
4.97
5.08
5.24
5.11
5.15
9.71
9.49
9.43
9.35
9.28
9.31
9.19
9.33
0.56
0.66
0.63
0.61
9.75
1.38
1.41
1.41
1.33
0.89
1.12
0.95
0.94
34
-------
Table 6-1 Ob. Rates of False Positives for the Non-Technical Operator (continued)
Description
Alum Creek Reservoir water
RB
QCS
Total # samples <10 ppb by
reference method
Total # false positive (Y)
Percent false positive
False Positive
Unit #1 (Y/N) Unit #1
(ppb) Qualitative
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
8.83 N
8.46 N
50
1
2%
False Positive
Unit #2 (Y/N) Unit #2
(ppb) Qualitative
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
<1 N
7.76 N
9.34 N
50
0
0%
Reference
(ppb)
1.50
1.48
1.42
1.50
<0.
<0.
<0.
<0.
<0.
<0.
<0.
<0.
9.97
9.87
35
-------
Table 6-11 a. Rates of False Negatives for the Technical Operator
Description
PT- 30 ppb As
PT- 100 ppb As
PT- 10 ppb As + low level
interferents
PT- 10 ppb As + high level
interferents
Olentangy River water LFM
Residential well water LFM
Alum Creek Reservoir water
LFM
QCS
Total # samples >10 ppb by
reference method
Total # false negatives (Y)
% False negatives
Unit #1
(ppb)
25.0
24.1
27.8
27.8
93.1
105.0
105.0
95.9
10.3
10.6
10.0
10.6
11.3
11.2
11.6
9.63
8.36
9.30
8.38
9.47
7.34
8.04
9.60
7.68
8.90
9.64
9.22
False
Negative
(Y/N)
Unit #1
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Y
N
N
N
N
N
N
27
1
4%
Unit #2
(ppb)
25.7
25.1
23.9
21.0
81.8
84.1
85.6
73.7
8.48
8.87
9.58
8.39
8.56
8.07
9.51
7.63
8.27
7.90
8.16
9.11
7.22
6.84
6.86
6.26
8.71
8.35
8.96
False
Negative
(Y/N)
Unit #2
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
N
N
N
Y
N
Y
N
N
Y
Y
Y
Y
N
N
N
27
6
22%
Reference
(ppb)
29.7
29.4
29.9
29.9
99.8
100.8
100.2
100.8
10.2
10.3
10.2
10.1
10.3
10.1
10.1
10.6
10.7
10.8
10.1
10.2
10.2
10.3
10.2
10.2
10.0
10.0
10.2
36
-------
Table 6-1 Ib. Rates of False Negatives for the Non-Technical Operator
Description
PT- 10 ppb As
PT- 30 ppb As
PT- 100 ppb As
Olentangy River water LFM
Residential well water LFM
Alum Creek Reservoir water
LFM
QCS
Total # samples >10 ppb by
reference method
Total # false negatives (Y)
% False negatives
Unit #1
(ppb)
6.70
9.93
8.58
9.98
26.8
25.9
24.4
26.0
83.2
95.2
85.9
87.7
7.84
11.9
8.96
9.17
8.10
10.2
8.36
0.00
5.09
9.84
False
Negative
(Y/N)
Unit #1
Qualitative
Y
N
N
N
N
N
N
N
N
N
N
N
Y
N
N
N
N
N
N
Y
Y
N
22
4
18%
Unit #2
(ppb)
8.79
9.73
9.64
9.79
26.2
25.2
24.0
24.7
75.8
81.4
82.9
82.6
7.78
10.9
7.83
9.94
9.59
6.56
7.41
8.56
8.06
7.96
False
Negative
(Y/N)
Unit #2
Qualitative
N
N
N
N
N
N
N
N
N
N
N
N
Y
N
Y
N
N
Y
Y
N
N
N
22
4
18%
Reference
(ppb)
10.2
10.3
10.2
10.3
29.7
29.4
29.9
29.9
99.8
100.8
100.2
100.8
10.6
10.7
10.8
10.3
10.3
10.3
10.4
10.1
10.2
10.2
6.3 Other Factors
During testing activities, the technical and non-technical operators were instructed to keep a
record of their comments on ease of use, reliability, portability, and generation of waste
materials. This section summarizes these observations and other comments pertaining to any
problems encountered during testing. Cost information is also presented.
6.3.1 Ease of Use
The technical and non-technical operators both reported that the SafeGuard was very easy to use.
The manual and the software program were clear and easy to follow. No solution or sample
preparation is necessary. The sample bottle is screwed into the instrument, and the software is
started. Thirty to 50 minutes later, the arsenic concentration of the sample is presented on the
37
-------
screen. This reading is automatically recorded in a spreadsheet that will open with Microsoft™
Excel. The software program is designed for both technical and non-technical operators. The
basic measurement mode of operating is described above, and an administrator level of operation
produces analysis diagrams and shows more detailed information about performance. Control
over the communication and configuration of the instrument is also available when in
administrator mode. The ease of use of the administrator mode was not evaluated by the
operators in this test.
6.3.2 Analysis Time
When started, the SafeGuard goes through an automatic system initialization, which takes about
10 minutes. Subsequently, there is a regular calibration sequence that occurs every 4 hours or
every four readings. This operation is also automated and takes about 10 minutes as well. The
analysis time per sample at room temperature is 30 to 50 minutes. The software program displays
a timer that counts down the time remaining.
6.3.3 Reliability
Overall, the SafeGuard operated reliably throughout the test. Pop-up messages occurred for three
main reasons on both units. The computer (comm ports) lost communication with the instrument
periodically. This error was easily remedied by closing the software and rebooting the
SafeGuard, or reassigning the comm ports. Another error message displayed was the "Overflow
script text...," which was considered a non-detect. This occurred when the software used zero in
the denominator to calculate the arsenic concentration. When there was an issue with the
SafeGuard, a message appeared on the screen describing the problem and how to fix it. The
message was not because of an error, but prompted the user to perform maintenance. For
example, when the sensitivity was low, the SafeGuard notified the operator and displayed
instructions to clean the sensor and run calibration. When troubleshooting was necessary (for the
communication and overscript issues), technical support was provided by the vendor over the
phone. The issues were clearly explained and quickly remedied.
6.3.4 Waste Material
The SafeGuard used standard addition to make multiple readings in calculating the actual
concentration of the samples. Because of this, the SafeGuard generated a considerable amount of
liquid waste. The SafeGuard used approximately 15 mL of sample and generated about 50 mL of
waste per sample. It used a 500-ppb standard for calibration and standard additions. An estimated
maximum concentration of arsenic in the waste is about 5 ppb above the native sample concen-
tration. This includes all line priming, two standard additions, and line flushing for each sample.
Since arsenic is added to every sample by the SafeGuard, the waste water must be disposed of as
arsenic-containing waste.
6.3.5 Cost and Consumables
The listed price for the SafeGuard at the time of the verification test was $35,000. The reagent
kits for arsenic analysis, which can analyze 45 samples, were available for $80. These kits have a
6-month shelf life as received and are stored at room temperature. Also, reagents from the kits
may be purchased separately, if an entire kit is not consumed. Sample bottles for the SafeGuard
38
-------
are standard narrow-mouth, threaded polytetrafluoroethylene bottles that can be purchased from
most laboratory supply vendors.
39
-------
Chapter 7
Performance Summary
The SafeGuard was verified by evaluating the following parameters:
• Accuracy
• Precision
• Linearity
• MDL
• Matrix interference effects
• Operator bias
• Inter-unit reproducibility
• Rate of false positives/false negatives.
Accuracy was assessed by comparing the results to Method 200.8(2) results from ICP-MS
analysis. The quantitative assessment of accuracy indicated that the relative bias for the
SafeGuard ranged from -28% to 7% for the technical operator and -28% to 11% for the non-
technical operator (excluding residential well water samples at approximately 600% due to
matrix effect).
Precision was assessed by analyzing four replicates of each sample. For the technical operator,
precision expressed as RSD ranged from 3% to 44%; and, for the non-technical operator, it
ranged from 2% to 38%. The average RSD for PT samples only was 10% for the technical
operator and 9% for the non-technical operator. These results exclude samples for which one or
more of the replicate results was not detected by the SafeGuard.
The linearity of response was evaluated by plotting the SafeGuard results against the reference
analysis results for the PT samples. All linear regressions against the reference method results
had coefficients of determination (r2) greater than 0.99. The 95% confidence intervals for the
slopes indicate that only the technical operator data for Unit #1 were consistent with a slope of 1
and were not significantly different from the reference analysis results. The 95% confidence
intervals for the y-axis intercept included zero for both operators on both units indicating no
significant difference from the reference analysis results.
The MDL was assessed by analyzing seven replicates of a sample spiked at a level
approximately five times the manufacturer's estimated detection limit for the SafeGuard (i.e.,
1 ppb x 5 = 5 ppb). The MDLs calculated using the precision data from these replicates ranged
from 2.0 ppb to 3.8 ppb.
40
-------
Results for samples containing low and high levels of interfering compounds indicated that
neither level of interference appeared to affect the detection of arsenic, with bias ranging from
-19% to 7%, consistent with the bias observed in the absence of interferences. The SafeGuard
performance was affected by one of the environmental samples, the residential well water. The
native (unspiked) replicates of this sample from both operators and both SafeGuard units
reported an arsenic concentration from 2.50 ppb to 9.70 ppb, whereas the reference method
reported this sample at 0.89 ppb to 1.12 ppb.
Operator bias was evaluated by comparing the SafeGuard results above the detection limit
produced by the technical and non-technical operators for all PT and environmental samples. The
95% confidence interval includes a slope of 1 for Unit #2, but the 95% confidence interval does
not include a slope of 1 for Unit #1, indicating a significant operator bias (technical results
> non-technical results) with that unit. Paired t-tests of the two sets of data indicate that the
SafeGuard results were not significantly different at a 0.05 level of significance depending on the
operator. Overall, these results indicate, at most, a small operator bias with one of the two
SafeGuard units.
Inter-unit reproducibility was evaluated by comparing the data for the two SafeGuard units.
Linear regressions of the data for each unit show that Unit #2 readings were lower than Unit #1
readings with both operators, but more strongly with the technical operator. Neither 95%
confidence interval includes a slope of 1, indicating a significant inter-unit bias that is more
pronounced with the technical operator than with the non-technical operator. A paired t-test of
the data indicated that the results from the two units with the technical operator were
significantly different at a 0.05 level of significance; however, the results from the two units with
the non-technical operator were not significantly different. Overall, these results show an inter-
unit bias with the technical operator, but minimal bias with the non-technical operator.
The rates of false positives for the SafeGuard were 0% for both units for the technical operator
and 2% and 0% for the non-technical operator (Units #1 and #2, respectively). The rates of false
negatives for the SafeGuard units were 4% and 22% for the technical operator and 18% for both
units for the non-technical operator. By averaging these rates, the results indicate that the
SafeGuard correctly identified water below the federal drinking water standard (<10 ppb) 99.5%
of the time (0.995 = sensitivity) and identified water that did not meet the federal standard
(>10 ppb) 85.4% of the time (0.854 = specificity).
The SafeGuard was easy to use, and the manual and software program were clear and easy to
follow. All reagent mixing and instrument flushing are automated. No solution or sample
preparation was necessary. Because the SafeGuard uses standard addition (two additions per
sample) to make multiple readings to calculate arsenic concentrations, roughly 50 mL of arsenic-
containing waste is generated per sample analyzed, which requires special disposal. The analysis
time per sample at room temperature was 30 to 50 minutes. The listed price for SafeGuard at the
time of the verification test was $35,000, and the cost for a 45-sample reagent kit was $80.
Replacement reagents and supplies are available without purchasing entire kits.
41
-------
Chapter 8
References
1. Test/QA Plan for Verification of Portable Analyzers, Battelle, Columbus, Ohio, Version 1.
December 8, 2000 (amended).
2. U.S. EPA Method 200.8, Determination of Trace Elements in Waters and Wastes by
Inductively Coupled Plasma Mass Spectrometry, Revision 5.5, October, 1999.
3. Federal Register, Vol. 66 No. 14, January 22, 2001. Part VIII, Environmental Protection
Agency. 40 CFR Parts 9, 141, and 142: National Primary Drinking Water Regulations;
Arsenic and Clarifications to Compliance and New Source Contaminants Monitoring: Final
Rule. http://www.epa.gov/safewater/ars/arsenic_finalrule.pdf.
4. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Center,
Version 6.0, U.S. EPA Environmental Technology Verification Program, Battelle,
Columbus, Ohio, November 2005.
42
------- |