August 2006 Environmental Technology Verification Report TRACEDETECT SAFEGUARD TRACE METAL ANALYZER Prepared by Battelle Baltelle I "he Husiness *.*/ Innovation Under a cooperative agreement with ^jf CrTr\ U.S. Environmental Protection Agency ET1/ET1/ET1/ ------- ------- August 2006 Environmental Technology Verification Report ETV Advanced Monitoring Systems Center Trace Detect SafeGuard Trace Metal Analyzer by Anne Gregg Tom Kelly Zachary Willenberg Amy Dindal Karen Riggs Battelle Columbus, Ohio 43201 ------- Notice The U.S. Environmental Protection Agency (EPA), through its Office of Research and Development, has financially supported and collaborated in the extramural program described here. This document has been peer reviewed by the Agency. Mention of trade names or commercial products does not constitute endorsement or recommendation by the EPA for use. 11 ------- Foreword The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the nation's air, water, and land resources. Under a mandate of national environmental laws, the Agency strives to formulate and implement actions leading to a compatible balance between human activities and the ability of natural systems to support and nurture life. To meet this mandate, the EPA's Office of Research and Development provides data and science support that can be used to solve environmental problems and to build the scientific knowledge base needed to manage our ecological resources wisely, to understand how pollutants affect our health, and to prevent or reduce environmental risks. The Environmental Technology Verification (ETV) Program has been established by the EPA to verify the performance characteristics of innovative environmental technology across all media and to report this objective information to permitters, buyers, and users of the technology, thus substantially accelerating the entrance of new environmental technologies into the marketplace. Verification organizations oversee and report verification activities based on testing and quality assurance protocols developed with input from major stakeholders and customer groups associated with the technology area. ETV consists of six environmental technology centers. Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/. Effective verifications of monitoring technologies are needed to assess environmental quality and to supply cost and performance data to select the most appropriate technology for that assessment. Under a cooperative agreement, Battelle has received EPA funding to plan, coordinate, and conduct such verification tests for "Advanced Monitoring Systems for Air, Water, and Soil" and report the results to the community at large. Information concerning this specific environmental technology area can be found on the Internet at http://www.epa.gov/etv/ centers/center 1 .html. in ------- Acknowledgments The authors wish to acknowledge the support of all those who helped plan and conduct the verification test, analyze the data, and prepare this report. We would like to thank Kenneth M. Hill and Paul J. Ponturo, Suffolk County Department of Health Services; Marty Link, Water Quality Division, Nebraska Department of Environmental Quality; and Jeff Adams, U.S. Environmental Protection Agency National Risk Management Research Laboratory for their careful review of this verification report. IV ------- Contents Notice ii Foreword iii Acknowledgments iv Contents v List of Abbreviations viii List of Abbreviations viii Chapter 1 Background 1 Chapter 2 Technology Description 2 Chapter 3 Test Design and Procedures 4 3.1 Introduction 4 3.2 Test Design 4 3.3 Test Samples 5 3.3.1 QC Samples 6 3.3.2 PT Samples 7 3.3.3 Environmental Samples 8 3.4 Reference Analysis 8 3.5 Verification Schedule 9 Chapter 4 Quality Assurance/Quality Control 10 4.1 Laboratory QC for Reference Method 10 4.2 Audits 13 4.2.1 Performance Evaluation Audit 13 4.2.2 Technical Systems Audit 13 4.2.3 Data Quality Audit 13 4.3 QA/QC Reporting 14 4.4 Data Review 14 Chapters Statistical Methods 15 5.1 Accuracy 15 5.2 Precision 15 5.3 Linearity 16 5.4 Method Detection Limit 16 5.5 Matrix Interference Effects 16 5.6 Operator Bias 16 ------- 5.7 Inter-Unit Reproducibility 16 5.8 Rate of False Positives/False Negatives 17 Chapter 6 Test Results 18 6.1 QC Samples 18 6.2 PT and Environmental Samples 22 6.2.1 Accuracy 22 6.2.2 Precision 22 6.2.3 Linearity 27 6.2.4 Method Detection Limit 29 6.2.5 Matrix Interference Effects 29 6.2.6 Operator Bias 29 6.2.7 Inter-Unit Reproducibility 30 6.2.8 Rate of False Positives/False Negatives 31 6.3 Other Factors 37 6.3.1 Ease of Use 37 6.3.2 Analysis Time 38 6.3.3 Reliability 38 6.3.4 Waste Material 38 6.3.5 Cost and Consumables 38 Chapter 7 Performance Summary 40 Chapters References 42 Figures Figure 2-1. TraceDetect SafeGuard 2 Figure 6-1 a. Linearity of SafeGuard Results for the Technical Operator 29 Figure 6-lb. Linearity of SafeGuard Results for the Non-Technical Operator 29 Figure 6-2. Comparison of SafeGuard Results for Technical and Non-Technical Operators 31 Figure 6-3. Comparison of SafeGuard Results for Units # 1 and # 2 32 Tables Table 3-1. Test Samples for Verification of the SafeGuard 6 Table 3-1. Test Samples for Verification of the SafeGuard (continued) 7 VI ------- Table 3-2. Schedule of Verification Test Days at Battelle Laboratory 9 Table 4-1. Reference Method QCS Analysis Results 11 Table 4-2. Reference Method Analytical Spike Results 12 Table 4-3. Reference Method Duplicate Analysis Results 12 Table 4-4. Reference Method PE Audit Results 13 Table 4-5. Summary of Data Recording Process 14 Table 6-la. RB Sample Results for the Technical Operator 19 Table 6-lb. RB Sample Results for the Non-Technical Operator 19 Table 6-2a. QCS Results for the Technical Operator 20 Table 6-2b. QCS Results for the Non-Technical Operator 20 Table 6-3a. LFM Sample Results for the Technical Operator 21 Table 6-3b. LFM Sample Results for the Non-Technical Operator 21 Table 6-4a. SafeGuard and Reference Sample Results for the Technical Operator 23 Table 6-4b. SafeGuard and Reference Sample Results for the Non-Technical Operator 25 Table 6-5. Quantitative Evaluation of Accuracy for the SafeGuard(a) 26 Table 6-6. Precision Results for the SafeGuard 27 Table 6-7. Summary of Linear Regression Equations for SafeGuard and Reference Results 28 Table 6-8. Detection Limit Results for SafeGuard 29 Table 6-9. Summary of Linear Regression Equations for Assessing Operator Bias and Inter- unit Reproducibility 31 Table 6-10a. Rates of False Positives for the Technical Operator 32 Table 6-10b. Rates of False Positives for the Non-Technical Operator 34 Table 6-1 la. Rates of False Negatives for the Technical Operator 36 Table 6-1 Ib. Rates of False Negatives for the Non-Technical Operator 37 vn ------- List of Abbreviations AMS As ASTM ASV EPA ETV HOPE ICP-MS L LFM MCL MDL mL mm ND NIST ppb ppm PE PT QA QA/QC QC QCS QMP r2 RB RPD RSD ISA Advanced Monitoring Systems arsenic American Society for Testing and Materials anodic stripping voltammetry U.S. Environmental Protection Agency Environmental Technology Verification high-density polyethylene inductively coupled plasma mass spectrometry liter laboratory-fortified matrix maximum contaminant level method detection limit milliliter millimeters non-detect National Institute of Standards and Technology parts per billion parts per million performance evaluation performance test quality assurance quality assurance/quality control quality control quality control sample Quality Management Plan coefficient of determination reagent blank relative percent difference relative standard deviation technical systems audit Vlll ------- Chapter 1 Background The U.S. Environmental Protection Agency (EPA) supports the Environmental Technology Verification (ETV) Program to facilitate the deployment of innovative environmental technologies through performance verification and dissemination of information. The goal of the ETV Program is to further environmental protection by accelerating the acceptance and use of improved and cost-effective technologies. ETV seeks to achieve this goal by providing high- quality, peer-reviewed data on technology performance to those involved in the design, distribution, financing, permitting, purchase, and use of environmental technologies. ETV works in partnership with recognized testing organizations; with stakeholder groups consisting of buyers, vendor organizations, and permitters; and with the full participation of individual technology developers. The program evaluates the performance of innovative technologies by developing test plans that are responsive to the needs of stakeholders, conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing peer-reviewed reports. All evaluations are conducted in accordance with rigorous quality assurance (QA) protocols to ensure that data of known and adequate quality are generated and that the results are defensible. The EPA's National Exposure Research Laboratory and its verification organization partner, Battelle, operate the Advanced Monitoring Systems (AMS) Center under ETV. The AMS Center recently evaluated the performance of the TraceDetect SafeGuard trace metals analysis system. SafeGuard was used to measure total arsenic in water in this verification test. ------- Chapter 2 Technology Description The objective of the ETV AMS Center is to verify the performance characteristics of environmental monitoring technologies for air, water, and soil. This report provides results for the verification testing of the TraceDetect SafeGuard trace metal analyzer. Following is a description of the technology, based on information provided by the vendor. The information provided below was not verified in this test. TraceDetect's SafeGuard is designed to automatically measure total arsenic (As) concentrations in drinking water samples (including raw water and treated water) over a range from 1 part per billion (ppb) to over 100 ppb. Once the operator has introduced the sample vial and selected "measure" on the control computer, all calibrations, dilutions, reductions, standard additions, and measurements are performed by the SafeGuard with the results displayed and logged in a data file. The software program is designed for both technical and non-technical operators, by having a basic mode of operation and an administrator mode of operation. This level of operation produces analysis diagrams and shows more detailed information about performance. Control over the communication and configuration of the instrument is also available when in "administrator" mode. Figure 2-1. TraceDetect SafeGuard The SafeGuard consists of three main components: the expert system, the fluidics system, and TraceDetect's patented NanoBand™ sensor and potentiostat. Each of these components has a part in the measurement process from controlling the pumps, to adding chemicals, to making measurements and interpreting the results. ------- The SafeGuard uses anodic stripping voltammetry (ASV) and the method of standard addition to make metals measurements. ASV is an electro-analytical method that detects ions in a solution by the potential at which they oxidize and strip away from the surface of an electrode. The SafeGuard is able to measure As (III) and reduce As (V) to As (III) to measure total arsenic. It can also be configured to analyze copper, lead, zinc, cadmium, and mercury in water. The SafeGuard stores data for every measurement and operation. The base of the SafeGuard is 15 inches by 28 inches (381 millimeters [mm] by 711 mm). It is 22 inches (559 mm) high and requires a computer, mouse, monitor, and keyboard. The TraceDetect SafeGuard as configured for measuring arsenic during this verification test was priced at $35,000, excluding options that the customer may require for unique sample preparation (e.g., copper removal from samples, filters for high turbidity samples). ------- Chapter 3 Test Design and Procedures 3.1 Introduction This verification test was conducted according to procedures specified in the Test/QA Plan for Verification of Portable Analyzers^ as amended December 4, 2002; January 24, 2003; and December 19, 2005. The verification was based on comparing the arsenic results from the SafeGuard to those from a laboratory-based reference method. The reference method for arsenic analysis was inductively coupled plasma mass spectrometry (ICP-MS) performed according to EPA Method 200.8.(2) The SafeGuard performance was verified by analyzing laboratory- prepared performance test (PT) samples, treated and untreated drinking water samples, and fresh surface water samples. All samples were tested using both the SafeGuard and the reference method. The test design and procedures are described further below. 3.2 Test Design The SafeGuard was verified by evaluating the following parameters: • Accuracy • Precision • Linearity • Method detection limit (MDL) • Matrix interference effects • Operator bias • Inter-unit reproducibility • Rate of false positives/false negatives. All sample preparation and analyses were performed according to the vendor's recommended procedures and the test/QA plan. The results from the SafeGuard were compared to those from the reference method to assess accuracy and linearity. Multiple aliquots of PT samples, drinking water samples, and surface water samples were analyzed to assess precision. Multiple aliquots of a low-level PT sample were analyzed to assess the detection limit of the SafeGuard. Potential matrix interference effects were assessed by challenging the SafeGuard with PT samples of known arsenic concentrations that also contained either low levels or high levels of potentially interfering substances. All samples were analyzed using two identical SafeGuard units ------- (designated Unit #1 and Unit #2). Results of analyses from the two units were statistically compared to evaluate inter-unit reproducibility. Operator bias was assessed by two Battelle staff members of differing technical expertise who operated the SafeGuard. Both the relatively skilled (technical) and relatively unskilled (non- technical) operators used both Units #1 and #2 to analyze identical sets of samples. The results from the two operators were reported separately, and the two sets of results were compared to determine operator bias. The rates of false positive and false negative results were evaluated relative to the 10-ppb maximum contaminant level (MCL) for arsenic in drinking water.(3) Other factors that were qualitatively assessed during the test included ease of use, time required for sample analysis, and reliability. 3.3 Test Samples Three types of samples were analyzed in the verification test, as shown in Table 3-1: quality control (QC) samples, PT samples, and environmental water samples. The QC and PT samples were prepared from National Institute of Standards and Technology (NIST)-traceable standards purchased from a commercial supplier and subject only to dilution as appropriate. Under the Safe Drinking Water Act, the EPA lowered the MCL for arsenic from 50 ppb to 10 ppb in January 2001; public water supply systems were required to comply with this standard by January 2006.(3) Therefore, the QC sample concentrations targeted the 10-ppb arsenic level. The PT samples ranged from 10% to 1,000% of the 10-ppb level (i.e., from 1 ppb to 100 ppb). The seven MDL replicates were randomly distributed throughout the PT samples. The environmental water samples were collected from various drinking water and surface freshwater sources. According to the test/QA plan (Section 4.1), the QC and PT samples were to be prepared within two days of analysis and stored at approximately 4°C until use. The QC and PT samples were prepared for each operator to analyze on two instruments and as needed in 1-liter batches. However, because of the length of time needed to acquire each measurement (30 to 50 minutes), preparing solutions at the suggested rate would have meant that there would have been many more samples for the reference analysis, as well as the extra labor to prepare and analyze these extra samples. Extending the holding time to three weeks is acceptable because arsenic is a very stable element, and the PT and QC samples were prepared in a clean matrix of ASTM Type 1 water. Also, PT and QC samples were stored at room temperature instead of 4°C, because the vendor's recommended analysis conditions for the SafeGuard are at room temperature. Allowing the samples to come to room temperature from 4°C every day would have substantially decreased the average number of samples analyzed per day, extending the length of the test. The environmental samples were stored at 4°C until analysis to minimize bacterial growth. These deviations from the test/QA plan did not affect the integrity of the samples or the final results of the test. The ICP-MS results of all samples were within 10% of the prepared concentration, confirming the stability of the samples. The analysis was done under the vendor's suggested conditions and is consistent with arsenic analysis in drinking water by ICP-MS. Fresh ------- QC and PT samples were prepared for each operator. This kept the age of the samples under 3 weeks and the samples provided to the two operators as similar as possible. Each sample was assigned a unique sample identification number when prepared in the laboratory or collected in the field. Each replicate of a sample had the identification number and a consecutive letter (i.e., a, b, c, or d). All SafeGuard samples were analyzed at room temperature without preservative. All samples were analyzed without pretreatment except the drinking water samples collected from plumbing (Battelle and residential well drinking waters). Following the vendor's instructions, samples that traveled through pipes were filtered to remove possible copper contamination from copper piping and brass fittings before analysis. 3.3.1 QC Samples QC samples included laboratory reagent blanks (RB), quality control samples (QCS), and laboratory-fortified matrix (LFM) samples (Table 3-1). The RB samples consisted of the same American Society for Testing and Materials (ASTM) Type I water used to prepare all other samples and were subjected to the same handling and analysis procedures as the other samples. The RB samples were used to verify that no arsenic contamination was introduced during sample handling and analysis. Ten percent of all samples analyzed were RB samples. The QCS consisted of ASTM Type I water spiked in the laboratory to a concentration of 10 ppb of arsenic with a NIST-traceable arsenic standard. QCSs were used to ensure the proper calibration of the SafeGuard. The SafeGuard was factory calibrated so no additional calibration was performed by the operators. However, QCSs were still analyzed (without defined performance limits) by the SafeGuard to demonstrate its proper functioning to the operator. QCSs were analyzed as the first and last samples, as well as after every tenth sample. The LFM samples consisted of environmental samples that were spiked in the laboratory to increase the arsenic concentration by 10 ppb. One LFM sample was prepared from each environmental sample. Table 3-1. Test Samples for Verification of the SafeGuard Type of Sample Quality Control(b) Sample Description Reagent blank (RB) Quality control sample (QCS) Laboratory-fortified matrix (LFM) Arsenic Concentration(a) -Oppb 10 ppb 10 ppb above native level No. of Replicates 10% of all First, last, and every 10th sample 1 per site ------- Table 3-1. Test Samples for Verification of the SafeGuard (continued) Type of Sample Performance Test(b) Sample Description Prepared arsenic solution Prepared arsenic solution Prepared arsenic solution Prepared arsenic solution Prepared arsenic solution Prepared arsenic solution for MDL Arsenic Concentration^ Ippb 3ppb lOppb 30 ppb lOOppb 5ppb No. of Replicates 4 4 4 4 4 7 Prepared arsenic solution spiked with low levels of interfering substances Prepared arsenic solution spiked with high levels of interfering substances lOppb lOppb Environmental Battelle drinking water (treated drinking) Olentangy River water (surface) Residential well water (untreated drinking) Alum Creek reservoir water (surface) 0.6 ppb 1 .4 ppb 1.0 ppb 1.5 ppb 4 4 4 4 (a) Target concentration for QCS and PT samples; measured native concentration for environmental samples (average of four replicate reference measurements). (b) Prepared in ASTM Type I water 3.3.2 PT Samples Three types of PT samples were used in this verification test (Table 3-1): spiked samples ranging from 1 ppb to 100 ppb arsenic, a low-level spiked sample for evaluation of the SafeGuard's MDL, and matrix interference samples that were spiked with potentially interfering substances. All PT samples were prepared in the laboratory using ASTM Type I water and NIST-traceable arsenic standards. Five PT samples containing arsenic at concentrations from 1 ppb to 100 ppb were prepared to evaluate the SafeGuard's accuracy and linearity. Four aliquots of each sample were analyzed to assess precision. ------- To determine the MDL of the SafeGuard, a PT sample was prepared with an arsenic concentration approximately five times the vendor-stated detection limit (i.e., 1 ppb x 5 = 5 ppb). Seven non-consecutive replicates of this 5-ppb arsenic sample were analyzed to provide precision data with which to estimate the MDL. The matrix interference samples were spiked with 10 ppb of arsenic, as well as potentially inter- fering species commonly found in natural water samples. One sample contained relatively low levels of interfering substances that consisted of 1 part per million (ppm) of iron, 0.1 ppm of sodium sulfide, and 3 ppm of sodium chloride. The second sample contained relatively high levels of interfering compounds at concentrations of 10 ppm of iron, 1.0 ppm of sodium sulfide, and 30 ppm of sodium chloride. Four replicates of each sample were analyzed to assess potential interferences. 3.3.3 Environmental Samples The environmental samples listed in Table 3-1 included two drinking water samples (treated and untreated) and two surface water samples collected in Columbus, Ohio. All environmental samples were collected in 1-liter (L) high density polyethylene (HDPE) bottles and analyzed in the Battelle laboratory. The Battelle drinking water sample was collected directly from a drinking water fountain without purging. Residential well water was collected from the spigots directly after the pressurized holding tank. Four aliquots of each sample were analyzed using the SafeGuard in the Battelle laboratory as soon as possible after collection. One aliquot of approximately 100 milliliters (mL) of each sample was preserved with nitric acid and submitted to the reference laboratory for reference analysis. One surface water sample was collected from Alum Creek Reservoir in Columbus, Ohio, and another was collected from the Olentangy River in Columbus, Ohio. These samples were collected near the shoreline by submerging 1-L FtDPE sample containers no more than one inch below the surface of the water. Four such containers were filled at one accessible location from each water source. The samples were transported to Battelle, and the four samples from each site were combined into a single 4-L volumetric flask to ensure homogeneity. The 4-L samples were then split into five samples, four for the replicate unspiked samples and one for the LFM, spiked at 10 ppb. One aliquot of approximately 100 mL of each sample was preserved with nitric acid and submitted for reference analysis. 3.4 Reference Analysis The reference arsenic analyses were performed in a Battelle laboratory using a Perkin Elmer Sciex Elan 6000 ICP-MS according to EPA Method 200.8, Revision 5.5.(2) The sample was introduced through a peristaltic pump by pneumatic nebulization into a radio frequency plasma where energy transfer processes caused desolvation, atomization, and ionization. The ions were extracted from the plasma through a pumped vacuum interface and separated on the basis of their mass-to-charge ratio by a quadrupole mass spectrometer. The ions transmitted through the quadrupole were registered by a continuous dynode electron multiplier, and the ion information was processed by a data handling system. ------- The ICP-MS was tuned, optimized, and calibrated according to Method 200.8 requirements and Battelle procedures. The calibration was performed using 11 calibration standards at concen- trations ranging from 0.1 to 250 ppb, and a minimum coefficient of determination (r2) of 0.999 was required. Internal standards were used to correct for instrument drift and physical inter- ferences. These standards were introduced in line through the peristaltic pump and analyzed with all blanks, standards, and samples. 3.5 Verification Schedule The verification test took place from March 2 through April 4, 2006. Table 3-2 shows the daily activities that were conducted during this period by the two operators. The reference analysis was performed on April 6, 2006, once all samples were analyzed by both operators. Subsamples for reference method analysis were collected and preserved with nitric acid when analyzed by SafeGuard. Table 3-2. Schedule of Verification Test Days at Battelle Laboratory Sample Preparation/ Collection Date Sample Analysis Date Non- Technical Technical Operator Operator Activity 3/2/06-4/4/06 3/21/06- 4/4/06 3/9/06 3/9/06 3/10/06 3/10/06 3/23/06 3/21/06 3/22/06 3/22/06 3/2/06 - Preparation and analysis of PT and associated QC 3/17/06 samples. 3/17/06 Collection and analysis of Battelle drinking water and associated QC samples. 3/14/06 Collection and analysis of Olentangy River water and associated QC samples. 3/16/06 Collection and analysis of residential well water and associated QC samples. 3/15/06 Collection and analysis of Alum Creek reservoir water and associated QC samples. ------- Chapter 4 Quality Assurance/Quality Control Quality assurance/quality control (QA/QC) procedures were performed in accordance with the quality management plan (QMP) for the AMS Center(4) and the test/QA plan for this verification test, except for the deviations noted in Section 3.3.(1) These deviations adapted the holding time and temperature of PT and QC samples. QA/QC procedures and results are described below. 4.1 Laboratory QC for Reference Method All reference analyses, including QC samples, were conducted on April 6, 2006. Laboratory QC for the reference method included the analysis of KB, QCS, analytical spike samples, and analytical duplicate samples. Laboratory KB samples were analyzed to ensure that no contamination was introduced by the sample preparation and analysis process. The accuracy of the ICP-MS calibration was verified after the analysis of every 10 samples by analyzing a QCS at 25 ppb. The percent recovery of the QCS was calculated from the following equation: % Recovery = ^xlOO ^ s where Cs is the measured concentration of the QCS and s is the spike concentration. If the QCS analysis differed by more than 10% from the true value of the standard, the instrument was recalibrated before continuing the test. As shown in Table 4-1, all reference QCS analyses were within the required range. 10 ------- Table 4-1. Reference Method QCS Analysis Results Sample ID CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 CCV25 Measured (ppb) 25.13 24.99 25.05 24.74 25.11 25.04 25.11 24.75 24.80 24.78 24.67 24.46 23.29 Actual (ppb) 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 25.00 % Recovery (R) 101 100 100 99 100 100 100 99 99 99 99 98 93 Spiked samples were analyzed to assess whether matrix effects influenced the reference method results. There was an analytical spike every 10th sample in the sequence per EPA Method 200.8.(3) The analytical spike percent recovery (R) was calculated from the following equation: R = Cs~C xlOO (2) s where Cs is the measured concentration of the spiked sample, C is the measured concentration of the unspiked sample, and s is the spike concentration. If the percent recovery of an analytical spike fell outside the range from 85% to 115%, a matrix effect was suspected. As shown in Table 4-2, all of the LFM sample results were within this range. Analytical duplicate samples were analyzed to assess the precision of the reference analysis. There was an analytical duplicate sample every 10 samples in the sequence per EPA Method 200.8.(3) The relative percent difference (RPD) of the duplicate sample analysis was calculated from the following equation: RPD= "g x10Q (3) (C + CD)/2 where C is the concentration of the sample analysis, and CD is the concentration of the duplicate sample analysis. If the RPD was greater than 10%, the instrument was recalibrated before continuing the test. As shown in Table 4-3, the RPDs for the duplicate analyses were all less than 10%; in two samples non-detects were seen in both duplicate analyses. 11 ------- Table 4-2. Reference Method Analytical Spike Results Amount Sample ID 51385-02-06-B 51385-11-10-D 51385-02-28-C 51385-13-06-A 51385-15-02-D 51385-18-05-A 51385-15-14-B 51385-21-07-A 51385-23-03-C 51385-23-10-D 51385-17-25 51385-17-13 Matrix ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water ASTM Type I water Surface water Drinking water Unspiked (ppb) 9.87 5.08 2.99 99.83 9.33 10.14 5.00 3.02 10.12 10.12 10.83 9.75 Table 4-3. Reference Method Duplicate Analysis Sample Concentration Sample ID 51385-02-02-A 51385-11-05-C 51385-11-10-E 51385-02-28-D 51385-13-06-B 51385-18-02-A 51385-18-05-B 51385-15-14-C 51385-21-07-B 51385-23-03-D 51385-16-15-A 51385-17-02-A (ppb) <0.10 10.33 5.24 3.01 100.8 <0.10 10.22 5.02 3.05 9.93 1.38 0.89 Spiked (ppb) 36.65 31.09 29.67 128.5 35.19 36.90 32.11 29.42 36.30 36.40 38.34 38.48 Results Duplicate Concentration (ppb) <0.10 10.40 5.23 3.02 102.0 <0.10 10.15 5.05 3.05 10.16 1.45 0.83 Spiked (ppb) 25 25 25 25 25 25 25 25 25 25 25 25 R (%) 107 104 107 115 103 107 108 106 105 105 110 115 RPD (%) - 0.7 0.2 0.3 1.2 - 0.7 0.6 0.0 2.3 4.9 6.3 12 ------- 4.2 Audits Three types of audits were performed during the verification test: a performance evaluation (PE) audit of the reference method, a technical systems audit (TSA) of the verification test performance, and a data quality audit. Audit procedures are described further below. 4.2.1 Performance Evaluation Audit A PE audit was conducted to assess the quality of the reference measurements made in this verification test. For the PE audit, two independent NIST-traceable reference materials were obtained from different commercial suppliers. One was used for the calibration standards in the reference analysis and the other used to prepare the QCS, PT, LFM, and PE samples. Accuracy of the reference method was verified by comparing the arsenic concentration measured based on the calibration standards to those obtained using the independently certified PE standard. RPD as calculated by Equation 3 was used to quantify the accuracy of the results. Agreement of the standard within 10% was required for the measurements to be considered acceptable. As shown in Table 4-4, the PE sample analysis was within the required range. Table 4-4. Reference Method PE Audit Results Measured Independent Standard Arsenic Concentration Concentration RPD Sample ID (ppb) (ppb) PE 26.60 25.00 6.2 4.2.2 Technical Systems A udit An independent Battelle Quality management staff member conducted a TSA to ensure that the verification test was being conducted in accordance with the test/QA plan(1) and the AMS Center QMP.(4) As part of the TSA, test procedures were compared to those specified in the test/QA plan,(1) and data acquisition and handling procedures as well as the reference method procedures were reviewed. Observations and findings from the TSA were documented and submitted to the Battelle Verification Test Coordinator for response. None of the findings of the TSA required corrective action. TSA records are permanently stored with the Battelle Quality Manager. 4.2.3 Data Quality Audit At least 10% of the data acquired during the verification test were audited. Battelle's Quality Manager traced the data from the initial acquisition, through reduction and statistical analysis, to final reporting to ensure the integrity of the reported results. All calculations performed on the data undergoing the audit were checked. 13 ------- 4.3 QA/QC Reporting Each audit was documented in accordance with Sections 3.3.4 and 3.3.5 of the QMP for the ETV AMS Center.(4) Once the audit reports were prepared, the Battelle Verification Test Coordinator ensured that a response was provided for each adverse finding or potential problem and imple- mented any necessary follow-up corrective action. The Battelle Quality Manager ensured that follow-up corrective action was taken. The results of the TSA were submitted to the EPA. 4.4 Data Review Records generated in the verification test received a one-over-one review before these records were used to calculate, evaluate, or report verification results. Table 4-5 summarizes the types of data recorded and reviewed. Data were reviewed by a Battelle technical staff member involved in the verification test. The person performing the review added his/her initials and the date to a hard copy of the record being reviewed. Table 4-5. Summary of Data Recording Process Data to be Recorded Where Recorded How Often Recorded Disposition of Data Dates, times of test events Test parameters (temperature, analyte/ interferent identities, and all SafeGuard portable analyzer results) Reference method sample analysis, chain of custody, and results ETV laboratory record book and data acquisition system ETV laboratory record book and data acquisition system Laboratory record books or data acquisition system, as appropriate Start/end of test event and sample analysis Throughout sample handling and analysis process Throughout sample handling and analysis process Used to organize/check test results; transferred to spreadsheets electronically Used to organize/check test results, transferred to spreadsheets and manually incorporated in data spreadsheets as necessary Transferred to spreadsheets electronically 14 ------- Chapter 5 Statistical Methods The statistical methods used to evaluate the performance factors listed in Section 3.2 are presented in this chapter. Qualitative observations were also used to evaluate verification test data. 5.1 Accuracy All samples were analyzed by both the SafeGuard and reference methods. For each sample, accuracy was expressed in terms of a relative bias (B) as calculated from the following equation: CD (4) where d is the average difference between the SafeGuard results and the reference method results, and CR is the average of the reference measurements. 5.2 Precision When possible, the standard deviation (S) of the results for the replicate samples at each concentration was calculated and used as a measure of SafeGuard precision. Standard deviation was calculated from the following equation: S = 1 n- (5) where n is the number of replicate samples, Ck is the concentration measured for the kth sample, and C is the average concentration of the replicate samples. Precision was reported in terms of the relative standard deviation (RSD) as follows: RSD = S_ C xlOO (6) 15 ------- 5.3 Linearity Linearity was assessed by performing a linear regression of SafeGuard results against the reference results, with linearity characterized by the slope, intercept, and coefficient of determination (r2). Linearity was tested using the four analyses of each of the five PT samples over the range of 1 ppb to 100 ppb arsenic. Samples with results below the vendor-stated detection limit were not included in the analysis. Results from both SafeGuard units were plotted against the corresponding reference concentrations, and separate regressions were performed. 5.4 Method Detection Limit The MDL for the SafeGuard was assessed using results from both units for seven replicate analyses of a sample spiked with 5 ppb of arsenic. The standard deviation of the seven replicate results was calculated using Equation 5. The MDL was then calculated using the following equation: MDL = txS (7) where t is the Student's t value for a 99% confidence level, and S is the standard deviation of the seven replicate results. 5.5 Matrix Interference Effects The potential effect of interfering substances on the sensitivity of the SafeGuard was evaluated by calculating accuracy (expressed as bias) using Equation 4. These results were compared with accuracy results for PT samples containing only arsenic to assess whether there was a positive or negative effect due to matrix interferences. 5.6 Operator Bias The effect of operator skill level on the performance of the SafeGuard was assessed by comparing results from the two operators for all samples producing results above the detection limit. Two types of statistical evaluations were conducted. First, linear regression of SafeGuard results against reference results was conducted for all analyses by each operator, and the two regressions were compared to one another. Second, a paired t-test of the two data sets was conducted to assess whether the means of the results from the two operators were significantly different. This t-test was done separately for results from SafeGuard Unit #1 and Unit #2 and would indicate a significant difference at the 0.05 level if the two means differed by more than about 10%. 5.7 Inter-Unit Reproducibility Inter-unit reproducibility was assessed by performing a linear regression of sample results generated by the two units. The slope, intercept, and r2 were used to evaluate the degree of inter- 16 ------- unit reproducibility. A paired t-test was also conducted to evaluate whether the two sets of sample results were significantly different at a 95% confidence level. 5.8 Rate of False Positives/False Negatives The SafeGuard produced quantitative results over a range from 1 ppb to over 100 ppb. The purpose of the false positive/negative evaluation was to assess whether the SafeGuard produced comparable results to the reference value regarding the MCL level. The rates of false positives and false negatives produced by the SafeGuard were assessed relative to the 10-ppb target arsenic level for PT, QC, and environmental samples. A false positive result is defined as any result reported to be greater than 10 ppb and greater than 125% of the reference value, when the reference value is less than or equal to that guidance level. (The additional criterion to compare the SafeGuard result to 125% of the reference value was used to account for analytical uncertainty.) Similarly, a false negative result is defined as any result reported as below or equal to 10 ppb and less than 75% of the reference value, when the reference value is greater than that guidance level. The rates of false positives and false negatives were expressed as a percentage of total samples analyzed for each type of sample. The inverses of these rates are the specificity and sensitivity of the SafeGuard. These are given as probabilities. The specificity of the SafeGuard is the probability of correctly identifying arsenic levels above 10 ppb. The sensitivity of the instrument is the probability of correctly identifying arsenic levels below 10 ppb. 17 ------- Chapter 6 Test Results The SafeGuard is automated by a computer and a specific software program that displays the arsenic measurement in ppb and prompts the user when regular maintenance is necessary. When the measurement result was lower than the detection limit of 1 ppb, the SafeGuard reported the measurement as "less than" a concentration, for example < 0.52 ppb. These readings are reported here as < 1 ppb. Values denoted as non-detects (ND) are ones that had an error message that read, "Overflow script text - error # 6 - Line # 1213 - Column # 8." This error originated when the software system did not detect arsenic and consequently divided an equation by zero. TraceDetect gave instruction that this error message should be recorded as a non-detect. One of the method detection performance test sample results for the technical operator on unit #1 gave this error message. The accuracy, precision, and MDL were calculated excluding this result. 6.1 QC Samples As described in Section 3.3.1, the QC samples analyzed with the SafeGuard included KB, QCS, and LFM samples. Ten percent of all samples analyzed were KB samples, and the results were used to verify that no arsenic contamination was introduced during sample handling and analysis. QCSs were analyzed first, last, and after every tenth sample. The QCS results were used to verify that the system was operating properly; however, since the SafeGuard is not calibrated by the operator, they were analyzed without defined performance limits. One LFM sample was prepared from each environmental sample to evaluate potential matrix interferences. KB sample results for the SafeGuard are presented in Tables 6-la and b for the technical and non-technical operators, respectively. One replicate by the technical operator on Unit #1 was reported at 1.11 ppb. This was just above the detection limit specified by the vendor. All other KB results with both units and both operators, including an aliquot of the same KB before and after this result were below the SafeGuard's detection limit. All KB samples were analyzed by the reference method and were below the 1-ppb detection limit of the SafeGuard. It was concluded that arsenic contamination resulting from sample handling did not occur. QCS results for the technical and non-technical operators are presented in Tables 6-2a and 6-2b, respectively. The QCSs were analyzed first, last, and after every tenth sample, as required, except for one, when the non-technical operator inadvertently switched the last two samples of the test. The QCS was prepared at 10 ppb for the percent recovery calculated using Equation 1 (Section 4.1). The QCS percent recovery for the technical operator ranged from 63% to 96%. The QCS percent recovery for the non-technical operator ranged from 0% to 102%, because one 18 ------- QCS was not detected by SafeGuard Unit #1. Excluding this reading, the percent recovery for the non-technical operator ranged from 51% to 102%. One LFM sample was prepared from each environmental sample to evaluate potential matrix interferences. The LFM sample results for the technical and non-technical operators are presented in Tables 6-3a and 6-3b respectively. The R value associated with each LFM sample was calculated using Equation 2 (Section 4.1). Reference method results are also provided for comparison. One spiked sample of Battelle drinking water was not detected by Unit #1 when analyzed by the technical operator. Except for that sample, the lowest recoveries for the SafeGuard from both operators on both units (12% to 45%) were associated with the residential well water LFM sample. This was because of the substantial level of arsenic measured in the unspiked sample. However, arsenic was not detected above 1 ppb by the reference method in the unspiked sample. This indicates that a matrix effect was exaggerating the level of arsenic in the unspiked residential well water sample as reported by the SafeGuard. The other environmental samples did not noticeably affect the instrument because good recoveries were observed for those samples. Table 6-la. RB Sample Results for the Technical Operator Sample ID 51385-18-02-A 51385-18-02-B 51385-18-02-C 51385-18-02-D 51385-18-02-E 51385-18-02-F 51385-22-03-G 51385-22-03-H 51385-22-03-1 Replicate 1 2 3 4 5 6 1 2 3 Unit #1 Analysis Date (ppb) 3/20/2006 <1 3/21/2006 <1 3/22/2006 <1 3/23/2006 <1 3/27/2006 <1 3/30/2006 <1 4/3/2006 <1 4/4/2006 1.11 4/4/2006 <1 Unit #2 (ppb) <1 <1 <1 <1 <1 <1 <1 <1 <1 Table 6-lb. RB Sample Results for the Non-Technical Operator Sample ID Replicate 51385-02-02-A 1 51385-02-02-B 2 51385-11-02-C 1 51385-11-02-D 2 51385-1 1-02-E 3 51385-16-02-F 1 51385-16-02-G 2 51385-16-02-H 3 Unit #1 Analysis Date (ppb) 3/1/2006 <1 3/3/2006 <1 3/7/2006 <1 3/8/2006 <1 3/9/2006 <1 3/9/2006 <1 3/15/2006 ND 3/17/2006 <1 Unit #2 (ppb) <1 <1 <1 <1 <1 <1 <1 <1 ND = non-detect 19 ------- Table 6-2a. QCS Results for the Technical Operator Analysis Unit #1 Sample ID Replicate Date (ppb) 51385-18-05-A 51385-18-05-B 51385-18-05-C 51385-18-05-D 51385-18-05-E 51385-18-05-F 51385-22-08-G 51385-22-08-H 51385-22-08-1 Table 6-2b. QCS 1 2 3 4 5 6 1 2 O Results 3/20/2006 3/21/2006 3/22/2006 3/23/2006 3/27/2006 3/30/2006 4/3/2006 4/4/2006 4/4/2006 8.38 9.47 7.34 8.04 9.60 7.68 8.90 9.64 9.22 for the Non-Technical Amount Unit #2 Spiked (ppb) 8.16 9.11 7.22 6.84 6.86 6.26 8.71 8.35 8.96 Operator Unit#l Unit #2 Sample ID Replicate 51385-02-06-A 51385-02-06-B 51385-11-05-C 51385-11-05-D 51385-11-05-E 51385-16-06-F 51385-16-06-G 51385-16-06-H 51385-16-06-1 1 2 1 2 3 1 4 5 6 Analysis Date 3/2/2006 3/3/2006 3/7/2006 3/8/2006 3/9/2006 3/9 & 3/ll/2006a 3/11/2006 3/15/2006 3/17/2006 (ppb) 8.83 8.46 9.17 8.10 10.2 8.36 ND 5.09 9.84 (ppb) 7.76 9.34 9.94 9.59 6.56 7.41 8.56 8.06 7.96 (ppb) 10 10 10 10 10 10 10 10 10 Amount Spiked (ppb) 10 10 10 10 10 10 10 10 10 % Recovery Unit #1 84 95 73 80 96 77 89 96 92 % Recovery Unit #1 88 85 92 81 102 84 0 51 98 % Recovery Unit #2 82 91 72 68 69 63 87 84 90 % Recovery Unit #2 78 93 99 96 66 74 86 81 80 ND = non-detect, reported error message by SafeGuard due to division by zero a QCS was analyzed on Unit #1 (3/11/06) and Unit #2 (3/9/06) on different days 20 ------- Table 6-3a. LFM Sample Results for the Technical Operator Unspiked(a) Description (ppb) Battelle drinking water LFM Unit # 1 Unit # 2 Reference Olentangy River water LFM Unit # 1 Unit # 2 Reference Residential well water LFM Unit # 1 Unit # 2 Reference Alum Creek Reservoir water LFM Unit # 1 Unit # 2 Reference ND ND 0.62 <1 <1 1.38 6.31 7.11 0.98 <1 <1 1.48 Spiked Amount Spiked (ppb) (ppb) ND 9.40 9.75 9.63 7.63 10.6 8.36 8.27 10.7 9.30 7.90 10.8 10 10 10 10 10 10 10 10 10 10 10 10 R(b) (%) 0 94 91 96 76 92 21 12 97 93 79 94 (a) Average of four replicates. Non-detects and <1 ppb results were assigned a value of zero for calculation of average (b) Non-detects and < 1 ppb results were assigned a value of zero for calculation of R ND = non-detect, reported error message by SafeGuard due to division by zero Table 6-3b. LFM Sample Results for the Non-Technical Operator Description Battelle drinking water LFM Unit # 1 Unit # 2 Reference Olentangy River water LFM Unit # 1 Unit # 2 Reference Residential well water LFM Unit # 1 Unit # 2 Reference Alum Creek Reservoir water LFM Unit # 1 Unit # 2 Reference Unspiked00 (ppb) <1 <1 0.62 <1 <1 1.38 7.38 6.49 0.98 <1 <1 1.48 Spiked Amount Spiked (ppb) (ppb) 7.04 8.00 9.75 7.84 7.78 10.6 11.9 10.9 10.7 8.96 7.83 10.8 10 10 10 10 10 10 10 10 10 10 10 10 R(b) (%) 70 80 91 78 78 92 45 44 97 90 78 94 (a) Average of four replicates. < 1 ppb results were assigned a value of zero for calculation of average (b) Non-detects and <1 ppb results were assigned a value of zero for calculation of R 21 ------- 6.2 PT and Environmental Samples Tables 6-4a and 6-4b present the results for the PT and environmental samples for the technical and non-technical operators, respectively. Each table includes the SafeGuard results for both units and the reference method results. The SafeGuard results below the detection limit were assigned a value of <1 ppb. Results for each performance factor are presented below. 6.2.1 Accuracy Table 6-5 presents the accuracy results for the SafeGuard, expressed as relative percent bias as calculated by Equation 4 (Section 5.1). Percent bias was not calculated if any result for a set of replicates was below the detection limit (<1 ppb). The bias ranged from -28% to 629% for the technical operator and -28% to 657% for the non-technical operator. The high end of these ranges is due to the residential well water. The high bias for this environmental sample confirms the apparent matrix effect observed in the LFM sample with this matrix (see Section 6.2.5). Excluding the residential well water sample, the bias ranged from -28% to 7% and -28% to 11% for the technical and non-technical operators, respectively. 6.2.2 Precision Precision results for the SafeGuard are presented in Table 6-6. The RSD was determined as a percentage according to Equation 5 (Section 5.2). The RSD was not calculated if any of the results for a set of replicates were below the detection limit. The RSDs ranged from 3% to 44% for the technical operator and from 2% to 38% for the non-technical operator. The average RSD of the PT samples for the technical operator was 10% and the average RSD for the non-technical operator was 9% 22 ------- Table 6-4a. SafeGuard and Reference Sample Results for Description PT- 1 ppb As PT- 3 ppb As PT- 10 ppb As PT- 30 ppb As PT- 100 ppb As Detection Limit PT- 10 ppb As + low level interferents Sample ID 5 1385-2 1-02-A 51385-21-02-B 51385-21-02-C 51385-21-02-D 5 1385-2 1-07-A 51385-21-07-B 51385-21-07-C 5 1385-2 1-07-D 51385-21-12-A 51385-21-12-B 51385-21-12-C 51385-21-12-D 51385-13-02-A 51385-13-02-B 51385-13-02-C 51385-13-02-D 51385-13-06-A 51385-13-06-B 51385-13-06-C 51385-13-06-D 51385-15-14-A 51385-15-14-B 51385-15-14-C 51385-15-14-D 51385-22-13-E 51385-22-13-F 51385-22-13-G 51385-23-10-A 51385-23-10-B 51385-23-10-C 51385-23-10-D Replicate 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 5 6 7 1 2 3 4 Analysis Date 3/29/2006 3/29/2006 3/29/2006 3/29/2006 3/29/2006 3/29/2006 3/29/2006 3/30/2006 3/30/2006 3/30/2006 3/31/2006 3/31/2006 4/3/2006 4/3/2006 4/3/2006 4/3/2006 4/3/2006 4/3/2006 4/3/2006 4/3/2006 3/27/2006 3/29/2006 3/30/2006 3/31/2006 4/3/2006 4/3/2006 4/4/2006 4/4/2006 4/4/2006 4/4/2006 4/4/2006 the Technical Operator Unit #1 (ppb) <1 <1 <1 <1 2.48 2.87 1.77 2.50 8.52 8.70 9.32 8.65 25.0 24.1 27.8 27.8 93.1 105.0 105.0 95.9 <1 4.43 4.91 4.50 5.50 7.35 6.04 10.3 10.6 10.0 10.6 Unit #2 (ppb) 1.08 <1 1.11 <1 3.07 2.65 2.41 2.96 8.49 7.81 6.31 8.35 25.7 25.1 23.9 21.0 81.8 84.1 85.6 73.7 3.77 4.67 3.44 5.01 4.48 6.27 4.58 8.48 8.87 9.58 8.39 Reference (ppb) 1.12 1.09 1.09 1.10 3.02 3.05 3.04 3.01 9.56 9.88 9.85 9.96 29.7 29.4 29.9 29.9 99.8 100.8 100.2 100.8 5.10 5.00 5.02 5.05 5.20 5.23 5.11 10.2 10.3 10.2 10.1 23 ------- Table 6-4a. SafeGuard and Reference Sample Results for the Technical (continued) Description PT- 10 ppb As + high level interferents Battelle drinking water Battelle drinking water LFM Olentangy River water Olentangy River water LFM Residential well water Residential well water LFM Alum Creek Reservoir water Alum Creek Reservoir water LFM Sample ID 51385-23-03-A 51385-23-03-B 51385-23-03-C 51385-23-03-D 51385-16-10-A 51385-16-10-B 51385-16-10-C 51385-16-10-D 51385-17-13 51385-16-15-A 51385-16-15-B 51385-16-15-C 51385-16-15-D 51385-17-17 51385-17-02-A 51385-17-02-B 51385-17-02-C 51385-17-02-D 51385-17-21 51385-17-08-A 51385-17-08-B 51385-17-08-C 51385-17-08-D 51385-17-25 Replicate 1 2 3 4 1 2 3 4 1 1 2 3 4 1 1 2 3 4 1 1 2 3 4 1 Analysis Date 4/4/2006 4/4/2006 4/4/2006 4/4/2006 3/23/2006 3/23/2006 3/23/2006 3/23/2006 3/21/2006 3/21/2006 3/21/2006 3/21/2006 3/21/2006 3/20/2006 3/22/2006 3/22/2006 3/22/2006 3/22/2006 3/21/2006 3/22/2006 3/22/2006 3/22/2006 3/22/2006 3/20/2006 Unit #1 (ppb) 11.3 11.2 11.6 9.35 ND ND ND ND ND <1 <1 1.16 <1 9.63 6.20 7.27 6.90 4.85 8.36 <1 <1 <1 <1 9.30 Operator Unit #2 (ppb) 8.56 8.07 9.51 8.91 ND ND ND ND 9.40 <1 2.92 <1 <1 7.63 7.97 8.58 2.50 9.39 8.27 <1 <1 <1 <1 7.90 Reference (ppb) 10.3 10.1 10.1 9.93 0.56 0.66 0.63 0.61 9.75 1.38 1.41 1.41 1.33 10.6 0.89 1.12 0.95 0.94 10.7 1.50 1.48 1.42 1.50 10.8 ND = Non-detects, reported error message by SafeGuard due to division by zero. 24 ------- Table 6-4b. SafeGuard and Reference Sample Results for the Non-Technical Operator Description PT- 1 ppb As PT- 3 ppb As PT- 10 ppb As PT- 30 ppb As PT- 100 ppb As Detection Limit PT- 10 ppb As + low level interferents PT- 10 ppb As + high level interferents Battelle drinking water Battelle drinking water LFM Sample ID 51385-02-23-A 51385-02-23-B 51385-02-23-C 51385-02-23-D 51385-02-28-A 51385-02-28-B 51385-02-28-C 51385-02-28-D 51385-11-14-A 51385-11-14-B 51385-11-14-C 51385-11-14-D 51385-13-02-A 51385-13-02-B 51385-13-02-C 51385-13-02-D 51385-13-06-A 51385-13-06-B 51385-13-06-C 51385-13-06-D 51385-02-15-A 51385-02-15-B 51385-02-15-C 51385-11-10-D 51385-11-10-E 51385-11-10-F 51385-15-14-G 51385-15-08-A 51385-15-08-B 51385-15-08-C 51385-15-08-D 51385-15-02-A 51385-15-02-B 51385-15-02-C 51385-15-02-D 51385-16-10-A 51385-16-10-B 51385-16-10-C 51385-16-10-D 51385-17-13-A Replicate 1 2 3 4 1 2 o 3 4 1 2 3 4 1 2 3 4 1 2 o 3 4 1 2 o 3 4 5 6 7 1 2 3 4 1 2 o 3 4 1 5 7 9 1 Analysis Date 3/2/2006 3/2/2006 3/2/2006 3/2/2006 3/3/2006 3/3/2006 3/3/2006 3/3/2006 3/6/2006 3/6/2006 3/6/2006 3/6/2006 3/7/2006 3/7/2006 3/7/2006 3/7/2006 3/7/2006 3/7/2006 3/7/2006 3/7/2006 3/2/2006 3/3/2006 3/3/2006 3/6/2006 3/7/2006 3/7/2006 3/8/2006 3/8/2006 3/8/2006 3/9/2006 3/9/2006 3/9/2006 3/9/2006 3/9/2006 3/9/2006 3/9/2006 3/17/2006 3/17/2006 3/17/2006 3/17/2006 Unit #1 (ppb) 1.14 1.16 1.09 1.05 3.52 3.30 2.94 3.30 6.70 9.93 8.58 9.98 26.8 25.9 24.4 26.0 83.2 95.2 85.9 87.7 6.06 4.39 3.90 5.71 6.14 4.90 3.50 4.90 6.37 8.72 11.8 8.01 7.24 8.67 8.41 ND <1 ND <1 7.04 Unit #2 (ppb) 1.12 <1 <1 1.04 2.81 3.00 3.04 3.29 8.79 9.73 9.64 9.79 26.2 25.2 24.0 24.7 75.8 81.4 82.9 82.6 4.36 4.03 5.08 5.52 5.46 5.78 5.35 8.76 8.93 8.77 9.15 7.58 7.34 7.40 7.57 1.08 <1 <1 <1 8.00 Reference (ppb) 1.06 1.05 1.05 1.02 3.02 2.92 2.99 3.01 10.2 10.3 10.2 10.3 29.7 29.4 29.9 29.9 99.8 100.8 100.2 100.8 5.08 5.02 4.97 5.08 5.24 5.11 5.15 9.71 9.49 9.43 9.35 9.28 9.31 9.19 9.33 0.56 0.66 0.63 0.61 9.75 25 ------- Table 6-4b. SafeGuard (continued) Description Olentangy River water Olentangy River water LFM Residential well water Residential well water LFM Alum Creek Reservoir water Alum Creek Reservoir water LFM and Reference Sample Results for the Non-Technical Operator Sample ID 51385-16-15-A 51385-16-15-B 51385-16-15-C 51385-16-15-D 51385-17-17-A 51385-17-02-A 51385-17-02-B 51385-17-02-C 51385-17-02-D 51385-17-21-A 51385-17-08-A 51385-17-08-B 51385-17-08-C 51385-17-08-D 51385-17-25-A Replicate 1 2 3 4 1 1 6 7 8 1 1 2 3 4 1 Analysis Date 3/14/2006 3/14/2006 3/14/2006 3/14/2006 3/14/2006 3/15/2006 3/16/2006 3/16/2006 3/16/2006 3/16/2006 3/14/2006 3/15/2006 3/15/2006 3/15/2006 3/15/2006 Unit #1 (ppb) 1.65 7.84 9.70 4.75 7.48 7.60 11.9 ND 8.96 Unit #2 (ppb) J! 7.78 6.88 6.62 4.80 7.65 10.9 J! 7.83 Reference (ppb) 1.38 1.41 1.41 1.33 10.6 0.89 1.12 0.95 0.94 10.7 1.50 1.48 1.42 1.50 10.8 ND = Non-detects, reported by SafeGuard as error message because of division by zero. Table 6-5. Quantitative Evaluation of Accuracy for the SafeGuard (a) Bias Description PT Samples PT- 1 ppb As PT- 3 ppb As PT- 10 ppb As PT- 30 ppb As PT- 100 ppb As Detection limit PT- 10 ppb As + low level interferents PT- 1 0 ppb As + high level interferents Environmental Samples Battelle drinking water Battelle drinking water LFM Olentangy River water Olentangy River water LFM Residential well water Residential well water LFM Alum Creek Reservoir water Alum Creek Reservoir water LFM Technical Operator Unit #1 NA -21% -10% -12% -1% 7%00 2% 7% NA NA NA -9% 547% -22% NA -14% Technical Operator Unit #2 NA -8% -21% -19% -19% -10% -13% -13% NA -4% NA -28% 629% -23% NA -27% Non-Technical Operator Unit #1 6% 9% -14% -13% -12% -3% -16% -13% NA -28% NA -26% 657% 11% NA -17% Non-Technical Operator Unit #2 NA 2% -7% -16% -20% 0% -6% -19% NA -18% NA -27% 565% 2% NA -28% (a) Percent bias calculated according to Equation 4, Section 5.1. (b) One replicate result = non-detect, calculated bias excluding this result. NA = one or more replicates below the detection limit. 26 ------- Table 6-6. Precision Results for the SafeGuard RSD Description FT Samples PT- 1 ppb As PT- 3 ppb As PT- 10 ppb As PT- 30 ppb As PT- 100 ppb As Detection limit PT- 10 ppb As + low level interferents PT- 10 ppb As + high level interferents Average RSD for PT samples with interferents Average RSD for PT samples Environmental Samples Battelle drinking water Olentangy River water Residential well water Alum Creek Res water Average RSD Technical Operator Unit #1 NA 19% 4% 7% 6% 20%(a) 3% 9% 6% 10% ND NA 17% NA 13% Technical Operator Unit #2 NA 11% 13% 9% 7% 20% 6% 7% ND NA 44% NA Non-Technical Operator Unit #1 4% 7% 18% 4% 6% 21% 38% 8% Non-Technical Operator Unit #2 NA 7% 5% 4% 4% 13% 2% 2% 12% 9% NA(b) NA 27% NA(b) NA NA 19% NA 11% (a) One replicate result = non-detect, calculated RSD excluding this result. (b) Includes values reported as ND. NA = One or more replicates below detection limit. ND = Non-detects, reported by SafeGuard as error message because of division by zero. 6.2.3 Linearity The linearity of the SafeGuard measurements was assessed by performing a linear regression of the SafeGuard results against the reference method results for the five PT samples ranging from 1 ppb to 100 ppb of arsenic. Figure 6-la presents the results of the linear regression for the two SafeGuard units when operated by the technical operator and Figure 6-lb for the two units when operated by the non-technical operator. In these regressions, results reported below the detection limit (<1 ppb) by the SafeGuard or identified as ND due to error message were not included. The slope, intercept, and coefficient of determination (r2) for each regression equation are shown on the charts. Table 6-7 summarizes the equations for the linear regressions and presents the 95% confidence interval for the slopes as ± error. All linear regressions compared to the reference method results had coefficients of determination greater than 0.99. The 95% confidence intervals for the slopes indicate that only the technical operator data for Unit #1 were consistent with a slope of 1 and were not significantly different from the reference analysis results. The 95% confidence intervals for the y-axis intercept included zero for both operators on both units indicating no significant difference from the reference analysis results. 27 ------- •fl 1 60 / = 1.0045x-1.6184 Technical Operator Unit #1 Technical Operator Unit #2 - Linear (Technical Operator UnitfH) -Linear (Technical Operator Unit #2) 60 Reference concentration (ppb) Figure 6-la. Linearity of SafeGuard Results for the Technical Operator 60 40 O 1 R = 0.9961 i Non-Technical Operator Unit Non-Technical Operator Unit #2 Linear (Non-Technical Operator Unit #1) Linear (Non-Technical Operator Unit #2) 40 60 i Reference concentration (ppb) Figure 6-lb. Linearity of SafeGuard Results for the Non-Technical Operator Table 6-7. Summary of Linear Regression Equations for SafeGuard and Reference Results Description Safeguard Unit #1, technical operator Safeguard Unit #2, technical operator Safeguard Unit #1, non-technical operator Safeguard Unit #2, non-technical operator Slope (± Error) 1.005 (0.044) 0.808 (0.034) 0.874 (0.027) 0.796(0.019) Intercept (± Error) -1.618(2.32) 0.060(1.70) 0.155(1.27) 0.960 (0.96) Coefficient of Determination 0.9942 0.9936 0.9961 0.9979 28 ------- 6.2.4 Method Detection Limit The MDL was assessed by analyzing seven replicates of a sample spiked at approximately five times the vendor-stated detection limit for the SafeGuard. Table 6-8 lists the replicate results, provides the standard deviations for the replicate results for the SafeGuard results, and shows the calculated MDLs. The calculated MDL values for the technical operator were 3.75 ppb and 2.87 ppb for Units #1 and #2 respectively. The MDL values for the non-technical operator calculated were 3.33 ppb and 2.04 ppb for Units #1 and #2. Table 6-8. Detection Limit Results for SafeGuard Sample Concentration (ppb) 5 5 5 5 5 5 5 Standard Deviation t (n=7) MDL Technical Unit #1 (ppb) ND(a) 4.43 4.91 4.50 5.50 7.35 6.04 1.11 3.37(b) 3.75 Operator Non-Technical Unit #2 (ppb) 3.77 4.67 3.44 5.01 4.48 6.27 4.58 0.91 3.14 2.87 Unit #1 (ppb) 6.06 4.39 3.90 5.71 6.14 4.90 3.50 1.06 3.14 3.33 Operator Unit #2 (ppb) 4.36 4.03 5.08 5.52 5.46 5.78 5.35 0.65 3.14 2.04 included in MDL calculation (b) t(n = 6) 6.2.5 Matrix Interference Effects Matrix interference effects were assessed by comparing the calculated percent bias for the samples containing low-level and high-level concentrations of interfering substances with the bias reported for the other PT samples (Table 6-5). The biases for the samples with low and high levels of interfering compounds ranged from -19% to 7%, which is within the range of the PT samples (-21% to 9%). As such, neither the low nor the high levels of interferences tested appeared to have affected the arsenic levels measured by the SafeGuard. As discussed in Section 6.2.1, residential well water clearly affected the SafeGuard measurement, because the native (unspiked) replicates from both operators and both SafeGuard units reported an arsenic concentration from 2.50 ppb to 9.70 ppb, whereas the reference method reported this sample at 0.89 ppb to 1.12 ppb. Thus a positive unknown interference exists in the residential well water sample. Battelle drinking water, Olentangy River water, and Alum Creek reservoir water did not appear to have matrix interference effects. 6.2.6 Operator Bias Operator bias was evaluated by comparing the SafeGuard results above the detection limit produced by the technical and non-technical operators for all PT and environmental samples. Linear regression results for the two sets of data are shown in Figure 6-2. The slopes of the regressions show little difference between operators with Unit #2 (slope = 0.98), but slightly 29 ------- higher results overall from the technical operator with Unit #1 (slope = 0.87). The 95% confidence intervals were calculated for the Unit #1 and Unit #2 regressions in Figure 6-2 and are shown as ± error in Table 6-9 (Section 6.2.7). The 95% confidence interval includes a slope of 1 for Unit #2, but the 95% confidence interval does not include a slope of 1 for Unit #1, indicating a significant operator bias (technical operator results > non-technical operator results) with that unit. Paired t-tests of the two sets of data indicate that the SafeGuard results were not significantly different at a 0.05 level of significance depending on the operator. The t-test finds a significant difference if the means of the data sets from the two operators differ by more than about 10%. The respective operator means for Unit #1 differed by 8% and those for Unit #2 differed by 2%. Overall, these results indicate at most a small operator bias with one of the two SafeGuard units. Technical Operator (ppb) Figure 6-2. Comparison of SafeGuard Results for Technical and Non-Technical Operators 6.2.7 Inter-Unit Reproducibility Inter-unit reproducibility was evaluated by comparing the data for the two SafeGuard units. All detected results for the PT and environmental samples were included in the analysis. Linear regressions of the data for each unit are shown in Figure 6-3 and show that Unit #2 readings were lower than Unit # 1 readings with both operators, but more strongly with the technical operator. The 95% confidence intervals were calculated for the technical and non-technical operator regressions in Figure 6-3 and are shown as ± error in Table 6-9. Neither 95% confidence interval includes a slope of 1, indicating a significant inter-unit bias that is more pronounced with the technical operator than with the non-technical operator. A paired t-test of the data indicated that the results from the two units with the technical operator were significantly different at a 0.05 level of significance; however, the results from the two units with the non-technical operator were not significantly different. The t-test finds a significant difference if the means of the two SafeGuard units differ by more than about 10%. 30 ------- 100 40 20 y=0.9066x+0.8642 y = 0.8024x+1.2084 R2 = 0.9897 Non-Technical Operator Techical Operator Linear (Non-Technical Operator) Linear (Techical Operator) 20 40 60 Unit # 1 100 Figure 6-3. Comparison of SafeGuard Test Results for Units # 1 and # 2 Table 6-9. Summary of Linear Regression Equations for Assessing Operator Bias and Inter-unit Reproducibility Description Unit #1, operator bias Unit #2, operator bias Technical operator, Inter-unit reproducibility Non-technical operator, Inter-unit reproducibility Slope (± Error) 0.872(0.033) 0.983 (0.033) 0.802 (0.029) 0.907 (0.023) Intercept (± Error) 0.957(1.18) 0.566 (0.98) 1.208(1.03) 0.864 (0.74) Coefficient of Determination 0.9886 0.9909 0.9897 0.9947 The means for Units #1 and #2 differed by 17% when used by the technical operator and by 7% when used by the non-technical operator. Overall, these results show an inter-unit bias with the technical operator, but minimal bias with the non-technical operator. 6.2.8 Rate of False Positives/False Negatives Tables 6-10a and b show the false positives for the technical and non-technical operators respectively, and Tables 6-1 la and b present the false negative data for the two operators. These calculations included all PT, QC, and environmental results. The rates of false positives for the SafeGuard were 0% for both units for the technical operator and 2% and 0% for the non- technical operator (Units #1 and #2, respectively). The rates of false negatives for the SafeGuard units were 4% and 22% for the technical operator (Units #1 and #2, respectively), and 18% for both units for the non-technical operator. The false positive and negative rates of the technical and non-technical operators were averaged to determine sensitivity and specificity. The results indicate that the SafeGuard correctly identified arsenic concentrations below the federal drinking water standard (<10 ppb) 99.5% of the time (0.995 = sensitivity) and identified arsenic 31 ------- concentrations that did not meet the federal standard (>10 ppb) 85.4% of the time (0.854 = specificity). Table 6-10a. Rates of False Positives for the Technical Operator Description PT- 1 ppb As PT- 3 ppb As PT- 10 ppb As Detection limit PT- 10 ppb As + high-level interferents Battelle drinking water Battelle drinking water LFM Olentangy River water Residential well water Unit #1 (ppb) 0.0 0.0 0.0 0.0 2.48 2.87 1.77 2.50 8.52 8.70 9.32 8.65 0.0 4.43 4.91 4.50 5.50 7.35 6.04 9.35 ND ND ND ND 0.0 <1 <1 1.16 <1 6.20 7.27 6.90 4.85 False Positive (Y/N) Unit #1 Qualitative N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N Unit #2 (ppb) 1.08 0.0 1.11 0.0 3.07 2.65 2.41 2.96 8.49 7.81 6.31 8.35 3.77 4.67 3.44 5.01 4.48 6.27 4.58 8.91 ND ND ND ND 9.40 <1 2.92 <1 <1 7.97 8.58 2.50 9.39 False Positive (Y/N) Unit #2 Qualitative N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N Reference (ppb) 1.12 1.09 1.09 1.10 3.02 3.05 3.04 3.01 9.56 9.88 9.85 9.96 5.10 5.00 5.02 5.05 5.20 5.23 5.11 9.93 0.56 0.66 0.63 0.61 9.75 1.38 1.41 1.41 1.33 0.89 1.12 0.95 0.94 32 ------- Table 6-1 Oa. Rates of False Positives for the Technical Operator (continued) Description Alum Creek Reservoir water RB Total # samples <10 ppb by reference method Total # false positive (Y) Percent false positive False Positive Unit #1 (Y/N) Unit #1 (ppb) Qualitative <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N 1.11 N <1 N 46 0 0% False Positive Unit #2 (Y/N) Unit #2 (ppb) Qualitative <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N 46 0 0% Reference (ppb) 1.50 1.48 1.42 1.50 <0. <0. <0. <0. <0. <0. <0. <0. <0. 33 ------- Table 6-10b. Rates of False Positives for the Non-Technical Operator Description PT- 1 ppb As PT- 3 ppb As Detection limit PT- 10 ppb As + low level interferents PT- 10 ppb As + high level interferents Battelle drinking water Battelle drinking water LFM Olentangy River water Residential well water Unit #1 (ppb) 1.14 1.16 1.09 1.05 3.52 3.30 2.94 3.30 6.06 4.39 3.90 5.71 6.14 4.90 3.50 4.90 6.37 8.72 11.8 8.01 7.24 8.67 8.41 ND <1 ND <1 7.04 1.65 <1 <1 <1 9.70 4.75 7.48 7.60 False Positive (Y/N) Unit #1 Qualitative N N N N N N N N N N N N N N N N N N Y N N N N N N N N N N N N N N N N N Unit #2 (ppb) 1.12 0.0 0.0 1.04 2.81 3.00 3.04 3.29 4.36 4.03 5.08 5.52 5.46 5.78 5.35 8.76 8.93 8.77 9.15 7.58 7.34 7.40 7.57 1.08 <1 <1 <1 8.00 <1 <1 <1 <1 6.88 6.62 4.80 7.65 False Positive (Y/N) Unit #2 Qualitative N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N Reference (ppb) 1.06 1.05 1.05 1.02 3.02 2.92 2.99 3.01 5.08 5.02 4.97 5.08 5.24 5.11 5.15 9.71 9.49 9.43 9.35 9.28 9.31 9.19 9.33 0.56 0.66 0.63 0.61 9.75 1.38 1.41 1.41 1.33 0.89 1.12 0.95 0.94 34 ------- Table 6-1 Ob. Rates of False Positives for the Non-Technical Operator (continued) Description Alum Creek Reservoir water RB QCS Total # samples <10 ppb by reference method Total # false positive (Y) Percent false positive False Positive Unit #1 (Y/N) Unit #1 (ppb) Qualitative <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N 8.83 N 8.46 N 50 1 2% False Positive Unit #2 (Y/N) Unit #2 (ppb) Qualitative <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N <1 N 7.76 N 9.34 N 50 0 0% Reference (ppb) 1.50 1.48 1.42 1.50 <0. <0. <0. <0. <0. <0. <0. <0. 9.97 9.87 35 ------- Table 6-11 a. Rates of False Negatives for the Technical Operator Description PT- 30 ppb As PT- 100 ppb As PT- 10 ppb As + low level interferents PT- 10 ppb As + high level interferents Olentangy River water LFM Residential well water LFM Alum Creek Reservoir water LFM QCS Total # samples >10 ppb by reference method Total # false negatives (Y) % False negatives Unit #1 (ppb) 25.0 24.1 27.8 27.8 93.1 105.0 105.0 95.9 10.3 10.6 10.0 10.6 11.3 11.2 11.6 9.63 8.36 9.30 8.38 9.47 7.34 8.04 9.60 7.68 8.90 9.64 9.22 False Negative (Y/N) Unit #1 Qualitative N N N N N N N N N N N N N N N N N N N N Y N N N N N N 27 1 4% Unit #2 (ppb) 25.7 25.1 23.9 21.0 81.8 84.1 85.6 73.7 8.48 8.87 9.58 8.39 8.56 8.07 9.51 7.63 8.27 7.90 8.16 9.11 7.22 6.84 6.86 6.26 8.71 8.35 8.96 False Negative (Y/N) Unit #2 Qualitative N N N N N N N N N N N N N N N Y N Y N N Y Y Y Y N N N 27 6 22% Reference (ppb) 29.7 29.4 29.9 29.9 99.8 100.8 100.2 100.8 10.2 10.3 10.2 10.1 10.3 10.1 10.1 10.6 10.7 10.8 10.1 10.2 10.2 10.3 10.2 10.2 10.0 10.0 10.2 36 ------- Table 6-1 Ib. Rates of False Negatives for the Non-Technical Operator Description PT- 10 ppb As PT- 30 ppb As PT- 100 ppb As Olentangy River water LFM Residential well water LFM Alum Creek Reservoir water LFM QCS Total # samples >10 ppb by reference method Total # false negatives (Y) % False negatives Unit #1 (ppb) 6.70 9.93 8.58 9.98 26.8 25.9 24.4 26.0 83.2 95.2 85.9 87.7 7.84 11.9 8.96 9.17 8.10 10.2 8.36 0.00 5.09 9.84 False Negative (Y/N) Unit #1 Qualitative Y N N N N N N N N N N N Y N N N N N N Y Y N 22 4 18% Unit #2 (ppb) 8.79 9.73 9.64 9.79 26.2 25.2 24.0 24.7 75.8 81.4 82.9 82.6 7.78 10.9 7.83 9.94 9.59 6.56 7.41 8.56 8.06 7.96 False Negative (Y/N) Unit #2 Qualitative N N N N N N N N N N N N Y N Y N N Y Y N N N 22 4 18% Reference (ppb) 10.2 10.3 10.2 10.3 29.7 29.4 29.9 29.9 99.8 100.8 100.2 100.8 10.6 10.7 10.8 10.3 10.3 10.3 10.4 10.1 10.2 10.2 6.3 Other Factors During testing activities, the technical and non-technical operators were instructed to keep a record of their comments on ease of use, reliability, portability, and generation of waste materials. This section summarizes these observations and other comments pertaining to any problems encountered during testing. Cost information is also presented. 6.3.1 Ease of Use The technical and non-technical operators both reported that the SafeGuard was very easy to use. The manual and the software program were clear and easy to follow. No solution or sample preparation is necessary. The sample bottle is screwed into the instrument, and the software is started. Thirty to 50 minutes later, the arsenic concentration of the sample is presented on the 37 ------- screen. This reading is automatically recorded in a spreadsheet that will open with Microsoft™ Excel. The software program is designed for both technical and non-technical operators. The basic measurement mode of operating is described above, and an administrator level of operation produces analysis diagrams and shows more detailed information about performance. Control over the communication and configuration of the instrument is also available when in administrator mode. The ease of use of the administrator mode was not evaluated by the operators in this test. 6.3.2 Analysis Time When started, the SafeGuard goes through an automatic system initialization, which takes about 10 minutes. Subsequently, there is a regular calibration sequence that occurs every 4 hours or every four readings. This operation is also automated and takes about 10 minutes as well. The analysis time per sample at room temperature is 30 to 50 minutes. The software program displays a timer that counts down the time remaining. 6.3.3 Reliability Overall, the SafeGuard operated reliably throughout the test. Pop-up messages occurred for three main reasons on both units. The computer (comm ports) lost communication with the instrument periodically. This error was easily remedied by closing the software and rebooting the SafeGuard, or reassigning the comm ports. Another error message displayed was the "Overflow script text...," which was considered a non-detect. This occurred when the software used zero in the denominator to calculate the arsenic concentration. When there was an issue with the SafeGuard, a message appeared on the screen describing the problem and how to fix it. The message was not because of an error, but prompted the user to perform maintenance. For example, when the sensitivity was low, the SafeGuard notified the operator and displayed instructions to clean the sensor and run calibration. When troubleshooting was necessary (for the communication and overscript issues), technical support was provided by the vendor over the phone. The issues were clearly explained and quickly remedied. 6.3.4 Waste Material The SafeGuard used standard addition to make multiple readings in calculating the actual concentration of the samples. Because of this, the SafeGuard generated a considerable amount of liquid waste. The SafeGuard used approximately 15 mL of sample and generated about 50 mL of waste per sample. It used a 500-ppb standard for calibration and standard additions. An estimated maximum concentration of arsenic in the waste is about 5 ppb above the native sample concen- tration. This includes all line priming, two standard additions, and line flushing for each sample. Since arsenic is added to every sample by the SafeGuard, the waste water must be disposed of as arsenic-containing waste. 6.3.5 Cost and Consumables The listed price for the SafeGuard at the time of the verification test was $35,000. The reagent kits for arsenic analysis, which can analyze 45 samples, were available for $80. These kits have a 6-month shelf life as received and are stored at room temperature. Also, reagents from the kits may be purchased separately, if an entire kit is not consumed. Sample bottles for the SafeGuard 38 ------- are standard narrow-mouth, threaded polytetrafluoroethylene bottles that can be purchased from most laboratory supply vendors. 39 ------- Chapter 7 Performance Summary The SafeGuard was verified by evaluating the following parameters: • Accuracy • Precision • Linearity • MDL • Matrix interference effects • Operator bias • Inter-unit reproducibility • Rate of false positives/false negatives. Accuracy was assessed by comparing the results to Method 200.8(2) results from ICP-MS analysis. The quantitative assessment of accuracy indicated that the relative bias for the SafeGuard ranged from -28% to 7% for the technical operator and -28% to 11% for the non- technical operator (excluding residential well water samples at approximately 600% due to matrix effect). Precision was assessed by analyzing four replicates of each sample. For the technical operator, precision expressed as RSD ranged from 3% to 44%; and, for the non-technical operator, it ranged from 2% to 38%. The average RSD for PT samples only was 10% for the technical operator and 9% for the non-technical operator. These results exclude samples for which one or more of the replicate results was not detected by the SafeGuard. The linearity of response was evaluated by plotting the SafeGuard results against the reference analysis results for the PT samples. All linear regressions against the reference method results had coefficients of determination (r2) greater than 0.99. The 95% confidence intervals for the slopes indicate that only the technical operator data for Unit #1 were consistent with a slope of 1 and were not significantly different from the reference analysis results. The 95% confidence intervals for the y-axis intercept included zero for both operators on both units indicating no significant difference from the reference analysis results. The MDL was assessed by analyzing seven replicates of a sample spiked at a level approximately five times the manufacturer's estimated detection limit for the SafeGuard (i.e., 1 ppb x 5 = 5 ppb). The MDLs calculated using the precision data from these replicates ranged from 2.0 ppb to 3.8 ppb. 40 ------- Results for samples containing low and high levels of interfering compounds indicated that neither level of interference appeared to affect the detection of arsenic, with bias ranging from -19% to 7%, consistent with the bias observed in the absence of interferences. The SafeGuard performance was affected by one of the environmental samples, the residential well water. The native (unspiked) replicates of this sample from both operators and both SafeGuard units reported an arsenic concentration from 2.50 ppb to 9.70 ppb, whereas the reference method reported this sample at 0.89 ppb to 1.12 ppb. Operator bias was evaluated by comparing the SafeGuard results above the detection limit produced by the technical and non-technical operators for all PT and environmental samples. The 95% confidence interval includes a slope of 1 for Unit #2, but the 95% confidence interval does not include a slope of 1 for Unit #1, indicating a significant operator bias (technical results > non-technical results) with that unit. Paired t-tests of the two sets of data indicate that the SafeGuard results were not significantly different at a 0.05 level of significance depending on the operator. Overall, these results indicate, at most, a small operator bias with one of the two SafeGuard units. Inter-unit reproducibility was evaluated by comparing the data for the two SafeGuard units. Linear regressions of the data for each unit show that Unit #2 readings were lower than Unit #1 readings with both operators, but more strongly with the technical operator. Neither 95% confidence interval includes a slope of 1, indicating a significant inter-unit bias that is more pronounced with the technical operator than with the non-technical operator. A paired t-test of the data indicated that the results from the two units with the technical operator were significantly different at a 0.05 level of significance; however, the results from the two units with the non-technical operator were not significantly different. Overall, these results show an inter- unit bias with the technical operator, but minimal bias with the non-technical operator. The rates of false positives for the SafeGuard were 0% for both units for the technical operator and 2% and 0% for the non-technical operator (Units #1 and #2, respectively). The rates of false negatives for the SafeGuard units were 4% and 22% for the technical operator and 18% for both units for the non-technical operator. By averaging these rates, the results indicate that the SafeGuard correctly identified water below the federal drinking water standard (<10 ppb) 99.5% of the time (0.995 = sensitivity) and identified water that did not meet the federal standard (>10 ppb) 85.4% of the time (0.854 = specificity). The SafeGuard was easy to use, and the manual and software program were clear and easy to follow. All reagent mixing and instrument flushing are automated. No solution or sample preparation was necessary. Because the SafeGuard uses standard addition (two additions per sample) to make multiple readings to calculate arsenic concentrations, roughly 50 mL of arsenic- containing waste is generated per sample analyzed, which requires special disposal. The analysis time per sample at room temperature was 30 to 50 minutes. The listed price for SafeGuard at the time of the verification test was $35,000, and the cost for a 45-sample reagent kit was $80. Replacement reagents and supplies are available without purchasing entire kits. 41 ------- Chapter 8 References 1. Test/QA Plan for Verification of Portable Analyzers, Battelle, Columbus, Ohio, Version 1. December 8, 2000 (amended). 2. U.S. EPA Method 200.8, Determination of Trace Elements in Waters and Wastes by Inductively Coupled Plasma Mass Spectrometry, Revision 5.5, October, 1999. 3. Federal Register, Vol. 66 No. 14, January 22, 2001. Part VIII, Environmental Protection Agency. 40 CFR Parts 9, 141, and 142: National Primary Drinking Water Regulations; Arsenic and Clarifications to Compliance and New Source Contaminants Monitoring: Final Rule. http://www.epa.gov/safewater/ars/arsenic_finalrule.pdf. 4. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Center, Version 6.0, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus, Ohio, November 2005. 42 ------- |