July 2002 Environmental Technology Verification Report Quick™ Arsenic Test Kit Prepared by £EPA Barteiie . . . Putting Technology To Work Battelle Under a cooperative agreement with U.S. Environmental Protection Agency etV ety etV ------- July 2002 Environmental Technology Verification Report ETV Advanced Monitoring Systems Center Quick™ Arsenic Test Kit by Adam Abbgy Thomas Kelly Charles Lawrie Karen Riggs Battelle Columbus, Ohio 43201 ------- Notice The U.S. Environmental Protection Agency (EPA), through its Office of Research and Development, has financially supported and collaborated in the extramural program described here. This document has been peer reviewed by the Agency and recommended for public release. Mention of trade names or commercial products does not constitute endorsement or recommendation by the EPA for use. 11 ------- Foreword The U.S. Environmental Protection Agency (EPA) is charged by Congress with protecting the nation's air, water, and land resources. Under a mandate of national environmental laws, the Agency strives to formulate and implement actions leading to a compatible balance between human activities and the ability of natural systems to support and nurture life. To meet this mandate, the EPA's Office of Research and Development provides data and science support that can be used to solve environmental problems and to build the scientific knowledge base needed to manage our ecological resources wisely, to understand how pollutants affect our health, and to prevent or reduce environmental risks. The Environmental Technology Verification (ETV) Program has been established by the EPA to verify the performance characteristics of innovative environmental technology across all media and to report this objective information to permitters, buyers, and users of the technology, thus substantially accelerating the entrance of new environmental technologies into the marketplace. Verification organizations oversee and report verification activities based on testing and quality assurance protocols developed with input from major stakeholders and customer groups associated with the technology area. ETV consists of six environmental technology centers. Information about each of these centers can be found on the Internet at http://www.epa.gov/etv/. Effective verifications of monitoring technologies are needed to assess environmental quality and to supply cost and performance data to select the most appropriate technology for that assess- ment. In 1997, through a competitive cooperative agreement, Battelle was awarded EPA funding and support to plan, coordinate, and conduct such verification tests for "Advanced Monitoring Systems for Air, Water, and Soil" and report the results to the community at large. Information concerning this specific environmental technology area can be found on the Internet at http://www.epa.gov/etv/centers/centerl.html. 111 ------- Acknowledgments The authors wish to acknowledge the support of all those who helped plan and conduct the verification test, analyze the data, and prepare this report. In particular we would like to thank A. J. Savage, Raj Mangaraj, Daniel Turner, and Bea Weaver of Battelle. We also acknowledge the assistance of AMS Center stakeholders Vito Minei, Dennis Goldman, Geoff Dates, and Marty Link, who reviewed the test/QA plan and verification reports. IV ------- Contents Notice ii Foreword iii Acknowledgments iv List of Abbreviations viii 1. Background 1 2. Technology Description 2 3. Test Design and Procedures 3 3.1 Introduction 3 3.2 Test Design 3 3.3 Test Samples 4 3.3.1 QC Samples 4 3.3.2 PT Samples 6 3.3.3 Environmental Samples 6 3.4 Reference Analysis 7 3.5 Verification Schedule 7 4. Quality Assurance/Quality Control 9 4.1 QC for Reference Method 9 4.2 Audits 11 4.2.1 Performance Evaluation Audit 11 4.2.2 Technical Systems Audit 11 4.2.3 Audit of Data Quality 12 4.3 QA/QC Reporting 12 4.4 Data Review 12 5. Statistical Methods 14 5.1 Accuracy 14 5.2 Precision 15 5.3 Linearity 15 5.4 Method Detection Limit 15 5.5 Matrix Interference Effects 16 v ------- 5.6 Operator Bias 16 5.7 Rate of False Positives/False Negatives 16 6. Test Results 17 6.1 Accuracy 17 6.2 Precision 24 6.3 Linearity 24 6.4 Method Detection Limit 27 6.5 Matrix Interference Effects 27 6.6 Operator Bias 28 6.7 Rate of False Positives/False Negatives 28 6.8 Other Factors 31 6.8.1 Costs 31 6.8.2 Data Completeness 31 7. Performance Summary 32 8. References 34 Figures Figure 2-1. Industrial Test Systems, Inc., Quick™ Arsenic Test Kit 2 Figure 6-1. Comparison of Quick™ Test Kit to Reference Method Results from PT Samples 26 Tables Table 3-1. Test Samples for Verification of the Quick™ Test Kit 5 Table 3-2. Schedule of Verification Test Days 8 Table 4-1. Reference Method QCS Analysis Results 10 Table 4-2. Reference Method LFMl Analysis Results 10 Table 4-3. Reference Method Duplicate Analysis Results 11 Table 4-4. Reference Method PE Audit Results 11 Table 4-5. Summary of Data Recording Process 13 Table 6-la. Results from Laboratory Performance Test Sample Analyses 18 vi ------- Table 6-lb. Results from Drinking Water Analyses 19 Table 6-lc. Results from Freshwater Analyses 20 Table 6-2a. Accuracy of the Quick™ Test Kit with Laboratory Performance Test Samples 21 Table 6-2b. Accuracy of the Quick™ Test Kit with Drinking Water Samples 22 Table 6-2c. Accuracy of the Quick™ Test Kit with Freshwater Samples 23 Table 6-3. Summary of Qualitative Accuracy Results 23 Table 6-4a. Precision Results for Quick™ Test Kit from Laboratory Performance Test Samples 25 Table 6-4b. Precision Results for Quick™ Test Kit from Drinking Water Samples 26 Table 6-5a. Results from Laboratory Performance Test Samples with Low-Level Interferences 27 Table 6-5b. Results from Laboratory Performance Test Samples with High-Level Interferences 28 Table 6-6. Rate of False Positives from Quick™ Test Kit 29 Table 6-7. Rate of False Negatives from Quick™ Test Kit 30 vii ------- List of Abbreviations AMS Advanced Monitoring Systems ASTM American Society for Testing and Materials DW drinking fountain water EPA U.S. Environmental Protection Agency ETV Environmental Technology Verification FW freshwater HDPE high-density polyethylene HI high interference ICPMS inductively coupled plasma mass spectrometry LBC Little Beaver Creek LC Lytle Creek LFM laboratory-fortified matrix LI low interference MDL method detection limit NIST National Institute of Standards and Technology ppb parts per billion ppm parts per million PE performance evaluation PT performance test QA quality assurance QA/QC quality assurance/quality control QC quality control QCS quality control standard QMP Quality Management Plan RB reagent blank RPD relative percent difference RSD relative standard deviation SR Stillwater River TSA technical systems audit TW treated well water WW well water Vlll ------- Chapter 1 Background The U.S. Environmental Protection Agency (EPA) has created the Environmental Technology Verification (ETV) Program to facilitate the deployment of innovative environmental tech- nologies through performance verification and dissemination of information. The goal of the ETV Program is to further environmental protection by substantially accelerating the acceptance and use of improved and cost-effective technologies. ETV seeks to achieve this goal by providing high-quality, peer-reviewed data on technology performance to those involved in the design, distribution, financing, permitting, purchase, and use of environmental technologies. ETV works in partnership with recognized testing organizations; with stakeholder groups consisting of buyers, vendor organizations, and permitters; and with the full participation of individual technology developers. The program evaluates the performance of innovative tech- nologies by developing test plans that are responsive to the needs of stakeholders, conducting field or laboratory tests (as appropriate), collecting and analyzing data, and preparing peer- reviewed reports. All evaluations are conducted in accordance with rigorous quality assurance (QA) protocols to ensure that data of known and adequate quality are generated and that the results are defensible. The EPA's National Exposure Research Laboratory and its verification organization partner, Battelle, operate the Advanced Monitoring Systems (AMS) Center under ETV. The AMS Center recently evaluated the performance of four portable analyzers for arsenic in water. This verification report presents the procedures and results of the verification test for Industrial Test Systems, Inc., Quick™ test kit arsenic analysis systems. The Quick™ test kit is an inexpensive, portable, rapid device designed for on-site analysis of arsenic in water. 1 ------- Chapter 2 Technology Description The objective of the ETV AMS Center is to verify the performance characteristics of environmental monitoring technologies for air, water, and soil. This verification report provides results for the verification testing of the Quick™ test kit for arsenic in water. Following is a description of the test kit, based on information provided by the vendor. The information The Quick™ test kit can be used to test for total arsenic in water. Up to 2.0 mg/L of hydrogen sulfide is tolerated without test result interference, and up to 5 parts per million (ppm) of antimony is tolerated. The Quick™ test kit consists primarily of two reaction bottles, two caps for holding the test strip, three spoons, three bottles of reagent, and one bottle of arsenic test strips in a waterproof, plastic case. The three reagents are added sequentially to the water sample and shaken. A test strip is placed into the turret of the cap. The test strip is exposed to arsine gas evolved from the sample solution, resulting in a color change in the test strip. When the reaction is complete, the test strip is compared with a color chart provided with the kit. The intensity of the yellow/brown color developed on the test strip relative to the color chart is proportional to the arsenic concentration in the sample and, therefore, provides a semi-quantitative analysis of the arsenic concentration. The color chart consists of the gradations: 0, 5, 10, 20, 40, 60, 100, 200, 300, and 500 parts per billion (ppb). The kits are available in three sizes: for two tests, 50 tests, or 100 tests. provided below was not verified in this test. Figure 2-1. Industrial Test Systems, Inc., Quick™ Arsenic Test Kit 2 ------- Chapter 3 Test Design and Procedures 3.1 Introduction This verification test was conducted according to procedures specified in the Test/QA Plan for Verification of Portable Analyzers,(1) The verification was based on comparing the arsenic results from the Quick™ test kit to those from a laboratory-based reference method. The reference method for arsenic analysis was inductively coupled plasma mass spectrometry (ICPMS), performed according to EPA Method 200.8(2) The Quick™ test kit does not require calibration, but relies on comparisons to a color chart provided with the test kit to allow semi-quantitative measurements of arsenic concentrations. The test kit was verified by analyzing laboratory- prepared performance test samples, treated and untreated drinking water, and fresh surface water, with both the test kit and the reference method. 3.2 Test Design The Quick™ test kit was verified in terms of its performance on the following parameters: # Accuracy # Precision # Linearity # Method detection limit (MDL) # Matrix interference effects # Operator bias # Rate of false positives/false negatives. All preparation and analyses were performed according to the manufacturer's recommended pro- cedures. Results from the Quick™ test kit were recorded manually. The results from the Quick™ test kits were compared to those from the reference method to assess accuracy, linearity, and detection limit. Multiple aliquots of performance test samples and drinking water samples were analyzed to assess precision. Identical sets of samples were analyzed independently by two separate operators (a technical and a non-technical Battelle staff member). The technical operator was a research technician at Battelle with three years of laboratory experience and a B.S degree. The non-technical operator 3 ------- was a part-time temporary helper at Battelle with a general education development certificate. Because the reagents of the Quick™ test kits are consumed in use, it was not feasible for the two operators to switch kits as a means of quantitatively assessing operator bias. However, each operator used multiple kits in order to analyze all the samples, so it is assumed that kit-to-kit variability was similar for both operators. Consequently, qualitative observations could be made on operator bias. Matrix interference effects were assessed by challenging the test kit with performance test samples of known arsenic concentrations containing both low-level and high-level interferences. False positives and negatives were evaluated relative to the recently established 10-ppb maximum contaminant level for arsenic in drinking water. In addition to the analytical results, the time required for sample analysis and operator observations concerning the use of the test kit (e.g., frequency of calibration, ease of use, maintenance) were recorded. In a few instances, the test kit operator interpolated between the test kit gradations in reporting an arsenic value. This is not unusual in use of such kits, and typically resulted in an arsenic reading midway between two gradation values (e.g., 30 ppb, between gradations of 20 and 40 ppb). 3.3 Test Samples Three types of samples were used in the verification test, as shown in Table 3-1: quality control (QC) samples, performance test (PT) samples, and environmental water samples. The QC and PT samples were prepared from National Institute of Standards and Technology (NIST) traceable purchased standards. Under the Safe Drinking Water Act, the EPA lowered the maximum contaminant level for arsenic from 50 ppb to 10 ppb, effective in January 2006. Therefore, the QC sample concentrations for arsenic were targeted at that 10-ppb level. The PT samples were targeted to range from 10% to 1,000% of that level, i.e., from 1 to 100 ppb. The environmental water samples were collected from various drinking water and surface water sources. All samples were analyzed using the Quick™ test kits and a reference method. Every tenth sample was analyzed twice by the reference method to document the reference method's precision. 3.3.1 QC Samples As Table 3-1 indicates, prepared QC samples included laboratory reagent blanks (RB), laboratory-fortified matrix (LFM) samples, and quality control samples. The RB samples consisted of water collected from the same tap and were exposed to handling and analysis procedures identical to the other prepared samples. These samples were used to help ensure that no sources of contamination were introduced during sample handling and analysis. Two types of LFMs were prepared. The LFMf samples consisted of aliquots of environmental samples that were spiked in the field to increase the analyte concentration by 10 ppb of arsenic. These samples were analyzed by the test kits in the field both before and after spiking. The spike solution used 4 ------- for the LFMf samples was prepared in the laboratory and brought to the field site. The LFMl samples were aliquots of environmental samples that were spiked in the laboratory to increase the analyte concentration by 25 ppb of arsenic. These samples were used to help identify whether matrix effects influenced the reference method results. At least 10% of all the prepared samples analyzed were RBs, and at least one sample taken from each sampling site was an LFMf. Table 3-1. Test Samples3 for Verification of the Quick™ Test Kit Type of Sample Sample Characteristics Concentration No. of Samples Reagent Blank (RB)b ~0 10% of all Laboratory Fortified Mixture (LFMp)b 10 ppb above native level 1 per site Quality Control LFMl13 25 ppb above native level 6 Quality Control Sample (QCS)b 10 ppb 10% of all Prepared arsenic solution (PT6) 25 ppb 7 Prepared arsenic solution (PT1) 1 ppb 4 Prepared arsenic solution (PT2) 3 ppb 4 Prepared arsenic solution (PT3) 10 ppb 4 Performance Test Prepared arsenic solution (PT4) 30 ppb 4 Prepared arsenic solution (PT5) 100 ppb 4 Prepared arsenic solution 10 ppb with low Q spiked with interference (LI) interference O Prepared arsenic solution 10 ppb with high Q spiked with interference (HI) interference o Columbus municipal drinking water (DW) Unknown 4 Well water (WW) Unknown 4 Environmental Treated well water (TW) Unknown 4 Stillwater River (SR) Unknown 4 Lytle Creek (LC) Unknown 4 Little Beaver Creek (LBC) Unknown 4 " Listing is for clarity; samples were analyzed in random order for the verification testing. b See Section 3.3.1 for descriptions of these samples. 5 ------- Quality control standards (QCS) were used as calibration checks to verify that the Quick™ test kit reference instrument was properly calibrated and reading within defined control limits. These standards were purchased from a commercial supplier and were subject only to dilution as appropriate. Calibration of the test kit and the reference instrument was verified using a QCS before and after the testing period, as well as after every tenth sample. An additional independent QCS was used in a performance evaluation (PE) audit of the reference method. 3.3.2 PT Samples The two types of PT samples used in this verification test (Table 3-1) were prepared in the laboratory using tap water as the water source. One type of PT solution contained arsenic at various concentrations and was prepared specifically to determine Quick™ test kit accuracy, linearity, and detection limit. To determine the detection limit of the Quick™, a solution with a concentration five times the vendor's estimated detection limit was used. Seven non-consecutive replicate analyses of this 25-ppb arsenic solution were made to obtain precision data with which to estimate the MDL. Five other solutions were prepared to assess the linearity over a 1- to 100-ppb range of arsenic concentrations. Four aliquots of each of these solutions were pre- pared and analyzed separately to assess the precision of the test kit, as well as the linearity. The second type of PT sample was used to assess the effects of matrix interferences on the per- formance of the Quick™ test kit. These samples were solutions with known concentrations of arsenic spiked with potentially interfering species likely to be found in typical water samples. One sample (designated LI) contained low levels of interferences that consisted of 1 ppm of iron, 3 ppm of sodium chloride, and 0.1 ppm of sulfide per liter at a pH of 6. The second sample (designated HI) contained high levels of interferences that consisted of 10 ppm of iron, 30 ppm of sodium chloride, and 1.0 ppm of sulfide per liter at a pH of 3. Eight replicate samples of each of these solutions were analyzed. 3.3.3 Environmental Samples Drinking water samples listed in Table 3-1 include Columbus municipal water collected from a Battelle drinking fountain (DW), well water (WW), and treated well water (TW) from a school near Columbus, Ohio. The WW was pumped from a 250-foot well and collected directly from an existing spigot with no purging. The TW was treated by running the WW through a Greensand filtration system in the basement of the school. These samples were collected directly from the tap into 2-L high-density polyethylene (HDPE) containers. Four aliquots of each sample were analyzed in the field at the time of collection by each set of the test kits being verified. One aliquot of each sample was preserved with nitric acid and returned to Battelle for reference analysis. The remaining collected sample was stored at 4°C for later use, if necessary. Freshwater (FW) samples from the Stillwater River (SR), Lytle Creek (LC), and the Little Beaver Creek (LBC) (in Ohio) were collected in 2-L HDPE containers. The samples were collected near the shoreline by submerging the containers no more than one inch below the surface of the water. Each body of water was sampled at four distinct locations. An aliquot of each sample was 6 ------- analyzed in the field at the time of collection by each set of each test kit being verified. One aliquot of each sample was preserved with nitric acid and returned to Battelle for reference analysis. The remaining collected sample was preserved and stored at 4°C for later use, if necessary. 3.4 Reference Analysis The reference arsenic analysis was performed using a Perkin Elmer Sciex Elan 6000 ICPMS according to EPA Method 200.8, Revision 5.5.(2) The sample was introduced through a peristaltic pump by pneumatic nebulization into a radiofrequency plasma where energy transfer processes cause desolvation, atomization, and ionization. The ions were extracted from the plasma through a pumped vacuum interface and separated on the basis of their mass-to-charge ratio by a quadrupole mass spectrometer. The ions transmitted through the quadrupole were registered by a continuous dynode electron multiplier, and the ion information was processed by a data handling system. The ICPMS was tuned, optimized, and calibrated daily. The calibration was performed using a minimum of five calibration standards at concentrations ranging between 0.1 and 250 ppb, and a required correlation coefficient of a minimum of 0.999. Internal standards were used to correct for instrument drift and physical interferences. These standards were introduced in line through the peristaltic pump and analyzed with all blanks, standards, and samples. 3.5 Verification Schedule The verification test took place over a 19-day period from October 25 to November 12, 2001. The environmental samples were collected and analyzed over the seven-day period from November 2 through November 8, 2001. Table 3-2 shows the daily testing activities that were conducted during these periods. In all field locations, the samples were analyzed shortly after collection using the Quick™ test kit by both the technical and the non-technical Battelle staff member. The reference analyses on all samples were performed on December 21, 2001, approximately six weeks after sample collection. 7 ------- Table 3-2. Schedule of Verification Test Days Test Day Testing Location Activity 10/25-11/12/01 Battelle Preparation and analysis of PT and associated QC samples. 10/25/01 Battelle Collection and analysis of DW and associated QC samples within Battelle. 11/02/01 Ohio Field Location Collection and analysis of WW samples, TW samples, and associated QC samples at Licking Valley Middle School. 11/06/01 Ohio Field Location Collection and analysis of environmental and associated QC samples at four locations on Little Beaver Creek. 11/07/01 Ohio Field Location Collection and analysis of environmental and associated QC samples at four locations on Lytle Creek. 11/08/01 Ohio Field Location Collection and analysis of environmental and associated QC samples at four locations on the Stillwater River. 8 ------- Chapter 4 Quality Assurance/Quality Control Quality assurance/quality control (QA/QC) procedures were performed in accordance with the quality management plan (QMP) for the AMS Center(3) and the test/QA plan for this verification test.(1) 4.1 QC for Reference Method Field and laboratory RB samples were analyzed to ensure that no sources of contamination were present. The test/QA plan stated that if the analysis of an RB sample indicated a concentration above the MDL for the reference instrument, any contamination source was to be corrected and proper blank readings achieved before proceeding with the verification test. A total of three field RB and one laboratory RB were analyzed. All of the blanks analyzed were below the 0.1-ppb reference MDL for arsenic. The instrument used for the reference method was initially calibrated using 11 calibration standards, with concentrations ranging between 0.1 and 250 ppb of arsenic. The accuracy of the calibration also was verified after the analysis of every 10 samples by analyzing a 25-ppb QCS. If the QCS analysis differed by more than ±10% from the true value of the standard, the instrument was recalibrated before continuing the test. As shown in Table 4-1, the QCS analyses were always within this required range. The maximum bias from the standard in any QCs analysis was 6.04%. LFMl samples were analyzed to assess whether matrix effects influenced the results of the reference method. The percent recovery (R) of these LFMl samples was calculated from the following equation: C - C R = x 100 (1) S where Cs is the analyzed concentration of the spiked sample, C is the analyzed concentration of the unspiked sample, and 5 is the concentration equivalent of the analyte spike. If the percent recovery of an LFMl fell outside the range from 85 to 115%, a matrix effect was suspected. As shown in Table 4-2, all of the LFMl sample results were well within this range, so no matrix effect on the reference analyses is inferred. 9 ------- Table 4-1. Reference Method QCS Analysis Results Measured Actual Sample ID Date of Analysis Arsenic (ppb) Arsenic (ppb) Percent Bias QCS 12/21/2001 24.1 25.0 3.56% QCS 12/21/2001 23.5 25.0 6.04% QCS 12/21/2001 23.8 25.0 4.64% QCS 12/21/2001 23.9 25.0 4.32% QCS 12/21/2001 244 25_0 2.52% Table 4-2. Reference Method LFMl Analysis Results Unspiked Sample Spiked Sample Spiked Amount LFMl Date of Arsenic Arsenic Arsenic Percent Sample ID Analysis (ppb) (ppb) (ppb) Recovery Laboratory RB 12/21/01 <0.1 23.8 25.0 95.3% Field QCS 12/21/01 10.9 35.7 25.0 99.0% DW LFMf 12/21/01 10.6a 34.6 25.0 96.2% LBC 3 Duplicate 12/21/01 2.26 26.6 25.0 97.5% LC 4 12/21/01 1.37 26.3 25.0 99.7% SR4 12/21/01 1.88 26.4 25.0 98.0% 1 Amount of arsenic in the sample after it was spiked in the field. Duplicate samples were analyzed to assess the precision of the reference analysis. The relative percent difference (RPD) of the duplicate sample analysis was calculated from the following equation: RPD = (C-CD) (C + CD)/ 2 : 100 (2) Where C is the concentration of the sample analysis, and CD is the concentration of the sample duplicate analysis. If the RPD was greater than 10%, the instrument was recalibrated before continuing the test. As shown in Table 4-3, the RPDs for the duplicate analysis were all less than 10%. The maximum RPD in any duplicate analysis was 4%. 10 ------- Table 4-3. Reference Method Duplicate Analysis Results Sample ID Date of Analysis Sample Arsenic (ppb) Duplicate Sample Arsenic (PPb) RPD PT QCS 12/21/2001 9.80 9.81 0% PT1 (tap) 12/21/2001 1.76 1.76 0% WW 1 12/21/2001 86.6 86.1 1% LBC4 12/21/2001 2.54 2.44 4% SR QCS 12/21/2001 9.33 9.37 0% 4.2 Audits 4.2.1 Performance Evaluation Audit A PE audit was conducted to assess the quality of the reference measurements made in this verification test. For the PE audit, an independent, NIST-traceable, reference material was obtained from a different commercial supplier than the calibration standards and the field QCS. The PE standard was prepared from Claritas PPT™ Grade Standard purchased through SPEX CertiPrep. Accuracy of the reference method was determined by comparing the measured arsenic concentration using the verification test standards to those obtained using the independently certified PE standard. Percent difference was used to quantify the accuracy of the results. Agreement of the standard within 10% was required for the measurements to be considered acceptable. Failure to achieve this agreement would have triggered recalibration of the reference instrument with the original QC standards and a repeat of the PE comparison. As shown in Table 4-4, the PE sample analysis was well within this required range. Table 4-4. Reference Method PE Audit Results Measured Actual Concentration Date of Arsenic Arsenic Percent Sample ID Analysis (ppb) (ppb) Agreement PE-1 12/21/01 23.7 25.0 5.2% 4.2.2 Technical Systems Audit The Battelle Quality Manager conducted a technical systems audit (TSA) between October 22 and December 21, 2001, to ensure that the verification test was being performed in accordance with the test/QA plan(1) and the AMS Center QMP.(3) The standard solution preparation and PT sample preparation were observed on October 22, the environmental testing (drinking water) on 11 ------- October 25, the testing with PT samples on October 26, and the reference method performance on December 21. As part of the audit, the reference standards and method used were reviewed, actual test procedures were compared to those specified in the test/QA plan, and data acquisition and handling procedures were reviewed. Observations and findings from this audit were docu- mented and submitted to the Verification Test Coordinator for response. No findings were documented that required any corrective action. The records concerning the TSA are permanently stored with the Battelle Quality Manager. 4.2.3 Audit of Data Quality At least 10% of the data acquired during the verification test was audited. Battelle's Quality Manager traced the data from the initial acquisition, through reduction and statistical analysis, to final reporting, to ensure the integrity of the reported results. All calculations performed on the data undergoing the audit were checked. 4.3 QA/QC Reporting Each assessment and audit was documented in accordance with Sections 3.3.4 and 3.3.5 of the QMP for the ETV AMS Center.(3) Once the assessment report was prepared, the Verification Test Coordinator ensured that a response was provided for each adverse finding or potential problem and implemented any necessary follow-up corrective action. The Battelle Quality Manager ensured that follow-up corrective action was taken. The results of the TSA and the audit of data quality were sent to the EPA. 4.4 Data Review Records generated in the verification test received a one-over-one review within two weeks of generation before these records were used to calculate, evaluate, or report verification results. Table 4-5 summarizes the types of data recorded. The review was performed by a Battelle technical staff member involved in the verification test, but not the staff member that originally generated the record. The person performing the review added his/her initials and the date to a hard copy of the record being reviewed. 12 ------- Table 4-5. Summary of Data Recording Process Data to be Responsible Where How Often Disposition of Recorded Party Recorded Recorded Data3 Dates, times of test events Battelle Laboratory record books or ETV field data sheets Start/end of test event Used to organize/check test results; manually incorporated in data spreadsheets as necessary Test parameters (temperature, analyte/ interferant identities, and Quick™ test kit results) Battelle Laboratory record books or ETV field data sheets When set or changed, or as needed to document test Used to organize/check test results, manually incorporated in data spreadsheets as necessary Reference method sample analysis, chain of custody, and results Battelle Laboratory record books, data sheets, or data acquisition system, as appropriate Throughout sample handling and analysis process Transferred to spreadsheets 1 All activities subsequent to data recording are carried out by Battelle. 13 ------- Chapter 5 Statistical Methods The statistical methods presented in this chapter were planned for verifying the performance factors listed in Section 3.2. In a few cases, qualitative comparisons are reported. 5.1 Accuracy When possible, accuracy was assessed relative to the results obtained from the reference analyses. Samples were analyzed by both the reference method and the test kit being verified. For each sample, accuracy was expressed in terms of a relative bias (B) as calculated from the following equation: x 100 (3) where d is the difference between the reading from the Quick™ test kit and that from the reference method, and CR is the reference measurement. Because of the semi-quantitative nature of the visual test kit results, it was not possible to make this determination for many of the results. For this reason, all of the data also were judged by a qualitative measure that was not specified in the test/QA plan. If the result from the test kit agreed within 25% of the reference result, the measurement was considered accurate; if it did not, the measurement was considered not to be accurate. The percentage of accurate measurements was determined for each of the three types of water samples as calculated from the following equation: Y A = — x 100 (4) T where A is the percent of accurate measurements, 7is the number of measurements within the ±25% criterion, and T is the total number of measurements. The criterion of 25% for agreement was based on the measurement resolution of the several portable arsenic analyzers tested and on scientific judgment of the required degree of accuracy for these analyzers. Readings below the B a 14 ------- detection limit (e.g., <10 ppb) were judged to be in agreement with the reference result if the reference value was in the specified "less than" range. 5.2 Precision When possible, the standard deviation (S) of the results for the replicate samples was calculated and used as a measure of Quick™ test kit precision at each concentration. S = k=l 1/2 (5) where n is the number of replicate samples, Ck is the concentration measured for the kth sample, and C is the average concentration of the replicate samples. The instrumental precision at each concentration was reported in terms of the relative standard deviation (RSD), e.g., C RSD x 100 (6) 5.3 Linearity Linearity was assessed by linear regression of Quick™ test kit results against the reference results, with linearity characterized by the slope, intercept, and correlation coefficient (r). Linearity was tested using PT samples over the range 1 to 100 ppb of arsenic. The color chart for the Quick™ test kit has a range of concentration from 5 to 500 ppb. If the concentration of arsenic for any sample is greater than 500 ppb, a smaller sample size can be used to extend the linearity beyond 500 ppb. 5.4 Method Detection Limit The MDL for the Quick™ test kit was assessed from the seven replicate analyses of a fortified sample with an analyte concentration of 25 ppb, i.e., five times the manufacturer's estimated detection limit of 5 ppb. The MDL was calculated from the following equation: MDL = t x S (7) where t (= 3.14) is the Student's t value for a 99% confidence level with n = 7, and S is the standard deviation of the replicate samples.(4) 15 ------- 5.5 Matrix Interference Effects The effect of interfering matrix species on the response of the Quick™ test kit to arsenic is typically calculated as the ratio of the difference in analytical response to the concentration of interfering species. For example, if adding 500 ppb of an interfering species results in a difference of 10 ppb in the analytical result, the relative sensitivity of the test kit to that interferant would be calculated as 10 ppb/500 ppb = 2%. In this test, three interfering species were added to the samples, all at either low or high concentrations (Section 3.3.2). Thus, it is not possible to determine which of these compounds would be responsible for any observed interferences. Only qualitative observations could be made assessing whether there was a positive or negative effect due to matrix interferences. 5.6 Operator Bias To assess operator bias for the Quick™ test kit, in all tests the results obtained from each operator were compiled independently and subsequently compared. However, because of the semi- quantitative nature of the test kit data and the inability of the operators to independently use the same test kits, quantitative assessments of operator bias could not be made. Qualitative observations were made concerning the results from the two operators. 5.7 Rate of False Positives/False Negatives The rates of false positives and false negatives of the Quick™ test kit were assessed relative to the 10-ppb target arsenic level. A false positive result is defined as any result reported to be equal to or greater than the guidance level (10 ppb) and greater than 125% of the reference value, when the reference value is less than that guidance level. Similarly, a false negative result is defined as any result reported below the guidance level and less than 75% of the reference value, when the reference value is greater than that guidance level. The rates of false positives and false negatives were expressed as a percentage of total samples analyzed for each type of sample. 16 ------- Chapter 6 Test Results The results of the verification test of the Quick™ test kits are presented in this section. 6.1 Accuracy Tables 6-la-c present the measured arsenic results from analysis of the PT, drinking water, and FW samples, respectively. Both reference analyses and Quick™ test kit results are shown in the tables, and Quick™ test kit results are shown for both the technical and non-technical operators. Some Quick™ test kit results could not be distinguished from blank sample results and were assigned a value of <5 ppb. The field spike results indicate apparent inconsistencies in some of the spike concentrations. The WW LFMf and LBC-4 LFMf samples apparently were not spiked in the field and the TW LFMf sample may have been spiked twice. However, these spiking errors have no effect on the usefulness of the data. Tables 6-2a-c show the percent accuracy of the Quick™ test kit results. Shown in the second and third columns in each of Tables 6-2a-c are the percent bias values determined according to Equation 3, in Section 5.1. Bias was not calculated for values reported as <5 ppb. The percent bias values that are shown in Tables 6-2a-c range from 8 to 83% for the non-technical operator and 8 to 84% for the technical operator for the PT samples, 8 to 92% for the non-technical operator and 8 to 54% for the technical operator for the drinking water samples, and 2 to 320% for both the non-technical operator and for the technical operator for the FW samples. In general, the larger bias values were associated with lower arsenic concentrations. In addition to the quantitative bias results, the qualitative accuracy was compared using Equation 4 in Section 5.1. The fourth and fifth columns in Tables 6-2a-c show the assignment of each Quick™ test kit result, in terms of whether that result fell within 25% of the reference value, or within a corresponding "less-than" range. The results of this qualitative evaluation of accuracy are shown in Table 6-3, which lists the overall percent of results meeting the qualitative accuracy criteria for each operator and sample type. Table 6-3 shows that the qualitative accuracy for the Quick™ test kit for the PT samples was 71% for the non-technical operator and 55% for the technical operator. The qualitative accuracy for the drinking water samples was 57% for the non- technical operator and 52% for the technical operator. The qualitative accuracy for the FW 17 ------- Table 6-la. Results from Laboratory Performance Test Sample Analyses Non-Technical Technical Reference Method3 Sample Arsenic (ppb) Arsenic (ppb) Arsenic (ppb) Laboratory RB <5 <5 <0.1 Laboratory RB NA <5 <0.1 Laboratory RB NA <5 <0.1 Laboratory RB NA <5 <0.1 Laboratory RB NA <5 <0.1 Laboratory RB NA <5 <0.1 QCS 10 20 10.9 QCS 10 20 10.9 QCS 10 20 10.9 QCS NA 20 10.9 QCS NA 20 10.9 PT1-1 <5 <5 1.76 PT1-2 <5 <5 1.76 PT1-3 <5 <5 1.76 PT1-4 <5 <5 1.76 PT2-1 <5 5 3.97 PT2-2 <5 5 3.97 PT2-3 <5 5 3.97 PT2-4 <5 5 3.97 PT3-1 5 10 10.9 PT3-2 10 10 10.9 PT3-3 10 10 10.9 PT3-4 10 10 10.9 PT4-1 30 40 34.8 PT4-2 30 40 34.8 PT4-3 20 40 34.8 PT4-4 30 20 34.8 PT5-1 100 100 113 PT5-2 100 100 113 PT5-3 100 100 113 PT5-4 100 100 113 PT6-1 5 20 29.6 PT6-2 5 20 29.6 PT6-3 10 20 29.6 PT6-4 10 20 29.6 PT6-5 10 20 29.6 PT6-6 10 20 29.6 PT6-7 20 20 29.6 " Only one aliquot of each sample was analyzed by the reference method (except for the laboratory RB). Multiple aliquots of each sample were analyzed by Quick™ test kit. NA: Not analyzed. 18 ------- Table 6-lb. Results from Drinking Water Analyses Non-Technical Technical Reference Method3 Sample Arsenic (ppb) Arsenic (ppb) Arsenic (ppb) Laboratory RB <5 <5 <0.1 QCS 10 10 10.9 DW-1 <5 <5 0.87 DW-2 <5 <5 0.87 DW-3 <5 <5 0.87 DW-4 <5 <5 0.87 DWLFMp 5 5 10.6 Laboratory RB <5 <5 <0.1 QCS 10 10 10.9 WW-1 100 60 86.6 WW-2 60 60 86.6 WW-3 60 40 86.6 WW-4 60 60 86.6 WWLFMp 70 60 82.1 Laboratory RB <5 <5 <0.1 QCS 5 10 10.9 TW-1 10 40 26.0 TW-2 10 40 26.0 TW-3 10 40 26.0 TW-4 50 40 26.0 TWLFMp 40 60 50.8 " Only one aliquot of each sample was analyzed by the reference method. Multiple aliquots of each sample were analyzed by Quick™ test kit. 19 ------- Table 6-lc. Results from Freshwater Analyses Non-Technical Technical Reference Method a Sample Arsenic (ppb) Arsenic (ppb) Arsenic (ppb) Laboratory RB <5 <5 <0.1 QCS 10 10 9.33 SR-1 <5 <5 1.73 SR-2 <5 5 1.72 SR-2 Duplicate <5 5 1.71 SR-3 <5 <5 2.03 SR-4 <5 5 1.88 SR-1 LFMp 10 20 11.6 Laboratory RB <5 <5 <0.1 QCS 10 10 9.43 LC-1 <5 5 2.13 LC-2 <5 5 1.30 LC-3 <5 5 1.44 LC-4 <5 <5 1.37 LC-4 Duplicate <5 5 1.36 LC-3 LFMp 10 10 12.0 Laboratory RB <5 <5 <0.1 QCS 10 10 9.81 LBC-1 <5 5 2.48 LBC-2 <5 5 2.60 LBC-3 <5 <5 2.14 LBC-3 Duplicate <5 <5 2.26 LBC-4 <5 <5 2.54 LBC-4 LFM, 10 10 2.38 20 ------- Table 6-2a. Accuracy of the Quick™ Test Kit with Laboratory Performance Test Samples Sample Bias" Non-Technical Bias3 Technical Within Range (Y/N)b Non-Technical Within Range (Y/N)b Technical Laboratory RB c - Y Y Laboratory RB NA - Y Laboratory RB NA - Y Laboratory RB NA - Y Laboratory RB NA - Y Laboratory RB NA - Y QCS 8% 84% Y N QCS 8% 84% Y N QCS 8% 84% Y N QCS NA 84% N QCS NA 84% N PT1-1 - - Y Y PT1-2 - - Y Y PT1-3 - - Y Y PT1-4 - - Y Y PT2-1 - 26% Y N PT2-2 - 26% Y N PT2-3 - 26% Y N PT2-4 - 26% Y N PT3-1 54% 8% N Y PT3-2 8% 8% Y Y PT3-3 8% 8% Y Y PT3-4 8% 8% Y Y PT4-1 14% 15% Y Y PT4-2 14% 15% Y Y PT4-3 43% 15% N Y PT4-4 14% 43% Y N PT5-1 12% 12% Y Y PT5-2 12% 12% Y Y PT5-3 12% 12% Y Y PT5-4 12% 12% Y Y PT6-1 83% 32% N N PT6-2 83% 32% N N PT6-3 66% 32% N N PT6-4 66% 32% N N PT6-5 66% 32% N N PT6-6 66% 32% N N PT6-7 32% 32% N N s Percent bias calculated according to Equation 3, Section 5.1. b Y = result within ±25% of reference, or reference value within < range; N = result not within ±25% of reference, or reference value not within < range. c Non-detect, no calculation of bias can be made. NA: not analyzed. 21 ------- Table 6-2b. Accuracy of the Quick™ Test Kit with Drinking Water Samples Sample Bias3 Non-Technical Bias3 Technical Within Range (Y/N)b Non-Technical Within Range (Y/N)b Technical Laboratory RB _C - Y Y QCS 8% 8% Y Y DW-1 - - Y Y DW-2 - - Y Y DW-3 - - Y Y DW-4 - - Y Y DW LFM, 53% 53% N N Laboratory RB - - Y Y QCS 8% 8% Y Y WW-1 15% 31% Y N WW-2 31% 31% N N WW-3 31% 54% N N WW-4 31% 31% N N WWLFMp 15% 27% Y N Laboratory RB - - Y Y QCS 54% 8% N Y TW-1 62% 54% N N TW-2 62% 54% N N TW-3 62% 54% N N TW-4 92% 54% N N TWLFMp 21% 18% Y Y s Percent bias calculated according to Equation 3, Section 5.1. b Y = result within ±25% of reference, or reference value within < range; N = result not within ±25% of reference, or reference value not within < range. c Non-detect, no calculation of bias can be made. 22 ------- Table 6-2c. Accuracy of the Quick™ Test Kit with Freshwater Samples Sample Bias" Non-Technical Bias3 Technical Within Range (Y/N)b Non-Technical Within Range (Y/N)b Technical Laboratory RB _C - Y Y QCS 7% 7% Y Y SR-1 - - Y Y SR-2 - 191% Y N SR-2 Duplicate - 192% Y N SR-3 - - Y Y SR-4 - 166% Y N SR-1 LFMp 14% 72% Y N Laboratory RB - - Y Y QCS 6% 6% Y Y LC-1 - 135% Y N LC-2 - 285% Y N LC-3 - 247% Y N LC-4 - - Y Y LC-4 Duplicate - 268% Y N LC3 LFMp 17% 17% Y Y Laboratory RB - - Y Y QCS 2% 2% Y Y LBC-1 - 102% Y N LBC-2 - 92% Y N LBC-3 - - Y Y LBC-3 Duplicate - - Y Y LBC-4 - - Y Y LBC-4 LFM, 320% 320% N N 1 Percent bias calculated according to Equation 3, Section 5.1. b Y = result within ±25% of reference, or reference value within < range; N = result not within ±25% of reference, or reference value not within < range. c No calculation of bias can be made. Table 6-3. Summary of Qualitative Accuracy Results Percent Accurate Percent Accurate Within 25% Within 25% (Non-Technical Operator) (Technical Operator) Laboratory performance test samples 71% 55% Drinking water samples 57% 52% Freshwater samples 96% 54% 23 ------- samples was 96% for the non-technical operator and 54% for the technical operator. Many of the Quick™ results judged qualitatively accurate were the result of sample concentrations below the manufacturer's estimated detection limit of 5 ppb. For the 25 samples in Tables 6-la and b with reference arsenic values between 26 and 113 ppb, the qualitative accuracy was 40% or less with both operators. 6.2 Precision Tables 6-4a and b, respectively, show the data used to evaluate the RSD of the Quick™ test kit results for the replicate laboratory PT and drinking water samples, along with the percent RSD value for each set of replicate analysis. The percent RSD was determined according to Equation 6 in Section 5.2. Percent RSD was not calculated if all of the results for a set of replicates were <5 ppb. These data sets illustrate the consistency in the Quick™ test kit replicate analyses. Seven of the 14 replicate sets for the PT and QCS samples (Table 6-4a) showed an RSD of 0%. The results for three of the replicate sets were <5 ppb. The remaining replicate sets for the non-technical operator had an RSD ranging from 29 to 50%, and the remaining replicate set for the technical operator had an RSD of 29%. For the drinking water samples (Table 6-4b), all results for two of the replicate sets were <5 ppb. The remaining sets had an RSD of 29 to 100% for the non- technical operator and 0 to 18% for the technical operator. 6.3 Linearity The linearity of the Quick™ test kit readings was assessed by means of a linear regression of the Quick™ test kit results against the reference method results, using the 27 data points from the PT samples (Table 6-1 a). In this regression, results reported as below detection limit by the Quick™ test kit were assigned a value of half the detection limit (2.5 ppb). Figure 6-1 shows a scatter plot of the Quick™ test kit data from both non-technical and the technical operators versus the reference method results. The one-to-one line is also shown in Figure 6-1. A linear regression of the data in Figure 6-1 gives the following regression equations: with the Quick™ test kit for the non-technical operator, ppb = 0.90 (±0.086) x (reference, ppb) - 5.2 (±4.1) ppb, with r = 0.974, and with the Quick™ test kit for the technical operator, ppb = 0.88 (±0.056) x (reference, ppb) - 0.45 (±2.7) ppb, with r = 0.988 where the values in parentheses represent the 95% confidence interval of the slope and intercept. Both regressions show slopes that are significantly different from 1.0. 24 ------- Table 6-4a. Precision Results for Quick™ Test Kit from Laboratory Performance Test Samples Reference Non-Technical3 Technical Concentration (ppb) Arsenic (ppb) Arsenic (ppb) QCS 10.9 10 20 QCS 10 20 QCS 10 20 QCS 20 QCS 20 %RSD 0 0 PT1-1 1.76 <5 <5 PT1-2 <5 <5 PT1-3 <5 <5 PT1-4 <5 <5 %RSD _b _b PT2-1 3.97 <5 5 PT2-2 <5 5 PT2-3 <5 5 PT2-4 <5 5 %RSD _b 0 PT3-1 10.9 5 10 PT3-2 10 10 PT3-3 10 10 PT3-4 10 10 %RSD 29 0 PT4-1 34.8 30 40 PT4-2 30 40 PT4-3 20 40 PT4-4 30 20 %RSD 29 29 PT5-1 113 100 100 PT5-2 100 100 PT5-3 100 100 PT5-4 100 100 %RSD 0 0 PT6-1 29.6 5 20 PT6-2 5 20 PT6-3 10 20 PT6-4 10 20 PT6-5 10 20 PT6-6 10 20 PT6-7 20 20 %RSD 50 0 s For the purpose of calculating %RSD, all "less than" values are considered zero. b No %RSD could be calculated. 25 ------- Table 6-4b. Precision Results for Quick™ Test Kit from Drinking Water Samples Reference Non-Technical3 Technical3 Concentration (ppb) Arsenic (ppb) Arsenic (ppb) DW-1 0.87 <5 <5 DW-2 <5 <5 DW-3 <5 <5 DW-4 <5 <5 % RSD _b _b WW-1 86.6 100 60 WW-2 60 60 WW-3 60 40 WW-4 60 60 % RSD 29 18 TW-1 26.0 10 40 TW-2 10 40 TW-3 10 40 TW-4 50 40 %RSD 100 0 " For the purpose of calculating %RSD, all "less than" values are considered zero. b No %RSD could be calculated. « Non-Technical Arsenic, ppb ¦ Technical Arsenic, ppb Linear (Non-Technical Arsenic, ppb) Linear (Technical Arsenic, ppb) Linear (1:1 Line) 140 120 100 ¦Q Q. Q. O *3 a o 20 40 60 80 100 120 140 Reference Arsenic, ppb Figure 6-1. Comparison of Quick™ Test Kit to Reference Method Results from PT Samples 26 ------- 6.4 Method Detection Limit The manufacturer's estimated detection limit for the Quick™ test kit is 5 ppb. An attempt was made to determine the MDL by analyzing seven replicate samples at approximately 25 ppb (PT6 samples, Table 6-la). The Quick™ results for both operators were all less than the reference value, but in particular the technical operator's results were all identical (20 ppb), providing no variation with which to quantitatively assess the MDL.(4) The non-technical operator reported arsenic between 5 and 20 ppb. Since the Quick ™ test kit is only semi-quantitative, no MDL was calculated from these data. Qualitative indication of the Quick™ test kit MDL can be obtained from the results of the PT2 and PT3 samples (Table 6-la) of concentrations 3.97 and 10.9 ppb, respectively. With the 3.97-ppb samples, the non-technical operator reported results of <5 ppb, whereas the technical operator reported results of 5 ppb. With the 10.9-ppb samples, all Quick™ results were 10 ppb except for one result of 5 ppb with the non-technical operator. 6.5 Matrix Interference Effects Tables 6-5a and b show the analytical results from laboratory performance test samples containing about 10.5 ppb arsenic, with low and high levels of interference, respectively. The Quick™ test kit produced positive readings on all the matrix interference samples with both operators, with a small increase in readings with the higher interference levels. For example, the non-technical operator reported 10 ppb in five of eight analyses of the LI samples, with three readings of 5 ppb, and reported 10 ppb in seven of eight analyses, with only one reading of 5 ppb. for the HI samples. Similarly, the technical operator reported six of eight values at 10 ppb and two at 5 ppb with the LI samples, but five of eight at 10 ppb and three at 20 ppb with the HI samples. These results indicate a minor tendency toward higher readings (3 ppb on average) from the Quick™ test kit at the higher interference levels. Because of the study design, it was not possible to determine which ion was responsible for the observed result. Table 6-5a. Results from Laboratory Performance Test Samples with Low-Level Interferences Non-Technical Arsenic (ppb) Technical Arsenic (ppb) LI-1 10 5 LI-2 5 10 LI-3 5 10 LI-4 10 10 LI-5 10 10 LI-6 10 5 LI-7 5 10 LI-8 10 10 " Only one aliquot of LI solution was analyzed by the reference method. Eight aliquots of LI solution were analyzed by Quick™ test kits. 27 ------- Table 6-5b. Results from Laboratory Performance Test Samples with High-Level Interferences Non-Technical Arsenic (ppb) Technical Arsenic (ppb) HI-1 10 10 HI-2 5 10 HI-3 10 10 HI-4 10 20 HI-5 10 10 HI-6 10 20 HI-7 10 10 HI-8 10 20 " Only one aliquot of HI solution was analyzed by the reference method. Eight aliquots of HI solution were analyzed by Quick™ test kits. 6.6 Operator Bias The effect of operator skill level does not appear to be a major factor with the Quick™ test kit. The non-technical operator had a higher percentage of accurate results, although the greater frequency of non-detects with the non-technical operator played a part in that outcome. On the other hand, the technical operator had fewer false positive and negative results (see Section 6.7). 6.7 Rate of False Positives/False Negatives Tables 6-6 and 6-7, respectively, show the data and results for the rates of false positives and false negatives obtained from the Quick™ test kit. All PT and environmental samples (Table 3-1) were considered for this evaluation. Table 6-6 shows that 24 samples had reference arsenic concentrations less than the target decision level of 10 ppb. Of the samples tested by the non-technical operator, in only one sample did the Quick™ test kit results indicate a concentration of 10 ppb or higher. The result was a false positive rate of 4% relative to the 10 ppb value. The samples tested by the technical operator had a false positive rate of 0%, with no Quick™ test kit results at or above the 10-ppb decision level. Table 6-7 shows that 43 samples had reference arsenic concentrations greater than the target decision level of 10 ppb. In seven of the 43 samples, the analyte was detected at a level less than 10 ppb by the non-technical operator (i.e., a false negative rate of 16%). The technical operator reported only two such results, for a false negative rate of 5%. 28 ------- Table 6-6 Rate of False Positives from Quick™ Test Kit Non-Technical Technical Reference Non-Technical Technical Arsenic Arsenic Method False Positive False Positive (PPb) (PPb) Arsenic (ppb) (Y/N) (Y/N) PT1-1 <5 <5 1.76 N N PT1-2 <5 <5 1.76 N N PT1-3 <5 <5 1.76 N N PT1-4 <5 <5 1.76 N N PT2-1 <5 5 3.97 N N PT2-2 <5 5 3.97 N N PT2-3 <5 5 3.97 N N PT2-4 <5 5 3.97 N N DW-1 <5 <5 0.87 N N DW-2 <5 <5 0.87 N N DW-3 <5 <5 0.87 N N DW-4 <5 <5 0.87 N N SR-1 <5 <5 1.73 N N SR-2 <5 5 1.72 N N SR-3 <5 <5 2.03 N N SR-4 <5 5 1.88 N N LC-1 <5 5 2.13 N N LC-2 <5 5 1.3 N N LC-3 <5 5 1.44 N N LC-4 <5 <5 1.37 N N LBC-1 <5 <5 2.48 N N LBC-2 <5 <5 2.6 N N LBC-3 <5 <5 2.14 N N LBC-4 10 <5 2.54 Y N Total number of applicable samples 24 24 Total false positive 1 0 Percent false positive 4% 0% Y = yes N = no 29 ------- Table 6-7 Rate of False Negatives from Quick™ Test Kit Non-Technical Arsenic (PPb) Technical Arsenic (PPb) Reference Method Arsenic (PPb) Non-Technical False Negative (Y/N) Technical False Negative (Y/N) PT3-1 5 10 10.9 Y N PT3-2 10 10 10.9 N N PT3-3 10 10 10.9 N N PT3-4 10 10 10.9 N N PT4-1 30 40 34.8 N N PT4-2 30 40 34.8 N N PT4-3 20 40 34.8 N N PT4-4 30 20 34.8 N N PT5-1 100 100 113 N N PT5-2 100 100 113 N N PT5-3 100 100 113 N N PT5-4 100 100 113 N N PT6-1 5 20 29.6 Y N PT6-2 5 20 29.6 Y N PT6-3 10 20 29.6 N N PT6-4 10 20 29.6 N N PT6-5 10 20 29.6 N N PT6-6 10 20 29.6 N N PT6-7 20 20 29.6 N N LI-1 10 5 10.6 N Y LI-2 5 10 10.6 Y N LI-3 5 10 10.6 Y N LI-4 10 10 10.6 N N LI-5 10 10 10.6 N N LI-6 10 5 10.6 N Y LI-7 5 10 10.6 Y N LI-8 10 10 10.6 N N HI-1 10 10 10.4 N N HI-2 5 10 10.4 Y N HI-3 10 10 10.4 N N HI-4 10 20 10.4 N N HI-5 10 10 10.4 N N HI-6 10 20 10.4 N N HI-7 10 10 10.4 N N HI-8 10 20 10.4 N N WW-1 100 60 86.6 N N WW-2 60 60 86.6 N N WW-3 60 40 86.6 N N WW-4 60 60 86.6 N N TW-1 10 40 26.0 N N TW-2 10 40 26.0 N N TW-3 10 40 26.0 N N TW-4 50 40 26.0 N N Total number of applicable samples Total false negative Percent false negative 43 7 16% 43 2 5% Y = yes N = no 30 ------- 6.8 Other Factors The operators felt that the Quick™ test kit was easy to use and free of maintenance. The kit is lightweight, easy to transport by car, and can be carried through fields and wooded areas. The reaction bottles, however, are tall, narrow, and lightweight, making them susceptible to falling over with a moderate breeze. The Quick™ test kit allows two samples to be analyzed simultaneously. The total reaction time is less than 15 minutes. The reagents are ready to use and do not require preparation. Three sizes of scoops are included in the Quick™ test kit, making it easy to add the three reagents to the sample. However, the narrow top of the reaction bottles makes it difficult to add the reagents. The reagent bottles can be cleaned and reused. However, the operators experienced some difficulty with the reagents sticking to the reaction vessel. This can be remedied by washing in a dilute acid solution. This kit requires no liquids or concentrated acids, making it safe and easy to carry in the field. The solid reagents contain no toxic materials. 6.8.1 Costs The Quick™ test kit is available in three sizes. The smallest kit costs $12.99 and is capable of analyzing two samples. The 50-sample test kit costs $79.99. The large kit, capable of analyzing 100 samples, sells for $139.99. 6.8.2 Data Completeness All portions of the verification test were completed, and all data that were to be recorded were successfully acquired. The non-technical operator analyzed only one of the three required laboratory reagent blanks, otherwise data completeness was 100%. 31 ------- Chapter 7 Performance Summary An assessment of quantitative accuracy showed that percent bias values ranged from 8 to 83% for the non-technical operator and 8 to 84% for the technical operator for the PT samples. The percent bias ranged from 8 to 92% for the non-technical operator and 8 to 54% for the technical operator for the drinking water samples. For the FW samples, the percent bias ranged from 2 to 320% for both the non-technical and technical operators. An additional qualitative criterion for accuracy was the percentage of samples for which the Quick™ test kit result was within 25% of the reference result or within a corresponding "less than" range. By this criterion, the Quick™ test kit yielded a qualitative accuracy for the PT samples of 71% for the non-technical operator and 55%) for the technical operator. The qualitative accuracy for the drinking water samples was 57%) for the non-technical operator and 52% for the technical operator. The qualitative accuracy for the freshwater samples was 96% for the non-technical operator and 54% for the technical operator. Percent RSD data illustrate consistency in the Quick™ test kit replicate analyses. Seven of the 14 replicate sets for the PT samples showed an RSD of 0% (i.e., all replicate results were identical). The remaining replicate sets for the non-technical operator had an RSD ranging from 29 to 50%), and the remaining replicate set for the technical operator had an RSD of 29%. For the drinking water samples, the RSDs for the non-technical operator ranged from 29 to 100%), and the RSDs for the technical operator ranged from 0 to 18%). The linearity of response of the Quick™ test kit was assessed using the PT samples containing 2 to 112 ppb arsenic. The linear regression for the Quick™ test kit results for the non-technical operator was ppb = 0.90 (±0.086) x (reference, ppb) - 5.2 (±4.1) ppb, with a correlation coefficient (r) of 0.974. The corresponding equation for the results for the technical operator was ppb = 0.88 (±0.056) x (reference, ppb) - 0.45 (±2.7) ppb, with a correlation coefficient (r) of 0.988. The manufacturer's estimated detection limit for the Quick™ test kit is 5 ppb. Seven replicate samples of 25-ppb arsenic produced Quick™ readings of 5 to 20 ppb with the non-technical operator and seven identical readings of 20 ppb with the technical operator. No MDL was calculated quantitatively from these data. The Quick™ test kit showed a minor tendency toward higher readings (3 ppb on average) with higher levels of sodium chloride, iron, sulfide, and acidity. Because of the study design, it was not possible to determine which ion was responsible for the observed result. 32 ------- The operator skill level does not appear to be a major factor determining Quick™ test kit results. The rates of false positives and false negatives for the Quick™ test kit were assessed relative to the reference method using 10 ppb of arsenic as the decision level. The rate of false positives for the Quick™ test kit was 4% for the non-technical operator and 0% for the technical operator. The rate of false negatives was 16% for the non-technical operator and 5% for the technical operator. The Quick™ test kit is available in three sizes. The smallest is capable of analyzing two samples and costs $12.99. The 50-sample test kit costs $79.99. The large kit, capable of analyzing 100 samples, sells for $139.99. The test kit allows two samples to be analyzed simultaneously. The total reaction time is less than 15 minutes. The reagents are ready to use and do not require preparation. Three scoop sizes are included in the Quick™ test kit, making addition of the reagents simple, but the size and shape of the reaction vessels limit the ease of use of the test kit. 33 ------- Chapter 8 References 1. Test/QA Plan for Verification of Portable Analyzers, Battelle, Columbus, Ohio, Version 2.0. 2. U.S. EPA Method 200.8, Determination of Trace Elements in Waters and Wastes by Inductively Coupled Plasma Mass Spectrometry, Revision 5.5, April 1991. 3. Quality Management Plan (QMP) for the ETV Advanced Monitoring Systems Pilot, Version 2.0, U.S. EPA Environmental Technology Verification Program, Battelle, Columbus, Ohio, October 2000. 4. U.S. Code of Federal Regulations, Title 40, Part 136, Appendix B. 34 ------- |